JPH10301675A - Multimodal interface device and multimodal interface method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an interface which proceeds interaction in the natural foot to a person like interaction between persons. SOLUTION: This device is provided with a personification image presenting means 103 which supplies a personified agent image, a detecting means 101 which detects a user's object to be closely observed, other media inputting means 102 which acquires at least more than one input information among voice input information, operation information and image input information of the user and a controlling means 107 which receives input information from the other media inputting means, controls the state of a recognition operation, recognizes any part of an agent image that presents the user's object to be closely observed by the means 103 based on closely observing object information that is acquired by the means 101 and receives and selects an input from the other media input recognizing means in accordance with the recognition result.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multimodal interface which is optimally applied to a multimodal dialogue device which interacts with a user through at least one input or output of natural language information, voice information, visual information and operation information. The present invention relates to an apparatus and a multimodal interface method.

[0002]

2. Description of the Related Art In recent years, in a computer system including a personal computer, multimedia information such as voice information and image information has been input and output in addition to the conventional input by a keyboard and a mouse and the output of characters and image information by a display. It is becoming possible to do so.

[0003] In addition to these circumstances, due to the progress of natural language analysis, natural language generation, speech recognition, speech synthesis technology, and dialog processing technology, there is a demand for a speech dialogue system for interacting with a user for speech input / output. "TOSBURG-II" which is a dialogue system that can be used by voice input by free speech (IEEE Transactions, Vol. J77-D-II, No. 8, pp1)
417-1428, 1994), and various speech dialogue systems have been researched and developed and published.

Further, in addition to such voice input / output, for example, visual information input using a camera is utilized, or a touch panel, a pen, a tablet, a data glove or a foot switch, a human sensor, a head, or the like. Mount display, force display
For example, there is an increasing demand for a multi-modal interaction system that interacts with a user using information that can be exchanged with the user through various input / output devices.

[0005] That is, by making full use of a multi-modal interface using such various input / output devices, various information can be transmitted and received. Therefore, a user can have a natural conversation with the system. It is attracting attention because it can be one powerful method for realizing a natural and easy-to-use human interface.

[0006] In other words, even in the dialogue between humans, communication is not performed using only one medium (channel) such as voice, but non-verbal sent and received through various media such as gestures, hand gestures and facial expressions. By interacting with messages, natural and smooth interaction is performed (“Intelligent Multimedia”).
Interfaces ", Maybury MT, E
ds. , The AAAI Press / The M
IT Press, 1993).

Considering this, in order to realize a natural and easy-to-use human interface, in addition to voice input / output, visual information input using a camera, touch panel, pen, tablet, data glove, Expectations are growing for the realization and application of dialogue using linguistic and non-verbal messages using various input / output media such as foot switches, interpersonal sensors, head mounted displays, and force displays.

However, there are the following situations (i) and (ii). [Background (i)] Conventionally, due to the problem of low analysis accuracy of input from each medium and the fact that the properties of each input / output medium have not been sufficiently clarified, it can be newly used. A multi-modal interface that efficiently uses each input or output media or a plurality of input / output media, is highly efficient, effective, and reduces the burden on the user has not been realized.

That is, since the analysis accuracy of the input from each media is insufficient, for example, erroneous recognition due to ambient noise in the voice input may occur, or in the gesture input recognition process, the input device may instantly recognize the input. A malfunction occurs, for example, due to a failure to cut out a signal portion intended by the user as an input message from the obtained signals, and the like, resulting in a burden on the user.

[0010] Further, media used not only when a user uses a computer or the like as a current operation target but also when the user speaks to other nearby people, such as voice input and gesture input, is used. In the interface device used, even if the user talks to another person beside him / her or shows a gesture instead of the interface device, it is determined that the interface device is an input to itself. Performs recognition processing and the like, resulting in malfunction. The user must take measures to cancel the malfunction and restore the effects of the malfunction, and the user must pay constant attention to avoid the malfunction. large.

In addition, even in a situation where a decision is not necessary, the processing of the input signal is continuously performed. Therefore, depending on the processing load, the execution speed and the utilization efficiency of other services related to the apparatus being used. Have problems such as a decrease in

In order to solve this problem, a method of changing the mode by a special operation, such as pressing a button or selecting a menu, when inputting a voice or a gesture, for example, is also adopted. However, such a special operation is an operation that does not exist when it is a conversation between humans, so that not only an unnatural interface, but also complicated for the user or depending on the type of operation, The necessity of training for learning has increased the burden on users unnecessarily.

In addition, for example, in a case where the input / output of voice input is switched by a button operation, the advantage of the voice media cannot be utilized. In other words, voice media input allows you to communicate using only the mouth. For example, even if there is work done by hand, it can be used simultaneously without disturbing it. However, in the case where a mechanism is required to switch whether or not voice input is possible with a button operation, such an inherent advantage of the voice media cannot be utilized.

In addition, the information to be presented, such as audio output, moving image information, characters and image information over a plurality of screens, may disappear immediately or may change every moment. In many cases, it is necessary to present information to the user using media, but in such a case, if the user does not pay attention to the information, the user can receive some or all of the presented information There was a problem that there was no case.

Conventionally, when presenting information to a user using a temporary medium, the user is presented with information for each amount that can be received at a time, and the user performs a confirmation operation through some special operation. Therefore, there is a method of presenting the next information to be continued, but in this case, the burden on the user increases due to the confirmation operation, and if the user is not accustomed to the operation, the operation is confused and the operation efficiency of the system is reduced. The problem of getting worse remains.

Further, in the conventional multimodal interface, it is said that, because of the undeveloped use technology, it plays an important role in communication between humans, such as gaze matching (eye contact), gaze position, gesture,
Gestures such as gestures and non-verbal messages such as facial expressions cannot be used effectively.

[Background (ii)] As another viewpoint, when looking at a conventional actual multimodal interface, voice input, touch sensor input, image input, distance sensor input, and the like are handled. Consider

In the case of voice input, for example, it is assumed that voice input has been made by a user, and in that case, the input voice waveform signal is converted, for example, from analog to digital, and the power per unit time is calculated. A section is detected and analyzed by, for example, a method such as FFT (Fast Fourier Transform) and, for example, is matched with a speech recognition dictionary, which is a standard pattern prepared in advance, by using a method such as HMM (Hidden Markov Model). , Etc., to estimate the content of the utterance, and perform processing according to the result.

When a pointing gesture is input by a user through a contact-type input device such as a touch sensor, coordinate information, time-series information, or input pressure, which is output information of the touch sensor, is input. A process for identifying the pointed-to destination is performed using information or an input time interval.

In the case of using an image, one or more cameras are used to photograph, for example, a user's hand, and the observed shape or operation is described, for example.
“Uncalibrated Stereo Visi
on With Pointing for a Ma
n-Machine Interface (R. Cip
olla, et. al. , Proceedings o
f MVA'94, IAPR Workshop on
Machine Vision Applicati
on, pp. 163-166, 1994. ), It is possible to input a pointing target in the real world or a pointing target on a display screen, which is pointed out by the user.

A distance sensor, in this case, for example, a distance sensor using infrared rays or the like is used. The distance sensor is used to analyze the position, shape, or movement of the user's hand in the same manner as in the case of an image. By analyzing and recognizing, the referents in the real world indicated by the user,
Alternatively, it is possible to input a pointing gesture to an instruction target or the like on the display screen.

In addition, as input means, a user's hand
For example, by mounting a magnetic sensor, an acceleration sensor, or the like, the spatial position, movement, or shape of the hand can be input, or virtual reality (VR = Virtual Real) can be input.
(it) The data pointed out by the user by analyzing the movement, position, or shape of the user's hand or body by wearing the data glove or data suit developed for the technology. It is possible to adopt input of an instruction target all over the world or an instruction target on a display screen.

However, in the conventional pointing gesture input method, the interface method implemented using, for example, a touch sensor has a problem that the pointing gesture cannot be performed from a remote position or without touching the device. there were. Furthermore, there is a problem that an interface method realized by a user wearing a data glove, a magnetic sensor, an acceleration sensor, or the like cannot be used unless a device is worn.

In the interface method implemented by detecting the shape, position, or movement of the user's hand or the like using a camera or the like, sufficient accuracy cannot be obtained. It is difficult to properly extract only gestures intended to be gestures, and as a result, hand movements, shapes, and the like that are not intended to be input by the user or gestures are mistaken for gesture input. In some cases, a gesture intended to be recognized or input by the user cannot be correctly extracted if the input is a gesture input.

As a result, for example, it is necessary to correct the influence of a malfunction caused by erroneous recognition, or a gesture input intentionally performed by a user is not correctly input to the system. There is a problem that the user needs to input again, which increases the burden on the user.

Also, since the gesture input by the user is obtained at the time when the analysis is completed, at the time when the user starts the gesture intended to be input or at the time when the input is being performed, the gesture is performed by the system. I don't know if the input was extracted correctly.

For this reason, for example, a gesture that the user is currently inputting is actually performed because the start time of the gesture is wrong, or it is not possible to correctly detect that the gesture input is being performed by the user. Is not correctly extracted by the system, and as a result, erroneous recognition is caused or the user has to input again, thereby increasing the burden on the user.

Alternatively, a malfunction may occur when the system erroneously recognizes that a gesture has been started even though the user has not performed a gesture input.
The effect must be corrected.

In a gesture recognition method using a contact-type input device such as a touch sensor or a tablet, for example, a user points to a part of the contact-type input device itself. There is a problem that it is not possible to input pointing gestures to refer to real world places and things, etc.
For example, a pointing gesture input recognition method that uses a non-contact input method using a camera, an infrared sensor, an acceleration sensor, or the like can point to an object or a place in the real world, but the system uses There was a problem that there was no way to properly indicate the location, or which object or which part of it was received.

[0030]

As described above in the background (i), the conventional multi-modal interface has a low analysis accuracy for input information from each input / output medium, Because the characteristics of input / output media are not fully understood, various newly available input / output media or a plurality of input / output media can be effectively used to efficiently There is a problem that a multi-modal interface that reduces the burden is not realized.

That is, since the analysis accuracy of the input from each medium is insufficient, for example, in the occurrence of erroneous recognition due to ambient noise or the like in voice input or in the recognition processing of gesture input, it is obtained from the input device every moment. For example, if the user fails to extract the signal part intended as an input message from the
There was a problem that a malfunction occurred and the burden on users increased.

Further, media such as voices and gestures are important as a multi-modal interface. This media is used not only by a user as input to a computer or the like which is a current operation target, but also by, for example, surroundings. It is also used for dialogue with other people.

Therefore, in the interface device using such a medium, even when a user speaks or shows a gesture to a person next to the user instead of the interface device, the interface device can be used. If the input is erroneously determined to be an input to itself, recognition processing of the information is performed, and a malfunction may be caused. Therefore, the user must deal with the cancellation of the malfunction and recovery of the effects of the malfunction, and the user must pay constant attention to prevent such a malfunction. There has been a problem that the burden on the user increases when it disappears.

Also, in a multimodal device,
Even when information recognition processing is not required, input signals are monitored and processed continuously, and the processing load reduces the execution speed and utilization efficiency of other services related to the device being used. There was a problem.

In order to solve this problem, when inputting a voice or a gesture, the mode is changed by a special operation such as forcing the user to press a button or selecting a menu. Although such special operations may be used, such special operations are not inherent in human-to-human conversations, so an interface that requires such operations is only an unnatural interface for the user. In addition, there is a problem that the burden on the user is increased because the user feels troublesome and cumbersome or, depending on the type of operation, requires training for learning.

Also, the input by the voice media is originally
Since communication can be performed using only the mouth, there is an advantage that it is possible to use both at the same time without interrupting work done by hand,
For example, in a case where the input / output of voice input is switched by a button operation, there is a problem in that the inherent advantage of such a voice media is lost.

Further, for example, audio output, moving image information, text and image information over a plurality of screens, etc., may disappear as soon as the presentation information is presented or may change momentarily. When presenting information to a user using such a transient medium, the user receives some or all of the information presented if the user has not paid attention to the information. There was a problem that there was no case.

Conventionally, when information is presented to a user using a temporary medium, the user is presented with information for each amount that can be received at a time, and the user performs a confirmation operation through some special operation. Therefore, a method of presenting the next information to be continued may be used. However, such a method increases the burden on the user due to the confirmation operation,
There was a problem that the operation efficiency of the system deteriorated.

In the conventional multimodal interface, gestures such as eye-gaze matching (eye contact), gaze position, gesture, hand gesture, etc., which are said to play an important role in human-to-human communication due to inexperience in applied technology, Then, there is a problem that non-verbal messages such as facial expressions cannot be used effectively.

As described in the background (ii), in an actual input means for a multi-modal interface, in the case of inputting a pointing gesture, in an interface method using a contact-type input device, a remote location is used. There is a problem that the pointing gesture cannot be performed without touching the device or the device, and the wearing interface method cannot be used unless the device is worn.

In the interface method for performing gesture recognition remotely, sufficient accuracy cannot be obtained, so that the user may erroneously perform gestures such as hand movements, shapes, and the like that are not intended to be input as gestures. There has been a problem that a gesture that is erroneously recognized as an input or a gesture intended by the user to be input frequently cannot be correctly extracted as a gesture input.

Further, at the point in time when the user starts a gesture intended for input or during input, the system does not know whether or not the gesture input has been correctly extracted. There is a problem that the burden on the user is increased because the user has to cause an input or need to input again.

In the gesture recognition method using the contact-type input device, a pointing gesture for referring to a place or an object in the real world other than the contact-type input device itself cannot be input. On the other hand, in the method of recognizing pointing gesture input using a non-contact input method,
Although it is possible to indicate an object or place in the real world, there is a problem in that there is no way to appropriately indicate which place, which object, or which part thereof has been received by the system.

Further, as a problem of the conventional method induced by the above-described problem, for example, it is necessary to correct the influence of a malfunction, or it is necessary to input again, or a user makes an input. At this time, there is a problem that the user's burden increases due to anxiety and the like, because it is not known whether the current input is correctly input to the system.

Therefore, the object of the present invention is to solve the problem of the background (i) by firstly making it possible to use a plurality of types of input / output media efficiently and effectively. It is an object of the present invention to provide a multi-modal interface capable of reducing a burden on a user and enabling a natural conversation in a state close to communication between humans.

Further, a second object of the present invention is to provide a method for detecting a malfunction due to insufficient analysis accuracy of an input from each medium,
Or multi-modal that eliminates the burden on users due to malfunctions due to malfunctions due to ambient noise or failure to cut out the signal part intended by the user as an input message from signals obtained from the input device every time Provides an interface.

Third, an interface device using a medium, such as a voice or a gesture, which is used not only by a user as an input to a computer or the like as a current operation target but also used by a human for dialogue. Then, the user
Instead of using the interface device of the operating multi-modal system, for example, when speaking to another person beside you or showing a gesture, because the user is near the multi-modal system, The interface device of the multi-modal system will judge that it is an input to itself and cause a malfunction, but even in such a case, such a situation can be resolved and the cancellation operation accompanying the malfunction and the influence of the malfunction It is an object of the present invention to provide a multi-modal interface that can eliminate a burden on a user, including a measure for recovery and a load that the user must constantly pay attention to avoid malfunction.

Fourth, even in a situation where information identification of media input is originally unnecessary from the processing operation state of the system, input signal processing is continuously performed, so that interrupt processing is performed. The execution speed and utilization efficiency of other services related to the device being used by eliminating the processing load on media input in unnecessary situations to eliminate the adverse effect of delaying work during processing It is an object of the present invention to provide a multi-modal interface capable of suppressing a decrease in the number.

Fifth, when inputting voices and gestures, a special operation such as pressing a button or changing a mode by selecting a menu is not required. An object of the present invention is to provide a multi-modal interface that is natural, does not require any training for learning, and does not impose a burden on a user without any complexity.

Sixth, when audio media is used, for example, unnecessary operations such as switching between the input and output of audio by a button operation are completely eliminated, and necessary audio information is obtained. It is an object of the present invention to provide a multi-modal interface capable of performing such operations.

Seventh, it is an object of the present invention to provide a multimodal interface that allows a user to receive information in a form in which presentation is temporary, without overlooking the information.

Eighth, when presenting information by means of a transient medium, when the user is presented with the information divided into small amounts that can be received at one time, the burden on the user such as a special operation is reduced. An object of the present invention is to provide an interface capable of presenting information smoothly without imposing.

Ninth, it is said that it plays an important role in human-to-human communication, but it is difficult to effectively use the conventional multi-modal interface. It is an object of the present invention to provide an interface that can effectively use non-verbal messages such as gestures such as contact, gaze position, gesture, hand gesture, and facial expressions.

Another object of the present invention is to solve the problem of the background (ii), in which the user attaches the device to a position away from the system or without touching the device. Without pointing, it is possible to perform pointing gestures remotely and input instructions, and to eliminate false recognition and gesture extraction failures caused by insufficient accuracy of the gesture recognition method. To provide a multimodal interface device and a multimodal interface method. Also, at the point when the user starts the gesture intended to be input or during the input, it is not known whether the system has correctly extracted the gesture input, and as a result, misrecognition may occur. Another object of the present invention is to provide a multi-modal interface device and a multi-modal interface method capable of suppressing a user's burden caused by the user having to input again.

Further, in response to a pointing gesture input from a user for referring to a place or thing in the real world,
An object of the present invention is to provide a multimodal interface device and a multimodal interface method capable of appropriately displaying which place, which object, or which part thereof has been received.

Further, the problems of the conventional method induced by the above-mentioned problems, such as the correction of the influence of the malfunction, the burden on the user caused by the re-input, and the user's anxiety at the time of the input by the user. It is an object of the present invention to provide a multi-modal interface device and a multi-modal interface system which can eliminate the burden of the above.

Further, by using an interface apparatus and an interface method using an anthropomorphic interface, it is possible to generate an appropriate facial expression of an agent in consideration of the field of view of the user and the anthropomorphic agent from the anthropomorphic agent, and present it as feedback. An object of the present invention is to provide a multi-modal interface device and a multi-modal interface method that can be performed.

[0058]

In order to achieve the above object, the present invention is configured as follows. In order to solve the problem relating to the background (i), [1] First, a detection unit that detects a gaze target of a user, and among voice input information, operation input information, and image input information of the user, Control means for receiving at least one or more pieces of input information and controlling the state of the recognition operation.

The multi-modal interface according to the present invention is a gaze detection process using visual information input from a camera for observing a user or a camera worn by a user, and an eye tracker for detecting a movement of a gaze of a user. Or a head tracker that detects the movement of the user's head, a seating sensor, an interpersonal sensor, etc., can be used to determine the location, area, direction, object, or part thereof that the user is currently looking at or facing. Detecting means for detecting and outputting as gaze target information, voice input, gesture input, keyboard input, input using a pointing device, visual input information from a camera, voice input information from a microphone, , Keyboard, touch panel, mouse, pointing device such as mouse, operation input information from data glove, etc. At least one other media input processing means for receiving and processing input information from a user representing an object other than the user's gazing target, and the control means controlling the at least one other media input processing means in accordance with the gazing target information. The media input processing means appropriately controls the operation acceptability of input input, or the operation status such as start, end, interruption, restart, and adjustment of the processing level of the processing or recognition operation.

[2] Second, anthropomorphic image providing means for supplying an anthropomorphized agent image, detecting means for detecting a gaze target of the user, voice input information of the user, operation input information, Among the image input information, other media input means for acquiring at least one or more input information, for receiving input information from the other media input means and controlling the state of a recognition operation, wherein the detection means Based on the obtained gazing target information, the gazing target of the user recognizes any part of the agent image presented by the anthropomorphic image presenting means, and inputs from the other media input recognizing means in accordance with the recognition result. And control means for selecting the reception of

According to this configuration, an anthropomorphic agent image responding to the user is specifically anthropomorphized as a person, a creature, a machine, a robot, or the like, who provides services while facing the user. There is an anthropomorphic image presenting means for presenting image information of a still person or a moving image of the agent person to the user, and according to the gazing object information obtained by the detecting means, an anthropomorphic image presentation of the user Depending on whether or not the agent person presented by the means points to the whole, or part of the face, eyes, mouth, ears, etc., the control means selects input reception from other media input recognition means. It was done.

[3] Third, character information, voice information,
Feedback presentation means for presenting a feedback signal to the user by presenting at least one signal such as still surface image information, moving image information, and force, and input from the media input recognition means with reference to the gaze target information. And a control means for controlling so as to appropriately present a feedback signal to the user through the feedback presenting means when making a selection of reception.

In this case, there is a feedback presenting means for presenting a feedback signal to the user by presenting at least one signal such as text information, voice information, still image information, moving image information, and force. Is
When switching whether or not to accept input from the media input recognition means with reference to the gaze target information, control is performed so as to appropriately present a feedback signal to the user through the feedback presentation means.

[4] Fourth, an image of an anthropomorphized agent person who provides a service facing the user. The agent person image provides the user with necessary gestures and facial expression changes. Anthropomorphic image presentation means for presenting the image as a non-verbal message with the image having
A control means for controlling to appropriately present a signal by a non-verbal message to the user through the anthropomorphic image presenting means when receiving and selecting an input from the media input recognizing means with reference to the gaze target information. It is characterized by the following.

In this case, the personification image presenting means includes a person, a creature, a machine,
Or, the image information of the agent personified as a robot or the like by a still image or a moving image, and gestures such as nodding, gesture, hand gesture to the user,
Any number and any kind of agent person images, such as changes in facial expressions, can be prepared or appropriately generated, and these images can be used to present non-verbal messages. Thus, when the input from the media input recognition means is received and selected with reference to the gaze target information, control is performed so as to appropriately present a signal by a non-verbal message to the user through the anthropomorphic image presentation means.

[5] Fifth, a detecting means for detecting a gaze target of the user, an information output means for outputting voice information, operation information, and image information to the user, and voice input information from the user First control means for receiving at least one of input information among operation input information and image input information and controlling a state of a recognition operation; and at least one information output means by referring to the gazing target information. A second control means for appropriately controlling an operation state such as start, end, interruption, resumption of output, or adjustment of presentation speed.

In the case of this configuration, detection means for detecting a gaze target, specifically, a gaze detection process using visual information input from a camera for observing a user or a camera worn by the user, An eye tracker that detects the movement of the user's line of sight, a head tracker that detects the movement of the user's head, a seating sensor, an interpersonal sensor, etc., the user is currently viewing or facing, There is a detection means for gaze target detection that detects an area, a direction, an object, or a part thereof and outputs the information as gaze target information, and further provides text information, voice information, still image information, and moving image information to a user. There is at least one information output means for outputting information by the presentation of at least one signal such as the presentation of a force, and the control means refers to the gazing target information and outputs at least one information. Output means, the start of the output, ends, interrupted, and controls restart, or the operation conditions such as adjustment of the presentation rate appropriate.

[6] Sixth, character information, voice information,
By presenting at least one of the static surface image information, the moving image information, and the presentation of the force, the information to be watched when presenting the information from the information output means and the alerting means for calling the user's attention by presenting at least one signal. Depending on the
And a second control means for controlling so as to appropriately present a signal for calling the user's attention.

In the case of this configuration, there is an alerting means for alerting the user by presenting at least one signal such as character information, voice information, still image information, moving image information, and presentation of force. The control means, when presenting the information from the information output means, controls so as to appropriately present a signal for calling the user's attention through the attention calling means in accordance with the attention target information.

[7] Seventh, by using at least one of input means such as gaze target information or input means such as a camera, a microphone, a keyboard, a switch, a pointing device, a sensor, etc. Reaction detection means for detecting a user's response to the signal and outputting this as user response information, and at least one of an operation status of the information output means and a warning means according to the content of the user response information. Control means for controlling is provided.

In such a configuration, the gaze target information or input means such as a camera, a microphone, a keyboard, a switch, a pointing device, and a sensor is used.
There is reaction detection means for detecting a user's response to the signal for alerting and outputting it as user response information, and the control means controls the operation status of the information output means and the operation status of the information output means according to the content of the user response information. (4) At least one of the warning means is appropriately controlled.

[8] Eighthly, a detecting means for detecting a gaze target of the user and at least one or more input information among the user's voice input information, operation input information, and image input information are obtained. An image of an anthropomorphic agent person providing a service facing the user with other media input means, wherein the agent person image is a non-verbal message of an image having a required gesture and facial expression change to the user. Anthropomorphic image presenting means for presenting the image, character information, audio information, still image information, moving image information,
By presenting at least one signal out of the presentation of power or the like, the information output means for outputting information to the user and the presentation of a non-verbal message through the anthropomorphic image presenting means draw the user's attention. A user who responds to the signal for alerting by referring to at least one of information to be alerted and information to be watched or input information from a camera, a microphone, a keyboard, a switch, a pointing device, a sensor, or the like. Response detection means for detecting the response of the user, and output as user response information, and at least one of the other media input processing means according to the gaze target information, whether or not input acceptance, or start or end of processing or recognition operation, Appropriately control the operation status such as suspending, resuming, adjusting the processing level, and referencing the gaze target information to recognize the media input. When switching between accepting and not accepting input from the stage, use text information, voice information, still image information, moving image information, force presentation, or non-verbal message to the user through anthropomorphic image presentation means Controlling to present a signal as appropriate, referring to the gaze target information, at least one information output unit, start of output,
To properly control the operation status such as termination, interruption, resumption, adjustment of the processing level, etc., and to present the information from the information output means, according to the information to be watched, to call the attention of the user through the warning means Control to present the signal of
Control means for appropriately controlling at least one of the operation status of the information output means and the alerting means according to the content of the user reaction information.

In such a configuration, detection means for detecting a gaze target, specifically, a gaze detection process using visual information input from a camera for observing a user or a camera worn by the user, or the like. , An eye tracker that detects the movement of the user's line of sight, a head tracker that detects the movement of the user's head, a seating sensor, an interpersonal sensor, etc.
There is a detecting means for detecting a place, an area, a direction, an object, or a part thereof, which the user is currently looking at or facing, and outputs the information as gaze target information, such as voice input, gesture input, Keyboard input, input using a pointing device, visual input information from a camera, voice input information from a microphone, pointing devices such as a keyboard, touch panel, pen, and mouse,
Receives input information from the user that indicates something other than the user's gaze, such as operation input information from a data glove,
At least one other media input processing means for performing processing;
Image information of still or moving images of an agent personified as a machine or robot,
Anthropomorphic image presentation means for presenting a user with gestures such as nodding, gesture, hand gesture, and any number of non-verbal messages such as facial expression changes,
To the user, text information, audio information, still image information, moving image information, at least one signal output such as presentation of force, at least one information output means for outputting information,
To alert the user by presenting at least one signal such as text information, audio information, still image information, moving image information, or force, or by presenting a non-verbal message through an anthropomorphic image presentation means The user's response to the signal for alerting is detected by referring to the alerting means to be watched and the information to be watched or the input information from the camera, microphone, keyboard, switch, pointing device, sensor, etc. There is reaction detection means for outputting as reaction information, and the control means controls whether at least one other media input processing means can accept input, or starts, ends, suspends, or resumes processing or recognition operation in accordance with the gaze target information. , Control the operation status such as adjusting the processing level as appropriate, refer to the gaze target information, and When switching between accepting and not accepting force, the user is provided with text information, audio information, still image information, moving image information, presentation of force, or a non-verbal message to the user through anthropomorphic image presentation means as appropriate. Control to present, referring to the gaze target information, at least one of the information output means, output start, end, interruption, restart, appropriately control the operation status such as adjustment of the processing level, from the information output means When presenting information, control is performed so as to appropriately present a signal for calling the user's attention through the alerting means according to the attention target information, and according to the content of the user reaction information, the information output means And at least one of the alerting means is appropriately controlled.

[9] Ninth, as a multi-modal interface method, a gaze target of a user is detected, and at least one of a user's voice, gesture, operation information of the user by operation means, and the like is used. Regarding the processing of the information, the operation status such as selection of input reception or start, end, interruption, restart, and adjustment of the processing level of the processing or recognition operation is appropriately controlled in accordance with the gaze target information. It also detects the user's gaze target, presents an image of an anthropomorphic agent person who provides services facing the user to the user as image information, and based on the gaze target information, Acceptance of the user's voice, gesture, operation information of the user by the operation means, and the like is selected depending on which part of the agent person image is the target.

That is, in the multi-modal input, a gaze detection process using visual information input from a camera for observing a user or a camera worn by the user,
An eye tracker that detects the movement of the user's line of sight, a head tracker that detects the movement of the user's head, a seating sensor, an interpersonal sensor, etc., the user is currently viewing or facing, Detects the area, direction, object, or part thereof and outputs it as gaze target information,
Voice input, gesture input, keyboard input, input using a pointing device, visual input information from a camera, voice input information from a microphone, pointing device such as keyboard, touch panel, mouse, mouse, data glove Such as operation input information from
Regarding the processing of at least one input information from the user that indicates other than the user's gaze target, depending on the gaze target information, input acceptability, or start, end, interruption, restart, processing level of processing or recognition operation This is a method for appropriately controlling the operation status such as adjustment.

Further, still image or moving image information of an agent personified as a person, a creature, a machine, a robot, or the like providing a service facing the user is presented to the user, and the object to be watched is displayed. According to the information, other media input recognition is performed depending on whether or not the gaze target points to the entirety of the agent person presented by the anthropomorphic image presentation means or a part such as a face, eyes, mouth, and ears. It switches the acceptability of the input from the means.

When switching the acceptability of the input from the media input recognizing means by referring to the gazing target information, the user is presented with character information, voice information, still image information, moving image information, force information, and the like. The presentation of the at least one signal provides a feedback signal.

Also, the image information of a person, an anthropomorphic person, such as a person, a creature, a machine, or a robot, who provides a service to the user in a face-to-face or a moving image, and the user, nodding, gesture, Gestures,
When presenting an arbitrary number and type of non-verbal messages such as gestures and facial expression changes, referring to the gaze target information, and switching whether or not to accept input from the media input recognition means, anthropomorphic image presentation means Through a non-verbal message to the user.

[10] Tenth, among character information, audio information, still image information, moving image information, and presentation of power,
In providing information to a user by presenting at least one signal, a gaze target of the user is detected, and referring to the detected gaze target information, start, end, suspension, restart, and processing of the presentation are performed. Control operating conditions such as level adjustment.

When presenting information, at least one signal of character information, voice information, still image information, moving image information, force presentation, etc., is presented to the user in accordance with the gaze target information. To draw the user's attention. It also detects the user's response to the signal for alerting and obtains it as user response information, and acquires the user's voice input information, operation input information, and image input information according to the content of the user response information. And at least one of the alerts is controlled.

As described above, the gaze target of the user is detected, and the information is obtained as the gaze target information. Specifically, gaze detection processing using visual information input from a camera that observes the user or a camera worn by the user, an eye tracker that detects the movement of the user's gaze, a user's head A head tracker that detects movement, a seating sensor, an interpersonal sensor, etc., detects the location, area, direction, object, or part thereof that the user is currently looking at or facing, and as gaze target information obtain. Then, to the user, text information, audio information, still image information, moving image information,
When outputting information by presenting at least one signal such as the presentation of force, by referring to this gaze target information, the operation status such as start, end, interruption, restart, and adjustment of the processing level is appropriately controlled. .

When information is presented from the information output means, at least one signal such as character information, audio information, still image information, moving image information, and force information is presented to the user in accordance with the gaze target information. The presentation calls for the user's attention.

Further, the user's response to the signal for alerting is detected by using gazing target information or input means such as a camera, a microphone, a keyboard, a switch, a pointing device, and a sensor, and the user response information is obtained. The information is output and at least one of the operation status of the information output means and the alerting means is appropriately controlled in accordance with the content of the user response information.

[11] Eleventh is an anthropomorphic agent person image which detects a user's gaze target, outputs it as gaze target information, and provides services to the user. The image should be presented to the user as a non-verbal message with the required gestures and facial expression changes, as well as textual information, audio information,
By presenting at least one signal among still image information, moving image information, and force presentation, the information is output to the user, and at least one of the user's voice input information, gesture input information, and operation input information is output. When receiving more than one input information and performing the processing, depending on the gaze target information, whether or not input reception is possible, or the start or end of the processing or recognition operation,
Controls operating conditions such as suspension, resumption, and adjustment of processing levels. In addition, when switching input acceptability with reference to the gaze target information, it is necessary to present text information, voice information, still image information, moving image information, force, or anthropomorphic image to the user. Make a presentation.

[12] A twelfth is a personified agent person image that detects a user's gaze target, outputs it as gaze target information, and provides a service to the user. The image should be presented to the user as a non-verbal message with the required gestures and facial expression changes, as well as textual information, audio information,
By presenting at least one signal among still image information, moving image information, and force presentation, the information is output to the user, and at least one of the user's voice input information, gesture input information, and operation input information is output. When receiving more than one input information and performing the processing, depending on the gaze target information, whether or not input reception is possible, or the start or end of the processing or recognition operation,
It is characterized by controlling operation states such as suspension, resumption, and adjustment of the processing level.

[0086] When switching input acceptability with reference to the gaze target information, the user is presented with character information, audio information, still image information, moving image information, force, or
It is characterized in that a required presentation is made using a personified image person image.

This is a gaze detection process using visual information input from a camera for observing the user or a camera worn by the user, an eye tracker for detecting the movement of the gaze of the user, a head track of the user. A head tracker that detects the movement of the part, a seating sensor, an interpersonal sensor, etc. detects the place, area, direction, object, or part thereof that the user is currently looking at or facing Information is output, and still or moving image information of an agent personified as a person, creature, machine, robot, or the like who provides services facing the user, and nods and gestures to the user , Hand gestures, gestures, facial expression changes, and any number of other types of non-verbal messages. Stop image information, sliding surface image information,
By outputting at least one signal such as the presentation of force, information is output, voice input, gesture input, keyboard input, input using a pointing device, visual input information from a camera, voice input from a microphone When receiving and processing input information from a user other than the user's gaze target, such as information and operation input information from a keyboard, touch panel, pointing device such as mouse, mouse, data glove, etc., the gaze target information is processed. In accordance with the method, it is a method of appropriately controlling the input / output acceptability, or the operation status such as start, end, interruption, restart, and adjustment of the processing level of the processing or recognition operation.

When switching the acceptability of input with reference to the gaze target information, when the user is presented with character information, audio information, still image information, moving image information, force, or
This is a method of appropriately presenting a signal based on a non-verbal message to a user through an anthropomorphic image presenting means.

Further, by referring to information to be watched or input information from a camera, a microphone, a keyboard, a switch, a pointing device, a sensor, and the like, a user's response to the signal for alerting is detected, and the user's response is detected. The information is output as information, and at least one of the operation status of the information output means and the alerting means is appropriately controlled in accordance with the content of the user response information.

As described above, the present invention detects the user's gaze target by using the technology of gaze detection and the like, and accepts or rejects input from other media, and performs recognition processing or output in accordance with the detected gaze target. The use of non-verbal messages in human-to-human communication, such as controlling the presentation method, interruption, confirmation, etc., especially in anthropomorphic interfaces, such as enabling conversations to be started by looking at faces It is applied to a system to simulate laws and roles.

Therefore, according to the present invention, a plurality of types of input / output media can be used efficiently and effectively, the burden on the user can be reduced, and natural dialogue can be performed in a state close to communication between humans. An interface can be provided.

[0092] Further, the user intends as an input message from a malfunction due to insufficient analysis accuracy of an input from each medium, a malfunction due to ambient noise, or a signal obtained from the input device every moment. It is possible to provide an interface that eliminates a burden on the user due to a malfunction due to a failure to cut out a signal portion or the like.

In an interface device using a medium such as a voice or a gesture that is used not only by the user as an input to a computer or the like as a current operation target but also by a medium used for human-to-human dialogue, Because the user is near the multi-modal system, not when using the interface device of the operating multi-modal system, for example, when talking to another person next to him or showing a gesture, The interface device of the multi-modal system will judge that it is an input to itself and cause a malfunction, but even in such a case, such a situation can be resolved and the cancellation operation accompanying the malfunction and the effect of the malfunction Measures for recovery of
An interface can be provided that can reduce the burden on the user, including the burden that the user must constantly pay attention to avoid malfunction.

Further, even in a situation where it is not originally necessary to identify the information of the media input from the processing operation state of the system, the input signal processing is continuously performed, so that the interruption of the input signal causes the work currently being processed to be interrupted. By eliminating the processing load on media input in unnecessary situations in order to eliminate the adverse effect of causing delay, it is possible to suppress a decrease in the execution speed and utilization efficiency of other services related to the device being used. Such an interface can be provided.

Further, when inputting voices and gestures, there is no need for a special operation such as pressing a button or changing a mode by selecting a menu. It is possible to provide an interface that is natural, does not require training for learning, and does not impose a burden on the user.

Further, according to the present invention, in the case of input by voice media, communication can be performed by using only the mouth, so that, for example, it is possible to use both of them at the same time without interrupting the work performed by hand. It is possible to provide an interface that can utilize the inherent advantages of the voice media, which is said to be possible.

Further, for example, a transient medium in which presented information such as audio output, moving image information, text and image information over a plurality of screens disappears immediately after presentation, or changes every moment. When presenting information to a user by using, even if the user did not pay attention to the information, the user was not able to receive part or all of the presented information Can provide an interface.

Further, when presenting information to a user using a temporary medium, the user presents information for each quantity that can be received at one time, and when presenting the next information to be continued, It is possible to provide an interface that can smoothly present information without imposing a burden such as performing some special operation.

Also, various present situations are displayed in the personification agent person image, the user's gaze is detected, and the user is aware of what the user is paying attention to and takes action. In addition, it is possible to provide an interface that enables the system and the human to proceed with the dialog in a form similar to the communication between humans.

Also, in order to enable the non-contact remote operation, prevent erroneous recognition, and eliminate the burden on the user, the problem related to the background (ii) is to give the anthropomorphic agent the gesture of the gesture indicated by the user. Attention should be paid to the instruction target, so that the user can intuitively understand whether the system can no longer recognize it or the recognition result on the system is incorrect. The invention is configured as follows. [13] A microphone that captures voice input from the user, a camera that observes the user's movements and expressions, an eye tracker that detects the movement of the user's eyes, or a head tracker that detects the movement of the head , Or at least one of a motion sensor that detects the movement of part or all of the body such as hands and feet, or an interpersonal sensor that detects the approach, departure, or seating of the user. An input unit that outputs the captured input information, and receives input information obtained from the input unit, and performs at least one of a voice detection process, a voice recognition, a shape detection process, an image recognition, a gesture recognition, a facial expression recognition, a gaze detection process, and a motion recognition. By performing one process, the input from the user is being received, that the reception has been completed,
An input recognition unit that outputs, as operation status information, a reception status of an input from the user such as recognition success or recognition failure, a warning sound, a synthetic voice,
An output unit that presents to the user as feedback using a character string, an image, or a moving image, and a control that presents feedback information to the user through the output unit in accordance with the operation status information obtained from the input recognition unit. Means are provided.

[14] Further, an image of a user is captured by image input means such as a camera (imaging device) and input means for outputting, for example, analog-to-digital converted image information as input information, is obtained from the input means. For image information, for example, by applying a method such as extraction of a difference from an image at a previous time point or an optical flow, for example, a moving region is detected, and collation is performed by, for example, a method such as a pattern matching technology. An input recognition means for extracting a gesture input and outputting the progress of each of these processes as operation status information as needed, and a character string or an image or a buzzer sound or the like in accordance with the operation status information obtained from the input recognition means. It has a control unit that controls to output audio signals from output means such as a CRT display and a speaker. It is characterized by the following.

[15] Further, according to at least one of the input information obtained from the input means and the operation status information obtained from the input recognizing means, feedback information to be presented as feedback to the user is provided. It is characterized by comprising feedback information generating means for generating.

[16] In addition, the user is presented with image information of a person, a living thing, a machine, or an agent personified as a robot or the like who provides a service facing the user, as a still image or a moving image. Feedback information generating means for generating an anthropomorphic image, and at least one of the facial expression or action of the anthropomorphic image to be presented to the user is determined according to the operation status information obtained from the input recognizing means. Feedback information generating means for generating feedback information that is a facial expression to be watched, such as a pointing destination of a pointing gesture, or a part or a part thereof where the user realizes a gesture expression, such as a fingertip, face, or eyes; The feedback information generated by the feedback information generating means is And control means for presenting feedback information from the output means to the user.

[17] Also, the spatial position of the input means,
And information on the spatial position of the output means, and arrangement information storage means for holding at least one of the information on the spatial position of the user as arrangement information, a reference object of the pointing gesture input by the user, the user, An input recognition unit that outputs reference object position information indicating a spatial position of a user's face, hand, or the like; arrangement information obtained from the arrangement information storage unit; reference object position information obtained from the input recognition unit;
Feedback means is provided for determining at least one of the operation, expression, or control timing of the anthropomorphic agent with reference to at least one of the operation status information and outputting the determined result as feedback information.

[18] Also, a microphone that captures voice input from the user, a camera that observes the user's movements and facial expressions, an eye tracker that detects the user's eye movement, or a head movement detection Head tracker, or a motion sensor that detects the movement of part or all of the body such as hands or feet, or the approach, departure,
An input step comprising at least one of an interpersonal sensor for detecting seating and the like, receiving an input from a user as needed, and outputting the input information; receiving the input information obtained by the input step; By performing at least one of shape detection processing, image recognition, gesture recognition, facial expression recognition, gaze detection processing, or motion recognition, input from the user is being received, that reception has been completed, Successful recognition,
Alternatively, an input recognition step of outputting, as operation status information, an input reception status from the user such as recognition failure, and the like, a warning sound, a synthesized voice, a character string, an image,
Alternatively, it is characterized in that the output step is controlled based on the operation status information obtained by the output step of presenting the user as feedback and the input recognition step using a moving image, and the feedback is presented to the user.

[19] Also, from the input recognition step, image information of still or moving images of an agent personified as a person, a creature, a machine, a robot, or the like who provides a service facing the user is obtained. A feedback information generating step of generating as anthropomorphic image information to be presented to the user according to the operating state information to be provided, and a feedback information generating step and an output step based on the operating state information obtained by the input recognition step. By doing so, for example, at the time the speech input is made,
It is characterized by presenting feedback to the user, such as presenting an expression of “nodding”.

[20] Also, a recognition step of outputting position information which is information relating to the spatial position of the pointing gesture input by the user, the user, the user's face and hands, and the like. A location information storage step of storing, as location information, at least one of a location, information on a spatial location of the output unit, and information on a spatial location of the user; and at least one of location information, location information, and operation status information. According to the present invention, for example, feedback is presented to the user by, for example, presenting an expression for gazing at a reference object which is a target of the pointing gesture of the user at any time.

The present system having such a configuration is as follows.
A microphone that captures voice input from the user, a camera that observes the user's movements and facial expressions, an eye tracker that detects the user's eye movement, a head tracker that detects the movement of the head, hands, feet, etc. At any time, input from the user input from at least one of the input means such as a motion sensor that detects the movement of a part or the whole body or an interpersonal sensor that detects the approach, departure, seating, etc. of the user Capture, obtain as input information, and perform at least one of recognition processing of voice detection processing, voice recognition, shape detection processing, image recognition, gesture recognition, facial expression recognition, eye-gaze detection processing, or motion recognition. Information on the reception status in response to the input from the user, that is, reception is being performed, reception has been completed, recognition has been successful. Information or information on the status of reception of input from the user, such as the fact that recognition has failed, is obtained as operation status information, and based on the obtained operation status information, a warning sound, a synthesized voice, a character string, an image, or A moving image is presented to the user as feedback from the system to the user (that is, a reaction from the system to the recognition situation of the user).

Further, an operation in which image information in a still image or a moving image of an agent personified as a person, a creature, a machine, a robot, or the like who provides a service while facing the user is obtained from the feedback information recognition means. By generating as anthropomorphic image information to be presented to the user in accordance with the situation information and displaying this, for example, when a voice input is made, an anthropomorphic agent presents an expression such as "nodding", for example. Give feedback to the person.

Further, the image recognition is performed by the recognition means to obtain position information which is information relating to the spatial position of the pointing gesture input by the user, the user, the user's face and hands, and the like. The spatial position of the input,
And information on the spatial position of the output unit, and holds at least one of the information on the spatial position of the user as arrangement information, position information, and arrangement information, according to at least one of the operation status information, for example, It provides feedback to the user, such as presenting an expression to look at the reference object that is the target of the pointing gesture of the user as needed.

As described above, the user can recognize a position distant from the system or a pointing gesture performed in a non-contact state with the device, and can input an instruction based on the gesture, and can perform erroneous recognition. It is possible to provide a multi-modal interface device and a multi-modal interface method that can perform gesture recognition without error and eliminate failure in gesture extraction. Also, at the time when the user starts the gesture intended to be input or at the time when the input is being performed, the user can know whether or not the system has correctly extracted the gesture input. It is possible to provide a multi-modal interface device and a multi-modal interface method that can eliminate an indispensable burden. In addition, in response to a pointing gesture input from a user for referring to a place or thing in the real world, it is possible to appropriately display which place, which object, or which part has been received as the pointing destination. A multimodal interface device and a multimodal interface method that can be provided.

[0112]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, embodiments of the invention as a solution to the above-mentioned background (i) will be described.

(First Embodiment) The present invention uses a technique of gaze detection or the like to determine whether or not an input from another medium can be accepted, a recognition process, or a method of presenting an output, according to the user's gaze target. It controls interruption, confirmation, etc., especially in the anthropomorphic interface, by simulating the use and role of non-verbal messages in human-to-human communication, for example, by starting a conversation by looking at a face. In addition, a user interface that is natural, has no burden on the user, and is surely realized.

Hereinafter, a multi-modal dialogue device according to the first embodiment of the present invention will be described in detail with reference to the drawings.

The present invention relates to a human interface in a multi-modal dialog device which enables a more natural conversation to proceed using various media. The subject of the present invention is a human interface (multi-modal interface). However, since various components of the interface part can be realized by extracting and combining necessary components and their functions from the entire multi-modal interaction device, here, the one related to the multi-modal interaction device is described. An embodiment will be described.

<Description of the Configuration of the Present Apparatus> FIG. 1 is a block diagram showing a configuration example of a multi-modal interactive apparatus as an example of the present invention. As shown in FIG. Other media input unit 102, anthropomorphic image presentation unit 103, information output unit 104, alert unit 105,
It comprises a reaction detection unit 106 and a control unit 107.

Of these, the gaze target detection unit 101
The gaze direction of the user of the multi-modal interaction device is detected, and the “place”, “region”,
It is a device that detects “direction”, “object”, or “part” thereof, and outputs it as gaze target information. The gaze target detection unit 101 includes, for example, an eye tracker device for observing the user's eye movements, a head tracker device for detecting the movement of the user's head, a seating sensor,
Japanese Patent Application Laid-Open No. 8-059071 "Viewing Location Estimation Apparatus and Method"
By processing image information obtained from a camera for observing the user or a camera worn by the user by the method disclosed in, for example, and detecting the direction of the user's line of sight, Is detected, or the "place", "area", "direction", "object", or "part" thereof to which the user is facing is detected, and the gaze target information is output.

The gaze target detection unit 101 defines in advance a set of all or a position of an arbitrary gaze target, an arbitrary gaze target region, and a description (name or the like) of the gaze target. By storing the information, the gaze target information including the gaze target description and the information on the time at which the user gazes at the gaze target are output.

FIG. 2 shows an example of the gaze target information output by the gaze target detection unit 101. The gaze target information includes “gaze target information ID”, “gaze target description information A”, and “time information”. B ", and so on.

In the gaze target information shown in FIG. 2, “P101”, “P102”,
.., “P201”,..., Etc., the identification symbols of the corresponding gaze target information are recorded.

In the column of “attention object description A”, the attention object detection unit 101 detects “personalized image”, “other person”, “output area”, “out-of-screen area”, and so on. The description of the watched object is recorded, and "t3", "t10", "t1"
Time information about the point in time at which the user gazes at the corresponding gaze target, such as “5”, “t18”,.

That is, the user takes a gaze action, and every time it is detected, "P101", "P102", "P1"
03 ”,“ P104 ”,“ P105 ”,..., In that order, an ID (identification code) is assigned, what is the target of the detected gaze behavior, and Whether or not there is is output as gaze target information.

In the example of FIG. 2, the information with the ID “P101” is
The gaze target is “personified image”, and the point of occurrence is “t”.
3 ”, the information with the ID“ P102 ”indicates that the gaze target is“ other person ”, the time of occurrence is“ t10 ”, and the ID
The information of “P106” indicates that the gaze target is the “output area” and the time of occurrence is “t22a”.

The other media input unit 102 in FIG.
FIG. 3 shows a detailed configuration example for acquiring input information from a user obtained from various input devices.

That is, as shown in FIG. 3, the other media input unit 102 is divided into an input device unit and a data processing unit. Of these, the components of the data processing unit are the speech recognition device 102a, Character recognition device 102b,
The language analysis device 102c, the operation input analysis device 102d, the image recognition device 102e, the gesture analysis device 102f, and the like correspond to this. Further, as components of the input device unit, a microphone (microphone) 102g, a keyboard 1
02h, pen tablet 102i, OCR (optical character recognition device) 102j, mouse 102k, switch 102
1, touch panel 102m, camera 102n, data glove 102o, data suit 102p, eye tracker, head tracker, interpersonal sensor, seating sensor,
... and so on.

Of these, the speech recognition device 102a
The character recognition device 102b analyzes a voice output signal of the microphone 102g and sequentially outputs the information as word information.
Is for recognizing a character based on character pattern information obtained from the pen tablet 102i or the OCR 102j, and outputs the recognized character information.

The language analysis device 102c is provided with a character code information from the keyboard 102h and a speech recognition device 102c.
This is a device that performs language analysis on the basis of character information from a and character recognition device 102b and outputs the contents intended by the user as user input information.

The operation input analysis device 102d is a device that analyzes the operation information of the user using the mouse 102k, the switch 102l, the touch panel 102m, or the like, and outputs the content intended by the user as the user input information. . The image recognition device 102e is a device for sequentially recognizing a user's silhouette, line of sight, face direction, and the like from a user's image obtained by the camera 102n and outputting the information.

The data glove 102o is provided with various sensors at various places, and outputs information such as bending of the finger, opening of the finger, and movement of the finger by putting the glove on the user's hand. The data suit 102p is a device in which various sensors are attached to various places, and by wearing the data suit 102p on the user, various types of information on the body movement of the user can be obtained. .

The gesture analysis device 102f uses the information from the data suit 102p and the data glove 102o or the information from the image recognition device 102e to
The gesture of the action indicated by the user is analyzed, and the information corresponding to the analyzed gesture is output as user input information.

That is, the other media input unit 102 includes the microphone 102g, the camera 102n, and the keyboard 102
h, touch panel 102m, pen tablet 102i,
Then, a pointing device such as a mouse 102k (or a trackball), a data glove 102o, a data suit 102p, an eye tracker, a head tracker, an OCR 102j, and further, although not shown in FIG. Capture, sample, code, digitize, and filter the input of voice, visual, and operation information from the user obtained through at least one of these input devices, including sensors. , Signal conversion, recording, storage, pattern recognition, language / sound / image / operation / operation analysis, understanding, intention extraction, etc. User input information.

FIG. 3 shows only an example of the configuration of the other media input unit, and the components, the number thereof, and the connection relationship between the components are not limited to this example.

The personified image presentation unit 103 in FIG.
Is a device for presenting gestures such as gestures, hand gestures, changes in facial expressions, etc. to the user as images. FIG. 4 shows an example of a display screen including the output of the anthropomorphic image presentation unit 103.

In FIG. 4, 103a is a display area for presenting a personified image, and 102b is a display area for outputting information. Anthropomorphic image presentation part 1
Reference numeral 03 denotes a multimodal dialogue device that enables the user to present an intention to be presented in the form of gestures such as gestures, hand gestures, changes in facial expressions, and the like when interacting with the user. Control from
“Affirmation”, “calling”, “voice can be heard”, “communication failed”, and the like are appropriately presented to the user with a gesture image.

Therefore, the user can intuitively recognize the current state by looking at the gesture image. That is, here, as in a dialogue between humans, the situation, the degree of understanding, and the like are indicated by gestures, so that communication between the machine and the human can be performed smoothly and communication can be achieved.

The information output unit 104 shown in FIG. 1 presents information such as “characters”, “still images”, “moving images”, “sounds”, “warning sounds”, and “power” to the user. FIG. 5 shows a configuration example of the information output unit 104.

As shown in FIG. 5, the information output unit 104 includes a character image signal generation device 104a and an audio signal generation drive device 1.
04b, a device control signal generation device 104c, and the like. Among these, the character image signal generation device 104a
It is a device for generating a character image signal which is an image signal of a character string to be displayed based on the output information from the control unit 107.
Based on the output information from 7, a voice signal to be transmitted to the user is generated and provided to a voice output device such as a speaker, a headphone, or an earphone included in the multi-modal interactive device, and is driven. Also, the device control signal generation device 104
c generates, based on the output information from the control unit 107, a control signal for a force display (power supply device) that returns an operation as a response to the user by physical force, and a control signal for lamp display and the like. It is a device to do.

In the information output unit 104 having such a configuration,
As information to be output to the user, the information output unit 10
4 receives the output information passed from the problem-solving device or database device which is a component of the multi-modal dialogue device to which it is connected, and controls output devices such as character and image displays, speakers and force displays (power devices). Then, information such as characters, still images, moving images, sounds, warning sounds, and power is presented to the user.

That is, the multimodal dialogue device is a device that interprets a question to be asked by a user, a problem to be solved or a problem to be solved by interpreting a request, a request, or a puzzle, and seeks a solution. A solution apparatus and a database (including a knowledge base) used by the problem solution apparatus are provided. Then, it receives output information passed from a problem solving device or a database device, and controls output devices such as a character and image display, a speaker and a force display (power supply device), and provides a user with “characters”, “ "Still face image", "moving image", "audio",
Information is presented using various communication means such as "warning sound" and "power".

The alerting unit 105 in FIG.
This is a device that alerts the user by calling or issuing a warning sound. Under the control of the control unit 107, the alerting unit 105 presents a warning sound, a specific linguistic expression for a call, the name of the user, or the like to the user as an audio signal, or a screen display unit. By presenting as a character signal, repeatedly inverting (flashing) the display screen, presenting an optical signal using a lamp, or using a force display,
The physical force signal is presented to the user, or through the anthropomorphic image presentation unit 103, for example, gestures, hand gestures, facial expression changes, image information simulating body movements, and the like are performed. To draw the attention of the elderly.

It is to be noted that the alerting unit 105 is configured as one independent element, or is configured to use the output unit 104 to present a signal for alerting the user. Is also possible.

The reaction detecting section 106 in FIG. 1 detects whether or not the user has responded to an action from the multi-modal interactive device. The reaction detection 106 is performed by using a camera, a microphone, a keyboard, a switch, a pointing device, a sensor, or other input means, and when a warning is presented to the user by the warning unit 105, a user-specified identification is performed. Operation, emits a predetermined specific voice, detects that a predetermined specific gesture is performed, or refers to gaze target information obtained from the gaze target detection unit 101. Thus, it is determined whether or not the user has responded to the signal for alerting, and is output as user response information.

The reaction detecting section 106 can be constituted as one independent component, or can be realized by being incorporated in the other media input section 102 as a function.

The control unit 107 in FIG. 1 is responsible for various controls of this system, arithmetic processing, judgment, and the like, and plays a central role in control and arithmetic of this system.

The control unit 107 controls the other components of the apparatus to realize the operation of the apparatus of the present invention and obtain the effect of the apparatus of the present invention. The procedure will be described later in detail.

FIG. 6 shows an example of the internal configuration of the control unit 107.
As shown in the figure, the control unit 107 controls the control process execution unit 20
1. Control rule storage unit 202 and interpretation rule storage unit 20
3 and the like.

Of these, the control processing execution unit 201
A status register S for holding status information of each element and an information type register M for holding an information type are provided therein.
In addition to receiving signals from each component, such as the operation status of each component of the multi-modal dialogue device, gaze target information, user response information, output information, and the like, these signals, the contents of the status register S, With reference to the contents of the control rule storage unit 202 and the interpretation rule storage unit 203, processing according to the processing procedure A described below is performed, and control signals to each component of the multimodal interface device are transmitted in accordance with the obtained result. By outputting, the functions and effects of the multimodal interface device are realized.

The control rule storage unit 202 stores a predetermined control rule.
03 holds a predetermined interpretation rule.

FIG. 7 shows an example of the contents of a control rule stored in the control rule storage unit 202. Here, the information of each control rule is classified and recorded as “rule ID”, “current state information A”, “event condition information B”, “action list information C”, “next state information D”, and the like. I have to.

In each entry of the control storage unit 202, an identification symbol for each control rule is recorded in “rule ID”.

In the “current status information A” column, restrictions on the contents of the status register S, which is a condition for applying the control rule of the corresponding entry, are recorded. In the “event information B” column, In addition, a limit on an event serving as a condition for applying the control rule of the corresponding entry is recorded.

In the column of “action list information C”, information on the control process to be performed when the corresponding control rule is applied is recorded. In the column of “next state information D”, When the control rule of the corresponding entry is executed, information on a state to be recorded as an update value is recorded in the state register S.

Specifically, in each entry of the control storage unit 202, “Q1”, “Q
An identification symbol for each control rule is recorded in the form of “2”, “Q3”, “Q4”, “Q5”, etc. In “current state information A”, “input / output standby”, “input in progress” "," Checking availability "," Outputting "," Preparing "," Suspended "," Calling ", etc. as conditions for applying the control rules of the entry by each rule ID. The contents of the status register S must be set in correspondence with the rule ID.

The "event condition information B" includes "input request", "output control reception", "output start request", "output preparation request", "input completed",. Rule ID indicates what event must be the condition for applying the control rule
It is set to correspond. “Action information C” is
“[Input reception FB input reception start]”, “[]”,
“[Output start]”, “[output enable / disable]”, “[input acceptance stop input complete FB]”, “[input acceptance stop cancel FB presentation]”, “[output start]”, “[interrogation]”, .., Etc., when a corresponding control rule is applied, what kind of action is to be performed is set corresponding to the rule ID.

In the control processing recorded in the column of “action information C”, “[input reception FB (feedback)]” indicates to the user that an input from the other media input unit 102 of the present apparatus is required. It presents feedback indicating that it has become possible, for example, presents a sound signal such as a character string, image information, or a sound such as a chime or a hammer with a positive meaning, or presents an anthropomorphic image The processing of presenting to the user, for example, directing the user's gaze through the unit 103 or displaying a gesture of placing a hand on the ear is shown.

In addition, “[input completion FB (feedback)]” and “[confirmation reception FB (feedback)]”
Is a process of presenting feedback to the user that the communication has been performed correctly or that the user has correctly received the intention of confirmation from the user in response to the call to the user.

In the control processing recorded in the column of “action list information C”, “[input reception FB (feedback)]” is provided to the user by inputting from the other media input unit 102 of the apparatus. Is presented as feedback indicating that the state has become possible. For example, the presentation method is “character string”, “plane image information”, or “chime” or affirmative. "Aoi hammer"
Reaction to the user, such as presenting with a sound signal, such as the voice of a person, or turning the gaze to the user through the anthropomorphic image presentation unit 103, or displaying an image of a gesture of placing a hand on the ear. Is presented.

In addition, “[input completion FB (feedback)]” and “[confirmation reception FB (feedback)]”
Is a process for presenting feedback indicating that communication has been correctly performed to the user or that the intention of confirmation from the user in response to the call to the user has been correctly received. (Feedback)] As in the case of "), a signal by sound, voice, text, or image is presented, or the personified image presentation unit 1
Through 03, a process of presenting a gesture such as “nodding” is shown.

“[Cancel FB (feedback)]” is used in communication with the user.
This is a process of presenting a user with feedback indicating that a problem has occurred. For example, a warning sound, a character string or an image indicating a warning is presented, or the palm is displayed through the anthropomorphic image presentation unit 103. This represents a process of presenting a gesture of spreading the hands while bending both hands.

Also, “[input reception start]” and “[input reception stop]” respectively correspond to the other mode input unit 10.
2 is a process for starting and stopping the input of “2.” Similarly, “[output start]”, “[output interruption]”, “[output restart]”, and “[output stop]” are used from the information output unit 104. The process of starting, suspending, resuming, and stopping the output of information to the user respectively.

“[Output availability check]” refers to the gaze target information output from the gaze target detection unit 101 and the information to be presented to the user with reference to the contents of the interpretation rule storage unit 203. This represents a process of checking whether or not it can be currently presented to the user.

[0162] In addition, "[calling]" presents, for example, a warning sound or an interjection voice during calling in order to draw the user's attention when presenting information to the user. For example, a gesture of presenting the user's name, flashing the screen (temporarily repeatedly inverting the display), presenting a specific image, or waving a hand to the right or left through the anthropomorphic image presenting unit 103 Is presented.

"[Input reception FB (feedback)]"
In the same manner as above, the process of presenting a signal such as a sound, a voice, a character, or an image, or presenting a gesture such as “nodding” through the anthropomorphic image presenting unit 103 is shown.

"[Cancel FB (feedback)]" indicates that communication with the user
This is a process of presenting a user with feedback indicating that a problem has occurred. For example, a warning sound, a character string or an image indicating a warning is presented, or a palm is displayed through the anthropomorphic image presentation unit 103. Represents a process of presenting a gesture of spreading the hands while bending both hands with the "?"

Also, “[input reception start]” and “[input reception stop]” respectively correspond to the other mode input unit 10.
2 is a process for starting and stopping the input of “2.” Similarly, “[output start]”, “[output interruption]”, “[output restart]”, and “[output stop]” are used from the information output unit 104. The process of starting, suspending, resuming, and stopping the output of information to the user respectively.

The “[output possibility check]” refers to the gaze target information output from the gaze target detection unit 101 and the information to be presented to the user with reference to the contents of the interpretation rule storage unit 203. This represents a process of checking whether or not it can be currently presented to the user.

"[Interrogation]" is for presenting information to the user, for example, by presenting a warning sound or presenting a spoken word during interrogation in order to draw the user's attention. , Presenting the user's name, flashing the screen (temporarily inverting the display), presenting a specific image, or through the anthropomorphic image presenting unit 103,
For example, a process of presenting a gesture of waving a hand left and right is shown.

The “next state information D” includes “inputting”,
Updates the status register S when the control rule of the corresponding entry is executed, such as “Checking availability”, “Output”, “Preparing”, “I / O standby”, “Interrogation”, etc. The information to be recorded as a value (information about the state) is described in Rule I.
D is set.

Therefore, when the "rule ID" is "Q1", the contents of the status register S, which is a condition to which the control rule of the corresponding entry is applied, are "input / output standby" and "Q1".
When the entry "1" occurs, if the content of the status register S is "input / output standby", an "input request" occurs as an event. At this time, a control process of "input reception feedback and input reception start" is performed. This control rule indicates that the contents of "inputting" are written to the status register S, and the contents of the status register S are updated from "input / output standby" to "inputting". Will be.

Similarly, when the “rule ID” is “Q5”,
If the content of the status register S, which is a condition for applying the control rule of the corresponding entry, is "inputting" and an entry "Q5" occurs, if the content of the status register S is "inputting", This control rule indicates that "input completion" occurs as an event, and at this time, control processing of "input acceptance stop and input completion feedback" is performed, and the status register S changes its contents to "input / output standby". It will be.

FIG. 8 shows an example of the contents of the interpretation rule storage unit 203. Information regarding each interpretation rule includes “current state information A”, “gazing target information B”, “input / output information type information C”,
And "interpretation result information D".

In each entry of the interpretation rule storage unit 203, an identification symbol of a corresponding rule is recorded in a “rule ID” column. In the column of "current state information A", restrictions on the state register S when the corresponding interpretation rule is applied are recorded.

In the field of “gazing target information B”, the control processing executing unit 201
The information on the gaze target for comparison and collation with the column of “gaze target information A” of the gaze target information to be interpreted is recorded.

In the column of “input / output information C”, there are restrictions on the type of information input by the user at the time of input.
At the time of output, restrictions on the type of information to be presented to the user are recorded.

In the field of "interpretation result information D", an interpretation result when the interpretation rule is applied to the received gaze target information is recorded.

Specifically, “Rule ID” includes “R
1 "," R2 "," R3 "," R4 "," R5 "," R
The identification code of the corresponding rule is recorded in the form of “6”, etc. The “current state information A” contains “input / output standby”, “input being performed”, “checking availability”, “outputting”. , "In preparation", "interruption", and so on, the contents to be held by the information held in the status register S when the corresponding interpretation rule is applied are recorded.

The “gazing target information B” includes “input request area”, “personified image”, “microphone area”, “camera area”, “output request area”, “cancel request area”, and “output”. The “gaze target” of the gaze target information received from the gaze target detection unit 101 and interpreted by the control processing execution unit 201 in a manner such as “other than the requested area”, “other person”, “output area”, “front of the device”,. Information on the gaze target for comparison and collation with the column of information A ″ is recorded.

The “input / output information type information C” includes “audio information”, “visual information”, “moving picture information”, “other than moving picture information”, “still picture information”, and so on. In, restrictions on the type of information input from the user are recorded, and at the time of output, restrictions on the type of information presented to the user are recorded.

The "interpretation result information D" includes "input request", "output preparation", "cancel request", "interrupt required",
The interpretation result when the interpretation rule is applied to the received gaze target information is recorded in a state such as “startable”, “reunitable”, “confirmation detected”,.

Therefore, for example, if the “rule ID” is “R2”
When the rule is applied, the contents of the status register S need to be “input / output standby”, the area to be watched is “anthropomorphic image”, and “voice information” is used at the time of input and output. However, it indicates that the interpretation result is “input request”.

The configuration of the control unit 107 has been described above.

Next, the details of the processing in the control processing execution section 201 which plays a central role in the apparatus of the present invention will be described.

The processing in the control processing execution unit 201 which is a component of the control unit 107 is performed according to the following processing procedure A.

FIG. 9 is a flowchart showing the flow of the processing procedure A.

<Processing Procedure A> [Step A1] First, the control processing unit 201
Performs initialization processing. This initialization process is performed by the status register S
And the information type register M are set to the initial state. By this initialization processing, the information of the content "I / O standby" is set in the status register S, and the content of "undefined" is set in the information type register M. Is set, and the other media input unit 1 is set.
02 is set in an input non-accepting state (initialization).

[Step A2] After initialization, input / output determination is made. Waiting for an input to the control unit 107, if there is an input, the input is from the gaze target detection unit 101, that is, the gaze target information Gi that is the detection output from the gaze target detection unit 101.
Is sent, the process proceeds to step A3 for performing the gaze information interpretation process. Although the details are not described because they are not directly related to the present invention, the output information Oj is supplied to the control unit 107 from a problem solving device, a database device, or a service providing device which is a main component of the multi-modal interactive device. If it is, the process goes to step A12 in step A2 which is an input / output determination step.

That is, when the control unit 107 receives the output information Oj from the solving device, the database device, or the service providing device in A2, the process proceeds to step A2.
Proceed to 12. The output information Oj is a control signal for outputting information to the user using the information output unit 104,
It contains information content Cj to be presented to the user and information type Mj, which is the type of information (input / output determination).

[Step A3] The processing here is interpretation of gaze information. The contents of the state register S, the contents of the gaze target information Gi, the contents of the information type register M,
“Current state information A” of each entry in the interpretation rule storage unit 203
Is compared with the contents of the “gazing attention target information B” and the “input / output information type information C”, and the interpretation rule Ri (i =
1, 2, 3, 4, 5 ...) (interpretation of gaze information).

[Step A4] In step A3, if no interpretation rule Ri that satisfies the condition is found, the flow proceeds to step A11, and if found, the flow proceeds to step A5 (interpretability determination).

[Step A5] The "interpretation result information D" corresponding to the found interpretation rule Ri is referred to, and the interpretation result Ii described in the "interpretation result information D" is obtained.
Then, the process proceeds to Step A6 (interpretation result determination).

[Step A6] The contents of the status register S and the interpretation result Ii are compared with the contents of “current state information A” and the contents of “event condition information B” in the control rule storage unit 202, respectively. Then, a corresponding control rule Qi is searched. Then, the process proceeds to step A7 (control rule search).

[Step A7] If no interpretation rule Qi that satisfies the condition is found in the process of step A6, the process proceeds to step A11. On the other hand, if an interpretation rule Qi that meets the conditions is found, the process proceeds to step A8.
Proceed to (control rule presence / absence determination).

[Step A8] Here, control rule Q
i, a list of control processes to be executed [Ci1. Ci2, ...]. Then, the process proceeds to step A9 (control processing list acquisition).

[Step A9] List of control processes to be executed [Ci1. ..] Are obtained, a list of the obtained control processes [Ci1. ..] Are sequentially executed in accordance with <Processing Procedure B> (described later) (execution of each control process).

[Step A10] In the status register S,
The contents of “next state information D” of Qi are recorded. Then, the process proceeds to step A11 (state update).

[Step A11] The processing related to the gaze target information Gi is completed, and the process returns to step A2 (return processing).

[Step A12] In step A2, when the output information Oj is given, the control section 107 proceeds to the process of step A12. In this step, the information type register M stores the information type Mj of the output information Oj. Is referred to, and the control rule stored in the control rule storage unit 202 is referred to, and the content of “current status A” matches the content of the status register S and the content of “event condition information B” is Entry Qk (k = 1, 2, 2) which is “output control reception”
3, 4, 5, ...). Then, the process proceeds to step A13 (control rule search).

[Step A13] Here, in step A12, from among the rule IDs of Q1 to Qx, a control rule ID Qk (k = 1, 2, 3,
(4,... K-1, k, k + 1, k + 2,... X) are not found, the process proceeds to step A17, and if a control rule Qk that meets the conditions is found, the process proceeds to step A14 (of the corresponding control rule). Determination).

[Step A14] In step A14, of the "action information C" in the control rules stored in the control rule storage section 202, the "action information C" corresponding to the found control rule Qk is referred to and executed. List of power control processes [Ck1. Ck2, ... "(control processing list acquisition).

[Step A15] For each element of the control processing list [Ck1, Ck2,...], Control processing is sequentially executed in accordance with <processing procedure B> (described later) (execution of each control processing).

[Step A16] The status register S stores "next status information D" corresponding to the rule ID of Qk.
Record the contents (status update).

[Step A17] The processing related to the information information Oj is completed, and the process returns to step A2 (return processing).

The above is the contents of the processing procedure A. It is determined whether the information entered is from the user or presented to the user, and the former (from the user) is determined. Information), interpret the gaze information, determine the interpretation result,
The control rule corresponding to the determined interpretation result is searched, and if there is a corresponding control rule, what kind of control is performed is listed from among the control rules, and control of the listed control content is performed, If the latter (presented to the user), search for control rules for output,
If there is a corresponding control rule, what kind of control is performed is listed from the control rule, and output control processing of the listed control content is performed, and audio, video, camera, keyboard, When communicating with various input / output devices such as mice and data gloves using analysis processing and control technology, rules such as what to pay attention to, such as communication between humans, should be followed. Decided and divided into information to be used and other information according to the flow of the dialog and the device used, and proceeded with control for the dialog, so it was possible to eliminate the capture of noise components, In order to prevent malfunction, depending on the situation, attention is raised, the understanding level, the situation of dialogue and the reaction are displayed with gestures using anthropomorphic images, and natural It made it possible to.

Next, the processing procedure B will be described. In the processing procedure B, the following presentation operation and control operation are performed according to the content of the action information.

<Processing Procedure B> [Step B1] First, when the control processing Cx as the action information is “input reception FB”, for example, a character string such as “input is possible” or “a microphone is circled” Image information such as "tae", or chime sound, or a hammer such as "yes" having a positive meaning, etc., are presented by voice or text, or turned to the user through the anthropomorphic image presentation unit 103, Display a gesture of placing your hand on the ear.

[Step B2] When the control process Cx is “input completed FB”, for example, a character string such as “input completed”, image information such as “picture of cross mark on microphone”, or “chime sound” , Such as “Yes” or “Ok,” which have a positive meaning, are presented by voice or character, or anthropomorphic image presentation unit 103
A gesture is displayed as an image, such as presenting an image that directs the user's gaze to the user or presenting a nodding image.

[Step B3] The control process Cx is
In the case of the "acknowledgment FB", for example, a character string such as "confirmation", image information, a chime sound, a souvenir such as "yes" or "yes" having a positive meaning, or the like, may be voice or the like. The gesture is displayed by using an image such as presenting a character, or turning the gaze to the user through the anthropomorphic image presenting unit 103 or nodding.

[Step B4] The control process Cx
In the case of “cancellation FB”, a warning sound, a character string, a symbol, or an image indicating a warning is presented, or an anthropomorphic image presenting unit 103 is used, for example, while bending both hands with the palm up. The gesture is presented by using an image such as spreading.

[Step B5] The control process Cx is
In the case of “input reception start” and “input reception stop”, the input from the other mode input unit 102 is started and stopped, respectively.

[Step B7] The control process Cx
In the case of “output start”, “output interruption”, “output restart”, and “output stop”, the output of information from the information output unit 104 to the user is started, interrupted, resumed, and stopped, respectively. .

[Step B8] The control process Cx
In the case of "calling", for example, a warning sound is presented,
For example, presenting the interjection voice of a call such as "Hello", presenting the user's name, flashing the screen (temporarily highlighting), presenting a specific image, or anthropomorphizing For example, a gesture of waving a hand right and left is presented through the image presentation unit 103.

[0212] The information type register M is adapted to appropriately record the type of output information when the information is to be presented to the user.

The above is the configuration and functions of the present apparatus.

<Explanation Using Specific Example> Next, the above-described multimodal interface device and multimodal interface method will be described in more detail.

Here, a gaze target extraction unit 101 having a function of detecting the user's line of sight and head direction, a function of recognizing a user in front of the apparatus and a person recognizing another person, and other media input means 102 are provided. , An anthropomorphic image presentation unit 103 capable of presenting a gesture based on gestures, hand gestures, and facial expression changes to a user, and image output and voice of character information, still image information, and moving image information as an information output unit 104 A situation where a user uses a device having an output unit will be described as a specific example.

FIG. 10 shows the internal state of the apparatus at each time.

[T0] In the control unit 107, the status register S and the information type register M are set to “input / output standby” by the processing of step A1 in “processing procedure A”.
And “undefined” are recorded, whereby the audio input unit, which is one of the components of the other media input unit 102, is in the “input non-acceptance” state.

[T1] Here, it is assumed that noise (noise) is generated around the present apparatus. However, since voice input is in a non-accepted state, this noise is not picked up as voice, and therefore, no malfunction due to noise occurs.

[T2] Subsequently, the user tries to start voice input by looking at the face of the anthropomorphic image presentation unit 103. That is, as shown in FIG. 4, the anthropomorphic image presenting section 103 presents an image of a receptionist on a display screen so that the user can communicate with a gesture.
02a, and an information output area 102b for outputting information such as characters and images. At the initial stage, the anthropomorphic image presentation unit 103 is controlled so as to present the upper body of the receptionist in a standby state as shown in FIG. Accordingly, the user unawarely watches the appearance of the receptionist.

[T3] The gaze target detection unit 101 detects this, and as the gaze target information, ID = P1 in FIG.
The gaze target information shown in the 01 column is output.

[T4] According to the determination at step A2 in “procedure A”, the process proceeds to step A3, where the corresponding interpretation rule is retrieved from the interpretation rule storage unit 203, and at this time, the contents of the “status register S” are Since the “gaze target information A” of the gaze target information of ID = P101 is “anthropomorphic image”
From the interpretation rule storage unit 203 shown in FIG.
2 are extracted ("rule ID" in FIG. 8).
Is interpreted as “input request”, which is “interpretation result information D” corresponding to “R2”).

[T5] At step A5 in “processing procedure A”, an “input request” is obtained as the interpretation result from the contents of “interpretation result information D” of “interpretation rule R2”.

[T6] By the processing of step A6 in “processing procedure A”, a search is performed from the control rule storage unit 202, and the current state information (“gazing target information A” in FIG. 2) is “input standby”. In addition, since the event condition information (“time information B” in FIG. 2) is “input request”, a control rule with an ID “rule ID” of [Q1] in FIG. 7 is selected, and step A8 is performed. As a result, as the content of the “action information C” corresponding to the “control rule Q2”,
“[Input reception FB, input reception start]” is obtained.

[T7] By the processing of step A9 in the “procedure A” and the processing in the “procedure B”, for example, through the anthropomorphic image presentation unit 103, the “hand in the ear” as shown in FIG. The image of the gesture “over” is presented to the user, the voice of “yes” is presented to the user, the acceptance of the voice input is started, and the status register S and the information type register M are registered in steps A10 and A11. The content is updated.

[T8] The voice input from the user is completed, "input completed" is notified to the control unit as a control signal (event), and the interpretation rule Q5 is selected by the processing according to "processing procedure A". / Executed and after the voice input is rejected, a character string such as "input completed", image information such as a picture of an X mark on the microphone, or a chime sound is presented to the user by "processing procedure B2". Is done.

According to the processing exemplified above, in a “scene where no voice input is required”, by setting the input to “non-acceptance”, a malfunction due to noise or the like can be prevented.
In “scenes where voice input is required”, voice input can be performed simply by facing the personified image. At that time, feedback is presented to the user through gestures, etc. Being aware of the changes will make it easier for the user to understand, without any malfunctions, and without the burden of special operations. A multi-modal interface suitable for a human interface with no burden is realized.

[T9] Next, it is assumed that another person x who is not the user approaches the user, and the user turns to the direction of the person x.

[T10] Here, the gaze target detecting unit 10
1 detects this, and as the gaze target information, “other person”, which is “gaze target information A” shown in the column of ID “P102” in “gaze target information ID” in FIG. Outputs the gaze target information.

[T11] The same processing as at time t4 is performed. However, since there is no interpretation rule that meets the condition in this case, the flow proceeds to step A11, and the processing relating to the watch target information ends.

[T12] Further, if the user is “person x”
When the control unit 107 remains in the state of
For example, it is assumed that an output control signal for presenting output information Oj of the information type M = “moving image information” to the user is provided.

[T13] At step A2 in the “control procedure A”, the process proceeds to step A12, where “moving picture information” is recorded in the information type register M, and the control rule storage section 20
2, as the entry in which the “current status information A” matches the content “input / output standby” of the status register S and the “event condition information B” is “output control reception”, rule ID = Q2 Are extracted.

[T14] Through the processing of steps A13 to A17 in the “control procedure A”, it is found from the “action information C” corresponding to the “control rule Q2” that “there is no control processing to be executed”. , Step A16
With reference to the "next state information D" corresponding to the "control rule Q2", "under confirmation of availability" is recorded in the state register S, and the process proceeds to step A2.

[T15] Subsequently, the user sets “person X”
2, the gaze target information having the ID “P103” among the gaze target information IDs in FIG. 2 can be obtained from the gaze target detection unit 101.

[T16] Through the processing of steps A2 to A5 in the “procedure A”, the content of the status register S is “confirmation of propriety” and the “gaze target information A” of the gaze target information P103 is Since it is “other person” and the content of the information type register M is “moving image information”, the entry of the rule ID = R11 in FIG. 8 is extracted,
"Unable to output" is obtained as the interpretation result.

[T17] Step A of “processing procedure A”
Through the processing of steps 6 to A9, the “control rule Q9” is selected by the processing similar to the time point t6 to t8, and the processing of step B8 of the processing procedure B provides the user with, for example, a screen flash or a name. Is called.

[T18] Here, the user turns to the screen area on which the moving image information is presented, so that the gaze target detection unit 101 outputs the gaze target information of the gaze target ID “P104” in FIG. By the same processing as described above, “confirmation detection” is obtained as an interpretation result from “interpretation rule R22”, and “control rule Q14” in FIG. FB presentation, output start].

[T19] By the processing of step A9 in “procedure A” and step B3 in “procedure B”, for example, a hammer such as “yes” is presented to the user by voice or characters, and then “process The output of the moving image information to be presented to the user is started by the processing of step B7 of procedure B ", and the state register S is output in step A10.
Is updated to "outputting".

With the above processing, the present apparatus appropriately controls the start of output in accordance with the user's gaze target and the type of information to be presented, Control of each part according to the user's response to the user's attention, the user's attention is directed differently, and if the presentation of the information is started in that state, the user cannot receive part or all of the presented information That problem has been solved.

[T20] Further, during the presentation of the moving image information, the user turns to another “person X” again, which is detected by the gazing target detecting unit 101, and the gazing target information ID is “P101”. It is assumed that the gaze target information is output.

[T21] As a result, the interpretation rule storage unit 2
"Interpretation rule R14" in the storage information of FIG.
As a result, "interpretation result information D" which is "interrupted" is obtained,
The output is interrupted by the control rule of the rule ID of “control rule Q11”, which is the control rule corresponding to the “event condition information B” of “required interruption” in the information stored in the control rule storage unit 202, and the status register is "Paused".

[T22a] Thereafter, if the user gazes again at the output area, "gaze target information P106" is output, and output is restarted by "interpretation rule R19" and "control rule Q12".

[T22b] Alternatively, for example, when the user continues to pay attention to another, a control signal of an interruption timeout is output due to elapse of a predetermined time or the like, and according to the “control rule Q13”, The interruption of the output of the moving image is reported.

As described above, the present apparatus appropriately controls the presentation of information according to the gaze target to which the user pays attention, the operation status of the apparatus, and the type and nature of the information to be presented. By doing so, the user may fail to receive information that is difficult to receive correctly in a distracted state, or may perform a special operation when interrupting the output of information or resuming the interrupted output. The problem that the burden on the user increases due to the necessity can be solved.

Further, although not included in the above operation example, by using the control rules Q4, Q12, Q13, etc. in FIG. 7, the user is watching the output area as in the case of, for example, moving picture information. If you start output without any
When presenting information that may cause the user to fail to receive a part or all of the presented information, when the information output request is received, the output is not started, the state is being prepared, and standby is performed.
At the stage that the user has noticed that the user has gazed at the output target area from the gazing target information, it is detected that the presentation of information can be started by using the interpretation rules R13, R14, R15, etc. By starting the presentation of
It is also possible to avoid these problems.

Alternatively, interpretation rule R3, interpretation rule R4,
By using the interpretation rule R18, the interpretation rule R21, or the like, for example, a configuration in which voice input is accepted when the user gazes at the microphone, an image input is started when the user gazes at the camera, or when the user gazes at the speaker, It is also possible to configure so that audio output is started.

Although the above is a specific example of the multimodal dialogue device, as described above, the components of the interface of the present invention are different from the multimodal dialogue device of the present embodiment in terms of necessary components. It can be realized by extracting and combining the functions.

More specifically, the apparatus of the invention of [1] in the section of means for solving the problems is characterized in that the gaze target detecting section 10
1 and the other media input unit 102 and the control unit 107.

The apparatus of the invention [2] and the apparatus of the invention [4] can be realized by adding a personification image presenting unit 103 to them, and the apparatus of the invention [3] can be realized by the apparatus [4]. In the apparatus of the present invention, the presentation of the feedback to the user through the anthropomorphic image presentation unit 103 is performed by presentation of at least one signal such as text information, audio information, still image information, moving image information, and force presentation. It can be realized by adding a function that performs

[0249] Further, the apparatus of the present invention according to [5] comprises a gaze target detecting section 101, an information output section 104, and a control section 107.
The device of the invention of [6] can be realized by adding the alerting unit 105 to the device of the invention of [5]. , [6] can be realized by adding a reaction detection unit 106 to the apparatus of the invention of [6]. The above is the configuration and functions of the present apparatus.

The present invention shown in the first embodiment can be applied as a method, and the processing procedures, flowcharts, interpretation rules, and control rules shown in the above specific examples are described as programs. Then, the same functions and effects can be obtained by mounting, executing, and using a general-purpose computer system.

That is, the present invention can be realized by a general-purpose computer. In this case, as shown in FIG. 12, a general-purpose computer including a CPU 301, a memory 302, a large-capacity external storage device 303, a communication interface 304, and the like is used. , Input interfaces 305a to 305
n, input devices 306a to 306n, and output interfaces 307a to 307m and output device 30
8a to 308m, and input devices 306a to 306m.
n, microphone, keyboard, pen tablet, OC
R, a mouse, a switch, a touch panel, a camera, a data glove, a data suit, and the like, and as output devices 308a to 308m, a display, a speaker, a force display, etc.
By the software control by the PU 301, the above-described operation can be realized.

The solution relating to the background (i) has been described above. Next, the background (i
An embodiment of the invention as a solution to i) will be described.

Presenting an anthropomorphic agent so that a user can naturally and smoothly input a non-verbal message such as a voice or a gesture intended for input requires a user to interact with a natural person. It is possible to expect a remarkable improvement in operability.However, by taking this one step further, it is possible to use the gesture target indicated by the user by displaying it so that the anthropomorphic agent gazes at it. The user can intuitively understand whether the pointing destination of the gesture of the user is not recognized on the system side or if the recognition result on the system side is incorrect, etc. For the user, the operability as if the guide of a natural person was responding more carefully was obtained. Worry that burden is eliminated. Therefore, an embodiment for realizing such a system will be described as a second embodiment.

(Second Embodiment) Here, a gesture input from a user is detected in order to allow a user to naturally and smoothly input a non-verbal message such as a voice or a gesture intended to be input. At that time, the hand of the gesture input is gazed at any time depending on the expression of the anthropomorphic agent, or the pointing target is gazed at the reference target, thereby providing natural feedback to the user (that is, the system side). To respond to the user's recognition situation)
In addition, at that time, the view of the user and the personification agent,
Alternatively, an example will be described in which the anthropomorphic agent can be controlled to be moved and displayed in an appropriate place in consideration of the spatial position of a reference target or the like.

Further, in the second embodiment, the purpose is, of course, not only instructing by mounting the device or touching the device, but also in addition to the above, one is provided from a remote position or in a non-contact manner with the device. It is also possible to perform a pointing gesture and recognize it remotely without wearing a device, and
An embodiment will be described in which erroneous recognition and gesture extraction failure that occur due to insufficient accuracy of the gesture recognition method can be suppressed. Also, at the point when the user starts the gesture intended to be input or during the input, it is not known whether the system has correctly extracted the gesture input, and as a result, misrecognition may occur. Alternatively, in order to suppress the burden on the user caused by the user having to perform the input again, a technique for preventing such a situation will be described.

Also, in response to a pointing gesture input from a user for referring to a place or thing in the real world,
It is an object of the present invention to provide a technology capable of appropriately displaying which place, which object, or which part thereof has been received as the pointing destination. In addition, the problem of the conventional method induced by the above-mentioned problem, that is, the correction of the effect of malfunction or the burden on the user caused by re-input, and the burden on the user due to anxiety at the time of user input. So that it can be resolved.

Further, with an interface apparatus and an interface method using an anthropomorphic interface, it is possible to generate an appropriate facial expression of an agent in consideration of the field of view of the user and the anthropomorphic agent from the anthropomorphic agent, and present it as feedback. Make it possible.

A multimodal interface device and a multimodal interface system according to a second embodiment of the present invention will be described below with reference to the drawings. First, the configuration will be described.

<Structure> FIG. 13 is a block diagram showing the outline of the structure of a multimodal interface device according to the second embodiment of the present invention. As shown in FIG. Unit 1102, feedback generation unit 1103, output unit 1104, arrangement information storage unit 11
05 and a control unit 1106.

The input unit 1101 receives an audio signal from a user of the multimodal interface device,
Alternatively, it can capture an input such as an image signal or an operation signal at any time, such as a microphone that captures a voice input from a user, a camera that observes a user's operation or facial expression, or a user's eye. An eye tracker that detects movement, a head tracker that detects movement of the head, a motion sensor that detects movement of part or all of the body such as the user's hands and feet, or a user approaching, leaving, sitting, etc. It comprises at least one of an interpersonal sensor and the like to be detected.

When assuming a voice input as an input from the user, the input unit 1101 is composed of, for example, a microphone, an amplifier, an analog / digital (A / D) converter, and the like. When an image input is assumed as an input from the user, the input unit 1
The 101 is constituted by, for example, a camera, a CCD device (solid-state imaging device), an amplifier, an A / D converter, an image memory device, and the like.

[0262] The recognizing unit 1102 is provided with an input unit 1101.
It analyzes the input signal input from the OPC at any time, and outputs a recognition result by, for example, extracting a temporal section or a spatial section of the input intended by the user, or performing collation processing with a standard pattern.

More specifically, the recognition unit 110
2 detects a speech section by calculating power per time, for example, with respect to a speech input, performs frequency analysis by, for example, a method such as FFT (Fast Fourier Transform), and performs, for example, HMM (Hidden Markov Model). )
The recognition result is output by collation discrimination processing using a neural network or the like, or collation processing with a speech dictionary that is a standard pattern using a method such as DP (dynamic programming).

For image input, for example, “Un
calibrated Stereo Vision
with Pointing for a Man-M
Achine Interface "(R. Cipol
la, et. al. , Proceedings of
MVA'94, IAPR Works on Ma
chine Vision Application,
pp. 163-166, 1994. ), The region of the user's hand is extracted, and the shape, spatial position, orientation, movement, and the like are output as a recognition result.

FIG. 14 shows an example of the internal configuration of the input unit 1101 and the recognition unit 1102 of the embodiment when image input is assumed.

In FIG. 14, reference numeral 1201 denotes a camera;
02 is an A / D converter, 1203 is an image memory, and the input unit 1101 is composed of these. Camera 1201
Captures the whole body of a user or a part such as a face or a hand, and outputs an image signal using, for example, a CCD element. A / D converter 1202
Converts an image signal obtained from the camera 1201 to a digital image signal such as a bitmap. The image memory 1203 records a digital image signal obtained from the A / D converter 1202 as needed.

In FIG. 14, reference numeral 1204 denotes an attention area estimation unit; 1205, a recognition dictionary storage unit; 1206, a collation unit;
Is composed.

Of the components of the recognizing unit 1102, the attention area estimating unit 1204 refers to the contents of the image memory 1203, and uses, for example, a difference image or an optical flow to obtain, for example, the face, eyes, and mouth of the user. , Or attention area information such as a hand or an arm performing a gesture input. The recognition dictionary storage unit 1205 stores a representative image to be recognized, abstracted feature information, and the like as a standard pattern prepared in advance. Also, the matching unit 1206 refers to the image memory 1203, the contents of the attention area information obtained from the attention area estimation unit 1204, and the contents of the recognition dictionary storage unit 1205, and performs, for example, pattern matching, DP
(Dynamic programming), HMM (Hidden Markov Model), neural network, and other methods are used to compare and collate the two, and output a recognition result.

The operation states of the attention area estimation unit 1204 and the comparison unit 1206 are stored in the control unit 1 as operation state information.
106 is notified at any time. Further, the attention area estimation unit 1204 and the collation unit 1206 can be realized as the same module that performs both processes collectively.

The above is the description of the input unit 1101 and the recognition unit 1102.
It is the details of.

Returning to the configuration of FIG. 13, the description will be continued. The feedback generation unit 1103 in FIG.
Generates information to be presented as feedback to the user, for example, to alert the user, to notify the operating status of the system, to select a warning sound, a character string, an image prepared in advance, or , A dynamic waveform, or a speech waveform generated from a character string to be presented using a synthetic speech technique, or the "multimodal dialogue apparatus and multimodal dialogue method" described in the first embodiment. , Or a body motion generation device and a body motion control method proposed by the present inventors and applied for a patent (Japanese Patent Application No. 8-5796).
7)), for example, using CG (computer graphics) to face the user and provide services such as "human", "animal", or "robot". The generated character generates a still image or a moving image expressing, for example, facial expressions, gestures, and hand gestures.

The output unit 1404 includes, for example, a lamp, a CRT display, an LCD (liquid crystal) display, a plasma display, a speaker, an amplifier, and an HMD.
(Head Mount Display), Power Display,
It is composed of at least one output device such as a headphone, an earphone, etc., and presents the feedback information generated by the feedback generation unit 1103 to the user.

Here, the feedback generation unit 11
In the case where a multi-modal interface device that generates an audio signal at 03 is realized, the output unit 1104 is provided by an output device for outputting an audio signal, such as a speaker.
When the feedback generation unit 1103 implements, for example, a multi-modal interface device that generates an anthropomorphic image, for example, C
The output unit 1104 is configured by the RT display.

The arrangement information storage unit 1105 obtains position information, which is information relating to the spatial position of the pointing gesture input by the user, the user, the user's face, hands, and the like, and obtains the spatial information of the input unit. At least one of the position, the information on the spatial position of the output unit, and the information on the spatial position of the user is held as arrangement information, and at least one of the position information, the arrangement information, and the operation state information For example, this method is used when a method of presenting feedback to the user, such as presenting an expression to be watched at any time, is performed on the reference object that is the target of the pointing gesture of the user.

The arrangement information storage unit 1105 includes an output unit 1104 that is referred to when generating feedback information to be presented to the user when the device accepts a pointing gesture to the real world from the user. Output unit 1 for calculating direction information required when pointing from a spatial position of
Information such as the spatial position of 104 or the arrangement direction (spatial position information or direction information referred to when generating feedback information to be presented to the user,
01, the spatial position of the reference destination intended by the user included in the reference object position information recognized and output by the recognizing unit 1102 from the spatial position of the output unit 1104. Output unit 1 for calculation
104, or information such as the arrangement direction).

FIG. 15 shows an example of the contents held in the arrangement information storage unit 1105.

Each entry of the arrangement information storage unit 1105 as an example shown in FIG. 15 includes the designated place obtained by the recognition unit 1102 which is a component of the present apparatus, the designated object, the position of the hand and face of the user, and Information regarding the position and direction of the reference destination of the pointing gesture is classified into “label information A”, “representative position information B”, “direction information C”, and the like, and is recorded as needed.

Here, in each entry of the arrangement information storage unit 1105, a label for identifying the place or object where the position information and the direction information are recorded in the entry is recorded in the column of “label information A”. Is done. In the column of "representative position information B", the position (coordinate) of the corresponding place or thing is recorded. In the column of "direction information C", a value of a direction vector for expressing the direction of the corresponding place or object is recorded as necessary.

[0279] The "representative position information B" and "direction information C" are described based on a predetermined coordinate system (world coordinate system).

In each entry of FIG. 15, the symbol "-" indicates that the content of the corresponding trouble is empty, and the symbol "-" is a symbol in which unnecessary information is omitted in the description of the present embodiment. The symbol ":" indicates that unnecessary entries are omitted in the description of the present invention (the same applies hereinafter).

Also, the control unit 1106 in FIG.
Input unit 1101 and recognition unit 110 in the system of the present invention
2. It controls the operation of each component such as the feedback unit 1103, the output unit 1104, and the arrangement information storage unit 1105, and controls the exchange of information input and output between these components.

In the present system, the control unit 110
Since the operation 6 plays an important role in realizing the system of the present invention, this operation will be described later in detail.

The above is the device configuration of this system and its functions. Next, the flow of processing of the system of the present invention performed under the control of the control unit 1106 will be described.

<Contents of Control by Control Unit 1106> The flow of processing of the system of the present invention under the control of the control unit 1106 will be described. From here on, as shown in FIG. 14, the input unit 1101 includes an image input unit by the camera 1201 and, for example, “Uncali
bladed Stereo Vision with
Pointing for a Man-Machin
e Interface "(R. Cipolla, e
t. al. , Proceedings of MVA
'94, IAPR Works on Mac
hine Vision Application, p
p. 163-166, 1994. ), The user's pointing gesture to the real world place or object is recognized, and the position of the user's pointing gesture reference target, and the position and orientation of the user's face are output. An anthropomorphic image presentation unit 1 having a recognition unit 1102 and using, for example, the “multimodal interaction apparatus and multimodal interaction method” described in the first embodiment.
03 or a technique already applied for a patent, “Body motion generation device and body motion motion control method (Japanese Patent Application No. Hei 8-
No. 57967)), using, for example, CG (computer graphics) to designate a person, such as a human, an animal, or a robot, who faces the user and provides services. A feedback generation unit 1103 that generates a still image or a moving image, such as a facial expression with a gaze in the specified direction, a facial expression or gesture representing “surprise” or “apology”, or a facial expression or action of a personification agent having a gesture. An embodiment of the present invention will be described using a multi-modal interface device having an output unit 1104 having at least one CRT display as an example.

Control Unit 11 in the Second Embodiment System
06 is “<procedure AA>”, “<procedure BB”
>","<ProcedureCC>","<procedureDD>",
And the control operation according to the processing according to “<procedure EE>”.

Here, “<procedure AA>” is a “main routine of processing”, and “<procedure BB>” is
This is a processing procedure of “determining whether or not the gesture input position of the user can be watched from the anthropomorphic agent”. “<Processing procedure CC>” is “when the presentation position Lc of a certain anthropomorphic agent is assumed, This is a procedure for “determining whether or not the user can observe the anthropomorphic agent”. “<Processing procedure DD>” is “when the presentation position Ld of a certain anthropomorphic agent is assumed, From, a pointing gesture G currently focused on
Is a processing procedure for determining whether or not the pointing object R is gazeable. “<Processing procedure EE>” is an anthropomorphic agent expression that generates “an expression of an anthropomorphic agent gazing at the gazing object Z”. This is a generation procedure.

<Processing Procedure AA> [Step AA1]: It waits until the user detects the start of gesture input (Gi) from the operation status information of recognition section 1102, and if detected, proceeds to step (AA2).

[Step AA2]: “<procedure BB
> ”,“ The present presentation position L of the anthropomorphic agent
j, it is determined that the place Li where the gesture input Gi is performed can be watched by the anthropomorphic agent ", and the" presentation position L "is determined by"<processing procedure CC>".
It is determined that the user can observe the anthropomorphic agent presented in j. ", and the process proceeds to step AA6; otherwise, the process proceeds to step AA3.

[Step AA3]: arrangement information storage unit 1
105, “<processing procedure BB>” and “<processing procedure CC” for entries corresponding to all the presentation positions.
By performing the condition judgment using “>”, “the place where the gesture input Gi is performed can be watched by the anthropomorphic agent” and “the user can observe the anthropomorphic agent”. The presentation position Lk of such an anthropomorphic agent is searched.

[Step AA4]: If the presentation position Lk is found, the process proceeds to step AA5. If not, the process proceeds to step AA7.

[Step AA5]: The output unit 1104 is controlled to move the personification agent to the presentation position Lk.

[Step AA6]: The feedback generation unit 1103 and the output unit 1104 are controlled, and the location L where the gesture input is performed by “<processing procedure EE>”
A facial expression of the anthropomorphic agent gazing at i is generated and presented, and the process proceeds to step (AA12).

[Step AA7]: “<Processing procedure CC
>, It is checked whether or not "the user can observe the anthropomorphic agent". As a result, if observable, the process proceeds to step AA11; otherwise, the process proceeds to step AA8.

[Step AA8]: Arrangement information storage unit 1
105, a condition judgment using “<procedure procedure CC>” is sequentially performed on the entries corresponding to all the presentation positions, so that the user can observe the personification agent from the user. The presentation position Lm of the agent is searched.

[Step AA9]: When the presentation position Lm exists, the process proceeds to Step AA10, and otherwise, the process proceeds to Step AA12.

[Step AA10]: Output unit 1104
To move the personification agent to the presentation position Lm.

[Step AA11]: The feedback generation unit 1103 is controlled to generate an expression such as “Nodding” indicating that “the system is currently accepting a pointing gesture input from the user”, and output the expression. Part 1
104 is controlled and presented to the user.

[Step AA12]: If input unit 1
101 or the location Li where the gesture Gi is input, based on the operation status information obtained from the recognition unit 1102.
Proceeds to step AA13 if deviated from the observation range of the input unit 1101, otherwise proceeds to step AA1.
Proceed to 4.

[Step AA13]: The feedback generation unit 1103 is controlled to generate a facial expression such as “surprise”, for example, which indicates that the system has failed to analyze the input of the pointing gesture from the user, which is currently being received. Controlling the unit 1104 to present it to the user,
Proceed to 1.

[Step AA14]: Recognition Unit 1102
If the end of the gesture input Gi input by the user is detected from the operation status information obtained from the above, the process proceeds to step AA15, and if not, the process proceeds to step AA26.

[Step AA15]: Recognition unit 1102
If the recognition result of the gesture input Gi obtained from is a pointing gesture (pointing gesture), the process proceeds to step AA16; otherwise, the process proceeds to step AA21.

[Step AA16]: "<Processing procedure D
D> ”, the anthropomorphic agent determines that the pointing target Rl of the pointing gesture Gi can be watched, and“ <procedure CC> ”determines that the user can observe the anthropomorphic agent. If so, the process proceeds to step AA20; otherwise, the process proceeds to step AA17.

[Step AA17]: Referring to the arrangement information storage unit 1105, conditions using “<procedure procedure DD>” and “<procedure procedure CC>” are sequentially applied to entries corresponding to all presentation positions. By making the determination, the presentation position Ln of the anthropomorphic agent is searched from the anthropomorphic agent such that the pointing target R1 of the pointing gesture Gi can be watched and the user can observe the anthropomorphic agent.

[Step AA18]: If the presentation position Ln exists, the process proceeds to Step AA19, and if not, the process proceeds to Step AA21.

[Step AA19]: Output unit 1104
To move the anthropomorphic agent to the presentation position Ln.

[Step AA20]: “<procedure E
Using E>", the feedback generation unit 1103 is controlled to generate an anthropomorphic agent expression for gazing at the reference destination Rl of the gesture Gi, and the output unit 1104 is controlled and presented to the user, and the process proceeds to step AA1.

[Step AA21]: "<procedure C
C> ”, it is checked whether or not“ the user can observe the anthropomorphic agent ”. As a result, if observable, the process proceeds to step AA25; otherwise, the process proceeds to step AA22.

[Step AA22]: By referring to the arrangement information storage unit 1105 and sequentially performing a condition judgment using “<processing procedure CC>” for the entries corresponding to all the presentation positions, use is made. The presentation position Lo of the anthropomorphic agent that enables the person to observe the anthropomorphic agent is searched.

[Step AA23]: If the presentation position Lo exists, the process proceeds to Step AA24; otherwise, the process proceeds to Step AA1.

[Step AA24]: Output unit 1404
To move the anthropomorphic agent to the presentation position Lo.

[Step AA25]: Next, the control unit 11
06 controls the feedback generation unit 1103 to generate a facial expression such as “nodding” indicating that “the system is currently accepting the pointing gesture input from the user”, and controls the output unit 1104 for use. To the person,
The process returns to step AA1.

[Step AA26]: Control unit 1106
If it is determined from the operation status information obtained from the recognizing unit 1102 that the analysis of the gesture input during the input reception from the user has failed, the process proceeds to step AA27; otherwise, the process proceeds to step AA12.

[Step AA27]: Control unit 1106
Controls the feedback generation unit 1103 to generate a facial expression such as “apology” indicating that the system failed to analyze the gesture input from the user, and further to the output unit 110
4 and presents it to the user, and returns to step AA1.

FIG. 17 is a flowchart showing the above-mentioned “<processing procedure AA>” by the control unit 1106. The arrow line with the symbol “T” indicates that the branch condition is satisfied. It is assumed that the arrow line with the symbol “F” represents the branch direction when the branch condition is not satisfied. 18 to FIG.
Part of the flowchart of FIG.

Next, “<procedure BB>” will be described. In the "<procedure procedure BB>", by performing the following procedure, when the presentation position Lb of a certain anthropomorphic agent is assumed, a gesture input G such as a tip of a user's finger is input from the anthropomorphic agent. It is determined whether or not the position Lg where is performed can be watched.

<Processing Procedure BB> [Step BB1]: The control unit 1106 refers to the arrangement information storage unit 1105, and obtains “entry Hb” corresponding to the presentation position Lb.

[Step BB2]: By referring to the arrangement information storage unit 1105 and checking the column of the label information A, an “entry Hg” corresponding to the position G where the gesture is performed is obtained.

[Step BB3]: “Entry Hb”
And the “entry Hg” are obtained, the control unit 1106 stores the “entry Hg” stored in the arrangement information storage unit 1105.
b ”(Xb, Yb, Zb) of“ representative position information B ”,
And the value of the “direction information C” (Ib, Jb, Kb) and the value of the “representative position information B” of the “entry Hg” (X
g, Yg, Zg), and the vector (Xb−Xg, Y
b-Yg, Zb-Zg) and vector (Ib, Jb, K
Calculate the value Ib of the inner product of b).

[Step BB4]: Then, the control section 1
Then, 106 checks whether the value Ib of the inner product, which is the calculation result, is a positive value or a negative value. If the value is a positive value, the presentation position Lb corresponding to the “entry Hb” is determined. Is determined that the position Lg where the gesture G corresponding to “entry Hg” is performed is “gazeable”, and if the position Lg is negative, “gaze impossible”
Is determined to be.

As described above, the processing of “determining whether or not the gesture input position of the user can be watched from the anthropomorphic agent” can be performed.

Similarly, if the presentation position Lc of a certain anthropomorphic agent is assumed, whether or not the user can observe the anthropomorphic agent is determined by the following “<procedure CC>”.

<Processing Procedure CC> [Step CC1]: The control unit 1106 refers to the arrangement information storage unit 1105 and obtains “entry Hc” corresponding to the presentation position Lc.

[Step CC2]: arrangement information storage unit 1
The “entry Hu” corresponding to the position of the user's face is obtained by checking the contents of the label information A with reference to 105.

[Step CC3]: “Entry Hc”
And the “entry Hu” are obtained.
06 is “entry H” based on the arrangement information storage unit 1105.
c ”(Xc, Yc, Zc) of“ representative position information B ”;
And the value (Ic, Jc, Kc) of “direction information C” and the value (X of “representative position information B” of “entry Hu”
u. Yu. Zu) and the vector (Xc-Xu, Y
c-Yu, Zc-Zu) and vector (Ic, Jc, K
Calculate the value Ic of the inner product of c).

[Step CC4]: Next, the controller 110
6 determines whether the value of the inner product Ic is a positive value or a negative value.
The anthropomorphic agent presenting at the presentation position Lc corresponding to c "determines that" observable from the user ", and if negative, determines that" observable ".

Similarly, the following “<procedure DD>”
Accordingly, it is determined whether or not, assuming the presentation position Ld of a certain anthropomorphic agent, the anthropomorphic agent can gaze at the pointing target R of a certain pointing gesture G that is currently focused on.

<Processing Procedure DD> [Step DD1]: The control unit 1106 refers to the arrangement information storage unit 1105 and obtains “entry Hd” corresponding to the presentation position Ld.

[Step DD2] Also, by referring to the arrangement information storage unit 1105 and examining the contents of “label information A”, “entry Hr” corresponding to “instruction target R” is obtained.

[Step DD3]: “Entry Hd”
And “entry Hr” are obtained, the control unit 1106
Is the value of the “representative position information B” of the “entry Hd” (Xd,
Yd, Zd) and the value of “direction information C” (Id, J
d, Kd) and the value (Xr, Yr, Zr) of the “representative position information B” of the “entry Hr”, and the vector (Xd−Xr, Yd−Yr, Zd−Zr) and the vector (Id, Id, Jd, Kd) is calculated.

[Step DD4]: Next, the controller 110
6 determines whether the obtained inner product value Id is a positive value or a negative value. As a result, if it is positive,
From the anthropomorphic agent presenting at “presentation position Ld” corresponding to “entry Hd”, “reference destination R” of pointing gesture G corresponding to “entry Hr” is “gazeable”
Is determined, and if negative, it is determined that “gazing is impossible”.

Further, according to the following “<procedure EE>”, when a certain presentation position Le is assumed by the feedback generation unit 1103, the anthropomorphic agent can, for example, perform a gesture or a pointing operation. A facial expression of an anthropomorphic agent that gazes at “gaze target Z”, such as a gesture reference destination, is generated.

<Processing Procedure EE> [Step EE1]: The control unit 1106 refers to the arrangement information storage unit 1105 and obtains “entry He” corresponding to the presentation position Le.

[Step EE2] Also, by referring to the arrangement information storage unit 1105 and examining the contents of “label information A”, “entry H” corresponding to the watch target z is obtained.
z ″.

[Step EE3]: Next, the control unit 110
6 is the value (X) of the “representative position information B” of the “entry He”.
e, Ye, Ze) and the value (Xz, Yz, Zz) of the “representative position information B” of the “entry Hz”, and the vector Vf = (Xe−Xz, Ye−Yz, Ze−Ze). obtain.

[Step EE4]: “Entry He”
And “vector Vf” are obtained, the control unit 110
Next, when the reference direction of the presentation position Le obtained from the “direction information C” of the “entry He” is set to the front, an anthropomorphic agent creates a facial expression pointing in the direction of “vector Vf”. The technique disclosed in, for example, “Body motion generation device and body motion motion control method (Japanese Patent Application No. 8-57967)” proposed by the present inventors and applied for a patent can be applied to such expression creation. is there.

In this manner, the control unit 1106 determines whether or not the gesture input position of the user can be watched from the anthropomorphic agent, and when the presentation position Lc of a certain anthropomorphic agent is assumed, the control unit 1106 It is determined whether or not the anthropomorphic agent can be observed, and assuming a presentation position Ld of a certain anthropomorphic agent, the anthropomorphic agent can gaze at the pointing target R of a certain pointing gesture G that is currently focused on. Is determined, and if gaze is possible, a facial expression of an anthropomorphic agent that gazes at the gaze target Z is generated. In addition, when gaze is impossible or recognition fails, a gesture anthropomorphic agent that indicates the fact is displayed.

The above is the configuration and functions of the multi-modal interface device and the multi-modal interface method according to the present invention and the flow of main processing. Next, the operation of the multi-modal interface device according to the present invention will be described in more detail using a specific example with reference to the drawings.

<Specific operation example of the second specific example> [0338] Here, the input unit 1101 using a camera and image recognition technology use a hand gesture for pointing and pointing the user's face and pointing. A recognition unit 1102 that obtains information on the position, direction, and position of a reference destination that is being performed, and a feedback generation unit 11 that generates a CG of an anthropomorphic agent that is important for advancing a natural conversation between the user and the system.
03, and a specific operation will be described with a setting in which the user points and performs a gesture input toward a multimodal interface device according to the second embodiment of the present invention, which has two display devices as the output unit 1104.

FIG. 16 is a diagram for explaining the situation of this operation example. In FIG. 16, X, Y, and Z represent coordinate axes in the world coordinate system. Further, P1, P2, P3 to P9 are locations, respectively. Of these, location P1 (coordinate of P1 = (10, 20, 40)) represents a representative position of “presentation location 1”, Arrow V1 drawn from location P1
(Coordinate of the tip end position of V1 = (10, 0, 1)) is a vector representing the normal direction of “presentation location 1”.

Similarly, location P2 (coordinate of P2 = (− 2
0, 0, 30)) represents the representative position of the “presentation position 2”, and the arrow V2 (the tip position coordinate of V2 = (10, 10, −1)) drawn from the place P2 is “presentation position”. This is a vector representing the normal direction of location 2 ″.

The location P3 (coordinate of P3 = (40,3
(0, 50)) represents the representative position of the current user's face obtained from the recognition unit 1102, and is indicated by an arrow V3 (tip position coordinates of V3 = (− 4, −3, −)) drawn from the place P3. 1
0)) is a vector representing the orientation of the user's face. Further, a location P4 (coordinates of P4 = (40, 10, 20))
Represents the tip position of the finger when the user points and makes a gesture at a certain time point (T2 to T8),
V4 drawn from the place P4 (the tip position coordinates of V4 = (−
1, -1, -1)) is a vector representing the direction of the pointing gesture.

The location P5 (coordinate of P5 = (20, 1
(0, 20)) represents the tip position of the finger when the user points and makes a gesture at a certain point in time (T14 to T15), and V5 (tip position coordinates of V5 = (tip position of V5) drawn from the place P5) -1, -1, -1)) is a vector representing the direction of the pointing gesture.

The location P8 (the coordinates of P8 = (30,
0, 10)) represents a representative position of the “object A” as a pointing target of the pointing gesture performed by the user at a certain time point (T2 to T8). In addition, place P9 (P9
Coordinates = (0, -10, 0)) at a certain time point (T14 to
At T15), the representative position of “object B”, which is the pointing target of the pointing gesture performed by the user, is shown.

The information on the representative position and the direction described above is prepared in advance, or is detected by the recognition unit 1102 that analyzes the image information or the like obtained from the input unit 1101, and is recorded in the arrangement information storage unit 1105 as needed. I am trying to.

Next, description will be made along the flow of processing.

<Processing Example 1> Here, a description will be given of a processing example for presenting, to the user, an expression of an anthropomorphic agent who gazes at a reference destination as feedback information when the user performs a pointing gesture input. I do.

[T1]: First, it is assumed that the anthropomorphic agent is displayed at “presentation place 1” corresponding to place P1.

[T2]: Here, the user is “object A”
It is assumed that the pointing gesture (G1) has been started.

[T3]: The recognizing unit 1102, which analyzes the input image from the input unit 1101, detects the start of the gesture G1, and notifies the control unit 1106 as operating status information.

[T4]: The control section 1106 advances the processing from step AA1 of "<procedure AA>" to AA2.

[T5]: The control unit 1106 proceeds to step A
In the process of A2, first, “entry Q1” and “entry Q4” of the arrangement information storage unit 1105 shown in FIG.
, It is determined that the position P4 where the gesture G1 is performed can be watched from the present position P1 of the anthropomorphic agent.

[T6]: Also, “entry Q1” and “entry Q” in the arrangement information storage unit 1105 shown in FIG.
Through the processing based on “<procedure procedure CC>” with reference to “3”, it becomes clear from the current user face position P3 that the current presentation position P1 of the anthropomorphic agent can be observed.

[Step T7]: Next, the controller 1106
Goes to the processing of step AA6, and “<procedure EE
By performing the processing based on >>, the feedback generation unit 1103 generates a facial expression of an anthropomorphic agent that gazes at the gesture G1 currently performed by the user, and presents the expression to the user via the output unit 1104.

According to the above processing, when the user starts gesture input, the expression of the anthropomorphic agent who gazes at the hand or finger of the user who is performing the gesture input is presented to the user as feedback information. You can do it.

[T8]: Next, the control unit 1106 proceeds to the process of step AA12. Here, it is determined whether or not the gesture G1 is out of the observation range of the input unit 1101.

The gesture G1 does not deviate from the observation range of the input unit 1101.
It is assumed that the vehicle has proceeded to A14.

[T9]: The control unit 1106 proceeds to step A
In A14, it is determined from the operation status information of the recognition unit 1102 whether or not the user's gesture has instructed termination.
Now, it is assumed that the end of the gesture G1 has been notified from the recognition unit 1102 as operation status information. Therefore, in this case, the control unit 1106 recognizes the end of the gesture G1.

[T10]: Next, the control unit 1106 proceeds to the process of step AA15. In this process, it is determined whether the gesture is a pointing gesture. In this case, since the gesture G1 is a pointing gesture, the process proceeds to step AA16 based on the operation status information obtained from the recognition unit 1102.

[T11]: In the process of step AA16, the control unit 1106 first reads “entry Q1” and “entry Q” in the arrangement information storage unit 1105 shown in FIG.
Then, the process based on “<procedure D>” referring to “8.” Then, it is known that “object A”, which is the indication target of the gesture G1, can be watched from the anthropomorphic agent.

[T12]: Also, “entry Q1” and “entry Q” in the arrangement information storage unit 1105 shown in FIG.
It is also found that the user can observe the anthropomorphic agent by the processing based on “<procedure CC” referring to “3”, and the process proceeds to step AA20.

[T13] In step AA20,
The control unit 1106 controls the arrangement information storage unit 110 shown in FIG.
5 performs a process based on “<procedure EE>” with reference to “entry Q1” and “entry Q8”, whereby an agent expression that gazes at a place P8 of “object A” to which the gesture G1 is referred Is presented to the user. Then, the process returns to step AA1.

According to the above-described processing, when the user performs the pointing gesture input, the expression of the anthropomorphic agent watching the reference destination can be presented to the user as the feedback information.

Next, another processing example with different conditions will be described.

<Processing Example 2> [T21]: From user, "object B" at location P9
It is assumed that the input of the pointing gesture G2 has begun.

[T22]: An anthropomorphic agent expression that gazes at the gesture G2 is presented to the user by the same processing as the processing in steps T2 to T7.

[T23]: In step AA16, first, referring to “entry Q1” and “entry Q9” in the arrangement information storage unit 1105 shown in FIG.
By the processing based on B>", it is determined that the position P9 where the gesture G2 is performed cannot be watched from the present position P1 of the anthropomorphic agent.

[T24]: At step AA17, entry Q1 in arrangement information storage 105 shown in FIG.
And the entries corresponding to all the presentation positions such as the entry Q2 are determined by the processing based on “<processing procedure DD>”, so that the anthropomorphic agent can gaze at the object B, which is the pointing target of the gesture G1, and An observable presentation position is searched from the user's position P3, and a place P2 corresponding to the presentation position 2 is obtained.

[T25]: Proceed to step AA19,
Place the anthropomorphic agent at the location P2 through the output unit 1104
To step AA20.

[T26]: By the same processing as in T13, the expression of the anthropomorphic agent gazing at the "object B" to be pointed is presented to the user as feedback to the gesture G2.

As a result of the above processing by the control unit 1106,
Even when the reference point of the pointing gesture made by the user is in a place where the anthropomorphic agent cannot gaze, the anthropomorphic agent is moved to an appropriate position to provide appropriate feedback to the user. It becomes possible.

In addition, when the anthropomorphic agent cannot gaze at the gesture input made by the user, the anthropomorphic agent is moved to an appropriate position by the processing of step AA3, so that appropriate feedback is provided to the user. It can be presented. If such movement is not possible, steps AA7 to AA11
, The expression of “nodding” is presented as feedback.

If the hand performing the gesture input is out of the field of view of the camera during the gesture input performed by the user, for example, the processing of steps AA12 to AA13 indicates “surprise”. The "expression" is presented to the user as feedback.

Also, when the gesture input by the user is of a type other than the pointing gesture, the display position of the anthropomorphic agent is moved by the processing of steps AA21 to AA25 as necessary.
The expression "nodding" is presented as feedback. Also, when the gesture input by the user fails to be recognized, the expression of “apology” of the anthropomorphic agent is presented to the user as feedback by the processing of step AA27.

Thus, according to the present device configured as described above, the user can perform a pointing gesture remotely from a remote position, without touching the device, and without mounting the device. It is possible to suppress erroneous recognition and gesture extraction failure that occur due to insufficient accuracy of the gesture recognition method.

Also, at the point in time when the user starts a gesture intended to be input or in the middle of inputting, it is not known whether or not the system has correctly extracted the gesture input. Alternatively, it is possible to reduce the burden on the user which occurs when the user has to perform the input again.

Also, in response to a pointing gesture input from a user for referring to a place or thing in the real world,
As the pointing destination, it is possible to appropriately display which place, which object or which part thereof has been received. In addition, correction of the effects of malfunctions, which are problems of the conventional method induced by the aforementioned problems,
Alternatively, the burden on the user caused by the input again and the burden on the user due to anxiety at the time of the input by the user can be eliminated.

Further, in the interface device and the interface method using the anthropomorphic interface, it is possible to generate an appropriate facial expression of an agent in consideration of the field of view of the user and the anthropomorphic agent from the anthropomorphic agent and to present it as feedback. It becomes possible.

The embodiments of the multimodal interface device and the multimodal interface method according to the present invention are not limited to the above-described examples.
For example, in the above-described embodiment, recognition processing of the position and orientation of the user's gesture, face, and the like is performed from the image captured using the camera.
It can also be realized by a method using a magnetic sensor, an infrared sensor, a data glove, a data suit, or the like. Further, in the above-described embodiment, the feedback of the pointing destination is realized by the expression of the gaze of the anthropomorphic agent. For example, the feedback of the pointing destination is performed by, for example, performing an operation of pointing the pointing target by hand by the anthropomorphic agent. It is also possible to realize.

Also, in the above-described embodiment, the input of the pointing gesture by pointing to one place has been described as an example. However, for example, a circling gesture by an operation of surrounding an area having a certain spread in the space may be used. Thus, for example, it is also possible to adopt a configuration in which the fingertip performing circling is fed back by the anthropomorphic agent gazing at any time.

Further, in the above-described embodiment, an entry relating to, for example, the output unit in the contents of the arrangement information storage unit is prepared in advance. It is configured so that the contents of the arrangement information storage unit are dynamically updated when the output unit or the position of the user is changed, if necessary, by observing changes in the surrounding environment using the input unit or the like. It is also possible.

Further, in the above-described embodiment, the anthropomorphic agent is configured to gaze at the object pointed by the gesture pointed by the user, so that the system cannot recognize the object or the recognition result on the system cannot be obtained. Although the user was intuitively aware of whether or not there was a mistake, the conversely, for example, when an anthropomorphic agent tells the user the physical location of the floppy drive, for example, By displaying the agent so as to look in the direction, it is also possible to configure so that the user can easily recognize the position of the target by the instruction by the look of the anthropomorphic agent.

Alternatively, in the above embodiment, for example,
From the user or the anthropomorphic agent, whether a certain position can be watched or observed is determined by the positional relationship with a plane perpendicular to their direction vector,
For example, the configuration may be such that the determination is performed based on a conical area, or the determination is performed based on an area shape simulating an actual human visual field pattern. Alternatively, in the above-described embodiment, the embodiment using the anthropomorphic agent displayed on the CRT display has been described. However, for example, the present invention may be realized using an output unit using a three-dimensional display technology such as holography. It is possible.

The output section of the present invention can be realized by one display device, can be physically realized by using a plurality of display devices, or can be physically realized. Can be realized using a plurality of regions of one display device. Alternatively, for example, using a general-purpose computer as shown in FIG.
The present invention can also be realized by recording the data in an external storage medium such as a floppy disk, reading the data into a memory, and executing the read data in, for example, a CPU (Central Processing Unit).

As described above, the present invention shown in the second embodiment is a microphone for taking in a voice input from a user, a camera for observing a user's motion or facial expression, or an eye tracker for detecting a user's eye movement. , Or a head tracker that detects the movement of the head, or a motion sensor that detects the movement of part or all of the body such as hands and feet, or a data glove, or a data suit that is worn by the user and captures the movement, or An input unit that includes at least one of an interpersonal sensor and the like that detects approach, departure, and seating of the user, and that takes in an input from the user as needed and outputs it as input information; and an input obtained from the input unit. Receiving information, voice detection processing, voice recognition, shape detection processing, image recognition, gesture recognition, facial expression recognition, gaze detection processing, By There is subjected to at least one processing operation recognition, the input from the user,
"Accepted", "Accepted",
Input recognition means for outputting, as operation status information, information on the status of acceptance of input from the user, such as "recognition success" or "recognition failure"; a warning sound, a synthesized voice, a character string, an image, Alternatively, an output unit that presents the user with feedback using a moving image, and a control unit that presents feedback information to the user through the output unit in accordance with the operation status information obtained from the input recognition unit. It is a feature.

Alternatively, the input means is a camera (imaging device)
The user's image is captured by image acquisition means such as
As input information, for example, a unit that outputs analog-to-digital converted image information is used, and the input recognizing unit outputs the image information obtained from the input unit, such as a difference extraction from an image at a previous point in time or an optical flow. By applying the method, for example, a moving region is detected and, for example, a gesture input is extracted from an input image by performing collation by a method such as a pattern matching technique, and a progress status of each of the processes is output as motion status information as needed. The control unit outputs a character string, an image, a buzzer sound, a sound signal, or the like from an output unit such as a CRT display or a speaker, according to the operation status information obtained from the input recognition unit. This is characterized in that it is means for such control. Further, according to at least one of the input information obtained from the input means and the operation status information obtained from the input recognizing means, feedback information generation for generating feedback information to be presented as feedback to the user. Means. In addition, the person, creature,
Image information of still or moving images of an agent personified as a machine or robot,
Feedback information generating means to be generated as an anthropomorphic image to be presented to the user, and at least one of the facial expression or action of the anthropomorphic image to be presented to the user is determined according to the operation status information obtained from the input recognition means, Through the output means, for example, feedback information generation for generating feedback information which is a facial expression to which the user gazes, such as a pointing destination of a pointing gesture, or a part or part thereof where the user realizes the gesture expression, for example, a fingertip, face, or eyes. Means is further provided, and the control means is provided with a function of presenting the feedback information generated by the feedback information generating means to the user as feedback information from the output means to the user. Furthermore, a location information storage means for holding at least one of the spatial position of the input means, the information relating to the spatial position of the output means, and the information relating to the spatial position of the user as placement information is provided, and the input recognition means is provided. Is
In addition to providing a function of outputting position information indicating a spatial position such as a reference object of the pointing gesture input by the user, the user, the user's face or hand, the layout information obtained from the layout information storage means and the input A feedback unit that determines at least one of the operation of the anthropomorphic agent or an expression or control timing by referring to at least one of the position information and the operation status information obtained from the recognition unit, and outputs the determined information as feedback information; It is configured.

[0386] The present system having such a configuration is as follows.
A microphone that captures voice input from the user, a camera that observes the user's movements and facial expressions, an eye tracker that detects the user's eye movement, a head tracker that detects the movement of the head, hands, feet, etc. At any time, input from the user input from at least one of the input means such as a motion sensor that detects the movement of a part or the whole body or an interpersonal sensor that detects the approach, departure, seating, etc. of the user Capture, obtain as input information, and perform at least one of recognition processing of voice detection processing, voice recognition, shape detection processing, image recognition, gesture recognition, facial expression recognition, eye-gaze detection processing, or motion recognition. Information on the reception status in response to the input from the user, that is, reception is being performed, reception has been completed, recognition has been successful. Information or information on the status of reception of input from the user, such as the fact that recognition has failed, is obtained as operation status information, and based on the obtained operation status information, a warning sound, a synthesized voice, a character string, an image, or The video is presented to the user as feedback using a moving image.

Also, an operation in which image information in a still image or a moving image of an agent personified as a person, a creature, a machine, a robot, or the like who provides a service to face a user is obtained from feedback information recognition means. By generating as anthropomorphic image information to be presented to the user in accordance with the situation information and displaying this, for example, when a voice input is made, an anthropomorphic agent presents an expression such as "nodding", for example. Give feedback to the person.

[0388] Further, image recognition is performed by the recognition means to obtain position information which is information relating to the spatial position of the pointing gesture input by the user, the user, the user's face and hands, and the like. The spatial position of the input,
And information on the spatial position of the output unit, and holds at least one of the information on the spatial position of the user as arrangement information, position information, and arrangement information, according to at least one of the operation status information, for example, It provides feedback to the user, such as presenting an expression to look at the reference object that is the target of the pointing gesture of the user as needed.

As described above, the user can recognize the gesture by pointing to the position away from the system or the device in a non-contact state, and input the instruction, and perform the gesture recognition without erroneous recognition. Thus, it is possible to provide a multi-modal interface device and a multi-modal interface method that can eliminate failure in gesture extraction. Also, at the time when the user starts the gesture intended to be input or at the time when the input is being performed, the user can know whether or not the system has correctly extracted the gesture input. It is possible to provide a multi-modal interface device and a multi-modal interface method that can eliminate an indispensable burden. In addition, in response to a pointing gesture input from a user for referring to a place or thing in the real world, as a pointing destination,
It is possible to provide a multi-modal interface device and a multi-modal interface method capable of appropriately displaying which place, which object, or which part thereof has been received.

The present invention shown in the second embodiment can be applied as a method, and the processing procedures and flowcharts shown in the above-described specific examples are described as programs, implemented, and The same functions and effects can be obtained by executing the above-mentioned computer system. That is, in this case, as shown in FIG.
U 301, memory 302, large-capacity external storage device 30
3, a general-purpose computer including a communication interface 304 and the like, input interfaces 305a to 305n,
Input devices 306a to 306n, output interfaces 307a to 307m, and output devices 308a to 308n
308m, input devices 306a to 306n, such as a microphone, a keyboard, a pen tablet, an OCR, a mouse, a switch, a touch panel, a camera, a data glove, and a data suit, and as output devices 308a to 308m, CPU 3 using speakers, force display, etc.
01 by the software control, the above-described operation can be realized.

In other words, the methods described in the first and second embodiments can be executed by a computer as a program such as a magnetic disk (floppy disk, hard disk, etc.), an optical disk (CD-ROM, DV
D) can be stored and distributed on a recording medium such as a semiconductor memory. If the computer reads a program using the recording medium and causes the CPU 301 to execute the program, the multi-modal interactive device of the present invention is realized. You can do it.

[0392]

As described above, the present invention uses the technology of gaze detection and the like to determine whether or not to accept input from other media according to the user's gaze target, to perform recognition processing, or to present or suspend output. Simulates the use and role of non-verbal messages in human-to-human communication, such as allowing a person to start a conversation by looking at a face, especially in anthropomorphic interfaces. It is applied in such a way that
Therefore, according to the present invention, it is possible to realize a multi-modal interface that efficiently utilizes a plurality of input / output media, is efficient, effective, and reduces the burden on the user.

In addition, since the analysis accuracy of the input from each medium is insufficient, for example, in the case of occurrence of erroneous recognition due to ambient noise in voice input, or in the recognition processing of gesture input, it may be obtained from the input device every moment. An interface that does not cause a malfunction due to a failure to cut out a signal portion intended by the user as an input message from among the signals to be input can be realized. In addition, an interface using media that is used not only as an input to the computer, etc. that the user is currently operating, such as voice input and gesture input, but also used when talking to other people around, for example In the device, even if the user speaks or shows a gesture to another person besides the interface device instead of the interface device, for example, the user mistakenly judges that the interface device is an input to the user. , By performing recognition processing, etc., to cause a malfunction, cancel the malfunction, recover the effects of the malfunction, and eliminate the load that the user must constantly pay attention to avoid the malfunction Thus, the burden on the user can be reduced.

Further, since it is possible to prevent the processing of the input signal from being performed continuously in a scene that is not originally required, it is possible to improve the execution speed and the use efficiency of other services related to the apparatus in use. I can do it.

Also, no special operation for changing the input mode or the like is required, it is not complicated for the user, there is no need for learning and training, and the same natural conversation as between humans without burdening the user. Interface can be realized.

[0396] Also, for example, an interface that effectively utilizes the inherent advantages of voice media, that voice input does not disturb the work being performed by hand and that both can be used simultaneously can be realized. Can be done.

Also, when presenting information to a user using a temporary medium in which the information to be presented disappears immediately after presentation or changes every moment, the user fails to receive the information. Interface can be realized.

Also, when presenting information to a user using a transient medium, a special operation is also performed when presenting information for each amount that can be received at a time and presenting the next information to be continued. Can realize an unnecessary interface.

In addition, non-verbal messages such as gaze matching (eye contact), gaze position, gestures such as gestures and hand gestures, and facial expressions, which were impossible in the conventional multi-modal interface, can be effectively used.

That is, according to the present invention, it is possible to realize an interface that uses a plurality of input / output media efficiently, is efficient, effective, and reduces the burden on the user.

Further, the present invention enables a user to naturally and smoothly input voices and gestures intended for input by a user. When a gesture input from a user is detected, the anthropomorphic agent is activated. Depending on the facial expression, the user performs a gesture input at any time, or, for the pointing gesture, gazing at the reference target to present natural feedback to the user,
At this time, the anthropomorphic agent is controlled to be moved and displayed at an appropriate place in consideration of the view of the user and the anthropomorphic agent, or the spatial position of the reference target, etc. According to the present invention, it is possible for a user to perform a pointing gesture remotely without contacting a device at a remote location or with a device, and without wearing a device.
In addition, it is possible to suppress erroneous recognition and gesture extraction failure that occur due to insufficient accuracy of the gesture recognition method.

Also, at the point in time when the user starts a gesture intended to be input or in the middle of inputting, it is not known whether the system has correctly extracted the gesture input. It is possible to reduce the burden on the user caused by causing the user or having to perform the input again by the user. In addition, in response to a pointing gesture input from a user for referring to a place or thing in the real world, it is possible to appropriately display which place, which object, or which part has been received as the pointing destination. It becomes possible. Furthermore, it is possible to generate an appropriate expression of the agent in consideration of the field of view of the user and the field of view from the anthropomorphic agent, and to present it as feedback.

Further, the problem of the conventional method induced by the above-mentioned problem, such as the correction of the influence of the malfunction, the burden on the user caused by the re-input, and the user's anxiety at the time of the input by the user. Thus, a great effect can be obtained in practical use, such as the burden on the user can be eliminated.

[Brief description of the drawings]

FIG. 1 is a diagram for explaining the present invention, showing a configuration example of a multimodal device as one specific example of the present invention.

FIG. 2 is a diagram for explaining the present invention, showing an example of gaze target information output by the present invention device.

FIG. 3 is a diagram for explaining the present invention, showing a configuration example of another media input unit 102 in the device of the present invention.

FIG. 4 is a view for explaining the present invention, showing an example of a display screen including an output of the anthropomorphic image presentation unit 103 in the apparatus of the present invention.

FIG. 5 is a diagram for explaining the present invention, showing a configuration example of an information output unit 104 in the device of the present invention.

FIG. 6 is a diagram for explaining the present invention, showing an example of an internal configuration of a control unit 107 in the device of the present invention.

FIG. 7 is a diagram for explaining the present invention, showing an example of the contents of a control rule storage unit 202 in the device of the present invention.

FIG. 8 is a view for explaining the present invention, showing an example of the contents of an interpretation rule storage unit 203 in the apparatus of the present invention.

FIG. 9 is a diagram for explaining the present invention, showing a flow of a processing procedure A in the device of the present invention.

FIG. 10 is a diagram for explaining the present invention, illustrating the internal state of the device at each point in the device of the present invention.

FIG. 11 is a diagram for explaining the present invention, showing an image of a personification agent person as an example used in the personification image presentation unit 103 of the device of the present invention.

FIG. 12 is a diagram for explaining the present invention, and is a block diagram showing an example of a device configuration for realizing the present invention with a general-purpose computer.

FIG. 13 is a diagram for explaining the present invention, and is a block diagram showing a configuration example of a multimodal interface device according to a second embodiment of the present invention.

FIG. 14 is a diagram for explaining the present invention, and illustrates an input unit 11 in a second embodiment when image input is assumed.
FIG. 2 is a block diagram showing a configuration example of a recognition unit and a recognition unit.

FIG. 15 is a diagram for explaining the present invention, showing an example of contents held in an arrangement information storage unit 1105 according to the second embodiment of the present invention.

FIG. 16 is a diagram for explaining the present invention, and is an explanatory diagram of a situation showing an operation example in the second embodiment of the present invention.

FIG. 17 is a diagram for explaining the present invention, and is a flowchart showing an example of the content of “<processing procedure AA>” in the control unit 1106 according to the second embodiment of the present invention.

FIG. 18 is a diagram for explaining the present invention, showing a part of the flowchart of FIG. 17 in detail according to the second embodiment of the present invention;

FIG. 19 is a diagram for explaining the present invention, showing a part of the flowchart of FIG. 17 in detail according to the second embodiment of the present invention;

FIG. 20 is a diagram for explaining the present invention, showing a part of the flowchart of FIG. 17 in detail according to the second embodiment of the present invention;

[Explanation of symbols]

 101: Gaze target detection unit 102: Other media input unit 102a: Voice recognition device 102b: Character recognition device 102c: Language analysis device 102d: Operation input analysis device 102e: Image recognition device 102f: Gesture analysis device 102g: Microphone 102h: Keyboard 102i ... pen tablet 102j ... OCR 102k ... mouse 102l ... switch 102m ... touch panel 102n ... camera 102o ... data glove 102p ... data suit 103 ... personification image presentation unit 104 ... information output unit 104a ... character image signal generation device 104b ... audio signal generation Driving device 104c: device control signal generating device 105: alerting unit 106: reaction detecting unit 107: control unit 201: control processing executing unit 202: control rule storage unit 203: interpretation rule storage unit. 1101 input unit 1102 recognition unit 1103 feedback generation unit 1104 output unit 1105 placement information storage unit 1106 control unit 1201 camera 1202 A / D conversion unit 1203 image memory 1204 attention area estimation unit 1205 verification Unit 1206: recognition dictionary storage unit

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Takehide Yano 8-6-26 Motoyama Minamicho, Higashinada-ku, Kobe City, Hyogo Prefecture Inside the Toshiba Kansai Research Institute Co., Ltd. 8-6-26 Toshiba Kansai Research Institute Co., Ltd.

Claims (21)

[Claims] 1. A detecting means for detecting a gaze target of a user, and receiving at least one or more of voice input information, operation input information, image input information, and user operation information of the user, A control unit for selecting information necessary for controlling the target device from the information, a multimodal interface device. 2. An anthropomorphic image providing means for supplying an anthropomorphized agent image corresponding to a user; a detecting means for detecting a gaze target of the user; voice input information of the user; Input information, media input means for acquiring at least one or more input information among image input information, and a gaze target of the user is provided by anthropomorphic image presentation means based on gaze target information obtained by the detection means. Recognizing which part of the presented agent image,
Control means for receiving and selecting an input from the media input recognition means according to the recognition result and the state of the agent image. 3. An image of an anthropomorphized agent person who provides a service in a face-to-face manner with a user, wherein the agent person image is a non-verbal image by an image having a required gesture and facial expression change. Anthropomorphic image presenting means for presenting the image as a message, non-verbal communication to the user through the anthropomorphic image presenting means when the input from the media input recognizing means is selected by referring to the gaze target information. 3. The multimodal interface device according to claim 2, further comprising control means for performing control so as to present a signal based on a message. 4. A detecting means for detecting a gaze target of a user, an information output means for outputting voice information or operation information or image information to the user, a voice information, operation information or image for the user First control means for receiving at least one or more pieces of information from among the information and selecting information necessary for controlling the target apparatus from the information, and at least one information output means by referring to the gazing target information; A second control means for controlling an output start, end, interruption, restart, or adjustment operation of the presentation speed. 5. A detecting means for detecting a gaze target of a user, a media input means for acquiring at least one or more of voice input information, operation input information, and image input information of the user; An image of an anthropomorphic agent person providing a service in a face-to-face manner, wherein the agent person image is presented to the user as a non-verbal message with an image having a required gesture and facial expression change. Image presenting means, information output means for outputting information to the user by presenting at least one signal of character information, audio information, still image information, and moving image information; and An alerting means for alerting the user by presenting a non-verbal message; Of the input information, by referring to at least one of information to detect the reaction of the user with respect to the signal for the reminder,
Response detection means for outputting as user response information, and at least one media input processing means for accepting input, starting, ending, suspending, resuming, and processing level adjustment of at least one media input processing means according to the gaze target information. When controlling the operation and switching the acceptability of the input from the media input recognition means by referring to the gaze target information, presenting the character information, voice information, still image information, moving image information, or anthropomorphic image to the user Means for controlling the presentation of a non-verbal message to the user through the means, and referring to the watch target information, at least one of the information output means for starting, ending, suspending, resuming, and adjusting the processing level. When presenting information from the information output means, presents a signal for calling the user's attention through the attention calling means in accordance with the gaze target information. Control means for controlling the operation of the information output means and at least one of the alerting means according to the content of the user reaction information,
A multimodal interface device comprising: 6. A gaze target of a user is detected, and processing on at least one of the user's voice, gesture, operation information of the user by operation means, and the like is performed according to the gaze target information. , Selection of input reception, start, end, suspension, restart of processing or recognition operation,
A multimodal interface method, wherein an operation of adjusting a processing level is controlled. 7. An image of an anthropomorphic agent person who detects a gaze target of a user, changes in response to a situation, and provides a service facing the user, and presents the image to the user as image information. Further, based on the gaze target information, depending on which part of the agent person image the gaze target is, the voice of the user,
A multimodal interface method, wherein selection of acceptance of operation information of a user by a gesture and operation means is selected. 8. A system for providing a user with information by presenting at least one signal of character information, audio information, still image information, and moving image information, detects a user's gaze target, and detects the detected gaze. A multi-modal interface method comprising controlling the operations of starting, ending, suspending, resuming, and adjusting the processing level with reference to target information. 9. The multimodal interface method according to claim 8, wherein when presenting information, at least one of character information, voice information, still image information, and moving image information is provided to the user according to the gaze target information. A multimodal interface method characterized by calling a user's attention by presenting two signals. 10. The multimodal interface method according to claim 9, wherein a reaction of a user to a signal for alerting is detected,
In addition to obtaining user response information, the user's voice input information,
A multi-modal interface method for controlling at least one of acquisition of operation input information and image input information and alerting. 11. An anthropomorphic agent person image for detecting a user's gaze target and outputting as gaze target information, and providing a service to the user, wherein the agent person image is the user's Required gesture, a non-verbal message by an image having a change in facial expression, and, by presenting at least one signal of character information, audio information, still image information, and moving image information, information is provided to the user. When receiving and processing at least one of the user's voice input information, gesture input information, and operation input information, when performing processing, input acceptability, processing or recognition is performed according to the gaze target information. Start, end, suspend, resume,
A multi-modal interface method, wherein the operation of processing level adjustment is controlled. 12. The multimodal interface method according to claim 11, wherein when switching the acceptability of the input with reference to the gaze target information, the user is provided with character information, voice information, still image information, and moving image information. Alternatively, a multimodal interface method in which required presentation is performed using a personified image person image. 13. An input means for receiving various information from a user as needed to obtain input information, and processing the input information obtained from the input means to determine whether or not input from the user has been accepted. An input recognition unit that outputs information, an output unit that presents at least one of a warning sound, a synthesized voice, a character string, an image, and a moving image to a user as a reaction result, and an operation state obtained from the input recognition unit Depending on the information,
A control unit for controlling the output unit so as to obtain a reaction result to be presented to a user. 14. A microphone for taking in a voice input from a user, a camera for observing a user's movement or facial expression, an eye tracker for detecting a user's eye movement, or a head tracker for detecting a head movement. ,
Or, it consists of at least one of a motion sensor that detects the movement of part or all of the body such as the user's hands and feet, or an interpersonal sensor that detects the approach, departure, or seating of the user. An input means for taking in an input as needed and outputting it as input information; receiving input information obtained from the input means, and performing voice detection processing, voice recognition, shape detection processing, image recognition, gesture recognition, facial expression recognition, gaze detection processing, or operation An input recognizing unit that performs a necessary process corresponding to the type of the input information in the recognition process to output a reception status of an input from a user as operation status information; a warning sound, a synthesized voice, a character string, and an image Output means for presenting to the user as a response to the user by at least one of the following: operation state information obtained from the input recognition means; And Flip,
A control unit for controlling the output unit so as to obtain a reaction result to be presented to a user. 15. An input means for capturing an image of a user by an image obtaining means and outputting it as image information; extracting a gesture input from the image information obtained from the input means; An input recognizing unit that outputs as needed, and according to the operation status information obtained from the input recognizing unit,
Means for presenting a response to a user's input as a character string, an image, or a sound signal. 16. A microphone for taking in a voice input from a user, a camera for observing a user's movement and facial expression, an eye tracker for detecting a user's eye movement, or a head tracker for detecting a head movement. ,
Or, it consists of at least one of a motion sensor that detects the movement of part or all of the body such as the user's hands and feet, or an interpersonal sensor that detects the approach, departure, or seating of the user. An input means for taking in an input as needed and outputting it as input information; receiving input information obtained from the input means, and performing voice detection processing, voice recognition, shape detection processing, image recognition, gesture recognition, facial expression recognition, gaze detection processing, or operation Among the recognition processing, input recognition means for outputting a reception state of an input from a user as operation state information by performing necessary processing corresponding to the type of the input information, and input information obtained from the input means, and According to at least one of the operation status information obtained from the input recognition means, information to be presented as feedback to the user Multimodal interface device being characterized in that includes a feedback information generating means for forming, the. 17. An operation obtained from feedback information generating means for generating image information of an anthropomorphized agent person who provides a service facing a user as an anthropomorphic image for user presentation, and operation obtained from input recognition means. According to the situation information, at least one of the facial expression or the motion of the anthropomorphic image to be presented to the user is determined, and through the output means, the feedback information that is the facial expression that the user gazes at the part where the gesture expression is realized is provided. 17. The apparatus according to claim 16, comprising: feedback information generating means for generating; and control means for presenting the feedback information generated by the feedback information generating means as feedback information from an output means to a user. Multimodal interface device. 18. An arrangement information storage means for holding at least one of information on a spatial position of an input means, information on a spatial position of an output means, and information on a spatial position of a user as arrangement information; An input recognition unit that outputs position information representing a spatial position such as a reference object of the input pointing gesture, a user, a user's face or a hand, and arrangement information obtained from the arrangement information storage unit, and from the input recognition unit. Feedback means for determining at least one of the operation of the anthropomorphic agent, the facial expression, or the control timing by referring to at least one of the obtained position information and the operation status information, and outputting the information as feedback information. The multimodal interface device according to claim 17, wherein: 19. An image of a voice from a user, an image showing a user's motion or facial expression, or a detection of information such as movement of a part or whole of the user's body, approaching, leaving, or sitting on the user. An input step consisting of at least one of the above, and outputting as input information from a user; based on the input information obtained by the input step, a voice detection process, a voice recognition, a shape detection process, an image recognition, a gesture recognition, An input recognition step of performing a process corresponding to the type of input information among facial expression recognition, gaze detection processing, or motion recognition processing to output a reception status of an input from the user as motion status information; Based on the operation status information obtained by the input recognition step, the result of the response to the input from the Multimodal interaction method characterized by comprising the steps of: presenting a more user. 20. Anthropomorphic image information of an anthropomorphized agent person who provides a service facing a user as anthropomorphic image information to be presented to a user in accordance with operation status information obtained from an input recognition step. 20. The method according to claim 19, further comprising: a feedback information generating step of generating; and a user feedback presenting step of presenting a facial expression of a response by the anthropomorphic agent as a response when an input from the user is made. Multi-modal interface method. 21. Voices from the user, images showing the user's movements and facial expressions, etc., or movements of part or all of the user's body, or detection information of the user's approach, departure, seating, etc. An input step of outputting at least one of the input information from the user, and a voice detection process, a voice recognition, a shape detection process, an image recognition, a gesture recognition, a facial expression recognition, By performing processing according to the input information type in the eye gaze detection processing or the motion recognition processing,
An input recognition step of outputting an input reception status from the user as operation status information; and a recognition step of outputting information relating to a reference object of the pointing gesture input by the user, the user, and a spatial position of a specific part of the user. A step, an arrangement information storage step of storing at least one of information on a spatial position of the input unit, information on a spatial position of the output unit, and information on a spatial position of the user as arrangement information; and an input from the user. The result of the reaction to is presented to the user as a warning sound, a synthesized voice, a character string, or an image based on the operation status information obtained by the input recognition step, and the position information, the arrangement information, and the operation status are displayed. Presenting a reaction image to the user's pointing gesture according to at least one of the information. Le interface method.