JPH1124813A - Multi-modal input integration system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To carry out the desired processing in an input mode that can eliminate the user burden to an information device in a multi-modal input integration system which contains a user interface to deal with the information device. SOLUTION: An input part 1 recognizes the contents which are inputted via an input means having plural modes and outputs the result of recognition. A planner part 2 receives the words of recognition result from the part 1 and converts the inputted words into a generalized command script. Then the command script sent from the part 2 is interpreted and executed at an agent part 4. Thus, the inputs of various types are integrated into a significant meaning and commanded to an information device.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a multimodal input integration system having a user interface for handling information equipment.

In recent years, various terminals such as computers, electronic organizers, ATM terminals and the like, and various information devices such as robots have been used. As their input devices, keyboards and mice have been used. With the development of multimedia, information has been diversified, and accordingly, new input devices such as touch panels and tablets have been used. Further, non-verbal input methods, such as voice input and gesture input, which have different styles (types), have been researched and developed.

[0003] Such an interface that allows the user to select various different types of input from usage conditions and preferences or to use it redundantly is a multimodal (mo
dal: type, style) interface.
The present invention relates to an input integration method for realizing the multimodal interface and a system architecture.

[0004]

2. Description of the Related Art In a personal computer or the like, in order to start an application, a method of hitting an application name with a keyboard or clicking an icon indicating the application with a mouse is generally used.

FIG. 16 shows a conventional application starting method. In this example, a user has a purpose of "reading a mail" for an information device such as a personal computer. In this case, the user determines and determines what the application handles mail.
Next, use the keyboard or mouse to search for the determined application from the display, and type a command to start the application into a predetermined location, or search for the icon corresponding to the application on the desktop and click it. Start the application by

Another example is the Internet, "I want to access the Internet (www)."
For this purpose, the user must determine that an application conventionally called a browser is to be used, and the user must start the browser by taking an action such as hitting a keyboard or clicking a mouse.

[0007]

The above-mentioned conventional method has a problem that the user has to remember the correct spelling of the application to be started and the shape of the icon, which imposes a heavy burden on the user.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a multi-modal input integration system that can execute a target process on an information device by an input that does not burden a user. It is another object of the present invention to provide a multimodal input integration system that can operate not only information processing devices such as personal computers as information devices but also various robots without burdening a user.

[0009]

SUMMARY OF THE INVENTION According to the present invention, a user simply inputs an operation by activating a target application, and the system automatically selects an application capable of achieving the target. It starts automatically.

FIG. 1 is a diagram showing the principle of the present invention. In the figure,
1 is a multi-modal input unit having a plurality of input means 1a to 1c corresponding to inputs of various types (styles) such as a user's voice and gestures (hand movements, etc.). A planner unit 3 that identifies input contents in order to integrate the input information, translates the input contents into words (or high-order words) of the superordinate concept, and generates a command script (character string) representing the target and purpose of execution, 3 A word dictionary for converting (translating) input into words of a higher concept based on an object-oriented concept having a plurality of attributes. 4 executes a script generated by the planner unit 2 and starts a corresponding application. The agent unit 5 is an application group composed of a large number of applications.

The input means 1a to 1c constituting the input unit 1
c may be provided with individual means for identifying inputs (speech, gestures, etc.) corresponding to each style (type), or may be configured to identify inputs in a plurality of styles with one input unit 1. it can.

When a user inputs an expression corresponding to a purpose in one of various forms (types) in order to execute a desired process, the corresponding input means 1a to 1c recognizes the contents of the expression. This expression depends on the input format,
The recognized result is supplied to the planner unit 2. In the word dictionary 3, standardized high-level concept words for integrating concepts identified according to various input formats in advance are defined, and are created based on an object-oriented concept having several attributes. The planner unit 2 determines the recognition result from the input unit 1 and uses the word dictionary 3 to generate a command script (character string) that can be executed by the agent unit 4.

When the agent section 4 receives the command script, it executes the script to activate the corresponding one of the application groups 5 to "read" the "mail". There are a plurality of levels of conversation between the user and the system. There are a top (system) level relating to commands to the system, an agent level relating to conversation with the agent unit, and an application level relating to conversation with the application.

[0014]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 2 is a block diagram of an embodiment. In the drawing, reference numeral 10 denotes a real input device, and 10-1 denotes a speech input for recognizing a user's speech input and outputting a word as a recognition result. A device 10-2, a motion input device for recognizing the movement of the user's hand or facial expression as an image input and outputting a word as a recognition result; 10-3, another input device (including a mouse or a keyboard); This is an input model, and is 11─1 corresponding to each of the real input devices 10 # 1 to 10 # 3.
-11 モ デ ル 3 input models are provided, each of which functions as a driver for a corresponding input device, and includes a dictionary (or template) of words handled by the corresponding input device in each input model. 12 is a planner section that replaces words input by the user through various real input devices with words of a higher concept using the word dictionary 120 and generates a script that can be executed by the agent section, and 13 is a planner section generated by the planner section. An agent unit that executes a script.

An application model 14 stores an application name and options to be started, a message output by the agent when the application is started, and a message output by the agent when the application ends. 1
Reference numeral 5 denotes an application group that executes a target process according to the application model 14. Reference numeral 16 denotes a real computer system (hardware, OS, etc.), and reference numeral 17 denotes a system model for monitoring the internal information of the computer and the status of external sensors. When the agent unit 13 executes a script, such system information is displayed. If you need, you can retrieve that information by asking this object. Specifically, the date, time, presence / absence of the user sitting, and the like are obtained.

Reference numeral 18 denotes a user model, which stores personal information such as a preference of a format (screen display, voice output, printout, electronic mail, etc.) when the user receives the output information, or a habit. By storing user-specific habits and preferences in the user model 18, it is possible to return a response in a style (type) according to the user. Therefore, the agent unit 13
If a response is made, there is no need to worry about the output device at all, thus simplifying the processing of the agent. Further, the output device can be easily changed by disconnecting the relationship between the agent unit 13 and the output device (described later). This is because it is not necessary to change the description of the agent unit 13 to change the output device. With such a configuration, output devices can be freely arranged, and application to various systems is possible.

Reference numeral 19 denotes an output model.
-2, and each output model is composed of the actual output devices 20-1 and 20-2 constituting the actual output device 20.
-2, and all commands for the actual output device 20 are replaced with commands for this object. Reference numeral 20 denotes an actual output device, which includes a display device, an audio output device, and the like for outputting an execution result or a response such as rejection to an instruction from a user in a different format such as an image, a print, a voice, and an e-mail.

The contents of each part of the configuration shown in FIG. 2 will be described in the order of operation of the system. Voice input device 10─1
Is an input device for recognizing a voice uttered by a user, and outputs a word recognized using a voice dictionary provided in the voice input device using a voice recognition technology such as pattern matching using a conventionally known technology. I do. The motion input device 10 # 2 is used by the user to input a hand movement or an image of face information as an input, and to track the movement.
This recognizes each motion by comparing it with a template (not shown) preset in # 2 and outputs a recognition result. For example, this technique has been previously proposed as a "correlation tracking system" (Japanese Patent Application No. 7-342320) by the same applicant as the present invention, and can be implemented by using the technique. The technical contents will be described with reference to FIGS.

FIG. 3 is a flowchart of the proposed correlation tracking. First, an image of an object (hand or face) taken by a video camera is taken in (S1 in FIG. 3), and the range to be searched next is taken as a full screen. Specify (S2). On the other hand, a plurality of templates (an example is shown in FIG. 4 to be described later) representing a plurality of different shapes in which an object (a hand shape or a facial expression) changes in advance.
Is prepared, one template is selected from among them (S3 in FIG. 3), pattern matching is performed while sequentially shifting the template over the entire screen, the degree of coincidence is calculated (the same S4), and the best matching The degree is stored (S5). When one template is completed, the same processing is performed for the next template. When the processing is completed for all templates, the template with the highest matching degree and its position are detected, and tracking is started from that position (S7). ).

That is, an image is fetched (S8),
A search block of a predetermined size is set (S9), one template is selected (S10), and a position where the degree of coincidence is maximized in the search block is searched (S11). When the process is completed for each template, the reference block is moved to the position indicated by the template having the highest degree of coincidence, and the movement vector at this time is obtained (S14). In this way, the object is tracked, and when the shape of the object changes, a match with any of the templates is detected.

FIGS. 4A and 4B show examples of hand shape recognition based on the above proposed principle, wherein FIG. 4A shows an example of the immediately preceding frame, FIG. 4B shows an example of the current frame, and FIG. In the previous frame, the object in the reference block has the highest degree of matching with the open hand template, but the object in the current reference block shown in (b) matches the template with two fingers out. This is an example where the degree is calculated as being the largest. In this case, the movement vector is represented by a line with an arrow in FIG. 4, and a numeral "2" is output as a recognition result. OK (or GOOD), NOGOO
D, OPEN, CLOSE, etc. can be defined so as to recognize the shape of each finger and output a corresponding recognition result.

The configuration of the word dictionary 120 used by the planner unit 12 of FIG. 2 to generate a script will be described. FIG. 5 shows the data structure of the word dictionary. FIG. 6 shows the configuration of words (synonyms) having the same superordinate concept, and FIG. 7 shows an example of the attributes of the superordinate concept word.

The word dictionary is shown in FIG. 5 has a data structure as shown in FIG. 5 and includes, for each word, its part of speech, broader concept word, and issuer list (list of operation targets). As shown as an example, the word "plan book" is a noun and means a document, and the issue destination is a "document agent" (meaning an agent handling a document). The word "read"
The verb and the broader concept word are "read", and the issuer includes a document agent and an assistant agent. “Assistant” is a proper noun, and the broader concept word is an assistant agent, indicating that there is no issue destination.

FIG. 6 is an example of a word having the same general concept. In this example, a user's voice input is a “Yes”, b
"Yes" and input by the finger of the user c
Words synonymous with "good" and "ok" (synonyms)
, Which indicates that the part of speech is a verb, the broader concept word is yes, and the destination is a document agent or an assistant agent.

FIG. 7 shows an example of the attribute of the broader concept word. As shown in the table, each attribute is composed of a character string representing the meaning, a list of paired upper concept words (combinable upper concept words), the path of the application to be started, and the attributes of the application to be started. You. B. of FIG. The top word "mai
For "l", a specific example of the data of each of the following items is shown.

FIGS. 8 and 9 show the processing flow (part 1) and (part 2) of the planner section. First, the input port (the port where the output of the input model 11 in FIG. 2 is generated) is checked (S1 in FIG. 8), and it is determined whether a word has been input (S1 in FIG. 8).
2). When input, the input word is stored in a stack (not shown) in the planner unit 12 (S in FIG. 8).
3) It is determined whether or not the number of words on the stack is two (S
4). This is a combination of noun (target) and verb (method,
That is, it is necessary to input two of the processing contents. Therefore, when two words are input, the part of speech of the word is checked (S5 in FIG. 8), and it is determined whether the combination is a combination of a noun and a verb (S6). If the combination is not possible, the stack is cleared. (S7).
In this case, the fact is notified to the user. If the combination is possible, the superordinate concept word is extracted from the attribute of the input word (S8), and the combination is checked (S9).

With this check, it is determined whether or not the combination of the noun and the verb is possible with reference to FIG. 7 (S10 in FIG. 9). (Execution agent section) is determined (S11), and a method (processing content) is determined from the upper concept word attribute of the verb (S12). Next, an argument (target of processing) is determined from the superordinate concept word attribute of the noun (S1).
3), a script (character string) is created by combining the destination, the method, and the argument (S14), the stack is cleared (S15), and the process ends.

FIG. 10 shows an example of script generation by the processing in the planner unit 12. This example is shown in FIG.
B. The words entered are “mail” and “read” as shown in, and “mail” is a noun according to the word dictionary, “read” is a verb, and the combination is possible. Using FIG. Create a script based on the rules shown in. According to this rule, C.I. As shown in the example script format, the destination is "assist Agent" (proper noun indicating the destination of the mail attribute), and the method (verb) is re.
A script is generated in which the ad (method of read) is mail.

When the script generated by the planner unit 12 is supplied to the agent unit 13, the script is checked by the agent unit 13, and if necessary information is insufficient, the user model 18 is informed. Output a message to notify. The user model 18 responds to the output model 19 corresponding to the output format according to the user's preference (habit) stored in advance, and the output model 19 drives the corresponding real output device 20.
The user inputs necessary information or instructions to the actual input device 10 by recognizing the contents of the response output to the actual output device 20. When the necessary information is input, the agent unit 13 issues an instruction to the corresponding application model 14, and the application model 14 sends an instruction to drive the designated application (in the application group 15) to a method registered in advance. And the application executes the received instruction.

FIG. 11 is a processing flow for presenting output to the user. In the user model 18 (18 in FIG. 2), the output contents are extracted from the output instruction of the agent section (13 in FIG. 2) (S1 in FIG. 11), and the output destination is determined from the accumulated information (like, habit, etc.) of the user model. It is determined (S2). Next, it is determined whether or not the output is a screen display. In this case, a message to be displayed on the output model (19 in FIG. 2) corresponding to the actual output device for screen display is output. Next, it is determined whether the output is a voice output, and if the voice output is applicable, the corresponding output model outputs a voice (FIG. 1).
1, S5 and S6), and then determines whether the output is an e-mail. If the output is applicable, the e-mail is output (S7 and S8). , S10). Finally, it is determined whether the file is a file output, and if the file is output, the file is output (S1 in FIG. 11).
1, S12).

According to the present invention, a necessary application is started and instructed through a conversation between the user and the multi-modal input integration system to execute the application, and the output from the system is presented to the user. To that end, the conversation between the user and the system is divided into three levels.

FIG. 12 shows the level of conversation between the user and the system, where 30 indicates the top level, 31 indicates the agent level, and 32 indicates the application level. The top level 30 represents a conversation between the user and the planner, and corresponds to a purpose instruction such as “mail” or “read”. Next, the agent level 31 represents a conversation between the user and the agent unit, and is a conversation between the user and the user when there is insufficient information when the agent unit starts the application.

For example, when a user inputs an instruction in a top-level conversation such as “report” and “change”, the target file is unknown at the agent level. Is output to the user. On the other hand, the user, for example, "File 3"
Output. The application level is a conversation with the application, and the operation of the application includes, for example, an instruction from the user to the application such as “end”, “open”, and “search”, and an output from the application to the user. .

As described above, the conversation between the user and the system is classified into three levels, and each level uses a different word. In an input device that performs pattern matching such as voice input, as the number of words to be registered increases, the search range of the template increases, and the recognition rate of recognition deteriorates. Therefore, as shown in FIG. 13, a voice template group corresponding to each level is prepared, and each time the level changes, the voice template is switched to reduce the search range, thereby improving the recognition rate.

FIG. 13 is an explanatory diagram of the input operation depending on the conversation level. In this example, a voice input device is used as the actual input device. In this example, the top level, the agent level, and the application level shown in FIG. 13, reference numeral 40 denotes a voice input device; 41, a voice dictionary (template);
An audio dictionary corresponding to 3 and 45 is a planner section.

FIG. Indicates the relationship between the voice input device and the planner unit. 3 in the voice input device
It has a voice dictionary corresponding to two conversation levels. The planner receives an inquiry about level information from the input device and returns the current level. When the voice input device recognizes the user's voice input, first, the current conversation level is inquired to the planner unit, and the search range of the voice dictionary is limited by the level information obtained from the planner unit. For example, in FIG. If the level information obtained from the planner unit is level 3 as shown in (1), a search is performed using only the voice dictionary 44 corresponding to level 3 to perform a sound recognition operation.

In this way, the search range of speech recognition is reduced to improve the speed of recognition and to prevent recognition mistakes.
It is possible to improve the recognition rate. FIG. 14 shows an example of the configuration of a multimodal input integration system for instructing a remote agent.

In the figure, reference numerals 10 to 20 denote the same parts as those of the embodiment shown in FIG. 2
Reference numeral 1 denotes a communication network including the Internet.
In the configuration of FIG. 14, the configuration of FIG.
2. The difference is that the user model 18 and the agent unit 13 are connected by the communication network 21. With this multi-modal input integration system, the user executes various processes by performing multi-modal input conversations with the agent unit 13 (including the application model 14 and the application group 15) at a remote location. And obtain the result.

Next, FIG. 15 shows an example of the configuration of a multi-modal input integration system for instructing a group of remote robots. In FIG. 15, reference numerals 10 to 12, and 18 to 20 are the same as those of the embodiment shown in FIG.
21 is a communication network including the Internet, 22,
Reference numerals 23 denote robots A and B which have a built-in mechanism having the function of an agent unit (13 in FIG. 2) and execute various operations, and are substantially provided with two agent units.

In this system, a real user (robot operator) gives a command to the remote robots 22 and 23 using a desired input format such as voice or hand gesture. This command is executed by the actual input device 10, the input model 11,
It is sent to the destination robot 22 or 23 via the planner unit 12 and a communication network 21 such as the Internet. The state of the robot 22 or 23 can be displayed by the actual output device 20. In particular,
Instructions can be given by intuitive words such as "robot A" and "movement", and a script generated by the planner unit 12 is sent to each robot. Thus, the burden on the operator can be reduced by the multi-modal input.

[0041]

According to the present invention, it is possible for the user to input a voice corresponding to the purpose of calculation, or a familiar voice or action used in ordinary conversation between humans by gesture. The burden of remembering commands and application names can be greatly reduced.

[Brief description of the drawings]

FIG. 1 is a principle configuration diagram of the present invention.

FIG. 2 is a configuration diagram of an embodiment.

FIG. 3 is a flowchart of a proposed correlation tracking.

FIG. 4 is a diagram showing an example of recognizing a hand shape according to the proposed principle.

FIG. 5 is a diagram showing a data structure of a word dictionary.

FIG. 6 is a diagram showing a configuration of words having the same superordinate concept.

FIG. 7 is a diagram illustrating an example of attributes of a broader concept word.

FIG. 8 is a view showing a processing flow of a planner unit (part 1);
It is.

FIG. 9 is a view showing a processing flow of a planner unit (part 2);
It is.

FIG. 10 is a diagram illustrating an example of script generation.

FIG. 11 is a diagram showing a processing flow of output presentation to a user.

FIG. 12 illustrates the level of conversation between the user and the system.

FIG. 13 is an explanatory diagram of an input operation based on a conversation level.

FIG. 14 is a diagram illustrating a configuration example of a multimodal input integration system that issues a command to a remote agent.

FIG. 15 is a diagram illustrating a configuration example of a multimodal input integration system that commands a robot group in a remote place.

FIG. 16 is a diagram showing a conventional application starting method.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Multimodal input part 2 Planner part 3 Word dictionary 4 Agent part 5 Application group

Claims (7)

[Claims] An input unit for recognizing any one of a plurality of styles of input by voice, gesture, etc. and outputting a recognition result, determining the recognition result output from each of the input units, and executing a generalized command A planner part to convert to script,
An agent unit for interpreting and executing the command script from the planner unit, and integrates inputs of different formats into higher-level meanings and issues a command to the information device. 2. The multimodal input integration system according to claim 1, wherein said input unit recognizes an input by voice or gesture expressing a purpose of use of the information device, generates a corresponding word, and outputs the word to said planner unit. . 3. The planner section according to claim 2, further comprising: a word dictionary in which words representing superordinate concepts are set and attributes associated with the words representing the superordinate concepts are set. The multi-modal input integration system according to claim 1, wherein the planner unit uses the word dictionary to generate a script representing a command to be executed by the agent unit, using the word dictionary. 4. The planner unit according to claim 3, wherein a list of words that can be combined with the word is set as an attribute associated with a word representing a superordinate concept stored in the word dictionary of the planner unit. A multi-modal input integration system, wherein it is determined whether a combination of words is possible at the time of generation of a word. 5. The agent unit according to claim 1, wherein the agent unit generates output information including a necessary input instruction and a processing state in executing the application specified by the input script. A user model for receiving the output, the user model holding information indicating the user's preference for the output format in advance;
A multimodal input integration system, wherein output information from the agent unit is output in a format corresponding to the held information. 6. The system according to claim 1, wherein a user's input from the input unit and a dialogue with an information device are divided into a plurality of levels, and the input unit recognizes an input such as a voice corresponding to each level. A multi-modal input integration system, comprising: a dictionary for identifying a current level of an input; and performing recognition using a corresponding dictionary. 7. The multimodal input according to claim 1, wherein the planner unit and the agent unit are connected by a network, and a script generated by the planner unit is provided to an agent unit provided at a remote place. Integrated system.