JPH09251368A - Construction supporting system for intelligent agent - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a function for freely constructing an agent responding by using plural media such as still images, moving images, voice and effective sound, etc., by controlling a computer and an application software, etc., instead of a user based on an instruction using the plural media such as a keyboard, a mouse, a pen, the voice and gestures, etc., even by the user incapable of performing programming. SOLUTION: By writing descriptions for input contents and input timings in the plural media such as keyboard input, mouse input, pen input, voice input and gesture input, etc., in a rule condition part 8 and writing the output information descriptions, execution command descriptions and the output timing descriptions of text display output, still image output, moving image output, voice output and effective sound output, etc., in a rule conclustion part 9, the operation of the agent is described.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a user interface technology for a personal computer, a portable information device and the like, and more specifically, based on an instruction using a plurality of media such as a keyboard, a mouse, a pen, a voice and a gesture.
The present invention relates to a technology for constructing an agent that controls a computer, application software, etc. on behalf of a user and responds using a plurality of media such as still images, moving images, voices, and sound effects.

[0002]

2. Description of the Related Art At present, in a personal computer, a portable information device and the like, input using a keyboard and a mouse is the mainstream. However, since keyboard and mouse inputs are complicated, software called agents, which act on behalf of users to operate computers and applications, is drawing attention.

For example, reference (a) "P. Mayes: Agent"
s that Reduce Work and Innovation Overload,
Communications of the ACM, Vol, 37, No.
7, pp. 30-40, 1994 ”describes an agent that advises whether to read a large amount of emails or discard them. This agent monitors the user's keyboard and mouse operations over the user's shoulders and learns the user's habits of operation.

In addition, reference (b) “D. Smith, A. Cyp
her and J. Spohrer: Kid Sim: Programming Age
nts Whitout a Programming Language, Communicat
ions of the ACM, Vol, 37, No. 7, pp. 54
-67, 1994 ”proposes a method in which even a child can program by using a graphical rule, and argues that application of this method can be used for building an agent.

[0005]

However, the above technique is
It specializes in keyboards and mice, and controls computers and application software on behalf of users based on instructions using multiple media including keyboards and mice, such as pens, voice, and gestures, and still images and videos. It was difficult to apply to agents that respond using multiple media such as voice, sound effects, etc.

An object of the present invention is that even a user who cannot program can control a computer, application software, etc. on behalf of the user based on an instruction using a plurality of media such as a keyboard, a mouse, a pen, a voice, and a gesture so that the user can stand still. It is to provide a function for freely constructing an agent that responds by using a plurality of media such as images, videos, voices, and sound effects.

[0007]

In order to achieve the above object, (1) In order to enable even a user who cannot program, to easily create an agent that responds to a multimodal input and responds with a multimodal output. ,
In the rule condition part, write the description about the input content and input timing in multiple media such as keyboard input, mouse input, pen input, voice input, gesture input, and write the text display output, still image output, video output, voice in the rule conclusion part. By writing output information description such as output and sound effect output, execution command description, and output timing description of them, the operation of the agent can be described.
(2) In order to enable multi-modal rule condition description, it is possible to specify the input contents for each input mode such as voice or keyboard input on a common time axis.
(3) In order to enable multi-modal rule conclusion part description, it is possible to specify output contents for each output mode such as voice and screen output on a common time axis. (4) User is a computer Operation is decided by looking at the output screen of, and since the operation result of the user can be described as the state change of the computer output screen, it is possible to specify the state of the computer screen in the multimodal condition description and the multimodal output description. The description of the computer screen in the multimodal conditional description is a condition that becomes true when the computer screen actually appears on the output screen of the computer, and the description of the computer screen appearing in the multimodal output description becomes that computer screen. To control the computer and operate application software. Interpretation as command execution (5) In addition, even if the user does not explicitly describe the multimodal rule as described above, the user can obtain multiple correct examples and By giving an incorrect example in a multimodal manner, the description of the rule condition part and the actual part of the rule is automatically generated. (6) Furthermore, as described above, the user does not have to explicitly describe the operation example. Also, the user's operation history includes input information using multiple media such as a keyboard, mouse, pen, voice, and gesture, output information such as still images, videos, voices, and sound effects, and an operation history consisting of user operations. The configuration is provided with means for saving, detecting patterns that often appear in them, and automatically generating multimodal rules.

[0008]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 shows a first embodiment of the present invention.

In this embodiment, as shown in FIG. 1, an agent 2 existing in a personal computer composed of a monitor 1, a speaker 3, a camera 4, a microphone 5 and the like is a target. The agent 2 is based on an instruction using a plurality of media such as the keyboard, mouse, pen, voice, and gesture of the user in front of this personal computer,
It has the function of controlling the personal computer system itself and application software running on this personal computer on behalf of the user, and responding by using multiple media such as still images, moving images, voices, and sound effects. There is.

A feature of the present system is that it has a function for freely adding and correcting the operation contents of this agent software without the user programming, that is, an intelligent agent construction support function.
The state of the main window 7 is displayed on the monitor 6. The main window 7 displays a multi-modal rule for defining the behavior of the agent,
It is composed of a multi-modal rule condition section 8 and a multi-modal rule execution section 9.

The intelligent agent construction support function will be described with reference to FIG. Clicking on Agent 2 with the mouse will bring up Menu 2 for that agent.
1 appears. In the menu 21, three submenus of multimodal rule creation 22, operation history storage start 23, and learning from operation history 24 are displayed.

The overall operation flow will be described with reference to FIG. First, it is judged whether or not there is a menu starting operation by clicking the mouse on the intelligent agent 2 displayed on the monitor 1 (31). If there is a menu starting operation, the menu shown in FIG. 2 is displayed. Do (3
2).

When the user selects "create multi-modal rule" in the menu (33), the multi-modal rule editor shown in FIG. 5 is started (34). A method of creating a multi-modal rule that defines the operation of the agent using the multi-modal rule editor will be described later with reference to FIG.

When the user selects "start operation history storage" in the menu (35), the history storage switch is set on (36).

When the user selects "learn from operation history" in the menu (37), the history file is analyzed,
A multimodal rule is automatically generated (38).

Further, when the history storage switch is on (39), the multimodal history storage necessary for automatic generation of the multimodal rule is written in the history file (4).
0). The multi-modal history information necessary for automatically generating the multi-modal rule will be described later with reference to FIG.

As described above, when the user selects "Create multi-modal rule" in the menu, the multi-modal rule editor is activated and the window shown in FIG. 5 is displayed. By operating this window,
It is possible to create a multi-modal rule by a user or automatically create an example based on a user operation.

Therefore, first, the operation when the window shown in FIG. 5 appears will be described using the operation flow shown in FIG. Multimodal Rule Editor window 51
Shows a rule name 52, a rule condition part 55 and rule conclusion part 58 which are rule definition information. The learning start button 53 and the learning end button 54 are buttons for controlling the learning of one rule as a whole, and the learning start button 56 and the learning end button 57 are buttons for controlling the learning of the conditional part of the rule, and the learning start button 59 and the learning. The end button 60 is a button for controlling learning of the rule execution unit.

When the rule learning button 53 is pressed while the window 51 of the multimodal rule editor is displayed (41), the learning end button 5
The operation history is stored in the format illustrated in FIG. 9 until 4 is pressed, and the result is analyzed to generate one rule (42). In addition, when the rule condition part learning button 56 is pressed (4
3) Until the learning end button 57 is pressed, the operation history is stored in the format illustrated in FIG. 9, the result is analyzed, and one rule condition part is generated (44). When the rule execution unit learning button 59 is pressed (45), the learning end button 60
The operation history is stored in the format illustrated in FIG. 9 until is pressed, and the result is analyzed to generate one rule execution unit (4
6). A method of holding the operation history and a learning method will be described later with reference to FIG. 9, and a method 47 of defining the multimodal rule by the user will be described below.

FIG. 6 is an enlarged view of the rule condition section 55 of FIG. As input media, a total of 7 types of "sound" 61, "voice" 62, "keyboard" 63, "mouse" 64, "pen" 65, "gesture" 66, and "screen" 67 are prepared. The difference between "sound" and "voice" is "sound" 61
Represents sound effects that cannot be expressed in words, while “voice” 6
2 represents a verbalizable sound. The "screen" 67 represents a part of the screen currently displayed on the monitor of the personal computer. This will be described later with reference to FIG.

An editor window 68 is prepared for each of the seven types of input media. The operation of the editor window is different for each medium, but has a time coordinate 69. Further, a triangle mark 70 is used to clearly indicate the start time.

In the "sound" editor window, the time at which the sound should be generated is selected, the sound is actually generated and collected by the microphone, and its waveform is displayed.

In the editor window for "Voice", select the time at which the voice should occur and enter the text of that voice. The start time may be clearly indicated by the triangle mark 70.

In the editor window for "keyboard", enter the text to be entered from the keyboard.

In the editor window for "mouse",
Clicks that should be input from the mouse are input using the triangles that represent the coordinates and time. Two consecutive triangles represent a double click.

In the "pen" editor window, stroke data to be input from the pen is input.

In the editor window for "gesture",
The time when the gesture should be input is selected, the gesture is actually generated and collected by the camera, and the state is displayed.

In the editor window for "screen", a part of the screen to be displayed on the screen of the personal computer is cut and input from the screen.

As described above, the condition part of the multimodal rule can be defined by describing each input medium. When the inputs described in the media specified in this condition part are observed at the specified time, it is determined that this multimodal rule can be executed.

Next, the method of defining the execution part of the multimodal rule will be described with reference to FIG. FIG. 7 is an enlarged view of the rule execution unit 58 of FIG. As output media,
“Sound” 71, “Voice” 72, “Text display” 73,
A total of 6 types of "animation" 74, "command execution" 75, and "screen" 76 are prepared. "Sound" and "Voice"
The difference is that, like the input section, "sound" represents a sound effect that cannot be expressed in words, while "speech" represents a verbalizable sound.

An editor window 79 is prepared for each of the six types of output media. The operation of the editor window differs depending on the medium, but having the time coordinate 77 and using the triangular mark 78 to clearly indicate the start time has the same meaning as the input section.

In the "sound" editor window, the time at which the sound should be generated is selected, the sound is actually generated and collected by the microphone, and its waveform is displayed.

In the editor window for "voice", select the time at which the voice should occur and enter the text of that voice. The start time may be clearly indicated by the triangle mark 78.

In the editor window for "text display", the text to be displayed as a character string on the monitor is input.

In the editor window for "animation", an animation to be displayed as a moving image on the monitor is input.

In the editor window for "command execution", the command to be executed by the personal computer is explicitly described. In the example of FIG. 7, a file (Fil
The command (Delete File 01) for deleting e01) is described.

In the editor screen for "screen", after executing this rule, a part of the screen to be displayed on the screen of the personal computer is cut and input from the screen.
This interpretation will be described later with reference to FIG.

As described above, by describing each output medium, the execution part of the multimodal rule can be defined.

When the multi-modal rule is executed, the content described in the medium defined in the rule execution part is output at the designated time. The exception is
These are “command execution” 75 and “screen” 76. The content described in the "command execution" 75 is such that the command itself is executed at the designated time, and in the case of the "screen" 76, the state of the screen described there is set at the designated time. The command sequence will be estimated and executed sequentially.

FIG. 8 shows an example of a multi-modal rule consisting only of definitions relating to "screen". The screen description 81 in the condition section represents the state after the change of the icon appearing on the desktop of the personal computer when the fax is received. That is, this icon 81 appears when the fax arrives at the personal computer. The description 82 in the actual part of the rule represents the state of the screen when the file is placed on the holder A, that is, the storing operation is performed. Therefore, this multi-modal rule can be executed when a fax is received, and a command sequence for saving the received file in the holder A is executed.

FIG. 9 shows a storage format of history information used for learning the entire rule, learning the rule condition part, and learning the rule conclusion part. The storage format 91 is a fusion of the definition formats of the rule condition part and the conclusion part, and the item is "sound".
92, "voice" 93, "keyboard" 94, "mouse"
95, “pen” 96, “gesture” 97, and “screen” 98. From these, it is possible to directly generate the rule condition part and the rule conclusion part. Further, a commonly known clustering method is used to extract the common operation.

[0043]

As described above, according to the embodiments of the present invention, a computer or application software is operated on behalf of the user based on the user's instructions using a plurality of media such as a keyboard, a mouse, a pen, voice, and a gesture. It is possible to define an intelligent agent that controls the above and responds using a plurality of media such as still images, moving images, voices, and sound effects without programming by the user himself.

[Brief description of drawings]

FIG. 1 is a diagram showing a first embodiment of the present invention.

FIG. 2 is a diagram showing a menu relating to an agent.

FIG. 3 is an explanatory diagram showing an overall operation flow.

FIG. 4 is a diagram illustrating an operation flow of a multi-modal rule window.

FIG. 5 is a diagram showing an entire multi-modal rule editor.

FIG. 6 is a diagram illustrating a condition part of a multimodal rule editor.

FIG. 7 is a diagram illustrating a real rear part of a multimodal rule editor.

FIG. 8 is a diagram illustrating an example of a multi-modal rule including only a description regarding a screen.

FIG. 9 is a diagram showing an example of multi-modal history information.

[Explanation of symbols]

1 ... Monitor of personal computer, 2 ... Agent, 3 ... Speaker, 4 ... Camera,
5 ... Mike, 7, 51 ... Multi-modal rule window, 8, 55 ... Multi-modal rule condition part,
9, 58 ... Multimodal rule execution unit, 21 ... Agent menu, 81, 82 ... Example of multimodal rule consisting of only screen definition, 91 ... Multimodal history storage information.

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Tamunori Tomita 1099 Ozenji Temple, Aso-ku, Kawasaki City, Kanagawa Prefecture Inside the Hitachi, Ltd.System Development Laboratory (72) Inventor Hirokazu Aoshima 1099 Address Ozenji Temple, Aso-ku, Kawasaki City, Kanagawa Prefecture Hitachi Systems Development Laboratory

Claims (6)

[Claims] 1. A still image, a moving image, a sound, a sound effect by controlling a computer, application software, etc. on behalf of the user based on a user's instruction using a plurality of media such as a keyboard, a mouse, a pen, a voice and a gesture. In a computer system having an agent that responds by using multiple media such as a multi-modal condition description consisting of description of input content and input timing in multiple media such as keyboard input, mouse input, pen input, voice input, and gesture input , Output information description such as text display output, still image output, video output, audio output, sound effect output, execution command description, and their output timing description, indicating the contents to be executed or responded when the condition description is satisfied Of the multimodal execution part description, which consists of A system for supporting construction of an intelligent agent characterized in that the behavior of an agent is defined by a set of multimodal rules composed of two sets. 2. The system according to claim 1, wherein the multimodal condition description can specify input contents for each input mode such as voice or keyboard input on a common time axis. Construction support system for intelligent agents. 3. The system according to claim 1, wherein the multimodal execution description can specify output contents for each output mode such as voice and screen output on a common time axis. Construction support system for intelligent agents. 4. The system according to claim 1, wherein the multimodal condition description and the multimodal output description include designation of a state of a computer screen, and the description of the computer screen in the multimodal condition description is such that the computer screen is actually a computer. As a condition that becomes true at the time of appearing on the output screen of, the description of the computer screen appearing in the multimodal output description is as the execution of the command for controlling the computer so that the computer screen becomes the computer screen or operating the software application.
A construction support system for intelligent agents characterized by being interpreted. 5. The system according to claim 1, wherein the user automatically generates a multimodal conditional part description and a multimodal execution part description by giving a plurality of correct and incorrect examples to the multimodal. Construction support system for intelligent agents. 6. The system according to claim 1, wherein the operation history of the user is input information using a plurality of media such as a keyboard, a mouse, a pen, voice, and a gesture, and output of a still image, a moving image, a voice, a sound effect, and the like. An intelligent agent construction support system characterized by automatically generating a multi-modal rule by storing and analyzing a multi-modal operation log consisting of information and user's operations.