JPH09258945A - Individual information equipment - Google Patents

Abstract

PROBLEM TO BE SOLVED: To use a computer with a user interface in a human-centered form and not in a computer-centered form by making the computer interactive in a multi-modal form such as voice, handwriting and gesture. SOLUTION: Paper 2 on a real disk 1, an environment viewer 3, a pen input computer 4, a keyboard 9, a pen with scanner 8, a microphone 6, a speaker 5 and a camera 7 are provided. The pen input computer 4 displays the file of a virtual desk top and executes application as the interface of real paper and virtual paper by using the pen with scanner 8, which incorporates a small CCD camera in the pen. The desk top is displayed in the environment viewer 3 and an intelligent agent 10 and the like are displayed on the monitor. The displayed agent 10 extracts a user intention from the multi-modal indication from an instruction (by pen, voice and gesture) for the system, solves a problem and outputs a result.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a user interface technology for a computer such as a personal computer and a portable information device. More specifically, it controls a computer, application software, etc. on behalf of the user based on an instruction using a plurality of media such as a keyboard, a mouse, a pen, a voice, and a gesture so that a still image, a video, a voice, a sound effect, The present invention relates to a multi-modal personal information device that responds using multiple media.

[0002]

2. Description of the Related Art Currently, computers such as personal computers are rapidly spreading in offices and homes. It is thought that the driving force is the dramatic improvement in cost performance in terms of hardware and the improvement of user friendliness by introducing a graphical user interface in terms of software. For example, the document “JA Sohnson, et. Al .: ACE: Building Interac
tive Graphical Applications, Communications of the
ACM, Vol, 36, No. 4, pp. 400-55, 1993 ”describes the superiority of the graphical user interface.

[0003]

However, a computer having a graphical user interface-based user interface based on the above-mentioned technology is used in an office,
As the computer spreads at home, the following three problems (problems (1) to (3)) of the current computer are being recognized.

Problem (1) Unnatural work environment When a computer is not required, it is possible to freely use the desk. However, once the computer is needed, the user is tied to a narrow desktop on the monitor.

This is because the computer ignores our world of activity and limits our place of activity to the desktop on the computer's manageable virtual monitor.

Problem (2) Limited input / output method The input / output of the current computer is a keyboard, a mouse,
Limited to on-screen display. Widening the input / output band is necessary for a smooth interface with a computer.

Problem (3) Ignorant user interface The operation to the computer must be explicitly instructed by direct manipulation by a graphical user interface, for example. In addition to what they can do, the user must know how to do it. Also, even though a lot of data is stored in the computer, it is not recognized by the computer. Furthermore, it is difficult to use even though valuable data is stored on the network.

It is an object of the present invention to provide a computer use environment having a more human-centered user interface rather than a computer center.

[0009]

In order to achieve the above object, the present invention is a computer designed by human-centered / real-world-centered thinking and capable of interacting in a multimodal form such as voice, handwriting, and gesture. And

Here, in order to solve these problems, it is necessary to grasp each problem comprehensively, not independently. In order to solve the problem (1), a new computing environment must be presented. The input / output (problem (2)) required there is closely related to the environment. Further, the intelligent processing (problem (3)) is deeply related to the computing environment, and more advanced input / output cannot be put to practical use without intelligent processing.

First, the problem (1) occurs because the user interface environment has been designed with the logic only on the computer side. At present, the method for solving this problem is considered to be the adoption of complete virtual reality or the return to the real world, but in the present invention, the solution is to fuse the real and virtual worlds. Specifically, the solution is a combination of "real desk + virtual desk + real paper + virtual paper".

That is, the world in which actual paper (real paper, 2 in FIG. 1) is placed on a real desk is a computerless world, in which a monitor representing a virtual desk and a virtual paper are represented. It is an environment with several pen input computers.

More specifically, the problem is solved by the following means.

(1) An environment viewer, which is a device for virtually displaying a business environment such as a desktop, a virtual paper, a microphone, a speaker, a camera, which has an interface similar to paper realized by a pen input computer integrated with display and input, The pen and keyboard are the components.

(2) It is better to distinguish and display the following three types on the environment viewer. First of all, there are virtual files such as electronic files and folders (folders). Second, it exists in the real world and can be operated from a computer, such as a networked air conditioner, video, light, etc. Finally, it is something that exists in the real world and cannot be operated from a computer, such as an actual piece of paper placed on a desk.

(3) More preferably, those existing in the real world and operable by a computer, and those existing in the real world which cannot be operated by the computer, input the situation of the business environment from a camera and display it. The position is specified by the analysis, and the position is displayed so as to be superimposed on the corresponding position in the virtual world such as the desktop displayed in the virtual viewer.

(4) More preferably, only the rough state of the environment is displayed on the environment viewer. At this time, the detailed contents of the text file displayed on the desktop or the like on the environment viewer are displayed on the virtual paper. At this time, it does not have to be displayed on the environment viewer.

(5) Furthermore, it is more preferable that the environment viewer simultaneously displays a plurality of environments such as one's office, another person's office, and a library.

(6) Finally, when a plurality of virtual papers are placed side by side, they can be operated as one large pad in conjunction with each other.

The problems (2) and (3) are solved by introducing an intelligent agent capable of multimodal input / output realized by the following means.

(1) An agent for managing the entire personal information equipment is provided. The agent processes with multi-modal input using at least two inputs of voice, gesture, pen gesture, pen and keyboard.

(2) When a plurality of environments are displayed on the environment viewer, it is better for the agent to execute an instruction for the plurality of environments.

(3) More preferably, the voice or pen input content input by the user is temporarily displayed on the virtual paper as a voice recognition result or a handwritten character recognition result.
When the recognition result can be executed as an instruction of the agent, the display color is changed to notify the user, and when the instruction is executed, it is erased from the virtual paper, and when not executed, the recognition result is left as it is on the virtual paper.

(4) Further, the agent is at least two of voice, sound effect, environment viewer display, and virtual paper.
Respond with a multimodal output that uses the output of one display at the same time.

(5) The figure of the agent is displayed on the environment viewer as a symbol of the agent. When it recognizes an executable instruction, it changes the expression of the displayed symbol and executes it while displaying the content in text.

(6) If the instruction content of the user is inferred from the situation, and if the portion recognized as the command displayed in a different color on the virtual paper is tapped,
The agent executes the instruction when saying “OK” to the agent, when making an OK gesture, when touching the agent, or when the user nods at the agent.

(7) Displayed in the environment viewer by double tapping the electronic file in the environment viewer, overlaying the electronic file on the virtual paper, and using voice or gesture to instruct "Open this". It is possible to easily display existing electronic files on virtual paper.

[0028]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows the appearance of the first embodiment of the present invention. As shown in FIG. 1, this embodiment has the following constituent elements. An actual paper 2 on the real desk 1, an environment viewer 3 for displaying a virtual desk, a pen (handwriting) input computer 4 for representing virtual paper, a keyboard 9, and a small size.
It is composed of a pen 8 with a scanner having a CCD camera built in the pen, a microphone 6, a speaker 5, and a camera 7 for inputting a state on a desk. In the present invention, these constituent requirements are configured so that the environment is easy for the user to use. The agent 10 is displayed on the environment viewer 3.

On the environment viewer 3, a desktop with a graphical user interface similar to the conventional one is displayed. Files, folders, mail inboxes, intelligent agents 10, etc. are displayed on this monitor. The real thing on the desk (eg real paper 2)
By monitoring the real desk through the camera 7,
Displayed on this virtual desktop.

The pen input computer 4 corresponds to virtual paper. The pen input computer 4 displays files on the virtual desktop and executes applications.
The pen input computer 4 is virtual paper. Therefore, it is operated with a pen. Here, the pen input computer 4 has characteristics of actual paper such as “having an ink mode without recognition” and “two pen input computers are placed sideways to form one large virtual paper”.

A pen 8 with a scanner having a small CCD camera built in the pen is used to interface between the real paper and the virtual paper. That is, the information of actual paper is captured by a pen-type scanner, and the recognized data or raw scan data is transferred to a designated area on the pen input computer.

A plurality of pen input computers, an environment viewer, a pen with a scanner and the like are connected by an indoor (or desktop) LAN such as infrared rays. And these are operated cooperatively.

Instructions to the agent are made using a plurality of modes of pen, voice, gesture and keyboard.
There are multiple input media (such as multiple pen input computers, monitors, microphones) that support the same mode. For example, pen input can be made on any pen input computer, and gestures can be made on any pen input computer or environment viewer. That is, the input mode is distributed to a plurality of input media (devices).

As an output, in addition to being displayed on an environment viewer or a pen input computer, a voice, a sound effect, a graphical display, etc. are used in a multimodal manner by utilizing voice synthesis or the like.

The agent 10 may display it on the environment viewer and use it as a symbol for intelligent processing. Specifically, commands to the system (pen, voice, gesture)
Will be processed by this intelligent agent.

The agent extracts the intention of the user from the multimodal instruction, solves the problem based on the intention, and outputs the result.

FIG. 2 shows a block diagram of this system. This system consists of three workstations 101, 102,
103 and four pen personal computers 105, 106, 1
07, 108, personal computer 104 for input / output, personal computer 1 for executing applications in the background
09.

Workstations 101, 102, 10
The software structure of 3 is the OS 110, and the communication software 111 between the workstation and the personal computer on top of it.
Consists of The workstation 103 is dedicated to voice recognition and camera input, and therefore the voice recognition server 120 and the image input server 121 are operating. In the workstation 101, a control program 112 for the entire personal information environment, a multimodal analyzer (MMA) 113, a print character recognition server 114, and a handwritten character recognition server 115, which will be described later, are operating. In addition to the control program 112 for the entire personal information environment, the workstation 102 includes a multimodal problem solver (MMIP) 117 and a multimodal response generator (MM) which will be described later.
G) 118, the voice synthesis server 119, and the general application program 116 are operating.

In this system, pen personal computers (pen input computers) 105, 106, 107, 108
Also, the personal computer 109 has an OS 127 on which communication software 12 between the workstation and the personal computer is installed.
6. The control program 125 of the entire personal information environment, the virtual paper software 123, and the control program 124 from the outside of the application on the OS 127 are operating. The personal computer 104 also has an OS 127, on which communication software 1 between the workstation and the personal computer is installed.
26, control program 125 for the entire personal information environment, pen-type scanner input 128, keyboard input 12
9. The program of environment viewer management 130 is running.

In this system, as shown in FIG. 3, the intelligent agent 32 is an application 31 running on each pen personal computer (pen input computer) or environment viewer.
It is regarded as a system that interposes the user. The user
It is possible to operate the application directly without going through an intelligent agent, or to use the application with the help of an intelligent agent.

Next, a method of using this personal information device will be described. The user uses it in the following environment as shown in FIG. (A) Multiple pen personal computers (pen input computers) 4, (b) Environment viewer (virtual desktop, intelligent agent) 3, (c) Pen 8 with scanner 8 Used in an environment where hardware and actual paper are mixed. . This environment allows the user to mix and interface pens, voices, and gestures (pen, touch) at the same time.

(1) Basic operation: When the intelligent agent function is not used: Various documents are displayed on the environment viewer (virtual desktop) 5. Here, when the user selects (“opens”) a desired one from the displayed ones, the selected one appears on at least one of the pen input computers in hand, and the operation can be performed.

That is, the user can select from a plurality of operating methods as described below. Operations can be performed using handwriting with a pen, tapping with a pen, a pen-type scanner, and a keyboard. In this case, the merits for the user are as follows.

(A) The work environment is expanded on the desk, and the user can naturally work on the desk using paper or a pen. (B) The virtual paper can be used with the same metaphor as the actual paper. (C) The real paper and the virtual paper. Interface of the pen-type scanner can be taken (for example, if important data exists on the actual paper, it can be easily brought to the virtual paper by the pen-type scanner). (2) Applied operation: Intelligent agent function When to use When using the intelligent agent function, multimodal input / output corresponding to the following input configuration requirements becomes possible. The configuration requirements are (a) pen, (b) voice, (c) gesture (pen), (d) keyboard, and the like. The information possessed by the intelligent agent includes (a) information about the application (meaning of stored data and a method of operating from the outside), (b) personal information (information taught by the user),
(C) Language information (word, thesaurus, meaning), etc.

When the intelligent agent uses the knowledge about the application (address book), the following operations are possible.

For example, when writing a letter with a pen input computer and faxing it to Mr. 500 heads, it is difficult to give his first and last name. Therefore, it is possible to give an instruction as shown in FIG. 4 using voice, pen and gesture (41). In this process, first, semantic analysis of multimodal input is performed. That is, the file of the pen input computer pointed by the pen is "this letter", and it is necessary to consider the voice and the pen input as one sentence (this is the work of multimodal semantic processing (MMIP) described later). ). Next, since the fax requires Mr. 500 flag head's fax number, the intelligent agent searches for the fax number of Mr. 500 flag head with an application called an address book.
At this time, the intelligent agent needs to know the information about the application that the name and the telephone number are stored in the application called the address book, and how the operation of searching can be performed from the outside. I won't. On the contrary, in order for an application to be used by an intelligent agent, it is necessary to have a method of transmitting such information and the function calling function to the agent from the outside.

Also, in the above example, fa
Although the x number is obtained from the address book application, the information taught to the intelligent agent can be used directly. For example, "The fax number of Mr. 500 flag head is xxx. , You don't need an address book application.

FIG. 5 shows a fifth application example using a pen-type scanner.
1 is shown. The user is trying to fax an order form to a mail-order company, but the fax number is a newspaper (actual paper).
Since it is above, input with the pen-type scanner at the same time while instructing "here" by voice. In this case, it is not necessary to strictly scan only the fax number, and if a rough scan is performed, only the appropriate one is cut out and used from the meaning (predicted to be a telephone number in this case).

The user's merit in this application operation is that (a) natural and easy instructions can be given by multimodal input, and (b) an intelligent agent can use complicated applications (compared to other systems). Information can be automatically referred to) and (c) Personal information (memo) is taught to agents in natural language (multimodal input is possible), and it is used as knowledge.

FIG. 6 shows an example of use in office work. "Send this part of the picture to the director by voice or handwriting." When saying "this part" and enclosing the target part in a circle with a pen (Example 61), the agent cuts out a picture from the target part and automatically sends it to the director. Fax to.

[Voice or handwriting "Copy the graph in the budget document here." If you tap the target location with the pen (Example 62) while saying "here" while instructing "," the agent automatically searches for budget materials or stored files, and holds a graph from them. Copy to the place you came and tapped.

[Voice or handwriting: "My face photo is this. If you encircle the photo displayed on the pen input computer or scan the actual photo with a pen-type scanner (Example 63) while saying "this", the agent Save this as knowledge and use it later.

Then, by voice or handwriting, "Display my face picture. ] (Example 64), the agent displays a photo based on the information taught in the previous conversation.

Also, by voice or handwriting, "Tanaka-san's address is here, and the telephone number is this." ],
When saying "here" and "this" and scanning the corresponding part of the business card with the pen-type scanner (example 65), the agent saves this as multimedia knowledge and uses it later.

FIG. 7 shows the overall processing flow. The intelligent recognition mechanisms 71, 73 and 76 have a two-stage feedback loop. The first feedback loop exists inside the intelligent recognition mechanism 71, 73, 76. Here, the recognition result is corrected using the dictionary and morpheme information of the intelligent recognition mechanisms 71, 73, and 76. The second feedback is performed using the constraint information and the prediction information from the multimodal input mechanism 78. The basic recognition engine 72, 74, 75 is realized by using a publicly available ordinary algorithm.
These feedbacks are used in the pen recognition mechanism 75 for cutting out a line and a character, and in the print character recognition mechanism 73 for cutting out a meaningful region, a line, and a character.

On the other hand, the voice recognition mechanism 71 takes a different approach because the recognition rate is worse than that of the above two because of a significant difference. That is, the constraint-driven recognition is mainly performed by first informing the voice recognition of a constraint from the multi-modal input mechanism 78 in the form of a dictionary of recognition candidates or a grammar of the recognition candidate and then recognizing it.

These intelligent recognition mechanisms are shown in FIG.
It is implemented as software modules 114, 115, 120, 121 on the workstations 101, 103.

The multimodal input mechanism 78 uses the expression “syntax description + constraint description (meaning, time)” to represent the knowledge base 77.
Use the data stored in. That is, the syntax description that emphasizes the form and the constraint description that emphasizes the meaning are used together, and in the processing (parsing process), levels from syntax emphasis to constraint emphasis are provided. In other words, if syntax is important, efficient processing becomes possible. If the constraint description is emphasized, it will be possible to analyze even if it is non-syntactic if it takes time. When the constraint description is emphasized, the amount of calculation increases explosively in combination, but the efficiency is increased to a level that can be calculated by using it together with the conventional high-speed purging algorithm.

The multi-modal input mechanism 78 uses the knowledge held in the knowledge base 77 to inform the problem solving mechanism 79 of the result when it can recognize it as an input, but if it cannot recognize it, how to recognize it again. This is transmitted to the intelligent recognition mechanisms 71, 73 and 76.

The main module of the multimodal input mechanism 78 is implemented as a software module 113 on the workstation 101 in FIG.

The problem solving function 79 analyzes the meaning of the recognition result input from the multimodal input mechanism 78 to the problem solving mechanism 79 using the knowledge base 80, and informs the multimodal generating mechanism 82 of the result. If the recognition result from the multimodal input mechanism 78 is a fact, it is registered in the knowledge base 80. The main module of the problem solving mechanism 79 is the workstation 10 in FIG.
2 is implemented as a software module 117.

Based on the response data input from the problem solving mechanism 79 to the multi-modal generation mechanism 82, the multi-modal generation mechanism 82 outputs the characters / images 83, 85 from the environment viewer and virtual paper, the environment viewer, the virtual viewer. The output 84 is output by paper and voice from the agent. How to select output media and output method
Is derived using. For example, the stress and speed of speech are adaptively set based on the relationship between the question and the answer.

The main module of the multi-modal generation mechanism 82 is implemented as a software module 118 on the workstation 102 in FIG.
Also, the voice output module is the workstation 10.
2 above is the software module 119.

In the processing flow of the system shown in FIG. 7,
The input 87 from the subsystem represents, for example, a report of the operating state from an application operating on the virtual paper. Using this, the problem solving mechanism 79 executes problem solving in consideration of the current state of the application.

The output 86 to the subsystem is not only the control instruction to the application running on the virtual paper but also the control instruction to the environment viewer, the agent, and the virtual viewer, which are the constituent elements of the personal information environment. Including. That is, in the problem solving mechanism, in addition to the response policy to the user, the execution control policy of the own system and the other application is simultaneously generated.

[0066]

As described above, according to the embodiments of the present invention, the human-centered / real-world-based design is adopted.
It becomes possible to realize personal information having a computer use environment having a more human-centered user interface capable of interacting in a multimodal form such as voice, handwriting, and gesture.

[Brief description of drawings]

FIG. 1 is a diagram showing a first embodiment of the present invention.

FIG. 2 is a diagram showing a program structure and a device configuration.

FIG. 3 is a diagram showing the positioning of agents.

FIG. 4 is a diagram illustrating an example of a usage form.

FIG. 5 is a diagram illustrating an example of a usage form.

FIG. 6 is a diagram illustrating an example of a usage form.

FIG. 7 is a diagram illustrating a processing flow of the present invention.

[Explanation of symbols]

1 ... Real desk, 2 ... Actual paper, 3 ... Environmental viewer displaying virtual desk, 4 ... Pen input computer representing virtual paper, 5 ... Speaker, 6 ... Microphone, 7 ... Camera, 8
… Pen with scanner that has a built-in small CCD camera, 9
… Keyboard, 10… Agent

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Taminori Tomita 1099, Ozenji, Aso-ku, Kawasaki-shi, Kanagawa Ltd. System Development Laboratory, Hitachi, Ltd. (72) Hirokazu Aoshima 1099, Ozen-ji, Aso-ku, Kawasaki, Kanagawa Ceremony company Hitachi Systems Development Laboratory

Claims (1)

[Claims] 1. A personal information device for supporting a worker's work, wherein a recognition means for recognizing what is actually present among those necessary for the work, and a recognized real thing that can be operated by a computer Distinguishing between those that are virtually impossible, those that exist virtually only, those that actually exist and can be operated by the computer, and those that exist in the real world and cannot be operated by the computer. A personal information device, comprising: display means for displaying, wherein the display of the display means displays the thing that actually exists in correspondence with the position where it exists.