JPH0863319A - Information processor - Google Patents

Abstract

PURPOSE: To simultaneously combine plural input means with each other for input operations in order to give an operation to such a part that cannot be directly operated since it is not visualized on a display and also to attain an operation to a visualized part with presence of plural corresponding objects of selection. CONSTITUTION: An information processor contains a voice input means such as a microphone 101, etc., a pointing input means such as a pen 102 or a mouse, etc., a command execution means 109, a screen display means 110 which changes the screen display in response to a command, and a screen output means 111 such as a display, etc., that outputs a screen. Furthermore an information integration means 108 is added to integrate the voice input information and the pointing input information and to interpret them. In such a constitution, the information processor can possess a function to interpret the operations including such non-direct operations as an operation of an invisible part of a screen, a collective selection operation of plural objects, etc., which are performed by a user by means of the voice input means and the pointing input means.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a user interface for file management, document editing, graphic editing, etc. mounted on OA equipment such as personal computers, workstations and word processors, and provides a user-friendly information input method. .

[0002]

2. Description of the Related Art A conventional graphical user interface realizes a simple operation by visualizing character information so that it can be directly operated. At that time, as the input means, a pointing device such as a mouse or a pen, or a keyboard for inputting characters,
It accepts only voice input alone. In part,
There is also an interface that allows you to operate multiple objects by dragging and dropping with a pointing device while pressing a specific key on the keyboard, and there are examples that deal with operating methods that use multiple input means at the same time, In a conventional interface that mainly uses a keyboard as a character input means, such an operating method is unnatural and difficult for an unskilled user to remember.

[0003]

In the above-mentioned prior art, in many situations, only input by a single input means is accepted. Therefore, it is possible to perform an operation on a part that cannot be directly operated because it is not visualized, or an operation in a situation where selection is troublesome such as when there are multiple selected objects that are visualized. Inevitably, the menu becomes complicated. Therefore, an object of the present invention is to use the keyboard, which is an input means that requires skill of the user, as much as possible in order to make the operation natural and easy in the above situation where it is difficult to operate with only a single input means. It is to avoid the input, and to accept the input used by combining a plurality of input means at the same time.

[0004]

In order to solve the above problems, at least a voice input means such as a microphone, a pointing input means such as a pen or a mouse, and a voice recognition means for recognizing a voice input by a user. ,
Information processing apparatus having pointing input information detection means for detecting pointing input information input using pointing input means, command execution means for executing a command, and screen display means for changing the screen display in response to the command In the above, an information integration means for integrating and interpreting the voice input information and the pointing input information is provided.

[0005]

At least a voice input means such as a microphone, a pointing input means such as a pen or a mouse, and a voice recognition means for recognizing voice input by a user.
Information processing apparatus having pointing input information detection means for detecting pointing input information input using pointing input means, command execution means for executing a command, and screen display means for changing the screen display in response to the command In the above, by providing an information integration means for integrating and interpreting the voice input information and the pointing input information, the voice input information and the pointing input information which are simultaneously input by the user using the voice input means and the pointing input means. By integrating and interpreting, it becomes possible to process operations including indirect operations, such as operations of invisible parts on the screen that are difficult to realize with a single input means and batch selection operations of multiple objects.

[0006]

EXAMPLES Examples will be described in detail below with reference to the drawings.
Here, a description will be given assuming a graphical user interface for managing files in a personal computer or the like as an information processing device. However, the present invention can be applied not only to the interface but also to general user interfaces such as database search, graphic editing, and document editing.

FIG. 1 is a block diagram showing an embodiment of an information processing apparatus of the present invention. In FIG. 1, the voice input means 101 is a tool such as a microphone used for inputting voice. The voice detection unit 103 is a unit that detects the voice section and the input time of the user from the information input by the voice input unit 101. Voice recognition means 104
Is a means for analyzing the detected voice of the user and outputting a corresponding word string. Voice input information analysis means 105
Is a means for dividing or integrating voice input information into units having a meaning and analyzing the meaning information. Pen input means 1
Reference numeral 02 is a tool such as an electronic pen capable of inputting to a computer. The pen input information detecting means 106 is the pen input means 1
02 is a means for detecting the content of the information input by 02 and the input time. The pen input information analysis unit 107 is a unit that analyzes the semantic information of the pen input information. The information integration unit 108 is a unit that interprets by integrating voice input information in which semantic information is divided or integrated into semantic units and semantic information is analyzed, and pen input information in which semantic information is analyzed. The command execution unit 109 is a unit that executes a file management command. The screen display means 110 is means for changing the screen display in response to the execution of a command. The screen output unit 111 is a unit such as a display that outputs characters or graphics.

The present invention is different from the conventional one in that the information integrating means 108 is provided.

In the embodiment shown in FIG. 1, first of all, the voice of the user is detected by the voice detecting means 103 from the information inputted using the voice inputting means 101. As the voice of the user, as a command utterance related to file management, word voices such as “move” and “right”, and continuous voices such as “I want to move this file here” can be input. After these voices are input using the voice input means 101, the voice detection means 103 detects the voice section and at the same time,
The start time and end time of the detected voice section are measured.
The voice recognition unit 104 performs a normal voice recognition process on the voice section detected by the voice detection unit 103, and outputs a word string as a recognition result. The voice input information analysis unit 105, for the word string that is the output of the voice recognition result,
Semantic information is analyzed by dividing or integrating into semantic information units. On the other hand, the information input using the pen input unit 102 is detected by the pen input information detection unit 106, and at that time, the start time and the end force time of the pen input are measured. Note that the pen input information may be a discrete operation such as pressing the pen to point to an object such as an icon showing the concept of a file or directory displayed on the screen, or moving with the pen pressed. There is input information obtained by various continuous operations. The pen input information analysis means 107 means the name of the object pointed to when the object is pointed to, as the meaning of the information input by the pen.
When a gesture is input, the command name corresponding to the gesture is analyzed and output as the meaning of pen input information. The information integration unit 108 interprets the intention of the user's entire operation based on the semantic information of the voice input information and the semantic information of the pen input information regarding the information input simultaneously using the voice input unit 101 and the pen input unit 102. . The intention corresponds to the command executed by the command executing means 109.

FIG. 2 is a block diagram showing an embodiment of the information integration means 108 of FIG. In FIG. 2, the time synchronization analysis unit 201 is a unit that analyzes the temporal correspondence between the voice input information and the pen input information and determines the time synchronism. The meaning collating means 202 is means for comparing and collating the meanings of the voice input information and the pen input information which are synchronized in time. The frame creating means 203 is means for creating a frame in which the condition for executing the command by the command executing means 109 is described from the voice input information and the pen input information, using the result of the meaning matching means 202.

First, from the voice input information analysis means 105,
The semantic information of the input voice input information, the input start time and the end time are output, and the pen input information analysis unit 107 is output.
From, the semantic information of the pen input information and the input start time and end time are output. The time synchronization analysis means 201 receives these pieces of information as input and analyzes the time synchronization of the voice input information and the pen input information. For example, if the time pointed by pressing the pen on the file icon displayed on the screen is within a predetermined time difference with respect to the appearance time of the voice section "This file", It is determined that the pen input information and the voice input information are synchronized. Here, when the voice input information and the pen input information are synchronized, a heuristic for determining that the input for the same concept is made by both the voice input means 101 and the pen input means 102 in FIG. To use. With this heuristic, the time synchronization analysis means 2
01 is linked as information showing the same concept with respect to voice input information and pen input information which have temporal synchronization,
Output. The meaning matching unit 202 is the time synchronization analysis unit 2.
The meaning of the voice input information and the pen input information linked by 01 is collated. If the respective meanings match, the linked voice input information and pen input information can be integrated into one. If the meanings do not match, the content of the pen input information is preferentially adopted and the content of the voice input information is discarded in view of the reliability of the input information. However, if the content of the pen input information is ambiguous, the meaning of the voice input information is used to clarify the content of the pen input information. Although the unit of the voice input information is assumed to be a phrase unit here, the content of the process is essentially the same in a word unit, a sentence unit, and a phrase unit.
The frame creating means 203 interprets from the voice input information and the pen input information, which are information obtained by integrating the information showing the same concept in the meaning collating means 202. The result of the interpretation is a frame consisting of commands for moving and deleting files and their parameters.

FIG. 3 is a flow chart showing an embodiment of the time synchronization analysis means 201 of FIG. From the voice input information analysis means 105 of FIG.
From 7, the meaning information of the pen input information and the input start time,
The end time is input to the time synchronization analysis means 201. Upon receiving these pieces of information, the time synchronization analysis unit 201 analyzes what kind of temporal correspondence the voice input information and the pen input information have, as shown in the flowchart of FIG.
Figure 3 shows the temporal correspondence between voice input information and pen input information.
There are 6 types shown in [Type of temporal correspondence]. Of these, it can be said that there is always synchronism in the correspondence relationship of (1), (2), (3), and (4), and in (5) and (6), the appearance time of the voice input information and the pen input information is set in advance. It can be said that there is synchronism if the condition of being within the defined time difference is satisfied.

For example, as shown in the legend of FIG. 3, it is assumed that the input start time and end time of the voice input information are Tss and Tse, respectively. When the input start time and end time of pen input information are Tps and Tpe, respectively, Tss = 100ms, Tse = 1200m
It is assumed that s, Tps = 300ms and Tpe = 500ms. In this case, in the flowchart of FIG. 3, first, s301, s302,
Following the flow of s308 and s310, it is determined that the voice input information and the pen input information have synchronism. The temporal correspondence relationship in this case is shown in [Type of temporal correspondence relationship] of FIG.
Corresponds to (1).

For example, Tss = 600ms, Tse = 1500ms, Tps =
In the case of 300ms, Tpe = 400ms, if c = 500ms is set in s305, the flow of s301, s302, s303, s304, s305, s306 will be followed, and it will be judged that the voice input information and pen input information correspond. It The temporal correspondence in this case is
This corresponds to (5) of [Type of temporal correspondence] in FIG.

In this way, the time synchronization analysis means 201 of FIG. 2 determines the synchronism between the voice input information and the pen input information, and further links the synchronized voice input information and the pen input information and outputs them.

FIG. 4 is a block diagram showing an embodiment of the meaning collating means 202 of FIG. The time synchronization analysis unit 201 of FIG. 2 outputs the voice input information and the pen input information which are linked by being synchronized in time, and the semantic information collating unit 401 of FIG. 4 outputs the linked voice input information. The semantic information of the pen input information is collated. Since it is assumed that the voice input information and the pen input information indicate the same concept if they are synchronized in time, they can have the same semantic information, but the voice input information includes “this”, “here”, etc. There is a possibility that input information having different meaning information may be linked even if they are synchronized in time, due to a word having meaning information that does not exist in the pen input information or an error in voice recognition. Therefore, the semantic information is collated, and if they do not match, the content of the pen input information is preferentially adopted because of the reliability of the input information. However, if there is some ambiguity in the interpretation, such as whether it should be judged as the object or the destination by just pointing the directory icon with the pen, it cannot be interpreted only by the pen input information. . In order to deal with such a case, the semantic feature information collating unit 402 uses the semantic feature information of the voice input information corresponding to the pen input information whose interpretation is ambiguous to determine the ambiguity of the interpretation of the pen input information. Resolve.

FIG. 5 is a diagram showing a data flow of an embodiment of the meaning collating means 202 of FIG. User says "File 1" while saying "Copy file 1 here"
While talking, point the file1 icon on the screen with the pen,
This is an example of an operation of pointing the window2 area on the screen with a pen while uttering "here". At this time, since the information of the expression "here" as the voice input information and the information of the object window2 as the pen input information are temporally synchronized, they are linked by the time synchronization analysis means 201 of FIG. . The semantic information collating means 401 of FIG. 4 compares and collates the semantic information “here” of the voice input information 2 and the semantic information “window2” of the pen input information 2 of FIG. 5, and returns a determination result of non-coincidence. Next, the semantic feature information collating means 402 of FIG. 4 compares and collates the semantic feature information “location” of the voice input information 2 and the semantic feature information “location” of the pen input information 2 and returns a match determination result. As a result, the linked voice input information 2 and pen input information 2 are integrated, and "window2" as the semantic information,
It is replaced with the integrated information 2 having "location" as the semantic feature information.

FIG. 6 is a diagram showing input / output data of an embodiment of the frame creating means.

The time synchronization analysis unit 201 and the semantic matching unit 202 of FIG. 2 integrate the information about the voice input information and the pen input information which are judged to exhibit the same concept by synchronizing in time, and the integrated input is made. A frame is created by the frame creating means 203 in FIG. 2 from the information. The frame here is a result of interpretation of the intention of the user's operation, and is composed of commands relating to file management such as file movement and deletion, and parameters required for command execution.

For example, in FIG. 6, the integrated information shown in (1) is information obtained by integrating the voice input information and the pen input information which already have the same concept. In the command frame shown in (2), the types of commands executed by the command execution means 109 of FIG. 1, such as file movement, deletion, and copy related to file management, the object name necessary for execution, the destination name, etc. It consists of parameters. In FIG. 6, for example, from the semantic feature “object” of the integrated information 1, the semantic information of the integrated information is described in the slot of object in the command frame. Similarly, describe the semantic information of the integrated information 2 to 4 in the slot of the command frame,
Create a command frame.

FIG. 7 is a diagram showing a form of use of an embodiment of the device of the present invention. In this figure, the display 701 is a screen output means. The window 702 is an area for displaying the contents of the directory. The icon 703 is a graphic showing the concept of a file.

In this figure, while uttering "Please move the icon on the back here," the user points the area (1) on the display with a pen while uttering "icon", By pointing the area (2) on the display with a pen during the utterance of "ni", it is possible to operate an object that is not visible on the screen. By providing the information integration means 108 of FIG. 1 in the device of the present invention, the above-mentioned operation and the directory or file below or above the object displayed on the screen, which is not displayed on the screen, can be designated by voice. You can interpret the operation of the invisible part such as the operation.

FIG. 8 is a diagram showing a form of use of an embodiment of the device of the present invention.

In this figure, the user utters "Please copy this to the window above" while pointing at the area indicated by (1) on the display while uttering "This" It is possible to directly specify the relative position by voice without pointing at the copy destination area. In the device of the present invention, the information integration means 108 of FIG.
By providing, it is possible to interpret an indirect operation that is an alternative to the direct operation, such as the above operation.

FIG. 9 is a diagram showing a usage pattern of an embodiment of the apparatus of the present invention.

In this figure, the user utters "Please select all icons of this shape", while pointing "(1)" on the display during the utterance of "this shape", By specifying the attribute of shape, it is possible to collectively select a plurality of objects. By providing the information integration means 108 of FIG. 1 in the device of the present invention, it is possible to interpret the above-described operation and the operation of specifying attributes such as color and size by voice.

[0027]

EFFECTS OF THE INVENTION At least voice input means such as a microphone, pointing input means such as a pen or a mouse, voice recognition means for recognizing voice input by a user, and pointing input information input using the pointing input means. Pointing input information detection means for detecting the, and command execution means for executing the command,
An information processing apparatus having a screen display unit for changing the screen display in response to the command includes an information integration unit for integrating and interpreting voice input information and pointing input information, so that the user can perform the voice input unit. And the voice input information and the pointing input information which are simultaneously input using the pointing input means are integrated and interpreted, and the operation of the invisible part on the screen and the batch selection operation of a plurality of objects which are difficult to realize with the single input means are performed. It made it possible to process operations, including indirect operations.

[Brief description of drawings]

FIG. 1 is a block diagram showing an embodiment of an information processing device.

FIG. 2 is a block diagram showing an embodiment of information integration means.

FIG. 3 is a flowchart showing an embodiment of time synchronization analysis means.

FIG. 4 is a block diagram showing an embodiment of a meaning matching unit.

FIG. 5 is a diagram showing a data flow of an embodiment of a meaning matching unit.

FIG. 6 is a diagram showing input / output data of an embodiment of a frame creating means.

FIG. 7 is a diagram showing a usage pattern of an embodiment of the device of the present invention.

FIG. 8 is a diagram showing a usage pattern of an embodiment of the device of the present invention.

FIG. 9 is a diagram showing a usage pattern of an embodiment of the device of the present invention.

[Explanation of symbols]

101 ... Voice input means, 102 ... Pen input means, 103
... voice detection means, 104 ... voice recognition means, 105 ... voice input information analysis means, 106 ... pen input information detection means, 1
07 ... Pen input information analysis means, 108 ... Information integration means,
109 ... Command execution means, 110 ... Screen display means, 1
11 ... Screen output means.

Claims (7)

[Claims] 1. A voice input means such as at least a microphone, a pointing input means such as a pen or a mouse, a voice detection means for detecting a voice of a user inputted by the voice input means, and a detected voice input. Voice recognition means for recognizing information by voice, voice input information analysis means for analyzing the voice input information, pointing input information detection means for detecting pointing input information input using the pointing input means, and the pointing Information having pointing input information analysis means for analyzing input information, command execution means for executing a command, screen display means for changing the screen display corresponding to the command, and screen output means for outputting a screen such as a display In the processing device, the voice input information and the pointing An information integration unit that integrates and interprets input information is provided, and a user uses the voice input unit and the pointing input unit, such as an operation of an invisible portion on a screen or a collective selection operation of a plurality of objects, An information processing apparatus having a function of interpreting an operation including an indirect operation. 2. The information processing apparatus according to claim 1, wherein the information integration unit is a time period of voice input information input using the voice input unit and pointing input information input using the pointing input unit. Command as a result of the interpretation from the result of the meaning matching means, the time synchronization analysis means for analyzing the temporal synchronism, the meaning matching means for matching the meanings of the voice input information and the pointing input information which are synchronized in time. An information processing apparatus having a frame creating means for creating an execution condition, and having a function of interpreting an operation in which the voice input information and the pointing input information are asynchronously input. 3. The information processing apparatus according to claim 1, wherein the time synchronization analysis means includes the voice input information input using the voice input means and the pointing input information input using the pointing input means. An information processing apparatus having a function of analyzing temporal correspondence and further determining time synchronism. 4. The information processing apparatus according to claim 1, wherein the meaning collating means has meanings and meaning features with respect to the voice input information and the pointing input information judged to have temporal synchronism by the time synchronization analysis means. An information processing device having a function of comparing and collating and complementing semantic concepts such as. 5. The information processing apparatus according to claim 1, wherein the information integration unit integrates the voice input information regarding the object invisible in the screen output unit with other input information to form an invisible portion. An information processing apparatus having a function of interpreting the operation of. 6. The information processing apparatus according to claim 1, wherein the information integration means integrates voice input information relating to a relative position with respect to a designated object with other input information, thereby producing a voice. An information processing apparatus having a function of interpreting an operation in which designation of an object is an alternative to a direct operation. 7. The information processing apparatus according to claim 1, wherein said information integration means integrates voice input information relating to attributes such as color, shape and size with other input information.
An information processing apparatus having a function of interpreting an operation of collectively selecting a plurality of objects having common attributes such as color, shape, and size by voice.