JPH07110734A - Multimodal input analysis system - Google Patents

Abstract

PURPOSE:To provide the multimodal input analysis system where it is unneces sary to point an object at each time and an input means is freely selected without forcing a user to input a voice. CONSTITUTION:This system is provided with a pointing input means 1, a voice input means 2, an object information storage part 3 where the position and feature information of each object are stored, an operation object holding part 4 where information of an object as the operation object is stored, a pointing event table 5 where the processing to be executed correspondingly to pointing input is described, a voice event table 6 where the processing to be executed correspondingly to the pattern of voice input is described, a history holding part 7 where the input history is held, an integrating and interpreting part 8 which integrates and interprets input data from the pointing input means and the voice input means to determine a processing, and a processing execution part 9 which executes the determined processing.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an input analysis system, and more particularly to a multimodal analysis system for integrating pointing input and voice input for analysis.

[0002]

2. Description of the Related Art As an interface integrating a conventional pointing device and a voice input, a voice input of "move this", a pointing input to an object corresponding to "this", and a position corresponding to "here" There is a method of interpreting by combining with pointing input to (Reference: "Examination of information integration model in multimodal interface", Ando et al., Proc. Of the 8th Human Interface Symposium, 1992).

[0003]

In the conventional multi-modal input method, when instructing an operation, it is necessary to input the pointing to the operation target and an instruction word such as "this" in combination. For example, when moving a certain object, it is necessary to perform a voice input "Move this to here", pointing to the object corresponding to "This" at almost the same time, and pointing to a position corresponding to "Here". is there. Even when the operation is repeated for the same object, the object must be pointed again. Further, there is a drawback that the operation instruction must be given by voice.

[0004]

SUMMARY OF THE INVENTION It is an object of the present invention to provide a multimodal input analysis system which does not require pointing of an object each time and which can freely select input means without forcing voice input. .

Therefore, the multimodal input analysis system of the present invention includes an object information storage section for storing the position and characteristic information of each operation target object, and an operation target holding section for storing information of the current operation target object. A history holding unit for storing an input history, a pointing event table describing a process to be performed according to pointing input, a voice event table describing a process to be performed according to a voice input pattern, and pointing input means And an input determination unit that receives input data from the voice input unit and distributes processing according to the input data, and receives only pointing input data from the input determination unit and refers to the pointing event table to determine a process to be executed. Pointing event alone Section and the list of voice input data and pointing input data from the input determination unit, compares the number of pointing expressions in the voice input data with the number of pointing input data, and if there is less pointing input data, the history Whether the latest input data in the holding unit is a pointing input, whether the object information in the operation target holding unit matches the instruction expression that appears first in the voice input data, and whether the next input is a pointing input In this order, and when the corresponding data is found, the pointing information acquisition unit that adds the data to the pointing input data list, and the pointing input data list and the voice input data are received, and the meaning in the voice input data is received. Representation and pointing input data list The pointing data is compared in the order of appearance, and if the meaning expression matches the meaning expression, the identification processing unit that replaces the meaning expression with the pointing data and the meaning expression after the identification processing are received, and executed by referring to the voice event table. It is characterized by having a process determining unit that determines a process and a process executing unit that executes the determined process and reflects the result on the object storage unit and the operation target holding unit.

[0006]

Embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a multimodal input analysis system which is an embodiment of the present invention. This multimodal input analysis system includes a pointing input means 1 such as a mouse and a joystick, a voice input means 2 for inputting a voice, an object information storage section 3 for storing the position and characteristic information of each object, and an operation target. The operation target holding unit 4 for storing the information of the object, the pointing event table 5 describing the processing to be performed in response to pointing input, and the processing to be performed in response to the voice input pattern are described. The voice event table 6, the history holding unit 7 that stores the input history, the integrated interpretation unit 8 that integrates and interprets the input data from the pointing input unit and the voice input unit, and determines the processing are determined. And a process execution unit 9 that executes a process.

The integrated interpretation unit 8 includes an input determination unit 21, a pointing event independent processing unit 22, a pointing information acquisition unit 23, an identification processing unit 24, and a processing determination unit 25.
Composed of and.

The voice input means 2 outputs a voice input start signal when voice input is started. When the voice input is completed, the content of the input voice is recognized and the recognition result is output. The recognition result is represented by a semantic expression holding the input order. For example, the voice input "Move this Maru here" is represented by a list as shown in FIG.

The input determination section 21 receives data from the pointing input means 1 and the voice input means 2 and sorts the processing according to the received data. Input determination unit 21
Sends the data received from the pointing input unit 1 to the pointing event independent processing unit 22 while not receiving the voice input start signal from the voice input unit 2. When the voice input start signal is received, the data received from the pointing input means 1 is stored as a pointing information list until the voice input ends, that is, until the next voice input data is received. When the voice input data is received, the voice input data and the pointing information list are stored in the pointing information acquisition unit 23.
Send to.

Pointing event independent processing unit 22
Refers to the pointing event table 5 to extract the processing procedure corresponding to the input data, and the processing execution unit 9
Send the processing procedure to. The pointing information acquisition unit 23,
When the voice input data and the pointing information list are received from the input determination unit 21, the number of instruction expressions in the voice input data is compared with the number of pointing inputs in the pointing information list, and if the number of pointing inputs is small, the history is retained. The pointing information is acquired by referring to the section 7 and the operation target holding section 4 or by obtaining the next pointing input, and added to the pointing information list.

The identification processing unit 24 receives the voice input data and the pointing information list from the object information acquisition unit 23 and identifies the pointing expression and the pointing information in the voice input data.

The process determining section 25 refers to the voice event table 6 and extracts a processing procedure corresponding to the pattern of the voice input data and sends it to the process executing section 9.

The processing execution unit 9 performs the actual processing according to the received processing procedure, and changes the information in the object information storage unit 3 and the operation target holding unit 4 according to the processing result.

FIG. 2 is a diagram showing the flow of processing of the integrated interpretation unit 8 of this embodiment. The details of the process of the integrated interpretation unit 8 of the present embodiment will be described with reference to FIG. 2 by using an actual example.

Step S31 The input from the pointing input means 1 and the voice input means 2 is awaited.

Step S32: It is determined whether the received input data is a voice input start signal. If the received input is a voice input start signal, the process from step S34 is performed. If not, that is, if the input is pointing input data, the input data is passed to the pointing event independent processing unit 22.

In step S33, the pointing event independent processing section 22 uses the pointing event table 5
With reference to, the processing procedure corresponding to the input data is extracted and sent to the processing execution unit 9. The processing execution unit 9 sequentially executes processing according to the received processing procedure.

Step S34 If the received input data is the voice input start signal, the input judging section 21 waits until the next voice input data is received. The pointing input data input during this period is stored in the pointing information list. This process ends when the voice input data is received, and the voice input data and the pointing information list are passed to the pointing information acquisition unit 23.

Step S35 The pointing information acquisition unit 23 compares the number of instruction expressions in the voice input data with the number of pointing inputs in the pointing information list. For example, when the voice input data is the semantic expression shown in FIG. 5, the number of instruction expressions is the number of elements having the "instruction word ()" item in the sublist, which is 2 in this example. If the number of pointing inputs is larger than the number of pointing expressions, the process proceeds to step S36, if it is less, the process proceeds to step S37, and if the same, the process proceeds to step S38.

In the above example, when the pointing information list is a list consisting of one element such as [pointing (X, Y)], the number of instruction expressions is 1 and the number of pointing inputs is smaller. Therefore, the process proceeds to step S38.

Step S36 If the number of pointing inputs is larger than the number of pointing words, error processing such as displaying an error message is performed, and step S31 is performed.
Return to.

Step S37 If the number of pointing inputs is less than the number of instruction expressions, the history holding unit 7 and the operation target holding unit 4 are referenced to acquire the missing pointing information.

The acquisition of pointing information is performed in the following order. The pointing information may be pointing input data or information of a specific object.

FIG. 3 shows an object information acquisition method when the number of pointing inputs is one less than the number of instruction expressions.

Step S51 Referring to the history holding unit 7, it is checked whether or not the last input was the pointing input. If it is pointing input, add it to the beginning of the pointing information list and end.

Step S52 If the first instructional expression is the designation of the object, the operation object holding unit 4 is referred to,
If there is a current operation target object, the object information is added to the beginning of the pointing information list and the processing ends. At this time, if the instruction expression includes a condition other than the instruction word, it is checked whether the operation target object meets the condition. For example, in the example of FIG. 5, the "type (maru)" item becomes a condition, and it is checked whether the shape of the operation target object is "maru". If it does not meet the conditions, it is considered a failure.

Step S53 Wait for the next input. At this time, it waits until there is a pointing input or a predetermined time elapses after the voice input ends. If there is pointing input, add it to the end of the pointing information list and exit. If it times out, it fails and ends.

FIG. 4 shows a method for obtaining object information when the number of pointing inputs is two smaller than the number of pointing expressions.

Step S61 Referring to the history holding unit 7, it is checked whether or not the last input was the pointing input. If it is a pointing input, it is added to the head of the pointing information list and the process proceeds to step S63.

Step S62 If the instruction expression that appears first is the designation of the object, the operation object holding unit 4 is referred to,
If there is a current operation target object, that object information is added to the top of the pointing information list. If there is no current operation target object, it fails and ends. At this time, if the instruction expression includes a condition other than the instruction word, it is checked whether the operation target object meets the condition. If it does not meet the conditions, it is considered a failure.

Step S63 Wait for the next input. At this time, it waits until there is a pointing input or a predetermined time elapses after the voice input ends. If there is pointing input, add it to the end of the pointing information list and exit. If it times out, it fails and ends.

Returning to FIG. 2, if the acquisition of pointing information fails in the above process, error processing is performed in step S36 and the process returns to step S31.

Step S38 The identification processing section 24 carries out an identification process of the pointing expression and the pointing information in the voice input data.

The pointing expression in the semantic expression of the voice input data and the pointing information in the pointing information list are identified in the order of appearance. This identification is performed by the following procedure.

When the designated expression is the designation of the object and the pointing information is the object information, the designated expression is replaced with the object information.

When the instruction expression is the designation of an object and the pointing information is pointing input data, that is, (X, Y) coordinate data, the object information storage unit is referenced to locate at the (X, Y) coordinates. Get information about an object. If conditions are added at this time,
Find an object that meets the conditions. Replace the instruction expression with object information.

When the designated expression is the designation of the position, the designated expression is replaced with the (X, Y) coordinates of the pointing information.

For example, the semantic representation of FIG. 5 can be rewritten as shown in FIG.

Step S39 Audio event table 6
With reference to, a voice input pattern that matches the semantic expression of the input generated in step S38 is searched, and a processing procedure corresponding to the voice input pattern is extracted and sent to the processing execution unit 9.

Step S40 The input data is stored in the history holding unit 7, and the process returns to step S31.

[0042]

As described above, since the multimodal input analysis system according to the present invention uses the immediately preceding input history and the current operation target object for interpretation,
When the operation is repeated for the same target object, it is not necessary to point the target object every time. Even if there is no voice input, the pointing input is processed independently, so
No voice input is forced.

[Brief description of drawings]

FIG. 1 is a block diagram showing an embodiment of a multimodal input analysis system of the present invention.

FIG. 2 is a diagram showing an example of a processing flow of an integrated interpretation unit of the present invention.

FIG. 3 is a diagram showing an example of a processing flow of a pointing information acquisition unit of the present invention.

FIG. 4 is a diagram showing another example of the processing flow of the pointing information acquisition unit of the present invention.

FIG. 5 is a diagram showing an example of a semantic representation of voice input data.

FIG. 6 is a diagram showing an example of a semantic expression after identification processing.

[Explanation of symbols]

 1 Pointing Input Means 2 Voice Input Means 3 Object Information Storage Section 4 Operation Target Holding Section 5 Pointing Event Table 6 Voice Event Table 7 History Holding Section 8 Integrated Interpretation Section 9 Processing Execution Section 21 Input Judgment Section 22 Pointing Event Independent Processing Section 23 Pointing Information acquisition unit 24 Identification processing unit 25 Processing determination unit

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI technical display location G10L 3/00 531 D 9379-5H

Claims (3)

[Claims] 1. An object information storage unit that stores the position and characteristic information of each operation target object, an operation target holding unit that stores information about the current operation target object, and a history holding unit that stores an input history. Section, a pointing event table that describes processing that should be performed according to pointing input, a voice event table that describes processing that should be performed according to a voice input pattern, and input data received from pointing input means and voice input means, An input determination unit that distributes processing according to input data; a pointing event independent processing unit that receives only pointing input data from the input determination unit and refers to the pointing event table to determine a processing to be executed; The voice input data and poi The list of pointing input data is received, the number of pointing expressions in the voice input data is compared with the number of pointing input data, and if the pointing input data is less, the missing pointing information is acquired and the pointing input is performed. The pointing information acquisition unit to be added to the data list, and the pointing input data list and the voice input data are received, and the semantic representation in the voice input data and the pointing data in the pointing input data list are compared in the order of appearance to obtain the semantic representation. If they match, the identification processing unit that replaces the semantic expression with pointing data, the processing determination unit that receives the semantic expression after the identification processing and determines the processing to be executed by referring to the voice event table, and the determined processing Execute and the result is the object Multimodal input analyzing system characterized by comprising a processing execution section to be reflected in the storage unit and the operation target holding unit. 2. The lacking pointing information is acquired by determining whether the latest input data in the history holding unit is pointing input, and the object information in the operation target holding unit appears first in the voice input data. 2. The multimodal input analysis system according to claim 1, wherein the multimodal input analysis system is performed by checking whether or not the specified expression is satisfied and whether or not the next input is a pointing input in this order, and finding the corresponding data. 3. The multimodal input analysis system according to claim 2, wherein when the corresponding data is found, the data is added to the pointing input data list.