KR20080051479A - Apparatus and method for processing multimodal fusion - Google Patents

Abstract

A device and a method for processing multimodal fusion are provided to infer various kinds of user input modality fusion easily and conveniently in a system including various kinds of modality recognition engines even if the system has a compact and low specification. An input processor(110) processes inferable information by receiving single or various kinds of user input recognition information through voice and gesture recognizers(210,220). A rule storing part(150) stores modality combination rule information by parsing XML(eXtensible Markup Language) type modality combination rule information. An inference engine(120) infers action corresponding to the processed user input recognition information by referring to the modality combination rule information. The XML type modality combination rule information includes a first element defining one action, a second element arranging modalities received as input of the action, a third element presenting a method for combining the arranged input modalities significantly, a fourth element arranging instructions of the modality, a fifth element pointing one instruction, and a sixth element combining many actions.

Description

Apparatus and method for processing multimodal fusion

1 is a block diagram showing a multi-modal fusion machine according to an embodiment of the present invention,

2 is a flowchart illustrating a modality fusion method of a multi-modal fusion device according to an exemplary embodiment of the present invention.

* Description of the symbols for the main parts of the drawings *

100: multi-modal fusion machine 110: input processing unit

120: inference engine 130: verification unit

140: rule storage unit 150: feedback generation unit

210: speech recognizer 220: gesture recognizer

The present invention relates to a multi-modal fusion processing method and apparatus, and more particularly, to a multi-modal fusion processing method and apparatus for easily performing a multi-modality fusion in the system.

Currently, with the development of technology, the performance of recognition engines for various modalities is improved, and interest in multimodal input / output is increasing.

Accordingly, systems such as general computers, wearable computers, wireless communication terminals, portable multimedia players, and the like have not been able to accurately recognize user input due to complex and low performance modality recognition engines in implementing multimodal input / output. Although difficult, the implementation is facilitated by improved performance modality recognition engines.

On the other hand, such a system should be equipped with a multi-modal fusion system to infer what the intention of the user input is recognized by the modality recognition engine, the current multi-modal fusion system is very complicated to infer the user intention In order to deduce intention, it usually requires a high-end server-class system, a dialogue management system based on natural language processing, and a large amount of knowledge information system.

In the system requiring multi-modal input and output, even though the performance of the recognition engines for various modalities is improved, as the multi-modal fusion system provided requires a high specification system and a large amount of knowledge information system, Multi-modal input and output implementations are difficult with systems and low-spec systems.

On the other hand, in order to solve such a problem, a technique for allowing a multimodal input / output system to recognize a variety of modality user inputs without the need for a high-end server-class system is under development, but the recognition technique is not yet defined.

A first object of the present invention for solving the above problems is to provide a multi-modal fusion processing method and apparatus for easily and simply performing a variety of user input modality fusion inference in a system including a multi-modality recognition engine. have.

It is a second object of the present invention to provide a multimodal fusion processing method and apparatus for allowing a plurality of modality fusions to be implemented in a miniaturized low specification system.

Multi-modal fusion device of the present invention for achieving the object of the present invention as described above, the input processing unit for receiving a single or multiple types of user input recognition information through the corresponding recognizer to process the inference information; A rule storage unit for parsing and storing modality combination rule information; And an inference engine that infers an action corresponding to the processed user input recognition information with reference to the modality combination rule information.

The rule storage unit may parse the modality combination rule information in the XML format.

The modality combination rule information in the XML form may include a first element defining one action, a second element listing modalities that may be input as a corresponding action, and the above listed for the corresponding action. A third element presenting a method for a meaningful combination of input modalities, a fourth element enumerating instructions of the modality, and a fifth element referring to one of the instructions of the modality And at least one of a sixth element (set) for combining several actions.

The third element is a child element of at least one of an element indicating that only one condition needs to be satisfied for at least one or more combination conditions according to the input modalities listed above, and an element indicating that all the conditions must be satisfied. Wherein the fourth element includes an element representing the modality instruction list as a child element, and the sixth element defines a previously inferred action order to represent each action. And at least one or more of an element including an action name as a child element and an element representing a meaningful maximum input time for each of the actions as a child element. .

The first element may include a name of a corresponding action, a state value indicating whether the corresponding action is multi or single, a subname for distinguishing actions of the same name, and a system input. It includes at least one or more of the value (send) indicating whether it can be used as an attribute, the third element includes a weight value (weight) for calculating the weighting value, the fourth element is a name At least one or more of an attribute value (name) and an input modality mode are included as attributes, and the sixth element includes at least one or more of an order value and a time value indicating an input order of the plurality of actions. Characterized in that.

The action may be a system command of the system.

The inference engine determines whether the processed user input recognition information is fusionable and fusion necessary information with reference to the modality combination rule information and the pre-generated action, and if the information is fusionable and necessary, infer the new action. It characterized in that to perform.

Preferably, the multi-modal fusion device further includes a result storage unit for storing the actions generated from the inference engine in chronological order and providing pre-stored actions as information for inference of the inference engine.

Preferably, the multi-modal fusion device verifies whether the action is valid when the action is generated from the inference engine unit, and if the action is determined to be valid, delivers the action to a system input and says that the action is not valid. If determined, characterized in that it further comprises a verification unit for generating action error information.

Preferably, the multi-modal fusion device, characterized in that it further comprises a feedback generation unit for converting the generated action error information to the information for informing the user to deliver to the system output.

The multi-modal fusion method of the present invention for achieving the above object of the present invention, parsing and storing the modality combination rule information at the initial stage of the system; If single or multiple types of user input recognition information is input from the corresponding recognizer, processing the same as action inferable information and storing the same; Inferring an action corresponding to the processed user input recognition information with reference to the modality combination rule information.

The parsing step may include parsing modality combination rule information in an XML format.

The inferring of the action may be performed by referring to the modality combination rule information and the pre-generated action, and sequentially determining whether the processed user input recognition information is fusionable and necessary information, and then convert the information into the fusionable and necessary information. If it is determined, the action is inferred.

Advantageously, the multimodal fusion method comprises: validating the inferred action; If it is determined that the inferred action is valid, passing the action to a system input and generating action error information if it is determined to be invalid; The method may further include converting the generated action error information into information for informing a user and transferring the generated action error information to a system output.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. However, in describing in detail the operating principle of the preferred embodiment of the present invention, if it is determined that the detailed description of the related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

In addition, the same reference numerals are used for parts having similar functions and functions throughout the drawings.

The present invention described below is configured in a multimodal interface device of a system including various types of modality recognition engines and infers a single or multiple types of modality inputs into system commands.

1 is a block diagram showing a multi-modal fusion device according to an embodiment of the present invention.

As shown in FIG. 1, the multimodal fusion machine 100 includes an input processor 110, an inference engine 120, a result storage unit 130, a verification unit 140, a rule storage unit 150, and feedback. It may include a generation unit 170.

In the multi-modal fusion apparatus 100 having such a configuration, the input processing unit 110 receives recognition information corresponding to a user input such as a voice or a gesture through corresponding recognizers 210 and 220, and infers this from the induction engine 120. It performs the function of processing for use.

That is, the input processing unit 110 processes the user input recognition information provided through the voice recognizer 210, the gesture recognizer 220, and the like into modality input information including an input start event, an input end event, and a result value event. Do this. Here, the input start event includes the start point of the user input, the input end event includes the end point of the user input, and the result value event includes the recognition result value of the user input.

The inference engine unit 120 determines what single or multiple user inputs such as voice and gesture input to the multi-modal fusion apparatus 100 mean and generates a system command corresponding to the determination. That is, the inference engine unit 120 converts a single or multiple user inputs determined according to the modality input start event, input end event, and result value event input from the input processing unit 110 into system commands in a single or fused state. do.

The reasoning engine unit 120 infers the system command corresponding to the current user input with reference to the modality combination rule information and the previously generated system command.

Meanwhile, in the present invention, a system command inferred by the inference engine unit 120 is called an action. Here, the action may be defined in various ways according to the system to apply the modality fusion device. For example, when the action is applied to a system that wants to control the media player, the action may be defined as STOP, PLAY, PAUSE, and the like. If applied, it may be defined as SELECT, MOVE, DELETE, or the like.

The result storage unit 130 stores the actions inferred from the inference engine unit 120 in order of time, and performs a function of providing pre-stored actions in the inference process of the inference engine unit 120.

The verification unit 140 performs a function of verifying the action inferred from the inference engine unit 120. The verification unit 140 transmits error information on the corresponding action to the feedback generator 170 when an error occurs during the verification of any action, and transfers the corresponding action to the system input when the verification ends without generating an error. do.

The feedback generator 170 defines the action error information transmitted from the verifier 140 as a method for notifying the user and delivers the action error information to the system output.

The rule storage unit 150 parses and stores the modality combination rules from the system, and provides the inference engine unit 120 with modality combination rule information for action inference.

Accordingly, the system may preset and store modality combination rules for providing to the multimodal fusion apparatus 100, and may allow the stored modality combination rules to be further changed by the user and the producer according to system characteristics. That is, the system allows the modality combination rule to be easily changed in case a new input modality is added or expansion and repair of the system occurs.

Meanwhile, the present invention allows a modality combination rule to be defined as an XML document in one embodiment, but XML according to the present invention is defined as ActionXML as it recognizes and supports all kinds of modality inputs such as voice and gesture. Also, in another embodiment, the present invention may be defined as a document in a markup language other than XML or a document in a language other than a markup language, but the present invention is described only for XML. do.

Table 1 shows the elements of ActionXML defined according to an embodiment of the present invention.

TABLE 1

Element Explanation Remarks adxml ActionXML element action An element that defines an action, the highest element ※ Child element-input, integration input An element that lists the modalities that can be entered as input to the action ※ Child element -command integration An element that suggests a meaningful combination of the input modalities listed in the input for the action. ※ Child element -or, and command an element that lists the commands of that modality as a child element of input ※ Child element -item item Modality Instruction List or An element element of the integration element, used to represent the rules of modality combinations, and an element indicating that only one of the conditions coming into the child element needs to be satisfied. ※ Child element -modality, or, and and A child element of the integration element, used to represent the rules of modality combinations, and an element indicating that it must contain all of the elements coming into the child element. ※ Child element -modality, or, and modality an element that refers to one of the commands of the input element set -An element that combines several actions according to the property of the action element. It has a sequence element as a child element. ※ Child element -sequence, time sequence A child element of the set element, which defines the order of actions coming into the user's input based on the attributes of the action element. ※ Child element -actionname time -The meaningful maximum input time of the actions listed under the set element, as a child of the set element. actionname an element that points to each action as a child element of the sequence element

As shown in Table 1, ActionXML defines and includes elements such as adxml, action, input, integation, command, item, or, and, modality, set, sequence, time, actionname, and the like. It consists of used ActionXML document.

Table 2 below shows, as an example, attributes of ActionXML elements.

TABLE 2

Element Attribute adxml version (optional): holds the version value of ActionXML-encoding (optional): holds the character encoding of the document action -name (required): the name of the action-type (optional): may have a value of 'multi', 'single', the default value is single-subname (optional): used to distinguish actions of the same name Alternate name-send (optional): Attribute indicating whether it can be used as input to the system. It has 'yes' and 'no' values. The default value is 'yes'. command name (required): The name attribute value. In the modality element, this name can refer to command. mode (mandatory): The input modality mode. has values such as 'voice', 'gesture', etc. or weight (optional): a weight value for calculating the weight value. and weight (optional): a weight value for calculating the weight value. modality weight (optional): a weight value for calculating the weighting value. c-name (mandatory). The name of the command element. sequence value (required): Takes a numeric value indicating the order. time value (required): has a threshold time value-unit: milisecond actionname a-name (mandatory): a value indicating the action-a-subname (optional): a value indicating the subname of the action

Next, the operation flow of the multi-modal fusion apparatus 100 having such a configuration will be described in detail. First, when the system is initialized, the input processing unit 110 of the multi-modal fusion apparatus 100 includes at least one recognizer 210. 220 and initializes itself by deleting the user input recognition information being stored from the pre-input, and the rule storage unit 150 parses and stores the ActionXML file.

The initialized input processing unit 110 is a modality input information including a modality start event, an input end event and a result value event when recognition information corresponding to a user input such as a voice or a gesture is input from the corresponding recognizers 210 and 220. The process outputs to the inference engine unit 120.

The reasoning engine 120 refers to the modality combination rule information of the rule storage unit 150 to sequentially determine whether the input modality input information is fusionable information and whether the fusion is necessary information. That is, the inference engine unit determines whether the single or multiple user inputs are inputs that can be inferred action according to the modality combination rule and whether the inputs are required for action inference.

In addition, when the inference engine unit 120 determines that the modality input information inputted as a result of the determination is fusionable information and necessary information, the existing action of the modality combination rule information of the rule storage unit 150 and the result storage unit 130 is determined. Infer new actions from the information. However, the inference engine unit 120 stops the operation until new modality input information is input without action inference when the input modality input information is non-fusion information or is determined to be unnecessary convergence information.

When a new action is input from the inference engine unit 120, the verification unit 140 performs the input action verification, and determines that the input action is invalid as a result of the verification, or when an error occurs during verification, action error information. The control unit outputs the to the feedback generator 170, and transmits the corresponding action to the system input when it is determined that the input action is valid.

Accordingly, the system can grasp the action delivered to the system input and provide the user with an output corresponding to the action.

Meanwhile, when action error information is input from the verification unit 140, the feedback generator 170 defines the input action error information as a method for informing the user, and then delivers the action error information to the system output, thereby causing a problem in the user input. Make sure you do it.

Next, look at the modality rule information that is set through the ActionXML document through the embodiment, and look at how the action is inferred using this.

<? xml version = "1.0" encoding = "ksc5601"?>

<adxml version = "1.0">

<action name = "PLAY" type = "single">

<input>

<command mode = "voice" name = "voice1">

<item> Run </ item>

<item> Playback </ item>

<item> Play </ item>

</ command>

<command mode = "gesture" name = "gesture1">

<item> Circle </ item>

</ command>

<command mode = "gesture" name = "gesture2">

<item> ToRight </ item>

<item> SpinRight </ item>

</ command>

</ input>

<integration>

<or>

<modality weight = "0.9" value = "voice1" />

<modality weight = "0.9" value = "gesture1" />

<and weight = "1.0">

<modality value = "voice1" />

<modality value = "gesture1" />

</ and>

<and weight = "0.8">

<modality value = "voice1" />

<modality value = "gesture2" />

</ and>

</ or>

</ integration>

</ action>

In one embodiment, the ActionXML document is input from the user according to a combination of a voice defined as “execute”, “play” or “play”, and a gesture defined as “Circle”, “ToRight” or “SpinRight”, respectively. In addition, we present a modality combination rule that allows an action called "PLAY" to be inferred.

In detail, the ActionXML document lists voice, gesture1, and gesture2 as input modalities for inferring a "PLAY" action using an input element, and the voice commands among the listed input modals are "execute", "play", and " Play "," Circle "as the gesture1 command," ToRight "and" SpinRight "as the gesture2 command are defined as item elements, and the combination of the input modalities defined above is defined through an integration element.

That is, "PLAY" is a case where voice1 and gesture1 are separately input from voice1, gesture1 and gesture2 defined as input modalities according to the ActionXML document, or when voice1 and gesture1 are input in combination, or voice1 and gesture2 are combined. If entered, it can be inferred as a system command.

Accordingly, in a system equipped with a microphone and a touch screen, a user may input a voice of "play", "play" or "play", or "circle", with a voice input of "play", "play" or "play". By performing a gesture corresponding to "ToRight" or "SpinRight", the system may be informed that the user requests execution of any image, music, or the like.

On the other hand, the ActionXML document, that is, the modality combination rule defines that the combination of voice1 and gesture1 has the highest reliability through the modality element, and the combination of voice1 and gesture2 has the lowest reliability.

The ActionXML document below illustrates modality combination rules for inferring "SELECTOBJECT", "SELECTOBJECT (FROM)", "SELECTOBJECT (TO)", "MOVE" and "MOVE (TOOBJEC)" in another embodiment, which is also described above. It is configured in the same way as the ActionXML document for inferring "PLAY".

That is, in the following ActionXML document, voice1 of "this" or "selection" and voice2 of "that" or "selection" are respectively independently input to "SELECTOBJECT", or voice1 or voice2 and gesture1 of "Pointing" When inputted in combination, it is inferred as a system command, and for "SELECTOBJECT (FROM)", it is inferred as a system command when the voice1 of "this" or "that" and the gesture1 of "Pointing" are input. For "SELECTOBJECT (TO)", the voice1 of "here", "here", "here", or "here" and the gesture1 of "Pointing" are combined to be inferred as a system command. .

In addition, the following ActionXML document is inferred as a system command when only the voice1 of "Move" or "Move" is input for "MOVE", and the voice1 of "Move" or "Move" for "MOVE (TOOBJEC)". And gesture1 of "Pointing" are inferred as system commands only when input is combined.

<action name = "SELECTOBJECT">

<input>

<command mode = "voice" name = "voice1">

<item> this </ item>

<item> Choice </ item>

</ command>

<command mode = "voice" name = "voice2">

<item> that </ item>

<item> Choice </ item>

</ command>

<command mode = "gesture" name = "gesture1">

<item> Pointing </ item>

</ command>

</ input>

<integration>

<or>

<modality value = "gesture1" />

<and>

<modality value = "voice1" />

<modality value = "gesture1" />

</ and>

<and>

<modality value = "voice2" />

<modality value = "gesture1" />

</ and>

</ or>

</ integration>

</ action>

<action name = "SELECTOBJECT" subname = "FROM">

<input>

<command mode = "voice" name = "voice1">

<item> this </ item>

<item> that </ item>

</ command>

<command mode = "gesture" name = "gesture1">

<item> Pointing </ item>

</ command>

</ input>

<integration>

<or>

<and>

<modality value = "voice1" />

<modality value = "gesture1" />

</ and>

</ or>

</ integration>

</ action>

<action name = "SELECTOBJECT" subname = "TO">

<input>

<command mode = "voice" name = "voice1">

<item> here </ item>

<item> To here </ item>

<item> Over there </ item>

<item> Over there </ item>

</ command>

<command mode = "gesture" name = "gesture1">

<item> Pointing </ item>

</ command>

</ input>

<integration>

<or>

<and>

<modality value = "voice1" />

<modality value = "gesture1" />

</ and>

</ or>

</ integration>

</ action>

<action name = "MOVE" send = "no">

<input>

<command mode = "voice" name = "voice1">

<item> Go </ item>

<item> Move </ item>

</ command>

</ input>

<integration>

<or>

<modality value = "voice1" />

</ or>

</ integration>

</ action>

<action name = "MOVE" subname = "TOOBJECT" send = "no">

<input>

<command mode = "voice" name = "voice1">

<item> Go </ item>

<item> Move </ item>

</ command>

<command mode = "gesture" name = "gesture1">

<item> Pointing </ item>

</ command>

</ input>

<integration>

<or>

<and>

<modality value = "voice1" />

<modality value = "gesture1" />

</ and>

</ or>

</ integration>

</ action>

Meanwhile, the following ActionXML document is a modality combination rule for inferring an action of another embodiment, and a new action "MOVE2OBJECT" is inferred by referring to the actions of the above-described ActionXML documents.

That is, the following ActionXML document is an example showing a modality combination rule for an action that can be inferred when at least one action is continuously input.

In detail, the following ActionXML document defines the action element attribute as "multi", and then uses the set element and the sequence element to perform the "SELECTOBJECT (FROM)", "SELECTOBJECT (TO)", and "MOVE" actions. The "MOVE2OBJECT" action is inferred when the corresponding user modalities are sequentially input or when the user modalities corresponding to the "SELECTOBJECT (FROM)" and "MOVE (TOOBJEC)" actions are sequentially input.

Accordingly, the inference engine unit 120 of the multi-modal fusion apparatus 100 refers to the action information corresponding to the current user input by referring to the existing action information of the result storage unit 130 together with the modality combination rule information of the rule storage unit 150. To infer.

Here, the inference engine unit of the multi-modal fusion apparatus 100 may infer an action corresponding to the current user input with reference to only an action generated within a preset time.

<action name = "MOVE2OBJECT" type = "multi">

<integration>

<set>

<sequence value = "1" option = "one-of">

<actionname name = "SELECTOBJECT" subname = "FROM" />

</ sequence>

<sequence value = "2">

<actionname name = "SELECTOBJECT" subname = "TO" />

</ sequence>

<sequence value = "3">

<actionname name = "MOVE" />

</ sequence>

<time value = "7000" />

</ set>

<set>

<sequence value = "1">

<actionname name = "SELECTOBJECT" subname = "FROM" />

</ sequence>

<sequence value = "2">

<actionname name = "MOVE" subname = "TOOBJECT" />

</ sequence>

<time value = "5000" />

</ set>

</ integration>

</ action>

</ adxml>

As described through the above embodiments, the multimodal fusion apparatus 100 according to the present invention parses and stores the modality combination rules of an ActionXML document from the system to infer at least one or more actions, and then stores the stored modality combination rule information and By referring to the generated action information, it can be seen that the action is inferred according to single or multiple user inputs.

Next, the modality fusion method of the multi-modal fusion device 100 for a single or multiple user inputs will be described with reference to the accompanying drawings.

2 is a flow chart illustrating a modality fusion method of the multi-modal fusion device 100 according to an embodiment of the present invention.

Referring to FIG. 2, when the initial power is applied to the system or when the operating system is rebooted, the multi-modal fuser 100 deletes user input and stored user modality input information and modality rule information configured as an ActionXML document. Pars and store it (S101).

The multi-modal fusion apparatus 100 checks whether user input recognition information such as voice or gesture is input through a single or multiple kinds of recognizers (S102).

When the multimodal fusion apparatus 100 confirms that the user input recognition information is input from the corresponding recognizer, the multimodal fusion apparatus 100 processes and stores the modality input information that can be inferred as action inference (S103). The parsed modality combination rule information and the pre-generated action information Reference is made to determine whether the corresponding modality input information is information that can be converged and whether information is required for convergence (S104, S105).

If the multimodal fusion apparatus 100 determines that the modality input information is fusionable and necessary fusion information, the multi-modal fusion apparatus 100 infers the action corresponding to the user input with reference to the modality combination rule information and the pre-generated action information (S106). ).

However, if the modality input information is non-fused information or unnecessary convergence information, the multi-modal fusion apparatus 100 stops inferring action on the modality input information, and new user input recognition information is received from a single or multiple types of recognizers. Check whether it is input (S102).

On the other hand, the multi-modal fusion apparatus 100 inferring the action for any modality input information, verify the validity of the inferred action once again (S107), and if appropriate, input the inferred action system input Transfer to (S108).

However, the multi-modal fusion apparatus 100 defines a method for informing the user that a user input corresponding to the action is an invalid input when it is determined that an inferred action is not valid or an error occurs during verification. Transfer to the system output, so that the user input is performed again from the user who confirmed this (S109).

The present invention described above is not limited to the above-described embodiments and the accompanying drawings, and it is common in the art that various substitutions, modifications, and changes can be made without departing from the technical spirit of the present invention. It will be apparent to those skilled in the art.

In particular, although the above-described present invention describes modality combination rules according to fusion of voice and gesture input as an embodiment, the modality combination rules may be applied to various modality inputs in another embodiment.

The multi-modal fusion processing method and apparatus according to the present invention as described above correspond to a single or multiple user inputs through ActionXML modality combination rule information defined to recognize and support all kinds of modality inputs such as voice and gestures. By inferring the system instructions to be inferred, the convergence of the various modalities can be easily and simply performed even in a compact low-end system.

In addition, the multimodal fusion processing method and apparatus according to the present invention simplify the maintenance of the system by allowing a user or a producer to modify only an ActionXML document including modality combination rules in order to add a new modality or add a command. In addition, it is easy to add a new input modality, and the expansion and development of the system can be easily performed.

Claims (16)

A multimodal fusion device of a system having a single or multiple modality recognizers, An input processing unit which receives single or multiple types of user input recognition information through corresponding recognizers and processes the information into inferable information;  A rule storage unit for parsing and storing modality combination rule information; And And an inference engine that infers an action corresponding to the processed user input recognition information with reference to the modality combination rule information. The method of claim 1, The rule storage unit, Multimodal fusion device characterized in that for parsing the modality combination rule information in the XML form. The method of claim 2, The modality combination rule information of the XML form is A first element defining an action, a second element that lists the modalities that can be input to the action, and a method for a meaningful combination of the above listed input modalities for the action. A third element presenting, a fourth element enumerating instructions of a corresponding modality, a fifth element referring to one of the instructions of the modality, and a combination of several actions; 6. A multimodal fusion device comprising at least one of six elements (set). The method of claim 3, The third element is a child element of at least one of an element indicating that only one condition needs to be satisfied for at least one or more combination conditions according to the input modalities listed above, and an element indicating that all the conditions must be satisfied. Including, The fourth element includes an element representing the modality instruction list as a child element. The sixth element defines a sequence of previously inferred actions, an element including an element representing an action of each action as a child element, and a meaningful maximum input time for each action. Multimodal fusion device, characterized in that it comprises at least one or more of the elements (time) representing as a child element. The method of claim 3, The first element may include a name of a corresponding action, a state value indicating whether the corresponding action is multi or single, a subname for distinguishing actions of the same name, and a system input. Contains at least one of the values indicating whether it can be used as an attribute, The third element includes a weight value for calculating a weighting value as an attribute, The fourth element includes at least one of a name attribute value name and an input modality mode as an attribute. And the sixth element includes at least one or more of an order value and a time value representing an input order of the plurality of actions as attributes. The method of claim 1, The action is, And a system command of the system. The method of claim 1, The inference engine unit, With reference to the modality combining rule information and the pre-generated action, it is determined whether the processed user input recognition information is fusionable and fusion necessary information, and if the information is fusionable and necessary, the new action inference is performed. Multimodal fusion device. The method of claim 1, And a result storage unit for storing the actions generated from the inference engine in chronological order and providing pre-stored actions as information for inference of the inference engine. The method of claim 1, When the action is generated from the inference engine unit, verify whether the action is valid, and if it is determined that the action is valid, deliver the action to the system input, and if it is determined that the action is not valid to generate action error information Multimodal fusion device characterized in that it further comprises a verification unit. The method of claim 9, And a feedback generation unit converting the generated action error information into information for informing a user and delivering the result to a system output. In the multi-modal fusion method of a system having a single or multiple modality recognizer, Parsing and storing modality combination rule information at system initial stage; If single or multiple types of user input recognition information is input from the corresponding recognizer, processing the same as action inferable information and storing the same; And And inferring an action corresponding to the processed user input recognition information with reference to the modality combination rule information. The method of claim 11, The parsing step, A multimodal fusion method characterized by parsing modality combination rule information in an XML form. The method of claim 12, The modality combination rule information of the XML form is A first element defining an action, a second element that lists the modalities that can be input to the action, and a method for a meaningful combination of the above listed input modalities for the action. A third element to present, a fourth element that lists instructions of the modality, a fifth element to combine one of the instructions of the modality, and multiple actions A multimodal fusion method comprising at least one of six elements (set). The method of claim 11, The action is, And a system instruction of the system. The method of claim 11, Inferring the action, With reference to the modality combination rule information and the pre-generated action, sequentially determine whether the processed user input recognition information is fusionable and fusion necessary information, and infer the action when it is determined to be the fusionable and necessary information. Multi-modal fusion method characterized in that. The method of claim 10, Verifying whether the inferred action is valid; If it is determined that the inferred action is valid, passing the action to a system input and generating action error information if it is determined to be invalid; And And converting the generated action error information into information for informing a user and transferring the generated action error information to a system output.

Images