KR100758789B1 - Multi-modal system - Google Patents

Abstract

A multi-modal system is provided to process a plurality of modalities in parallel at the same time by using SCXML(State Chart eXtensible Markup Language), and offer a user and an application maker a convenient/efficient user interface by easily processing a sequential or concurrent multi-modal I/O(Input/Output) command with the SCXML. A markup language interpreter(240) interprets an SCXML document. A markup language execution engine(250) concurrently processes the plurality of different modality information by activating a state corresponding to the number of modalities used by a client in parallel based on the interpreted SCXML document. The different modalities are voice and non-voice modality. The markup language execution engine receives voice and non-voice modality information from the client in parallel, and fuses the inputted modality information with other modality input information if the inputted modality information is the partial information.

Description

Multi-modal System

1 is a diagram illustrating an outline of an operation scenario for multi-modality parallel processing according to an embodiment of the present invention in a Harrel State Chart;

FIG. 2 is a diagram illustrating a specific operational scenario for multi-modality parallel processing according to an embodiment of the present invention in a Harrel State Chart. FIG.

3 is an SCXML document for the statechart scenario of FIG.

4 is a configuration diagram of one embodiment of a multi-modal system according to the present invention;

5 illustrates an embodiment of a session structure in accordance with the present invention;

6 is a detailed configuration diagram of an embodiment of a client according to the present invention;

7 is a detailed configuration diagram of an embodiment of a multi-modal server according to the present invention

8 is a detailed configuration diagram of one embodiment of a multi-modal server according to the present invention;

9 is a flowchart illustrating a process of parallel processing multi-modality input information according to the present invention.

Explanation of symbols on the main parts of the drawings

100: Client 200: Multimodal Server

110: voice modality processing unit 130: communication unit

120: non-voice modality processing unit 111: microphone

112: Speaker 113: VXML Browser

121: key input unit 122: display unit

123: HTML browser

The present invention relates to a multimodal system, and more particularly, to a multimodal system that processes voice or non-voice modality input information in parallel at the same time.

Multimodal means multiple modalities. Modality means each sensory channel such as visual, auditory, tactile, taste, and olfactory, and it is called multimodal interaction.

Multimodal systems allow users to send information such as voice, data, video, audio or other information, and email, weather updates, banking and news or other information to one or more modalities through a user agent program such as a graphical browser or voice browser. It allows access to and receive information in different modalities. In particular, after a user submits an information fetch request with one or more modalities, such as speaking a fetched request to the microphone, the user fetches it, such as through a graphical browser that provides the returned information as visual information on the display screen. Information can be received with the same modality (ie, voice) or different modality (ie, graphics). Within a communication device, the user agent program operates in a similar manner to a web browser or other suitable software program residing on a device connected to a network or other terminal device.

Since a multimodal system provides an interface for multiple modalities, each modality must be synchronized with each other to switch between modalities. For example, the use of non-voice modality can be changed to voice modality, and vice versa, and it must be possible to use it at the same time.

Currently, various multimodal systems have been proposed that allow more than two user input modalities. One suggestion using the markup language is that when a user program and a Hyper Text Markup Language (HTML) browser are running on the client, the user can enter text information using the markup language of the page after inputting voice. There is an X + V type multimodal system. X + V is a form suggested by IBM, etc. It is a structure in which HTML browser, which is a graphic browser, and Voice Extensible Markup Language (VXML), which is a voice browser, are embedded in a multimodal server.

However, the proposal is easy to integrate and process input / output synchronization and sequential or simultaneous input modality, while multi-modal input / output is processed under the same program. There is a problem that it is difficult to meet the requirements of the individual modality for distributed processing because it must be modified for the modal system.

On the other hand, as another proposal, for the implementation of concurrent modality, the markup-based that the modality for the user program includes VXML or HTML and acquires a specific command based on the simultaneous modality tag to integrate multi-modal input information or to synchronize the input and output signals There is a multi-modal system of (see Korean Patent Publication No. 10-2004-0089677). However, the proposal has a problem in that separate hardware or software / firmware configurations, such as a simultaneous multimodal synchronization regulator and a multimodal fusion engine, are required to synchronize a plurality of modalities.

Accordingly, there is a need for an implementation of an improved multimodal system that can efficiently handle two or more modalities that are sequentially input or output simultaneously.

SUMMARY OF THE INVENTION The present invention has been proposed to meet the above demands, and an object thereof is to provide a multimodal system capable of simultaneously processing a plurality of modalities in parallel by using a statechart markup language, and a multimodality input processing method thereof.

Other objects and advantages of the present invention can be understood by the following description, and will be more clearly understood by the embodiments of the present invention. Also, it will be readily appreciated that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the claims.

The present invention for achieving the above object is a multi-modal system for processing different modality information in parallel by connecting to a network and a client using different modality information, to interpret a predetermined state chart markup language document Markup language interpreter for; And a markup language execution unit for simultaneously processing different modality information by activating in parallel a state corresponding to the number of modalities used by the client according to the interpreted state chart markup language.

On the other hand, the present invention is a multi-modal input information processing method for processing different modality information in parallel at the same time, comprising: a user session creation step of generating a user session according to the connected client; A parallel state step of generating a modality component session corresponding to the number of modalities used by the client as a sub-session of the user session based on a state chart markup language and activating in parallel; And a receiving step in which each of the modality component sessions activated in the parallel state step receives different modality input information in parallel.

The following merely illustrates the principles of the invention. Therefore, those skilled in the art, although not explicitly described or illustrated herein, can embody the principles of the present invention and invent various devices that fall within the spirit and scope of the present invention. Furthermore, all conditional terms and embodiments listed herein are in principle clearly intended for the purpose of understanding the concept of the invention and are not to be limited to the specifically listed embodiments and states. Should be. In addition, it is to be understood that all detailed descriptions, including the principles, aspects, and embodiments of the present invention, as well as listing specific embodiments, are intended to include structural and functional equivalents of these matters. In addition, these equivalents should be understood to include not only equivalents now known, but also equivalents to be developed in the future, that is, all devices invented to perform the same function regardless of structure.

Thus, the functionality of the various elements shown in the figures, including functional blocks represented by a processor or similar concept, can be provided by the use of dedicated hardware as well as hardware capable of executing software in conjunction with appropriate software. When provided by a processor, the functionality may be provided by a single dedicated processor, by a single shared processor or by a plurality of individual processors, some of which may be shared. In addition, the use of terms presented in terms of processor, control, or similar concept should not be interpreted exclusively as a citation of hardware capable of executing software, and without limitation, ROM for storing digital signal processor (DSP) hardware, software. (ROM), RAM, and non-volatile memory are to be understood to implicitly include. Other hardware for the governor may also be included.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. In describing the present invention, when it is determined that the detailed description of the related known technology may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a diagram illustrating an outline of an operation scenario for multi-modality parallel processing according to an embodiment of the present invention in a Harrel State Chart.

According to the present invention, a method of describing an operation scenario for multi-modality parallel processing such as voice modality and non-voice modality according to a scenario in the form of a harle state chart is described in XML. The scenario description scheme in the form of a state chart has an intuitive / visual and event-critical process structure.

As shown in FIG. 1, the statechart includes a super state 1, states 2 and 9, parallel states 6 and 7, and transition 8. , Action (4) and (5).

Statecharts created according to any operating scenario are described in a statechart XML (SCXML) document. In other words, statechart XML refers to a document that maps statechart scenarios to XML style.

The multi-modal system according to the present invention creates an operation scenario for parallel processing of multi-modality using a statechart, and maps it to a machine-readable XML document to use a plurality of voice or non-speech input or output sequentially or simultaneously. Process all Daranti in parallel in the same way as the statechart scenario.

FIG. 2 is a diagram illustrating a specific operational scenario for multi-modality parallel processing according to an embodiment of the present invention in a Haral State Chart, and FIG. 3 is an SCXML document for the state chart scenario in FIG.

First, each state of this embodiment is summarized according to the reference numerals as follows.

101: initial state of idle state, 110: idle state, 120: system_on super state, 130: super parallel state part, 140: lower parallel non-voice modality state, 170: lower parallel speech aural state, 141 : Initial state of non-voice modality (visual), 150: HTML reception state, 160: HTML input reception state, 180: VXML reception state, 190: VXML input reception state, 195: integration state, 199: end state, 111: turn_on event transition, 121: turn_off event transition, 141: initial state of visual, 151: html_on event occurrence and conditional statement transition from state 150 to state 160 when html_value is partial information, 152: html_on event occurrence and conditional statement html_value is complete Transition from state 150 to state 195 for information, 161: timeout when timer> = waiting_time and transition from state 160 to state 150, 162: increase timer 1 by conditional statement time <waiting_time, 163: increase from state 160 to 195 Transformation, 171: Seconds of aural State, 181: vxml_on event occurrence and condition statement transition from state 180 to state 190 when vxml_value is partial information, 182: vxml_on event occurrence and condition statement transition from state 180 to state 195 when vxml_value is complete information, 191: conditional statement timer> = Timeout at waiting_time to transition from state 190 to state 180, 192: conditional timer <increment timer 1 at waiting_time, 193: transition from state 190 to state 195.

As shown in FIG. 2, in accordance with the scenario of the present embodiment, two voice and visual modality components are activated by a login event 111 generated by a user in a client in an initial state 101. XML state charts are mapped to two parallel states. State 130 is started in state 120 indicating that it is in parallel and its substate 140 and state 170 are activated at the same time.

State 140 includes its substates 150 and 160, and state 170 also includes its substates 180 and 190.

Accordingly, when the event 111 occurs, the process automatically enters the states 150 and 180 and waits for a user input. States 160 and 190 are temporary active states that will remain in their current location for a limited time. State 195 synchronizes or fuses the value to the last state 199 after the parallel state ends.

According to this multi-modality parallel processing operation scenario, the present invention provides a multi-modal synchronization coordinator when a multi-modal application is started after a client connects to and connects to a multi-modal server and requests the user information in both voice mode and non-voice mode. User multi-modality input is fused and processed by an SCXML execution engine that executes a markup language based on the SCXML document of FIG. 3 without a separate machine such as a fusion engine.

For example, if a user clicks on a point on the map and says the words "Zoom" or "Zoom", the state of the statechart transitions according to the partial input information received, filling the fields with different modalities and fusing the input information. do. Also, voice signals such as "first time" or "reset" that do not require fusion in the individual modalities are already completed information, in which case the values of the other modalities are synchronized.

4 is a block diagram of an embodiment of a multi-modal system according to the present invention.

As shown in FIG. 4, in the multimodal system, a plurality of clients 100 are connected to the multimodal server 200 through a wired or wireless network 300.

The client 100 is a communication terminal device capable of driving application software capable of multimodal interaction. For example, a mobile or fixed terminal device such as a portable digital assistant (PDA), a mobile phone, or a personal computer (PC) may be used. Can be applied.

The client 100 receives a voice or non-voice modality signal from a user through an HTML browser and a VXML browser, and converts input content (HTML data, VXML data) and event information corresponding to the input signal into EMMA data to multiply. It transmits to the modal server 200, and outputs the output content as voice or non-voice modality according to the information (URL, event information) returned from the multi-modal server 200. Here, the EMMA data further includes meta information according to the EMMA standard together with the input content and event information.

EMMA (Extensible Multimodal Annotation Markup Language) is a standard language proposed by the W3C in 2004 to express input results of various modalities. It is a standard language that connects input elements and interaction managers. It is a markup language that expresses processing results by attaching semantic tags when a user uses various input means such as voice and key input. For example, various meta information such as input time, recognition result, semantic tag of the result, and reliability can be expressed. In the present invention, the result obtained through the EMMA interpreter is converted into a context usable in the statechart XML execution engine and used for markup control.

The multimodal server 200 is connected to the client 100 through the network 300 to process sequential or simultaneous multimodal input / output interactions with the client 100, thereby receiving EMMA data (input content, event information, URL and event information corresponding to the additional information) are transmitted to the client.

Referring to the multi-modal input and output interaction according to the present invention.

The multi-modal server 200 receives a predetermined operation scenario for multi-modality parallel processing described in statechart XML and activates N user sessions such as the number of clients 100, and the client as a sub-session of each user session. It handles multimodal I / O interactions by activating and parallelizing statechart sessions and modality component sessions (HTML sessions, VXML sessions) in response to the supporting modalities.

Here, the multimodal server processes the input content received from the client by interworking with an external component such as an external web server or a service bundle through such multimodal input / output interaction, and transmits corresponding output content and event information to the client. do.

5 is a diagram illustrating an embodiment of a session structure according to the present invention.

As shown in Fig. 5, a session according to the present invention consists of a user session, a statechart session (SC session) and a modality component session (HTML session, VXML session).

User sessions are maintained throughout the interaction with the client, while SC sessions and modality component sessions are created and maintained during the user session.

Lower sessions lie within the time axis of the upper session. That is, the modality component session may be shorter than the SC session, and each SC session and the modality component session have different durations.

For example, in the scenario where voice modality and non-voice modality cannot be used at the same time, if a voice component input is added while the user is interacting with a graphical user interface (hereinafter referred to as a GUI) through an HTML browser, the HTML session is voiced. Pause while I / O continues. Then, after the HTML session is added, the VXML session is processed, and then the interaction is completed by the HTML session.

In this case, there is one VXML session and two separate HTML sessions. The first HTML session starts and ends before the VXML session begins.

The duration of the VXML session in turn overlaps with the second HTML session and ends before that. The SC session will include all three modality component sessions (two HTML sessions and one VXML session).

On the other hand, in the scenario of using voice modality and non-voice modality simultaneously, HTML sessions and VXML sessions are processed sequentially or in parallel.

Once a user logs in, a user session is created, loaded and loaded into memory, creating an SC session underneath the user session. next,

Initialization of the user session is performed by a multi-modal server.

When a user application such as an HTML browser and a VXML browser is started, the multimodal server generates a user session through user authentication, reads an SCXML document stored in the multimodal server, creates and initializes an SC session, and initializes the SCXML. Depending on the scenario, create and initialize a modality component session for driving the HTML and VXML modality components.

In addition, the multimodal server sends a performance message as URL or event information to the client so that each component session is started upon initialization.

Each modality component session (HTML session, VXML session) terminates when the client notifies the end point in the markup language (eg <exit> tag), or a Halt event occurs according to the SCXML scenario.

As such, the user session generates a life cycle event necessary for the operation of the SC session, and the SC session performs the asynchronous communication with the client. When the parent session ends, all child sessions are automatically terminated.

6 is a detailed configuration diagram of an embodiment of a client according to the present invention.

As shown in FIG. 6, the client 100 may include a voice modality processor 110 for processing voice modality information, a non-voice modality processor 120 for processing non-voice modality information such as graphic modality, and various data. It includes a communication unit 130 for transmitting and receiving through a network.

The voice modality processor 110 includes a microphone 111 for receiving a voice signal, a speaker 112 for outputting a voice signal, and a VXML browser 113 which is a voice browser.

When the input time of the voice signal input through the microphone 111 continues above a predetermined threshold, the VXML browser 113 considers the voice input signal, extracts the start point and the end point of the voice input, and inputs the input voice signal data. Recognize and generate text data, convert it to EMMA data format and transmit EMMA data of the input voice signal to the multi-modal server 200 through the communication unit 130, and outputs the voice modality returned from the multi-modal server The audio output signal is synthesized according to the content and output through the speaker 112. Here, the general functions of the VXML browser related to interworking with an external web server are well known to those skilled in the art, and thus, more detailed description thereof will be omitted.

The detailed configuration of the VXML browser will be described later with reference to FIG. 7.

The non-voice modality processing unit 120 includes a key input unit 121 for receiving a non-voice modality signal from a user, a display unit 122 for outputting a graphic modality signal, and an HTML browser 123 as a graphic browser. .

The HTML browser 123 converts the input content corresponding to the user input signal received from the key input unit 121 into the EMMA data format and transmits it through the data transmission unit 130, and outputs the non-voice modality returned from the multi-modal server. The output signal corresponding to the content is displayed as visual information to the user through the display 122.

Since the HTML browser 123 is well known to those skilled in the art, detailed description thereof will be omitted. The key input unit 121 may be a keypad, a keyboard or a touch screen.

7 is a detailed block diagram of an embodiment of a VXML browser according to the present invention.

As shown in FIG. 7, the VXML browser includes a voice feature extractor 131, a VXML interpreter 132, a recognizer 133, and a synthesizer 134.

The voice feature extracting unit 131 considers the voice input signal when the input time of the voice signal input through the microphone 111 is longer than a predetermined threshold value, and extracts a start point and an end point of the voice input and input the voice signal data. Is obtained and output to the VXML interpreter 132. Here, the process up to the processing by the voice feature extraction unit 112 is called a voice preprocessing process.

The recognizer 133 recognizes the input voice signal and outputs text data, and the synthesizer 134 synthesizes the input text data into a voice signal and outputs the text data.

Here, the recognizer 133 may apply a general Automatic Speech Recognizer (ASR), and the synthesizer may also apply a general Text to Speech (TTS) engine.

The VXML interpreter 132 outputs a voice signal input from the voice feature extractor 131 to the recognizer 133 to instruct voice recognition, transforms the recognized text data into an EMMA data format, and transmits the received voice signal to the multimodal server 200. In addition, the synthesizer 134 outputs synthesis target text information to be synthesized according to the output content of the voice modality received from the multi-modal server 200 and instructs voice synthesis.

The synthesizer 134 receives synthesis target text information from the VXML interpreter 132, performs speech synthesis according to the text information, generates a speech synthesis signal, and outputs the synthesized speech signal through the speaker 112.

In this embodiment, the VXML browser 113 is configured to be included on the client 100 side, but only the voice preprocessing process is performed on the client 100 side, and the voice postprocessing process is performed on the server side. It may also be implemented as a distributed speech recognition structure included on the server side.

That is, according to the distributed processing policy of the server and the client, the start point / end point extraction and feature extraction occur in the voice preprocessing process of the client side, and the extracted voice information is transmitted to the VXML browser of the multimodal server and the VXML browser included in the server side ( 210 may be implemented as a structure in charge of voice post-processing.

8 is a detailed block diagram of an embodiment of a multi-modal server according to the present invention.

As shown in FIG. 8, the multi-modal server 200 may include a communication unit 210, an EMMA analyzer 220, a session manager 230, an SCXML interpreter 240, an SCXML execution engine 250, and a profile DB ( 250).

The communication unit 210 transmits and receives various data with the client through the network.

The EMMA interpreter 220 interprets the EMMA data received from the client, extracts input content, event information, and meta information included in the EMMA data, and outputs the extracted content to the session manager 230 and the SCXML execution engine 250.

The session manager 230 generates, maintains, activates, and terminates the user session, the SC session, and the modality component sessions, and transmits the output content through the activated session. That is, the session manager 230 creates and manages a session by giving a session ID every time a new user accesses the session. More specifically, the session manager 230 generates the sessions as described above with reference to FIG. 5 according to the information received from the EMMA interpreter 220 and the interaction according to the SCXML scenario through the SCXML execution engine 250. Handles maintenance, activation and shutdown processes.

The SCXML interpreter 240 receives and compiles the SCXML document, generates a Java object model corresponding to the session activated by the session manager 230, and outputs the generated Java object model to the SCXML execution engine. The SCXML parser according to the present embodiment basically operates similarly to a general XML markup parser.

The SCXML execution engine 250 executes a multi-modality parallel processing process according to the SCXML document using the Java object model generated by the SCXML interpreter 240. That is, the SCXML execution engine executes the script scenario when the SCXML markup script is converted into the Java object model from the SCXML interpreter 240 and inputted in a manner of transitioning states based on an event like a general XML execution engine, thereby executing the script scenario. Process the activated modality component sessions sequentially via 230 at the same time or in parallel.

The SCXML execution engine 250 uses the event received from the client and the profile information stored in the profile DB 260 to transition the state of the Java object model or directly send the corresponding output content through the communication unit, or request The signal is generated and transmitted to a web server or an external bundle (OSGi module), and the output content is generated according to the returned data and transmitted to the client through the communication unit.

The profile DB 260 is a database that stores and manages profile information.

The profile information is also called a delivery context component (DCI) and includes device environment information and user preference information. The device environment information is dynamically changed in the mobile environment such as GPS reception, battery information, and network information, and the user preference information is the view preference, the specific modality preference, the N-best selection, the intelligence level selection, and the simultaneous input. Information that can be selected differently for each client, such as timeout selection.

According to the present embodiment, when the SCXML interpreter 240 of the multi-modal server 200 operates to compile the SCXML document, generate a Java object model, and drive the SCXML execution engine, as described in the scenario according to the SCXML document. Each modality component, such as HTML or VXML browser, is instructed to parse the URL described in the SCXML document, and the client accordingly activates the HTML or VXML browser in a ready state capable of inputting and outputting to the user.

More specifically, the input content and event information of the voice modality is converted into EMMA data through the VXML browser and transmitted to the multimodal server, and the non-voice input content and event information such as pen or key input is transmitted through the HTML browser. Once converted to data and sent directly to the multimodal server, the multimodal server parallelizes the multimodality input content according to the multimodality parallelization scenario of the script in the SCXML document, and runs the client's VXML browser or HTML browser to display the output content. send. The multi-modality parallel processing will be described later with reference to FIG. 9.

9 is a flowchart illustrating a process of parallel processing multi-modality input information according to the present invention.

First, when a client connects, a user session is generated (910), a SCXML script is compiled to generate a Java object model corresponding to the user session to generate an SC session (920), and a modality component session supported by the client. Create an HTML session and a VXML session (930).

Subsequently, by activating the HTML session and the VXML session in parallel, the client's HTML browser and the VXML browser are activated to proceed to receive user input, respectively (950 and 960).

When the input information of each modality is received in parallel in step 950 or 960, it is determined whether the received input information is partial information (951, 961).

Here, the partial information does not have complete meaning by itself and refers to information that can be processed only when integrated with the partial information of the other modality. Complete information is information that is complete in its own right and can be processed without any other modality information.

For example, if a user clicks a point on the map displayed through an HTML browser and enters click information through the HTML browser, while entering voice information such as "zoom in" or "zoom out" through the VXML browser, the HTML And the click information and the voice information input from the VXML browser become partial information.

As a result of the determination (951, 961), if the received input information is complete information, the parallel state is terminated and the received input information is processed separately (960).

As a result of the determination (951, 961), if the received input information is partial information, it proceeds to the other modality input reception state (952, 962), and determines whether other modality input information has been received within a predetermined timeout time. (953, 963),

When the partial information of the different modalities is received, the parallel state is terminated and the received partial information of the different modalities are converged and processed (970 and 980).

Similarly, the multimodal output is initialized by the same number of modalities and processed through the same algorithm as that of the flow, so that each modality can be independently output through a speech synthesizer or a display device. Will be omitted.

In the above embodiment, it is assumed that an input device having two modalities, that is, voice modality and non-voice modality through a keypad, is only one example. In the present invention, a voice input and a keyboard are used as input devices for a multi-modal system. Obviously, various modality input devices such as input, mouse input, pen input, touch screen input, gesture input, and eye movement input can be applied.

As described above, the method of the present invention may be implemented as a program and stored in a recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto-optical disk, etc.) in a computer-readable form. Since this process can be easily implemented by those skilled in the art will not be described in more detail.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present invention for those skilled in the art to which the present invention pertains. It is not limited by the drawings.

As described above, the present invention has the effect of simultaneously processing voice or non-voice modality input / output information in parallel using SCXML markup language.

In addition, the present invention can easily process a sequential or simultaneous multi-modal input / output command using a markup language, there is an effect that can provide a more convenient and efficient user interface to users and application creators.

In addition, the present invention introduces and uses the SCXML language similar to the scenario description language of the client's VXML or HTML, so that independence and autonomy under the same structure without changing the function or operation range of the existing client modality from the service operator's point of view. It has the effect of guaranteeing the operation of a multimodal system.

Claims (11)

A multimodal system for processing different modality information in parallel by connecting a network with a client using different modality information, A markup language interpreter for interpreting a predetermined state chart markup language document; A markup language execution unit for simultaneously processing different modality information by activating in parallel a state corresponding to the number of modalities used by a client according to the interpreted state chart markup language; Multi-modal system comprising a. The method of claim 1, The markup language is a multi-modal system, characterized in that the state chart XML (SCXML) for a state chart that supports parallel state processing. The method of claim 1, Wherein said different modalities are voice modality and non-voice modality. The method of claim 3, wherein The markup language execution unit, And receiving voice modality information and non-voice modality information from the client in parallel, and processing the converged with other modality input information when the input modality information is partial information. The method of claim 4, wherein The modality information is received from a client in the form of EMMA data. The method of claim 1, A session manager for generating and managing a session corresponding to the connected client according to the input information of the connected client and the control of the markup language execution unit. Multimodal system further comprising. The method of claim 6, The session, A user session corresponding to each of the connected clients; A statechart session created according to the markup language as a sub-session of the user session; A subsession of the statechart session, the modality component session corresponding to the modality component used by the connected client. Multimodal system comprising a. The method of claim 7, wherein The modality component session, Multimodal system, characterized in that it is a voice modality session and a non-voice modality session. A multi-modal input information processing method for simultaneously processing different modality information in parallel, Generating a user session according to a connected client; A parallel state step of generating a modality component session corresponding to the number of modalities used by the client as a sub-session of the user session based on a state chart markup language and activating in parallel; And A receiving step in which each of the modality component sessions activated in the parallel state step receive different modality input information in parallel Multi-modal input information processing method comprising a. The method of claim 9, Determining whether the input information received in the receiving step is partial information; And In the determination result, when the input information is partial information, a fusion processing step of fusion processing with input information of another modality Multimodal input information processing method further comprising. The method of claim 10, In the determination result, if the input information is complete information, the individual processing step of processing separately regardless of the input information of other modalities Multimodal input information processing method further comprising.

Images