JP2005519363A - Simultaneous multimodal communication system and method - Google Patents

Abstract

The multimodal network element (14) facilitates simultaneous multimodal communication sessions through different user agent programs (30, 34) on one or more devices (12, 16). For example, a user agent program that communicates in a voice mode, such as the voice browser (34) of the voice gateway (16) including the speech engine and call / session termination, may be a graphical browser (30) on the mobile device (12). Synchronize with other user agent programs running in different modes. A plurality of user agent programs (30, 34) are coupled to the content server (18) and operate at the time of a session to perform simultaneous multimodal interaction.

Description

  The present invention relates generally to communication systems and methods, and more particularly to multimodal communication systems and methods.

Multimodal interaction to access information and services in new technology areas involving communication devices such as handheld devices, mobile phones, laptop computers, PDAs, Internet appliances, non-mobile devices, and other suitable devices There are application fields. Typically present in the communication device is a user agent program such as at least one such as a browser or other suitable software capable of operating as a user interface. The user agent program responds to fetch requests (entered by the user through the user agent program or from other devices or software applications), receives the fetched information, and uses an internal or external connection. The user can navigate through the content server and present information to the user. The user agent program can be a graphical browser, a voice browser, or any other suitable user agent program understood by those skilled in the art. Such user agent programs include J2ME application, Netscape (trademark), Internet Explorer (trademark), Java application, WAP browser, Instant
This includes, but is not limited to, Messaging, Multimedia Interface, Windows CE ™, or other suitable software implementation.

  By using multimodal technology, the user can in one mode via user agent program, information such as voice, data, video, audio or other information, e-mail, weather information, banking And access services such as news or other information and receive information in different modes. More specifically, the user submits an information fetch request in one or more modes, such as issuing a fetch request toward the microphone, after which the user can retrieve the fetched information in the same mode (ie, voice). Or in different modes, such as using a graphical browser that presents the returned information in a visible format on the display screen. Within the communication device, the user agent program operates in the same manner as a standard web browser or other suitable software program residing on a device connected to the network or other terminal device.

  Accordingly, multimodal communication systems have been proposed in which one or more user input and output interfaces can be used by a user to facilitate communication in multiple modes during a session. User agent programs can be located on different devices. For example, a network element such as a voice gateway can include a voice browser. For example, the handheld device may be a graphical browser such as a WAP browser or other suitable text-based user agent program. Therefore, when a multimodal function is provided, the user can receive information that is input in one mode and returned in a different mode.

For example, systems have been proposed that attempt to provide user input in two different modes, such as entering some information in a voice mode and entering other information in a haptic or graphical interface. For example, one that has been proposed is the use of a serial asynchronous approach that requires the user to speak first and then send a short message after the speech is complete. Users in such systems may have to manually switch modes during the same session. Therefore, such a proposal can be cumbersome.

  Other proposed systems use a single user agent program and markup language tags in existing HTML pages, and the user can, for example, use voice instead of typing a search word. The user can navigate to the web page and then enter text information on the same HTML page. For example, the user can obtain visual map information from the content server by uttering the word “city” and inputting an address. However, such a proposed method usually requires multi-mode input in different modes with the same user agent program on one device (input through the same browse). Thus, voice and text information is typically entered in the same HTML format and processed through the same user agent program. However, this proposal requires the use of a single user agent program running on a single device.

  As such, on less complex devices, such as mobile devices with limited processing power and storage capacity, the use of complex browsers can degrade device performance. In addition, such a system cannot facilitate simultaneous multimodal information input through different user agent programs. Furthermore, it may be desirable to perform simultaneous multimodal input on multiple devices and distribute processing across different applications or different devices.

  Other proposals use multimodal gateways and multimodal proxies, fetch content with multimodal proxies, and retrieve the content into user agent programs (eg, browsers) and voice browsers (eg, browsers) in communication devices. In the network element so that the system side can use both voice and text output for one device. However, this approach does not seem to allow the user to input information simultaneously in different modes through different applications. The reason is that this proposal also seems to be a single user agent approach that requires different modes of fetched information to be output to a single user agent program or browser.

  Therefore, there is a need for improved simultaneous multimodal communication devices and methods.

The present invention will now be described by way of example, but the present invention is not limited to the accompanying drawings in which like reference numerals indicate like elements.
Briefly, multimodal network elements can be used to facilitate simultaneous multimodal communication sessions through different user agent programs on one or more devices. For example, a user agent program that communicates in a voice mode, such as a speech engine and a voice gateway voice browser that includes call / session termination, can be used by other users operating in different modes, such as a graphical browser on a mobile device Synchronize with the user agent program. Multiple user agent programs can operate in conjunction with the content server during a session to conduct simultaneous multimodal interactions.

For example, multimodal network elements differ from each other, such as obtaining different markup language formats associated with different modes, such as HTML formats associated with text modes and voiceXML formats associated with speech modes. Get mode specific instructions for multiple user agent programs running in mode. Multimodal network elements in the session synchronize the outputs from multiple user agent programs for the user based on the resulting mode specific instructions. For example, since a voice browser synchronizes with audio output on one device and a graphical browser synchronizes with the display output of a screen on the same device or a different device at the same time, the user can -It can be entered through an agent program. When a user inputs input information through multiple user agent programs operating in different modes, the method and apparatus provides a receiving simultaneous multimodal input by the user in response to a request for different simultaneous multimodal information. The input information and the received simultaneous multimodal input information transmitted from a plurality of user agent programs are fused or linked. Thus, simultaneous multimodal input is facilitated through different user agent programs, using multiple devices or other devices during a simultaneous multimodal session, or multiple user agent programs on one device Can be used. Different proxies communicate with each of the different user agent programs specified by the multimodal network element and set to different modes.

  FIG. 1 shows an example of a multimodal communication system 10 according to one embodiment of the present invention. In this example, multimodal communication system 10 includes content sources such as communication device 12, multimodal fusion server 14, voice gateway 16, and web server 18. The communication device 12 is, for example, an Internet home appliance, a PDA, a mobile phone, a cable set top box, a telematics unit, a laptop computer, a desktop computer, or other mobile or non-mobile device. Further, depending on the type of communication desired, the communication device 12 may be a wireless local area network or wireless wide area network 20, a WAP / data gateway 22, a short messaging service center (SMSC / paging network). Network) 24, or other suitable network, can also operate. Similarly, the multimodal fusion server 14 can be any suitable device, network element or network including the Internet, Intranet, Multimedia Server (MMS) 26, Instant Messaging Server (IMS) 28, or other suitable network. Can communicate. Accordingly, the communication device 12 communicates with and operates on an appropriate network via communication links 21, 23, and 25. Similarly, the multimodal fusion server 14 can be appropriately linked to various networks via a conventional communication link, indicated at 27. In this example, the voice gateway 16 includes, but is not limited to, a conventional voice gateway including a voice recognition engine, a handwritten character recognition engine, a face recognition engine, session control, user provided algorithms, and an operation and maintenance controller as needed. A function can be provided. In this example, the communication device 12 uses a user agent program 30 such as a WAP browser, gesture recognition, tactile recognition, or a visual browser (eg, a graphical browser) in the form of another suitable browser, such as a telephone. It is provided with a telephone circuit including a microphone and speaker shown as circuit 32. Other suitable configurations can also be used.

The voice gateway 16 includes another user agent program 34, such as a voice browser, that outputs audio information in a format suitable for output from the speaker of the telephone circuit 32. However, the speakers are placed on different devices other than the communication device 12, such as a pager or other PDA, so that the sound is output to one device, and the visual browser via the user agent program 30 is further different. It will be appreciated that it can be prepared on the device. Also, although the user agent program 34 resides within the voice gateway 16, the user agent program 34 may be contained within the communication device 12 (shown as a voice browser 36) or other suitable device. It will also be understood that this is possible. To accommodate simultaneous multimodal communication, as described herein, a plurality of user agent programs, ie, user agent program 30 and user agent program 34, are provided in a given session. In different modes with respect to each other. Thus, the user signs up for the disclosed service and presets the mode preferences in the mode preference database 36 accessible via the web server 18 or other server (including MFS 14). Thus, each mode of the user agent program can be defined in advance. In addition, the user can select or change the mode of a given user agent program during a session, as is known in the art, if desired.

  The simultaneous multimodal synchronization coordinator 42 provides a mode specific instruction for one of the plurality of user agent programs to compensate for communication delays associated with mode specific instructions for other user agent programs during the session. A buffer memory for temporary storage may be provided. Thus, for example, if necessary, the synchronization coordinator 42 can take into account system delays or other delays waiting for mode specific instructions to be rendered simultaneously on different user agent programs and output to the proxy.

  Also, if necessary, the user agent program 30 can include an input interface that allows the user to mute several multimodes. For example, if the device or user agent program is capable of multi-mode operation, the user can be instructed to mute the mode for a specific duration. For example, if the output mode for the user is voice, but a loud sound is generated in the environment where the user is present, the user can mute the output to the voice browser, for example. The multi-mode mute data received from the user is stored by the multi-modal fusion server 14, for example, in the memory 602 (see FIG. 5) and can indicate which mode is to be muted during a given session. is there. Thereafter, the synchronization coordinator 42 may refrain from obtaining mode specific instructions for the mode identified as mute.

  Information fetcher 46 obtains mode-specific instructions 69 from multi-mode application 54 for a plurality of user agent programs 30 and 34. Mode specific instructions 68, 70 are sent to user agent programs 30 and 34. In this embodiment, the multi-mode application 54 includes data identifying different user agent programs and thus mode specific instructions associated with different modes, as described below. The simultaneous multimodal synchronization coordinator 42 operates in conjunction with the information fetcher 46 to receive mode specific instructions. The simultaneous multimodal synchronization coordinator 42 further operates in conjunction with a plurality of proxies 38a-38n to specify the proxies required for a given session.

  If the different user agent programs 30 and 34 are on different devices, the method sends a markup language format based on the first mode to one device and a format based on the second mode markup language. Including sending a request for simultaneous multimodal input information 68, 70 by sending to one or more other devices and allowing the user to request simultaneous information input to different devices in different modes during the same session. . Forms based on these markup languages were obtained as mode specific instructions 68,70.

The multimodal session controller 40 is used to detect incoming sessions, respond to sessions, modify session parameters, terminate sessions, and exchange session and media information with session control algorithms on the device. The multimodal session controller 40 is the primary session end point of the session if necessary, or the secondary session end if, for example, the user wishes to establish a session with another gateway such as a voice gateway The point then allows a session with the multimodal session controller 40 to be established.

  The synchronization coordinator 42 sends output synchronization messages 47 and 49, including requests for simultaneous multimodal input information, to each proxy 38a and 38n to synchronize their output with each of the plurality of user agent programs. Take. The proxies 38a and 38n transmit input synchronization messages 51 and 53 including the received multimodal input information 72 and 74 to the simultaneous synchronization coordinator 42.

  The simultaneous multimodal synchronization coordinator 42 sends and receives synchronization messages 47, 49, 51, and 53 using a proxy or, if the user agent program is capable, using a user agent program. When 38a and 38n receive incoming multimodal input information 72 and 74 from different user agent programs, proxies 38a and 38n send input synchronization messages 51 and 53 containing incoming multimodal input information 72 and 74 to synchronization coordinator 42. To do. The synchronization coordinator 42 transfers the received information to the multimodal fusion engine 44. Further, when the user agent program 34 sends a synchronization message to the multimodal synchronization coordinator 42, the multimodal synchronization coordinator 42 sends the synchronization message to the other user agent program 30 in the session. The simultaneous multimodal synchronization coordinator 42 can also perform message conversion and perform synchronization message filtering to make the synchronization system more efficient. The simultaneous multimodal synchronization coordinator 42 can maintain a list of current user agent programs used in a given session and track their notification destinations when synchronization processing is required.

  The multimodal fusion server 14 includes a plurality of multimodal proxies 38a-38n, a multimodal session controller 40, a simultaneous multimodal synchronization coordinator 42, a multimodal fusion engine 44, an information (eg, mode specific instructions) fetcher 46, and A voiceXML interpreter 50 is provided. At least a multimodal session controller 40, a simultaneous multimodal synchronization coordinator 42, a multimodal fusion engine 44, an information fetcher 46, and a multimodal markup language (eg, voiceXML) interpreter 50, one or more processing devices It can be implemented as a software module to be executed. Accordingly, executable instructions that, when read by one or more devices, perform the functions described herein for each of the software modules on one or more processing devices are stored in memory. . Thus, the multimodal fusion server 14 includes, but is not limited to, processing devices that may include digital signal processors, microcomputers, microprocessors, state machines, or other suitable processing devices. It is not limited to. Memory includes state or other data that causes one or more processing devices to operate as described herein when executed by ROM, RAM, distributed memory, flash memory, or processing devices. There are other suitable memories that can be stored. Alternatively, the functionality of the software module may be implemented as appropriate with any suitable combination of hardware or hardware, software, and firmware as required.

The multimodal markup language interpreter 50 can be a state machine or other suitable hardware, software, firmware, or any suitable combination thereof, in particular the markup language provided by the multimodal application 54. Execute.

  FIG. 2 shows an example of a method for executing multimodal communication by the multimodal fusion server 14. However, it will be understood that any of the steps described herein can be performed by a suitable device or devices in a suitable order. In a current multimodal session, the user agent program 30 (eg, WAP browser) sends a request 52 to the web server 18 to request content from a simultaneous multimodal application 54 that is accessible to the web server 18. This can be done, for example, by entering a URL or clicking an icon or using other conventional mechanisms. Also, as indicated by dashed line 52, user agent programs 30 and 34 can each send user mode information to markup interpreter 50. The Web server 18 functioning as a content server obtains the multimodal preference 55 of the communication device 12 from the mode preference database 36 that has already been input through the user subscription process to the simultaneous multimodal service. To do. Next, the web server 18 notifies the multimodal fusion server 14 through a notification 56 containing user preferences from the database 36, for example which user agent program is being used in simultaneous multimodal communication, which Indicates whether each of the user agent programs is set in the mode. In this example, the user agent program 30 is set to the text mode, and the user agent program 34 is set to the voice mode. Thereafter, the simultaneous multimode synchronization coordinator 42 determines which of a plurality of multimodal proxies 38a-38n is used for each of the user agent programs 30 and 34 during the session. Thus, the simultaneous multimode synchronization coordinator 42 designates the multimode proxy 38a as a text proxy for communicating with the user agent program 30 that is set to text mode. Similarly, the simultaneous multimode synchronization coordinator 42 designates the proxy 38n as a multimodal proxy that conveys voice information of the user agent program 34 operating in voice mode. The information fetcher is shown as a web page fetcher 46 and obtains mode specific instructions, such as markup language format or other data, from the web server 18 associated with the simultaneous multimodal application 54.

For example, if the multimodal application 54 asks the user to enter information in both voice and text modes, the information fetcher 46 will output the associated HTML markup to be output to the user agent program 30. Obtain the associated voiceXML format that is output to the user agent program 34 via the language format and request 66. These mode-specific instructions are then rendered as output by the user agent program (eg, output to the screen or output by a speaker). A simultaneous multimodal synchronization coordinator 42 synchronizes the outputs from the plurality of user agent programs 30 and 34 based on mode specific instructions during the session. For example, the simultaneous multimodal synchronization coordinator 42 may have the appropriate time to render text as it is output on the screen via the user agent program 30 when audio is rendered on the communication device 12. To each user agent program 30 and 34 in the appropriate markup language format representing the different modes. For example, the multimodal application 54 may be able to communicate via the user agent program 34 for information expected to be entered via a text browser while simultaneously waiting for text input from the user agent program 30. A command in the form of a voice command can be given to the user. For example, the multimodal application 54 requires a voice output of the phrase “please enter your desired destined city followed by your desired depart time” while simultaneously on the display device of the communication device through the user agent program 30 In the next line, the field is designated as “C” meaning city, and the next line is designated as “D” meaning destination. In this example, the multimodal application does not require simultaneous multimodal input by the user, but only requests input through one mode, the text mode. Other modes are used to send user commands.

Alternatively, if the multi-modal application 54 requires the user to enter input information through multiple user agent programs, the multi-modal fusion engine 14 may use different multi-modal user agent programs during the session. Merge user inputs that are input simultaneously. For example, while clicking on two positions on the map where the user is displayed, “directions from here to
utter the phrase "there", the voice browser or user agent program 34 writes "here" in the start position field and "the" as received input information 74 in the destination position field, while a graphical browser, That is, the user agent program 30 writes the geographical position (for example, latitude / longitude) of the first click point on the map in the start position field, and the geography of the second click point on the map in the destination position field. Write the target location (eg latitude / longitude). The multimodal fusion engine 44 obtains this information, fuses the input information entered by the user from multiple user agent programs operating in different modes, and the word “here” is the geographical location of the first click point. Corresponding to the position, it is determined that the word “there” corresponds to the geographical position (eg, latitude / longitude) of the second click point. In this way, the multimodal fusion engine 44 has a complete set of user commands. On the multimodal fusion engine 44 side, it may be desirable to send the fused information 60 back to the user agent programs 30 and 34 to have complete information related to simultaneous multimodal communication. At this time, the user agent program 30 can submit this information to the content server 18 to obtain desired information.

  As shown in block 200, for a session, the method includes obtaining mode specific instructions 68, 70 for a plurality of user agent programs operating in different modes relative to each other, eg, a plurality of Different types of markup languages specific to each mode of each of the user agent programs. As shown in block 202, the method synchronizes the output of a user agent program or the like during a session based on mode specific instructions that allow the user's simultaneous multimodal operation to be performed smoothly. Including doing. Therefore, the rendering of the markup language format is synchronized, and the outputs from the plurality of user agent programs are simultaneously rendered in different modes through the plurality of user agent programs. As shown in block 203, the simultaneous multimodal synchronization coordinator 42 performs mode specific instruction sets 68, 70 for different user agent programs 30 and 34 by the user using different user agent programs. Determine whether to request simultaneous input of information in different modes. If not, as indicated at block 205, the simultaneous multimodal synchronization coordinator 42 forwards input information received from only one user agent program to the destination server or Web server 18.

However, as shown in block 204, if mode specific instruction sets 68, 70 for different user agent programs 30 and 34 require simultaneous user input in different modes, the method operates in different modes. Fusing received user-entered multimodal input information sent back by user agent programs 30 and 34 to generate a fused multimodal response 60 associated with different user agent programs. As indicated at block 206, the method includes forwarding and returning the fused multimodal response 60 to the currently executing application 61 with the markup language interpreter 50. The currently running application 61 (see FIG. 5) is a markup language from the application 54 that is running as part of the interpreter 50.

  The detailed operation of the multimodal communication system 10 will be described with reference to FIGS. As shown in block 300, the communication device 12 sends a request 52 for web content or other information via the user agent program 30. As shown in block 302, the content server 18 determines the multimodal preferences from the user's mode preference database 36 that has been identified to obtain the device and mode preferences for the session. Reference data 55 is acquired. As shown in block 304, the method may include the content server to the multimodal fusion server 14, which user agent application on which device, which mode for a given concurrent different multimodal communication session. Including notifying you that you are working.

  As described above and shown in block 306, the simultaneous multimodal synchronization coordinator 42 assigns each proxy for each of the different modes based on mode preference information 55 from the mode preference database 36. Set up to determine. As indicated at block 308, the method includes receiving a user mode designation for each user agent program via the multimodal session controller 40 as needed. For example, the user can change the desired mode to be different from the preset mode preferences 55 stored in the mode preferences database 36. This can be done using conventional session messaging functions. If the user is changing the desired mode for a particular user agent program, such as when the desired user agent program is on a different device, different mode specific instructions are required, such as different markup language formats May be. If the user mode designation has changed, the information fetcher 46 fetches and requests the appropriate mode specific instructions based on the mode selected for the user agent application.

  Thereafter, as shown in block 310, the information fetcher 46 fetches from the content server 18 a mode specific instruction, shown as a fetch request 66, for each user agent program and thus for each mode. Thus, the multimodal fusion server 14 obtains a markup language representing different modes via the information fetcher 46 so that each user agent program 30 and 34 can operate in different modes based on the markup language. Can output information. However, it will be understood that the multimodal fusion server 14 can obtain any suitable mode specific command, not just information based on markup language.

If a mode specific instruction is fetched from the content server 18 for each user agent program and the CMMT is not associated with the mode specific instructions 68, 70, the received mode specific instruction 69 is transcoded 608 (see FIG. 5). Can be sent to. The transcoder 608 transcodes the received mode-specific instructions into a basic markup language format so that it can be interpreted by the interpreter 50 to provide different mode 6
A basic markup language form is created having data identifying mode specific instructions for 10. Thus, the transcoder transcodes the mode specific instructions to include data identifying the mode specific instructions for other user agent programs operating in different modes. For example, if the interpreter 50 uses a basic markup language such as voiceXML, and one mode-specific instruction set of the application 54 is in voiceXML format and the other is in HTML format, the transcoder 608 acquires the HTML format. Embed the CMMT that identifies the URL of a possible location or the actual HTML format itself in the voiceXML format. Further, if none of the mode-specific instructions are in the basic markup language, a set of mode-specific instructions is converted to the basic markup language, and thereafter, the other mode-specific instructions are referred to by the CMMT.

  Alternatively, the multimodal application 54 can provide the necessary CMMT information to facilitate smoothing of output synchronization by multiple user agent programs during a simultaneous multimodal session. An example of mode specific instructions for each user agent program is shown below in markup language format. The markup language form is supplied by the multimodal application 54 and is used by the multimodal fusion server 14 to perform simultaneous multimodal communication sessions. The multimodal voiceXML interpreter 50 assumes that the multimodal application 54 uses voiceXML as a basic language. Includes or enhances simultaneous multimodal tags (CMMT) such as voiceXML format extensions or HTML format indexes to allow multiple user agent programs to smoothly synchronize output on behalf of the user A multimodal application 54 can be created that generates an index. The CMMT identifies or includes information such as the actual HTML format that identifies the mode and is output by one of the user agent programs in the identified mode. CMMT is also used as multi-modal synchronization data, indicating that it is necessary to synchronize different mode specific instructions with different user agent programs by including CMMT.

  For example, if voiceXML is the base language of the multimodal application 54, it indicates that CMMT is in text mode. In this example, the CMMT can include a URL that includes HTML formatted text that is output by the user agent program, or can include HTML as part of the CMMT. The CMMT can be provided with a property of a markup language attribute extension function. The multimodal voiceXML interpreter 50 uses the information fetcher 46 to fetch mode specific instructions, parses (executes in this example) the mode specific instructions fetched from the multimodal application, and detects the CMMT. After being detected, the multimodal voiceXML interpreter 50 interprets the CMMT and, if necessary, obtains other mode specific instructions, such as HTML for text mode.

For example, the CMMT can indicate where to obtain text information for a graphical browser. Shown below is a voice output by the voice browser asking “where from” and “where to”, while the graphical browser is “from
FIG. 10 is a table showing an example of mode specific instructions for simultaneous multimodal cyclic application in the form of voiceXML for simultaneous multimodal applications that need to be labeled “city” and “to city”. In the fields labeled “from city” and “to city”, it is expected that the user will enter the received simultaneous multimodal information through different browsers.

Table 1
<Vxml version = “2.0”>
<Form>
<Block>
<Cmd mode = “html” src = “./ internally.html” /> non-speech mode is html (text), and source information is itinerary. Indicates that it is placed in url html </ block>
<Field name = “from_city”> Expected information text fragment to be collected through a graphical browser Need to create Where from? Prompt from voice browser </ field>
<Field name = "to_city"> Expects text <grammar src = "./ city.xml"/>
Where to? Voice generated by voice browser </ field>
</ Form>
</ Vxml>

Thus, the markup language format described above is written in a basic markup language that represents mode specific instructions for at least one user agent program, and the CMMT can be used by other user agent programs operating in different modes. An extension that specifies mode specific instructions for a program.

  As shown in block 311, if the user changes the preference, the method includes resetting the proxy to be consistent with the change. As shown in block 312, the multimodal fusion server 14 determines whether a reception wait point has been reached. If so, the next state is entered as shown in block 314. If yes, this process is complete. If not, the method includes performing a synchronization process of mode specific instructions for different user agent programs. In this example, the multimodal voiceXML interpreter 50 outputs HTML for the user agent program 30, voiceXML for the user agent 34, and the simultaneous multimodal synchronization coordinator 42, and outputs by a plurality of user agent programs. Synchronize. This can be executed based on the occurrence of a reception standby point, for example, as described above. This is indicated by block 316.

  As shown in block 318, the method sends synchronized mode-specific instructions 68, 70 to the corresponding proxies 38a and 38n, such as by a simultaneous multimodal synchronization coordinator 42, and varies from user to user in the same session. Requesting user input information in mode. Synchronized requests 68 and 70 are sent to each of the user agent programs 30 and 34. For example, requests for different simultaneous mode input information corresponding to multiple input modes associated with different user agent programs are shown as synchronized requests that include mode specific instructions 68 and 70. These can be, for example, synchronized markup language formats.

When user agent programs 30 and 34 render mode-specific instructions at the same time, the method determines whether user input is received within the timeout period as indicated in block 320, or other event has occurred. Including determining whether. For example, the multimodal fusion engine 44 may wait for a period of time and then determine whether multimodal input information entered by the user for fusion has been properly received from a plurality of user agent programs. This waiting period can be different depending on the mode setting of each user agent program. For example, if a user is expected to input both voice and text information at the same time, but the multimodal fusion engine has not received information about the fusion within a certain period of time, it is considered that an error has occurred. Furthermore, when the multimodal fusion engine 44 is used, since it takes a relatively long time for processing via the voice gateway 16 for voice information, it may take a longer time to return the voice information than for text information. is there.

  In this example, the user is required to input text via the user agent program 30 and at the same time utter voice information using a microphone and transmit it to the user agent program 34. Received simultaneous multimodal input information 72 and 74 is received from user agent programs 30 and 34 and passed to the respective proxies via appropriate communication links. The communication indicated by reference numeral 76 between the user agent program 34 and the microphone and speaker of the device 12 is performed in PCM format or other suitable format, in this example output by the user agent program. Note that it is not a mode-specific command format that can be done.

  When a user inputs information using a text browser and a voice browser simultaneously, and the multimodal fusion engine 44 receives simultaneous multimodal input information transmitted from a plurality of user agent programs, the multimodal fusion engine 44 Fuses the input information 72 and 74 received from the user as shown in block 322.

  FIG. 4 shows an example of the operation of the multimodal fusion engine 44. For the sake of explanation, for a certain event, “no input” means that the user has not entered anything in this mode. “No match” means that something was entered but not the expected value. The result is a set of slot (or field) name and corresponding value pairs from what was successfully entered by the user. For example, suitable input content is “City = Chicago” and “State = Illinois” and “street” = “first street” and a confidence weighting factor in the range of 0% to 100%, for example. As described above, whether the multimodal fusion engine 44 fuses information can vary depending on the time between receipt of the slot name (eg, variable) and value pair or the receipt of expected or other events. There is sex. This method assumes that a confidence level is assigned to the received information. For example, the synchronization coordinator and its weight reliability based on mode and information arrival time. For example, if it is possible to enter the same slot data in different modes during the same session (for example, speaking and keying the street name of the address), the entered data is more than the spoken data Is also assumed to be accurate. The synchronization coordinator receives from one of a plurality of user agent programs sent in response to a request for different simultaneous multimodal information based on the time of reception and on the confidence value of the individual results received. Combine multimodal input information.

As shown in block 400, the method includes determining if there was an event or result in non-voice mode. If yes, as shown in block 402, the method includes determining if there were any modes of events except for “no input” and “no match” events. If so, the method includes returning the received first such event to the interpreter 50, as shown in block 404. However, if there are no events from the user agent program other than “no input” and “no match”, the method may include two or more of the multimodal fusion engines as shown in block 406. For the mode that transmitted the result, combining the results of that mode in the order of the reception time. This may be useful when the user re-enters input for the same slot. The later value for a given slot name overwrites the previous value. The multimodal fusion engine adjusts the mode result confidence weight based on the individual result confidence weights that are components. The final result is one answer for each slot name for each mode. The method includes taking the results from block 406, as shown in block 408, and combining them into one combined result for all modes. The method includes starting with the least reliable result and proceeding to the most reliable result. For each slot name in the merged result, receive the slot value belonging to the most reliable input result including the definition of that slot.

  As shown in block 410, the method includes determining whether there are now combined results. That is, whether the user agent program has sent a result to the multimodal fusion engine 44. If yes, the method includes returning the combined result to the content server 18, as shown in block 412. If not, it means that there are zero or more “no input” or “no match” events, as shown in block 414. The method includes determining whether there is a “no match” event. If yes, the method includes returning a “no match” event, as shown in block 416. However, if there is no “no match” event, the method includes returning a “no input” event to the interpreter 50, as shown in block 418.

  Returning to block 400, if there is no event or result from the non-voice mode, the method determines whether the result is returned in voice mode, that is, whether the user agent program 34 has generated the received information 74. Including that. This is indicated by block 420. If yes, as indicated at block 422, the method includes returning a voice response to the received input information to the multimodal application 54. However, if the voice browser (eg, user agent program) did not output information, the method includes determining whether an event was returned in voice mode, as shown in block 424. . If yes, the event is reported to multimodal application 54 at 73, as shown in block 426. If a voice mode event has not been generated, the method includes returning a “no input” event, as shown in block 428.

Table 2 below shows an example of the method of FIG. 4 applied to hypothetical data.

Table 2
VoiceModeCollectedData
STREETNAME = Michigan
TIMESTAMP = 0
CONFIDENCELEVEL =. 85
NUMBER = 112
TIMESTAMP = 0
CONFIDENCELEVEL =. 99

TextModeCollectedData
STREETNAME = Michigan
TIMESTAMP = 0
CONFIDENCELEVEL = 1.0
STREETNAME = LaSalle
TIMESTAMP = 1
CONFIDENCELEVEL = 1.0

For example, if, at block 400, no result from non-voice mode is received, the method proceeds to block 402. If at block 402 no events are received, the method proceeds to block 406. At block 406, the fusion engine compresses TextModeCollectedData to one response per slot. VoiceModeCollectedData remains as it is.

VoiceModeCollectedData
STREETNAME = Michigan
TIMESTAMP = 0
CONFIDENCELEVEL =. 85
NUMBER = 112
TIMESTAMP = 0
CONFIDENCELEVEL =. 99
OVERALLCONFIDENCE =. 85

The voice mode remains the same. However,. Since 85 is the lowest confidence in the result set, the total confidence value. 85 is assigned.

TextModeCollectedData
STREETNAME = Michigan
TIMESTAMP = 0
CONFIDENCELEVEL = 1.0
STREETNAME = LaSalle
TIMESTAMP = 1
CONFIDENCELEVEL = 1.0

Since LaSale is written to the slot at a later time stamp, Michigan is deleted from the collected data in the text mode. The final result looks like this: And since 1.0 is the lowest confidence level in the result set, a total confidence level of 1.0 is assigned.

TextModeCollectedData
STREETNAME = LaSalle
TIMESTAMP = 1
CONFIDENCELEVEL = 1.0
OVERALLCONFIDENCE = 1.0

The data transmitted to block 408 is shown below.

VoiceModeCollectedData
STREETNAME = Michigan
TIMESTAMP = 0
CONFIDENCELEVEL =. 85
NUMBER = 112
TIMESTAMP = 0
CONFIDENCELEVEL =. 99
OVERALLCONFIDENCE =. 85

TextModeCollectedData
STREETNAME = LaSalle
TIMESTAMP = 1
CONFIDENCELEVEL = 1.0
OVERALLCONFIDENCE = 1.0

At block 408, effectively the two modes are merged into a single return result.
First, the entire result with the lowest confidence level is retrieved and placed in the final result (FinalResult) structure.

FinalResult

STREETNAME = Michigan
CONFIDENCELEVEL =. 85
NUMBER = 112
CONFIDENCELEVEL =. 99

The next lowest result element is then replaced in the final result.

FinalResult

STREETNAME = LaSalle
CONFIDENCELEVEL = 1.0
NUMBER = 112
CONFIDENCELEVEL =. 99

This end result is a blend of the two modes and is sent to the interpreter where it decides what to do next (fetch more information from the Web or need more information from the user? Determine whether and re-prompt based on current state).

FIG. 5 illustrates a multi comprising a simultaneous multi-modal session persistence controller 600 and a simultaneous multi-modal session status memory 602 coupled to the simultaneous multi-modal session persistence controller 600. Another embodiment of the modal fusion server 14 is shown. The simultaneous multimodal session persistence controller 600 may be a software module running on a suitable processing device, or suitable hardware, software, firmware, or a suitable combination thereof. it can. The concurrent multimodal session persistence controller 600 maintains concurrent multimodal session status information 604 in the form of a database or other suitable data structure for each user when in a non-session state. Simultaneous multimodal session status information 604 is status information for a plurality of user agent programs configured to perform different simultaneous mode communications during a session. Simultaneous multimodal session persistence controller 600 re-establishes a concurrent multimodal session that has already ended in response to access to simultaneous multimodal session status information 604. The multimodal session controller 40 notifies the simultaneous multimodal session persistence controller 600 when the user has joined the session. The multimodal session controller 40 also communicates with the simultaneous multimodal synchronization coordinator to synchronize with offline devices or to re-establish a simultaneous multimodal session. Synchronize with.

  The simultaneous multimodal session persistence controller 600 stores proxy ID data 906, such as a URL indicating the proxy used for a given mode during a previous simultaneous multimodal communication session, for example. If necessary, the simultaneous multimodal session state memory 602 also includes information indicating which fields or slots were written by input from the user during a previous simultaneous multimodal communication session along with the contents of such fields or slots. Can be stored. Further, the simultaneous multimodal session state memory 602 can include a current interaction state 606 for the simultaneous multimodal communication session. The state includes a case where the interpreter 50 is in the execution state of the application being executed. Information regarding which field the user has written can be in the form of fused input information 60.

  As shown in the figure, the web server 18 can include mode specific instructions for each mode type. In this example, the text is supplied in the form of HTML, the speech is supplied in the form of voiceXML, and the speech is further supplied in the WML format. The simultaneous multimodal synchronization coordinator 42 outputs the appropriate format to the appropriate proxy. As shown in the figure, the voiceXML format is output through the proxy 38a designated for the voice browser, while the HTML format is output to the proxy 38n for the graphical browser.

  Session persistence maintenance can be used when a session ends abnormally and the user wants to return to the same interactive state afterwards. In addition, the mode uses a transport mechanism with different delay characteristics that causes delays from input to output in different modes and requires information to be temporarily stored to compensate for the time delay. It is also beneficial to do.

As shown in FIGS. 6-7, the simultaneous multimodal session persistence controller 600 provides multimodal session status for multiple user agent programs for a given user during a given session. The information is retained and the user agent program is configured to allow different simultaneous mode communications during the session. This is indicated by block 700. As shown in block 702, the method includes re-establishing a previous simultaneous multimodal session in response to accessing the multimodal session status information 604. More specifically, as shown in block 704, during a simultaneous multimodal session, the simultaneous multimodal session persistence controller 600 stores per-user multimodal session status information 604 in memory 602. To do. As shown in block 706, the simultaneous multimodal session persistence controller 600 detects user participation in the session from the session controller and retrieves the user ID in memory so that the user Determine if you were involved in a multimodal session. Accordingly, as shown in block 708, the method accesses multimodal session status information 604 stored in memory 602 based on detection of session participation by the user.

  As shown in block 710, the method includes determining whether a session exists in memory 604. If not, the session is designated as a new session, a new entry is created, and the data necessary to record the new session in memory 602 is written to that entry. This is indicated by block 712. As shown in block 714, if a session exists, for example, if the session ID exists in memory 602, the method causes memory 602 to have the user run an existing application. Including executing the query, and if so, the user can execute the query if they want to re-establish communication with the application. If the user desires, the method includes retrieving the URL of the last fetched information from memory 602. This is indicated by block 716 (FIG. 7). As shown in block 718, the appropriate URL retrieved in block 716 is provided to the appropriate proxy 38a-38n. As shown in block 720, the method includes sending a request to the appropriate user agent program via the proxy based on the user agent state information 606 stored in the memory 602. .

  FIG. 8 is a diagram illustrating an example of the contents of the simultaneous multimodal session status memory 602. As shown in the figure, a user ID 900 can be used to specify a particular user and when the user stores multiple sessions in the memory 602, the session ID 902 can be associated with the user ID. Further, the user agent program ID 904 indicates, for example, a device ID related to a device executing a specific user agent program. The program ID may be a user program identifier, URL, or other address. Proxy ID data 906 indicates that a multimodal proxy is used in the previous simultaneous multimodal communication. Thus, the user can end the session and later continue from where it ended.

  Holding the device ID 904, among other things, allows the user to easily switch between devices during simultaneous multimodal communication because the system can hold the identity of the device being used during the execution of the simultaneous multimodal session.

  Thus, multiple inputs entered in different modes, through separate user agent programs distributed across one or more devices (or if included in the same device), are unified and consistent. It is fused in a sexual way. In addition, a mechanism is provided that synchronizes both the rendering of user agent programs and the information input by the user using those user agent programs. Further, the disclosed multimodal fusion server can be coupled to existing devices and gateways to allow simultaneous multimodal communication sessions.

  It will be understood that other variations and modifications of the invention in various aspects will be apparent to those skilled in the art and that the invention is not limited to the specific embodiments described. Let's go. For example, although the method of the present invention has been described with respect to several steps, it will be understood that the steps can be performed in an appropriate order as needed. Accordingly, some and all modifications, variations, or equivalents that fall within the spirit and scope of the basic principles disclosed and claimed herein are intended to be covered by the present invention.

1 is a block diagram illustrating an example of a multimodal communication system according to an embodiment of the present invention. 5 is a flowchart for explaining an example of a method of multimodal communication according to an embodiment of the present invention. 5 is a flowchart for explaining an example of a method of multimodal communication according to an embodiment of the present invention. 5 is a flowchart illustrating an example of a method for fusing received simultaneous multimodal input information according to an embodiment of the present invention. 1 is a block diagram illustrating an example of a multimodal network element according to an embodiment of the invention. 5 is a flow diagram illustrating an example method for maintaining multimodal session persistence according to one embodiment of the invention. 7 is a flowchart for explaining a part of the flowchart shown in FIG. 6. FIG. 4 is a block diagram illustrating an example of memory contents of a status of simultaneous multimodal sessions according to an embodiment of the present invention.

Claims (23)

A method of multimodal communication,
Obtaining mode specific instructions for multiple user agent programs operating in different modes relative to each other; and
Synchronizing the output from the plurality of user agent programs based on the mode specific instructions during a session.   Synchronizing output by the plurality of user agent programs transmits at least a portion of a markup language representing a different mode for use by the plurality of user agent programs, wherein each user agent The method of claim 1, comprising causing the program to output information in different modes based on the at least part of the markup language.   One of the plurality of user agent programs includes a graphical browser, the other one of the plurality of user agent programs includes a voice browser, and the output from the plurality of user agent programs is The method of claim 2, comprising user input entered simultaneously in different modes through the plurality of user agent programs.   Transmitting at least a portion of a markup language representing different modes for use by the plurality of user agent programs, transmitting markup language forms associated with the different modes to a plurality of different devices. The method of claim 2, wherein each of the devices is for operating one of the plurality of user agent programs.   Sending at least a portion of a markup language representing a different mode for use by the plurality of user agent programs sends a markup language form associated with the different mode to the same device. The method of claim 2, wherein the device is for operating the plurality of user agent programs in different modes.   The method of claim 1, comprising determining a proxy to communicate with each of the plurality of user agent programs during a session.   Obtaining a mode specific instruction for the plurality of user agent programs comprises communicating with an application that provides a different markup language format for each of the plurality of user agent programs; The method of claim 2, wherein the format represents a different mode.   Obtaining mode-specific instructions for the plurality of user agent programs operating in different modes comprises a markup language form written in a basic markup language that represents the mode-specific instructions for at least one user agent program. The method of claim 1 including fetching, wherein the markup language format includes data identifying mode specific instructions for other user agent programs operating in different modes. Synchronizing the output from the plurality of user agent programs based on the mode specific instructions analyzes the mode specific instructions fetched from the multimodal application to detect simultaneous multimodal tags (CMMT) And if detected,
The method of claim 1, comprising obtaining a mode specific instruction for at least one user agent program based on the CMMT. A multimodal network element,
An information fetcher that operates to obtain mode specific instructions for a plurality of user agent programs operating in different modes relative to each other during the same session;
A simultaneous multimodal synchronization coordinator that operates in conjunction with the information fetcher and operates to synchronize output from the plurality of user agent programs based on the mode-specific instructions during the session; Multimodal network element.   The multimodal network element of claim 10, wherein the simultaneous multimodal synchronization coordinator determines a proxy for communicating with each of the plurality of user agent programs during a session. A method of multimodal communication,
Sending a request for simultaneous multimodal input information corresponding to multiple input modes associated with multiple user agent programs operating in the same session; and
Fusing received simultaneous multimodal input information transmitted from the plurality of user agent programs transmitted in response to requests for different simultaneous multimodal information. Determining a proxy for each different mode associated with each application associated with a different mode in a given session prior to sending said request for simultaneous multimodal input information;
13. The method of claim 12, comprising synchronizing the request for different simultaneous multimodal input information with the plurality of user agent programs using the proxy determined for each different mode. .   13. The method of claim 12, comprising waiting for a period of time to determine whether the multimodal input information has been properly received for fusion.   15. The method of claim 14, comprising waiting for different periods depending on the mode of each user agent program.   Sending the request for simultaneous multimodal input information corresponding to multiple input modes sends mode specific instructions for multiple user agent programs operating in different modes to each other, and simultaneous information input of different modes 13. The method of claim 12, comprising requesting:   The step of transmitting a request for simultaneous multimodal input information corresponding to the plurality of input modes includes transmitting a first mode-based markup language format to the device and a second mode-based markup language format. 13. The method of claim 12, comprising transmitting to one or more devices to request simultaneous information input in different modes.   Sent from one of the plurality of user agent programs sent in response to a request for different simultaneous multimodal information based on the received time and on the confidence value of the individual results received The method of claim 12, comprising combining received multimodal input information.   The method of claim 12, comprising transcoding the mode specific instructions to include data identifying the mode specific instructions for other user agent programs operating in different modes. A multimodal network element,
A plurality of proxies each sending a request for simultaneous multimodal input information corresponding to a plurality of input modes associated with a plurality of user agent programs operating during the same session;
Acting to respond to received simultaneous multimodal input information sent from the plurality of user agent programs sent in response to requests for different simultaneous multimodal information, and different user agent programs during the same session A multimodal network element comprising: a multimodal fusion engine operable to fuse different multimodal input information transmitted from the plurality of user agent programs to provide simultaneous multimodal communication from . The request for different simultaneous multimodal information includes mode specific instructions for the plurality of user agent programs operating in different modes with respect to each other and requesting different modes of simultaneous information input, the multimodal network element comprising: ,
An information fetcher operative to obtain mode specific instructions for the plurality of user agent programs operating in different modes relative to each other during the same session;
A simultaneous multimodal synchronization coordinator that operates in conjunction with the information fetcher and the plurality of proxies and operates to synchronize received simultaneous multimodal input information output from the plurality of user agent programs during a session The multimodal network element of claim 20 comprising: Simultaneous multimodal operation of the plurality of user agent programs operating in conjunction with the plurality of user agent programs and configured for different simultaneous mode communication during a session for each user during a non-session state A simultaneous multimodal session persistence controller that maintains session status information and re-establishes the simultaneous multimodal session in response to accessing the simultaneous multimodal session status information;
The multimodal network element of claim 21, comprising: a memory operatively coupled to the simultaneous multimodal session persistence controller and including the simultaneous multimodal session status information.   Data comprising a markup language interpreter and a transcoder operating in conjunction with the markup language interpreter, the transcoder identifying mode specific instructions for other user agent programs operating in different modes 21. The multimodal network element of claim 20, transcoding mode specific instructions to include:

Images