ES2382747A1 - Multimodal interaction on digital television applications - Google Patents

Abstract

The invention relates to a method of multimodal interaction on digital television applications, wherein the multimodal application resides in a web server and is downloaded by a browser (110) residing in the actual television decoder (100). All the multimodal interaction analysis processes can be performed in real time using a distributed system of components and through the communications protocols. The system allows the interaction of the user with the application by means of using the remote control or voice.

Description

Multimodal interaction on applications of digital television

Field of the Invention

The present invention applies to the sector of digital television, more specifically to the field of interactions man-machine on terminals like Digital television decoders or mobile phones capable to run interactive applications that are displayed on a television.

Background of the invention

A multimodal system must allow simultaneously different methods or input mechanisms (keyboard, voice, images, etc.), collecting the information of each one of them as needed, for example, sometimes, the user could say something using a vocal command, but other times it could select a name from a list by using the keyboard and you could even select a menu or a part of the screen pointing with your own finger making the interface engine multimodal is able to detect the interaction method that the user has freely chosen (discarding incongruous information received through the other methods).

Regarding the interface design of user, these have traditionally been based on the metaphor of desk, developed decades ago in Xeros laboratories, and that tries to move all the objects and tasks that normally they are made in a real office to the world of computers; so by For example, both real and electronic files can be stored, the traditional typewriter has its equivalent in the word processor, the blank folio is equivalent to the document blank processor, etc. This way you get the mental model that the user has when performing these tasks traditional be kept with few changes when moving to field of computers, that is, try to reach the highest degree of familiarity between objects and actions. This metaphor of desktop has been implemented through the WIPM paradigm (in English, Windows, Icons, Pointers and Menus) which are the main elements that support the vast majority of interfaces Current graphics

However, this paradigm is clearly inappropriate in an interactive Digital TV environment for several reasons. The first one is related to the own one nature of the tasks that a user performs on an application interactive (more relaxed and close to an environment of entertainment, social, etc.) which make them very different from those of a real office. As a second point you have to point out that the device with which the user interacts (command distance) is very different in functionality and accessibility from the keyboard and mouse, which imposes many restrictions when it comes to perform tasks on a Digital TV environment (for example, the text entry via remote control to perform a simple search can become an expensive task). During quite a few years, and since its inception, the remote control used in the TV environment has become the device for excellence and through it it has been possible to control a great variety of devices and functions associated with them. Without However, the task models used in any of the interactive services currently deployed commercially about any of the distribution technologies and environments of development of the same, make numerous times their use is inefficient presenting great problems of usability, which translates into a demotivation and loss of interest in user exploration (usability is defined as the efficiency and satisfaction with which a product allows to achieve specific objectives, such as the purchase of a football match, to specific users, such as TV viewers, in a specific context of use, as per example the living room of a house).

If we also consider that numerous people have accessibility problems when using a remote traditional distance, we can conclude that clearly the mechanism traditional interaction with television has become outdated and surpassed by the new interactive services executed on The digital television decoders.

Tasks such as entering text with the remote control when performing an EPG search (Electronic Programming Guide) or the ability to send a message through an application Interactive TV, it can become an expensive task that will do Finally, the user may lose interest in his utilization. When entering this data, it is usually used a virtual keyboard that appears on the screen and can have a similar appearance to the keyboard of a mobile phone or that of a ANSI keyboard In any case the process is slow, not all the world is used to using the remote control as if it it is a keyboard of a mobile phone and they are also not infrequent mistakes made when using this mechanism (The remote control works by infrared which, depending on the ambient light, objects located between the user and the receiver, etc. you can make the keystroke not translate into a character entry). Almost all studies and tests of usability that are performed on interactive applications They point to this process as expensive for the user.

It should also be noted that TV has a character much more social, and the user is usually in a much more relaxed atmosphere, sitting at 3-4 meters of TV, and with an attitude of much less concentration than It requires working with a computer. It is clear that many of the tasks that are performed on a computer through a graphical interface traditional can not be done or will have to be made of A very different way. All of the above has made necessarily this desktop metaphor has been abandoned in Digital TV application developments.

Interactive applications on Digital TV, in addition, they run on a single window presented simultaneously (instead of several as the graphical interfaces of PC, for example) for all the restrictions indicated above. On this window arranges the different multimedia objects that make up the scene (texts, graphics, videos, etc.) trying to make all of them are synchronized based on a timeline generating a set of scenes that describe the different actions or steps that the user must complete until they get their objective. For example, when buying a movie from a system interactive video on demand, the user must initially enter that section, perform a content search based on some criteria, enter the data, select the content, enter a purchase PIN, etc. In a synchronized way they go appearing the different objects in the scene as the User interacts with them.

Although this concept based on the description of scenes and simultaneous presentation of objects (videos, texts, graphics) may seem simple is costly from a point of graphic processing view and especially accentuates about those devices - like TV decoders - where business models, for cost reasons, impose restrictions important in the electronic components that constitute the device. However, technologies and technologies already exist in the industry mechanisms that support this paradigm at the layer level of presentation.

If we wanted to transfer the concept of synchronization of objects in the presentation layer to that of synchronization of the different interaction mechanisms (command to distance, interaction through vocal commands, etc.) we would see that about Digital TV environments have barely developed architectures that support this synchronization of mechanisms of interaction. For example, we might want to go back to the interactive application of buying a video movie under demand, perform the content search using a command vowel that is given to the system, but then introduce - for reasons Privacy - Purchase PIN via remote control traditional. Simultaneous management of the different mechanisms of interaction is complex from a semantic point of view (for example, when counter and simultaneous orders are given through a vocal and graphic interface) and expensive in processing resources computational If we add to this the fact that the applications multimodal interactive will run on devices with little Processing capacity (100 MHz CPUs and limited RAM 32-64 MB and much lower than the performance of any Home PC, 1 GHz CPUs and 1 GB RAM) and on which -by example, digital television decoders - it is impossible Perform real-time voice processing, can be concluded that there are important technical difficulties when offering Multimodal interfaces over Digital TV environments.

There are different patents that deal with multimodal interfaces, although none do it concretely Applied to the field of digital television.

US5265014 is focused on how resolve ambiguities that occur when using natural language As an interaction mechanism. US4829423 is similar to the previous one although it is more focused on how to solve the problems that they produce when using natural language in a multimodal environment. US6345111 is more related to a mechanism of image analysis so that the system is able to recognize on what object the user has his sights set, so that serve as an input mechanism in the selection of elements. In US5577165 describes in general how the mechanism is performed of dialogue between a machine and a person, taking into account the keywords detected during the recognition process as well as the different states that the system goes through during the dialogue.

Try to implement a complete architecture of use of multimodal interfaces on devices lightweight, such as digital TV decoders or mobile phones, it presents performance problems (mainly related to real-time voice processing or the incorporation of a multimodal interpreter, for example SALT or VoiceXML, within the own device that is also expensive in terms of memory processing and reservation), so it becomes necessary address new architectures that allow getting times of response in the interaction process using any of the mechanisms (visual or vocal commands) are the minimum possible as they are already produced in other more powerful devices such as PCs, where all voice processing (synthesis and recognition) is performed on the machine itself, without requiring the external processing So for example, the CPU and RAM necessary to perform a voice recognition process in real time with acceptable response times (less than 5 seconds) and allow the user not to lose attention about the system would imply the use of a current medium-type PC (512 RAM, 1 GHz CPU), which means that in any case you gave a lot of the processing capabilities of the decoders of digital television (32 MB RAM, 100 MHz), even those of more range high (128 MB RAM, 200 MHz CPU).

In short, the technical problem is that the limited processing capacity of decoders of digital television, prevent them from being developed authentic multimodal applications that use as mechanisms of interaction in addition to visual vocal interaction so simultaneous.

Object of the invention

The object of the present invention is to create a method, platform or system that allows for an environment of digital television live simultaneously and synchronously different interaction mechanisms, thus producing what is known as multimodal interfaces. For this, the invention proposes a method of multimodal interaction on applications Interactive Digital TV, where television is provided with a network decoder that incorporates an associated browser, and where The method consists mainly of the following steps:

to.

Connecting the browser to a network server and download of a multimodal application and its descriptive tags that are generated in response to an interaction event produced by a user during the dialogue man-machine

b.

Sending by the browser of the labels that characterize the multimodal application to an interpreter residing in A network server.

C.

Interpretation of labels by the interpreter, who orders the execution of actions corresponding to the tags.

d.

Repeat steps a-c until that the user leave the application.

       \ vskip1.000000 \ baselineskip
    

In step a. events can be graphic and / or voice. It is advantageous to associate an external module to the browser with the function of transferring descriptive voice dialogue tags to the interpreter of these labels by means of an IP protocol. This interpreter, preferably, coordinates and controls all the events of voice and communicates with one or more servers that provide Voice resources using the MRCP protocol. Also preferably, analyze the structure of the multimodal application and send the corresponding commands to the voice server that complies with the protocol MRCP Optionally it communicates with the external module and it transfers the necessary data for it to establish a session via SIP with the MRCP voice server. The decoder can receive and send voice data to the MRCP server through the RTP protocol The external module preferably establishes a communication with an RTP client, thus obtaining the status of communication between the decoder and the voice server MRCP The decoder in turn has an application capable of collect data from any external device that collect audio data and be able to send it via a connection IP to voice servers. This application is able to compress said audio data to the format compatible with an MRCP server and send them through the RTP protocol to the voice server. From Preferably, the decoder has an application capable of collect audio data from the RTP channel, decompress them to the playable format by the decoder and send them to an existing electronic device in it responsible for Audio generation

Communication between the existing browser in the decoder and the external module is done through a Application programming interface.

Optionally, multimodal applications executed in the browser are preprocessed, separating the logic multimodal service logic before being shown to Username.

The use of multimodal interfaces in the interactive TV environment is not about replacing the use of the remote control (visual interaction) but complement it and improve it according to the needs and wishes of the user.

Brief description of the figures

In order to help a better understanding of the present description, according to a preferred example of practical embodiment of the invention, a figure is attached, whose character is illustrative and not limiting and that describes the system architecture (figure 1).

Detailed description of the invention

The system of the invention is capable of performing all multimodal interaction analysis processes in time real, using a distributed system of components through the communications protocols described in figure 1. The System power is based on the distributed architecture of components, delegating to servers and external machines those CPU intensive processes.

The system of the invention must have:

?

A decoder with a integrated web browser and a return channel that provides the ability to access an external server. The browser must allow the use and execution of a scriptable language, for example, JavaScript language

?

This decoder will connect to the TV to allow viewing of the graphic part of The multimodal application and play the voice messages.

?

The browser must allow the development and installation of plugins (or modules external to browser) to provide the browser, and therefore the decoder, the specific functionality for interpretation and Multimodal application execution.

       \ global \ parskip0.930000 \ baselineskip
    

?

One or more external servers where voice resources reside (synthesis and recognition of speaks).

?

A server capable of interpreting the labels that provide multimodality (such as SALT / VoiceXML). The system has an internal machine of states so that it is updated throughout the dialogue man-machine SALT / VoiceXML tags (or others similar) are distributed throughout the website of the application, providing features such as recognition or voice synthesis. The location of the same depends on the own Website design and next to the traditional labels of HTML or JS would make up what we call multimodal application. So for example there may be a <prompt> "Text" tag that what it produces in the end is an audio synthesized from the message that It appears after the label. Similarly, there are other <listen>, <reco> tags that allow you to record the User commands for later recognition. In any case, the syntax of the tags depends on the language itself or specification being used (SALT / VoiceXML or others).

?

Web server where the multimodal application.

?

Any external device, such as a microphone connected to the decoder, a mobile phone, a bluetooth hands-free device, etc. that allow to pick up the voice and send it in digital format to decoder

       \ vskip1.000000 \ baselineskip
    

The system allows user interaction With the application by using the remote control or voice. Both methods are complementary allowing the user to decide which one you want to use in each case. To this user interaction with the application it is called dialogue man-machine

The system also allows synchronization between the events generated by the user through any of the possible modes (text / voice) next to what was presented towards the user, solving those inconsistencies that may occur during the interaction This synchronization is carried out by a external module that runs in the browser preventing actions unwanted or that may cause adverse effects on the system.

In summary, the system performs the Next steps:

one.

The multimodal application resides on a Web server. In a first step, the user selects said application (purchase of content, electronic banking, etc.) provided by your provider of Digital TV services. Being an application multimodal, the service provider will inform the user, telling you to connect the microphone, phone mobile, bluetooth headset to decoder (the details of this in any case, they are outside the scope of the invention). Once the device is connected, the decoder is downloaded said application through an http protocol or other mechanism- al web browser resident in the decoder, a plugin is executed and The multimodal application is executed in the browser. The plugin in This moment is already linked to the browser.

2.

In order to perform time processing real, the browser plugin sends the web page containing the tags that provide multimodality to an interpreter (SALT / VoiceXML, for example) external.

3.

Immediately, the browser processes the web page and He presents it on the television.

Four.

Simultaneously, the multimodal interpreter (SALT / VoiceXML), recognizes and processes multimodal tags communicating to the voice servers the actions they must perform at each moment (for example, at a given moment in the dialogue man-machine with the interactive application, the multimodal label indicates that an audio has to be reproduced certain message that appears on the screen).

5.

The voice server sends the data to the decoder audio, which are processed and adapted, so that they are playable by the television's audio device and ears by the user.

6.

At this point, the status of the dialogue has advanced a step, and the browser plugin informs the SALT interpreter that it can process the following multimodal label, repeating this process until the user leaves the application.

       \ vskip1.000000 \ baselineskip
    

The multimodal application consists of a HTML document composed of two frames - existing element in the HTML terminology that corresponds to a part of a page main web-: a frame containing the application where the web content itself resides (application frame, which is what shows in the graphical interface and what the user definitely sees in your terminal) and another frame for creation at runtime -instance- of the external module or plugin (module frame external). The frames are used to separate the content from the application. The application frame contains the elements with the that the user is able to interact both graphically and vocally The external module or plugin is an application that relates to the application frame to provide functionality specific to multimodal logic, that is, the one that allows maintain a man-machine dialogue through commands vowels and in a complementary way to the traditional graphic interface together with the remote control during the entire execution period of the application.

       \ global \ parskip1.000000 \ baselineskip
    

The structure of a multimodal application resides in an XML type document formatted to meet the specifications defined in SALT (Speech Application Language Tags) or VoiceXML ( http://www.w3.org/Voice/ ).

The external frame is hidden from the user since it does not contain a graphical interface and is only responsible for grouping the SALT / VoiceXML tags, mentioned in the previous paragraph, specific to each application, and the instantiation of the external module. This set of SALT / VoiceXML tags determines multimodal logic, that is, the interactions allowed in the application. These tags allow you to configure the synthesis and execution of the voice as well as the voice recognizer and the set of events that can be performed using the vocal interface. For example, during the man-machine dialogue with the application, the user could see on his television an editing field that invites him to enter a text with the remote control; in parallel in the application there will be a SALT / VoiceXML tag that will indicate that the system is currently waiting for the user of a vocal command. The user can then choose to enter the data with the remote control, or give the voice command
equivalent.

For there to be a concordance between what is shows on the screen, in this case television, and interactions or events that are performed vocally, both frames communicate with each other using a programming interface applications or API (Application Programming Interface) through the browser architecture itself that runs on the decoder This API defines the set of functions and communication procedures between the two frames getting a level of abstraction and separation between them. The API is defined in javascript since it is compatible with the browser integrated in the decoder

This frame structure allows you to separate the Service logic of multimodal logic. Multimodal logic (provided by the hidden frame where the module is executed external or plugin) is associated with the interaction management man-machine from any of the interfaces graphic or vocal, that is to say, it is responsible for the management of events from the vocal or graphic interface by launching the actions Timely to these events. It will also be responsible for resolving problems that pose simultaneous interactions between both interfaces The service logic (provided by the frame of application) be associated with the achievement of the objective The user has when using the application, such as the purchase of a movie on a pay-per-view TVD service. Too allows the external module or plugin to keep active the service that provides multimodality when navigating between different pages and contents of the main application. This allows while loading a new graphical interface can be performed text conversions to synthetic voice or TTS (Text To Speech), or voice conversions to a system understandable format too called SR (Speech Recognition). This provides an improvement of the user experience avoiding any type of wait since the service that provides multimodality is always working in background.

The plugin that provides multimodality requires that communications with the user be based on events, that is, it requires that some action be taken before the which occurs some process. So for example, at a time of dialogue the user can press a remote control button with the remote control accept or send the vocal command "Accept" for your recognition; in both cases the same code will be generated execution, not distinguishing through what mechanism of Interaction has come the event. If it is through the interface graphically, the plugin will inform the SALT / VoiceXML interpreter of the pressing that button, so that your state machine is synchronized with the execution of the application, while will execute the code associated with the key press. If he user sends a voice command and it is recognized correctly, The SALT / VoiceXML interpreter will inform the plugin using a command in this way and it will generate the corresponding event that will cause the code corresponding to the press of that one is executed event.

To manage all the events produced both by the user and by the system it is used preferably the Javascript programming language that allows schedule event handlers, which are responsible for capturing the actions produced in the system.

The browser used in the Digital TV decoders interprets the Javascript code that It is integrated into Web pages. In addition, the logic itself multimodal (the allowed interactions), determined by the SALT / XML tags, is encoded in JavaScript. For all this, the multimodal external module requires JavaScript, in order to Communicate with the browser.

The multimodal external module also needs access the audio hardware resources of the decoder to control basic actions such as play, stop, etc.

In figure 1 you can see the architecture high level of an example of a system capable of carrying out the method of the invention:

?

Decoder [100]:

?

[110] Browser with possibility to include an external module [120].

?

Custom software able to allow the incorporation of modules that enable a IP connection with other systems and any other protocol communication.

?

SIP Client (Start Protocol of Sessions) [140]: responsible for creating a session between the decoder [100] and the MRCP server (Media Resource Control Protocol or Multimedia Resource Control Protocol) [460].

?

RTP Client (Real Time Protocol - Real Time Protocol) [170].

?

Audio Recording [190] and audio playback [180].

?

Network connection [500] IP.

?

External Servers:

?

MRCP server [350] and [460].

?

SALT / Voice XML interpreter [300].

?

Voice Resources [400] (TTS or Text to Speech, SP or Speech Recognition).

       \ vskip1.000000 \ baselineskip
    

The communication processes between the client (decoder) [100] and MRCP server resources [460] are performed through the SIP protocol [610] (Start Protocol of Sessions) that allows the establishment of multimedia sessions by exchanging messages between the parties they want communicate. The decoder [100] implements a SIP module [140] that creates a session [610] by sending requests to MRCP server In this message you also send the features of the session you want to establish as Supported audio encoders / decoders, addresses, ports where you expect to receive them, transmission speeds, etc. that are necessary to perform the synthesis processes and voice recognition All these actions are coordinated by the interpreter SALT [300].

In the decoder part [100] an RTP module [170] (Real Time Protocol) may be necessary since the decoder may not support streaming playback via RTP. Therefore, it is necessary to use a player [180] capable of sending raw voice data to the system speaker, collected and stored in real time in the RTP client buffer [170].

The process of sending the audio [620] through the RTP channel up to the RTP element [411] is based on the use of an application [190] capable of recording and collecting voice data in gross from the audio input device (for example, a microphone) to then create a buffer with this data. Depending on the possibilities of the voice servers and the formats that you want to support, would be necessary to use different voice compressors / decompressors, such as PCMU-PCM (Pulse Code Modulation mu-law - Pulse code modulation).

This compressor / decompressor application [180] [190] implements the client / server paradigm being the object of the transaction the voice data on an RTP protocol [620]. The Application consists of two main modules:

?

RTP Player [180]: decompress and convert sound in PCMU (or PCMA) format to PCM, which comes from the RTP channel, and finally plays the sound.

?

RTP Recorder [190]: Read data of the audio input device, compress / convert PCM to PCMU (or PCMA) and finally sends it through the RTP channel [620].

       \ vskip1.000000 \ baselineskip
    

All these components are coordinated to your once by the SALT interpreter [300]. The most common actions that are They perform with them are PLAY, STOP, PAUSE and REC.

The invention can be applied to almost all the interactive services that run on Digital TV decoders that meet the characteristics previously mentioned. As examples of services interactive:

?

Control and navigation through EPGs (Electronic Program Guide) or ESG (Electronic Service Guide) or UEG (Unified Electronic Guide).

?

Control and navigation through VoD (Video on Demand), CoD (Content on Demand).

?

Control and navigation through electronic banking services (home banking), purchase / sale electronic access to product catalogs, etc.

?

Control and navigation through the features offered by messaging applications electronics and Internet browsing through browsers

Claims (16)

         \ global \ parskip0.900000 \ baselineskip
      

1. Multimodal interaction method on interactive applications of Digital TV, where the television is provided with a network decoder (100) that incorporates an associated browser (110), characterized by the following steps:

to.

Connecting the browser to a network server and download of a multimodal application and its descriptive tags that are generated in response to an interaction event produced by a user during the dialogue man-machine

b.

Sending by the browser of the labels that characterize the multimodal application to an interpreter (300) that It resides on a network server.

C.

Interpretation of labels by the interpreter, who orders the execution of actions corresponding to the tags.

d.

Repeat steps a-c until that the user leave the application.

         \ vskip1.000000 \ baselineskip
      

2. Method according to claim 1 characterized in that in step a. The events are graphics and / or voice. 3. Method according to claim 2 characterized in that an external module (120) is associated with the browser (110) with the function of transferring the descriptive tags of the voice dialogue to the interpreter of said tags (300) by means of an IP protocol. Method according to claim 3 characterized in that the interpreter (300) of the descriptive labels of the voice dialogue coordinates and controls all the voice events. Method according to claim 4, characterized in that the interpreter (300) of the descriptive tags of the voice dialogue communicates with one or more servers that provide voice resources by means of the MRCP protocol. Method according to claim 5, characterized in that the interpreter (300) of the descriptive labels of the voice dialogue analyzes the structure of the multimodal application and sends the corresponding commands to the voice server that complies with the MRCP protocol. Method according to claim 6, characterized in that the interpreter (300) of the descriptive labels of the voice dialogue communicates with the external module (120) associated with the decoder's browser and transfers the necessary data for it to establish a session via SIP with the MRCP voice server (460). Method according to claim 7, characterized in that the decoder receives and sends the voice data to the MRCP server (460) by means of the RTP protocol. 9. Method according to claim 8, characterized in that the external module (120) associated with the browser establishes a communication with an RTP client (170) thereby obtaining the communication status between the decoder and the MRCP voice server (460) . 10. Method according to any of claims 5-9 characterized in that the decoder (100) has an application (190) capable of collecting data from any external device that collects audio data and is capable of sending it via an IP connection to Voice servers Method according to claim 10, characterized in that said application is capable of compressing said audio data to the format compatible with an MRCP server and sending it through the RTP protocol to the voice server (400). 12. Method according to claim 11, characterized in that the decoder has an application (180) capable of collecting audio data from the RTP channel, decompressing them to the format reproducible by the decoder and sending them to an existing electronic device in charge of the audio generation 13. Method according to any of claims 3-12 characterized in that the communication between the browser (110) existing in the decoder and the external module (120) is carried out through an application programming interface. 14. Method according to any of the preceding claims, characterized in that the multimodal applications executed in the browser (110) are preprocessed, separating the multimodal logic from the service logic before being shown to the user. 15. System capable of carrying out any of the methods of claims 1 to 14. 16. Use of the system of claim 15 in a pay-per-view digital television service.