ES2382747B1 - MULTIMODAL INTERACTION ON DIGITAL TELEVISION APPLICATIONS - Google Patents

Abstract

The invention proposes a method of multimodal interaction on digital television applications where the multimodal application resides on a web server and is downloaded by a browser (110) that resides on the television decoder itself (100). Using a distributed system of components and through communication protocols, all multimodal interaction analysis processes can be performed in real time. The system allows user interaction with the application through the use of the remote control or voice.

Description

Multimodal interaction on digital television applications.

Field of the Invention

The present invention applies to the digital television sector, more specifically to the field of human-machine interactions on terminals such as digital television decoders or mobile phones capable of executing interactive applications that are displayed on a television.

Background of the invention

A multimodal system must simultaneously allow different input methods or mechanisms (keyboard, voice, images, etc.), collecting the information of each of them as needed, for example, sometimes, the user could say something through a command vowel, but other times you could select a name from a list by using the keyboard and you could even select a menu or a part of the screen pointing with your own finger making the multimodal interface engine able to detect the interaction method that the user has freely chosen (discarding incongruous information received through the other methods).

With regard to the design of user interfaces, these have traditionally been based on the desktop metaphor, developed decades ago in the Xeros laboratories, and which attempts to transfer all the objects and tasks normally performed in a real office to the world. of computers; Thus, for example, both real and electronic files can be stored, the traditional typewriter has its equivalent in the word processor, the blank sheet is equivalent to the blank document of the processor, etc. In this way it is achieved that the mental model that the user has when performing these traditional tasks is maintained with few changes when moving to the field of computers, that is, trying to achieve the highest degree of familiarity between objects and actions. This desktop metaphor has been implemented through the WIPM paradigm (in English, Windows, Icons, Pointers and Menus), which are the main elements that support the vast majority of current graphic interfaces.

However, this paradigm is clearly inappropriate in an interactive Digital TV environment for several reasons. The first one is related to the very nature of the tasks that a user performs on an interactive application (more relaxed and close to an entertainment, social, etc. environment) which make them very different from those of a real office . As a second point it should be noted that the device with which the user interacts (remote control) is very different in functionality and accessibility to the keyboard and mouse, which imposes many restrictions when performing tasks on a Digital TV environment (For example, entering text through the remote control to perform a simple search can become an expensive task). For many years, and since its inception, the remote control used in the TV environment has become the device par excellence and through it it has been possible to control a wide variety of devices and functions associated with them. However, the task models used in any of the interactive services currently deployed on a commercial level on any of the distribution technologies and their development environments, make their use inefficient in many cases, presenting great usability problems. which translates into a demotivation and loss of interest in the exploration by users (usability is defined as the efficiency and satisfaction with which a product allows to achieve specific objectives, such as the purchase of a football match, users specifics, such as television viewers, in a context of specific use, such as the living room of a dwelling).

If we also take into account that many people have accessibility problems when using a traditional remote control, we can conclude that clearly the mechanism of traditional interaction with television has become outdated and surpassed by the new interactive services executed on digital television decoders. .

Tasks such as entering text with the remote control when performing a search on an EPG (Electronic Programming Guide or Electronic Programming Guide) or the possibility of sending a message through an interactive TV application, can be converted in an expensive task that will finally cause the user to lose interest in its use. When entering this data, a virtual keyboard that appears on the screen and that can look similar to the keyboard of a mobile phone or that of an ANSI keyboard is usually used. In any case the process is slow, not everyone is accustomed to using the remote control as if it were a keyboard of a mobile phone and also the errors that are made when using this mechanism (the remote control are not uncommon) it works by infrared that depending on the light of the environment, objects located between the user and the receiver, etc. can make the keystroke not translate into an introduction of characters). Almost all studies and usability tests carried out on interactive applications point to this process as expensive for the user.

It should also be noted that the TV has a much more social character, and the user is usually in a much more relaxed environment, sitting 3-4 meters from the TV, and with an attitude of much less concentration than that required to work with a computer. It is clear that many of the tasks performed on a computer through a traditional graphical interface cannot be performed or will have to be performed in a very different way.

All of the above has meant that this desktop metaphor has necessarily been abandoned in the development of Digital TV applications.

In addition, interactive applications on Digital TV, in addition, run on a single window presented simultaneously (instead of several such as the graphical PC interfaces, for example) for all the restrictions indicated above. On this window the different multimedia objects that make up the scene are arranged (texts, graphics, videos, etc.) trying to ensure that all of them are synchronized based on a timeline generating a set of scenes that describe the different actions or steps that should be complete the user until his goal is achieved. For example, when purchasing a movie from an interactive video on demand system, the user must initially enter that section, perform a content search based on some criteria, enter the data, select the content, enter a PIN purchase, etc. In a synchronized way the different objects appear in the scene as the user interacts with them.

Although this concept based on the description of scenes and the simultaneous presentation of objects (videos, texts, graphics) may seem simple, it is costly from a graphic processing point of view and it is especially accentuated on those devices - such as TV decoders - where Business models, for cost reasons, impose significant restrictions on the electronic components that constitute the device. However, technologies and mechanisms that support this paradigm at the presentation layer level already exist in the industry.

If we wanted to transfer the concept of synchronization of objects in the presentation layer to the synchronization of the different interaction mechanisms (remote control, voice command interaction, etc.) we would see that architectures have hardly been developed on Digital TV environments that support this synchronization of interaction mechanisms. For example, we might want, by returning to the interactive application of the purchase of a video movie on demand, to search for the content by means of a vocal command that is given to the system, but then enter -for privacy reasons-the purchase PIN by traditional remote control. Simultaneous management of the different interaction mechanisms is complex from a semantic point of view (for example, when opposite and simultaneous orders are given through a vocal and graphic interface) and expensive in computational processing resources. If we add to this the fact that multimodal interactive applications will run on devices with low processing capacity (100 MHz CPUs and 32-64 MB limited RAM and much lower than the performance of any home PC, 1 GHz CPUs and 1 GB RAM) and on which - for example, digital television decoders - it is impossible to perform voice processing in real time, it can be concluded that there are important technical difficulties when offering multimodal interfaces over Digital TV environments.

There are different patents dealing with multimodal interfaces, although none do so specifically applied to the field of digital television.

US5265014 is focused on how to solve ambiguities that occur when using natural language as an interaction mechanism. US4829423 is similar to the previous one although it is more focused on how to solve the problems that occur when using natural language in a multimodal environment. The US6345111 patent is more related to an image analysis mechanism so that the system is able to recognize on which object the user has the view, so that it serves as an input mechanism in the selection of elements. In US5577165 it is described in a general way how the mechanism of dialogue between a machine and a person is performed, taking into account the keywords detected during the recognition process, as well as the different states through which the system goes through during the dialogue.

Trying to implement a complete architecture for the use of multimodal interfaces on light devices, such as digital TV decoders or mobile phones, presents performance problems (mainly related to real-time voice processing or the incorporation of a multimodal interpreter, for example SALT or VoiceXML, within the device itself that is also expensive in terms of processing and memory reservation), so it is necessary to address new architectures that allow to achieve response times in the interaction process using any of the mechanisms (visual or vocal commands) are the minimum possible as they already occur in other more powerful devices such as PCs, where all voice processing (synthesis and recognition) is performed on the machine itself, without requiring external processing. For example, the CPU and RAM memory needed to perform a real-time voice recognition process with acceptable response times (less than 5 seconds) and allowing the user not to lose attention to the system would imply the use of a current medium-type PC (512 RAM, 1 GHz CPU), which in any case gave much of the processing capabilities of digital television decoders (32 MB RAM, 100 MHz), even the range higher (128 MB RAM, 200 MHz CPU).

In summary, the technical problem is that the limited processing capacity of digital television decoders prevents them from developing authentic multimodal applications that use simultaneous interaction as vocal mechanisms in addition to the visual one.

Object of the invention

The object of the present invention is to create a method, platform or system that allows different interaction mechanisms to coexist simultaneously and in a synchronized manner, thus producing what are known as multimodal interfaces. For this, the invention proposes a multimodal interaction method on interactive applications of Digital TV, where the television is provided with a network decoder incorporating an associated browser, and where the method is mainly composed of the following steps:

to.

Connecting the browser to a network server and downloading a multimodal application and its descriptive tags that are generated in response to an interaction event produced by a user during the man-machine dialogue.

b.

Sending by the browser of the labels that characterize the multimodal application to an interpreter residing in a network server.

C.

Interpretation of the labels by the interpreter, which orders the execution of actions corresponding to the labels.

d.

Repeat steps a-c until the user exits the application.

In step a. the events can be graphics and / or voice. It is advantageous to associate an external module to the browser with the function of transferring the descriptive tags of the voice dialogue to the interpreter of said tags by means of an IP protocol. This interpreter preferably coordinates and controls all voice events and communicates with one or more servers that provide voice resources through the MRCP protocol. Also preferably, it analyzes the structure of the multimodal application and sends the corresponding commands to the voice server that complies with the MRCP protocol. Optionally, it communicates with the external module and transfers the necessary data for it to establish a session via SIP with the MRCP voice server. The decoder can receive and send the voice data to the MRCP server using the RTP protocol. The external module preferably establishes a communication with an RTP client, thus obtaining the communication status between the decoder and the MRCP voice server. The decoder in turn has an application capable of collecting data from any external device that collects audio data and is capable of sending it via an IP connection to the voice servers. Said application is capable of compressing said audio data to the format compatible with an MRCP server and sending it through the RTP protocol to the voice server. Preferably, the decoder has an application capable of collecting audio data from the RTP channel, decompressing it to the format reproducible by the decoder and sending it to an existing electronic device in charge of generating audio.

Communication between the existing browser in the decoder and the external module is done through an application programming interface.

Optionally, multimodal applications executed in the browser are preprocessed, separating multimodal logic from service logic before being shown to the user.

The use of multimodal interfaces in the interactive TV environment is not about replacing the use of the remote control (visual interaction) but rather complementing and improving it according to the needs and wishes of the user.

Brief description of the fi gures

In order to help a better understanding of the present description, according to a preferred example of practical embodiment of the invention, a figure is attached, the character of which is illustrative and not limiting and which describes the architecture of the system (Figure 1).

Detailed description of the invention

The system of the invention is capable of performing all multimodal interaction analysis processes in real time, using a distributed system of components through the communication protocols described in Figure 1. The power of the system is based on the distributed architecture. of components, delegating to processes and external machines those processes of intensive use of the CPU.

The system of the invention must have:

•

A decoder with an integrated web browser and a return channel that gives you the ability to access an external server. The browser must allow the use and execution of a scriptable language, for example, the JavaScript language.

•

This decoder will be connected to the TV to allow viewing of the graphic part of the multimodal application and play the voice messages.

•

The browser must allow the development and installation of plugins (or modules external to the browser) that provide the browser, and therefore the decoder, the specific functionality for the interpretation and execution of multimodal applications.

•

One or more external servers where voice resources reside (speech synthesis and recognition).

•

A server capable of interpreting the tags that provide multimodality (such as SALT / VoiceXML). The system has an internal state machine so that it is updated throughout the man-machine dialogue. SALT / VoiceXML (or similar) tags are distributed throughout the application's website, providing features such as speech recognition or speech synthesis. The location of the same depends on the design of the web page itself and together with the traditional HTML or JS tags would form what we call multimodal application. Thus, for example, there may be a <prompt> "Text" tag that produces a synthesized audio of the message that appears after the tag at the end. Similarly, there are other <listen>, <reco> tags that allow you to record user commands for later recognition. In any case, the syntax of the tags depends on the language or speci fi cation being used (SALT / VoiceXML or others).

•

Web server where the multimodal application resides.

•

Any external device, such as a microphone connected to the decoder, a mobile phone, a Bluetooth hands-free device, etc. that allows to pick up the voice and send it in digital format to the decoder.

The system allows user interaction with the application through the use of the remote control or voice. Both methods are complementary allowing the user to decide which one they want to use in each case. This user interaction with the application is called the man-machine dialogue.

The system also allows synchronization between the events generated by the user through any of the possible modes (text / voice) along with the one presented to the user, resolving those inconsistencies that may occur during the interaction. This synchronization is carried out by an external module that runs in the browser preventing unwanted actions or that may cause adverse effects on the system.

In summary, the system performs the following steps:

one.

The multimodal application resides on a Web server. In a first step, the user selects said application (content purchase, electronic banking, etc.) provided by his Digital TV service provider. As it is a multimodal application, the service provider will inform the user of this fact, instructing him to previously connect the microphone, mobile phone, bluetooth headset to the decoder (the details of this step in any case, are beyond the scope of the invention ). Once the device is connected, the decoder downloads said application through an http protocol or other mechanism-to the web browser resident in the decoder, a plugin is executed and the multimodal application is executed in the browser. The plugin is currently linked to the browser.

2.

In order to perform real-time processing, the browser plugin sends the web page containing the tags that provide multimodality to an external interpreter (SALT / VoiceXML, for example).

3.

Immediately, the browser processes the web page and presents it on the TV.

Four.

Simultaneously, the multimodal interpreter (SALT / VoiceXML), recognizes and processes multimodal labels communicating to the voice servers the actions that they must perform at each moment (for example, at a given moment of the man-machine dialogue with the interactive application , the multimodal label indicates that a certain message that appears on the screen must be reproduced by audio).

5.

The voice server sends the audio data to the decoder, which is processed and adapted, so that it is reproducible by the television's audio device and heard by the user.

6.

At this point, the status of the dialogue has advanced one step, and the browser plugin informs the SALT interpreter that it can process the next multimodal tag, repeating this process until the user leaves the application.

The multimodal application is composed of an HTML document composed of two frames - an element existing in the HTML terminology that corresponds to a part of a web-main page: a frame that contains the application where the web content itself resides (application frame, which is what is shown in the graphical interface and what the user de fi nitively sees in his terminal) and another frame for creation at runtime - instantiation of the external module or plugin (external module frame). The frames are used to separate the content of the application. The application frame contains the elements with which the user is able to interact both graphically and vocally. The external module or plugin is an application that is related to the application frame to provide the specific functionality of multimodal logic, that is, the one that allows to maintain a man-machine dialogue by means of vocal commands and in a complementary way to the traditional graphical interface together with the remote control during the entire period of application execution.

The structure of a multimodal application resides in an XML type document formatted to meet the specifications defined in SALT (Speech Application Language Tags) or VoiceXML (http://www.w3.org/Voice/).

The external frame is hidden from the user since it does not contain a graphical interface and is only responsible for grouping the SALT / VoiceXML tags, mentioned in the previous paragraph, specifics of each application, and the instantiation of the external module. This set of SALT / VoiceXML tags determines multimodal logic, that is, the interactions allowed in the application. These tags allow you to configure the synthesis and execution of the voice as well as the voice recognizer and the set of events that can be performed using the vocal interface. For example, during the man-machine dialogue with the application, the user could see on his television an editing field that invites him to enter a text with the remote control; in parallel in the application there will be a SALT / VoiceXML tag that will indicate that the system is currently waiting for the user of a vocal command. The user can then choose to enter the data with the remote control, or give the equivalent vocal order.

So that there is a concordance between what is shown on the screen, in this case television, and the interactions

or events that are performed vocally, both frames communicate with each other using an application programming interface or API (Application Programming Interface) through the browser architecture itself that runs in the decoder. This API defines the set of functions and communication procedures between the two frames, achieving a level of abstraction and separation between them. The API is defined in javascript since it is compatible with the browser integrated in the decoder.

This frame structure allows to separate the service logic from the multimodal logic. Multimodal logic (provided by the hidden frame where the external module or plugin is executed) is associated with the management of human-machine interaction from any of the graphic or vocal interfaces, that is, it is responsible for managing events from the vocal or graphic interface launching the appropriate actions before these events. It is also responsible for solving problems that pose simultaneous interactions between both interfaces. The service logic (provided by the application frame) will be associated with the achievement in itself of the objective that the user has when using the application, such as the purchase of a movie in a pay-per-view TVD service. It also allows the external module or plugin to keep active the service provided by multimodality when browsing between the different pages and contents of the main application. This allows that while loading a new graphical interface, text conversions to synthetic voice or TTS (Text To Speech), or voice conversions to a format understandable by the system also called SR (Speech Recognition) can be performed. This provides an improvement of the user experience avoiding any type of waiting since the service provided by multimodality is always running in the background.

The plugin that provides multimodality requires that communications with the user are based on events, that is, it requires that some action be taken before any process occurs. For example, at a time during the dialogue the user can press an accept button with the remote control or send the “Accept” vocal command for recognition; In both cases, the same execution code will be generated, not distinguishing through which interaction mechanism the event has arrived. If it is through the graphical interface, the plugin will inform the interpreter SALT / VoiceXML of the press of that button, so that its state machine is synchronized with the execution of the application, at the same time as the code associated with the key press. If the user sends a voice command and it is correctly recognized, the SALT / VoiceXML interpreter will inform the plugin of this fact by means of a command and will generate the corresponding event that will cause the code corresponding to the pulsation of that event to be executed.

To manage all the events produced by both the user and the system, the Javascript programming language is used, which allows programming event handlers, which are responsible for capturing the actions produced in the system.

The browser used in Digital TV decoders interprets the Javascript code that is integrated into the Web pages. In addition, the multimodal logic itself (the allowed interactions), determined by the SALT / XML tags, is coded in JavaScript. For all this, the multimodal external module requires JavaScript, in order to communicate with the browser.

The multimodal external module also needs to access the audio hardware resources of the decoder to control basic actions such as play, stop, etc.

In Figure 1 the high-level architecture of an example of a system capable of carrying out the method of the invention can be seen:

• Decoder [100]:

•

[110] Browser with the possibility of including an external module [120].

•

Customized software capable of allowing the incorporation of modules that enable an IP connection with other systems and any other communication protocol.

•

SIP Client (Session Initiation Protocol) [140]: responsible for creating a session between the decoder [100] and the MRCP (Media Resource Control Protocol) [460].

•

RTP Client (Real Time Protocol - Real Time Protocol) [170].

•

Audio Recording [190] and audio playback [180].

•

Network connection [500] IP.

•

External Servers:

•

MRCP server [350] and [460].

•

SALT / Voice XML interpreter [300].

•

Voice resources [400] (TTS or Text to Speech, SP or Speech Recognition).

The communication processes between the client (decoder) [100] and the resources of the MRCP server [460] are carried out through the SIP protocol [610] (Session Initiation Protocol) that allows the establishment of multimedia sessions through the exchange of messages between the parties that want to communicate. The decoder [100] implements a SIP module [140] that creates a session [610] by sending requests to the MRCP server. This message also sends you the characteristics of the session that you want to establish as supported audio encoders / decoders, addresses, ports where you expect to receive them, transmission speeds, etc. that are necessary to carry out the processes of synthesis and recognition of the voice. All these actions are coordinated by the SALT interpreter [300].

In the decoder part [100] an RTP module [170] (Real Time Protocol-Real Time Protocol) may be necessary since the decoder may not support streaming playback via RTP. Therefore, it is necessary to use a player [180] capable of sending raw voice data to the system speaker, collected and stored in real time in the RTP client buffer [170].

The process of sending the audio [620] through the RTP channel to the RTP element [411] is based on the use of an application [190] capable of recording and collecting raw voice data from the audio input device (for example, a microphone) and then create a buffer with this data. Depending on the possibilities of the voice servers and the formats that you want to support, it would be necessary to use different voice compressors / decompressors, such as PCMU-PCM (Pulse Code Modulation mu-law - Pulse code modulation).

This compressor / decompressor application [180] [190] implements the client / server paradigm being the object of the transaction voice data on an RTP protocol [620]. The application consists of two main modules:

•

RTP Player [180]: decompresses and converts sound in PCMU (or PCMA) format to PCM, which comes from the RTP channel, and finally plays the sound.

•

RTP Recorder [190]: reads data from the audio input device, compresses / converts PCM to PCMU (or PCMA) and finally sends it through the RTP channel [620].

All these components are coordinated in turn by the SALT interpreter [300]. The most common actions performed with them are PLAY, STOP, PAUSE and REC.

The invention can be applied to almost all of the interactive services that run on Digital TV decoders that meet the aforementioned characteristics. As examples of interactive services:

•

Control and navigation through EPGs (Electronic Program Guide) or ESG (Electronic Service Guide) or UEG (Uni fi ed Electronic Guide).

•

Control and navigation through the features of VoD (Video on Demand), CoD (Content on Demand).

•

Control and navigation through electronic banking services (home banking), electronic purchase / sale access to product catalogs, etc.

•

Control and navigation through the functionalities offered by electronic messaging applications and Internet browsing through browsers.

Claims (17)

1. Multimodal interaction method on interactive applications of Digital TV, where television is provided with a network decoder (100) that incorporates an associated browser (110), characterized by the following steps: Connecting the browser to a network server and downloading a multimodal application and its tags a. descriptions that are generated in response to an interaction event produced by a user during the man-machine dialogue. b. Sending by the browser of the labels that characterize the multimodal application to an interpreter (300) that resides on a network server. Interpretation of the labels by the interpreter, who orders the execution of corresponding actions. Teeth to tags. d. Repeat steps a-c until the user exits the application. 2. Method according to claim 1 characterized in that in step a. The events are graphic and / or voice.

3.

Method according to claim 2 characterized in that an external module (120) is associated with the browser (110) with the function of transferring the descriptive tags of the voice dialogue to the interpreter of said tags (300) by means of an IP protocol.

Four.

Method according to claim 3 characterized in that the interpreter (300) of the descriptive labels of the voice dialogue coordinates and controls all the voice events.

5.

Method according to claim 4, characterized in that the interpreter (300) of the descriptive labels of the voice dialogue communicates with one or more servers that provide voice resources by means of the MRCP protocol.

6.

Method according to claim 5 characterized in that the interpreter (300) of the descriptive tags of the voice dialogue analyzes the structure of the multimodal application and sends the corresponding commands to the voice server that complies with the MRCP protocol.

7.

Method according to claim 6, characterized in that the interpreter (300) of the descriptive labels of the voice dialogue communicates with the external module (120) associated with the decoder navigator and transfers the necessary data for it to establish a session via SIP with the MRCP voice server (460).

8.

Method according to claim 7, characterized in that the decoder receives and sends the voice data to the MRCP server (460) by means of the RTP protocol.

9.

Method according to claim 8, characterized in that the external module (120) associated with the browser establishes a communication with an RTP client (170) thereby obtaining the communication status between the decoder and the MRCP voice server (460).

10.

Method according to any of claims 5-9 characterized in that the decoder (100) has an application (190) capable of collecting data from any external device that collects audio data and is capable of sending it via an IP connection to the servers voice.

eleven.

Method according to claim 10 characterized in that said application is capable of compressing said audio data to the format compatible with an MRCP server and sending it through the RTP protocol to the voice server (400).

12.

Method according to claim 11, characterized in that the decoder has an application (180) capable of collecting audio data from the RTP channel, decompressing them to the format reproducible by the decoder and sending them to an existing electronic device in charge of generating Audio.

13.

Method according to any of claims 3-12 characterized in that the communication between the navigator (110) existing in the decoder and the external module (120) is carried out through an application programming interface.

14.

Method according to any of the preceding claims, characterized in that the multimodal applications executed in the browser (110) are preprocessed, separating the multimodal logic from the service logic before being shown to the user.

fifteen.

System capable of carrying out any of the methods of claims 1 to 14.

16.

Use of the system of claim 15 in a pay-per-view digital television service.

SPANISH OFFICE OF THE PATENTS AND BRAND Application no .: 200930385 SPAIN Date of submission of the application: 06.30.2009 Priority Date: REPORT ON THE STATE OF THE TECHNIQUE 51 Int. Cl.: See Additional Sheet RELEVANT DOCUMENTS

Category

56 Documents cited Claims Affected

X X

US 2008255850 A1 (CROSS ET AL.) 10/16/2008, the whole document. EP 1455282 A1 (CIT ALCATEL) 08/09/2004, the whole document. 1-16 1-4, 15, 16

Category of the documents cited X: of particular relevance Y: of particular relevance combined with other / s of the same category A: reflects the state of the art O: refers to unwritten disclosure P: published between the priority date and the date of priority submission of the application E: previous document, but published after the date of submission of the application

This report has been prepared • for all claims • for claims no:

Date of realization of the report 14.05.2012

Examiner J. Maldonado Bottle Page 1/4

REPORT OF THE STATE OF THE TECHNIQUE Application number: 200930385 CLASSIFICATION OBJECT OF THE APPLICATION G06F3 / 16 (2006.01) G06F17 / 30 (2006.01) G10L15 / 00 (2006.01) Minimum documentation sought (classification system followed by classification symbols) G06F, G10L Electronic databases consulted during the search (name of the database and, if possible, terms of search used) INVENES, EPODOC, WPI, NPL, XPESP, XPAIP, XPI3E, INSPEC. State of the Art Report Page 2/4  WRITTEN OPINION Application number: 200930385 Date of Written Opinion: 05.05.2012 Statement

Novelty (Art. 6.1 LP 11/1986)

Claims Claims 1-16 IF NOT

Inventive activity (Art. 8.1 LP11 / 1986)

Claims Claims 1-16 IF NOT

The application is considered to comply with the industrial application requirement. This requirement was evaluated during the formal and technical examination phase of the application (Article 31.2 Law 11/1986).  Opinion Base.- This opinion has been made on the basis of the patent application as published. State of the Art Report Page 3/4  WRITTEN OPINION Application number: 200930385 1. Documents considered.- The documents belonging to the state of the art taken into consideration for the realization of this opinion are listed below.

Document

Publication or Identification Number publication date

D01

US 2008255850 A1 (CROSS et al.) 16.10.2008

D02

EP 1455282 A1 (CIT ALCATEL) 08.09.2004

2. Statement motivated according to articles 29.6 and 29.7 of the Regulations for the execution of Law 11/1986, of March 20, on Patents on novelty and inventive activity; quotes and explanations in support of this statement Document D01 presents a system and method of interaction of a user with a multimodal application operating with a multimodal navigator in a multimodal device that supports multiple modes of interaction including voice and other different modes. The system ([paragraph 0064]) incorporates network servers to provide users, through protocols to use (HTTP, HDTP, WAP or similar), of XHTML + voice documents that are interpreted in the multimodal equipment by means of an XMLX voice interpreter or Well on a remote voice server. In one of the embodiments (paragraphs [0078] to paragraph [0083]), the multimodal device and a voice server are connected by a VOIP protocol and present a voice interface to the user that digitizes and encodes the outgoing voice and through the browser and a voice module presents the data in RTP protocol format, the process is the reverse for incoming data. An example of a multimodal device that includes a video bus and a video adapter is presented in paragraph [0090] of the document. Document D02 presents an operating software for a multimodal device with voice interaction through a browser that allows you to interact with applications that use XHTML and voiceXML. We consider that the object of the invention set forth in the claims from 1 to 16 derives directly and without any ambiguity from document D01. The claims from 1st to 4th, 15th and 16th are also set out in document D02. Therefore, the claims from 1 to 16 are not new in view of the state of the known art or present inventive activity. State of the Art Report Page 4/4