KR100826778B1 - Wireless mobile for multimodal based on browser, system for generating function of multimodal based on mobil wap browser and method thereof - Google Patents

Abstract

Disclosed are a browser-based wireless terminal for a multimodal, a browser-based multimodal server and a system for a wireless terminal, and a method of operating the same.

The system according to the present invention is a system for providing a multimodal function for voice and text processing based on a terminal WAP browser, and a multimodal plug-in on a WAP browser. A wireless terminal for receiving a corresponding content by transmitting a voice or keyboard information to the wireless Internet based on a multi-modal plug-in; After recognizing the multi-modal plug-in request, perform data conversion on voice and keyboard information, generate information recognition result based on data conversion, and output visual data of the corresponding content corresponding to the information recognition result on voice and keyboard. A multimodal platform for processing the information into a wireless terminal for processing; And a content server for extracting content for information recognition results and providing the same to a multimodal platform. Therefore, the present invention has an effect that the voice recognition search service can be provided, including the effect that the operation of the mobile device is convenient through two input methods (key input + voice input).

Wap, WAP, Browser, Multi-Modal, Plug-in, Voice Recording, Voice Recognition, Voice Output

Description

WIRELESS MOBILE FOR MULTIMODAL BASED ON BROWSER, SYSTEM FOR GENERATING FUNCTION OF MULTIMODAL BASED ON MOBIL WAP BROWSER AND METHOD THEREOF }

1 is a configuration diagram for explaining a conventional multimodal (Multimodal).

2 is a block diagram for explaining the multi-modal outline according to the present invention.

3 is a swap browser of a wireless terminal for explaining voice recording among multi-modal functions according to the present invention.

4 is a block diagram for explaining a voice recording plug-in according to the present invention.

5 is a swap browser of a wireless terminal for explaining voice recognition among multi-modal functions according to the present invention.

6 is a block diagram for explaining a voice recognition plug-in according to the present invention.

7 is a swap browser of a wireless terminal for explaining a voice output of the multi-modal function according to the present invention.

8 is a configuration diagram illustrating a voice output plug-in according to the present invention.

<Explanation of symbols for main drawings>

201: wireless terminal 203: multi-modal platform

205 content server 601 voice recognition server (ASR)

801: voice conversion server (TTS)

The present invention relates to a multimodal interface for a wireless Internet service, and more particularly, a part of a code written in a terminal program (WIPI) using a plug-in method (WAP) browser. The present invention relates to a browser-based wireless terminal for multimodality that can coexist with the contents of a wireless markup language (WML), a browser-based multimodal server and system for a wireless terminal, and a method of operating the same.

Currently, the keyboard and mouse are mainly used for the interface between humans and computers, but the more natural method for humans is to use the voices already used between humans. Although this method has already been tried, the computer does not understand the contents of the input voice differently from the conversation between people, but instead of the spoken contents, it is converted into text and the contents are reacted as if they were input by the keyboard. Technology is not yet in use. The reason for this is that voice understanding technology, like artificial intelligence, requires the intelligence of a computer. Currently, technology is impossible in general fields and only in extremely limited fields such as travel planning.

However, in recent years, various technologies combining a wireless terminal and a voice recognition module have been provided, and among them, a multimodal technology for providing various interfaces between a user and the wireless terminal has emerged. That is, the existing wireless Internet service provides one input / output, a screen output and a key input, but provides a single integrated interface by adding a voice input and output and synchronizing it with the screen input and output.

Referring to the presently disclosed multi-modal based on the accompanying drawings as follows. First, as shown in FIG. 1, using the task-specific template library 111, the task-specific template library 111, an MXML editor 112 that allows a user to create an MXML document, and stores the created MXML document. A multimodal Internet server 110 including an MXML document server for providing an MXML document at the request of a user; It is connected to the multi-modal Internet server 110 through the Internet, and interprets an MXML document to display HTML content on the screen, and a message to be synthesized by interpreting the voice XML content by using a speech synthesis engine to synthesize the message through a speaker or a telephone interface. The MXML browser 121 and the MXML browser 121 which prepare a speech recognition engine using a language model for speech recognition, recognize the contents of the speech spoken by the user, and perform the operations specified in the speech XML. TAPI (Telephony Application Programming Interface: 125) / which provides an interface when accessing a multimodal Internet client using a phone 130 and a speech recognition / synthesis engine 122 that performs speech recognition or speech synthesis according to the present invention. MTAPI (Multimedia Telephony Application Programming Interface: 126) and I / O (Input / Output) devices such as keyboards, mice, monitors, microphones, speakers, etc. 124, comprises a TAPI (125) and MTAPI (126) and multi-modal Internet client 120 including the I / O interface 123 that provides an interface between the MXML browser 121.

On the other hand, the multi-modal Internet server 110 serves as an existing web server serves to provide MXML document on request. The multi-modal Internet server 110 stores an MXML document, which is created using the MXML editor 112 and the task-specific template library 111. Unlike regular HTML, MXML needs to create a language model for speech recognition, which requires specialized knowledge of language processing, making it difficult for general HTML authors to write.

In addition, the multi-modal Internet client 120 is a terminal having a CPU, such as a conventional personal computer / work station (PC / WS). MXML browser 121, speech recognition / synthesis engine 122, I / O devices (keyboard, mouse, monitor, microphone, speaker, etc., 124) for interpreting and displaying MXML on screen, TAPI, MTAP, and MXML browser (121). ) Has an interface module, that is, an I / O interface 123.

The multi-modal function configured as described above can conveniently browse the Internet by adding a voice that is convenient to a person as an interface in addition to the interface method such as a keyboard, a mouse, and a monitor in the Internet browsing. In addition, it is possible to adopt a free form of conversational method used in daily life that is realized through meta grammar function, instead of using the existing fixed expression only.

However, as described above, the addition of a multimodal function to a terminal application, for example, WIPI, is based on a server flat that processes voice input and output and a terminal application that can interoperate with it. Therefore, there is a problem in that a terminal application must be developed separately, which leads to a very slow spreading speed.

The present invention was created to solve the above problems, and an object of the present invention is to add a multi-modal function to a WAP browser and provide a convenient input / output service to a user using a wireless Internet service regardless of the type of terminal. A browser-based wireless terminal for a modal, a browser-based multi-modal server and system for a wireless terminal and its operation method.

It is another object of the present invention to provide a form in which a portion of code written in WIPI coexists with WML contents of a WAP browser by using a plug-in method, thereby providing a terminal application-based service. The present invention provides a browser-based wireless terminal for multi-modality, a browser-based multi-modal server and system for wireless terminal, and an operation method thereof.

It is still another object of the present invention to provide a terminal browser-based multi-modal function by subdividing a plug-in method into a component of voice recording, voice recognition, and voice output. The present invention provides a browser-based wireless terminal for a multi-modal that can be easily applied to the modal function, a browser-based multi-modal server and system for the wireless terminal, and an operation method thereof.

A browser-based multimodal server for a wireless terminal according to the first aspect of the present invention for achieving the above object, to a service server for providing a multi-modal function for voice and text processing in a terminal swap (WAP) browser-based Recognizing a multimodal plug-in connection state of a wireless terminal, performing data conversion on voice or keyboard information received on the wireless Internet, generating an information recognition result based on the data conversion, and recognizing the information. And receiving a corresponding content corresponding to the result, and processing the visual data of the content into voice and keyboard information and transmitting the processed data to the wireless terminal.

According to a preferred embodiment of the present invention, the multi-modal plug-in may include one or more of a plug-in for voice recording, a plug-in for voice recognition, and a plug-in for voice output. .

In addition, the multi-modal server, in response to a voice recognition command requested from the wireless terminal for the voice recognition plug-in activation, characterized in that for performing a voice recognition process according to a predetermined voice recognition algorithm.

In addition, the multi-modal server, characterized in that for reproducing the voice output plug-in to reproduce the output or recorded information from the wireless terminal.

In addition, the multi-modal platform, it characterized in that the interworking with the TTS server for the reproduction output of the textified information.

In accordance with a second aspect of the present invention, a browser-based wireless terminal for multi-modality provides a wireless terminal for providing a multi-modal function for voice and text processing based on a terminal swap (WAP) browser. Requests a multimodal plug-in on a WAP browser, transmits voice or keyboard information to the wireless Internet based on the multi-modal plug-in, and then transmits the voice or keyboard information to the voice or keyboard information. And a plug-in operation mode for receiving corresponding corresponding content.

According to an embodiment of the present invention, the plug-in may perform a plug-in request using an object tag on a WAP browser of the wireless terminal.

The object tag may be included in a WML document when the wireless terminal transmits a message to a multimodal platform.

A system for providing a browser-based multimodal function for a wireless terminal according to a third aspect of the present invention for achieving the above object is a system for providing a multimodal function for voice and text processing in a terminal swap (WAP) browser-based. A wireless terminal for requesting a multimodal plug-in on a WAP browser and transmitting voice or keyboard information over the wireless Internet based on the multimodal plug-in to receive the corresponding content. ; After recognizing the multi-modal plug-in request, data conversion for the voice and keyboard information is performed, an information recognition result is generated based on the data conversion, and visual data of the corresponding content corresponding to the information recognition result. A multi-modal platform for processing a voice and keyboard information and transmitting the same to the wireless terminal; And a content server for extracting content for the information recognition result and providing the same to the multi-modal platform.

On the other hand, the voice recording method of the browser-based multi-modal function for the wireless terminal according to the fourth aspect of the present invention for achieving the above object, a) after the voice recording plug-in (Plug-In) is activated, the wireless If a button for voice recording is input from the terminal, transmitting a mobile telephone number (MDN) and voice recording parameters to the multi-modal platform; b) delivering user voice input to the wireless terminal to the multimodal platform via EVRC streaming; c) converting, by the multi-modal platform, the type of the user's voice into a predetermined audio type, and mapping and storing it in the mobile phone number (MDN); And d) recognizing a recording message upload signal from the wireless terminal, and transferring user voice information stored in the multi-modal platform to the content server using the mobile telephone number (MDN) as a key. It features.

Or voice recognition method of the browser-based multi-modal function for a wireless terminal according to the same aspect of the present invention, a) Voice recognition plug-in (Plug-In) is activated to enable the wireless terminal to communicate with the multi-modal platform After opening, delivering a voice recognition parameter to the multi-modal platform including a mobile telephone number (MDN); b) the multi-modal platform automatically detects the end point of the voice using Server-side End Point Detection (EPD) and transmits it to the content server; c) converting the voice file of the user transmitted in the EVRC format to the PCM format by the multi-modal platform, and delivering the voice recognition parameter and the pre-recorded voice file to a voice recognition server (ASR) for voice recognition processing; ; d) providing, by the voice recognition server (ASR), a voice recognition result to the multimodal platform, and the multimodal platform creating and storing a temporary mapping item for the mobile phone number (MDN) and the recognition result; e) providing, by the multimodal platform, a voice recognition result as text on a terminal browser of the wireless terminal; And f) setting, by the content server, a mobile telephone number (MDN) as a key in the multi-modal platform based on the user's approval of the voice recognition result, and querying a recent search result for the voice recognition result. Characterized in that made.

Alternatively, a voice output method of a browser-based multimodal function for a wireless terminal according to the same aspect of the present invention includes a) after a voice output plug-in is activated, the wireless terminal is connected to the multi-modal platform. Establishing a communication and delivering a mobile telephone number (MDN) and voice output parameters to the multimodal platform; b) the multimodal platform reading a text file or a voice file stored in the content server from the URL information included in the voice output parameter; c) when the URL information is a text file, transmitting the text to a TTS server for converting the text into voice; d) providing the voice converted by the TTS server to the multi-modal platform, wherein the multi-modal platform is converted into an EVRC format that can be reproduced by the wireless terminal; And e) the multimodal platform streaming the voice converted into the EVRC to a plug-in on a WP browser of a wireless terminal.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

First, the present invention is provided with a terminal browser through a plug-in method. That is, even if the multi-modal function is not applied at the time of terminal release, it can be installed after the release, and the user can easily install the plug-in. Unlike the terminal application, the plug-in works similarly to the Active X control, so the user can install it without any additional work. Accordingly, the present invention provides multimodal on the terminal browser through this plug-in method.

The multi-modal plug-in provided in the present invention is provided in three forms: a voice recording plug-in, a voice recognition plug-in, and a voice output plug-in. Each plug-in is operated by the multi-modal platform when the corresponding item is requested from the terminal.

2 is a block diagram for providing a terminal browser-based multi-modal function shown as an embodiment of the present invention. As shown in the figure, a multimodal plug-in is requested on a WAP browser, and voice or keyboard information is transmitted over the wireless Internet based on the multimodal plug-in to receive the corresponding content. After recognizing the multi-modal plug-in request, the wireless terminal 201 performs data conversion on the voice and keyboard information, generates an information recognition result based on the data conversion, and corresponds to the information recognition result. The multi-modal platform 203 processes the visual data of the corresponding content into voice and keyboard information and transmits it to the wireless terminal 201, and extracts the content of the information recognition result to the multi-modal platform 203. It consists of a content server 205 for providing.

The multi-modal plug-in is composed of a voice recording plug-in, a voice recognition plug-in, and a voice output plug-in, and any one plug-in is requested on a swap browser. The multi-modal platform 203 temporarily changes the EVRC format for the voice input from the wireless terminal 201 to WAV format and temporarily stores the voice recording plug-in for activation. do.

In addition, the voice recognition plug-in, in response to the voice recognition command requested from the wireless terminal 201, the multi-modal platform 203 performs a voice recognition process according to a predetermined voice recognition algorithm, and the result is the content server ( 205). In addition, the voice output plug-in function may reproduce the information recorded by the multi-modal platform 203 from the wireless terminal 201 or as part of the content, and also through the TTS server according to user selection. Is converted into voice information and provided to the wireless terminal 201.

The operation of the present invention will be described below. First, in order to use the plug-in, the wireless terminal 201 transmits an object tag to the multimodal platform 203 on a WAP browser. The object tag is a series of control commands for checking the installation status of a plug-in or performing an update according to a version. The object tag is to include the object tag into the WML document when transmitting a message from the wireless terminal 201 to the multi-modal platform 203,

It can be implemented in the form

The object tag checks whether the plug-in is installed while calling the corresponding document in the WAP browser of the wireless terminal 201, and if it is not installed or is installed in a lower version, the object tag is installed immediately after user confirmation. Once the plug-in is installed, the plug-in takes up part of the screen of the WAP browser. Therefore, the user performs a multi-modal function through the wireless terminal 201, and the multi-modal has a function of recording, recognizing, and outputting voice as described above. Start the audio and audio output plug-ins respectively.

3 is a swap browser of a wireless terminal 201 showing a voice recording plug-in function among multi-modal plug-ins according to an exemplary embodiment of the present invention. As shown, the voice recording plug-in functions to record voices on a browser and deliver them to a multimodal platform. The multimodal plug-in consists of a "record" button 301, a "listen" button 303, and a status display field 305.

When the user presses the "record" button 301, the status display field starts recording while indicating the passage of time. At this time, the "record" button 301 is changed to a "stop" button 307. When recording is completed, press the "stop" button 307 to stop recording. And press the "listen" button 303 can play by listening to the recorded voice. In addition, when the corresponding key button is operated according to the 'recording message upload' message provided on the WAP screen of the wireless terminal 201, the recorded voice may be transmitted to the content server 205.

On the other hand, the multi-modal platform 203 is to record the voice message provided from the wireless terminal 201, and converts to the type required by the content server 205 through a plurality of audio types (Audio Type). The content server 205 may request a plurality of message formats according to the server, and the multimodal platform 203 may use 'evrc: Enhanced Variable Rate Codec Format', 'alaw: A-Law Format', and 'mulaw: Mu- Law Format ',' pcm: Intel PCM Format ',' alaw-wav: A-Law Wav Format ',' mulaw-wav: Mu-Law Wav Format ',' pcm-wav: PCM Wav Format ' Support. The multimodal platform 203 sets a voice recording plug-in parameter to set this format.

The voice recording plug-in parameter includes a 'MaxLength' parameter for setting the maximum length of the voice recording input in seconds and a message output such as "record a message" such as a prompt message to the wireless terminal 201. Contains the 'UseIntro' parameter. The default value of the 'MaxLength' parameter is 60 seconds, and the default value of the 'UseIntro' parameter is set to '1' to indicate a message output. Such parameters are indicated by the user on the swap browser of the wireless terminal 201, and the indicated parameters are provided to the multimodal platform 203. Figure 4 is a block diagram illustrating such a voice recording procedure.

First, when the user presses the 'record' button 301 in step S401 transfers the mobile phone number (MDN) and the above-mentioned parameters, that is, the Audio Type, MaxLength parameters to the multi-modal platform 203. The wireless terminal 201 delivers the user's voice to the multimodal platform by EVRC streaming. The Audio Type parameter is based on the format required by the content, and the MaxLength parameter is set according to the voice capacity of the user to be stored.

In step S403, the multi-modal platform 203 converts the voice type in a manner specified by the Audio Type, and maps it to a mobile phone number (MDN) to temporarily store it. In operation S405, the user selects a 'record message upload' button provided on a WAP browser. This requests the next page, and the content server 205 receives the page request. The voice file itself is not known to the content server 205 at this time.

In step S407, the content server 205 transfers the mobile telephone number (MDN) to the multi-modal platform 203 and requests a pre-stored voice to obtain a recorded voice. The content server 205 receives it, stores it locally and utilizes it. In operation S409, the content server 205 moves to the next page.

5 is a swap browser of the wireless terminal 201 for explaining the voice recognition plug-in function.

The speech recognition plug-in consists of an Edit field consisting of an "record" button and an input box. When the user presses the "record" button, a prompt message such as "speak search word" is output, and then the voice input mode is switched. For example, when a user utters a corresponding search word to search for 'Hyo Lee', the voice is recorded and transmitted to the multimodal platform 203 by streaming. The voice transmitted to the multimodal platform 203 is converted into text through a speech recognizer and transmitted to a multimodal plug-in. The Edit field of the plug-in displays it on the screen.

The user can also move the Edit field focus to enter the keypad. In addition, since the speech recognition result is ambiguous, several candidates may be selected. In this case, the Edit field is changed to a combo box to display a plurality of candidates according to the speech recognition result. Thereafter, the user can search the search results by pressing the "GO" button provided by the WAP browser of the wireless terminal 201.

In this way, the voice recognition is performed by the voice recognition algorithm in the multi-modal platform 203, the wireless terminal 201 provides a parameter for the voice recognition to the multi-modal platform 203. The parameter for speech recognition is 'GrammarURI', which specifies Grammar for speech recognition. Grammar can be an HTTP URL or can point to a static Grammar.

In case of HTTP URL, Grammar in JGSF, ABNF GRXML format is supported, and content server 205 must provide this Grammar file dynamically (using Dynamic Grammar). For example, 'http://cp.nate.com/ music_gr.jgsf '. In the case of static Grammar, Grammar is already registered in the speech recognizer in the multimodal server. Because this method is highly efficient, it is usually used in large vocabulary Grammar. Static Grammar is of the form 'static :: grammar1'.

In addition, the parameters for the voice recognition plug-in are 'UseNbest' parameter for determining whether to use N-best, 'UseIntro' parameter for prompt message output, and 'MaxLength' indicating the maximum length of voice recognition input in seconds. The parameter is used. The default value of the 'UseNbest' parameter is '0' indicating not using N-best, and the default value of the 'UseIntro' parameter is '1' for outputting a guide message. Default of the MaxLength 'parameter may be exemplified by 10 seconds.

6 is a configuration diagram for explaining the operation of the voice recognition plug-in shown in the embodiment of the present invention. The ASR server, which has not been described above, is a voice recognition server 601, and works in conjunction with the multi-modal platform 203 to process voice recognition of a user.

First, in step S601, the voice recognition plug-in establishes communication with the multimodal platform 203. The wireless terminal 201 transmits parameters for voice recognition, ie, GrammarURI, UseNbest, MaxLength, etc., to the multimodal platform 203 including a mobile phone number (MDN). This is to deliver a user voice called 'Hyo Lee' by streaming, and the multi-modal platform 203 automatically detects an end point of a voice using a server-side EPD (End Point Detection) and transmits it to the content server 205. do.

In step S603, the multi-modal platform 203 changes the voice file of the user transmitted in the EVRC format to the PCM format, and delivers the GrammarURI parameter and the pre-recorded voice file to the voice recognition server 601. The speech recognition server 601 provides the recognition result to the multimodal platform 203. In step S605, the multi-modal platform 203 creates and stores a temporary mapping item for the MDN and the recognition result.

In step S607, the multi-modal platform 203 delivers the result of speech recognition, that is, 'Hyo Lee' as text through a plug-in on the terminal browser, and the plug-in displays it in an Edit field (input box). do. In step S609, the user requests a search by pressing a 'GO' button based on the textified search result. The search request signal is provided to the content server 205. In step S609, the voice recognition plug-in cannot transmit the input result through the ECMA Script, so the content server 205 may not know what the search item is. Accordingly, the content server 205 sets a mobile phone number (MDN) as a key in the multi-modal platform 203 to query a recent search result. In step S611, the content server 205 generates the next search result page based on the search result and transmits the next search result page to the swap browser of the wireless terminal 201.

7 is a WAP browser of a wireless terminal for explaining a voice output plug-in function according to an embodiment of the present invention. First, the voice output may output the recorded voice, or may convert the text into voice through the TTS server. The swap browser of the wireless terminal 201 provides a speaker-shaped icon to display the voice output plug-in status.

As illustrated, the voice output converts text information such as news or e-books composed of text into voice. Such voice conversion output is performed by the TTS server, and for this purpose, the wireless terminal 201 transmits a parameter according to the content to the multimodal platform 203. The first parameter is a 'AutoPlay' parameter for determining whether to output audio automatically when the plug-in is executed, and a 'UseTTS' parameter for determining whether to use a TTS server for converting text to voice, and the 'UseTTS' 'Use audio file in parameter setting, that is,' AudioType 'to select one of evrc, alaw, mulaw, pcm, alaw-wav, mulaw-wav and pcm-wav files when not using TTS server. Parameter, and 'AudioURI' parameter indicating the location of the voice file to be converted.

The 'AudioType' parameter should be text / plain or text / xml + ssml when the 'UseTTS' parameter is 1, the text / plain is plain text, and text / xml + ssml is Speech Synthesis Markup Language (SSML). Means. In addition, the 'AudioURI' parameter specifies plain text or an HTTP URL of an SSML file when the 'UseTTS' parameter is '1' configured to use a TTS server. When the 'UseTTS' parameter is '0' that is set to not use the TTS server, the HTTP URL of the voice file is specified.

8 is a view for explaining a voice output plug-in procedure according to an embodiment of the present invention. In this embodiment, the TTS server 801, which is a voice recognition server, is included and interworked with the multi-modal platform 203.

As shown, in step S801, the voice output plug-in establishes communication with the multi-modal platform 203, and transmits a mobile phone number (MDN) and parameters such as AutoPlay, UseTTS, AudioType, and AudioURI to the multi-modal platform ( 203). In step S803, the multimodal platform 203 reads a text file or a voice file stored in the content server 205 from URL information included in an AudioURI parameter. In operation S805, when the URL information is a text file, the multi-modal platform 203 transfers the speech to the TTS server 801.

The TTS server 801 provides the voice-converted information to the multimodal platform 203, and the multimodal platform 203 can play back in the wireless terminal 201 through step S807. Convert to In operation S809, the multimodal platform 203 streams the voice converted into the EVRC to a plug-in on a swap browser of the wireless terminal 201. The user listens to text information converted to EVRC through the wireless terminal 201.

As described above, the browser-based wireless terminal for the multi-modal, the browser-based multi-modal server and system for the wireless terminal and the operation method thereof according to the present invention, even if the function is not applied at the time of terminal release It can be installed at any time later, and the user can easily install the plug-in, so that the voice recording plug-in function, the voice recognition plug-in function, and the voice output plug-in function can be started separately. By realizing the modal, the voice recognition input is performed on the search word on the wireless Internet, and the user can receive the voice recognition search service including the effect of easy operation of the mobile device through two input methods (key input + voice input). It works.

In the present invention, by providing a multi-modal messaging service that can record the voice on the WAP browser, the receiver can play on the WAP browser, there is an effect that the community service is activated on the WAP browser, such as mobile cyworld.

In addition, in the present invention, by providing a service by reading a large amount of information together with text information and the core content by voice, there is an effect of increasing the readability of the wireless terminal limited in LCD size.

While the invention has been shown and described with respect to specific preferred embodiments thereof, the invention is not limited to the embodiments described above, and is commonly used in the art to which the invention pertains without departing from the spirit of the invention as claimed in the claims. Anyone with knowledge will be able to make various variations.

Claims (31)

A service server for providing a multi-modal function for voice and text processing on a terminal swap (WAP) browser basis, After recognizing a multimodal plug-in connection state of a wireless terminal, performing data conversion for voice or keyboard information of a user, which is formed and received from the wireless terminal through a multimodal function based on the terminal swap (WAP) browser. And generating an information recognition result based on the data conversion, and receiving the corresponding content corresponding to the information recognition result, processing the visual data of the content into voice and keyboard information, and transmitting the multimodal to the wireless terminal. Browser-based multi-modal server for a wireless terminal, characterized in that it comprises a server. The method of claim 1, wherein the multi-modal plug-in, Browser-based multi-modal server for a wireless terminal, characterized in that any one or more of the plug-in for voice recording, the plug-in for voice recognition, the plug-in for voice output. The method according to claim 1 or 2, The multi-modal server in response to a voice recognition command requested by the wireless terminal for the voice recognition plug-in activation, and performs a voice recognition process according to a predetermined voice recognition algorithm. Multi modal server. The method according to claim 1 or 2, The multi-modal server is a browser-based multi-modal server for a wireless terminal, characterized in that for reproducing the recorded or textual information from the wireless terminal for the voice output plug-in activation. The method of claim 4, wherein the multi-modal platform, Browser-based multi-modal server for a wireless terminal, characterized in that for interworking with the TTS server for the reproduction output of the textified information. A wireless terminal for providing a multi-modal function for voice and text processing on a terminal swap (WAP) browser basis, Requesting a multimodal plug-in on the WAP browser and performing a multi-modal function based on the WAP browser in a connected state based on the multi-modal plug-in; And a plug-in operation mode for receiving a corresponding content corresponding to the voice or keyboard information after forming and transmitting the voice or keyboard information of the user through the mobile terminal. The method of claim 6 wherein the multi-modal plug-in, Browser-based wireless terminal for multi-modal, characterized in that any one of a plug-in for voice recording, a plug-in for voice recognition, a plug-in for voice output. The method of claim 6 or 7, wherein the plug-in is And a plug-in request using an object tag on a WAP browser of the wireless terminal. The method of claim 8, wherein the object tag, Browser-based wireless terminal for multi-modal, characterized in that included in the WML document when transmitting a message from the wireless terminal to the multi-modal platform. A system for providing a multi-modal function for voice and text processing on a terminal swap (WAP) browser basis, Requesting a Multimodal Plug-In on the WAP browser, and through a multimodal function based on the WAP browser in a connected state based on the multimodal plug-in. A wireless terminal for forming and transmitting voice or keyboard information of a user and receiving corresponding contents corresponding to the voice or keyboard information of the user; After recognizing the multi-modal plug-in request, data conversion for the voice and keyboard information is performed, an information recognition result is generated based on the data conversion, and visual data of the corresponding content corresponding to the information recognition result. A multi-modal platform for processing a voice and keyboard information and transmitting the same to the wireless terminal; And A browser-based multi-modal function providing system for a wireless terminal, characterized in that the content server for extracting the content for the information recognition result to provide to the multi-modal platform. The method of claim 10 wherein the multi-modal plug-in, System for providing a browser-based multi-modal function for a wireless terminal, characterized in that any one or more of a plug-in for voice recording, a plug-in for voice recognition, a plug-in for voice output. The method of claim 10 or 11, The multi-modal platform is a browser-based multi-modal for the wireless terminal, characterized in that for changing the EVRC format for the voice input from the wireless terminal to the WAV format for transmitting the voice recording plug-in to the content server Function providing system. The method of claim 10 or 11, The multi-modal platform, in response to a voice recognition command requested from the wireless terminal for starting the voice recognition plug-in, performs a voice recognition process according to a predetermined voice recognition algorithm and provides the result to the content server. Browser-based multi-modal function providing system for a wireless terminal. The method of claim 10 or 11, The multi-modal platform is a browser-based multi-modal function providing system for a wireless terminal, characterized in that for outputting the recorded or textual information from the wireless terminal for the voice output plug-in activation. The method of claim 14, wherein the multi-modal platform, A browser-based multi-modal function providing system for a wireless terminal, characterized in that the interworking with the TTS server for the reproduction output of the textified information. The method of claim 10, wherein the plug-in, The browser-based multi-modal function providing system for a wireless terminal, characterized in that performed by transmitting an object (OBJECT) tag on the multi-modal platform on a WAP browser of the wireless terminal. The method of claim 16, wherein the object tag, Browser-based multi-modal function providing system for a wireless terminal, characterized in that included in the WML document when transmitting a message from the wireless terminal to the multi-modal platform. In the browser-based multi-modal function providing method for a wireless terminal, a) transmitting a mobile telephone number (MDN) and a voice recording parameter to the multi-modal platform when a button for voice recording is input from the wireless terminal after the voice recording plug-in is activated. ; b) delivering user voice input to the wireless terminal to the multimodal platform via EVRC streaming; c) converting, by the multi-modal platform, the type of the user's voice into a predetermined audio type, and mapping and storing it in the mobile phone number (MDN); And d) recognizing a signal for uploading a recorded message from the wireless terminal, and transferring user voice information stored in the multi-modal platform to the content server using the mobile telephone number (MDN) as a key. Browser-based multi-modal function providing method for a wireless terminal. The method of claim 18, wherein the voice recording parameter, Audio Type parameter and MaxLength parameter, the Audio Type parameter is based on the format required by the content, the MaxLength parameter provides a browser-based multi-modal function for the wireless terminal, characterized in that set according to the user's voice capacity to be stored Way. The voice recording parameter of claim 19, A method of providing a browser-based multimodal function for a wireless terminal, further comprising a 'UseIntro' parameter for setting a message output. The method of claim 18, wherein the shape conversion of the user's voice, 'evrc: Enhanced Variable Rate Codec Format', 'alaw: A-Law Format', 'mulaw: Mu-Law Format', 'pcm: Intel PCM Format', 'alaw-wav: A-Law Wav Format', 'mulaw -wav: Mu-Law Wav Format ', or' pcm-wav: PCM Wav Format 'format conversion, browser-based for the wireless terminal, characterized in that the selective conversion in response to the message format request of the content server To provide multimodal functionality. In the browser-based multi-modal function providing method for a wireless terminal, a) the voice recognition plug-in (Plug-In) is activated to establish a communication with the multi-modal platform, the wireless terminal, and transmits the voice recognition parameters including the mobile number (MDN) to the multi-modal platform step; b) the multi-modal platform automatically detects the end point of the voice using Server-side End Point Detection (EPD) and transmits it to the content server; c) converting the voice file of the user transmitted in the EVRC format to the PCM format by the multi-modal platform, and delivering the voice recognition parameter and the pre-recorded voice file to a voice recognition server (ASR) for voice recognition processing; ; d) providing, by the voice recognition server (ASR), a voice recognition result to the multimodal platform, and the multimodal platform creating and storing a temporary mapping item for the mobile phone number (MDN) and the recognition result; e) providing, by the multimodal platform, a voice recognition result in text on a terminal browser of the wireless terminal; And f) based on a user's approval of the voice recognition result, the content server sets a mobile telephone number (MDN) as a key in the multi-modal platform, and inquires a recent search result for the voice recognition result. Method for providing a browser-based multi-modal function for a wireless terminal, characterized in that. The method of claim 22, g) generating, by the content server, a next search result page based on the search result, and transmitting the generated search page to a swap browser of the wireless terminal. The method of claim 22, wherein the speech recognition parameter, As a 'GrammarURI' parameter for selecting a voice recognition scheme, a voice recognition algorithm (Grammar) is specified, and the algorithm is an HTTP URL or a static algorithm (Grammar). How to provide browser-based multimodal functionality. The method of claim 24, wherein the HTTP URL, A method for providing a browser-based multimodal function for a wireless terminal, which supports Grammar in the JGSF and ABNF GRXML formats, and the content server dynamically provides the Grammar file. The method of claim 24, wherein the static algorithm, Registered to the speech recognizer mounted on the multi-modal platform, wherein the static algorithm is applied in a large vocabulary Grammar browser-based multi-modal function providing method for a wireless terminal. The method of claim 24, wherein the speech recognition parameter, The parameters for speech recognition plug-in are 'UseNbest' parameter for deciding whether to use N-best, 'UseIntro' parameter for prompt message output, and 'MaxLength' parameter representing the maximum length of speech recognition input in seconds. Browser-based multi-modal function providing method for a wireless terminal characterized in that it further comprises. In the browser-based multi-modal function providing method for a wireless terminal, a) after the voice output plug-in is activated, the wireless terminal establishes communication with the multi-modal platform, and transmits a mobile telephone number (MDN) and a voice output parameter to the multi-modal platform; ; b) the multimodal platform reading a text file or a voice file stored in the content server from the URL information included in the voice output parameter; c) when the URL information is a text file, transmitting the text to a TTS server for converting the text into voice; d) providing the voice converted by the TTS server to the multi-modal platform, wherein the multi-modal platform is converted into an EVRC format that can be reproduced by the wireless terminal; And e) the multi-modal platform streaming the voice converted to the EVRC to the plug-in on the wireless browser of the wireless terminal, characterized in that the browser-based multi-modal function providing method for the wireless terminal. The method of claim 28, wherein the audio output parameter, In the 'UseTTS' parameter for determining whether to use the TTS server for converting text to speech, the 'AutoT' parameter for determining whether to output audio automatically upon execution of the plug-in, and the 'UseTTS' parameter setting Method for providing a browser-based multi-modal function for a wireless terminal, characterized in that the 'AudioType' parameter for setting the use of the voice file, 'AudioURI' parameter indicating the location information of the voice file to be converted. The method of claim 29, The audio file usage setting of the 'AudioType' parameter may select one of the evrc, alaw, mulaw, pcm, alaw-wav, mulaw-wav, and pcm-wav files when the TTS server is not used. Browser-based multi-modal function providing method for a wireless terminal characterized in that. The method of claim 29, When the 'UseTTS' parameter is set to use a TTS server, the 'AudioType' parameter is plain text or Speech Synthesis Markup Language (SSML), and the 'AudioURI' parameter indicates that the 'UseTTS' parameter indicates a TTS server. When it is set to use, it specifies the plain text or the HTTP URL of the SSML file, and when the 'UseTTS' parameter is set to not use the TTS server, it specifies the HTTP URL of the voice file. Browser-based multimodal function providing method for a wireless terminal.

Images