JP2004246865A - Audio response web system and its input/output control method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an audio response web system and an input/output controlling method for it capable of improving a usage feeling in a terminal device using a multimodal technology. <P>SOLUTION: This audio response web system accessed from a terminal device 2 via a network 3 is provided with a web server 4 executing web application and an audio server executing audio application. The web server 4 is provided with a control part 12 notifying a condition of a program working on the web server 4 to a program working on the audio server 5. The audio server 5 is provided with a control part 15 notifying a condition of a program working on the audio server 5 to the web server 4. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

  The present invention relates to a voice response web system that synchronizes between a web server and a voice server, thereby enabling seamless cooperation between voice access from a terminal device and web access. And an input / output control method in a response web system.

  With the development of mobile phones and personal digital assistants (PDAs), mobile phones having a function of accessing websites on the Internet and a function of displaying a screen in addition to a voice transmission function (so-called so-called personal digital assistants). Browser phones) and PDAs having a voice input / output function and a voice transmission function in addition to a character input function and a screen display function have been put to practical use. In these terminal devices, characters and commands can be input by a plurality of methods such as a keypad or a keyboard, a stylus pen, and voice commands. Such a technique that enables a plurality of input / output methods in a certain device is called a multimodal technique.

  Conventionally, when performing web access from a mobile phone, a keypad is generally used to enter characters required for access.However, by applying multimodal technology, character input can be performed using speech recognition. And voice synthesis can be used as web output. Therefore, a voice response web system using voice recognition as a character input unit and using voice synthesis as a text output unit has been proposed for web access from a mobile phone or the like. The voice response web system processes data input / output (web data input / output) associated with normal web access by executing a web application, and also executes processes related to voice recognition and voice output.

  As such a voice response web system, there is a system in which a voice server for performing voice recognition and voice synthesis and a web server are combined. In this system, a program running on a web server processes web access, and a program running on a voice server processes voice recognition and voice synthesis. The start of the process related to web access is notified from the terminal device to the web server, and the start of the process related to voice recognition and speech synthesis is notified from the terminal device to the voice server via the web server.

  However, in such a voice response web system, since the program running on the web server cannot input the state of the program running on the voice server, the terminal device starts the voice recognition and displays the voice recognition result. Had to be notified to the web server for each. For this reason, the user who operates the terminal device has to perform two operations for starting speech recognition and displaying the result. In other words, the user once performs, for example, a key (or button) operation to cause the voice server to recognize the input voice, and performs a web access based on the voice recognition result when the voice recognition result is returned from the voice server. Therefore, it is necessary to perform further key (or button) operation. Further, in the case of performing voice recognition after the voice as a result of voice synthesis is output from the terminal device, since the program running on the voice server cannot input the state of the program running on the web server, The terminal device has to notify the start of the speech recognition after the speech as a result of the speech synthesis is output from the terminal device. For this reason, the user who operates the terminal device has to wait for the output of the voice to end, and then perform the key (or button) operation for starting the voice recognition as the next character input.

  As a method of constructing a voice response web system, the terminal device itself has a voice recognition function and a voice synthesis function, accesses the web server according to the voice recognition result in the terminal device, and transmits the character information from the web server to the terminal. There is also a method of synthesizing speech with a device, but this method requires too much processing at the terminal device in terms of both hardware and software. For example, when the terminal device is a mobile phone, a small number of vocabularies However, there is a problem that the voice recognition can be performed only by the above, the continuous use time becomes extremely short, and the reduction in size and weight of the terminal device is hindered.

  Further, in a web application using a terminal device including a mobile phone or a PDA, an application having a process related to voice recognition and voice output in addition to input / output of web data is called a multimodal application. There are two ways to implement multi-modal applications: a language that is dedicated to the multi-modal application and a dedicated web browser or an additional function added to an existing web browser. There are two methods that are realized by linking a general voice server and a general web server that perform voice recognition and voice output processing without requiring a web browser.

In a method of using a special language in creating a multi-modal application, an application description for screen data and an application description for voice processing are described in a single web page, that is, an XML (extended markup language) -based text file. Although it is possible to do so, the method of linking the voice server and the web server cannot describe both applications in a mixed manner. Here, the application for screen data and the application for audio processing correspond to so-called contents (contents of information). In a method of linking a voice server and a web server, a combination of both applications corresponds to a multimodal application. That is, the multi-modal application refers to an application in which a normal web page (or web content) including only characters and images is associated with a web page (or web content) that performs a voice response process. (The multimodal application is controlled to be activated by the voice response web system. A program that performs the activation control is a system application that implements the voice response web system described here. (In order to distinguish it from an application, it will be called a module instead of an application.)
In a conventional multimodal application executed in a system in which a voice server and a web server are linked, a typical function is to output a screen in response to a voice input, and a displayed web page changes. Dynamic execution of voice applications, simultaneous output of voice and screen data, control of speech recognition and output for web data input, and context-sensitive processing. It was difficult to build a complicated application.

  In addition, a method of designating an application for screen data processing and an application for audio processing to be started is generally determined in a program of an apparatus that aims to link both servers. . Therefore, the application creator cannot describe in the program or file to be created a specification for specifying the screen data processing application and the audio processing application to be started, and the screen data processing application to be started. In addition, it has been difficult to create an application that can freely specify an application for voice processing according to a user operation. Also, the device for linking both servers and the application for screen data processing and audio processing must be placed on the same local host or in the same local network, and they must be completely separated. Can not run. For this reason, the application creator could not use a device for linking the web server and the voice server via the Internet.

  As described above, in the conventional voice response web system based on the multimodal technology, the operability of a terminal device accessing the voice response web system is poor, and events such as start of voice recognition and display of a voice recognition result are not performed. There is a problem that it is necessary to operate keys and buttons on the terminal device every time. In a voice response web system in which a web server and a voice server are linked, a typical function is to output a screen in response to a voice input, and to perform a voice input and / or a web data input. It was not possible to perform both audio output and screen output.

  Therefore, an object of the present invention is to provide a voice response web system capable of improving the usability in a terminal device and an input / output control method thereof.

  Another object of the present invention is to provide a voice response web system that can simultaneously output voice and screen, control voice recognition and voice output for voice input and web data input, and easily perform application creation. An object of the present invention is to provide an input / output control method.

  A first voice response web system according to the present invention is a voice response web system accessed by a terminal device, comprising a web server and a voice server, and operating a program running on the web server on a voice server. It has first control means for notifying the program and second control means for notifying the web server of the state of the program running on the voice server.

  A second voice response web system of the present invention is a voice response web system accessed from a terminal device, comprising: a web server that manages execution of a web application; and a voice server that manages execution of a voice application. It has first control means for notifying the state of the program running on the web server to the voice server, and second control means for notifying the state of the program running on the voice server to the web server.

  According to the present invention, when an input request or a voice output request operation by voice recognition is performed in the terminal device, the activation unit activates the voice application based on the identification information obtained by the web data input. be able to. As the identification information, for example, an attribute indicating that a voice application to be started for input processing by voice recognition can be set. By setting the identification information as described above, the activation unit that performs the activation process of the audio application executes the audio application specified to find the attribute in the value obtained by inputting the web data. By adding the attribute to the identifier of the voice application, the creator of the application can specify the voice application to be executed when an input request by voice recognition is made.

  Further, in the present invention, the first control means activates a web application held in a first application server different from the web server via a network such as the Internet, and the activation means executes the second application different from the voice server. The voice application held in the second application server may be started. With this configuration, the application creator can arbitrarily specify the application or file to be started. For example, a URL (Uniform Resource Locater) for specifying an absolute address is described as identification information (for example, an identifier) for starting an application, and a program of a voice response web system is called by specifying an absolute address, thereby starting a voice application and a web application. The application does not have to be on the same host as the voice response web system, but can be on any host. Also, if the initial page designating these application calls is described in a web-accessible language such as HTML (hypertext markup language), XML, or XHTML (extended HTML), it can be placed on any host and the Internet can be placed. And can be executed in a distributed manner.

  In the input / output control method of the present invention, for example, a web application started from the web page (an existing web application started via a start control module of the voice response web system) is used as the identification information described in the web page. It is possible to set an attribute indicating that audio is to be supported and an identifier for specifying a file in which information (voice operation tag) for voice is written. Dynamically embed data read from the file for voice support specified by the identifier into the web data output by the web application launched from the web page at the time of execution, and put the combined result in the final By using web data output, a voice input / output function can be added without modifying an existing application. In addition, for a web application that has already been voice-enabled, it is possible to support both web applications that have already been voice-supported and those that have not.

  Note that the present invention is applicable to both the case where the terminal device can access the web server and the voice server at the same time and the case where the terminal device can access only one of the web server and the voice server at a certain point in time. It is also possible to apply.

  ADVANTAGE OF THE INVENTION According to this invention, the cooperation operation | movement of a web server and a voice server is attained, and the usability of the terminal device in a voice response web system improves.

  Further, according to the present invention, there is provided a voice response web system for linking a web application and a voice application. According to the voice response web system, the application creator can use the identification information (for example, the identifier and the attribute) for controlling the activation of the application, without developing a new control program. A multi-modal application can be developed for simultaneous output of images and screens, and voice recognition and control of voice output, and a multi-modal application with a higher degree of freedom can be developed. The application creator describes the identification information (for example, a file name as an identifier) for starting the application by using the URL of the absolute address specification, and calls the program of the voice response web system by specifying the absolute address to start the application. The voice application and the web application can be placed on any host, and these applications can be distributed and executed from the main host (main server) of the voice response web system. In the example described below, a case in which HTML data is used is shown. However, as a language for describing an application, an identifier, or the like, another script language or a markup language can be used. For example, XML data, It is possible to adapt to any of the XHTML data cases. Further, VoiceXML data, which is currently being developed as an extension to XML voice data, can be used.

  Next, a preferred embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of a voice response web system according to an embodiment of the present invention. The voice response web system 1 is accessed from a terminal device 2 via a network 3 such as the Internet, and includes a web server 4 and a voice server 5. The web server 4 and the voice server 5 operate in cooperation with each other. The web server 4 and the voice server 5 may be integral with each other in terms of hardware, or may be separate components that are closely connected to each other via a network. In the present invention, the terminal device 2 only needs to have at least one of a voice transmission function and a screen display function, but in this embodiment, it is assumed that the terminal device 2 has both a voice transmission function and a screen display function. . Such a terminal device 2 includes a keyboard or a keypad, a display screen, a microphone for voice input, and a speaker for voice output. A mobile phone capable of simultaneously making voice calls can be preferably used.

  The web server 4 includes an application execution unit 11 that executes a web application, and a control unit 12 that performs attribute determination and controls activation of the web application. The voice server 5 includes an application execution unit 13 for executing a voice application, an activation unit 14 for activating the voice application, and a control unit 15 for transmitting a voice recognition result and controlling voice output. Since the control units 12 and 15 and the activation unit 14 are actually realized by software executed on the web server 4 or the voice server 5, in the following description, the control unit 12 of the web server 4 This is referred to as an attribute determination / web application activation control module, and the activation unit 14 and the control unit 15 in the audio server are referred to as an audio application activation module and a recognition result transmission / audio output control module, respectively.

  First, the operation of the voice response web system will be described focusing on the status notification processing between the web server 4 and the voice server 5 and the timing thereof. FIG. 2 shows a process in a case where an operation of starting speech recognition is performed after the end of the speech output, and FIG. 3 shows a process in a case where an operation of starting speech recognition is performed during the speech output. I have. Note that, in these figures, the flow of data and the like between the servers is shown in chronological order as time passes from left to right in the horizontal direction.

  As shown in FIG. 2, in the first processing for accessing the voice response web system from the terminal device 2, when a voice response request access is issued by a user operation of the terminal device 2, the voice is transmitted from the terminal device 2. The voice response request access 104 is sent to the server 5, and the voice response request access 104 is notified to the program 103 operating on the voice server. In response to this, the program 103 running on the voice server sends a voice synthesis status notification 106 to the program 102 running on the web server 4 and sends a voice output 105 to the terminal device 2. After the voice output 105 is completed, the program 103 running on the voice server sends a voice recognition start waiting state notification 107 to the program 102 running on the web server 4. The voice synthesis state notification 106 indicates that the voice server 5 is in a state of performing voice synthesis processing, and the voice recognition start waiting state notification 107 indicates that the voice server 5 is in a state of waiting for the start of voice recognition. It shows that it is.

  When a speech recognition request web access 108 is sent from the terminal device 2 to the web server 4 by a user operation on the terminal device 2, the speech recognition request web access 108 is notified to the program 102 running on the web server, The program 102 running on the server sends a voice recognition start notification 109 and a voice recognition result waiting state notification 110 to the program 103 running on the voice server. The voice recognition start notification 109 notifies that the voice recognition should be started, and the voice recognition result waiting state notification 110 notifies that the web server 4 is waiting for the voice recognition result. is there.

  Here, since it is assumed that the voice recognition request web access is performed after the voice output is completed, the program 103 operating on the voice server operates on the web server before the voice recognition request web access is performed. A voice recognition start wait state notification 107 is sent to the program 102 to be executed. At this point, the program 103 running on the voice server is in a wait state, and thus the voice recognition processing is immediately started.

  Thereafter, the voice recognition processing is executed by the program 103 running on the voice server, and when the voice recognition processing is completed, the program 103 running on the voice server performs the voice recognition processing on the program 102 running on the web server. A voice recognition completion status notification 111 indicating that the voice recognition is completed is sent. Upon receiving the voice recognition completion status notification 111, the program 102 running on the web server sends a voice recognition web display 112 to the terminal device 2. The speech recognition web display 112 is a display corresponding to the speech recognition result.

  On the other hand, if the voice recognition request web access is performed at the time when the voice output is not completed, as shown in FIG. 3, at the time when the voice recognition request web access 108 is issued, the program running on the voice server is executed. 103 has not issued the voice recognition start waiting state notification 107 yet. Then, the program 103 operating on the voice server is notified of the voice recognition start notification 109 and the voice recognition result waiting state notification 110 from the program 102 operating on the web server. Do not start. When the speech recognition start notification 109 is found at the end of the speech synthesis processing, the program 103 operating on the speech server executes the speech recognition processing because the state of the program 102 operating on the web server is in a speech recognition result waiting state. To start. Thereafter, the speech recognition completion state 111 is sent to the application running on the web server, as in the case shown in FIG. However, when a voice synthesis system capable of canceling processing during voice output is used as the voice server 5, the program 103 running on the voice server receives the voice recognition start notification 109 during the voice output. In this case, the voice output may be canceled at that time, and the voice recognition processing may be started immediately.

  In each of the example shown in FIG. 2 and the example shown in FIG. 3, the program 102 running on the web server refers to the state of the program 103 running on the voice server after the notification of the voice recognition web access 8 is notified. are doing. Then, when the voice recognition is completed, the program 102 operating on the web server sends a voice recognition web display 112 to the terminal device 2. As a result, the screen corresponding to the voice recognition result is displayed on the screen of the terminal device 2 based on the voice recognition web display 112.

  The program 102 running on the web server may execute an application that outputs a voice simultaneously with the voice recognition web display on the screen of the terminal device 102. In this case, the program 102 running on the web server sends the speech recognition web display 112 to the terminal device 2 and simultaneously notifies the program 103 running on the voice server of the start of speech synthesis. A start notification 113 is sent. Upon receiving the voice synthesis start notification 113, the program 103 operating on the voice server starts voice synthesis processing and executes voice output 114 to the terminal device 2. The audio output 114 here is the same as the audio output 105 described above, and the processing after the above-described audio output 105 is similarly repeated with the start of the audio synthesis processing.

  Next, activation control of the voice application and the web application in the voice response web system shown in FIG. 1 will be described. In the present embodiment, activation control of an application is performed using an identifier / attribute as identification information. FIG. 4 shows in more detail the portion from the speech recognition request web access to the speech recognition web display in the process flow shown in FIG. 2, and shows the process flow when the basic process is performed. FIG. 5 shows a description example of an attribute and an identifier used for controlling the activation of an application. In FIG. 5, attributes and identifiers are described as tags in HTML. FIG. 6 is a flowchart showing processing in the voice response web system. In this voice response web system, as is apparent from the following description, simultaneous output of voice and screen is performed based on voice input, only voice output is performed from web data input, and voice and screen are output from web data input. Although simultaneous output is performed, a flowchart is illustrated in FIG. 6 as being capable of coping with all these processes. The determination of any of these cases is made around steps S6, S12, and S14. It goes without saying that the order in which the steps S6, S12, and S14 for determining which case are performed can be changed as appropriate.

  In the following description, a module is a program created by a system developer that implements the voice response web system shown here, and is modularized by a function such as activation control of a voice application or a web application. That is. Usually, a module cannot be modified by an application developer, but is only allowed to execute. On the other hand, the voice application and the web application are mainly created by an application developer. Similarly, the audio output data is also created mainly by the application developer.

  Here, the voice application is an XML-based text file described in a language executable by a browser for voice processing, or a program that outputs an XML-based text file. The web application is, for example, an HTML text file or an XML-based text file in which data that can be displayed by a web browser is described, or a program that outputs an HTML text or an XML-based text. The voice application and the web application may be on the same local host as the voice response web system or on a different host. In this description, the voice application itself seems to perform voice recognition and voice synthesis. However, after the voice application is activated by the control module, the voice recognition and voice synthesis engines perform voice recognition and voice synthesis. Synthesis is performed.

《Basic processing》
An example of the basic processing will be described. As shown in FIG. 4, in the terminal device 2, a voice recognition request web access 208 is performed by a user operation such as clicking a button or link on a voice-enabled web page displayed on the screen of the terminal device 2. Occurs, the web application 205 receives the voice recognition request web access 208, and transmits an identifier / attribute notification 209 to the attribute determination / web application activation control module 204 (hereinafter, simply referred to as “activation control module 204”). 1, the web application 205 is executed by the application execution unit 11, and the function of the activation control module 204 is realized by the control unit 12. The activation control module 204 determines that the speech server 5 is in a state of waiting for speech recognition, and sends a speech recognition start notification and an identifier / attribute notification 10 to the speech application activation module 201 (hereinafter, simply referred to as “activation module 201”). Then, the voice application 202 is started. The function of the activation module 201 is realized by the activation unit 14 of the audio server 5, and the audio application 202 is executed on the application execution unit 13 of the audio server 5.

  The activation module 201 activates the voice application 202 that performs voice recognition in accordance with the instruction from the activation control module 204 and the contents of the identifier / attribute notification 210. At this time, if there is no special designation in the identifier / attribute notification 10, a default, in other words, a predetermined voice application is started.

  When a voice recognition result is generated after the execution of the voice application 202, the value of the recognition result is transmitted from the voice application 203 to the recognition result transmission / voice output control module 203 (hereinafter, simply referred to as “control module 203”). . As shown in FIG. 2, the activation control module 204 refers to the state of the control module 203. When the control module 203 sends the speech recognition completion state notification 11 to the activation control module 204, the activation control module 204 Is received, the web application 206 is immediately started according to the content, and the web application 206 is started. As a result, the speech recognition web display 213 is output from the web application 206, and the result of the speech recognition is displayed on the screen of the terminal device 2 as the speech recognition web display. At this time, the value of the speech recognition result includes a file name on the local host, a URL designating a web application on the Internet, a file name on the local host, or a parameter (query data) to the web application. It contains an identifier that can determine the information needed to launch the web application. That is, the identifier may be the file name of the web application itself, or a URL or a file name necessary for starting the web application may be referred from a data holding device including a database and a text file based on the identifier. You may. The identifier may be configured based on a plurality of keywords or parameters as long as the web application to be started can be specified.

  The activation control module 204 activates the web application 206 specified by these identifiers, and transfers control of the web server 4 to the web application 206. Generally, the web application 205 and the web application 206 are often different, but may be the same application.

  When the control module 203 finishes communication with the activation control module 204 of the web server 4, the control right of the voice server 5 is transferred to the activation module 201. Then, the activation module 201 sends the voice recognition start waiting state notification 107 of FIG. 2 to the activation control module 204, and enters a state of waiting for a notification from the activation control module 204 in preparation for the next voice processing request.

  As a result of the above processing, when the web application 206 is activated and screen output is performed, the user operates the web page displayed on the screen of the terminal device 2. When the operation of the voice processing request is performed, the web application 206 becomes the web application 205 described above, and the operation of the basic process illustrated in FIG. 4 is repeated.

  The identifier / attribute notification 209 transmitted by the web application 205 to the activation control module 204 may include an identifier for activating the voice application or the web application, and a parameter including a value added to the identifier. There are several ways to use this parameter, as described below. In this embodiment, the identifier may have an attribute in order to distinguish the usage form. The identifier having the attribute information or the value added to the identifier is used in the activation control module 204 of the web server 4 or the activation module 201 of the voice server 5 to perform a process according to the attribute. The attribute information may be a value described directly on a web page, or may be an identifier for referring to the value from a data holding device such as a file or a database.

《Usage of attributes》
Next, how to use attributes will be described.

  If the value of the speech recognition result is a value with a large number of characters such as a URL, for example, as described in the above <Basic Processing>, it is necessary for the application to describe all the character strings in each of the speech recognition results. It is cumbersome for the creator. Therefore, means for the application creator to briefly describe the character string of the speech recognition result will be described.

  As one of the attributes described above, an attribute indicating that the acquired value is “a value to be added before the value of the speech recognition result” is set. With reference to the flowchart shown in FIG. 6, a process when the attribute is set in this way will be described.

  First, when the user performs a voice recognition request operation, parameters are acquired from a web page, and the parameters are transmitted to the activation control module 204 (step S1). Then, in step S2, it is determined whether or not there is a value to be added to the speech recognition result. When such a value is present, in step S3, it is held as recognition result addition data. Since the processes in steps S4 to S9 are not relevant here and will be described later, when a voice input is performed thereafter, in step S10, a voice recognition result is obtained and the voice recognition result is transmitted to the activation control module 204. Can be Then, in step S11, the values of both the speech recognition result and the data for adding the recognition result are combined, and used in step S16 to activate the web application 205.

  FIG. 5A shows an example of a description of an attribute and a value. Here, “page1.html” is a speech recognition result sent from the speech recognition application 202. The INPUT tag (a tag starting with “<INPUT”) is described in the web application 205, and is used as an attribute of the Name of the INPUT tag indicating an attribute of a value specified by an attribute value of Value. I have. Instead of the INPUT tag, it may be passed as a query string parameter in an HTTP (hypertext transfer protocol) request. In this example, “add” in the INPUT tag is an attribute indicating that a value described below “value =” is added before the speech recognition result. “Http://aaaaa.co.jp/app1/” is a value actually added to “page1.html”. The launch control module 204 combines these values to launch the web application.

  Next, another example of attribute use will be described. A procedure will be described in which the content described in the speech recognition result is used not as an identifier for specifying the web application to be activated but as a parameter to the web application to be activated.

  As one of the attributes described in the above <Basic Processing>, "attribute indicating" identifier designating web application requesting parameter "" is set. FIG. 5B shows an example of a description when the recognition result is handled as a parameter. In this case, since the attribute may be added with a value before the speech recognition result, "add" in the INPUT tag can be used as in the case described above. The application developer describes the web application requesting the parameter as “http://aaaaa.co.jp/app2/servlet/stock?”, And the value described as the speech recognition result includes “code = 9999 ", The parameters to be passed to the web application are described.

  In the flowchart shown in FIG. 6, when the activation control module 204 finds the attribute of “add” in step S2, it recognizes “http://cywiz.co.jp/app2/servlet/stock?” In step S3. It is stored as additional data, and in step S11, this value is combined with the parameter "code = 9999" sent as the value of the speech recognition result, and the application is started.

《Audio support for existing applications》
One of the features of the present invention is that an existing general application can be directly used for audio. Here, a method of making an existing general application compatible with voice without modifying it will be described. However, only the first page for calling an existing application needs to be a newly created page. In this page, a URL is specified as described in the above << Use of Attributes >>, and an existing application is called. In addition, data that needs to be added to the existing application in order to support voice is prepared, and an identifier of the data (a file in which the data is described) is set to “data to be added to output data in order to support voice. It is described together with an attribute indicating that it is an "identifier".

  In the flowchart of FIG. 6, after the execution of step S2 or S3, the startup control module 204 describes, in step S4, a capture file, that is, data that needs to be added to an existing application when a parameter value is received. It is determined whether or not a file exists, and if such a file is found, the obtained value is treated as an “identifier of data to be added”. Then, in step S5, a file is read based on the identifier and stored as data to be added to the output of the existing application.

  A description example of the identifier of the data to be added is shown in (3) of FIG. In this example, “include” in the INPUT tag is an attribute indicating that the value “voice.html” described below “value =” is handled as an identifier for capturing a file, and is associated with voice. In the content of “voice.html”, a button or a link that triggers the voice input by calling the activation language module 204 is described. Further, the value indicated in “Use of attributes”, voice output described later, voice / screen, and the like are described. A value for output can be described. In addition, as long as an application that does not support audio is called, the specification of “include” is necessary. As described in << Usage of Attributes >>, the startup control module 204 retrieves the output data (usually HTML data) from the stream connected by the URL when calling the existing application, and adds "voice. "html" is combined and output (step S16). If the voice operation button is displayed at the end of the screen data, for example, the combination is embedded before "</ BODY>" in the HTML text. If an existing application is to be modified in advance, data corresponding to the content of "voice.html" must be written, and the attribute of "include" need not be specified.

《Specification of voice application》
Next, the designation of the voice application will be described.

  When the user requests input by voice recognition in the web application 205, the application creator does not use the default voice application but the voice application specified by the application creator, for example, the displayed web page. When it is desired to activate a corresponding voice application including a grammar for voice recognition, the identifier of the voice application to be activated is described in the web application 205. This identifier is described together with an attribute indicating that it is an "identifier for specifying a voice application to be activated for input processing by voice recognition". One example is shown in FIG. “Vxml” in the INPUT tag is an attribute indicating that a value described below “value =” is to be treated as an identifier of a voice application executed at the time of input processing by the next voice recognition, and “query.vxml” is This is the identifier of the voice application to be activated.

  In the flowchart shown in FIG. 6, when an input request by voice recognition is made by a user operation (step S6), the activation control module 204 sets the identifier of the voice application to be executed next (“query.vxml”) to the web application. When received as a parameter from 205, an attribute (“vxml”) of designating as a voice application to be executed next is added to the received identifier, and the identifier is set in the standby state for waiting for data from the activation control module 204. Send to module 201. In step S7, the activation module 201 determines whether there is an attribute of “vxml”. If such an attribute is found, in step S8, based on the received identifier, the corresponding voice application 202 ( "Query.vxml"). If the attribute of “vxml” is not found in step S7, the activation module 201 activates the default voice application in step S9. The processing after the execution of steps S8 and S9 is the same as the case of << basic processing >> described above as the processing after step S10. If it is determined in step S6 whether the request is a voice input request, and if the request is not a voice input request, the process proceeds to step S12 described below.

《Simultaneous output of audio and screen from audio input》
Next, a process for simultaneously outputting a voice and a screen based on a voice input will be described with reference to FIG.

  There are two methods for simultaneous output of sound and screen. One is an explicit designation method in which an identifier of audio data to be output is described in the web application 205. The identifier / attribute notification 209 sent from the web application 205 to the activation control module 204 includes the identifier of the audio output data 214. Upon receiving the voice recognition completion status notification 211, the activation control module 204 sends a voice output identifier notification 216 to the control module 203 together with a voice synthesis start notification. The control module 203 determines the audio output data 214 based on the audio output identifier, and executes the audio output 217. Here, the voice output data 214 is data that can be read by file access, database access, or web access, and may be on the same local host as the voice response web system or another host accessible by the network. It can be on top.

  The audio output data 214 is, for example, a message file to be executed by the audio synthesis engine of the audio server 5 or an audio file for audio output. In this case, the control module 203 takes in the data and executes it as an audio application. do it. Alternatively, the audio output data 214 may be in the form of an audio output application that can be executed from the beginning. Further, the web application 206 may dynamically generate data similar to the audio output data 214. Alternatively, the control module 203 or the activation control module 204 may dynamically extract data from the data of the speech recognition web display 213 to create the speech output data 214. Alternatively, the voice output data 214 may be determined by combining the voice output identifier and the value of the voice recognition result. Since the voice recognition completion status notification / voice recognition result notification 211 and the voice synthesis start notification / voice output identifier notification 216 are performed almost simultaneously in the form of message exchange, the voice output 217 is executed with a delay in FIG. Although visible, in practice the speech recognition web display 213 and speech output 217 are performed simultaneously.

  Another method for simultaneously outputting the voice and the screen is to use a common identifier that associates the voice output data 214 with the web application 205 started from the voice recognition result. The creator of the application creates the voice output data 214 so as to have the same identifier as the identifier of the web application 205 started from the voice recognition result. The difference from the above-described “basic processing” is that, as shown in the flowchart of FIG. 6, when the control module 203 transmits the speech recognition result to the activation control module 204, in step S17, the common identifier with the web application 205 is set. It is checked whether or not the audio output data 214 or the audio output designation is present, and if these are present, the data is read in step S18 to execute the audio output. As a result, the screen output by the activation of the web application by the activation control module 204 (step S19) and the audio output by the control module 203 are performed simultaneously. In the flowchart shown in FIG. 6, the process proceeds to step S17 after the execution of step S16. If there is no such audio output data 214 in step S17, the process directly proceeds to step S19.

《Sound and screen output from simultaneous input of sound and web data》
Next, a description will be given of a process in a case where voice and web data are input simultaneously and a voice and a screen are output based on the voice and web data.

  In the process of simultaneous output of the voice and the screen from the voice input illustrated in FIG. 7, when the user operates the web page displayed on the screen of the terminal device 2, the web application 205 sends the screen to the activation control module 204. The identifier / attribute notification 209 includes web data input by the user, for example, data such as position data of a map pointed to by the user, buttons and click points selected by the user, and text data input by the user. Simultaneous input of web data enables simultaneous output of sound and screen. If the audio output processing is not performed as described in the above <Basic processing>, the screen can be output from simultaneous input of audio and web data. In the process of using both the web data and the value of the speech recognition result, the combination of the values of the identifiers indicated by << use of attributes >> can be used.

《Only audio from web data input》
Next, a description will be given of a case where only sound is output from the web data input. In each of the above-described examples, the simultaneous output of the voice and the screen is performed by the voice recognition. However, there is a case where the voice output or the screen output is performed by the operation on the displayed web screen without performing the voice input. . FIG. 8 shows a process in the case of controlling only the audio output from the web data input. Here, the operation of the voice response request web access 218 that does not perform voice recognition is performed instead of the voice recognition request web access. In the web application 205, the identifier of the voice application 207 to be executed is described as in the case of the above-mentioned "designation of voice application". FIG. 5 (4) shows an example of a description of an identifier for executing a voice output by a user clicking a link.

  When a voice output request operation is performed in the web application 205, the activation control module 204 receives the identifier of the voice application 207 (“explain.vxml” in this example) from the web application 205 as a parameter. At this time, an attribute (“vout” in this example) indicating that the parameter is an “identifier for specifying an application for voice output” is added to the parameter. The start control module 204 determines whether or not there is an attribute of “vout” in step S14 as shown in the flowchart shown in FIG. The identifier “explain.vxml” of the voice application 7 is sent to the activation module 201 with the attribute “vxml” shown. The activation module 201 activates the audio application 207 in step S15, as in the case of << designation of audio application >>. In this case, the voice application 207 is different from the voice application 202 which requests a voice input described in << Basic Processing >> and is a voice application which performs only a voice output, and is described so as to pass control to the activation module 201 after execution. ing. When the activation control module 204 finds the attribute of “vout”, the activation control module 204 does not acquire the voice recognition result and does not update the current screen data or a terminal that needs to be updated for operation on the web screen. In the case of, the same web application 205 as the application on which the operation was performed is displayed.

《Simultaneous output of voice and screen from web data input》
Next, a process in the case of performing simultaneous output of sound and screen from web data input will be described with reference to FIG.

  In order to output the screen simultaneously with the audio output by inputting the web data, the following processing is executed. When a user operation is performed in the web application 205, the web application 205 sends the same data as the result of the voice recognition from the voice application 202 to the activation control module 204 as a parameter. FIG. 5 (5) shows an example of a description of an identifier on a web page for allowing a user to simultaneously output a voice and a screen by clicking a link. “Kinou.html” in the figure is a value corresponding to the result of speech recognition. “Vdout” is an attribute indicating that the acquired value is “a value corresponding to the result of speech recognition”.

  As shown in the flowchart of FIG. 6, when acquiring the value of “kinou.html”, for example, in step S12, the activation control module 204 determines whether there is an attribute of “vdout”. When the attribute of “vdout” is found, the value is sent to the activation module 201 which is in a state of waiting for data from the activation control module 204. If the attribute of “vdout” is found, the activation module 201 directly transmits “kinou.html” as the value of the recognition result to the control module 203 without activating the voice application 202 in step S13. Send (arrow B in FIG. 9). The control module 203 that has received the value performs the same processing as the above-described operation of “Simultaneous output of voice and screen from voice input”, that is, shifts to step S11, and as a result, voice output and screen output are performed. . Therefore, even in the speech recognition completion state notification / speech recognition result notification 211, the same data as when speech recognition is performed is sent without actually performing the speech recognition processing. If the attribute “vdout” is not found in step S12, the process proceeds to step S14 described above. In this case, if the attribute “vout” is not found in step S14, the above-described “ FIG. 6 shows a case in which none of the above-mentioned "simultaneous output of voice and screen based on voice input", "only voice output from web data input", and "simultaneous output of voice and screen from web data input" apply. As described above, in step S20, error processing or other processing is executed.

  With the above configuration, when creating an application that realizes simultaneous output of a screen and voice from a user operation on the web screen, the creator of the application sends the value sent as a result of voice recognition from the voice application 202 to the web application 205. It may be described or programmed so as to be transmitted as a parameter from the web application 205.

《Context processing》
According to the present invention, it is possible to perform processing in consideration of the context and the history of conversation with the user. Such processing will be referred to as context processing. Hereinafter, the context processing will be described.

  Information obtained from the execution result of the voice application 202 is classified into "information for determining a certain topic", "information belonging to a certain topic", or "command for controlling a dialogue". Which one of the speech recognition results is divided is determined by the identifier included in the speech input result. As a result, “information for determining a certain topic” is held in the activation control module 204. Here, the topic is, for example, the name of a certain product or individual, or the name of an individual. The information on topics is attribute information on those individuals, for example, properties, performance, price, characteristics, and the like.

  According to the above configuration, “information for determining a certain topic” is held in the activation control module 204, and the information held as a topic does not change as long as information belonging to the topic is continuously input. When information that belongs to a topic is sent to the activation control module 204, the information is combined with the current topic and handled as an identifier for activating a web application. How to use this identifier is the same as that described in the above <Basic processing>. For example, if the topic currently held is "car" and the user is talking about the topic, and an utterance such as "What is the characteristic?" The activation control module 204 activates the web application 205 using the keywords “car” and “feature” as identifiers for activating the web application. In addition to the start of the web application, the simultaneous output of the sound and the start of the web application can be performed as shown in << Simultaneous output of sound and screen from sound input >>. Further, in order to perform a context-dependent dialogue, it is necessary to switch the corresponding grammar, but this can be realized by the method described in << Specification of Voice Application >>.

  Further, according to the above-described configuration, in order to control not the conversation about a certain topic but the conversation, the startup control module 204 can hold not only the current topic but also the input topic as a history. it can. By doing so, when the user utters a command to control the dialogue, for example, when a speech recognition result such as "return to the previous topic" or "return to the topic with which the dialogue is currently transmitted" is sent, , The corresponding topic information can be searched and a web application can be started. Further, by combining the value of the history information holding a plurality of topics with the newly input speech recognition result, it is possible to realize more complicated context processing.

《Distributed execution》
As is apparent from the above description, in the voice response web system according to the present embodiment, the web application and the voice application created by the application creator are separated from each module such as the start control module created by the voice response web system developer. It is possible to do. Furthermore, by referring to these applications and each module using a URL that specifies an absolute address, a web application or a voice application can be executed even if the application is placed on any host on the Internet. FIG. 10 shows a configuration in which the web application and the voice application are stored in a server independent of the main body of the voice response web system. Here, an application server 6 that holds and executes a web application and an application server 7 that holds a voice application are respectively connected to the network 3. These application servers 6 and 7 are provided with application execution units 16 and 17 for executing a web application and a voice application, respectively. If the network 3 is the Internet, it can be specified by the resources and URLs on each of the servers 4 to 7, so that the web server 4 or the voice server 5 starts the application held in each of the application servers 6 and 7. Can be executed.

  Although the preferred embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment. In the above description, the terminal device 2 has the voice input / output function and the web display function. However, in applying the present invention, it is not always necessary that one terminal device has these functions. Absent. In the present invention, activation control of a voice application and a web application is realized by cooperation between servers, and it is not necessary to mount a special program or plug-in on a terminal device. Therefore, it is also possible to combine a terminal having an arbitrary web display function with a terminal having an arbitrary voice input function. For example, as a terminal having a voice input function, a fixed telephone, or a mobile phone (a web display function is not required), or an IP (Internet protocol) telephone, or a softphone, and as a terminal having a web display function, Personal computer, or PDA, or mobile phone with web display function, fixed phone with web display function, or display device such as PDP (plasma display panel) or LCD (liquid crystal display device) with web display function Alternatively, a kiosk terminal having a web display function can be used. Further, the present invention can be applied to a mobile phone that can access a web server and can communicate voice input / output data or voice feature extraction data by IP.

  Also, in the present invention, the request for voice recognition from the terminal device does not necessarily need to be transmitted to the voice response web system via web access.

  The above-described voice response web system of the present invention can be typically realized by a software program executed on a server computer. Such a program is generally read into a computer by a recording medium such as a magnetic tape or a CD-ROM, or via a network such as the Internet. The computer functions as the voice response web system described above by executing the read program.

It is a block diagram showing composition of a voice response web system of one embodiment of the present invention. FIG. 14 is a diagram illustrating a state notification process performed between the voice server and the web server and a timing thereof when a voice recognition start operation is performed after the voice output ends. FIG. 14 is a diagram illustrating a process and a timing of a status notification performed between the voice server and the web server when a voice recognition start operation is performed during voice output. FIG. 9 is a diagram illustrating a process for controlling activation of a voice application and a web application using an identifier / attribute and timings thereof. FIG. 5 is a diagram illustrating a description example of an attribute and an identifier used for controlling activation of an application. It is a flowchart which shows the detail of the flow of a process in a voice response web system. It is a figure which shows the process at the time of performing simultaneous output of a sound and a screen from a voice input, and its timing. It is a figure which shows the process at the time of performing only audio | voice output from web data input, and its timing. It is a figure which shows the process at the time of performing simultaneous output of a sound and a screen from web data input, and its timing. FIG. 3 is a block diagram illustrating a configuration of a voice response web system when a voice application and a web application are arranged on another server.

Explanation of reference numerals

DESCRIPTION OF SYMBOLS 1 Voice response web system 2 Terminal device 3 Network 4 Web server 5 Voice server 6, 7 Application server 11, 13, 16, 17 Application execution part 12, 15 Control part 14 Activation part

Claims (11)

  A voice response web system accessed from a terminal device, comprising: a web server and a voice server, a first control unit for notifying a program running on the voice server of a state of a program running on the web server; And a second control unit for notifying the web server of the status of the program running on the voice server. A voice response web system accessed from a terminal device,
A web server that manages execution of the web application, and a voice server that manages execution of the voice application,
A first control unit for notifying the state of the program running on the web server to the voice server; and a second control unit for notifying the state of the program running on the voice server to the web server. Voice response web system.   The content of processing in the voice server is determined based on a state notified from the first control means, and the content of processing in the web server is determined based on a state notified from the second control means. 3. The voice response web system according to 1 or 2.   The voice response web system according to claim 2, further comprising an activation unit that activates the voice application based on a notification from the first control unit.   The voice response according to any one of claims 2 to 4, wherein association between the voice application and the web application is performed based on a value obtained as a result of voice recognition and a value obtained through web data input. Web system.   The method according to any one of claims 2 to 4, wherein the first control unit activates a corresponding web application by combining a value obtained from a speech recognition result and a value obtained from web data input. Voice response web system.   5. The method according to claim 4, wherein, when an input request or a voice output request operation by voice recognition is performed in the terminal device, the activation unit activates a voice application based on identification information obtained by web data input. A voice response web system as described.   In the terminal device, when an operation of a request for simultaneous output of voice and screen is performed or when voice input is performed, screen data output from the web application is performed based on identification information obtained by web data input. The voice response web system according to any one of claims 2 to 4, wherein voice data output from the voice application and the voice application are performed simultaneously. The first control unit activates a web application held by a first application server different from the web server via a network, and the activation unit executes a second application server different from the voice server 5. The voice response web system according to claim 4, wherein the voice response web system is activated. An input / output control method in a voice response web system that is accessed from a terminal device and has a web server that manages execution of a web application and a voice server that manages execution of a voice application,
For the value obtained from the web data input, set identification information indicating that voice correspondence is to be performed and specify a file for voice operation,
Dynamically refers to the screen output data output by calling the web application, based on the identification information, embedded data read from the file,
An input / output control method for executing screen output data in which the data is embedded as final screen output. An input / output control method in a voice response web system that is accessed from a terminal device and has a web server that manages execution of a web application and a voice server that manages execution of a voice application,
Context processing is performed taking into account topics and dialog history during dialogue with the user,
An input / output control method for activating the web application and outputting a sound in accordance with a processing result of the context.

Images