JP2007033478A - Multi-modal dialog system and multi-modal application generation wizard - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily generate a prototype of basic multi-modal application and to improve development efficiency even for an application development engineer who is not familiar with a mounting method of voice dialog application and the multi-modal application. <P>SOLUTION: To automatically generate source codes of graphic user interface (GUI) application, a dialog scenario, voice recognition syntax for composing the multi-modal application by selecting the number of question items, a name regarding each question item, guidance information, an answer example and a correction method. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

The present invention relates to a technique for automatically generating a multimodal application program that can be operated using both voice operation and operation on a screen.

  A GUI (Graphical User Interface) capable of operating an application on a screen is widely used in personal computers, car navigation systems, mobile phones, and the like. There are many tools and reference books for developing such application programs, and the number of developers is very large.

  On the other hand, multimodal application programs that can be operated using both voice operations and on-screen operations are not yet widespread, and their development tools are not yet widespread.

  In addition, GUI applications use an event-driven program format in which GUI components such as buttons are arranged on the screen and the operation of the system is described when these components are operated by the user. In this case, sequential processing based on the dialogue flow is required, which is a threshold for GUI application developers.

  For a tool that allows a user who cannot perform programming to create a multi-modal application, Patent Literature 1 is disclosed.

JP-A-9-251368

  However, with respect to the method disclosed in Patent Document 1, the user directly creates a timing chart or creates an operation log for learning the timing chart for the timings of a plurality of input / output modalities. It took man-hours.

  The present invention has been made in view of the above problems, and the present invention provides a mechanism for realizing a multi-modal application in cooperation with a GUI application and a voice interaction scenario, as well as the number of question items and the names of question items. Guidance for each question item, answer example for each question item, GUI application source code that realizes multimodal application just by giving information such as correction method at the time of misrecognition, dialogue scenario linked with GUI application, dialogue Generate a speech recognition grammar template to be used in the scenario.

  With the above method, even an application developer who is not familiar with the implementation method of a voice interaction application or a multimodal application can easily generate a basic model of a multimodal application, thereby improving the development efficiency.

  FIG. 1 is a diagram showing an embodiment of a configuration of a multimodal application according to the present invention. The multimodal application is described in the speech recognition engine 101 operating on the OS 100, the speech synthesis engine 102, one or more interaction scenarios 105, one or more speech recognition grammars 106, and the interaction scenario 105. In accordance with the contents, the dialogue control unit 103 that controls dialogue with the user by using the speech recognition engine 101 and the speech synthesis engine 102, and a GUI application 104 are configured.

As the OS 100, any existing OS that supports GUI and voice input / output can be used. Any existing engine can be used as the speech recognition engine 101 or the speech synthesis engine 102. The dialog control unit 103 can use a VoiceXML interpreter implemented based on the specification of VoiceXML 2.0 recommended by World Wide Web Consortium (hereinafter abbreviated as W3C). It is also possible to use a program that performs control. The interaction scenario 105 can be described based on the specification of VoiceXML 2.0, but may be described based on other formats. The speech recognition grammar 106 can be described based on the specification of Speech Recognition Grammar Specification (hereinafter abbreviated as SRGS), which is a W3C recommendation, but may be described based on other formats.
The GUI application 104 is an application in a format that allows the user to control the operation of the application by selecting and operating GUI components such as buttons, drop-down lists, and radio boxes with a pointing device. , C ++, Java (registered trademark), and other languages.
In order for the GUI application 104 and the dialog control unit 103 that operate in separate threads or processes to cooperate with each other and to allow the user to control the application in both GUI and voice dialog, the operation of the dialog control unit 103 is performed from the GUI application 104. It is necessary to provide a mechanism for controlling, or conversely, a mechanism for controlling the operation of the GUI application 104 from the dialogue control unit 103. In general, an instruction that the user can give to the system with a single utterance is higher in complexity than an instruction that the user can give to the system with a single GUI operation. That is, in the case of a GUI operation, even if a four-step operation such as “file” → “print” → “printer selection” → “OK” is necessary, “print to XXX printer” is used for voice utterance. You can give instructions with a single utterance. This means that the voice dialog requires more complicated control, so the latter mechanism in which the dialog control unit 103 controls the operation of the GUI application 104 with simple control is appropriate. .

Therefore, as an embodiment, a mechanism in which the dialogue control unit 103 controls the operation of the GUI application 104 will be described.
The dialogue control unit 103 controls the dialogue based on the description content of the given dialogue scenario 105. FIG. 2 is an example of the dialogue scenario 105 described in the VoiceXML format. Based on this dialogue scenario, the dialogue control unit 103 uses the speech synthesis engine 102 to output guidance to the user such as “Welcome to the Shinkansen reservation system” and “Please say the name of the station to depart”. Next, the dialog control unit 103 recognizes the user's utterance using the speech recognition engine 101 based on the description content of the speech recognition grammar 106 specified by the file name “departure.abnf”. When the speech recognition result 107 is notified from the speech recognition engine 101, if the user utters an appropriate station name, the process proceeds to the next dialog for asking the destination, and if the user does not utter <noinput> , “Speak like Tokyo Station” is output and voice recognition is performed again. As described above, the dialog control unit 103 can control the dialog independently of the GUI application 104.

  In a general GUI application, when a GUI component such as a menu or a button arranged on the screen is selected and operated by a pointing device, a process associated with the GUI component is executed. However, in the GUI application 104 of the present invention, when a GUI component is selected and operated, the voice recognition result when the user utters a voice command having the same meaning as the GUI operation to the dialogue control unit 103. Equivalent pseudo speech recognition result information 108 is transmitted. Thus, regardless of whether a voice command is issued or a GUI operation is performed, the dialogue control unit 103 can control the voice dialogue in the same manner.

  On the other hand, when a GUI operation is performed or when a voice command is uttered, the result of the operation needs to be reflected in the GUI application 104. For this purpose, the dialogue control unit 103 needs to have a function of providing the GUI application 104 with information regarding the progress of the dialogue. An event system can be used as a mechanism for providing this information. An API (Application Program Interface) for registering an event listener may be prepared in the dialog control unit 103, and an event 109 related to the progress of the dialog generated during the operation of the dialog control unit 103 may be notified to the GUI application 104.

  As an event related to the progress of the dialog, when the dialog scenario 105 is described in VoiceXML, an event indicating the start of the dialog, an event indicating the start of executing the form, an event indicating the start of executing a specific item in the form, An event indicating that a value is assigned to a defined field by a user input, an event indicating that a value of a variable defined in an interaction scenario is changed, an event indicating the end of an interaction, or the like can be considered. The GUI application may receive these events and perform processing such as changing the screen display based on the contents. For example, FIG. 3 is an example of a screen of the GUI application 104 that realizes a multimodal application in cooperation with the dialogue scenario of FIG. 2, and the value of the list box 301 corresponding to the selection of “departure station” is displayed in the field department. What is necessary is just to change when the event showing that the value was substituted by the input was notified. When a user utters a response to multiple items in a single utterance, an event indicating that a value is assigned to the field defined in the form by user input is repeated multiple times. By issuing the command, it is possible to obtain the same effect as when the GUI application is operated a plurality of times.

  In the dialogue scenario illustrated in FIG. 2, a dialogue in which question items are sequentially asked to the user is performed. In the GUI application 104 of FIG. 3, all three input items are displayed. However, considering that the dialog control unit 103 controls the dialog, the input that the dialog control unit 103 cannot accept is the GUI application 104. Therefore, in the situation where the input of the departure station is requested, it is desirable that the list boxes 302 and 303 and the buttons 304 and 305 be in the input disabled state as shown in FIG. Similarly, when the voice conversation scenario 105 and the speech recognition grammar 106 are described so that inputs related to a plurality of input items can be received simultaneously, only the GUI component of the input items received simultaneously in the GUI application 104. It is desirable that the input is enabled and other GUI components are disabled.

Next, an embodiment of a multimodal application generation wizard according to the present invention will be described with reference to the drawings.
FIG. 11 shows an embodiment of the flow of the multimodal application generation wizard according to the present invention. According to this flow, the source code, dialogue scenario, and speech recognition grammar format to be generated will be explained.

  When the multimodal application generation wizard starts executing (S01), an application definition file in which information necessary for generating an application is written is read (S02). The application definition file must include at least three types of information: (1) the number of question items, (2) the name of each question item, and (3) typical answer examples for each question item.

  The application definition file may optionally include (4) information on whether or not to allow correction of the previous input, and (5) guidance information on each question item.

  4 (a) and 4 (b) are diagrams showing an example of a dialogue sequence of a multimodal application generated by the multimodal application generation wizard according to the present invention. In this example, (1) departure station name (departure), (2) arrival station name (destination), and (3) seat class (class) are items to be input by the user. Both are in a format that requires the user to input each input item in turn. The difference between (a) and (b) is whether or not the speech recognition engine 101 can correct the previous input on the spot when the recognition result is incorrect or the user makes an input error. Which type of dialog is generated is determined by the information in (4) described in the definition file. If no option is given, one of the forms of interaction can be created by default.

  Next, the speech recognition grammar 106 for the first question item is generated (S03). Since there is no possibility of correct utterance in the first question item, it is only necessary to generate a speech recognition grammar having the same format for both of the dialog formats shown in FIGS.

  For generation of the speech recognition grammar 106, information on typical answer examples for each question item in (3) can be used. When a plurality of combinations of pronunciation and notation are given as answer examples, a speech recognition grammar 106 as shown in FIG. 6A can be generated. This example of speech recognition grammar uses a notation method based on the ABNF format defined by SRGS recommended by W3C. The information provided by the developer is used in the form of “pronunciation {name of question item =“ notation ”}”. When only one pronunciation notation set is given, a speech recognition grammar 106 as shown in FIG. 6B can be generated. By generating such a speech recognition grammar 106, it is possible to inform the developer what kind of description should be made when the number of utterance examples is increased later.

  Next, the process is distributed depending on whether or not a dialog that can correct the previous input is generated (S04). First, an interactive application generation flow shown in FIG. 4A in which the previous input cannot be corrected will be described.

  In the case of the interactive format shown in FIG. 4A, it is not necessary to accept a corrected utterance, and the generated speech recognition grammar 106 may be generated in the format shown in FIG. 6 for all question items. Furthermore, the speech recognition grammar of the recognition vocabulary “Yes” and “No” is output as the speech recognition grammar 106 for final confirmation (S05).

  Next, a dialogue scenario 105 is generated (S06). FIG. 5 shows an example of an interaction scenario 105 constituting the interactive multimodal application shown in FIG. 4A generated by the multimodal application generation wizard according to the present invention, and is described in the VoiceXML format. In this description method, the number of question items in (1) can be handled as the number of <field> elements included in the generated VoiceXML document. The name of each question item in (2) can be treated as the value of the name attribute of each <field> element and the name of the speech recognition grammar 106 to be referred to. The voice guidance for each question item in (5) can be handled as the contents of the <prompt> element in each <field> element. It is sufficient to output the character string “guidance for prompting utterance” so that the developer can easily modify it later. In addition, a description example of guidance voice when there is no utterance, an example of confirmation guidance after all question items have been entered, and each processing when the confirmation is affirmative and negative By automatically generating examples, developers' man-hours can be reduced. As an example of confirmation guidance, it is effective to let the user hear the contents of each question item entered, so it is desirable to automatically generate an example of guidance text that refers to each input content using the <value> element. Developers can also modify the guidance at their own discretion.

  Next, the source code of the GUI application 104 is generated (S07). FIG. 7 shows an example of the source code of the GUI application 104 generated by the multimodal application generation wizard according to the present invention, which is written in Java.

  FIG. 7A shows a code for generating a GUI component corresponding to the question item. With the performance of current speech recognition systems, it is difficult to support free input such as a text box in the GUI, and even if speech recognition can be done correctly, understand the content and proceed with the dialogue. It is also difficult. Therefore, as the GUI component corresponding to each question item, a type that selects from a plurality of predetermined options, that is, a drop-down list, a radio box, or the like is suitable. You can ask the developer which GUI component to use, or ask the developer for the number of options, use the drop-down list if there are many, use the radio box if there are few, etc. Good. In the example of FIG. 7A, a drop-down list is generated. The name of the question item in (2) is used for the title of the label component explaining the drop-down list. Also, typical answer examples for each question item in (3) use reading and notation information. In the speech recognition grammar, multiple readings may be prepared for one option as shown in FIG. 6 (a). However, since the GUI component is redundant, only one answer is selected for the same answer example. It is desirable to use it. As many GUI component generation codes as (1) are generated.

  FIG. 7B is an example of source code representing registration of these GUI components and operations when the GUI components are operated. In general, the processing when the GUI component is operated differs depending on each application, so automatic generation is difficult, but here, only the code for notifying the dialogue control unit 103 of the pseudo speech recognition result is generated. is doing. As a result, even when a GUI operation is performed, it is possible to make the conversation proceed as in the case of operating with a voice. Whether the real speech recognition result is obtained or the pseudo speech recognition result is given, the dialogue control unit 103 may substitute a value into an appropriate field according to the VoiceXML specification based on the speech recognition result. The speech recognition result can be expressed by a description method based on SISR (Semantics Information for Speech Recognition) which W3C is standardizing. For example, for a question item whose name is “departure”, if the input “Tokyo” is made by GUI or voice, it will be in the form of a category / value pair to which the input content belongs and “{departure:“ Tokyo "}". Also in the source code of FIG. 7B, a character string expressing the above speech recognition result is generated for the item selected by the GUI. By using the format of category information and its value pairs, it becomes clear which question item is input regardless of the input method, and when multiple question items are input by voice input However, there is an advantage that it can respond.

  When a value is assigned to a field in the processing of the dialogue control unit 103, a corresponding event is notified to the event handler of the GUI application 104. FIG. 7C shows an example of an event handler to be generated. The dialogue control unit 103 notifies the field name to which the value is assigned and the assigned value. Since the question item name given by the developer is used as the field name, the question item name can be used here to determine which question item has been input and generate a code for distributing the process. . The process for input to each question item is a process of changing the selection item of the GUI component according to the notified input content (choice1.select (argValue)), and the GUI component can be input according to the progress of the dialogue / A code (setDialogStatus (1)) that calls the code that changes the impossible state may be generated.

  FIG. 7D shows the substance of the code that changes the input enabled / disabled state of the GUI component in accordance with the progress of the dialogue. In the dialog shown in FIG. 4 (a), it is desirable that all items other than the currently questioned item are in an input-impossible state. In response to this, a code for enabling input of the GUI component corresponding to the current question item may be generated.

  Next, the flow of the wizard when the developer instructs the wizard to allow the utterance to correct the input content immediately before and the GUI operation will be described.

  In the interactive form of FIG. 4 (b), after the departure station is input, the multimodal application is set so that the following four operations are allowed for the user when the input of the arrival station is requested. It is desirable to generate That is, (A) Select the arrival station (current question item) by voice operation, (B) Correct the departure station (immediate entry item) by voice operation, (C) Specify the arrival station (current question item) by GUI operation Select (D) Departure station (immediate input item) by GUI operation. The same applies to the following dialogue sequence.

First, a voice conversation scenario 105 that realizes (A) and (B) is generated (S08). FIG. 8 is an example of a dialogue scenario 105 automatically generated by the dialogue wizard according to the present invention. The number of <field> elements, the name of each element name attribute, the content of <prompt>, and the like are generated from information input by the developer, as in the example of the interactive scenario 105 in FIG. The difference from FIG. 5 is that the previous input content is presented to the user, the speech recognition grammar that accepts the corrected utterance in addition to the current question item is specified, and whether the input to the current question item or the corrected utterance is This is the part that performs processing accordingly. Specifically, in addition to the question item provided by the developer as the content of <prompt>, a sentence that confirms the previous input content (<value expr = ”name of the immediately previous question item” />?>) Automatically It is desirable to generate. Also, by setting the name of the speech recognition grammar 106 to be referred to as “current question item name_correct.abnf”, it is easy to understand that the correction is allowed when the developer sees the generated dialogue scenario. Become. In order to determine whether the user's utterance is an input to the current question item or a correction to the previous input, “{destination:“ Osaka ”}, which is the expression format of the speech recognition result already described, is used. So, there is not enough information, so we use a form like "{destination: {correct:" true "departure:" Nagoya "}}" as a representation of the correct utterance. In this example, in the step of inputting the destination, the correction utterance is performed, and the content of the previous department is corrected to Nagoya. On the other hand, the expression format representing the input to the current question item is “{destination: {correct:“ false ”destination:“ Tokyo ”}}”. Within each <field> element that allows correct utterance, the correct content is determined using the <if> element, and if it is determined to be corrected utterance, the previous input content is determined using the <asign> element. If it is determined that it is not a corrected utterance, it is automatically rewritten with a variable called `` Current Question Item Name.Previous Question Item Name '' and clears the field variable of the current question item that has not yet been entered. It is desirable to automatically generate a description for rewriting the value of the field variable of the current question item with the value of the variable “current question item name.current question item name”.
In the example of FIG. 8, the last “class” cannot be corrected in the final confirmation step, but it is desirable that the developer can select whether or not to allow correction utterance in the final confirmation step. When correction utterances for the input content immediately before in the final confirmation step are allowed, the name of the speech recognition grammar 106 to be referred to is set to an easily understood name such as “yesno_correct.abnf”, and it is determined whether or not the recognition result is a corrected utterance. <If> elements to be generated can be automatically generated.

Next, the speech recognition grammar 106 that allows a corrected utterance to the input content immediately before will be described. As a correction utterance for the input content immediately before, a phrase such as “Nagoya, not Osaka” can be considered. At this time, it is conceivable to automatically generate a speech recognition grammar 106 that can accept a phrase such as “yyyy instead of xxx” for all patterns. However, in this case, even if the input content immediately before is “Osaka”, the utterance such as “Nagoya instead of Tokyo” is accepted, and the size of the speech recognition grammar 106 is enough to accept all patterns. It's not optimal because it grows. As another method, a method of dynamically determining the name of the speech recognition grammar 106 to be referred to from the immediately preceding input content can be considered. For example, in the speech recognition grammar specification method described in W3C's VoiceXML 2.1 Working Draft, by writing “<grammar srcexpr =" departure + '_correct.abnf'"/>" For the input, the speech recognition grammar 106 “Osaka_correct.abnf” can be referred to in the next question item step. This method is advantageous because it can eliminate inappropriate language. As another method, the name of the speech recognition grammar 106 to be referred to is fixed to “current question item name_correct.abnf”, and the GUI application 104 automatically generates the speech recognition grammar 106 having this file name. Conceivable. Here, in order to explain based on the last automatic generation method, the next flow of the multimodal generation application is generation of source code for automatically generating a speech recognition grammar (S09).
FIG. 9 is an example of a speech recognition grammar 106 that is automatically generated by the GUI application 104 and that takes into account the previous input content. It is described in the SRGS format so as to obtain a recognition result in a format in which it can be determined whether or not the correction utterance is already explained. The structure of the grammar is the same regardless of the previous input content, and the part that changes depending on the previous input content is that the first word of $ correct is the previous input content and the correction candidate at the end of $ correction The previous entry is excluded from the list. In the wizard according to the present invention, the code for generating the speech recognition grammar 106 as described above can be automatically generated in the GUI application 104 by giving the reading and notation of the immediately preceding input content (S09).

Next, automatic source code generation (S10) of the GUI application 104 that can correct the input content immediately before and can cooperate with the dialogue scenario 105 will be described.
FIG. 10 shows an example of source code of the GUI application 104 automatically generated by the wizard according to the present invention. In FIG. 10 (a), a drop-down list is generated as in the example of FIG. 7 (a). The name of the question item in (2) is used for the title of the label component explaining the drop-down list. Also, typical answer examples for each question item in (3) use reading and notation information. FIG. 10B shows the first question item, and FIG. 10D shows the event handler when the GUI component for the subsequent question items is operated. The variable “state” is a variable that indicates the progress of the dialog, and even if the same GUI component is operated, it is distinguished whether the input is for the current question item or a correction input after moving to the next step. can do. Since FIG. 10B is an event handler for the first question item, there is no content to be corrected when it is currently in the step of the first question item (state = 0), so “{departure:“ Osaka ”} If the code for generating the recognition result expression like "" is automatically generated and the process moves to the next step (state = 1), "{destination: {correct:" true "departure:" Nagoya "}} As shown in Fig. 10 (d), a code for generating a recognition result expression similar to that in the case where the corrected utterance for the immediately preceding input content is performed in the next step may be automatically generated. The recognition result expression representing the input may be automatically generated in a description format that can determine whether the input is a correct input such as “{destination: {correct:“ false ”destination:“ Tokyo ”}}. Event handler code when a value is assigned to the <field> element of 10 (c) It is only necessary to automatically generate almost the same code as that shown in FIG. 7C, but the difference is that the code (makeDic () that calls the code for automatically generating the speech recognition grammar 106 in the next step depends on the input contents. 10 (e) is a code for realizing the restrictions (c) and (d) related to the user operation described above, and the wizard displays the current question item. In order to do so, the code that prevents the GUI component other than the previous question item from being operated can be automatically generated.For this, in the case of the first question item (step = 0), only the GUI component for the first question item is required. The code for enabling is automatically generated, and in the subsequent states, the code for enabling only the current question item and the immediately preceding question item may be automatically generated.

  The present invention can be used for developing a program for an information device equipped with a voice interaction function and a GUI operation function, such as a personal computer, a PDA, and a car navigation system.

The figure showing one Example of a structure of a multimodal application. The figure showing one Example of the example of a description of a dialogue scenario. The figure showing the example of the position of the screen of a GUI application. The figure showing one Example of the interaction sequence of the multimodal application produced | generated automatically. The figure showing one Example of the interactive scenario of the multimodal application produced | generated automatically. The figure showing one Example of the speech recognition grammar of the multimodal application produced | generated automatically. The figure showing one Example of the GUI application source code of the multimodal application produced | generated automatically. The figure showing one Example of the interactive scenario of the multimodal application produced | generated automatically. The figure showing one Example of the speech recognition grammar automatically produced | generated by the GUI application of the multimodal application produced | generated automatically. The figure showing one Example of the GUI application source code of the multimodal application produced | generated automatically. An example of the automatic generation flow of a multimodal application.

Explanation of symbols

100 ... OS
DESCRIPTION OF SYMBOLS 101 ... Speech recognition engine 102 ... Speech synthesis engine 103 ... Dialog control part 104 ... GUI application 105 ... Dialog scenario 106 ... Speech recognition grammar 107 ... Speech recognition result 108 ... -Pseudo-voice recognition result 109 ... Event 301 indicating the progress of dialogue ... Drop-down list 302 that is currently input target ... Drop-down list 303 that is not currently input target ... Not currently input target Drop-down list 304 ... Button that is not currently input 305 ... Button that is not currently input.

Claims (6)

  At least a GUI application, a voice dialogue control unit that controls voice dialogue based on a voice dialogue scenario, and a voice recognition unit are configured. The GUI application transmits a user operation content to the voice dialogue control unit. Notification is made as pseudo speech recognition result information in the same format as that outputted as a speech recognition result, and the speech dialogue control unit receives pseudo speech recognition result information from the GUI application when the speech recognition result is obtained from the speech recognition unit. If obtained, the multi-modal dialogue system is characterized by notifying the GUI application of information regarding the progress of the dialogue.   2. The multimodal dialog system according to claim 1, wherein the format shared by the speech recognition result and the pseudo speech recognition result information includes at least information on category name / value pairs.   The information on the progress status of the dialog notified to the GUI application when the voice recognition result is obtained from the voice recognition unit and the pseudo voice recognition result information is obtained from the GUI application according to claim 1. A multimodal dialog system characterized by adopting a GUI application operation sequence format. Receive at least information about the number of question items, each question item name, and a list of typical answers to each question item as input,
A source code of a program of a GUI application including a process of transmitting the pseudo voice recognition information to the voice dialogue control unit when an answer to the plurality of question items is input through the GUI interface;
A voice interaction scenario for allowing a user to answer the plurality of question items;
Generating a speech recognition grammar including a typical answer example for each of the plurality of question items,
Multimodal application generation wizard program.   5. The multi-modal application generation wizard program according to claim 4, wherein when a corrected utterance of the previous speech recognition result and an utterance for the current input item are received, a speech recognition grammar that can distinguish the corrected utterance from the input utterance is dynamically generated. A multi-modal application generation wizard program, wherein the source code is generated by including the code to be generated. 5. The multi-modal application generation wizard program according to claim 4, wherein the source code includes a code for prohibiting operation of a GUI component corresponding to an input item that cannot be input by a user according to progress of the voice dialogue scenario. A multi-modal application generation wizard program characterized by generating

Images