JP2005339513A - Information processor, control method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an information processor and a control method and program therefor capable of easily making various pieces of equipment function as operation equipment and control object equipment. <P>SOLUTION: Modality information for modality of the operation equipment is received. Interaction information for interaction of the control object equipment that is a control object of the operation equipment is received. Bind information conforming the modality information to the interaction information is generated. The bind information and the interaction information are transmitted to the operation equipment. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an information processing device that supports control between an operation device and a control target device via a network, an information processing device that operates as an operation device that controls the operation of the control target device, and an instruction from the operation device. The present invention relates to an information processing apparatus that operates as a control target device that executes processing, a control method thereof, and a program.

  Web browsing for browsing content written in a markup language called HTML using a browser is now a technology that has become popular worldwide. HTML is a markup language for displaying content. HTML has a mechanism called a form, and can have GUI parts such as buttons and text boxes. With this and a mechanism called CGI (Common Gateway Interface) of the Web server, it means not only browsing but also a client (a computer and software using functions and data provided by the server in the network. This client. For example, a computer connected to a network, a home personal computer, and a Web browser or viewer operating on the computer). Can be executed, and the result can also be returned to the client in the form of HTML. For example, a Web search engine is generally realized by this method.

  This mechanism can be applied not only to Web browsing but also to device control. That is, a Web server is mounted on the control target device, and an HTML file including a form including a GUI component for controlling itself is transmitted to the operation device in response to a request from the operation device as a client. The operating device displays this HTML file on the browser, and the user operates the GUI on the operating device. The operating device transmits the input to a control target device (for example, a Web server). In the device to be controlled, this input is transferred to the control program by the CGI mechanism, and control according to the input is performed.

  On the other hand, in recent years, the form of information devices has been diversified, such as PDAs, mobile phones, car navigation systems and other portable terminals, and it has become possible to connect to the Internet from such devices other than personal computers (hereinafter referred to as PCs). Yes. Along with this, markup languages replacing HTML such as WML have been developed and standardized. In addition, with the development of speech recognition / synthesis technology and the development of CTI technology, it becomes possible to access the Web by voice conversation via the telephone. With this, markup languages such as VoiceXML have been developed and standardized. Yes. Thus, a markup language suitable for a device form has been developed and standardized.

  In addition to diversification of device forms, UI modalities are diversified, such as GUIs for PCs and PDAs, and voice and DTMF for telephones. There is a growing interest in multimodal user interfaces that combine these diversified modalities to improve operability. The description of the multi-modal user interface includes a dialog description (description showing what input is received from the user, what output is made, and in what procedure the input / output proceeds) and a modality description (what UI components are included). At least a description indicating whether to use that input / output).

  The modality description is highly dependent on the client configuration. A general-purpose device such as a PC has abundant GUIs, and recently, some have a speech UI due to the development of speech recognition / synthesis technology. On the other hand, it is best to use voice for mobile phones. This is because a mobile phone supports a simple GUI component on a small liquid crystal screen, but does not have a pointing device and cannot always be said to be easy to use. In view of device control, there is a remote controller as an operation device. The remote control is generally operated by physical buttons.

  There are two kinds of methods for such dialog description and modality description.

  One is a method of clearly describing the modality input / output form in the dialog description, such as a certain input is a GUI button and a certain output is a voice. The other is a method of separating the dialog description and the modality description and describing the dialog description in a modality-independent manner.

  In the latter method, a dialog description, which is an operation procedure of a device to be controlled, is performed in a modality-independent manner, and in a form separated from this, for each client, a modality description adapted to the client is performed. , It becomes possible to operate one control target device from various clients. There is CML (Patent Document 1) as a markup language of a multi-modal user interface capable of separately describing dialog description and modality description.

  In CML, the dialog description itself is described in a modality independent form, and there is no framework for performing the modality description itself and its control description. A modality is generated by converting the dialog description part into an existing markup language such as HTML, WML, VoiceXML or the like. Alternatively, when the browser directly executes CML, the browser is assumed to have a known modality of the mounted device, and the specification is left to the browser which modality is associated with which input / output element in the dialog description.

  Japanese Patent Laid-Open No. 2004-228867 separates input / output elements of a dialog description as a logical UI and modalities of a modality description as a concrete UI, and dynamically associates the concrete UI with a logical UI of a control target device according to a client. Has proposed.

In the future, No-PC will progress, and it is expected that all types of devices will be equipped with CPUs and communication functions, and will cooperate with each other via a network to improve user convenience. From the UI point of view, it is not a dream to realize a device operating environment that does not select the type and location of devices, such as operating home appliances and vending machines instead of remote controls for mobile devices such as mobile phones and digital cameras. A Web mechanism based on the above markup language is effective in realizing such a device operating environment. Furthermore, a markup language that enables modality-independent description is effective for realizing a device operating environment where the type and location of the device are not selected.
JP 2001-154852 A JP 2001-217850 A

  As described above, HTML, WML, VoiceXML, and the like, which are existing markup languages, are modality-dependent languages that assume an operating device form. Therefore, in order to enable control from various types of devices, the control target device must prepare a plurality of types of markup languages in accordance with the assumed operation device. In addition, HTML, WML, VoiceXML, and the like are not suitable for the realization of a multimodal user interface because operations that combine multiple modalities are not assumed.

  Further, although CML is a modality-independent markup language, the method of converting to an existing markup language such as HTML, WML, VoiceXML, etc. is not suitable for realizing a multimodal user interface as described above. On the other hand, when the browser directly executes CML, the browser is assumed to have a known modality of the installed device, and the specification is left to the browser which modality is associated with which input / output element in the dialog description. Therefore, the correspondence between modalities and input / output elements is fixed, and there is no flexibility that can be controlled by description in a markup language.

  Furthermore, Japanese Patent Laid-Open No. 2001-217850 provides a very simple connection between a logical UI and a concrete UI, for example, a logical UI of a type selected from options is connected to a concrete UI such as a radio button or a pull-down menu. Is doing. In this case, there is some restriction on the concrete UI that can be associated with the logical UI because it is not possible to associate a physical button with each selection item of the option or to select an item by repeatedly pressing one button. There is a problem of end.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to provide an information processing apparatus capable of easily causing various devices to function as operation devices and control target devices, a control method therefor, and a program. And

In order to achieve the above object, an information processing apparatus according to the present invention comprises the following arrangement. That is,
First receiving means for receiving modality information relating to the modality of the operating device;
Second receiving means for receiving dialogue information relating to a dialogue of a control target device that is a control target of the operation device;
Generating means for generating binding information for associating the modality information with the dialogue information;
Transmitting means for transmitting the binding information and the dialogue information to the operating device.

Preferably, the modality information, the dialogue information, and the bind information are described in a markup language.
The markup language is
A dialog description indicating the content of the dialog of the control target device;
A modality description indicating a modality of the operating device, which is configured separately from the dialog description;
It includes a bind description that links the modality description and the dialog description.

  Preferably, the dialog description is composed of one or a plurality of minimum dialog elements which are input / output processing units, and the minimum dialog elements include zero or one input element and zero or one. The above output elements are included.

  Preferably, the modality description includes a hierarchical relationship between modality classes and modality instances, and the definition of the modality of the operating device refers to a general-purpose modality description describing a general-purpose modality definition, wherein Defined as a subclass and instance of the defined generic modality.

  Preferably, the binding description is a description for binding one or more modalities to one or more input elements, and when binding a plurality of modalities to one input element, Includes a description of how to combine modalities.

  Preferably, the binding description is a description for binding one or more modalities to one or more output elements, and when binding a plurality of modalities to one output element, Includes a description of how to combine modalities.

Preferably, the operating device includes a management unit that manages the modality according to the modality description;
Modality control means for controlling processing related to the modality and the dialogue according to the binding description,
The control target device includes dialog execution means for managing the dialog according to the dialog description;
Control means for executing control of the device to be controlled based on an instruction from the dialogue execution means.

Preferably, the apparatus further comprises storage means for storing a bind sample including the general-purpose modality, input / output elements that can be linked to the general-purpose modality, and description of appropriateness information of the binding,
The generation unit generates bind information that associates the modality information with the dialogue information with reference to the bind sample.

In order to achieve the above object, an information processing apparatus according to the present invention comprises the following arrangement. That is,
An information processing apparatus that operates as an operation device for controlling a control target device,
Receiving means for receiving dialogue information relating to dialogue of the control target device;
Generating means for generating the binding information that associates the modality information related to the modality of the information processing apparatus and the dialogue information based on the appropriateness information indicating the appropriateness of the association;
Modality control means for controlling processing related to the modality and the dialogue according to the binding information.

In order to achieve the above object, a method for controlling an information processing apparatus according to the present invention comprises the following arrangement. That is,
A first receiving step of receiving modality information related to the modality of the operating device;
A second receiving step of receiving dialogue information relating to a dialogue of a control target device that is a control target of the operation device;
A generation step of generating binding information that associates the modality information with the dialogue information;
A transmission step of transmitting the binding information and the dialogue information to the operating device.

In order to achieve the above object, a method for controlling an information processing apparatus according to the present invention comprises the following arrangement. That is,
A method for controlling an information processing apparatus that operates as an operation device for controlling a control target device,
Receiving step for receiving dialogue information related to dialogue of the control target device;
A generation step of generating bind information that associates the modality information related to the modality of the information processing apparatus and the dialogue information based on appropriateness information indicating the appropriateness of association;
A method for controlling an information processing apparatus, comprising: a modality control step of controlling processing related to the modality and the dialogue according to the binding information.

In order to achieve the above object, a program according to the present invention comprises the following arrangement. That is,
A program for causing a computer to control an information processing apparatus capable of communicating with an operation device and a control target device that is a control target of the operation device,
A program code of a first receiving step for receiving modality information relating to the modality of the operating device;
Program code of a second receiving step for receiving dialogue information relating to dialogue of the control target device;
A program code of a generation step for generating binding information that associates the modality information with the dialogue information;
And a program code of a transmission step for transmitting the binding information and the dialogue information to the operating device.

In order to achieve the above object, a program according to the present invention comprises the following arrangement. That is,
A program for causing a computer to function as a control device for an information processing device that operates as an operation device for controlling a control target device,
Program code of a receiving step for receiving dialogue information related to dialogue of the control target device;
A program code of a generating step for generating bind information that associates the modality information related to the modality of the information processing apparatus and the dialogue information based on the appropriateness information indicating the appropriateness of association;
Program code of a modality control step for controlling processing related to the modality and the dialogue according to the binding information.

  According to the present invention, it is possible to provide an information processing apparatus capable of easily causing various devices to function as an operation device and a control target device, a control method therefor, and a program.

Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the drawings.
[Embodiment 1]
In the first embodiment, as shown in FIG. 1, an example in which a copier is operated from a mobile phone via a network will be described.

  FIG. 1 is a configuration diagram of an information processing system according to the first embodiment of the present invention.

  In the figure, reference numeral 101 denotes a copier that is one of the control targets. Reference numeral 102 denotes a mobile phone used as an operation device by the user.

  Reference numeral 103 denotes a bind inference server that infers how to connect the modality of the operation device and the input / output element of the dialog description of the control target device, and automatically generates a bind layer.

  Reference numeral 104 denotes a network such as the Internet, a dedicated line, a wireless network, and an optical fiber network. Reference numeral 105 denotes an air conditioner that is one of the controlled objects. A digital camera 106 is used as an operation device by the user.

  1 can interpret at least a markup language and has a function of executing various processes based on the interpretation.

  Next, the configuration of the copy machine 101 will be described.

  FIG. 2 is a functional configuration diagram of the copier according to the first embodiment of the present invention.

  In the figure, reference numeral 1201 denotes a dialog execution unit which operates according to the description of the dialog layer of the markup language. Reference numeral 1202 denotes a device control unit. If there is an instruction related to device control from the dialogue execution unit 1201, device control is executed based on the instruction (for example, copying is executed in accordance with an instruction “copy”). A communication unit 1203 communicates with various devices via the network 104. 1204 is a dialogue markup language (ML) describing a dialogue layer.

  FIG. 3 is a block diagram showing the hardware configuration of the copier according to the first embodiment of the present invention.

  In the figure, reference numeral 201 denotes a CPU, which operates according to a program that realizes a flowchart to be described later. A RAM 203 provides a storage area, a work area, and a data temporary save area necessary for the operation of the program. Reference numeral 202 denotes a ROM which holds programs and various data for realizing a flowchart described later. Reference numeral 204 denotes a disk device, which holds various data such as an interactive markup language 1204, for example. Reference numeral 205 denotes a bus that interconnects the above-described components.

  Next, the configuration of the mobile phone 102 will be described.

  FIG. 4 is a functional configuration diagram of the mobile phone according to the first embodiment of the present invention.

  In the figure, reference numeral 1301 denotes an input / output management unit which manages data input / output and voice input / output. Reference numeral 1302 denotes a voice recognition unit which recognizes input voice. A voice synthesizer 1303 synthesizes voice output target data. A modality management unit 1304 operates according to the description of the modality layer of the markup language, and manages modalities (UI parts). Reference numeral 1305 denotes a modality control unit which operates according to the description of the markup language bind layer. A communication unit 1306 communicates with various devices via the network 104. Reference numeral 1307 denotes a modality markup language (ML) describing a modality layer.

  FIG. 5 is a configuration diagram showing a hardware configuration of the mobile phone according to the first embodiment of the present invention.

  In the figure, reference numeral 301 denotes a CPU, which operates according to a program that realizes a flowchart to be described later. A RAM 303 provides a storage area, a work area, and a data temporary save area necessary for the operation of the program. Reference numeral 302 denotes a ROM that holds programs and various data for realizing flowcharts to be described later. Reference numeral 304 denotes a microphone that performs voice input to the voice recognition unit 1302. Reference numeral 305 denotes a liquid crystal display (LCD). Reference numeral 306 denotes a speaker, which outputs a synthesized sound by the voice synthesis unit 1303. Reference numeral 307 denotes a physical button for executing various operations. Reference numeral 308 denotes a bus that interconnects the above-described components.

  Next, the configuration of the bind layer inference server 103 will be described.

  FIG. 6 is a functional configuration diagram of the bind layer inference server according to the first embodiment of the present invention.

  In the figure, reference numeral 1401 denotes a bind layer inference unit. Reference numeral 1402 denotes a bind sample indicating a sample bind description used for inference in the bind layer inference unit 1402. A communication unit 1403 communicates with various devices via the network 104.

  FIG. 7 is a configuration diagram showing a hardware configuration of the bind layer inference server according to the first embodiment of the present invention.

  In the figure, reference numeral 401 denotes a CPU, which operates according to a program for realizing a flowchart to be described later. A RAM 403 provides a storage area, a work area, and a data temporary save area necessary for the operation of the program. Reference numeral 402 denotes a ROM which holds programs and various data for realizing a flowchart described later. Reference numeral 404 denotes a disk device that holds various data such as a bind sample 1402. Reference numeral 405 denotes a bus that interconnects the above-described components.

  Before describing a specific operation example, a specification example of the multimodal user interface markup language (hereinafter referred to as MMML) of the present invention will be described.

Three-layer structure In the present invention, as shown in FIG. 8, a modality description depending on a device and a dialog description which is UI logic are described separately. In particular, in addition to a modality layer 801 indicating a modality description (portion enclosed by tags <modality> to </ modality>), an interaction layer 803 indicating an interaction description (portion enclosed by tags <dialog> to </ dialog>), It has a three-layer structure of a bind layer 802 that indicates a description (a portion enclosed by tags <Binds> to </ Binds>) that links a modality and a dialogue.

  Thus, by separating the modality description, it is possible to share one dialog description (UI logic) among a plurality of devices having different modalities. A specific example of this three-layer structure is shown in FIG.

  FIG. 8 shows the structure of a description (Mobile phone modalities) that can be operated from a mobile phone in addition to a copy machine modality description (Copier modalities) indicating the operation of the copy machine 101. As shown in this example, if one copier dialogue description (Copier Dialog) is configured, it is possible to perform operations from various devices only by describing a device-dependent modality layer and a bind layer for each device.

Modality class and instance In the description of the modality layer in MMML, it is difficult to predefine all vocabularies representing modalities, that is, UI components as MMML definition words, and lack of extensibility. Therefore, several types of UI component classes that are expected to be generally used are defined as MMML common vocabulary, and a new class can be defined by inheriting this class. Furthermore, UI parts used in actual UI descriptions can be defined in the form of instances of such classes. In the first embodiment, an RDF schema is used as a method for describing class definition, inheritance, hierarchical relationships between classes, and instance definitions in XML. The RDF schema is a markup language (see http://www.w3.org/TR/rdf-schema/) that is standardized by the W3C (see http://www.w3c.org), a Web standards organization. It is.

General Modality Class Schema Several GUI component classes that are expected to be commonly used are defined as MMML vocabulary in the form of an RDF schema. FIG. 9 shows an example in which this RDF schema (general-purpose modality schema) is schematically represented. As shown in FIG. 9, UI component classes can be defined hierarchically using the RDF schema. For example, “Button” can define “Physical Button” and “GUI Button” as subclasses. The former is, for example, a button of a copier, and the latter is, for example, a button displayed on a liquid crystal panel. Also, the XML representation of this RDF schema is as shown in FIGS. 19A to 19E.

Modality schema of operating device (client) Based on a general UI component class defined as a common vocabulary of MMML, a schema that defines an operating device-dependent UI component class and an instance is described. That is, this modality schema describes what modalities can be used in the operating device, and is assumed to be provided by the manufacturer of the operating device. For example, in contrast to the Generic Modalities Schema, a schematic example of a mobile phone modality schema corresponding to the mobile phone UI components (LCD, physical buttons, voice input, voice output, etc.) shown in FIG. As shown in FIG. The XML representation of this modality schema is as shown in FIGS. 20A and 20B.

Dialogue layer This is a layer for performing concrete dialog description, and description independent of modality is possible. In the MMML description, the dialog layer is described inside tags <dialog> to </ dialog>. The minimum unit of interaction is field (minimum interaction unit). The field has input / output elements such as input and output and an element called filled.

  Each of input and output is a tag that describes what information is input / output, and has an ID as an attribute. This ID is used when describing the binding with the modality described separately. This will be described later.

Describe the type of input you want to accept in the field to which the Input element belongs. The type of input is described in the type attribute. As attribute values, the following are pre-defined as MMML common vocabulary.

"SelectMe"
This is a type of input that accepts the fact that there was an input. Modality such as “button” and voice can be bound.

"SelectOne"
This is a type of input for selecting one of the options. For example, radio buttons and pull-down menus are typical modalities. The selectOne type input describes an option with an item tag. It is also possible to bind modalities such as “buttons” to each item.

"SelectMany"
This is a type of input for selecting a plurality of options. For example, a combo box (ComboBox) is a typical modality. Like the selectOne type, the option is described with the item tag.

"TextString"
This type of input accepts input of character strings. For example, GUI text boxes and voice input are typical modalities.

Describes the output in the field to which the Output element belongs. The output contents can be described with the content tag.

filled element Describes the action (action information) for the input in each field. There are various types of actions. This action includes interacting with an internal program other than the UI by script description or some description method (for example, receiving a “copy start” input and starting a processing program that actually performs copying). A typical action. Moreover, when responding to an input, an output tag can be described in filled.

Role of fields
Group fields. In general, UI logic is divided into several groups, and there are many cases where UI logic is defined in units of groups. Therefore, it is convenient to appropriately group fields (for example, the initial screen of a copier). And each setting mode).

Example of Copier Interaction Layer FIG. 12 shows a description example of the interaction layer of the copy machine UI, and FIG. 21A and FIG. 21B show its XML representation.

  In this example, a UI for executing copy (CopyStart), paper selection (PaperSize), and single-sided / double-sided (PrintSides) is described. In this example, modality description, that is, what kind of UI parts to use, such as whether to input with buttons, input with voice, display with text, notify with voice, etc. Note that it is not described (modality independent).

Since the modality is excluded in the dialog description independent of the bind layer modality, it is necessary to associate the modality with the dialog description. The binding layer is a layer that describes the association, and binds each modality defined in the modality layer to each input / output element in the interaction layer. The description of the association is performed by referring to the ID of the modality to be associated and the ID of the input / output element with a URI (Uniform Resource Identifier). The bind layer is described inside the tags <Binds> to </ Binds>. Each bind element is described inside tags <Binds> to </ Binds>. By binding a plurality of modalities to one input / output element of the interactive layer, it is possible to describe a multimodal UI using the selective use of multiple modalities and the cooperative use. In the bind layer, the following description about modality management is possible.

-Description of how to combine multiple modalities-Description of output contents-Description of activate / deactivate of modalities
Description of how to combine multiple modalities When binding multiple modalities to one input / output element, you can specify how to combine modalities such as selective use and cooperative use. The combination instruction is described by the BindType attribute. As attribute values of the BindType attribute, the following are predefined as MMML common vocabulary.

"Alt"
Any one of a plurality of modalities bound to input / output elements can be used. The priority of multiple modalities shall be in the order described. For example, when the radio button modality and the voice input modality are bound with the “Alt” attribute to the input element for selecting “A4 or B5” on the copier, this is the case of selection by radio button or voice input. Either means that the paper can be selected.

"Seq"
Apply multiple modalities bound to input / output elements in order. For example, it corresponds to an input such as “speak after pressing button A”.

"Coordinated"
Coordinate multiple modalities bound to input / output elements. For example, it corresponds to an input of pointing “Osaka” on the displayed map with a pointing device while saying “From Tokyo to here”. The type of cooperative operation depends on the specifications of the browser that executes MMML, and is not defined in the present invention.

Description of output content The output content can be described in a bind element that binds an output element of the dialog layer. The output contents may be written immediately inside the tags <Binds> to </ Binds>, or may be written inside each modality description to be bound. In this way, more appropriate output contents according to the modality can be described.

For example, for a speech synthesis modality, a description is given that the message “Copy completed” is output, and for the notification sound playback modality, the file name of an audio file that sounds a “beep” is given. it can. As described above, the output content can also be described in the output element of the dialog layer (inside the tags <output> to </ output>). These priorities are
Inside modality description in bind element> Just inside bind element> Inside output element of dialog layer.

Modality activate / deactivate description It is possible to activate / deactivate each bound modality. For example, it is possible to describe that the voice input modality is deactivated when the ambient noise is large. activate / deactivate is described by <activate modality = "..."/> tag and <deactivate modality = "..."/> tag, respectively.

Example of Bind Layer For example, in order to operate the copy machine 101 from the mobile phone 102, FIG. 13 shows an example of the bind layer that links the modality description (FIG. 11) of the mobile phone 102 and the dialog description (FIG. 12) of the copy machine. Further, the XML expression is as shown in FIG.

  In this way, in the present invention, the bind reasoning server 103 automatically generates this bind layer for any modality layer and any interaction layer, so that any operation device (for example, the mobile phone 102) can arbitrarily The control target device (copy machine 101) can be operated.

According to a markup language specification of the specification, such as the above operation example to operate the copy machine from a mobile phone, be an example to operate the copy machine from a mobile phone.

  The mobile phone 102 holds a markup language having a modality layer as shown in FIG. This is normally created by the manufacturer of the mobile phone 102 and placed on the mobile phone at the time of shipment. On the other hand, the copier 101 holds a markup language having an interactive layer as shown in FIG. This is normally created by the manufacturer of the copy machine 101 and placed on the copy machine 101 at the time of shipment.

  The bind layer inference server 103 holds a bind sample as shown in FIG. 14, for example. This binding sample shows whether or not each generic modality defined in the schema of FIG. 9 can be bound to an input / output element and appropriateness information indicating how appropriate the binding is (in the binding layer of FIG. 14). The number (value of 1 to 5) is the appropriateness, and the larger the number, the higher the appropriateness).

  For example, the modality “Button” can be bound to an input element of “selectMe” type with an appropriateness of 5, and can be further bound to each option (item) of an input element of the “selectOne” type with an appropriateness of 3. Show. The former is a correspondence such as “Copying starts when the start button of the copier is pressed”, and the latter is an option of “Tokyo”, “Osaka”, “Nagoya”. The association is “Osaka” when B is pressed and “Nagoya is selected when button C is pressed”.

  The modality “GUIRadioButton” indicates that it can be bound to an input element of “selectOne” type with an appropriateness of 5. Therefore, if the operating device has UI parts of both the “Button” class and the “GUIRadioButton” class, the appropriateness of the “selectOne” type input element is 5, so the modality of the “GUIRadioButton” class It would be more appropriate to bind.

  The bind layer inference server 103 binds the modality of the actual operation device and the input / output elements of the control target device based on such a bind sample. A specific example of the operation will be described in detail.

  FIG. 18 is a flowchart showing processing executed by the bind layer inference server according to the first embodiment of the present invention.

  First, when the user tries to operate the office copier 101 (control target device) with the mobile phone 102 (operating device) from outside, in step 101, first, the URL of the copier 101 and the mobile phone 102 Modality information including the modality layer (FIG. 11) is transmitted to the bind layer inference server 103.

  The bind layer inference server 103 receives modality information from the mobile phone 102 in step S201. Next, in step S202, a request for transmission of dialog information including a dialog description is issued to the copy machine 101 from the URL in the received modality information.

  Upon receiving this request from the bind reasoning server 103, the copier 101 transmits the dialog information regarding its own dialog layer (FIG. 12) to the bind reasoning server 103 in step S301.

  The bind layer inference server 103 receives the dialogue information from the copy machine 101 in step S203. Next, in step S204, the bind layer inference unit 1401 determines which modality of the mobile phone 102 should be bound to each input / output element of the dialog description in the dialog information. Specifically, with reference to the bind sample 1402 (FIG. 14), bind information including a bind layer that links the modality description of the mobile phone 102 and the dialog description of the copy machine 101 is generated as follows.

  For example, since the input element of the “CopyStart” field in the dialog description of the copier 101 is of the “selectMe” type, the modality appropriate for binding to this from FIG. 14 may be the modality of the “Button” class. Understand. Also, from FIG. 9 and FIG. 11, among the modalities of the mobile phone 102, “10Key-0” to “10Key-9” of the numeric keypads (0 to 9) are instances as subclass instances of the “Button” class. I know that there is. Therefore, “10Key-0”, which is an appropriate one, is bound to the input element of “CopyStart” field.

  Similarly, since the input element of the “PaperSize” field is of the “selecOne” type, the “GUIRadioButton” class, the “GUICheckBox” class, and the “SpeechInput” class are bound from FIG. You can see that you only need to bind to each option. From FIG. 9 and FIG. 11, among the modalities of the mobile phone 102, the voice input modality “MyASR” may be bound, or the numeric keys “10Key-0” to “10Key-9” may be bound to each option. I understand.

  Here, since “10Key-0” is already bound, “10Key-1” and “10Key-2” are bound to the choices “A4” and “B5”, respectively. This makes it possible to configure a multimodal interface that allows both voice input and button input by binding “MyASR”.

  Note that the recognition vocabulary of the voice input modality “MyASR” automatically generates “Ayeon” and “Vigo” from the choices. This can be done by reading from the descriptions “A4” and “B5” of the options by language analysis. Note that the reading process can be realized by a technique used in a technique such as speech synthesis, and the specific contents thereof are out of the scope of the present invention.

  The same applies to the input elements of the “PrintSides” field, binding “10Key-3” and “10Key-4” to each option, and further binding the voice input modality “MyASR”. In this way, the bind information including the bind layer as shown in FIG. 13 is generated. In step S <b> 205, the bind layer inference server 103 combines the bind information including this bind layer and the dialog information including the dialog layer (FIG. 12) of the copier 101 and transmits the combined information to the mobile phone 102.

  As a result, the mobile phone 102 has a markup language in which its modality and the dialog description of the copy machine 101 are combined in the bind layer, and various operations of the copy machine 101 are executed based on this markup language. can do.

  Hereinafter, an operation example between the mobile phone 102 and the copy machine 101 will be described.

  After receiving the bind layer and the conversation layer from the bind reasoning server 103, for example, when the user presses the numeric keypad “1” of the mobile phone 102, the input / output management unit 1301 of the mobile phone 102 detects this. . By this detection, the modality management unit 1304 knows that there is an input via the modality “10Key-1”.

  The modality control unit 1305 refers to the bind layer description (FIG. 13) received from the bind layer inference server 103 and finds that this is bound to the input element option “A4” of the “PaperSize” field of the copier 101. Therefore, the fact is transmitted to the copier 101 via the communication unit 1306 as XML data as shown in FIG.

  The copier 101 receives the XML data of FIG. 15 via the communication unit 1203, interprets the content in the dialog execution unit 1201, and knows that the paper size is set to “A4”. Subsequently, when the user presses the numeric key “0”, the input / output management unit 1301 of the mobile phone 102 detects this, and the modality management unit 1304 knows that there is an input through the modality “10Key-0”.

  The modality control unit 1305 refers to the bind layer description (FIG. 13) received from the bind layer inference server 103 and knows that this is bound to the input element “CopyStart” field of the copier 101. The XML data as shown in FIG. 16 is transmitted to the copier 101 via the communication unit 1306.

  Then, the copier 101 receives the XML data in FIG. 16 via the communication unit 1203, interprets the content in the dialog execution unit 1201, and starts copying by the device control unit 1202.

  As described above, the bind layer inference server 103 binds the modality of the operation device (for example, the mobile phone 102) and the input / output elements in the dialog description of the control target device (for example, the copy machine 101). Even if the digital camera 106 is switched to the digital camera 106, the copier 101 can be controlled in the same manner as long as the digital camera 106 has its own modality description and functions similar to those shown in FIG. Further, even if the control target device is the air conditioner 105, it can be controlled in the same way as long as the air conditioner has the same function as FIG. 2 and the dialog description (FIG. 17) which is its own control procedure.

As described above, according to the first embodiment, in the environment that implements the processing related to the operation device and the control target device based on the markup language, in particular, the markup language
Dialogue description of the control target device Modality description of the operation device Construct a bind description that links the dialogue description and the modality description.

  The operation device is linked to each modality according to the description of the modality description in the markup language and the modality management unit that performs input / output management using each modality and according to the description of the bind layer in the markup description language. A modality control unit that exchanges information with the input / output elements is configured. In addition, the control target device has a dialog execution unit that accepts input from each modality managed by the modality management unit and executes output instructions to each modality according to the content of the dialog description in the markup language. To do. Then, in accordance with the contents of the bind description in the markup language, a bind reasoning unit that passes information between the modalities and the input / output elements associated with each modality is configured in the bind reasoning server.

As a result, any device can be automatically configured as an operation device having any device as a control target device. The operating device can be configured as an operating device having a multimodal user interface.
[Embodiment 2]
In the first embodiment, a mobile phone and a digital camera are shown as the operation device, and a copy machine and an air conditioner are shown as the control target devices. However, as long as the operation device and the control target device have the same functions as described above, Arbitrary devices (for example, PDAs, remote controllers, etc.) to arbitrary devices (OA devices such as facsimiles, printers, scanners, etc., home appliances such as electric pots, refrigerators, televisions, etc.) may be used as operation devices and controlled devices, respectively. Needless to say, you can.
[Embodiment 3]
In the first embodiment, the network connecting the devices is the Internet. However, any type of network or any protocol may be used as long as it can transmit and receive text information described in XML.
[Embodiment 4]
In the first embodiment, the bind layer inference unit 1401 is configured on the bind layer inference server 103. However, the bind layer inference unit 1401 is configured as an operation device (client), and its own modality and the control target device are configured. You may make it produce | generate the description description of a dialog description.
[Embodiment 5]
In the first embodiment, a case has been described in which a program that realizes the operation executed by each of the above-described devices is held in the ROM. Good. Moreover, you may implement | achieve with the circuit which implement | achieves the same operation | movement.

  Although the embodiment has been described in detail above, the present invention may be applied to a system constituted by a plurality of devices, or may be applied to an apparatus constituted by one device.

  In the present invention, a software program (in the embodiment, a program corresponding to the flowchart shown in the drawing) that realizes the functions of the above-described embodiment is directly or remotely supplied to the system or apparatus, and the computer of the system or apparatus Is also achieved by reading and executing the supplied program code. In that case, as long as it has the function of a program, the form does not need to be a program.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In this case, the program may be in any form as long as it has a program function, such as an object code, a program executed by an interpreter, or script data supplied to the OS.

  As a recording medium for supplying the program, for example, floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card ROM, DVD (DVD-ROM, DVD-R) and the like.

  As another program supply method, a client computer browser is used to connect to an Internet homepage, and the computer program of the present invention itself or a compressed file including an automatic installation function is downloaded from the homepage to a recording medium such as a hard disk. Can also be supplied. It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  In addition to the functions of the above-described embodiments being realized by the computer executing the read program, the OS running on the computer based on the instruction of the program is a part of the actual processing. Alternatively, the functions of the above-described embodiment can be realized by performing all of them and performing the processing.

  Furthermore, after the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion board or The CPU or the like provided in the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

It is a block diagram of the information processing system of Embodiment 1 of this invention. 1 is a functional configuration diagram of a copier according to a first embodiment of the present invention. It is a block diagram which shows the hardware constitutions of the copying machine of Embodiment 1 of this invention. It is a functional block diagram of the mobile telephone of Embodiment 1 of this invention. It is a block diagram which shows the hardware constitutions of the mobile telephone of Embodiment 1 of this invention. It is a function block diagram of the bind layer inference server of Embodiment 1 of this invention. It is a block diagram which shows the hardware constitutions of the bind layer inference server of Embodiment 1 of this invention. It is a figure which shows the structure of the markup language of Embodiment 1 of this invention. It is the figure which schematized the schema description of the general-purpose modality of Embodiment 1 of this invention. It is a figure which shows the structural example of UI of the mobile telephone of Embodiment 1 of this invention. It is the figure which schematized the schema description of the modality of the mobile telephone of Embodiment 1 of this invention. FIG. 2 is a diagram schematically illustrating an interactive layer of the copier according to the first embodiment of the present invention. It is the figure which represented the description of the bind layer of Embodiment 1 of this invention schematically. FIG. 3 is a diagram illustrating a bind sample held by a bind layer inference unit according to the first embodiment of the present invention. It is a figure which shows the example of information transmitted to the copy machine from the mobile telephone of Embodiment 1 of this invention. It is a figure which shows another example of information transmitted to the copy machine from the mobile telephone of Embodiment 1 of this invention. It is the figure which schematized the interactive layer of the air conditioner of Embodiment 1 of this invention. It is a flowchart which shows the process performed with the bind layer inference server of Embodiment 1 of this invention. It is a figure which shows the XML expression example of the general purpose modality schema of Embodiment 1 of this invention. It is a figure which shows the XML expression example of the general purpose modality schema of Embodiment 1 of this invention. It is a figure which shows the XML expression example of the general purpose modality schema of Embodiment 1 of this invention. It is a figure which shows the XML expression example of the general purpose modality schema of Embodiment 1 of this invention. It is a figure which shows the XML expression example of the general purpose modality schema of Embodiment 1 of this invention. It is a figure which shows the XML expression example of the modality schema of the mobile telephone of Embodiment 1 of this invention. It is a figure which shows the XML expression example of the modality schema of the mobile telephone of Embodiment 1 of this invention. It is a figure which shows the example of XML expression of the interactive layer of the copier of Embodiment 1 of this invention. It is a figure which shows the example of XML expression of the interactive layer of the copier of Embodiment 1 of this invention. It is a figure which shows the XML expression example of the bind layer which binds the modality of the mobile telephone of Embodiment 1 of this invention, and the input / output element of the dialog description of a copy machine.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Copier 102 Mobile phone 103 Bind reasoning server 104 Network 105 Air conditioner 106 Digital camera 1201 Dialogue execution unit 1202 Device control unit 1203 Communication unit 1204 Dialogue description ML
1301 Input / output management unit 1302 Speech recognition unit 1303 Speech synthesis unit 1304 Modality management unit 1305 Modality control unit 1306 Communication unit 1307 Modality ML
1401 Bind layer inference unit 1402 Bind sample 1403 Communication unit

Claims (20)

First receiving means for receiving modality information relating to the modality of the operating device;
Second receiving means for receiving dialogue information relating to a dialogue of a control target device that is a control target of the operation device;
Generating means for generating binding information for associating the modality information with the dialogue information;
An information processing apparatus comprising: transmission means for transmitting the binding information and the dialogue information to the operating device. The modality information, the interaction information, and the bind information are described in a markup language.
The markup language is
A dialog description indicating the content of the dialog of the control target device;
A modality description indicating a modality of the operating device, which is configured separately from the dialog description;
The information processing apparatus according to claim 1, further comprising: a bind description that links the modality description and the dialog description. The dialog description is composed of one or a plurality of minimum dialog elements, which are input / output processing units, and the minimum dialog elements include zero or one input element and zero or one or more output elements. The information processing apparatus according to claim 2, further comprising: The modality description includes a hierarchical relationship between modality classes and modality instances, and the definition of the modality of the operation device refers to a general modality description that describes a general modality definition, and the general modality defined there The information processing apparatus according to claim 2, wherein the information processing apparatus is defined as a subclass and an instance. The binding description is a description for binding one or a plurality of modalities to one or a plurality of input elements. When a plurality of modalities are bound to a single input element, how to combine the modalities The information processing apparatus according to claim 2, further comprising a description. The binding description is a description for binding one or a plurality of modalities to one or a plurality of output elements. When a plurality of modalities are bound to a single output element, how to combine the modalities The information processing apparatus according to claim 2, further comprising a description. The operating device, according to the modality description, management means for managing the modality,
Modality control means for controlling processing related to the modality and the dialogue according to the binding description,
The control target device includes dialog execution means for managing the dialog according to the dialog description;
The information processing apparatus according to claim 2, further comprising: a control unit that executes control of the control target device based on an instruction from the dialogue execution unit. A storage means for storing a bind sample including the general-purpose modality, an input / output element that can be linked to the general-purpose modality, and a description of appropriateness information of the binding;
The information processing apparatus according to claim 4, wherein the generation unit generates bind information that associates the modality information with the dialogue information with reference to the bind sample. An information processing apparatus that operates as an operation device for controlling a control target device,
Receiving means for receiving dialogue information relating to dialogue between the control target devices;
Generating means for generating the binding information that associates the modality information related to the modality of the information processing apparatus and the dialogue information based on the appropriateness information indicating the appropriateness of the association;
An information processing apparatus comprising: modality control means for controlling processing related to the modality and the dialogue according to the binding information. A first receiving step of receiving modality information related to the modality of the operating device;
A second receiving step of receiving dialogue information related to a dialogue of a control target device that is a control target of the operation device;
A generation step of generating binding information that associates the modality information with the dialogue information;
And a transmission step of transmitting the binding information and the dialogue information to the operating device. The modality information, the interaction information, and the bind information are described in a markup language.
The markup language is
A dialog description indicating the content of the dialog of the control target device;
A modality description indicating a modality of the operating device, which is configured separately from the dialog description;
The information processing apparatus control method according to claim 10, further comprising: a bind description that links the modality description and the dialog description. The dialog description is composed of one or a plurality of minimum dialog elements, which are input / output processing units, and the minimum dialog elements include zero or one input element and zero or one or more output elements. The method for controlling an information processing apparatus according to claim 11, further comprising: The modality description includes a hierarchical relationship between modality classes and modality instances, and the definition of the modality of the operating device refers to a general modality description describing a general modality definition, and the general modality defined there The information processing apparatus control method according to claim 11, wherein the information processing apparatus is defined as a subclass and an instance of the information processing apparatus. The binding description is a description for binding one or a plurality of modalities to one or a plurality of input elements. When a plurality of modalities are bound to a single input element, how to combine the modalities The information processing apparatus control method according to claim 11, further comprising a description. The binding description is a description for binding one or a plurality of modalities to one or a plurality of output elements. When a plurality of modalities are bound to a single output element, how to combine the modalities The information processing apparatus control method according to claim 11, further comprising a description. The operation device manages the modality according to the modality description, and
A modality control step for controlling processing related to the modality and the dialogue according to the binding description;
The control target device manages the dialogue according to the dialogue description,
The information processing apparatus control method according to claim 11, further comprising: a control step of executing control of the control target device based on an instruction from the dialogue execution step. A storage step of storing in the storage medium a bind sample comprising the general-purpose modality, an input / output element that can be linked to the general-purpose modality, and a description of appropriateness information of the binding;
The information generation apparatus control method according to claim 13, wherein the generation step generates bind information that associates the modality information with the dialogue information with reference to the bind sample. A method for controlling an information processing apparatus that operates as an operation device for controlling a control target device,
Receiving step for receiving dialogue information related to dialogue of the control target device;
A generation step of generating bind information that associates the modality information related to the modality of the information processing apparatus and the dialogue information based on appropriateness information indicating the appropriateness of association;
A method for controlling an information processing apparatus, comprising: a modality control step of controlling processing related to the modality and the dialogue according to the binding information. A program for causing a computer to control an information processing apparatus capable of communicating with an operation device and a control target device that is a control target of the operation device,
A program code of a first receiving step for receiving modality information relating to the modality of the operating device;
Program code of a second receiving step for receiving dialogue information relating to dialogue of the control target device;
A program code of a generation step for generating binding information that associates the modality information with the dialogue information;
A program code comprising: a transmission process program code for transmitting the binding information and the dialogue information to the operating device. A program for causing a computer to function as a control device for an information processing device that operates as an operation device for controlling a control target device,
Program code of a receiving step for receiving dialogue information related to dialogue of the control target device;
A program code of a generating step for generating bind information that associates the modality information related to the modality of the information processing apparatus and the dialogue information based on the appropriateness information indicating the appropriateness of association;
A program comprising: modality control process program code for controlling processing related to the modality and the dialogue according to the binding information.

Images