JP3902959B2 - Information processing apparatus, control method therefor, and program - Google Patents

Description

上記例の「CopyComplete」のようなイベント名は、機器に応じて事前に定義され、ＭＭＭＬのブラウザは、これらの語彙を知っているものとする。
【０１２５】
イベントは、操作機器（クライアント）に関するイベントと制御対象（サーバー）に関するイベントに分けて考える必要がある。例えば、コピー機をＰＣからネットワーク経由で操作するケースを考える。この場合、「コピーが完了した」というイベントは、制御対象であるコピー機に関するイベントであり、コピー機の対話層に記述すべきものである。
【０１２６】
一方、ＰＣの「サウンドデバイスがｏｆｆされた」というイベントは、操作機器であるＰＣの中で閉じたイベントであり、コピー機の対話層には無関係なイベントである。このような場合、ＰＣの中で閉じたイベントの記述は、コピー機の対話層とは別のＰＣ固有の対話層を構成し、その中にイベントをキャッチするfieldを設ければよい。図１９に、イベント情報を含む記述例を示す。
［実施形態５］
上記実施形態１では、上述の各機器で実行される動作を実現するプログラムをＲＯＭに保持する場合について説明したが、これに限定されるものではなく、任意の記憶媒体を用いて実現してもよい。また、同様の動作を実現する回路で実現してもよい。
【０１２７】
以上、実施形態例を詳述したが、本発明は、複数の機器から構成されるシステムに適用しても良いし、また、一つの機器からなる装置に適用しても良い。
【０１２８】
尚、本発明は、前述した実施形態の機能を実現するソフトウェアのプログラム（実施形態では図に示すフローチャートに対応したプログラム）を、システム或いは装置に直接或いは遠隔から供給し、そのシステム或いは装置のコンピュータが該供給されたプログラムコードを読み出して実行することによっても達成される場合を含む。その場合、プログラムの機能を有していれば、形態は、プログラムである必要はない。
【０１２９】
従って、本発明の機能処理をコンピュータで実現するために、該コンピュータにインストールされるプログラムコード自体も本発明を実現するものである。つまり、本発明は、本発明の機能処理を実現するためのコンピュータプログラム自体も含まれる。
【０１３０】
その場合、プログラムの機能を有していれば、オブジェクトコード、インタプリタにより実行されるプログラム、ＯＳに供給するスクリプトデータ等、プログラムの形態を問わない。
【０１３１】
プログラムを供給するための記録媒体としては、例えば、フロッピーディスク、ハードディスク、光ディスク、光磁気ディスク、ＭＯ、ＣＤ−ＲＯＭ、ＣＤ−Ｒ、ＣＤ−ＲＷ、磁気テープ、不揮発性のメモリカード、ＲＯＭ、ＤＶＤ（ＤＶＤ−ＲＯＭ、ＤＶＤ−Ｒ）などがある。
【０１３２】
その他、プログラムの供給方法としては、クライアントコンピュータのブラウザを用いてインターネットのホームページに接続し、該ホームページから本発明のコンピュータプログラムそのもの、もしくは圧縮され自動インストール機能を含むファイルをハードディスク等の記録媒体にダウンロードすることによっても供給できる。また、本発明のプログラムを構成するプログラムコードを複数のファイルに分割し、それぞれのファイルを異なるホームページからダウンロードすることによっても実現可能である。つまり、本発明の機能処理をコンピュータで実現するためのプログラムファイルを複数のユーザに対してダウンロードさせるＷＷＷサーバも、本発明に含まれるものである。
【０１３３】
また、本発明のプログラムを暗号化してＣＤ−ＲＯＭ等の記憶媒体に格納してユーザに配布し、所定の条件をクリアしたユーザに対し、インターネットを介してホームページから暗号化を解く鍵情報をダウンロードさせ、その鍵情報を使用することにより暗号化されたプログラムを実行してコンピュータにインストールさせて実現することも可能である。
【０１３４】
また、コンピュータが、読み出したプログラムを実行することによって、前述した実施形態の機能が実現される他、そのプログラムの指示に基づき、コンピュータ上で稼動しているＯＳなどが、実際の処理の一部または全部を行ない、その処理によっても前述した実施形態の機能が実現され得る。
【０１３５】
さらに、記録媒体から読み出されたプログラムが、コンピュータに挿入された機能拡張ボードやコンピュータに接続された機能拡張ユニットに備わるメモリに書き込まれた後、そのプログラムの指示に基づき、その機能拡張ボードや機能拡張ユニットに備わるＣＰＵなどが実際の処理の一部または全部を行ない、その処理によっても前述した実施形態の機能が実現される。
【０１３６】
【発明の効果】
以上説明したように、本発明によれば、より汎用性が高く、かつ拡張が容易なモダリティの制御を実現できる情報処理装置及びその制御方法、プログラムを提供できる。
【図面の簡単な説明】
【図１】本発明の実施形態１のマークアップ言語の構造を示す図である。
【図２】本発明の実施形態１の汎用モダリティのスキーマ記述を図式化した図である。
【図３】本発明の実施形態１のコピー機のＵＩの構成例を示す図である。
【図４】本発明の実施形態１のコピー機のモダリティのスキーマ記述を図式化した図である。
【図５】本発明の実施形態１のＵＩ開発者によるコピー機のモダリティのスキーマ記述を図式化した図である。
【図６】本発明の実施形態１のコピー機の対話層を図式化した図である。
【図７】本発明の実施形態１のバインド層の記述の一部を図式化した図である。
【図８】本発明の実施形態１のバインド層の記述の一部を図式化した図である。
【図９】本発明の実施形態１のバインド層の記述の一部を図式化した図である。
【図１０】本発明の実施形態１のコピー機の機能構成図である。
【図１１】本発明の実施形態１のコピー機のハードウエア構成を示す構成図である。
【図１２】本発明の実施形態１のコピー機で実行される処理を示すフローチャートエである。
【図１３】本発明の実施形態２の情報処理システムの構成図である。
【図１４】本発明の実施形態２の情報処理システムの機能構成図である。
【図１５】本発明の実施形態２のコピー機から携帯電話に送信する情報例を示す図である。
【図１６】本発明の実施形態２の携帯電話からコピー機に送信する情報例を示す図である。
【図１７】本発明の実施形態３のエアコンを操作するリモコンの例を示す図である。
【図１８】本発明の実施形態３の操作機器固有の対話層を図式化した図である。
【図１９】本発明の実施形態４のイベント情報の記述を図式化した図である。
【図２０Ａ】本発明の実施形態１の汎用モダリティスキーマのＸＭＬ表現例を示す図である。
【図２０Ｂ】本発明の実施形態１の汎用モダリティスキーマのＸＭＬ表現例を示す図である。
【図２０Ｃ】本発明の実施形態１の汎用モダリティスキーマのＸＭＬ表現例を示す図である。
【図２０Ｄ】本発明の実施形態１の汎用モダリティスキーマのＸＭＬ表現例を示す図である。
【図２０Ｅ】本発明の実施形態１の汎用モダリティスキーマのＸＭＬ表現例を示す図である。
【図２１Ａ】本発明の実施形態１のコピー機のモダリティスキーマのＸＭＬ表現例を示す図である。
【図２１Ｂ】本発明の実施形態１のコピー機のモダリティスキーマのＸＭＬ表現例を示す図である。
【図２２Ａ】本発明の実施形態１のＵＩ開発者が新規定義するコピー機の対話層のＸＭＬ表現例を示す図である。
【図２２Ｂ】本発明の実施形態１のＵＩ開発者が新規定義するコピー機の対話層のＸＭＬ表現例を示す図である。
【図２２Ｃ】本発明の実施形態１のＵＩ開発者が新規定義するコピー機の対話層のＸＭＬ表現例を示す図である。
【図２２Ｄ】本発明の実施形態１のＵＩ開発者が新規定義するコピー機の対話層のＸＭＬ表現例を示す図である。
【図２３Ａ】本発明の実施形態１のコピー機の対話層のＸＭＬ表現例を示す図である。
【図２３Ｂ】本発明の実施形態１のコピー機の対話層のＸＭＬ表現例を示す図である。
【図２３Ｃ】本発明の実施形態１のコピー機の対話層のＸＭＬ表現例を示す図である。
【図２４Ａ】本発明の実施形態１のコピー機のバインド層のＸＭＬ表現例を示す図である。
【図２４Ｂ】本発明の実施形態１のコピー機のバインド層のＸＭＬ表現例を示す図である。
【図２４Ｃ】本発明の実施形態１のコピー機のバインド層のＸＭＬ表現例を示す図である。
【図２４Ｄ】本発明の実施形態１のコピー機のバインド層のＸＭＬ表現例を示す図である。
【図２５】本発明の実施形態２の携帯電話のモダリティスキーマのＸＭＬ表現例を示す図である。
【図２６】本発明の実施形態２の携帯電話のモダリティとコピー機の対話記述の入出力要素をバインドするバインド層のＸＭＬ表現例を示す図である。
【図２７Ａ】本発明の実施形態３の操作機器固有の対話層のＸＭＬ表現例を示す図である。
【図２７Ｂ】本発明の実施形態３の操作機器固有の対話層のＸＭＬ表現例を示す図である。
【符号の説明】
１０１ 入出力管理部
１０２ ＧＵＩ制御部
１０３ 音声認識部
１０４ 音声合成部
１０５ モダリティ管理部
１０６ モダリティ制御部
１０７ 対話実行部
１０８ 機器制御部
１０９ ＭＭＵＩマークアップ言語[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an information processing apparatus that has a plurality of types of modalities and controls the plurality of types of modalities based on a markup language, or an information processing apparatus that operates as an operation device for a control target device based on a markup language And its control method and program.
[0002]
[Prior art]
Web browsing for browsing content written in a markup language called HTML using a browser is now a technology that has become popular worldwide. HTML is a markup language for displaying content. HTML has a mechanism called a form, and can have GUI parts such as buttons and text boxes. With this and a mechanism called CGI (Common Gateway Interface) of the Web server, it means not only browsing but also a client (a computer and software using functions and data provided by the server in the network. This client. For example, a computer connected to a network, a home personal computer, and a Web browser or viewer operating on the computer). Can be executed, and the result can also be returned to the client in the form of HTML. For example, a Web search engine is generally realized by this method.
[0003]
This mechanism can be applied not only to Web browsing but also to device control. That is, a Web server is mounted on the control target device, and an HTML file including a form including a GUI component for controlling itself is transmitted to the operation device in response to a request from the operation device as a client. The operating device displays this HTML file on the browser, and the user operates the GUI on the operating device. The operating device transmits the input to a control target device (for example, a Web server). In the device to be controlled, this input is transferred to the control program by the CGI mechanism, and control according to the input is performed.
[0004]
On the other hand, in recent years, the form of information devices has been diversified, such as PDAs, mobile phones, car navigation systems and other portable terminals, and it has become possible to connect to the Internet from such devices other than personal computers (hereinafter referred to as PCs). Yes. Along with this, markup languages replacing HTML such as WML have been developed and standardized. In addition, with the development of speech recognition / synthesis technology and the development of CTI technology, it becomes possible to access the Web by voice conversation via the telephone. With this, markup languages such as VoiceXML have been developed and standardized. Yes. Thus, a markup language suitable for a device form has been developed and standardized.
[0005]
In addition to diversification of device forms, UI modalities are diversified, such as GUIs for PCs and PDAs, and voice and DTMF for telephones. There is a growing interest in multimodal user interfaces that combine these diversified modalities to improve operability.
[0006]
[Problems to be solved by the invention]
The description of the multi-modal user interface includes a dialog description (description showing what input is received from the user, what output is made, and in what procedure the input / output proceeds) and a modality description (what UI components are included). At least a description indicating whether to use that input / output).
[0007]
Dialog description is generally divided into three types: user-driven, system-driven, and mixed-driven. The user initiative allows the user to select an input / output procedure. For example, HTML-based Web browsing in which a form is filled or a button is pressed corresponds to this.
[0008]
On the other hand, the system initiative is a type in which the system determines input / output procedures, and the user proceeds with input according to instructions from the system. For example, a wizard-type installer corresponds to this, and a configuration in which voice input is performed along the voice guidance of the system or input by DTMF as in the CTI system is an example of this system initiative.
[0009]
Mixed initiative is a combination of both user initiative and system initiative. For example, VoiceXML is designed to allow system-driven and mixed-driven descriptions. In order to realize a more versatile UI description, it is desirable that UI developers can freely describe dialog strategies without being restricted by user-driven, system-driven, or mixed-driven, but existing HTML, WML, VoiceXML In such markup languages, there is a problem that it is biased toward either type of dialogue.
[0010]
The modality description is highly dependent on the client configuration. A general-purpose device such as a PC has abundant GUIs, and recently, some have a speech UI due to the development of speech recognition / synthesis technology. On the other hand, it is best to use voice for mobile phones. This is because a mobile phone supports a simple GUI component on a small liquid crystal screen, but does not have a pointing device and cannot always be said to be easy to use. In view of device control, there is a remote controller as an operation device. The remote control is generally operated by physical buttons.
[0011]
Looking at existing markup languages, HTML basically assumes GUI as a modality. VoiceXML assumes voice and DTMF as modalities. Both are modality-dependent markup languages and are not suitable for multimodal user interfaces that combine multiple modalities.
[0012]
There are two kinds of methods for such dialog description and modality description.
[0013]
One is a method of clearly describing the modality input / output form in the dialog description, such as a certain input is a GUI button and a certain output is a voice. The other is a method of separating the dialog description and the modality description and describing the dialog description in a modality-independent manner.
[0014]
In the latter method, a dialog description, which is an operation procedure of a device to be controlled, is performed in a modality-independent manner, and in a form separated from this, for each client, a modality description adapted to the client is performed. , It becomes possible to operate one control target device from various clients. There is CML (Japanese Patent Laid-Open No. 2001-154852) as a markup language of a multimodal user interface capable of separately describing dialog description and modality description.
[0015]
In CML, the dialog description itself is described in a modality independent form, and there is no framework for performing the modality description itself and its control description. A modality is generated by converting the dialog description part into an existing markup language such as HTML, WML, VoiceXML or the like. Alternatively, when the browser directly executes CML, the browser is assumed to have a known modality of the mounted device, and the specification is left to the browser which modality is associated with which input / output element in the dialog description.
[0016]
In this case, since a UI developer cannot describe how to combine a plurality of modalities, the advantage as a multimodal user interface cannot be exhibited. For example, it cannot be described that the operation is different between when the button A is pressed and when the button B is pressed and when the button B is pressed. In addition, there is a problem that the UI developer cannot customize or control the modality by describing the markup language. For example, it cannot be described that the voice input modality is turned off when the ambient noise is large.
[0017]
The present invention has been made to solve the above-described problems, and an object thereof is to provide an information processing apparatus, a control method thereof, and a program that can realize modality control that is more versatile and easy to expand. .
[0018]
[Means for Solving the Problems]
  In order to achieve the above object, an information processing apparatus according to the present invention comprises the following arrangement. That is, a control unit that controls processing executed by the information processing device based on data stored in the memory;
  A dialog description indicating a processing procedure of the process executed by the information processing apparatus, a modality description indicating a modality of the information processing apparatus configured separately from the dialog description, the modality description, and the dialog description. Storage means for storing description data indicating a binding description to be associated in the memory;
  Binding means for associating the input / output process included in the processing procedure indicated in the dialog description in the description data stored in the memory by the control means with the modality included in the modality description according to the bind description;
  When the control unit controls the information processing apparatus based on the dialog description in the description data stored in the memory, the data required by the input / output process corresponds to the input / output process by the bind unit. Execution means for executing control to and from the attached modality;
  Is provided.
[0019]
Preferably, the written description is composed of one or a plurality of minimum dialog elements which are input / output processing units, and the minimum dialog elements include zero or one input element and zero or one. The above output elements are included.
[0020]
Preferably, the minimum interaction element includes a description of event information related to the event.
[0021]
Preferably, the minimum interaction element includes a description of action information regarding an action for an input to the minimum interaction element.
[0022]
Preferably, the minimum interaction element has one of two states of active / inactive, and the minimum interaction element is a description for activating / deactivating itself or another minimum interaction element as the action information. including.
[0023]
Preferably, the output element includes a description indicating the output content.
[0024]
Preferably, the modality description includes at least one of definitions indicating class definition of the modality, inheritance, a hierarchical relationship between classes, and definitions of instances of the class.
[0025]
Preferably, each modality in the modality description has one of two states of active / inactive.
[0026]
Preferably, the binding description is a description for binding one or more modalities to one or more input elements, and when binding a plurality of modalities to one input element, Includes a description of how to combine modalities.
[0027]
Preferably, the binding description is a description for binding one or more modalities to one or more output elements, and when binding a plurality of modalities to one output element, Includes a description of how to combine modalities.
[0028]
Preferably, the bind description includes a description indicating the output contents in a bind element which is a description indicating a connection between the modality and the output element, and a description indicating the output contents for each modality linked to the output element. The output content described in the bind element including at least one has priority over the output content described in the output element that is the target of the binding, and the output described for each modality in the bind element The content takes precedence over the output content described in the bind element.
[0029]
Preferably, each binding element of the binding description includes a description for activating / deactivating a modality that is a binding target.
[0030]
  In order to achieve the above object, an information processing apparatus according to the present invention comprises the following arrangement. That is, a control unit that controls processing executed by the information processing device based on data stored in the memory;
  The modality description indicating the modality of the information processing apparatus and the bind description that associates the modality description with the dialog description, which are configured separately from the dialog description indicating the processing procedure of the process executed by the control target device. Storage means for storing descriptive data in the memory;
  Binding means for associating the input / output process included in the processing procedure indicated in the dialog description in the description data stored in the memory by the control means with the modality included in the modality description according to the bind description;
  When the control unit controls the control target device based on the dialog description in the description data stored in the memory, the data required by the input / output process corresponds to the input / output process by the bind unit. Execution means for executing control to and from the attached modality;
  Is provided.
[0031]
  In order to achieve the above object, a method for controlling an information processing apparatus according to the present invention comprises the following arrangement. That is,
  A control method for controlling an information processing apparatus having a plurality of types of modalities,
  A dialog description indicating a processing procedure of the information processing apparatus; a modality description indicating a modality of the information processing apparatus configured separately from the dialog description; and a bind description that associates the modality description with the dialog description. A reading step of reading the description data from a memory storing the description data by a control unit that controls processing executed by the information processing apparatus;
  A binding step of associating the input / output process included in the processing procedure indicated in the dialog description in the description data read from the memory by the control unit with the modality included in the modality description according to the bind description;
  When the control unit controls the information processing apparatus based on the dialog description in the description data read from the memory, the data requested by the input / output process is combined with the input / output process by the bind step. A control process for controlling the transfer to and from the associated modality;
  Is provided.
[0032]
  A method of controlling an information processing apparatus that operates as an operation device of a control target device,
  Stores description data indicating the modality description indicating the modality of the information processing apparatus and the binding description for associating the modality description and the dialog description, which are configured separately from the dialog description indicating the processing procedure of the control target device. Reading out the description data from the memory to be read by a control unit that controls processing executed by the information processing apparatus;
  A binding step of associating the input / output process included in the processing procedure indicated in the dialog description in the description data read from the memory by the control unit with the modality included in the modality description according to the bind description;
  When the control unit controls the control target device based on the dialog description in the description data read from the memory, the data requested by the input / output process is combined with the input / output process by the binding step. A control process for controlling the transfer to and from the associated modality;
  Is provided.
[0033]
  Control of information processing equipment with multiple modalities in a computerfunctionA program for
  A dialog description indicating a processing procedure of the information processing apparatus; a modality description indicating a modality of the information processing apparatus configured separately from the dialog description; and a bind description that associates the modality description with the dialog description. A program code for a reading process of reading the description data from a memory storing the description data by a control unit that controls processing executed by the information processing apparatus;
  A program of a binding step in which the control unit associates the input / output process included in the processing procedure indicated in the dialog description in the description data read from the memory with the modality included in the modality description according to the bind description. Code,
  When the control unit controls the information processing apparatus based on the dialog description in the description data read from the memory, the data requested by the input / output process is combined with the input / output process by the bind step. Program code of a control process that controls to pass to and from the associated modality;
  Is provided.
[0034]
  A program for causing a computer to control the control of an information processing device that operates as an operation device of a control target device,
  Stores description data indicating the modality description indicating the modality of the information processing apparatus and the binding description for associating the modality description and the dialog description, which are configured separately from the dialog description indicating the processing procedure of the control target device. Reading the program data from the memory by the control unit that controls the processing executed by the information processing apparatus;
  A program of a binding step in which the control unit associates the input / output process included in the processing procedure indicated in the dialog description in the description data read from the memory with the modality included in the modality description according to the bind description. Code,
  When the control unit controls the control target device based on the dialog description in the description data read from the memory, the data requested by the input / output process is combined with the input / output process by the binding step. Program code of a control process that controls to pass to and from the associated modality;
  Is provided.
[0035]
In order to achieve the above object, a program according to the present invention comprises the following arrangement. That is,
A program for causing a computer to control a plurality of types of modalities possessed by an information processing apparatus,
A program code comprising a dialog description indicating the content of the dialog of the information processing apparatus;
A program code comprising a modality description indicating a modality of the information processing apparatus, which is configured separately from the dialog description;
Program code comprising a bind description that links the modality description and the dialog description;
Each of the program codes is configured separately from each other.
[0036]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the drawings.
[Embodiment 1]
First, a specification example of a multimodal user interface markup language (hereinafter referred to as MMML), which is a feature of the present invention, will be described. In particular, here, a case where a UI of a copier is realized will be described as an example.
[0037]
Three-layer structure
In the present invention, as shown in FIG. 1, a modality description depending on a device and a dialog description which is UI logic are described separately. In particular, in addition to a modality layer 401 indicating a modality description (portion enclosed by tags <modality> to </ modality>), an interaction layer 403 indicating an interaction description (portion enclosed by tags <dialog> to </ dialog>), It has a three-layer structure of a bind layer 402 that indicates a description (portion enclosed by tags <Binds> to </ Binds>) that links a modality and a dialogue.
[0038]
Thus, by separating the modality description, it is possible to share one dialog description (UI logic) among a plurality of devices having different modalities. A specific example of this three-layer structure is shown in FIG.
[0039]
In FIG. 1, a copy machine modality description (Copier modalities) showing the operation of the copy machine, specifically, a description about an operation by a physical button of the copy machine, a GUI on the LCD, etc., and also enabling operation from a mobile phone. The structure of the description (Mobile phone modalities) is shown. As shown in this example, if one copier dialogue description (Copier Dialog) is configured, it is possible to perform operations from various devices only by describing a device-dependent modality layer and a bind layer for each device.
[0040]
Modality classes and instances
In the description of the modality layer in MMML, it is difficult to predefine all vocabularies representing modalities, that is, UI parts, as MMML definition words, and lack of extensibility. Therefore, several types of UI component classes that are expected to be generally used are defined as MMML common vocabulary, and a new class can be defined by inheriting this class. Furthermore, UI parts used in actual UI descriptions can be defined in the form of instances of such classes. In the first embodiment, an RDF schema is used as a method for describing class definition, inheritance, hierarchical relationships between classes, and instance definitions in XML. The RDF schema is a markup language (see http://www.w3.org/TR/rdf-schema/) that is standardized by the W3C (see http://www.w3c.org), a Web standards organization. It is.
[0041]
Generic modality class schema
Several types of GUI component classes that are expected to be commonly used are defined as MMML vocabulary in the form of an RDF schema. An example in which this RDF schema (general-purpose modality schema) is schematically represented is shown in FIG. As shown in FIG. 2, UI component classes can be hierarchically defined using the RDF schema. For example, “Button” can define “Physical Button” and “GUI Button” as subclasses. The former is, for example, a button of a copier, and the latter is, for example, a button displayed on a liquid crystal panel. Also, the XML representation of this RDF schema is as shown in FIGS. 20A to 20E.
[0042]
Modality schema of operation device (client)
Based on a general UI part class defined as an MMML common vocabulary, a schema that defines a UI part class and an instance dependent on an operation device is described. That is, this modality schema describes what modalities can be used in the operating device, and is assumed to be provided by the manufacturer of the operating device. For example, FIG. 4 shows a schematic example of the modality schema corresponding to the UI parts (LCD, physical buttons, voice input, voice output, etc.) of the copier in FIG. The XML representation of this modality schema is as shown in FIGS. 21A and 21B.
[0043]
Modality extension by UI developers
In the copier modality schema for the generic modality schema shown in FIG. 4, a thin box represents a class, and a thick box represents an instance. In this figure, "StartButton", "ResetButton", "10Key-1" ... "10Key-0" mean physical buttons such as a copy machine start button, reset button, and numeric keypad, respectively. ) Is defined as an instance of the physical button class “PhysicalButton”. Copier UI developers use these vocabularies directly to describe the UI.
[0044]
On the other hand, “CopierGUIButton” is a class representing a GUI button, not an instance. In other words, it only describes that “GUI buttons are available”, and it is left to the UI developer to determine what kind of GUI buttons to deploy. A UI developer can freely configure a GUI by defining an instance of a GUI button using such a device modality schema provided by a device manufacturer. For example, a schematic example of a modality schema (Author-defined Copier Modalities) newly defined by a UI developer with respect to a copier modality schema (Copier Modalities Schema) is shown in FIG. Also, the XML representation of this modality schema is as shown in FIGS. 22A to 22D.
[0045]
" Target Concept
Define a class called “Target” in the general-purpose modality schema. In addition, classes such as “Display” representing a display and “Window” representing a GUI window are defined as subclasses of “Target”. “Target” is an “existing place” of mainly visual UI parts such as GUI parts. In the XML description of each modality, the “location” of the modality can be described by the “target” tag.
[0046]
For example, in the GUI on the LCD of the copier, when the “paper selection mode button” is arranged on the initial screen and the “A4 button”, “B5 button”, etc. are arranged in the “paper selection subwindow”, the “paper selection mode button” Set “target” to “LCD” and “target” of “A4 button” and “B5 button” to “CopierWindow”. The “target” of “CopierWindow” itself is “LCD”. The actual display is left to XSL or CSS2. The display position of each component is a relative position with respect to “Target” indicated by “target”.
[0047]
Dialogue layer
This is a layer for performing a specific dialog description, and a description independent of the modality is possible. In the MMML description, the dialog layer is described inside tags <dialog> to </ dialog>. The minimum unit of interaction is field (minimum interaction unit). The field has input / output elements such as input and output and an element called filled.
[0048]
Each of input and output is a tag that describes what information is input / output, and has an ID as an attribute. This ID is used when describing the binding with the modality described separately. This will be described later.
[0049]
When there is an input (user operation) corresponding to input, the filled contents described immediately after that are executed. Each field takes one of two states, active / inactive. In addition, you can activate / deactivate yourself or another field. By describing the activate / deactivate of the field, the state transition of the dialog can be controlled. In the case of VoiceXML, system-driven is basic, so the process proceeds automatically every time one input operation is performed. That is, active fields move sequentially.
[0050]
On the other hand, in this dialog description, the active / inactive state of each field does not change unless it is explicitly activated / deactivated. To explicitly activate / deactivate, specify with the activate tag or deactivate tag in the filled tag. Hereinafter, the role of each element (tag described inside the field tag) of field and field will be described in detail in each section.
[0051]
field Role
The field represents the minimum unit of input / output (minimum dialog unit) in the UI dialog, and describes the type of input to be received, the operation for the input, or the content to be output. The input type is represented by an input tag, and the operation for the input is represented by a filled tag. The output is represented by an output tag. Only active fields can be input / output. Even if the input is completed as in the VoiceXML field, it does not advance to the next field. The description “go to the next field” is realized by explicitly describing the act of activating the next field and deactivating itself.
[0052]
Input element
Describe the type of input you want to accept in the field to which it belongs. The type of input is described in the type attribute. As attribute values, the following are pre-defined as MMML common vocabulary.
[0053]
"SelectMe"
This is a type of input that accepts the fact that there was an input. Modality such as “button” and voice can be bound.
[0054]
"SelectOne"
This is a type of input for selecting one of the options. For example, radio buttons and pull-down menus are typical modalities. The selectOne type input describes an option with an item tag. It is also possible to bind modalities such as “buttons” to each item.
[0055]
"SelectMany"
This is a type of input for selecting a plurality of options. For example, a combo box (ComboBox) is a typical modality. Like the selectOne type, the option is described with the item tag.
[0056]
"TextString"
This type of input accepts input of character strings. For example, GUI text boxes and voice input are typical modalities.
[0057]
Output element
Describes the output in the field to which it belongs. The output contents can be described with the content tag.
[0058]
filled element
Describes the action (action information) for the input in each field. There are various types of actions. Use the activate / deactivate tag to explicitly activate / inactivate itself or other fields, interact with internal programs other than the UI by scripting or some other description method (for example, receiving “copy start” input) In other words, a typical action is to start a processing program that actually performs copying. Moreover, when responding to an input, an output tag can be described in filled.
[0059]
fields Role
Group fields. In general, the UI logic is often divided into several groups and becomes active / inactive in units of groups. Therefore, it is convenient to appropriately group fields (for example, the initial screen of a copier). When the fields are activated / deactivated, all the fields belonging to the fields become active / inactive.
[0060]
Example of a copier dialogue layer
FIG. 6 shows a description example of the interaction layer of the copy machine UI, and FIGS. 23A to 23C show XML representations thereof.
[0061]
In this example, a UI for executing copy (CopyStart), paper selection (PaperSize), and single-sided / double-sided (PrintSides) is described. In this example, modality description, that is, what kind of UI parts to use, such as whether to input with buttons, input with voice, display with text, notify with voice, etc. Note that it is not described (modality independent).
[0062]
Bind layer
Since the modality-independent dialog description eliminates the modality, it is necessary to associate the modality with the dialog description. The binding layer is a layer that describes the association, and binds each modality defined in the modality layer to each input / output element in the interaction layer. The description of the association is performed by referring to the ID of the modality to be associated and the ID of the input / output element with a URI (Uniform Resource Identifier). The bind layer is described inside the tags <Binds> to </ Binds>. Each bind element is described inside tags <Binds> to </ Binds>. By binding a plurality of modalities to one input / output element of the interactive layer, it is possible to describe a multimodal UI using the selective use of multiple modalities and the cooperative use. In the bind layer, the following description about modality management is possible.
[0063]
・ Description of how to combine multiple modalities
・ Description of output contents
-Description of the activate / deactivate of the modality
Description of how to combine multiple modalities
When binding a plurality of modalities to one input / output element, it is possible to instruct how to combine modalities such as selective use and cooperative use. The combination instruction is described by the BindType attribute. As attribute values of the BindType attribute, the following are predefined as MMML common vocabulary.
[0064]
"Alt"
Any one of a plurality of modalities bound to input / output elements can be used. The priority of multiple modalities shall be in the order described. For example, when the radio button modality and the voice input modality are bound with the “Alt” attribute to the input element for selecting “A4 or B5” on the copier, this is the case of selection by radio button or voice input. Either means that the paper can be selected.
[0065]
"Seq"
Apply multiple modalities bound to input / output elements in order. For example, it corresponds to an input such as “speak after pressing button A”.
[0066]
"Coordinated"
Coordinate multiple modalities bound to input / output elements. For example, it corresponds to an input of pointing “Osaka” on the displayed map with a pointing device while saying “From Tokyo to here”. The type of cooperative operation depends on the specifications of the browser that executes MMML, and is not defined in the present invention.
[0067]
Description of output contents
The output content can be described in the bind element that binds the output element of the interaction layer. The output contents may be written immediately inside the tags <Binds> to </ Binds>, or may be written inside each modality description to be bound. In this way, more appropriate output contents according to the modality can be described.
[0068]
For example, for a speech synthesis modality, a description is given that the message “Copy completed” is output, and for the notification sound playback modality, the file name of an audio file that sounds a “beep” is given. it can. As described above, the output content can also be described in the output element of the dialog layer (inside the tags <output> to </ output>). These priorities are
Inside modality description in bind element> Just inside bind element> Inside output element of dialog layer
It has become.
[0069]
Modality activate / deactivate Description of
It is possible to activate / deactivate each bound modality. For example, it is possible to describe that the voice input modality is deactivated when the ambient noise is large. activate / deactivate is described by <activate modality = "..." /> tag and <deactivate modality = "..." /> tag, respectively.
[0070]
Copier UI bind layer description example
Examples of the binding layer description of the copy machine UI are shown in FIGS. The XML expressions are shown in FIGS. 24A to 24D.
[0071]
Next, a copy machine will be described as an example of a device that operates in accordance with the markup language having the above specifications.
[0072]
FIG. 10 is a functional configuration diagram of the copier according to the first embodiment of the present invention.
[0073]
In the figure, reference numeral 101 denotes an input / output management unit which manages data input / output and voice input / output. Reference numeral 102 denotes a GUI control unit that controls the GUI according to an operation from the user. Reference numeral 103 denotes a voice recognition unit that recognizes an input voice. A voice synthesis unit 104 synthesizes voice output target data.
[0074]
A modality management unit 105 operates according to the description of the modality layer of the markup language, and manages modalities (UI parts).
[0075]
A modality control unit 106 operates according to the description of the bind layer of the markup language, controls the activation / deactivate of the bound modality, the process of passing the input via the modality to the corresponding input element of the dialog layer, the dialog Depending on the output element of the layer, the output content is passed to the bound modality and output is performed through this.
[0076]
A dialog execution unit 107 operates according to the description of the markup language dialog layer, and executes actions such as dialog state transition (activate / deactivate field) and instructions related to device control. Reference numeral 108 denotes a device control unit, which executes an instruction related to device control from the dialog execution unit 107 (for example, executes copy in accordance with an instruction “copy”).
[0077]
As described above, the modality control unit 106 performs various types of control regarding processing between the modality management unit 105 and the dialogue execution unit 107.
[0078]
109 is a multi-modal user interface (MMUI) markup language.
[0079]
FIG. 11 is a block diagram showing the hardware configuration of the copier according to the first embodiment of the present invention.
[0080]
In the figure, reference numeral 201 denotes a CPU, which operates according to a program for realizing a flowchart to be described later. A RAM 203 provides a storage area, a work area, and a data temporary save area necessary for the operation of the program. Reference numeral 202 denotes a ROM which holds programs and various data for realizing a flowchart described later. A disk device 204 holds an MMUI markup language 109.
[0081]
Reference numeral 205 denotes a liquid crystal display (LCD), which displays GUI components such as icons generated by the GUI control unit 102. Reference numeral 206 denotes a microphone that performs voice input to the voice recognition unit 103. Reference numeral 207 denotes a physical button, such as the start button, reset button, or numeric keypad shown in FIG. Reference numeral 208 denotes a speaker, which outputs a synthesized sound from the speech synthesizer 104. Reference numeral 209 denotes a bus which interconnects the above-described components.
[0082]
Next, processing executed by the copier according to the first embodiment of the present invention will be described with reference to FIG.
[0083]
FIG. 12 is a flowchart showing processing executed by the copier according to the first embodiment of the present invention.
[0084]
First, in step S <b> 101, the dialogue management unit 107 refers to the MMUI markup language 109 and performs initialization processing on the device.
[0085]
Next, in step S102, the modality management unit 105 refers to the modality layer in the MMUI markup language 109 and is described in the modality layer based on the input from the input / output management unit 101 / modality control unit 106. Processes such as status management of various modalities are executed.
[0086]
Next, in step S103, the modality control unit 106 refers to the bind layer in the MMUI markup language 109, and based on the input from the modality management unit 105 / dialog execution unit 107, the modality and the input / output element corresponding thereto. Executes processes such as exchange of information and activate / deactivate modalities.
[0087]
Next, in step S104, the dialog execution unit 107 refers to the dialog layer in the MMUI markup language 109, receives input from the modality based on the input from the modality control unit 106 / device control unit 108, modality Processing such as instruction to output to, reception of event, state transition of dialogue, action for input and event, etc. are executed.
[0088]
Next, in step S <b> 105, the device control unit 108 executes device control based on the input from the dialogue execution unit 107.
[0089]
Note that FIG. 12 illustrates an example in which the processing from step S101 to step S105 is finally executed by the device control unit 108 by an input from the input / output management unit 101. Depending on the processing content of each step, the reverse processing, that is, the processing of step S105 to step S101 is executed, or the above steps are linked to each other, and the order of the steps is changed or omitted. Please note that.
[0090]
Next, a specific operation example of the copier shown in FIG. 10 will be described. Here, according to a markup language in which FIG. 5 is a modality layer (FIGS. 22A to 22D), FIG. 6 is an interaction layer (FIGS. 23A to 23C), and FIGS. 7 to 9 are bind layers (FIGS. 24A to 24D). An operation example will be described.
[0091]
First, the dialog execution unit 107 refers to the MMUI markup language 109 and executes the initialization process described in the tags <initial> to </ initial> in FIGS. 23A to 23C. Here, only the field "CopierTop" is made active. In this case, only the fields included in the “CopierTop” fields are active, that is, input / output is possible, and other fields are not subjected to interaction.
[0092]
In response to this, the modality control unit 106 sets only the modalities bound to the input / output elements in each field included in the “CopierTop” fields to be active, and sets the others to inactive.
[0093]
That is, only the modalities shown in FIG. 7 are active, and the corresponding voice input / output and GUI button can be pressed. On the other hand, the modality shown in FIG. 8 and FIG. 9 is inactive, and the corresponding GUI button is disabled. It should be noted that the GUI button pressable state and the non-pressable state are displayed on the GUI in different display forms (for example, the non-pressable state is displayed in gray, blinking, etc.) so as to distinguish each state. In FIG. 7, the start button and the voice input modality are bound to the input element of the “CopyStart” field of the interactive layer in the selection usage mode.
[0094]
Therefore, if the start button is pressed or if there is a voice input corresponding thereto, this input element accepts the input, and the dialog execution unit 107 executes the process described in the corresponding filled element.
[0095]
In the example of FIG. 7, after some output is executed by the output element, an instruction to execute copying is issued to the device control unit 108. Here, what is output is not described in the output element itself, and “copy” is described in the bind element. Since this is bound to the speech synthesis modality, it will be uttered with a synthesized sound saying “copy”. Further, since there is no description of field activate / deactivate in this filled element, the state of active / inactive of each field does not occur, and only the field and modality shown in FIG. 7 are also active.
[0096]
In FIG. 7, the GUI button and the voice input modality for entering the paper size selection mode are bound to the input element of the “IsPaperSizeMode” field of the interactive layer in the selection usage mode. Therefore, if this GUI button is pressed or if there is a voice input corresponding thereto, this input element accepts the input, and the dialog execution unit 107 executes the process described in the corresponding filled element. Since filled is described to activate “SelectPaperSizeMode” fields, each field in the fields becomes active.
[0097]
In response to this, the modality control unit 106 activates the modality bound to the input / output elements of these fields, that is, the modality shown in FIG. 8, so that, for example, the “ButtonA4” of the GUI button can be pressed. It becomes.
[0098]
With the above procedure, a multimodal user interface according to the markup language is realized.
[0099]
As described above, according to the first embodiment, particularly in an environment where processing related to a device is realized based on a markup language,
Device dialogue description
Device modality description
Bind description that links the dialog description and modality description
Configure.
[0100]
The device is linked to each modality according to the description of the modality description in the markup language and the modality management unit that performs input / output management using each modality according to the content of the modality description in the markup language, and according to the description of the bind layer in the markup description language. Modality control unit that passes information to and from input / output elements, and accepts input from each modality managed by the modality management unit according to the contents of the dialog description in the markup language, and instructs output to each modality The dialog execution part which performs is comprised.
[0101]
As a result, an effective combination of various modalities of devices can be freely described in a markup language, so that a multimodal user interface with higher usability can be realized.
[Embodiment 2]
In the first embodiment, the example in which the operation of the copier is performed through the modality of the copier itself has been described. However, a PC or a mobile phone other than the copy machine can be used as the operation device of the control target device. In the second embodiment, a markup language editing method for operating a copier from a mobile phone and its operation will be described.
[0102]
First, how the markup language is edited will be described with reference to FIG.
[0103]
FIG. 13 is a diagram showing a configuration of an information processing system according to the second embodiment of the present invention.
[0104]
First, the copy machine itself has already been installed with a markup language having the configuration described in FIGS. 5 to 9 and can be operated through the modality of the copy machine itself as described above. What operation the copier performs for what input, what output, and how the dialog state transitions are functions of the copier itself. It operates according to the description of 6 dialog layers.
[0105]
The development of a markup language that allows the copy machine to be operated from another device will describe a modality layer of the operation device itself and a bind layer that binds the modality layer and the interaction layer of the copy machine itself. In order to enable such development, the copy machine manufacturer discloses the interaction layer of the copy machine on a Web site provided on the copy machine manufacturer terminal 1201 of the company.
[0106]
On the other hand, since the modality layer of the mobile phone as the operation device is unique to the mobile phone, it is natural for the mobile phone device manufacturer to describe the modality layer. A mobile phone device maker discloses a mobile phone modality layer on a Web site provided on its mobile phone maker terminal 1202 or the like. FIG. 25 shows a markup language description example of the modality layer of the mobile phone. In this example, a “0” button (Button0) is defined.
[0107]
A UI developer who develops a markup language that describes an MMUI that operates a copier from this mobile phone refers to the mobile phone's modality layer and the copier's interactive layer, and creates a bind layer that binds both. Describe. FIG. 26 shows an example of markup language description of this bind layer. This example represents binding the “0” button of the mobile phone to the “copy start” input element.
[0108]
The UI developer publishes the developed markup language (bind layer) on a Web site provided on his UI developer terminal 1204.
[0109]
The user downloads this published markup language (bind layer) to his / her mobile phone 1205.
[0110]
As described above, the markup language of the present invention is not described by a single UI developer, but the modality layer is developed by the manufacturer of the operation device, and the dialog layer is developed by the manufacturer of the control target device. It enables division of labor such that the UI developer who wants to bind describes the bind layer.
[0111]
This is an advantage of the markup language of the present invention having the above-described three-layer structure. According to this configuration, it is possible to easily develop a markup language in which any device is not limited to a combination of a mobile phone and a copy machine, and any device is an operation device or a control target device. For example, if a bind layer that binds the modality layer of a mobile phone and the interactive layer of an air conditioner published by an air conditioner manufacturer, the air conditioner can be operated from the mobile phone.
[0112]
Next, operations of the copier and the mobile phone according to the second embodiment will be described with reference to FIG. Since the operation of each component in FIG. 14 is basically the same as the operation of each component in FIG. 10 described in the first embodiment, the exchange between the copier and the mobile phone will be described here.
[0113]
In FIG. 14, the device control unit 1301 and the dialog execution unit 1303 of the copier 1301 correspond to the device control unit 108 and the dialog execution unit 107 of FIG. Also, the input / output management unit 1308, the speech recognition unit 1310, the speech synthesis unit 1311, the modality management unit 1312, and the modality control unit 1313 of the mobile phone 1307 are the same as the input / output management unit 101, the speech recognition unit 103, and the speech synthesis unit in FIG. 104, the modality management unit 105 and the modality control unit 106. Furthermore, the MMUI markup languages 1306 and 1315 in FIG. 14 correspond to the MMUI markup language 109 in FIG. In addition, the DTMF management unit 1309 performs DTMF management of the mobile phone 1307. The communication units 1305 and 1314 communicate with each other via the network 1315.
[0114]
First, by transmitting an operation request from the mobile phone 1307 to the copy machine 1301 via the communication unit 1314, a dialogue between the mobile phone 1307 and the copy machine 1301 is started. The dialogue execution unit 1303 of the copier 1301 operates according to the description of the dialogue layer. The dialog execution unit 1303 sends information indicating which field is active at each step described in the dialog layer and the ID of the output element to the mobile phone 1307 via the communication unit 1305 if there is an output operation. Send.
[0115]
An example of this transmission information is shown in FIG. In this example, the input elements listed in <ActiveList> to </ ActiveList> are currently active, and output corresponding to the output element “CopyStart_Message” is performed.
[0116]
The mobile phone 1307 receives this transmission information via the communication unit 1314. Based on the transmission information and the MMUI markup language 1315, the modality control unit 1313 performs output via the modality bound to “CopyStart_Message” (for example, “synthesizes” and notifies with a synthesized voice), The modalities bound to the input elements listed in <ActiveList> to </ ActiveList> are set to active, and other modalities are set to inactive. If the user inputs to the mobile phone 1307, the ID of the input element bound to the modality and the input content are transmitted from the mobile phone 1307 to the copier 1301.
[0117]
An example of this transmission information is shown in FIG. This example means that the copier 1301 is notified that there has been an input to the input elements listed in <InputList> to </ InputList>. As described above, the copy machine can be operated using the mobile phone 1307 as an operating device.
[0118]
As described above, according to the second embodiment, in an environment in which the processing related to the operation device and the control target device is realized based on the markup language, in particular, in the markup language,
Dialogue description of controlled devices
Modality description of operating device
Bind description that links the dialog description and modality description
Configure.
[0119]
The operation device is linked to each modality according to the description of the modality description in the markup language and the modality management unit that performs input / output management using each modality and according to the description of the bind layer in the markup description language. A modality control unit that exchanges information with the input / output elements is configured. In addition, the control target device has a dialog execution unit that accepts input from each modality managed by the modality management unit and executes output instructions to each modality according to the content of the dialog description in the markup language. To do.
[0120]
As a result, an arbitrary device (for example, PDA, remote controller, etc.) is automatically controlled, and an arbitrary device (OA device such as a facsimile, printer, scanner, etc., home appliances such as an electric kettle, a refrigerator, a television) is set as a control target device. It can be configured as an operating device. The operating device can be configured as an operating device having a multimodal user interface.
[Embodiment 3]
In the second embodiment, when the operation device (mobile phone) and the control target device (copy machine) are different from each other, the example in which the interaction layer is only on the control target device side has been described. In the third embodiment, an example will be described in which a dialogue layer unique to the manipulation device can be described on the manipulation device side separately from the dialogue layer of the control target device.
[0121]
For example, consider a case where a UI for operating an air conditioner from a remote controller as shown in FIG. 17 is described in a markup language. In the air conditioner interaction layer, a field that selects the air volume from “light”, “normal”, and “strong wind” is defined, and an appropriate modality is bound to the input element (“selectOne” type) of this field. .
[0122]
If the operating device has sufficient types of GUI parts, for example, a radio button or a pull-down menu may be bound to this input element. However, when binding the button A1602 of the remote controller, there is only a description that the air volume is switched by repeatedly pressing the button A1602. However, since the meaning of the button press, that is, the corresponding air volume changes each time the button is pressed, it cannot be simply bound.
[0123]
In such a case, the markup language of the present invention can describe a dialog layer specific to the operating device (remote control in the third embodiment) separately from the dialog layer of the control target (the air conditioner in the third embodiment). it can. FIG. 18 shows a schematic example of binding to the remote control interaction layer specific to the remote control and the air conditioner interaction layer in this example. In this example, the modality of button A1602 is bound to the input element of the interactive layer unique to the remote controller simultaneously with the air volume setting input element of the interactive layer of the air conditioner. By describing the switching of the input value in the filled element of the field having the latter input element, the value input to the air volume setting input element of the air conditioner interactive layer can be switched each time the button A is pressed. Also, the XML representation of the configuration of FIG. 18 is as shown in FIGS. 27A and 27B.
[Embodiment 4]
In the above embodiment, only user operations (input / output) are handled. However, event processing is also an important factor when considering device control. The markup language of the present invention can also describe information related to events (event information) such as event reception, event processing, and event type, and the embodiment will be described here. Event reception is described by a catch tag inside the field tag in the dialog layer.
[0124]

The event name such as “CopyComplete” in the above example is defined in advance according to the device, and the MMML browser knows these vocabularies.
[0125]
Events need to be considered separately for events related to operating devices (clients) and events related to controlled objects (servers). For example, consider a case where a copier is operated from a PC via a network. In this case, the event “copy completed” is an event related to the copy machine to be controlled, and should be described in the interaction layer of the copy machine.
[0126]
On the other hand, the event “Sound device is turned off” of the PC is an event that is closed in the PC that is the operating device, and is an event that is irrelevant to the interactive layer of the copier. In such a case, the description of the event closed in the PC may constitute a dialog layer unique to the PC other than the dialog layer of the copier, and a field for catching the event may be provided therein. FIG. 19 shows a description example including event information.
[Embodiment 5]
In the first embodiment, a case has been described in which a program that realizes the operation executed by each device described above is held in the ROM. However, the present invention is not limited to this, and the program may be realized using any storage medium. Good. Further, it may be realized by a circuit that realizes the same operation.
[0127]
Although the embodiment has been described in detail above, the present invention may be applied to a system constituted by a plurality of devices, or may be applied to an apparatus constituted by one device.
[0128]
The present invention supplies a software program (in the embodiment, a program corresponding to the flowchart shown in the drawing) to the system or apparatus directly or remotely from the above-described embodiment, and the computer of the system or apparatus Is also achieved by reading and executing the supplied program code. In that case, as long as it has the function of a program, the form does not need to be a program.
[0129]
Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.
[0130]
In this case, the program may be in any form as long as it has a program function, such as an object code, a program executed by an interpreter, or script data supplied to the OS.
[0131]
As a recording medium for supplying the program, for example, floppy disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R).
[0132]
As another program supply method, a client computer browser is used to connect to an Internet homepage, and the computer program of the present invention itself or a compressed file including an automatic installation function is downloaded from the homepage to a recording medium such as a hard disk. Can also be supplied. It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.
[0133]
In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. It is also possible to execute the encrypted program by using the key information and install the program on a computer.
[0134]
In addition to the functions of the above-described embodiments being realized by the computer executing the read program, the OS running on the computer based on the instruction of the program is a part of the actual processing. Alternatively, the functions of the above-described embodiment can be realized by performing all of them and performing the processing.
[0135]
Furthermore, after the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion board or The CPU or the like provided in the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are also realized by the processing.
[0136]
【The invention's effect】
As described above, according to the present invention, it is possible to provide an information processing apparatus, a control method thereof, and a program that can realize modality control that is more versatile and easy to expand.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating a structure of a markup language according to a first embodiment of the present invention.
FIG. 2 is a diagram illustrating a schema description of a general-purpose modality according to the first embodiment of the present invention.
FIG. 3 is a diagram illustrating a configuration example of a UI of the copier according to the first embodiment of the present invention.
FIG. 4 is a diagram schematically showing the schema description of the modality of the copier according to the first embodiment of the present invention.
FIG. 5 is a diagram schematically showing a schema description of a copier modality by a UI developer according to the first embodiment of the present invention;
FIG. 6 is a diagram schematically showing an interactive layer of the copier according to the first embodiment of the present invention.
FIG. 7 is a diagram schematically illustrating a part of the description of the bind layer according to the first embodiment of the present invention.
FIG. 8 is a diagram schematically illustrating a part of the description of the bind layer according to the first embodiment of the present invention.
FIG. 9 is a diagram schematically illustrating a part of the description of the bind layer according to the first embodiment of the present invention.
FIG. 10 is a functional configuration diagram of the copier according to the first embodiment of the present invention.
FIG. 11 is a configuration diagram showing a hardware configuration of the copier according to the first embodiment of the present invention.
FIG. 12 is a flowchart illustrating processing executed by the copier according to the first embodiment of the present invention.
FIG. 13 is a configuration diagram of an information processing system according to a second embodiment of the present invention.
FIG. 14 is a functional configuration diagram of an information processing system according to a second embodiment of this invention.
FIG. 15 is a diagram illustrating an example of information transmitted from the copier according to the second embodiment of the present invention to a mobile phone.
FIG. 16 is a diagram showing an example of information transmitted from the mobile phone to the copier according to the second embodiment of the present invention.
FIG. 17 is a diagram illustrating an example of a remote controller for operating an air conditioner according to a third embodiment of the present invention.
FIG. 18 is a diagram schematically showing a dialogue layer unique to an operating device according to the third embodiment of the present invention.
FIG. 19 is a diagram schematically illustrating a description of event information according to the fourth embodiment of the present invention.
FIG. 20A is a diagram showing an XML expression example of a general-purpose modality schema according to the first embodiment of the present invention.
FIG. 20B is a diagram showing an XML expression example of a general-purpose modality schema according to the first embodiment of the present invention.
FIG. 20C is a diagram showing an XML expression example of a general-purpose modality schema according to the first embodiment of the present invention.
FIG. 20D is a diagram showing an XML expression example of a general-purpose modality schema according to the first embodiment of the present invention.
FIG. 20E is a diagram showing an XML expression example of a general-purpose modality schema according to the first embodiment of the present invention.
FIG. 21A is a diagram showing an XML expression example of a modality schema of the copier according to the first embodiment of the present invention.
FIG. 21B is a diagram illustrating an XML expression example of the modality schema of the copier according to the first embodiment of the present invention.
FIG. 22A is a diagram showing an XML expression example of the interaction layer of the copier newly defined by the UI developer according to the first embodiment of the present invention.
FIG. 22B is a diagram showing an XML expression example of the interaction layer of the copier newly defined by the UI developer according to the first embodiment of the present invention.
FIG. 22C is a diagram showing an XML expression example of the interaction layer of the copier newly defined by the UI developer according to the first embodiment of the present invention.
FIG. 22D is a diagram showing an XML expression example of the interaction layer of the copier newly defined by the UI developer according to the first embodiment of the present invention.
FIG. 23A is a diagram showing an XML expression example of an interaction layer of the copier according to the first embodiment of the present invention.
FIG. 23B is a diagram illustrating an XML expression example of an interaction layer of the copier according to the first embodiment of the present invention.
FIG. 23C is a diagram showing an XML expression example of an interaction layer of the copier according to the first embodiment of the present invention.
FIG. 24A is a diagram showing an XML expression example of a bind layer of the copier according to the first embodiment of the present invention.
FIG. 24B is a diagram showing an XML expression example of the bind layer of the copier according to the first embodiment of the present invention.
FIG. 24C is a diagram showing an XML expression example of the bind layer of the copier according to the first embodiment of the present invention.
FIG. 24D is a diagram showing an XML expression example of the bind layer of the copier according to the first embodiment of the present invention.
FIG. 25 is a diagram showing an XML expression example of a mobile phone modality schema according to the second embodiment of the present invention;
FIG. 26 is a diagram illustrating an XML expression example of a bind layer that binds input / output elements of a mobile phone modality and a copier's dialog description according to the second embodiment of the present invention;
FIG. 27A is a diagram showing an XML expression example of a dialog layer unique to an operating device according to the third embodiment of the present invention.
FIG. 27B is a diagram showing an XML expression example of the interaction layer unique to the operating device according to the third embodiment of the present invention.
[Explanation of symbols]
101 Input / output management unit
102 GUI control unit
103 Voice recognition unit
104 Speech synthesis unit
105 Modality Management Department
106 Modality control unit
107 Dialogue execution unit
108 Device control unit
109 MMUI markup language

Claims (28)

An information processing apparatus having a plurality of types of modalities,
Control means for controlling processing executed by the information processing apparatus based on data stored in the memory;
A dialog description indicating a processing procedure of the process executed by the information processing apparatus, a modality description indicating a modality of the information processing apparatus configured separately from the dialog description, the modality description, and the dialog description. Storage means for storing description data indicating a binding description to be associated in the memory;
Binding means for associating the input / output process included in the processing procedure indicated in the dialog description in the description data stored in the memory by the control means with the modality included in the modality description according to the bind description;
When the control unit controls the information processing apparatus based on the dialog description in the description data stored in the memory, the data required by the input / output process corresponds to the input / output process by the bind unit. An information processing apparatus comprising: execution means for executing control to and from the attached modality. The dialog description is composed of one or a plurality of minimum dialog elements, which are input / output processing units, and the minimum dialog elements include zero or one input element and zero or one or more output elements. The information processing apparatus according to claim 1, further comprising: The information processing apparatus according to claim 2, wherein the minimum interaction element includes a description of event information regarding an event. The information processing apparatus according to claim 2, wherein the minimum interaction element includes a description of action information regarding an action for an input to the minimum interaction element. The minimum interaction element has one of two states of active / inactive, and the minimum interaction element includes a description for activating / deactivating itself or another minimum interaction element as the action information. The information processing apparatus according to claim 4. The information processing apparatus according to claim 2, wherein the output element includes a description indicating the output content. 2. The information according to claim 1, wherein the modality description includes at least one of definitions indicating the class of the modality, inheritance, a hierarchical relationship between classes, and definitions of instances of the class. Processing equipment. The information processing apparatus according to claim 1, wherein each modality in the modality description has one of two states of active / inactive. The binding description is a description for binding one or a plurality of modalities to one or a plurality of input elements. When a plurality of modalities are bound to a single input element, how to combine the modalities The information processing apparatus according to claim 2, further comprising a description. The binding description is a description for binding one or a plurality of modalities to one or a plurality of output elements. When a plurality of modalities are bound to a single output element, how to combine the modalities The information processing apparatus according to claim 2, further comprising a description. The binding description includes at least one of a description indicating the output content in the bind element, which is a description indicating the connection between the modality and the output element, and a description indicating the output content for each modality linked to the output element,
The output content described in the bind element takes precedence over the output content described in the output element that is the binding target, and the output content described for each modality in the bind element is the binding content. The information processing apparatus according to claim 2, wherein the information processing apparatus has priority over the output contents described in the element. The information processing apparatus according to claim 11, wherein each binding element of the binding description includes a description for activating / deactivating a modality that is a binding target. An information processing apparatus that operates as an operation device of a control target device,
Control means for controlling processing executed by the information processing apparatus based on data stored in the memory;
The modality description indicating the modality of the information processing apparatus and the bind description that associates the modality description with the dialog description, which are configured separately from the dialog description indicating the processing procedure of the process executed by the control target device. Storage means for storing descriptive data in the memory;
Binding means for associating the input / output process included in the processing procedure indicated in the dialog description in the description data stored in the memory by the control means with the modality included in the modality description according to the bind description;
When the control unit controls the control target device based on the dialog description in the description data stored in the memory, the data required by the input / output process corresponds to the input / output process by the bind unit. An information processing apparatus comprising: execution means for executing control to and from the attached modality. A control method for controlling an information processing apparatus having a plurality of types of modalities,
A dialog description indicating a processing procedure of the information processing apparatus; a modality description indicating a modality of the information processing apparatus configured separately from the dialog description; and a bind description that associates the modality description with the dialog description. A reading step of reading the description data from a memory storing the description data by a control unit that controls processing executed by the information processing apparatus;
A binding step of associating the input / output process included in the processing procedure indicated in the dialog description in the description data read from the memory by the control unit with the modality included in the modality description according to the bind description;
When the control unit controls the information processing apparatus based on the dialog description in the description data read from the memory, the data requested by the input / output process is combined with the input / output process by the bind step. And a control step of controlling to transfer to and from the associated modality. The dialog description is composed of one or a plurality of minimum dialog elements, which are input / output processing units, and the minimum dialog elements include zero or one input element and zero or one or more output elements. The information processing apparatus control method according to claim 14, further comprising: The information processing apparatus control method according to claim 15, wherein the minimum interaction element includes a description of event information related to an event. The information processing apparatus control method according to claim 15 or 16, wherein the minimum interaction element includes a description of action information regarding an action for an input to the minimum interaction element. The minimum interaction element has one of two states of active / inactive, and the minimum interaction element includes a description for activating / deactivating itself or another minimum interaction element as the action information. An information processing apparatus control method according to claim 17. The information output apparatus control method according to claim 15, wherein the output element includes a description indicating an output content thereof. The information according to claim 14, wherein the modality description includes at least one of definitions indicating the class of the modality, inheritance, a hierarchical relationship between classes, and definitions of instances of the class. A method for controlling a processing apparatus. The method for controlling an information processing apparatus according to claim 14, wherein each modality in the modality description has one of two states of active / inactive. The binding description is a description for binding one or a plurality of modalities to one or a plurality of input elements. When a plurality of modalities are bound to a single input element, how to combine the modalities The information processing apparatus control method according to claim 16, further comprising a description. The binding description is a description for binding one or a plurality of modalities to one or a plurality of output elements. When a plurality of modalities are bound to a single output element, how to combine the modalities 15. The information processing apparatus control method according to claim 14, further comprising a description. The binding description includes at least one of a description indicating the output content in the bind element, which is a description indicating the connection between the modality and the output element, and a description indicating the output content for each modality linked to the output element,
The output content described in the bind element takes precedence over the output content described in the output element that is the binding target, and the output content described for each modality in the bind element is the binding content. The information processing apparatus control method according to claim 14, wherein priority is given to output contents described in an element. 25. The method of controlling an information processing apparatus according to claim 24, wherein each binding element of the binding description includes a description for activating / deactivating a modality that is a binding target. A method of controlling an information processing apparatus that operates as an operation device of a control target device,
Stores description data indicating the modality description indicating the modality of the information processing apparatus and the binding description for associating the modality description and the dialog description, which are configured separately from the dialog description indicating the processing procedure of the control target device Reading out the description data from the memory to be read by a control unit that controls processing executed by the information processing apparatus;
A binding step of associating the input / output process included in the processing procedure indicated in the dialog description in the description data read from the memory by the control unit with the modality included in the modality description according to the bind description;
When the control unit controls the control target device based on the dialog description in the description data read from the memory, the data requested by the input / output process is combined with the input / output process by the binding step. And a control step of controlling to transfer to and from the associated modality. A program for causing a control of the information processing apparatus having a plurality of types of modalities computer,
A dialog description indicating a processing procedure of the information processing apparatus; a modality description indicating a modality of the information processing apparatus configured separately from the dialog description; and a bind description that associates the modality description with the dialog description. A program code for a reading process of reading the description data from a memory storing the description data by a control unit that controls processing executed by the information processing apparatus;
A program of a binding step in which the control unit associates the input / output process included in the processing procedure indicated in the dialog description in the description data read from the memory with the modality included in the modality description according to the bind description. Code,
When the control unit controls the information processing apparatus based on the dialog description in the description data read from the memory, the data requested by the input / output process is combined with the input / output process by the bind step. And a program code of a control process for controlling to be transferred to and from the associated modality. A program for causing a computer to control the control of an information processing device that operates as an operation device of a control target device,
Stores description data indicating the modality description indicating the modality of the information processing apparatus and the binding description for associating the modality description and the dialog description, which are configured separately from the dialog description indicating the processing procedure of the control target device. Reading the program data from the memory by the control unit that controls the processing executed by the information processing apparatus;
A program of a binding step in which the control unit associates the input / output process included in the processing procedure indicated in the dialog description in the description data read from the memory with the modality included in the modality description according to the bind description. Code,
When the control unit controls the control target device based on the dialog description in the description data read from the memory, the data requested by the input / output process is combined with the input / output process by the binding step. And a program code of a control process for controlling to be transferred to and from the associated modality.

Images