JP2006222622A - Input control system, and controller and control program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an input control system capable of controlling an arbitrary device to be controlled that a user desires among many devices to be controlled by using various objects as input devices. <P>SOLUTION: The input control system is equipped with an input device, a photographing device, and a controller, which receives information on an operation kind given to the input device from the input device or a photographed image of the input device from the photographing device photographing the input device to recognize the operation kind of the input device based upon change of the photographed image from reference time. Then control information recorded in a control information storage means while being associated with the received information on the operation kind is read out and the control information is transmitted to the device 4 to be controlled which is indicated with input identification information of the device 4 to be controlled. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an input control system, a control device, and a control program for controlling a predetermined device.

In the ubiquitous society, computers exist everywhere, input devices corresponding to the computers exist, and control signals are transmitted from the input devices to specific control target devices corresponding to the input devices. It is very convenient to have an input device so that a signal can be sent to a control target device such as a computer at any location. Recently, a mechanism has been devised and implemented that uses a mobile phone and a TV capable of infrared communication, and operates the TV using a TV control application downloaded to the mobile phone. In this way, it is possible to send signals to various computers by downloading an application corresponding to the device or computer to which the signal should be sent. In order to send signals to various devices and computers using the input device, the input device is always carried around and needs to be a wearable device, so it is desirable to be compact. There is also a need to provide an easy-to-understand user interface for use by a wide variety of users.
In addition, a technique has been disclosed in which remote control data corresponding to remote operation is downloaded in advance for each operation target device from a stored server on the Internet, and the operation target device is controlled using the data (for example, , See Patent Document 1).
JP 2004-48777 A

However, since the mobile terminal is an input device in the above-described technique, the user can only input from the mobile terminal held by the user to the device to be controlled. Here, for example, in a future ubiquitous society, an input device is preliminarily provided somewhere on a street corner, and a user among many devices desires using an input device previously installed on the street corner. There is a possibility that a system for controlling a device to be controlled is required. Alternatively, there is a possibility that a system that uses a familiar object itself as an input device is required.
The above-described technique can send signals to various devices to be controlled. For example, when the mobile terminal is a mobile phone, the display unit that provides the user interface is a mobile phone. If the part is 3-4cm square and an easy-to-understand user interface is provided, the button display using a picture that anyone can understand is at best, and if the display is difficult to convey, the Help page will be displayed. It is necessary to provide separately and instruction manuals.
In addition, the above-mentioned technology can provide a very flexible input interface by providing a touch panel display on a mobile terminal. However, as described above, everyone understands that it will provide a user interface that is easy to understand on a single screen. It is necessary to provide a user interface that is easy to understand by making the button display using a picture that can be displayed, or by adding a description to the button with characters etc. The device itself gets bigger and harder to carry in wearables.

  Therefore, the present invention can control an arbitrary control target device desired by a user among many control target devices by using various objects as an input device, and the size of the user interface can be increased. An object of the present invention is to provide an input control system in which the function of the input device is not restricted by the restriction.

  The present invention has been made to solve the above-described problem, and is an input control system that includes at least an input device, a photographing device, and a control device, and controls a control target device, wherein the control device includes the control device. Identification information receiving means for receiving input of identification information of the target device and identification information of the input device, action type information receiving means for receiving information on the action type given to the input device from the input device, or One of operation type recognition means for receiving a captured image of the input device from the imaging device capturing the input device and recognizing the operation type of the input device based on a change from the reference time of the captured image, Control information for controlling the control target device for each operation type is recorded in association with the identification information of the control target device that has received the input and the identification information of the input device. Control information storage means that reads the control information recorded in the control information storage means in association with the received operation type information and identifies the control information device that received the input An input control system comprising: control information transmitting means for transmitting to the device to be controlled indicated by information.

  In the present invention, the control system includes a display device, and the control device controls the control target device in association with the identification information of the control target device and the identification information of the input device. Input auxiliary image storage means for storing an input auxiliary image for assisting the operation applied to the input device, identification information storage means for storing identification information of the display device in association with the input device, and the input received Input auxiliary image transmitting means for transmitting the input auxiliary image to the display device indicated by the identification information of the display device stored in the identification information storage means corresponding to the identification information of the input device, and the display The apparatus receives a photographed image of the input device photographed by the photographing apparatus and displays it on a display unit, and a photographed image of the input device displayed on the photographed image display means Characterized by and an input auxiliary image synthesizing means for displaying overlapping the input auxiliary image received from the controller.

  In the input control system according to the present invention, the operation type is information on a type of a button included in the input device, a position variation of the input device, or a rotation angle of the input device. To do.

  Further, the present invention is the control device in an input control system that includes at least an input device, an imaging device, and a control device, and controls the control target device, wherein the control target device identification information and the input device identification The identification information receiving means for receiving input of information, the action type information receiving means for receiving information on the action type given to the input apparatus from the input apparatus, or the imaging apparatus that is shooting the input apparatus One of operation type recognition means for receiving a captured image of the input device and recognizing the operation type of the input device based on a change from the reference time of the captured image, and identification information of the control target device that has received the input And control information storage means for storing control information for controlling the control target device for each operation type in association with the identification information of the input device, The control target device that reads the control information recorded in the control information storage means in association with the received operation type information, and indicates the control information by the identification information of the control target device that received the input And a control information transmitting means for transmitting to the control device.

  Further, according to the present invention, the control device described above assists an operation applied to the input device when controlling the control target device in association with the identification information of the control target device and the identification information of the input device. Input auxiliary image storage means for storing an auxiliary image, identification information storage means for storing identification information of a display device in association with the input device, and the identification corresponding to the identification information of the input device that has received the input Input auxiliary image transmitting means for transmitting the input auxiliary image to the display device indicated by the identification information of the display device stored in the information storage means.

  Further, the present invention is a program to be executed by a computer of the control device in an input control system that includes at least an input device, a photographing device, and a control device, and controls the control target device, the identification information of the control target device And an identification information receiving process for receiving an input with the identification information of the input device, an action type information receiving process for receiving information of an action type given to the input device from the input device, or photographing the input device. One of an operation type recognition process for receiving a captured image of the input device from the imaging device and recognizing an operation type of the input device based on a change from the reference time of the captured image, and receiving the input. In association with the identification information of the control target device and the identification information of the input device, control information for controlling the control target device for each operation type The control information storage means stores the control information recorded in association with the received operation type information, and the control information is indicated by the identification information of the device to be controlled that has received the input. This is a program for causing a computer to execute control information transmission processing to be transmitted to a device to be controlled.

  In addition to the processing of the above-described program, the present invention assists the operation to be added to the input device when controlling the control target device in association with the identification information of the control target device and the identification information of the input device. Corresponding to the identification information of the input device that received the input in the input auxiliary image storage processing for storing the input auxiliary image and the identification information storage means for storing the identification information of the display device in association with the input device. The program further causes the computer to execute input auxiliary image transmission processing for transmitting the input auxiliary image to the display device indicated by the stored identification information of the display device.

  According to the present invention, the user can remotely control the device to be controlled. In addition, when the input device is installed in advance in a predetermined place and the user wants to control an arbitrary control target device desired by the input device, the input auxiliary image of the arbitrary control target device is used. However, the control target apparatus can be controlled. Further, by using a head-mounted display or the like for the display unit, there is no restriction on the display unit such as a portable terminal, and the input auxiliary image of the input device can be synthesized and displayed in the entire field of view of the user.

  Hereinafter, an input control system according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the input control system according to the embodiment. In this figure, reference numeral 1 denotes a control server (control device) that controls the device to be controlled. Reference numeral 2 denotes an input device, which is provided with a predetermined input unit (such as a button). Reference numeral 3 denotes a head mounted display (hereinafter referred to as HMD) (display device) having a photographing function, and an input device photographed by the photographing function is displayed on the display. In the present embodiment, the input control system is described using the HMD 3, but a mobile terminal such as a mobile phone or a PDA (Personal Digital Assistants) may be used instead of the HMD 3. Reference numeral 4 denotes an apparatus to be controlled. For example, electrical appliances such as a television, an air conditioner, and an illuminator, and other industrial products that operate by inputting an electric signal are conceivable. And the control server 1, the input device 2, HMD3, and the control object apparatus 4 are connected via the communication network. Here, the control server 1 may be provided in the HMD 3 itself.

  In the control server 1, reference numeral 11 denotes a communication processing unit (identification information receiving unit, operation type information receiving unit) that transmits and receives information to and from other devices via a communication network. Reference numeral 12 denotes a control unit (input auxiliary image transmission unit) that controls each processing unit of the control server 1. Reference numeral 13 denotes a database (control information storage means, input auxiliary image storage means, identification information storage means) that stores various types of information. Reference numeral 14 denotes an operation type recognition unit (operation type recognition means) that recognizes the operation type based on information input to the input device or an operation given to the input device. Reference numeral 15 denotes a control information transmission processing unit (control information transmitting means) that transmits a signal for controlling the control target device 4 to the control target device 4 based on the recognized action type.

  In the input device 2, reference numeral 21 denotes a communication processing unit that transmits / receives information to / from other devices via a communication network. A control unit 22 controls each processing unit of the input device 1. Reference numeral 23 denotes a storage unit for storing various information. In this embodiment, the identifier of each button provided in the input device 2 is recorded in this storage unit. Reference numeral 24 denotes an operation notification processing unit, which performs processing of transmitting the identifier of the pressed button to the control server 1 by a signal in the present embodiment.

  In the HMD 3, reference numeral 31 denotes a communication processing unit that transmits / receives information to / from other devices via a communication network. A control unit 32 controls each processing unit of the HMD 3. Reference numeral 33 denotes a storage unit for storing various information. Reference numeral 34 denotes a photographing unit that performs photographing processing. Reference numeral 35 denotes a display unit (photographed image display means, input auxiliary image composition means) for displaying a photographed image.

  In the present embodiment, the control server 1 receives the information of the type of operation given to the input device 2 from the input device or the captured image of the input device received from the HMD 3 capturing the input device. Recognize more. Then, control information corresponding to the information of the operation type is transmitted to the control target device 4 to control the control target device 4.

Next, each table stored in the database will be described.
FIG. 2 is a diagram showing a registration control table.
As shown in FIG. 2, in the registration control table, identification information of the control target device 4, the input device 2, and the HMD 3 is recorded in association with each other by registration from the user.
FIG. 3 is a diagram showing a control information specifying table.
As shown in FIG. 3, in the control information identification table, the operation type of the input device (for example, the identifier of the button of the input device) can be determined in association with the identification information of the control target device 4 and the input device 2. ) And control information (such as a command to be transmitted to the control target apparatus 4) of the control target apparatus 4 corresponding to the operation type. Control information for controlling not only the control target apparatus 4 but also the HMD 3 itself may be used.
FIG. 4 is a diagram showing an input auxiliary image specifying table.
As shown in FIG. 4, in the input auxiliary image specifying table, the display unit 35 of the HMD 3 is confirmed when controlling the control target device 4 in association with the identification information of the control target device 4 and the input device 2. However, an input auxiliary image for assisting the operation applied to the input device 2 by the user is stored.

FIG. 5 is a diagram showing a processing flow of the input control system.
Next, processing of the input control system according to the present embodiment will be described with reference to FIG.
First, the user registers the identification information of the input device 2 and the identification information of the control target device 4 to be controlled in the control server 1 (step S1). For example, the user inputs the identification information of the input device 2 and the identification information of the control target device 4 from the input unit provided in the HMD 3, and the control unit 32 of the HMD 3 automatically inputs the identification information input to the control server 1. By performing the process of transmitting the identification information of the device (HMD3), each of the three pieces of identification information is recorded in association with the database 13 (registration table in FIG. 2). Here, the identification information may be an ID of each device, for example, an IP address or a name address.

  Next, the control server 1 maps the operation types recorded in the control information specifying table of the database 13 in association with the identification information of the registered input device 2 and control target device 4 and the corresponding control information. Is read and stored in a memory or the like (step S2). Then, the control unit 2 of the control server 1 waits for reception of information from the input device or the HMD 3.

  On the other hand, the user photographs the input device 2 with the camera provided in the HMD 3, and the photographing unit 34 generates an image (step S3). As a result, the display unit 35 displays the captured image of the input device (step S4). The control unit 32 of the HMD 3 instructs the communication processing unit 31 to transmit the captured image data to the control server 1, and the communication processing unit 31 transmits the captured image data of the input device 2 to the control server 1. Thereafter, the HMD 3 always transmits the captured image data of the input device 2 captured to the control server 1 using a streaming captured image distribution method or the like (step S5).

  When the control server 1 receives the captured image data of the input device 2, the control server 12 determines whether the captured image data is an input device by comparing it with an image stored in advance. Based on this, the direction of the vector on the predetermined surface of the input device 2 and the angle with respect to the camera are calculated. When the control unit 12 controls the control target device 4 from the input auxiliary image specifying table of the database 13 in association with the input device 2 and the control target device 4 input in advance by the user, the display unit 35 of the HMD 3 is used. The input auxiliary image for assisting the user to apply to the input device 2 is read while confirming (step S6). Then, the control unit 12 corrects the input auxiliary image so that it can be combined with the input device 2 based on the direction of the vector on the predetermined surface of the input device 2 and the angle with respect to the camera (step S7). That is, processing for matching the vector of the input auxiliary image with the direction and angle of the input device 2 photographed by the camera is performed using 3D modeling processing or the like. Then, the communication processing unit 11 of the control server 1 transmits the corrected input auxiliary image to the HMD 3 (step S8). Since the captured image is deformed in response to the transmission of the captured image from the HMD 3 by streaming, the control unit 12 of the control server 1 repeats the process of correcting the input auxiliary image at a predetermined time interval.

  Next, when the HMD 3 receives the input auxiliary image, the control unit 32 combines the received input auxiliary image with the captured image of the input device 2 already displayed on the display unit 35 and displays it on the display unit 35 ( Step S9). The user presses the button of the input device 2 while looking at the input auxiliary image displayed on the display unit 35 of the HMD 3 (step S10). For example, when the control target device 4 is an air conditioner, the input auxiliary image has four meanings of “rising”, “decreasing”, “large”, and “small” of the air flow on the four buttons of the input device 2, respectively. If it is, the user presses the button according to the meaning of each button of the input auxiliary image of the temperature and the air volume. Then, the operation notification processing unit 24 of the input device 2 reads the pressed button identifier or the like from the storage unit 23, and the communication processing unit 21 transmits the button identifier to the control server 1 via the communication network. (Step S11). Thereby, the operation type given to the input device 2 is notified to the control server 1.

  Next, the control server 1 receives the identifier of the button A. Then, the action type recognition unit 14 reads the control information from the mapping between the action type and the control information recorded in the memory corresponding to the identifier of the button A (step S12). The mapping information may be read directly from the control information specifying table of the database 13 instead of being stored in the memory in advance. And the control information transmission process part 15 performs the process which transmits the said control information to the control object apparatus 4 (step S13). Thereby, control information is transmitted from the communication processing unit 11 to the control target device 4. And the control object apparatus 4 performs a control process based on control information (step S14). For example, when the device to be controlled is an air conditioner, the control information is information indicating “rise”, “decrease” of the temperature, “large”, “small” of the air volume. Adjust the temperature of the device to “rise”, “fall”, and “large” and “small”.

  Through the above processing, the user can control the control target device 4 from a remote location. In addition, when the input device is installed in advance in a predetermined place and the user wants to control an arbitrary control target device 4 with the input device, an input auxiliary image of the arbitrary control target device 4 is displayed. The controlled device 4 can be controlled while being used. For example, as an application example, in remote medical instrument operation etc., the input device 2 has become an input device corresponding to many types of medical devices (for example, medical devices from different manufacturers), and using a head mounted display, There is an example in which a medical practice is performed by displaying an input auxiliary image corresponding to a medical device of a manufacturer desired by a doctor on a display unit. In the present embodiment, the configuration in which the control server 1 and the head mounted display 3 are connected via a communication network has been described. However, the control server 1 and the head mounted display 3 may be integrated. Further, by using the head mounted display 3, there is no limitation on the display unit such as a portable terminal, and the input auxiliary image of the input device 2 can be synthesized and displayed in the entire field of view of the user.

FIG. 6 is a diagram illustrating another example of the input auxiliary image.
The input auxiliary image in this figure is an input auxiliary image when the television is the control target device 4. For example, when there are five buttons on the input device 2, the meanings of the power source, channel (ch), volume, input switching (input), and timer are given to each. When the user presses the volume button, the input device 2 transmits an identifier corresponding to the button to the control server 1. In the control server 1, the control information corresponding to the received identifier is read from the mapping information stored in the memory in advance. Here, if the control information is a command that causes the device to transmit a new input image, the control server 1 newly reads an input auxiliary image for inputting a volume from the database 13 by the command, and the input auxiliary image is transferred to the HMD 3. Send. This input auxiliary image is an image that means, for example, low, high, and mute of the volume as buttons. When one of the small, large, and mute buttons is pressed by the user input device 2, the identifier of the button is transmitted to the control server 1, and the television as the control target device 4 adjusts the volume as described above. The process to do is performed.

FIG. 7 is a schematic diagram showing a process for recognizing an action type according to a change in an image of the input device.
The example in which the information input to the input device 2 is transmitted from the input device 2 to the control server 1 and the operation type of the control target device 4 is recognized based on the information transmitted from the input device has been described above. However, in the input control system of the present embodiment, the operation type may be recognized according to a change in the captured image of the input device 2.
For example, as shown in FIG. 7A, it is assumed that the input auxiliary image corresponding to the input device is a rotary channel image. A user rotates an input device (a pen in FIG. 7) as shown in this figure by a predetermined angle (rotates in accordance with the arrow direction of the input auxiliary image). Then, the captured image is transmitted to the control server 1. As described above, the control unit 12 of the control server 1 calculates the vector direction and the angle with respect to the camera in a predetermined plane of the input device 2 in the captured image. Then, using the rotation angle of the input device as the operation type, the control information corresponding to the rotation angle is read from the mapping information and transmitted to the control target device 4. That is, the control unit 12 recognizes the operation type of the input device based on the change of the captured image from the reference time. Thereby, the control object apparatus 4 can be controlled according to the change of the image of the input apparatus 2. In addition, since the control target device 4 can be controlled by a change in the movement of the input device, for example, various objects close to the user can be used instead of the input device.

  This example can also be applied to an example in which characters and the like are input by the locus of the pen tip <(b) in FIG. 7>. As a process for recognizing the pen locus, first, the user continuously presses the button of the input device. Then, the button identifier is transmitted to the control server 1 multiple times while the button is being pressed. As a result, the control server 1 determines that the pen locus is input, and analyzes the locus according to the change in the locus of the image of the input device while the button is pressed. Thus, for example, by transmitting information on the trajectory to the control target device 4, the control target device 4 performs a process of displaying the trajectory on the display unit.

FIG. 8 is a diagram showing marks for determining the direction of a vector on a predetermined surface of the input device.
As shown in FIG. 8, the input device is provided with, for example, black circle marks at three locations, and the control unit 12 of the control server 1 determines the direction of the vector on a predetermined surface of the input device according to the positions and intervals of the marks. calculate. In addition, even if it is not a black circle mark, the indicator which can determine the direction of the vector in the predetermined surface of an input device should just be attached to the input device.

FIG. 9 is a diagram showing an outline of processing for recognizing misperception of the movement of the input device.
When recognizing the movement of the input device 2 from a captured image of the camera, it is conceivable that the movement on the camera side is erroneously recognized as the movement of the input device 2, but it is necessary to eliminate (reduce) the erroneous recognition. Therefore, as a process of grasping the relative position of the camera, a non-moving mark in the vicinity of the input device 2 to be imaged is searched (or the mark is displayed on the display on the service side), and this is used as a fixed point. If the movement of the input device is calculated based on the standard, erroneous recognition due to camera shake is reduced to some extent. At this time, the calculation is performed based on the coordinates in the captured image of the fixed point and the coordinates in the image of the input device 2. However, the distance between the fixed point and the input device 2 needs to be shortened. The longer it is, the lower the recognition rate.

  The control server 1, the input device 2, the head mounted display 3, and the control target device 4 described above have a computer system inside. The process described above is stored in a computer-readable recording medium in the form of a program, and the above process is performed by the computer reading and executing this program. Here, the computer-readable recording medium means a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like. Alternatively, the computer program may be distributed to the computer via a communication line, and the computer that has received the distribution may execute the program.

  The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

It is a block diagram which shows the structure of the input control system by embodiment of this invention. It is a figure which shows the registration control table by this embodiment. It is a figure which shows the control information specific table by this embodiment. It is a figure which shows the input auxiliary image specific table by this embodiment. It is a figure which shows the processing flow of the input control system by this embodiment. It is a figure which shows the other example of the input auxiliary image by this embodiment. It is a schematic diagram which shows the process at the time of recognizing operation | movement classification according to the change of the image of the input device by this embodiment. It is a figure which shows the mark for determining the direction of the vector in the predetermined surface of the input device by this embodiment. It is a figure which shows the outline | summary of the process for recognizing the misperception of the motion of the input device by this embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Control server, 2 ... Input device, 3 ... Head mounted display (HMD), 4 ... Control object apparatus

Claims (7)

An input control system that includes at least an input device, a photographing device, and a control device, and controls the device to be controlled.
The control device is
Identification information receiving means for receiving input of identification information of the device to be controlled and identification information of the input device;
The action type information receiving means for receiving the action type information given to the input apparatus from the input apparatus, or the captured image of the input apparatus is received from the imaging apparatus that is shooting the input apparatus. One of action type recognition means for recognizing the action type of the input device based on a change from the reference time,
Control information storage means for storing control information for controlling the control target device for each operation type in association with the identification information of the control target device that has received the input and the identification information of the input device;
Read the control information recorded in the control information storage means in association with the received operation type information, and the control information is indicated by the identification information of the control target device that received the input Control information transmitting means for transmitting to the device;
An input control system comprising: The input control system comprises a display device;
The control device is
An input auxiliary image storage means for storing an input auxiliary image for assisting an operation applied to the input device when the control target device is controlled in association with the identification information of the control target device and the identification information of the input device; ,
Identification information storage means for storing identification information of the display device in association with the input device;
Input auxiliary image transmitting means for transmitting the input auxiliary image to the display device indicated by the identification information of the display device stored in the identification information storage means corresponding to the identification information of the input device that has received the input; With
The display device
A photographed image display means for receiving a photographed image of the input device photographed by the photographing apparatus and displaying the photographed image on a display unit;
Input auxiliary image combining means for displaying the input auxiliary image received from the control device on the captured image of the input device displayed on the captured image display means;
The input control system according to claim 1, further comprising:   The input according to claim 1, wherein the operation type is information on a type of a button included in the input device, a position variation of the input device, or a rotation angle of the input device. Control system. The control device in the input control system that includes at least an input device, a photographing device, and a control device, and controls the control target device,
Identification information receiving means for receiving input of identification information of the device to be controlled and identification information of the input device;
The action type information receiving means for receiving the action type information given to the input apparatus from the input apparatus, or the captured image of the input apparatus is received from the imaging apparatus that is shooting the input apparatus. One of action type recognition means for recognizing the action type of the input device based on a change from the reference time,
Control information storage means for storing control information for controlling the control target device for each operation type in association with the identification information of the control target device that has received the input and the identification information of the input device;
Read the control information recorded in the control information storage means in association with the received operation type information, and the control information is indicated by the identification information of the control target device that received the input Control information transmitting means for transmitting to the device;
A control device comprising: An input auxiliary image storage means for storing an input auxiliary image for assisting an operation applied to the input device when the control target device is controlled in association with the identification information of the control target device and the identification information of the input device; ,
Identification information storage means for storing identification information of a display device in association with the input device;
Input auxiliary image transmitting means for transmitting the input auxiliary image to the display device indicated by the identification information of the display device stored in the identification information storage means corresponding to the identification information of the input device that has received the input; ,
The control device according to claim 4, further comprising: An input control system that includes at least an input device, an imaging device, and a control device, and that is executed by a computer of the control device in an input control system that controls a control target device,
Identification information receiving processing for receiving input of identification information of the control target device and identification information of the input device;
An operation type information receiving process for receiving operation type information given to the input device from the input device, or a captured image of the input device received from the imaging device capturing the input device, One of the action type recognition processes for recognizing the action type of the input device based on the change from the reference time,
In the control information storage means for storing control information for controlling the control target device for each operation type in association with the identification information of the control target device that has received the input and the identification information of the input device, the reception Control information transmission processing for reading the control information recorded in association with the information of the action type and transmitting the control information to the control target device indicated by the identification information of the control target device that received the input When,
A program that causes a computer to execute. In addition to the processing of claim 5,
An input auxiliary image storage process for storing an input auxiliary image, which associates the identification information of the control target device and the identification information of the input device with each other and assists the operation applied to the input device when the control target device controls. ,
In the identification information storage means for storing the identification information of the display device in association with the input device, the display device indicated by the identification information of the display device stored corresponding to the identification information of the input device that received the input An input auxiliary image transmission process for transmitting the input auxiliary image;
A program that causes a computer to execute.

Images