JP2007116270A - Terminal and apparatus control system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To implement a user-friendly structure for selecting an apparatus to be remotely controlled by a terminal and creating and operating a user interface fitted to the apparatus. <P>SOLUTION: The terminal comprises a means for picking up an image, a means for holding the information representing the features of an apparatus, an image comparison means for determining whether the apparatus feature information is included in the image picked up or not, a means for selecting an apparatus to be operated, a means for acquiring a script describing the operation of the apparatus, a means for creating a user interface according to the script, a display control means for displaying the user interface on a display means, a means through which an operator inputs operational information by utilizing the user interface, and a communication means for communicating with the apparatus to be operated and transmitting control information to the apparatus. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a terminal device and a device control system for controlling a device by communication, and more particularly to a technique for controlling an unspecified device by giving a terminal device such as a mobile phone a function of a general-purpose remote controller. .

  In recent years, computerization in the home has progressed, and network / Internet related technologies have become widespread. Along with this, the information handled in the home has increased in both quality and quantity, and even in information home appliances that handle information, user needs for browsing information and operating devices with simple operations are increasing.

  However, with the increase in information handled at home, household information home appliances (hereinafter referred to as “devices”) continue to increase, and the general infrared remote control method for operating these devices is also proportional to the increase in devices. is increasing. In other words, a large number of infrared remote controllers are flooded in the home, which in turn has become a problem of reducing user convenience.

As a method for solving this problem, it is possible to realize a general-purpose remote controller called a universal controller in a form corresponding to a network by using a technique such as Patent Document 1, and a position where infrared rays do not reach. Certain devices can also be operated by remote control.
JP 2002-232978 A

  According to the technique described in Patent Document 1 described above, a general-purpose remote controller can be realized in a form corresponding to a network. However, in order to realize this, it is necessary to register the identifier of the target device in the remote controller in advance, and when the target device is newly added, the user has to register the identifier. Will occur.

  An object of the present invention is to realize a mechanism for selecting a device as a target for remote control operation by a terminal device and a mechanism for generating and operating a user interface suitable for the device without burdening an operator (user). There is to be able to do it.

  In one representative invention of the terminal device according to the present application, an image capturing unit that captures an image, a device feature information retaining unit that retains device feature information that expresses a feature of the device, and the device feature information is included in the captured image. Image comparison means for determining whether or not the device is selected, device selection means for selecting a device to be operated, script acquisition means for acquiring a script describing the operation of the device from a database or device, and a user interface according to the script (User interface image) generating user interface, display control means for displaying the user interface on the display means, input means for the operator to input operation information using the user interface, the operation information Information generation means for converting control information according to the script and generating control information And a communication means for communicating with the device to be operated to transmit the control information to the device comprises.

  According to the present invention, a mechanism for selecting a device as a target for remote control operation by a terminal device and a mechanism for generating and operating a user interface suitable for the device are realized without burdening the operator (user). it can.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration of a terminal device according to an embodiment of the present invention (hereinafter referred to as the present embodiment) and a device control system including the terminal device.

  The device control system includes a terminal device 1, one or more devices 5, and a device information database 7. The terminal device 1 includes a camera 4 as an imaging unit, a display device 2 as a display unit, and a touch panel device 3 as an input unit. The display device 2 and the touch panel device 3 are touch devices that are also input devices. It is comprised by the display panel apparatus. For example, a mobile phone is assumed as the terminal device 1. The device 5 is a home appliance used in the home such as an air conditioner or a refrigerator, or an information device that handles information such as a personal computer or a recorder device. The device 5 is communicated (wireless communication or, in some cases, wired communication). It is an information home appliance that can be operated remotely. The device information database 7 is a database device that stores information on the device 5. The device information database 7 is a database that communicates with the terminal device 1 wirelessly or by wire, or a type that is built in the terminal device 1. In short, any database may be used as long as it stores device information 210 to be described later, which is information related to the device 5, and has a mechanism that the terminal device 1 can freely use. Reference numeral 6 denotes a living body, and here, the finger of an operator who operates the terminal device 1 is assumed.

  In the terminal device 1, the imaging control unit 100, the image comparison unit 101, the device feature information holding unit 102, the device selection unit 103, the biological determination unit 104, the user interface generation unit 105, the script acquisition unit 106, the communication unit 107, an image Separation means 108, display control means 109, control information generation means 110, control means 111, input reception / interpretation means 112, coordinate information comparison means 113, positional relationship detection means 114, and the like are provided. Most of these means are realized by hardware resources such as CPU, ROM, RAM, and various I / Fs built in the terminal device 1 and various programs prepared in advance. ing.

  The imaging control unit 100 controls the operation of the camera 4, and appropriately processes and temporarily stores an image of the subject (here, the device 5 and the living body 6) captured by the camera 4 as necessary. The captured image is handed to the image comparison unit 101, the living body determination unit 104, and the positional relationship detection unit 114. The device feature information holding unit 102 holds device feature information 230, which will be described later, representing the features of the device. The image comparison unit 101 compares the image captured by the camera 4 with the device feature information 230 held by the device feature information holding unit 102 and determines whether the device 5 is the operation target. The device selection unit 103 refers to the output from the image comparison unit 101 and the device feature information 230 held by the device feature information holding unit 102 when the image comparison unit 101 determines that there is a device 5 to be operated. Then, the device 5 to be operated is identified, and the identified device 5 is selected as a communication partner by the communication unit 107 or selected as a remote control operation target by the terminal device 1. In addition, when the communication unit 107 communicates with the wireless tag (wireless IC tag) included in the device 5 and acquires the wireless tag identifier, the device selection unit 103 identifies the device 5 from the wireless tag identifier. This identified device 5 has a function of selecting it as a communication partner or selecting it as an object of remote control operation by the terminal device 1.

  The living body determination unit 104 determines whether or not the portion of the living body 6 (here, the finger of the operator who operates the terminal device 1) is captured in the image captured by the camera 4, that is, the camera 4 of the terminal device 1. It is determined whether or not the operator's fingers are held over (in the field of view of the camera 4). In this embodiment, the camera 4 captures an image by visible light, and the determination processing by the living body determination unit 104 includes appropriate image processing such as skin color extraction, contour extraction, feature extraction, and determination processing. Done by combining. Note that the camera 4 may have a function of performing image capturing with visible light and image capturing with infrared light. In this case, a human image is captured from an image captured with infrared light. What is necessary is just to judge whether the part of the biological body 6 is image | photographed in the image image | photographed with the camera 4 with the presence location of body temperature. The image separation means 108, when an image taken by the camera 4 includes a living body part, from an image taken by the camera 4 (image taken by visible light imaging), an image of the living body 6 (living body). The image is separated into a partial image) and an image of the other portion (non-biological partial image). The image separation means 108 identifies the finger partial area of the living body 6 by appropriate image processing such as skin color extraction, contour extraction, feature extraction, and the like, and extracts the identified partial area of the finger of the living body 6. Separated into a partial image and a non-biological partial image. The positional relationship detection unit 114 detects the positional relationship between the fingertip of the operator of the terminal device 1 and the terminal device 1 using the image taken by the camera 4 and the latest living body partial image. That is, the positional relationship detection means 114 detects the coordinate position of the fingertip in the shooting angle of view of the camera 4.

  The communication unit 107 communicates with the selected device 5, receives device information 210 to be described later from the device 5, and transmits control information to the device 5. The communication means 107 also communicates with the device information database 7 and exchanges device information 210 to be described later. When the device information database 7 is, for example, an IC memory or HDD connected by USB, the communication unit 107 communicates (information exchange) with the device information database 7 through a USB standard interface. The script acquisition unit 106 acquires a script 300 to be described later from the device information 210 to be described later acquired from the device 5 and the device information database 7, and holds this. The device feature information holding unit 102 acquires device feature information 230 to be described later from the acquired device information 210 to be described later, and holds this.

  The user interface generation unit 105 interprets a later-described script 300 handed over from the script acquisition unit 106 and generates a user interface (user interface image). The display control means 109 performs processing for generating various image data to be displayed on the display device 2, and sequentially displays the non-biological partial image, the user interface, and the biological partial image on the display device 2 from the bottom. It has a function of displaying on the display device 2 or displaying an image on which a living body partial image is superimposed in order from the bottom.

  When the operator's finger touches the touch panel device 3, the input reception / interpretation unit 112 acquires the touched coordinate position and displays the user interface displayed on the display device 2 below the transparent touch panel device 3. When it is determined that the icon inside is touched, the operation information by the user corresponding to the icon is interpreted, and the operation information is handed to the control information generation unit 110. The coordinate information comparison unit 113 refers to the coordinate information of the fingertip from the positional relationship detection unit 114 and the coordinate information of the display fingertip from the display control unit 109, and displays the superimposed information on the user interface displayed on the display device 2. It is determined whether or not operation information has been input by the operator according to the fingertip movement / position, that is, whether or not the operator has performed an operation information input operation without directly touching the input device 3. (Determining whether or not the icon in the displayed user interface is virtually touched), and if it is determined that the icon is virtually touched, the operation information by the user corresponding to the icon is interpreted. Then, the operation information is handed over to the control information generation means 110.

  The control information generation unit 110 generates control information for controlling the device 5 from the input operation information and a script 300 described later of the script acquisition unit 106. The generated control information is transferred to the communication unit 107 through the control unit 111, and is transmitted from the communication unit 107 to the device 5 under the control of the control unit 111. Note that the control unit 111 performs overall control of the entire terminal device 1, and each unit operates under the control of the control unit 111.

  Next, the device information management table 200 will be described with reference to FIG. The device information management table 200 is a data storage table that is held by the device information database 7 and includes one or more pieces of device information 210. The device information 210 includes a device identifier 220 for uniquely identifying the device information 210, a script 300 that describes the user interface and control information of the device, and model feature information 230 that describes the features of the device.

  The device identifier 220 is an identifier having a unique value for each device, and one device identifier 220 is, for example, an IP address in a TCP / IP (Transmission Control Protocol / Internet Protocol) protocol or a MAC (Media Access). Control) A value that designates a specific device such as an address. The communication unit 107 can communicate with the specific device using the value indicated by the device identifier 220.

  The script 300 is text-based information describing the user interface and control information of the device, and the configuration of the script 300 will be described later. By using this, the terminal device 1 can construct a user interface for controlling the device 5 and generate control information for controlling the device 5.

  The device characteristic information 230 includes a wireless tag identification ID 240 that is an identifier of the wireless tag of the device 5, a front image 231 that is an image of the front panel of the device 5, and a side image that is an image of the side of the device 5. 232, a model name 233 that holds the name of the device 5 in a text base, a switch placement position 234 that indicates where the switch of the device 5 is placed, and a display device position 235 that indicates the mounting position of the display device of the device 5 And a standby display image (initial display image) 236 that is an image displayed on the display unit of the device 5 when the device 5 enters the standby state.

  Next, the script 300 will be described with reference to FIG. The script 300 is text-based information describing the user interface and control information of the device. The script 300 indicates how to construct the user interface of the device 5 by interpreting the configuration, or which device 5 Information indicating how to control, for example, HTML (Hypertext Marcup Language), XML (Extensive Marcup Language), RSS (RDF Site Summary or Rich Site Summary or Really Simple Syndication), GML (Standard Generalized Marcup Language) ) In a text-based markup language. In FIG. 3, an example of the script 300 will be described using XML.

  The script 300 includes a user interface 310, a layer 1 (320), a function 1 (330), and a function 2 (331). The script 300 may include one or more of the user interface 310, layer 1 (320), function 1 (330), and function 2 (331).

  The user interface 310 includes a plurality of layers (311 and 312). The layer 311 is described as <layer 1 event = standby />. When the event “wait” occurs, the user interface described in “layer 1”, that is, the layer 1 (320) is executed. means. The layer 312 is described as <layer 2 event = activated state>, which means that when the event “activated state” occurs, the user interface described in “layer 2” is executed. The event “standby” indicates, for example, a stop state before the user presses the power button even when power is supplied to the device 5, and the event “activated state” indicates that the user presses the power button, for example. This means a state immediately after the device 5 is started up. That is, it is possible to acquire the state of the device 5 from the device 5 using the communication unit 107 and change the user interface of the terminal device 1 accordingly.

  The layer 1 (320) includes an image display (321), an image display (322), and a function 2 (323). The image display 321 is described as <image display x = 10 y = 20 res = background.jpg />. This is because when layer 1 (320) is executed, an image designated as res, that is, an image file “background.jpg” is acquired here, and coordinates (x, y) = (10, The operation of “image display” at the position 20) is performed. “Image display” is a control character string uniquely provided by the user interface of the terminal device 1 and may be any character string for displaying an image or an image on the display device 2.

  The image display 322 is described as <image display x = 30 y = 40 res = switch.jpg target = function 1 />. This is because when layer 1 (320) is executed, an image designated as res, that is, an image file “switch.jpg” is acquired here, and coordinates (x, y) = (30, The operation of “image display” is performed at the position 40). In addition, “function 1” is assigned as a hyperlink to this image. This means that when the user selects this “switch.jpg” using the input device, a script corresponding to “function 1” is executed.

  Function 1 (330) is described as <function 1 device = device 1 command = control command 1 status = activated state />. This is information describing how to execute the script “Function 1” described above. After sending the character string “Control Command 1” to “Device 1”, the status of the device 5 is changed to “ This means “starting state”. Here, “control command 1” refers to the control information described above, and is control information transmitted to “device 1”, that is, device 5. Function 2 (331) is described as <function 2 device = device 1 command = control command 2 status = standby state />. This means that after the character string “control command 2” is sent to “device 1”, the state of device 5 is set to “standby state”. Here, “control command 2” refers to the control information described above, and is control information transmitted to “device 1”, that is, device 5.

  Using the script 300 as described above, a user interface can be constructed and a method for sending a control command to a device can be described.

  Next, in this embodiment, an operation of selecting the device 5 photographed by the terminal device 1 as a target for remote control operation, generating a user interface suitable for the device 5 and operating the device 5 by remote control is shown in FIG. This will be described with reference to a flowchart.

  First, in order to determine whether or not a wireless tag (wireless IC tag) is attached to the device 5 using the communication means 107, a wireless tag identifier is acquired (step 1001), and the device 5 has a wireless tag. If so (if yes in step 1002), the communication means 107 opens the communication line with the device 5 and starts communication (step 1100). Next, the device information 210 is acquired from the device 5 (step 1101), the communication line is disconnected, and the communication is terminated (step 1102). Next, an image is acquired using the camera 4 of the terminal device 1 (step 1103), and step 1006 is executed. Note that the image acquired in step 1103 is an image including the device 5 and the finger of the operator of the terminal device 1 here.

  If it is determined in step 1002 that the device 5 does not have a wireless tag (if there is no wireless tag in step 1002), the communication unit 107 is used to communicate with the device information database 7, and the device information management table 200 All device information 210 is acquired. Moreover, an image is acquired using the camera 4 of the terminal device 1 (step 1004), and the device determination process of step 2000 is performed. Details of the device determination processing in step 2000 will be described later. Note that the image acquired in step 1104 is also an image including the device 5 and the finger of the operator of the terminal device 1 here.

  If the device identifier 220 of the device 5 cannot be found by referring to the model feature information 230 in the device determination processing of step 2000 (in the case of No determination in step 1005), the processing returns to step 1000 (or this flow). Exit). If the device identifier 220 of the device 5 can be found by referring to the model characteristic information 230 (in the case of Yes determination in step 1005), the script 300 is acquired from the device information 210 (step 1006). In addition, when the process proceeds from step 1103 to step 1006, the script 300 is acquired from the device information 210. Next, the script 300 is interpreted as described above, and a user interface (user interface image) is generated (step 1007). The interpretation of the script 300 basically constructs a user interface by reading a character string from the beginning of the script 300 and interpreting it according to the conventions of hypertext documents such as HTML, XML, RSS, GML.

  Next, the living body partial image and the non-living body partial image are separated from the image captured using the camera 4 in step 1103 or step 1004 (step 1008). Next, the non-biological partial image is displayed on the display device 2 (step 1009), and then a user interface (user interface image) is displayed on the non-biological partial image (step 1010). The living body partial image is displayed so as to overlap (step S1011). FIG. 7B shows a display example on the display device 2 after the processing in step S1011. Then, the input process and device control process in step 3000 are performed, and the process returns to step 1000. Details of the input processing and device control processing in step 3000 will be described later.

  Next, the device determination processing in step 2000 of FIG. 4 will be described using the flowchart shown in FIG.

  First, one piece of device information 210 is extracted from the device information 210 acquired from the device information management table 200 of the device information database 7 to the work area (step 2001), and the extraction to the last device information 210 has been completed. If there is no longer any device information 210 to be extracted (in the case of Yes determination in step 2002), the fact that the device 5 is not the operation target is output as the device determination process in FIG. 4 (step 2012), and this flow Is terminated (step 2014).

  If the device information 210 can be extracted in step 2002 (if No in step 2002), the device feature information 230 is extracted from the device information 210 (step 2003). Next, the image (image taken by the camera 4) acquired in step 1004 in FIG. 4 is taken out to the work area (step 2004), and the front image 231, the side image 232, the model name 233, and the switch arrangement position of the model characteristic information 230 234, the display device position 235, and the standby display image 236 are compared with the image photographed by the camera 4 (step 2004). In this comparison, the image taken by the camera 4 is expanded / contracted and the image is compared with the image in the model feature information 230, or the feature points of the image are acquired and the feature points match. Any procedure may be used as long as image comparison is performed, such as whether or not.

  In the image comparison process performed in step 2005, when it is determined that the front images 231 match (in the case of Yes determination in step 2006), the process proceeds to step 2013, and the device identifier 220 of the device information 210 is changed to FIG. The device determination process is output (step 2013), and this flow ends (step 2014). If it is determined in the image comparison process performed in step 2005 that the side images 232 match (in the case of Yes determination in step 2007), the process proceeds to step 2013, and the device identifier 220 of the device information 210 is set as follows. It is set as the output of the apparatus determination process of FIG. 4 (step 2013), and this flow is complete | finished (2014). If it is determined in the image comparison process performed in step 2005 that the model names 233 match (in the case of Yes determination in step 2008), the process proceeds to step 2013, and the device identifier 220 of the device information 210 is set as follows. It is set as the output of the apparatus determination process of FIG. 4 (step 2013), and this flow is complete | finished (step 2014). If it is determined in the image comparison process performed in step 2005 that the switch arrangement positions 234 match (if YES in step 2009), the process proceeds to step 2013, and the device identifier 220 of the device information 210 is set. Then, the output of the device determination process of FIG. 4 is set (step 2013), and this flow is ended (step 2014). If it is determined in the image comparison process performed in step 2005 that the display device positions 235 match (in the case of Yes determination in step 2010), the process proceeds to step 2013, and the device identifier 220 of the device information 210 is set. Then, the output of the device determination process of FIG. 4 is set (step 2013), and this flow is ended (step 2014). If it is determined in the image comparison process performed in step 2005 that the standby display images 236 match (if YES in step 2011), the process proceeds to step 2013, and the device identifier 220 of the device information 210 is set. Then, the output of the device determination process of FIG. 4 is set (step 2013), and this flow is ended (step 2014). If the standby display images do not match in step 2011 (in the case of No determination in step 2011), it is determined that the device information 210 acquired in step 2001 is not applicable, and the next device information 210 is acquired. Therefore, the process proceeds to step 2001 and the process is continued.

  Next, the operator directly operates the touch panel device 3 to input operation information, or obtains the operation information input by the operator from an image captured by the camera 4, thereby obtaining control information for the device 5. The input and device control process of step 3000 in FIG. 4, which is a process of generating and sending this to the device 5, will be described using the flowchart shown in FIG. 6.

  First, in step 1007 of FIG. 4, it is determined whether or not a user interface has been generated (step 3001). If it is determined that there is no user interface (in the case of No determination in step 3001), the operator's Since there is no need to accept an input, this flow ends (step 3015). If it is determined that there is a user interface (in the case of Yes determination in step 3001), a living body partial image separated from the image captured by the camera 4 in step 1008 in FIG. 3002), it is determined whether or not the operator has made an input by comparison with the previously captured biological partial images already developed in the work area (step 3003). This is because, for example, when the finger (displayed finger) of the living body partial image touches an icon of the user interface, the movement of the finger portion stops at that time, and thus the living body portion extracted from the previously captured image By comparing with the image, it is possible to determine whether or not the operator has input. If it is determined that there is no input (in the case of No determination in step 3003), the process shifts to input acceptance by the touch panel device 3 (step 3004), and whether or not an operator performs an input operation using the touch panel device 3 Is determined (step 3005). If there is no input, that is, if the operator is not touching the touch panel device 3, it is determined that there is no input from the operator (No determination is made in step 3005), and this flow is finished (step 3015). If there is an input, that is, if the operator is touching the touch panel device 3, it is determined that there is an input from the operator (Yes is determined in step 3005), and the processing in step 3007 is executed.

  If it is determined in step 3003 that there is an input from the operator (Yes in step 3003), it is determined where the fingertip is from the feature information of the biological partial image (step 3003). 3006), coordinate information of the fingertip is acquired (step 3007). If the input is from the touch panel device 2 (Yes in step 3005), the coordinate information of the touched input position is acquired (step 3007).

  Next, referring to the script 300, the hypertext is interpreted (step 3008). In the example of the script 300 in FIG. 3, for example, in the hypertext, the icon image and the display start position are described as in the image display 322 of layer 1. Together, a specific range can be assigned to function 1. It is determined whether the X and Y coordinates acquired in step 3007 are within the range (step 3009). This determination is performed for all icons described in the script 300. If it is determined that the icon is not within the range (in the case of No determination in step 3009), the icon is not pressed. This flow is finished (step 3015). If it is determined that it is within the range (in the case of Yes determination in step 3009), the function to be processed is acquired from the script 300 (step 3010). In the example of FIG. 3, since “function 1” corresponding to the image display 322 is described in the function 1 (330), device and command information is acquired from the function 1 (3301). Next, using the communication means 107 of the terminal device 1, communication is started with the device 5 specified in the device (step 3012), and the control command (control information) described in the command is sent to the device 5. Transmission is performed using the communication line (step 3013), and then the communication is stopped (step 3014). Then, this flow is terminated (3015), and transmission of all control commands is terminated.

  With the above operation, it is possible to select the device 5 photographed on the terminal device 1 as a target for remote control operation, generate a user interface suitable for the device 5, and realize the operation of operating the device 5 by remote control. .

  FIG. 7A shows a state in which a living body (operator's fingertip) 6 is photographed by the camera 4 of the terminal device 1 and a remote control operation is performed using the photographed fingertip. Represents. FIG. 7B shows an example of a display image on the display device 2 that is viewed by the operator's eyes 701 in FIG. In FIG. 7B, 702 is a non-biological partial image displayed on the lowermost side of the display screen, 703 is a user interface (user interface image) superimposed and displayed on the upper side of the non-biological partial image 702, Reference numeral 704 denotes a living body partial image superimposed on the user interface 703.

  As shown in FIGS. 7A and 7B, the user interface 703 displayed on the display device 2 is virtually operated by a fingertip photographed by the camera 4 without touching the touch panel device 3. Thus, when the remote control operation is performed, even if the terminal device 1 is a small device such as a mobile phone and the display device 2 is small in size, a living body partial image having a size smaller than the actual fingertip ( With the display fingertip image), it is possible to accurately operate fine icons in the user interface 703. In addition, many icons can be arranged on the user interface 703.

  In the above-described example, the non-biological partial image 702, the user interface 703, and the biological partial image 704 are displayed in a superimposed manner, and the operator recognizes the image of the device 5 being operated by the non-biological partial image 702. Although it is possible to display the non-biological partial image 702 when performing a remote control operation, the biological partial image 704 may be simply superimposed on the user interface 703.

It is a block which shows the structure of the terminal device which concerns on one Embodiment of this invention, and the apparatus control system comprised including it. It is explanatory drawing which shows the content of the equipment information management table which the equipment information database which the terminal device concerning one Embodiment of this invention uses hold | maintains. It is explanatory drawing which shows the example of the script of the apparatus information in the apparatus information management table of FIG. It is a flowchart which shows the processing flow of the terminal device which concerns on one Embodiment of this invention. It is a flowchart which shows the detail of the apparatus judgment process in FIG. 5 is a flowchart showing details of input and device control processing in FIG. 4. It is explanatory drawing which shows the example of remote control operation using the terminal device which concerns on one Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Terminal device 2 Display apparatus 3 Touch panel apparatus 4 Camera 5 Equipment 6 Living body 7 Equipment information database 100 Imaging control means 101 Image comparison means 102 Equipment characteristic information holding means 103 Equipment selection means 104 Living body judgment means 105 User interface generation means 106 Script acquisition means DESCRIPTION OF SYMBOLS 107 Communication means 108 Image separation means 109 Display control means 110 Control information generation means 111 Control means 112 Input reception / interpretation means 113 Coordinate information comparison means 114 Position relation detection means 200 Equipment information management table 210 Equipment information 220 Equipment identifier 230 Equipment characteristic information 231 Front image 232 Side image 233 Model name 234 Switch placement position 235 Display device position 236 Standby display image 240 Wireless tag identification ID
300 Script 310 User Interface 320 Layer 1
330 Function 1
331 Function 2

Claims (12)

A terminal device having a function of communicating with a device to be operated,
Imaging means for taking an image;
Device feature information holding means for holding device feature information expressing the features of the device;
Image comparison means for determining whether or not the device characteristic information is included in the captured image;
A device selection means for selecting a device to be operated;
Script acquisition means for acquiring a script describing the operation of the device from a database or device;
User interface generation means for generating a user interface (user interface image) according to the script;
Display control means for displaying the user interface on display means;
Input means for an operator to input operation information using the user interface;
Control information generating means for converting the operation information according to the script and generating control information;
Communication means for communicating with the device to be operated and transmitting the control information to the device;
A terminal device comprising the terminal device. The terminal device according to claim 1,
The script acquisition unit has a function of acquiring a script included in device information received from the device by the communication unit, and the device feature information holding unit has a function of acquiring device feature information included in the device information. A terminal device characterized by that. The terminal device according to claim 1,
A terminal device comprising a function of identifying an operation target device based on identification information acquired from a wireless tag (wireless IC tag) provided in the device. The terminal device according to claim 1,
A terminal device comprising a function of identifying a device to be operated with reference to the database from an image of a device photographed by the imaging means. The terminal device according to any one of claims 1 to 4,
The terminal device, wherein the display unit and the input unit are configured by a touch display panel which is an input device that also serves as a display. The terminal device according to claim 5,
Analyzing an image captured by the imaging means and determining whether or not a living body is included in the image, and a living body partial image from the captured image when a living body is included in the captured image Image separation means for separating the non-biological partial image,
A terminal device comprising a function of superposing and displaying at least the user interface and the living body partial image in order from the bottom on the display means. The terminal device according to claim 6,
A terminal device comprising a function of superimposing and displaying the non-biological partial image, the user interface, and the biological partial image on the display unit in order from the bottom. The terminal device according to claim 6 or 7,
A positional relationship detection unit that detects a positional relationship between the fingertip of the operator of the terminal device, which is a living body included in the image, and the terminal device from an image captured by the imaging unit;
Means for determining whether or not the operation information has been input according to the movement and position of the fingertip of the living body in the living body partial image superimposed and displayed on the user interface is provided so that the operator of the terminal device A terminal device characterized in that the operation information can be input without directly touching the input device. The terminal device according to claim 6,
The terminal device has a function of performing image capturing using visible light and image capturing using infrared light. It has a communication means having a function of communicating with a device to be operated, an imaging means for taking an image, a display means, and an input means. The display means and the input means are touch displays which are input devices that also serve as a display. A terminal device composed of panels,
The script describing the operation of the device acquired from the device using the communication means, the device identification information acquired from the device using the communication means, or the device identification specified from the image of the device taken by the imaging means A user interface (user interface image) for operating the device is automatically generated according to a script describing the operation of the device acquired from the database based on the information, and the generated user interface is displayed on the display means. A terminal device comprising means. It has a communication means having a function of communicating with a device to be operated, an imaging means for taking an image, a display means, and an input means. The display means and the input means are touch displays which are input devices that also serve as a display. A terminal device composed of panels,
Script acquisition means for acquiring a script describing the operation of the device to be operated from a database or device;
User interface generation means for generating a user interface (user interface image) according to the script;
Image separating means for separating a biological partial image and a non-biological partial image from an image photographed by the imaging means;
A positional relationship detecting unit that detects a positional relationship between the fingertip of the operator of the terminal device, which is a living body included in the image, and the terminal device, from an image captured by the imaging unit;
Display control means for displaying the living body partial image superimposed on the user interface on the display means,
The operator of the terminal device determines whether operation information has been input by the operator according to the movement / position of the fingertip of the living body in the living body partial image superimposed and displayed on the user interface. A terminal device characterized in that the operation information can be input without directly touching the input device.   12. A device control system comprising: the terminal device according to claim 1; and a device having one or more communication functions to be operated by the terminal device.

Images