JP2014036382A - Remote controller and remote control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a remote controller and a remote control method capable of controlling a controlled apparatus even if an image pick-up section is not directed toward the controlled apparatus, while achieving compaction and simplification of the device configuration.SOLUTION: A remote controller acquires apparatus identification information and a control command, previously stores in a storage section the first captured image of a controlled apparatus captured in an image pick-up section in association with the apparatus identification information and a control command, recognizes the controlled apparatus by extracting an image matching with a second captured image among the first captured images, generates the operation screen of input operation information corresponding to a control command of the controlled apparatus, displays the operation screen on a display, receives an input operation for the input operation information from an input section, and transmits a control command corresponding to the input operation information of input operation to the controlled apparatus.

Description

  The present invention relates to a remote control device and a remote control method for controlling a control target device from a position away from the control target device such as a household electrical appliance.

  As a technology of a remote control device that controls a control target device from a position away from the control target device, the control target device performs an operation on the control target device based on a photographed image of the control target device captured by the photographing unit. Techniques are conventionally known (see, for example, Patent Documents 1 and 2).

  Specifically, Patent Document 1 describes the position of one or more objects (control target devices) existing in the real space, and is obtained by photographing the real space using the imaging device (imaging unit). Can be operated based on object identification data for identifying objects that can be operated by the user from among the objects included in the environment map generated based on the input image and the feature data representing the appearance characteristics of each object There has been proposed a technique for selecting at least one object recognized as being a candidate object that is a candidate for an operation target by a user using a selection unit. In Patent Document 1, for example, in an output image generated from an input image obtained by photographing an environment, a frame surrounding the object is attached to an operable object, and a control command (specifically, an operation of the object is performed). A configuration is described in which a command column for displaying “Power ON”) is attached so that the user can easily select a control command to be transmitted to the operation target.

  Further, in Patent Document 2, an image input by an image input unit (photographing unit, specifically a network camera) is displayed on an image display unit (display unit, specifically a digital TV). A technique for performing an operation related to an image display function with an operation means (a remote control device, specifically, a remote controller) has been proposed. In Patent Document 2, for example, when the image input means receives that each button of the operation means has been pressed by the user through the image display means, the display screen displayed on the image display means by the region selection means A configuration is described in which the range of the area to be selected is enlarged or reduced, or the area to be selected is changed.

JP 2011-146996 A JP 2008-129946 A

  However, in the prior art described in Patent Document 1, since the control command is displayed on the shooting screen shot by the imaging device (shooting unit), when the object (control target device) is desired to be controlled, the imaging device is always attached to the object. It is necessary to keep it in the direction.

  Further, in the prior art described in Patent Document 2, the image input unit (imaging unit), the image display unit (display unit) for displaying an image input by the image input unit, and the image display function of the image display unit are related. Since the operation means (remote control device) for performing the operation is configured separately, the device configuration becomes large and complicated.

  Therefore, the present invention can control the control target device without setting the photographing unit in the direction of the control target device, and can realize a reduction in size and simplification of the apparatus configuration. An object is to provide a control device and a remote control method.

  In order to solve the above problems, the present invention provides the following remote control device and remote control method.

(1) Remote control device A remote control device that controls the device to be controlled from a position away from the device to be controlled, the photographing unit that photographs the device to be controlled, and the control that is photographed by the photographing unit A first captured image of a target device, device identification information for identifying the device to be controlled, a storage unit for storing a control command for controlling the device to be controlled, and corresponding to the device identification information and the control command A display unit that displays an operation screen of input operation information, an input unit that receives an input operation for the input operation information on the operation screen, a transmission unit that transmits the control command to the control target device, the device identification information, and An acquisition control means for acquiring the control command, and the first captured image of the control target device captured by the capturing unit are displayed on the device identification corresponding to the control target device. Registration information that is registered in advance in the storage unit in association with other information and the control command, and a matching image that matches the second captured image of the control target device captured by the imaging unit are stored in the storage unit. Recognition control means for recognizing the control target device by extracting from the registered first captured image, and the control command corresponding to the device identification information of the control target device recognized from the second captured image Generating control means for generating the operation screen of the input operation information corresponding to the control command by reading from the storage section, display control means for displaying the operation screen on the display section, and the operation from the input section An acceptance control means for accepting the input operation for the input operation information on the screen, and the input of the accepted input operation when the input operation is accepted from the input unit Remote control device, characterized in that the control command corresponding to the work information from the transmitting unit and a transmission control means for transmitting to the control target device.

(2) Remote control method A remote control method of a remote control device for controlling the control target device from a position distant from the control target device, the device identification information for identifying the control target device, and the control target A step of acquiring a control command for controlling the device, and a first captured image of the control target device captured by a capturing unit provided in the remote control device, the device identification information corresponding to the control target device And a step of registering in advance in a storage unit provided in the remote control device in association with the control command, and a coincidence image matching a second photographed image of the control target device photographed by the photographing unit, Extracting from the first captured image registered in the storage unit and recognizing the control target device; and identifying the device of the control target device recognized from the second captured image Reading the control command corresponding to information from the storage unit and generating an operation screen of input operation information corresponding to the control command; and displaying the operation screen on a display unit provided in the remote control device Receiving an input operation for the input operation information on the operation screen from an input unit provided in the remote control device, and the input of the input operation received when the input operation is received from the input unit And a step of transmitting the control command corresponding to the operation information from the transmission unit provided in the remote control device to the control target device.

  In the present invention, prior to controlling the control target device, first, the device identification information for identifying the control target device and the control command for controlling the control target device are acquired. Next, the first photographed image of the control target device photographed by the photographing unit is registered in advance in the storage unit in association with the device identification information and the control command corresponding to the control target device. Thus, in the state where the device identification information and the control command are registered in the storage unit in advance, the matching image that matches the second captured image of the control target device captured by the imaging unit, Extracting from the first captured image registered in the storage unit to recognize the control target device, and reading out the control command corresponding to the device identification information of the recognized control target device from the storage unit The operation screen of the input operation information corresponding to the control command is generated, and the generated operation screen is displayed on the display unit. In this way, when controlling the control target device, the photographing unit does not have to be in a state of being directed toward the control target device. That is, after the control target device is recognized, it is not always necessary to keep the photographing unit in the direction of the control target device. When the input operation for the input operation information on the operation screen is received from the input unit, the control command corresponding to the input operation information of the received input operation is transmitted from the transmission unit to the control target device. Send. This makes it possible to control the device to be controlled.

  In addition, since the photographing unit and the display unit are provided in the remote control device, the device configuration can be reduced in size and simplified.

  In the present invention, the registration control means associates the plurality of first photographed images photographed a plurality of times by the photographing unit with the device identification information and the control command as the same control target device in the storage unit in advance. A mode to be registered can be exemplified.

  In this specific matter, a plurality of the first captured images taken from different angles of the device to be controlled can be registered in advance in the storage unit as the same device to be controlled, whereby the second It is possible to improve the recognition accuracy of the device to be controlled based on the captured image.

  In the present invention, the recognition control means determines the recognition of the control target device when the control target device is put on hold even when the control target device is recognized and continues to recognize the control target device for a predetermined time. Can be illustrated.

  In this specific matter, even if the control target device is recognized, the device is put on hold, and when the control target device is continuously recognized for the predetermined time, the recognition of the control target device is confirmed. Even if the control target device is reflected, it is possible to reduce the probability that another control target device not desired by the user is mistakenly recognized as the control target device desired by the user.

  In this invention, the aspect further provided with the 1st alerting | reporting control means which alert | reports that the recognition of the said control object apparatus was put on hold can be illustrated.

  In this specific matter, when the user is photographing an article by notifying that the recognition of the device to be controlled has been put on hold by the first notification control unit, the control target is determined by the second captured image. When the device is recognized, the user can understand that the photographed article is recognized as the control target device.

  In the present invention, it is possible to exemplify an aspect further including second notification control means for notifying that the recognition of the device to be controlled is confirmed.

  In this specific matter, when the user is photographing an article by notifying that the recognition of the control target device is confirmed by the second notification control unit, the control target device is identified by the second captured image. When confirmed, it is possible to make the user understand that the photographed article is confirmed to be the device to be controlled.

  In the present invention, the acquisition control unit acquires the device identification information and the control command corresponding to a plurality of control target devices, and the registration control unit includes a plurality of the control target devices photographed by the photographing unit. The first captured image is registered in advance in the storage unit in association with the device identification information and the control command corresponding to the device to be controlled, and the recognition control means includes a plurality of images captured by the imaging unit. A plurality of matching images that respectively match the second captured images of the control target device are extracted from the first captured images registered in the storage unit, and a plurality of the control target devices are recognized. The generation control means outputs the plurality of control commands respectively corresponding to the device identification information of the plurality of control target devices recognized from the plurality of second captured images. A plurality of the operation screens of the input operation information corresponding to a plurality of the control commands read out from the storage unit are generated, and the display control unit displays the plurality of operation screens on the display unit, and the reception control A means can illustrate the aspect which receives the said input operation with respect to the said input operation information of the said some operation screen from the said input part.

  In this specific matter, by recognizing a plurality of devices to be controlled, the input operation for the input operation information of the plurality of operation screens displayed on one display unit can be received from the input unit, and accordingly The operation for controlling the plurality of devices to be controlled can be performed on the input operation information of the plurality of operation screens displayed on one display unit, and the convenience can be improved accordingly.

  In the present invention, the information processing apparatus further includes a receiving unit that receives the device identification information and the control command via a network from a server that stores the control command in association with the device identification information, and the acquisition control unit stores the control command in the server For example, the device identification information and the control command may be acquired by receiving the device identification information and the control command from the server via the network by the receiving unit.

  In this specific matter, since the device further includes the receiving unit that receives the device identification information and the control command from the server via the network, even if the model of the device to be controlled is added or the device identification information and / or Alternatively, even when the control command is changed, the device identification information and / or the control command corresponding to the control target device can be acquired from the server, and accordingly, various types of models of the control target device can be obtained. It is possible to easily cope with addition or change of the device identification information and / or the control command.

  As described above, according to the present invention, the device to be controlled can be controlled without setting the photographing unit in the state of the device to be controlled, and the apparatus configuration can be reduced in size and simplified. Can be realized.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a system configuration diagram schematically showing a remote control system that controls a device to be controlled by a portable terminal device that includes a remote control device that performs a remote control method according to an embodiment of the present invention. It is an external view which shows the external appearance of the portable terminal device shown in FIG. 1, (a) is a schematic front view which shows the front side of a portable terminal device, (b) is a schematic rear view which shows the back side of a portable terminal device. It is. It is a schematic block diagram which shows the internal structure of the remote control apparatus in the portable terminal device shown in FIG.1 and FIG.2. It is a schematic diagram which shows the data structure of the apparatus identification information in an external recording medium and a server, a control command, and input operation information. It is a schematic diagram which shows the data structure of the control object apparatus information database in the memory | storage part of a remote control apparatus. It is a schematic front view which shows the state in which the mode selection screen which selects and operates any one of apparatus information registration mode and remote control mode is displayed on the display screen of a display part in a remote control apparatus. It is a flowchart of the first half part which shows the flow of a process of the example of 1 control when apparatus information registration mode is selected. It is a flowchart of the latter half part which shows the flow of a process of the example of control when apparatus information registration mode is selected. It is a schematic front view which shows the state by which the reception screen which receives input operation of the model information of a control object apparatus is displayed on the display screen of a display part in apparatus information registration mode. It is a schematic front view which shows the state by which the imaging | photography screen image | photographed in the imaging | photography part in the apparatus information registration mode and the confirmation key which determines whether to register a picked-up image are displayed on the display screen of a display part. It is the flowchart of the first half part which shows the flow of the process by one example of control when the remote control mode is selected. It is a flowchart of the latter half part which shows the flow of the process by one example of control when the remote control mode is selected. It is a schematic front view which shows the state in which the imaging | photography screen on which the 2nd imaging | photography image of one control object apparatus was image | photographed in remote control mode is displayed on the display screen of a display part, (a) And (b) FIG. 5 is a diagram illustrating a state in which recognition of an indoor unit acting as a control target device is suspended and a confirmed state. It is a schematic front view which shows the state in which the imaging | photography screen on which the 2nd imaging | photography image of one control object apparatus was image | photographed in remote control mode is displayed on the display screen of a display part, (a) And (b) FIG. 6 is a diagram illustrating a state where recognition of a television acting as a control target device is suspended and a state where the recognition is confirmed. It is a schematic front view which shows the operation screen which displayed the input operation information corresponding to the apparatus identification information and control command of an indoor unit in remote control mode. It is a schematic front view which shows the operation screen which displayed the input operation information corresponding to the apparatus identification information and control command of a television in remote control mode. It is a flowchart of the first half part which shows the flow of the process by the other control example in case remote control mode is selected. It is a flowchart of the latter half part which shows the flow of the process by the other control example in case remote control mode is selected. It is a schematic front view which shows the state by which the imaging | photography screen on which the 2nd imaging | photography image of the some control object apparatus was image | photographed in remote control mode is displayed on the display screen of a display part, (a) is as control object apparatus It is a figure which shows the state which has hold | maintained recognition of the indoor unit and television which act, (b) is a figure which shows the state which confirmed the recognition of the outdoor unit and television which act as a control object apparatus. It is a schematic front view which shows the operation screen which displayed the input operation information corresponding to the apparatus identification information and control command of an indoor unit and a television in remote control mode. FIG. 21 is a schematic front view illustrating a modification of the operation screen illustrated in FIG. 20, in which (a) illustrates input operation information switched to a higher priority, and (b) illustrates priority It is a figure which shows the input operation information switched to the lower one. It is a schematic front view which shows the other modification of the operation screen shown in FIG. 20, (a) is a figure which shows the input operation information switched to the indoor unit, (b) is a television It is a figure which shows the input operation information switched to the direction.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The following embodiments are examples embodying the present invention, and do not limit the technical scope of the present invention.

[Overall configuration of remote control system]
FIG. 1 is a system configuration diagram schematically showing a remote control system that controls a device 200 to be controlled by a mobile terminal device 10 including a remote control device 100 that performs a remote control method according to an embodiment of the present invention. . FIG. 2 is an external view showing the external appearance of the mobile terminal device 10 shown in FIG. FIG. 2A is a schematic front view showing the front side of the mobile terminal device 10. FIG. 2B is a schematic rear view showing the back side of the mobile terminal device 10. FIG. 3 is a schematic block diagram showing an internal configuration of the remote control device 100 in the mobile terminal device 10 shown in FIGS. 1 and 2. FIG. 4 shows device identification information D2 (1) to D2 (ma), control commands D3 (1,1) to D3 (ma, n), and input operation information D4 (1,1) in the external recording medium ME and the server 300. It is a schematic diagram which shows the data structure of -D4 (ma, n). FIG. 5 is a schematic diagram illustrating a data structure of the control target device information database DB in the storage unit 120 of the remote control device 100.

  The remote control device 100 (see FIGS. 1 to 3) is a device that controls the control target device 200 from a position remote from the control target device 200 using wireless communication means such as infrared communication (so-called remote controller). Here, it is assumed to be incorporated in the mobile terminal device 10 such as a smartphone. The control target device 200 may be any electronic device that is controlled from a remote position by a remote controller, and is not limited thereto. For example, an indoor unit of an air conditioner system, a lighting device such as a fluorescent lamp, or the like Electronic devices such as home appliances, and AV products such as audio video (AV) devices such as televisions (TVs) and video recorders.

  The remote control device 100 (see FIG. 3) includes a photographing unit 110, a storage unit 120, a display unit 130, an input unit 140, a first communication unit 150, and a control unit 160. The photographing unit 110 and the storage unit 120 are provided. The display unit 130, the input unit 140, the first communication unit 150, and the control unit 160 are provided in a state of being housed in the remote control device main body 100a (specifically, the main body case) (see FIG. 2).

  The imaging unit 110 (see FIG. 2B) includes an imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and is electrically connected to an input system of the control unit 160. . The photographing unit 110 has a photographing function (so-called digital camera mechanism) for photographing a subject with a moving image and / or a still image digital image. Thereby, the imaging unit 110 can transmit the image data of the control target device 200 that captured the control target device 200 (see FIG. 1) to the control unit 160.

  The storage unit 120 (see FIG. 5) constitutes a storage device such as a RAM (Random Access Memory) or a ROM (Read Only Memory), and is electrically connected to the data line of the control unit 160. As a result, the control unit 160 can store the data processed by the control unit 160 in the storage unit 120 and can read data recorded in the storage unit 120 from the storage unit 120. The storage unit 120 controls the control program P, the first captured image D1 of the control target device 200 captured by the capturing unit 110, the device identification information D2 for identifying the control target device 200, and the control target device 200. It includes a control target device information database DB (see FIG. 5) that stores a control command D3 (control code) for executing the command.

  Here, the control command D3 is an operation command for causing the control target device 200 to perform various operations by being transmitted to the control target device 200.

  In the control target device information database DB, the first captured image D1 is stored in association with the device identification information D2 and the control command D3 together with the device identification information D2 and the control command D3. The device identification information D2 may be the name of the control target device 200 or an ID code indicating the control target device 200. When the device identification information D2 is an ID code, the ID code and the name of the control target device 200 corresponding to the ID code can be associated with the storage unit 120 and registered in advance. Here, the device identification information D2 is the name of the control target device 200 itself. In the present embodiment, the input operation information D4 corresponding to the control command D3 is registered in advance in the control target device information database DB of the storage unit 120. The input operation information D4 may be associated with the control command D3 and registered in advance in another recording unit, and the input operation information D4 corresponding to the control command D3 may be acquired from the other recording unit. .

  The display unit 130 (see FIGS. 2A and 3) includes a display device such as a liquid crystal display device or an organic EL (Electro-Luminescence) display device, and is electrically connected to the output system of the control unit 160. ing. Thereby, the display unit 130, based on the screen data sent from the control unit 160, the shooting screen Gp shot by the shooting unit 110 (see FIGS. 10, 13, 14, and 19 described later), device A mode selection screen Gs (see FIG. 6 to be described later) for selecting and operating one of the information registration mode and the remote control mode, and an operation screen for input operation information D4 corresponding to device identification information D2 and control command D3 Gr (see FIGS. 15 and 16 described later, and FIGS. 20 to 22) can be displayed.

  The input unit 140 (see FIG. 2 and FIG. 3) receives an input operation by the user, and is electrically connected to the input system of the control unit 160. Thereby, the input unit 140 causes the mode selection screen Gs (see FIG. 6), the reception screen Gu (see FIG. 9 described later), the shooting screen Gp (see FIG. 10), and the operation screen Gr (FIGS. 15, 16, and FIG. 20 (see FIG. 22 from FIG. 22), and the operation information can be transmitted to the control unit 160. Here, the input unit 140 is a touch input unit that is provided on the display screen G (see FIG. 2 and the like) of the display unit 130 and receives a touch input operation (screen position information) on the display screen G. That is, the input unit 140 constitutes a display panel (so-called touch panel) that is provided on the display screen G of the display unit 130 and can be touch-operated with the display unit 130, and is displayed on the display screen G of the display unit 130. A display key (so-called software key) can be input (specifically, a touch input operation). The input unit 140 is configured on the display screen G of the display unit 130 so that a display key (so-called software key) displayed on the display screen G can be input (specifically, touch input operation). In addition or alternatively, an operation key (so-called hardware key) provided on the remote control device main body 100a (see FIG. 2) separately from the display screen G can be input (specifically, a push input operation). It may be configured.

  The first communication unit 150 (see FIG. 3) communicates with the control target device 200 according to a specified communication standard, and includes a transmission unit 151 that transmits a control command D3 to the control target device 200. As communication standards, for example, IrDA standard, Bluetooth (registered trademark) standard (IEEE802.15.1), wireless LAN (Wireless Local Area Network) standard (IEEE802.11), NFC (Near Field Communication) standard (ISO / IEC18092). ) And the like.

  The control unit 160 includes a processing unit 161 such as a CPU (Central Processing Unit), and the processing unit 161 stores a control program P (see FIG. 5) stored in advance in the ROM of the storage unit 120 on the RAM of the storage unit 120. Various components are controlled by loading and executing them.

  Note that the photographing unit 110, the storage unit 120, the display unit 130, the input unit 140, the first communication unit 150, and the control unit 160 have the same configuration as a conventionally known configuration, and here, more detailed description is given here. Is omitted.

  The remote control device 100 includes a device information registration mode for registering information of the control target device 200 and a remote control mode for remotely controlling the control target device 200.

  The control unit 160 displays the mode selection screen Gs (see FIG. 6) on the display screen G of the display unit 130, and selects operation information (specifically, device information registration mode selection key Ka) on the mode selection screen Gs from the input unit 140. And an input operation to the remote control mode selection key Kb).

  Specifically, the control program P executed by the control unit 160 includes a device information registration program that executes the device information registration mode and a remote control program that executes the remote control mode.

  In the following description of the device information registration program and the remote control program, “ma” indicates the number of control target devices 200 to be acquired, “m” indicates the number of control target devices 200 already registered, and “n” indicates control. The number of the command D3 and the input operation information D4, “t” is the number of times the first photographed image D1 is photographed by the digital camera function, and “k” is the second photographed on the display screen G by the digital camera function. The number of photographed images D5, “i” represents the control target device 200 that has been input, and “j” represents the control command D3 and input operation information D4 that have been input. . The number of control commands D3 and input operation information D4 represented by “n” is a number determined for each type of the control target device 200. Further, “h” in FIGS. 1 and 5 is an arbitrary value of 1 to m.

  The device information registration program includes an acquisition control step and a registration control step.

  Specifically, the control unit 160 (see FIG. 2) functions as an acquisition control unit P1 and a registration control unit P2. That is, the device information registration program causes the control unit 160 to execute steps including an acquisition control step corresponding to the acquisition control unit P1 and a registration control step corresponding to the registration control unit P2.

  The acquisition control step (acquisition control means P1) includes device identification information D2 (1) to D2 (ma) and control corresponding to a plurality of control target devices 200 (1) to 200 (ma) (ma is an integer of 2 or more). Commands D3 (1, 1) to D3 (ma, n) (n is an integer of 1 or more) (see FIG. 4) are acquired. Here, the acquisition control step (acquisition control means P1) includes input operation information D4 (1,1) to D4 (ma, n) corresponding to the control commands D3 (1,1) to D3 (ma, n) (FIG. 4). The plurality of control target devices 200 (1) to 200 (ma) may be different types of devices, or may be devices of the same type and different models.

  Specifically, the remote control device 100 further includes a reading unit 170 (see FIG. 3) that reads information from the external recording medium ME (see FIG. 3). In place of or in addition to the reading unit 170, the remote control device 100 includes a server 300 (see FIG. 1) via a network 400 such as the public line 401 (see FIG. 1) and the Internet 402 (see FIG. 1) connected to the public line 401. 2) (see FIG. 3). The external recording medium ME is not limited to this, but a typical example is an SD (Secure Digital) memory card. In the present embodiment, remote control device 100 includes both reading unit 170 and second communication unit 180, and these reading unit 170 and second communication unit 180 are remote control device main body 100a (see FIG. 2). It is provided in a state of being housed inside.

  Note that the reading unit 170 and the second communication unit 180 have the same configuration as a conventionally known configuration, and a detailed description thereof is omitted here.

  As shown in FIG. 4, the control commands D3 (1,1) to D3 (ma, n) are stored in advance in the external recording medium ME and the server 300 in association with the device identification information D2 (1) to D2 (ma). Has been. In the present embodiment, the external recording medium ME and the server 300 further include input operation information D4 (1,1) to D4 (ma, n) corresponding to the control commands D3 (1,1) to D3 (ma, n). n) is stored in advance. The second communication unit 180 includes device identification information D2 (1) to D2 (ma), control commands D3 (1,1) to D3 (ma, n), and input operation information D4 (1,1) to D4 ( A receiving unit 181 for receiving ma, n) is provided. In the present embodiment, device identification information D2 (1) to D2 (ma), control commands D3 (1,1) to D3 (ma, n), and input operation information D4 (1,1) to D4 (ma , N) are stored in the external recording medium ME in advance, but the device identification information D2 (1) to D2 (ma) received (downloaded) from the server 300, and the control commands D3 (1, 1) to D3 (ma, n) and input operation information D4 (1, 1) to D4 (ma, n) may be stored in the external recording medium ME.

  The acquisition control step (acquisition control means P1) includes device identification information D2 (1) to D2 (ma) and control commands D3 (1,1) to D3 (ma, n) stored in the external recording medium ME, Furthermore, by reading the input operation information D4 (1,1) to D4 (ma, n) by the reading unit 170, the device identification information D2 (1) to D2 (ma) and the control commands D3 (1,1) to D3 ( ma, n) and input operation information D4 (1, 1) to D4 (ma, n) are acquired. The acquisition control step (acquisition control means P1) includes device identification information D2 (1) to D2 (ma) and control commands D3 (1,1) to D3 (ma, n) stored in the server 300, and By receiving the input operation information D4 (1, 1) to D4 (ma, n) from the server 300 via the network 400 by the receiving unit 181, device identification information D2 (1) to D2 (ma), a control command D3 ( 1, 1) to D3 (ma, n) and input operation information D4 (1, 1) to D4 (ma, n) are acquired. The acquisition control step (acquisition control means P1) includes device identification information D2 (1) to D2 (ma), control commands D3 (1,1) to D3 (ma, n) and input operation information D4 (1, 1) to D4 (ma, n) are temporarily stored in the RAM of the storage unit 120.

  In the present embodiment, the input operation information D4 (1, 1) to D4 (ma, n) includes an “illustration” button (icon image) (see FIG. 5) of a display key (so-called software key) and a display key. Position information (layout information) (see FIG. 5) to be displayed on the display screen G is included. Here, the position information is information (specifically, position information in the horizontal and vertical directions of the display screen G) that determines at which position on the display screen G the display key is displayed.

  The registration control step (registration control means P2) includes one or a plurality of first photographed images D1 (1, 1) photographed once or a plurality of times (a plurality of times in the example shown in FIG. 5) by the digital camera function of the photographing unit 110. ) To D1 (m, t) (t is an integer equal to or greater than 1), and device identification information D2 (1) to D2 () corresponding to the control target devices 200 (1) to 200 (m) as the same control target device 200. m) and the control commands D3 (1,1) to D3 (m, n), and further, the control target device information database DB of the storage unit 120 in association with the input operation information D4 (1,1) to D4 (m, n). (See FIG. 5). Here, the first captured images D1 (1,1) to D1 (m, t) may be still images or moving images. Here, the first captured images D1 (1, 1) to D1 (m, t) are still images.

  In the registration control step (registration control means P2), the first captured images D1 (1,1) to D1 (1, t) of the single control target device 200 (1) are converted into the single control target device 200. Associated with the device identification information D2 (1) and control commands D3 (1,1) to D3 (1, t) corresponding to (1), and further to the input operation information D4 (1,1) to D4 (1, t) Then, it may be previously registered in the control target device information database DB of the storage unit 120.

  The remote control program includes a recognition control step, a generation control step, a display control step, a reception control step, and a transmission control step.

  Specifically, the control unit 160 (see FIG. 2) functions as a recognition control unit P3, a generation control unit P4, a display control unit P5, an acceptance control unit P6, and a transmission control unit P7. That is, the remote control program corresponds to the recognition control step corresponding to the recognition control means P3, the generation control step corresponding to the generation control means P4, the display control step corresponding to the display control means P5, and the reception control means P6. The controller 160 is caused to execute a step including an acceptance control step and a transmission control step corresponding to the transmission control means P7.

  The recognition control step (recognition control means P3) is a second captured image D5 (k is an integer of 1 to m) of one or more control target devices 200 (1) to 200 (k) captured by the imaging unit 110. 1) to D5 (k), which match one or a plurality of matching images, which are registered in the control target device information database DB (see FIG. 5) of the storage unit 120. One or a plurality of control target devices 200 (1) to 200 (k) are recognized by extraction from D1 (m, t) (see FIGS. 13, 14, and 19). Here, the second captured images D5 (1) to D5 (k) may be still images or moving images. Here, the second captured images D5 (1) to D5 (k) are moving images.

  It should be noted that a conventionally known recognition method can be used for recognition of the control target devices 200 (1) to 200 (k), and detailed description thereof is omitted here.

  The generation control step (generation control means P4) includes one or more control target devices 200 recognized from one or more second captured images D5 (1) to D5 (k) in the recognition control step (recognition control means P3). Control commands D3 (1, 1) to D3 corresponding to device identification information D2 (1) to D2 (k) and device identification information D2 (1) to D2 (k) corresponding to (1) to 200 (k), respectively. (K, n), and further, input operation information D4 (1, 1) to D4 (k, n) are read from the control target device information database DB of the storage unit 120 and one or a plurality of device identification information D2 (1) to One or more operation screens Gr (1) of input operation information D4 (1,1) to D4 (k, n) corresponding to D2 (k) and control commands D3 (1,1) to D3 (k, n) ~ Gr (k) (from FIGS. 15, 16, and 20) See 22) to be generated.

  The display control step (display control unit P5) displays one or a plurality of operation screens Gr (1) to Gr (k) generated in the generation control step (generation control unit P4) on the display screen G of the display unit 130.

  In the reception control step (reception control means P6), the input unit 140 inputs one or more operation screens Gr (1) to Gr (k) displayed on the display unit 130 in the display control step (display control means P5). Input operations for the operation information D4 (1, 1) to D4 (k, n) (see FIGS. 15, 16, and 20 to 22) are accepted.

  The transmission control step (transmission control means P7) receives the input operation information D4 (i, j) (i is the received input operation) when the input operation is received from the input unit 140 in the reception control step (reception control means P6). The control command D3 (i, j) corresponding to any value of 1 to k and j is any value of 1 to n) is transmitted from the transmission unit 151 to the control target device 200 (i). By doing so, the remote control device 100 can remotely control the control target device 200 (i) in response to the control command D3 (i, j).

  Here, as the input operation information, for example, as illustrated in FIGS. 15, 16, and 20 to 22, when the control target device 200 is an indoor unit of an air conditioner system, Specifically, the power key), operation switching operation information for switching between cooling and heating (specifically, the cooling key and the heater key), temperature change operation information for changing the set temperature (specifically, temperature) For example, when the control target device 200 is a television, power operation information for turning on / off the power (specifically, a power key), channel change operation information for changing the channel (specifically, a channel) Key), volume change operation information for changing the volume (specifically, a volume key).

  In the present embodiment, the recognition control step (recognition control means P3) is put on hold even when the control target device 200 is recognized, and the control target device 200 is set in a predetermined state in the hold state of recognition of the control target device 200. When the recognition continues only for the time, the recognition of the control target device 200 is confirmed. In other words, the recognition control step (recognition control means P3) does not confirm the recognition of the control target device 200 unless the recognition of the control target device 200 is in a suspended state unless the recognition is continued for a predetermined time.

  In the present embodiment, the remote control program, when the recognition of the control target device 200 is put on hold, informs the user that the recognition of the control target device 200 is put on hold; And a second notification control step of notifying the user that the recognition of the control target device 200 is confirmed when the recognition of the control target device 200 is confirmed.

  Specifically, the control unit 160 (see FIG. 2) further functions as first notification control means P8 and second notification control means P9. That is, the remote control program causes the control unit 160 to execute a step further including a first notification control step corresponding to the first notification control unit P8 and a second notification control step corresponding to the second notification control unit P9.

  Here, as a mode of notifying the user that the recognition of the control target device 200 has been put on hold, the second captured images D5 (1) to D5 (1) to be displayed on the display screen G of the display unit 130 by the digital camera function. A mode in which an index is displayed for the control target devices 200 (1) to 200 (k) of D5 (k) and the display mode of the index is different from the case where the recognition of the control target device 200 is confirmed. Furthermore, the aspect which notifies by voice that recognition of the control object apparatus 200 was put on hold can be illustrated. Here, the frame W is displayed blinking while surrounding the second captured images D5 (1) to D5 (k) displayed on the display screen G with the frame W (FIGS. 13A and 14A). And a broken line in FIG. 19 (a)). In addition, as an example of informing that the recognition of the control target device 200 has been confirmed, the second captured images D5 (1) to D5 (k) displayed on the display screen G of the display unit 130 by the digital camera function. An aspect in which an index is displayed for the control target devices 200 (1) to 200 (k), and the display form of the index is different from the case where the recognition of the control target device 200 is in the hold state, or / A mode of notifying that the recognition of the control target device 200 has been confirmed can be exemplified. Here, the blinking frame W is lit (see solid lines in FIGS. 13B, 14B, and 19B).

[Control example by control unit]
FIG. 6 is a schematic diagram illustrating a state in which a mode selection screen Gs for selecting and operating one of the device information registration mode and the remote control mode in the remote control device 100 is displayed on the display screen G of the display unit 130. It is a front view.

  As illustrated in FIG. 6, the control unit 160 executes the device information registration mode when receiving an input operation of the device information registration mode selection key Ka, and receives the input operation of the remote control mode selection key Kb. Execute.

(One control example of device information registration mode)
First, an example of control when the device information registration mode is selected will be described below with reference to FIGS.

  7 and 8 are flowcharts showing the flow of processing in one control example when the device information registration mode is selected. FIG. 7 shows the first half, and FIG. 8 shows the second half. FIG. 9 is a schematic front view showing a state in which a reception screen Gu for accepting an input operation of model information of the control target device 200 is displayed on the display screen G of the display unit 130 in the device information registration mode. FIG. 10 shows a state in which a shooting screen Gp shot by the shooting unit 110 and a confirmation key for determining whether to register a shot image are displayed on the display screen G of the display unit 130 in the device information registration mode. It is a schematic front view which shows.

  In one control example of the device information registration mode, first, the control unit 160, as shown in FIG. 7, the device identification information D2 (1) to D2 of the control target devices 200 (1) to 200 (m + 1) to be remotely controlled. Prior to registering (m + 1), control commands D3 (1,1) to D3 (m + 1, n) and input operation information D4 (1,1) to D4 (m + 1, n) in the storage unit 120 of the remote control device 100. Then, the number m of the control target devices 200 already registered in the control target device information database DB (see FIG. 5) of the storage unit 120 is substituted into the variable mi (mi ← m) (step S101), and the device identification information D2 ( 1) to D2 (ma), control commands D3 (1, 1) to D3 (ma, n) and input operation information D4 (1, 1) to D4 (ma, n) (see FIG. 4) are stored in the external recording medium ME. Or from server 300 (Step S102). When nothing is registered in the control target device information database DB (when the control target device 200 is newly registered), the value of m is “0”.

  Next, the control unit 160 acquires the device identification information D2 (1) to D2 (ma), the control commands D3 (1,1) to D3 (ma, n), and the input operation information D4 (1,1) acquired in step S102. ) To D4 (ma, n) are temporarily stored in the RAM of the storage unit 120 (step S103).

Next, the control unit 160 displays an acceptance screen Gu that accepts input information of device identification information D2, which is model information of the device 200 to be controlled, at the input position Kc (see FIG. 9), and waits until the device identification information D2 is accepted. (Step S104: No),
Next, when receiving the device identification information D2 in step S104 (step S104: Yes), the control unit 160 adds 1 to mi (mi ← mi + 1) (step S105), and receives the received device identification information D2 (mi). Is temporarily stored in the RAM of the storage unit 120 (step S106).

  Next, the control unit 160 resets t (t ← 0) (step S107) and activates the photographing unit 110 (step S108).

  Next, as illustrated in FIG. 8, the control unit 160 adds 1 to t (t ← t + 1) (step S <b> 109), and when the first captured image D <b> 1 (mi, t) is captured by the capturing unit 110. (Step S110), the first photographed image D1 (mi, t) (see FIG. 10) photographed by the photographing unit 110 and the first photographed image D1 (mi, t) are registered in the display screen G of the display unit 130. A confirmation key Kd (see FIG. 10) for determining whether or not to perform (in this example, “register” Kd1, “not register” Kd2) is displayed (step S111). At this time, a confirmation key Ke described later is displayed in a light color and does not accept an input operation.

  Next, the control unit 160 waits until the confirmation key Kd is accepted (step S112: No). When the confirmation key Kd is accepted (step S112: Yes), the input operation of the “register” Kd1 has been accepted or “registered” It is determined whether or not “Kd2” has been accepted (step S113), and when an input operation of “not registered” Kd2 is accepted (step S113: not registered), the process proceeds to step S110 while “registering” Kd1 is input. When the operation is accepted (step S113: registration), the first captured image D1 (mi, t) is temporarily stored in the RAM of the storage unit 120 (step S114).

  Here, since it is considered that the user controls the control target device 200 from various angles using the remote control device 100, the recognition accuracy of the control target devices 200 (1) to 200 (k) in the remote control mode. In order to improve the image quality, the control unit 160 uses a confirmation key Ke (see the broken line in FIG. 10) for determining whether or not to further shoot by the photographing unit 110 (in this example, “Re-photograph” Ke1, “Do not re-photograph” Ke2). ) Is displayed (step S115). At this time, the confirmation key Kd is lightly displayed and does not accept input operation, while the confirmation key Ke is darkly displayed and accepts input operation.

  Next, the control unit 160 waits until the confirmation key Ke is accepted (step S116: No). When the confirmation key Ke is accepted (step S116: Yes), the control unit 160 has accepted an input operation of “Reshoot” Ke1 or “ It is determined whether an input operation of Ke2 “do not re-photograph” is accepted (step S117). If an input operation of Ke1 “re-photograph” is accepted (step S117: photographing), the process proceeds to step S109, while “ When an input operation of “Do not re-photograph” “Ke2) is received (step S117: Do not shoot), the shooting is terminated (step S118).

  Then, the control unit 160 stores the first captured images D1 (mi, 1) to D1 (mi, t) stored in the RAM of the storage unit 120 and the device identification information D2 stored in the RAM of the storage unit 120. (Mi) and control commands D3 (mi, 1) to D3 (mi, n) corresponding to the device identification information D2 (mi) and input operation information D4 (mi, 1) to D4 (mi, n) are stored in association with each other. The information is stored in the control target device information database DB (see FIG. 5) of the unit 120 (step S119), and the device information registration mode is terminated.

(Example of remote control mode)
Next, an example of control when the remote control mode is selected (an example in the case of recognizing one control target device 200 (1)) will be described below with reference to FIGS.

  FIGS. 11 and 12 are flowcharts showing the flow of processing according to an example of control when the remote control mode is selected. FIG. 11 shows the first half, and FIG. 12 shows the second half. 13 and 14, the shooting screen Gp in which the second shot image D5 (1) of one control target device 200 (1) is shot in the remote control mode is displayed on the display screen G of the display unit 130. It is a schematic front view which shows a state. FIG. 13A and FIG. 13B show a state in which recognition of an indoor unit acting as the control target device 200 (1) is suspended and a state in which the indoor unit is confirmed. FIG. 14A and FIG. 14B respectively show a state in which recognition of a television acting as the control target device 200 (1) is suspended and a state in which it is confirmed. FIG. 15 shows the input operation information D4 (1,1) to D4 (1) corresponding to the device identification information D2 (1) of the indoor unit and the control commands D3 (1,1) to D3 (1,5) in the remote control mode. , 5) is a schematic front view showing an operation screen Gr (1). FIG. 16 shows input operation information D4 (1, 1) to D4 corresponding to the television device identification information D2 (1) and the control commands D3 (1, 1) to D3 (1, 5) in the remote control mode. It is a schematic front view which shows the operation screen Gr (1) which displayed (1, 5).

  In one control example of the remote control mode, first, as shown in FIG. 11, the control unit 160 sets a predetermined value (an integer equal to or greater than 1) to an arbitrary value N (step S1). The arbitrary value N may be a fixed value determined on the program, or may be set in advance on an initial setting screen (not shown) that can be set by the user.

  Next, the control unit 160 activates the photographing unit 110 to recognize the control target devices 200 (1) to 200 (k) (step S202), and substitutes 1 for the counter ki (ki ← 1) ( In step S203, recognition of the control target devices 200 (1) to 200 (k) is started using the first captured images D1 (1,1) to D1 (m, t) registered in the device information registration mode (step S203). S204).

  Next, the control unit 160 matches the second captured image D5 (ki) (see FIGS. 13A and 14A) of the control target device 200 (ki) captured by the imaging unit 110. Are extracted from the first captured images D1 (1, 1) to D1 (m, t) registered in the control target device information database DB (see FIG. 5) of the storage unit 120, and the control target device 200 ( The recognition operation is continued until ki) is recognized (step S205: No). On the other hand, when the control unit 160 recognizes the control target device 200 (ki) (step S205: Yes), the control unit 160 resets the counter M (M ← 0) (step S206).

  Next, in order to indicate to the user that the control target device 200 (ki) is recognized, the control unit 160 causes the display unit 130 to capture the second captured image D5 (ki). The periphery of the control target device 200 (ki) displayed on the screen is surrounded by a frame W, and the frame W is blinked (see the broken lines in FIGS. 13A and 14A) (step S207).

  Next, after a predetermined time has elapsed (step S208), the control unit 160 determines whether or not the control target device 200 (ki) continues to be captured in the imaging unit 110 (step S209), and performs control. When the target device 200 (ki) is not continuously reflected in the photographing unit 110 (step S209: No), the process proceeds to step S205, while the control target device 200 (ki) is continuously reflected in the photographing unit 110. (Step S209: Yes), 1 is added to the counter M (M ← M + 1) (Step S210).

  Next, the control unit 160 compares whether or not the value of the counter M is equal to an arbitrary value N set in advance (step S211), and if the value of the counter M is not equal to the arbitrary value N (step S211). : No), the process proceeds to step S207, and the processes of step S205 to step S210 are repeated until the value of the counter M becomes equal to an arbitrary value N.

  By this process, the operation screen Gr (ki) (FIGS. 15 and 16) is changed only when the control target device 200 (ki) is continuously recognized for a certain period or longer. Therefore, in order to capture the control target device 200 (ki) desired by the user, another control target device that is not desired by the user in the process of adjusting the shooting angle is erroneously recognized as the control target device 200 (ki). The probability of being lost can be reduced.

  Next, when the counter M is equal to N (step S211: Yes), the control unit 160 displays the blinking frame W in a lit state (see solid lines in FIGS. 13B and 14B). (Step S212), the process proceeds to step S213 shown in FIG. In FIG. 13 and FIG. 14, ki is shown as 1.

  Next, as illustrated in FIG. 12, the control unit 160 ends the shooting by the shooting unit 110 (step S <b> 213), and a device corresponding to the control target device 200 (ki) recognized from the second shot image D <b> 5 (ki). Identification information D2 (ki) (here, the name of the control target device 200 (ki)), control commands D3 (ki, 1) to D3 (ki, n) and input operation information D4 corresponding to the device identification information D2 (ki) (Ki, 1) to D4 (ki, n) are read from the control target device information database DB of the storage unit 120, and device identification information D2 (ki) (see FIGS. 15 and 16) and a control command D3 (ki, 1) Operation screen Gr (ki) for displaying display operation keys of input operation information D4 (ki, 1) to D4 (ki, n) (here, n = 5) corresponding to .about.D3 (ki, n) (FIG. 15 and FIG. (See Figure 16) To formed (step S214).

  Next, the control unit 160 displays the operation screen Gr (ki) (see FIGS. 15 and 16) of the control target device 200 on the display unit 130 instead of the shooting screen Gp that has been displayed on the display unit 130 until now. It is displayed on the screen G (step S215). In FIG. 15 and FIG. 16, ki is shown as 1.

  On the operation screen Gr (ki), as shown in FIG. 15 and FIG. 16, among the information registered in the device information registration mode (see FIG. 5), the control target device 200 (device identification information D2 (ki)) ( ki) and input operation information D4 (ki, 1) to D4 (ki, n) ("illustration" button) whose positions are set on the display screen G by layout information are displayed as display keys.

  In the present embodiment, the device identification information D2 (ki) (here, the name of the control target device 200 (ki)) is generated and displayed on the display unit 130 as the operation screen Gr (ki). It is not always necessary to generate the device identification information D2 (ki), and therefore it is not necessary to display the device identification information D2 (ki). The same applies to other control examples described later shown in FIGS. 20 to 22.

  Further, on the display screen G, input operation information (“BACK” button) BK indicating that the process returns to the process of recognizing the control target devices 200 (1) to 200 (k) is displayed as a display key.

  Next, the control unit 160 stands by until an input operation by the user is received on the operation screen Gr (ki) (step S216: No). When the input operation is received (step S216: Yes), the received input operation is It is determined whether or not the display key (“BACK”) BK is an input operation (step S217). If it is determined that the display key (“BACK”) BK is an input operation (step S217: Yes), When the process proceeds to step S202 shown in FIG. 11 and it is determined that the input operation is not for the display key (“BACK”) BK (step S217: No), the accepted input operation is the input operation information D4 (1, 1). It is determined whether or not the input operation is to D4 (k, n) (step S218).

  Next, when the control unit 160 determines in step S218 that the input operation information D4 (1, 1) to D4 (k, n) is not an input operation (step S218: No), the process proceeds to step S216. If it is determined that the input operation information D4 (1,1) to D4 (k, n) is an input operation (step S218: Yes), the input operation information D4 (i, j) (i, j) ( Here, the control command D3 (i, j) corresponding to i = 1, j = 1 to 5) is transmitted from the transmission unit 151 to the control target device 200 (i) (step S219).

  Then, the control unit 160 determines whether or not there is an instruction to end the process in the remote control mode (step S220), and when there is no instruction to end the process (step S220: No), the process proceeds to step S216. If there is an instruction to end the process (step S220: Yes), the remote control mode is ended.

(Other examples of remote control mode)
By the way, since there may be a plurality of control target devices 200 to be controlled, in detail, when the user captures the control target device 200 with the photographing unit 110, the photographing angle may be gradually changed. Yes, it is conceivable that there will be a plurality of control target devices in the middle of photographing. Therefore, it is preferable to recognize a plurality of control target devices 200 (1) to 200 (k).

  Next, another control example (an example in the case of recognizing a plurality of control target devices 200 (1) to 200 (k)) when the remote control mode is selected will be described below with reference to FIGS. Explained.

  17 and 18 are flowcharts showing the flow of processing according to another control example when the remote control mode is selected. FIG. 17 shows the first half, and FIG. 18 shows the second half. FIG. 19 shows the display screen of the display unit 130 as the shooting screen Gp in which the second shot images D5 (1) to D5 (2) of the plurality of control target devices 200 (1) to 200 (2) are shot in the remote control mode. It is a schematic front view which shows the state currently displayed on G. FIG. 19A shows a state in which recognition of an indoor unit and a television acting as the control target device 200 is suspended. FIG. 19B shows a state where the recognition of the outdoor unit and the television acting as the control target device 200 is confirmed. FIG. 20 shows the input operation information D4 () corresponding to the device identification information D2 (1) to D2 (2) and the control commands D3 (1,1) to D3 (2,5) of the indoor unit and the television in the remote control mode. It is a schematic front view which shows the operation screens Gr (1) -Gr (2) which displayed (1,1) -D4 (2,5). FIG. 21 is a schematic front view showing a modification of the operation screens Gr (1) to Gr (2) shown in FIG. FIG. 21A shows the input operation information D4 (1,1) to D4 (2,3) switched to the higher priority, and FIG. 21B shows the lower priority. The switched input operation information D4 (1, 4) to D4 (2, 5) is shown. FIG. 22 is a schematic front view showing another modification of the operation screens Gr (1) to Gr (2) shown in FIG. FIG. 22A shows the input operation information D4 (1,1) to D4 (1,5) switched to the indoor unit, and FIG. 22B is switched to the television. The input operation information D4 (2,1) to D4 (2,5) is shown.

  In other control examples of the remote control mode shown in FIGS. 17 and 18, the basic flow is the same as that of the control example shown in FIGS. 11 and 12, and the same reference numerals are given.

  Another control example of the remote control mode shown in FIGS. 17 and 18 is the same as the control example shown in FIGS. 11 and 12, with the processing of step S210a to step S210b being provided between step S210 and step S211. Step S211a is provided between Step S212 and Step S212, Step S213a is provided between Step S213 and Step S214, and Steps S215a to S215b are provided between Step S215 and Step S216. It is a thing. Hereinafter, another control example of the remote control mode shown in FIGS. 17 and 18 will be described focusing on differences from the control example shown in FIGS. 11 and 12.

  That is, in step S205 illustrated in FIG. 17, the control unit 160 matches the second captured image D5 (ki) (see FIG. 19A) of the control target device 200 (ki) captured by the capturing unit 110. An image is extracted from the first captured images D1 (1,1) to D1 (m, t) registered in the control target device information database DB (see FIG. 5) of the storage unit 120, and the control target device 200 is extracted. The recognition operation is continued until (ki) is recognized (step S205: No).

  In step S207, the control unit 160 captures the second captured image D5 (ki) on the display unit 130 to indicate to the user that the control target device 200 (ki) is recognized. The periphery of the control target device 200 (ki) displayed on the shooting screen Gp is surrounded by a frame W, and the frame W is blinked (see the broken line in FIG. 19A).

  In step S210a, the control unit 160 determines whether or not the other control target device 200 is captured, and if it is determined that the other control target device 200 is captured (step S210a: Yes), 1 is added to ki (ki ← ki + 1) (step S210b), and the process proceeds to step S205. On the other hand, when the control unit 160 determines that the other device to be controlled 200 is not captured (step S210a: No), the control unit 160 proceeds to step S211 and when the counter M is equal to N in step S211 (step S210a). S211: Yes), the value of ki is substituted for k (k ← ki) (step S211a), the blinking frame W is lit (see the solid line in FIG. 19B) (step S212), and FIG. The process proceeds to step S213 shown in FIG. In FIG. 19, ki is shown as 1 and 2.

  In step S213a shown in FIG. 18, the control unit 160 substitutes 1 for ki (ki ← 1).

  In step S214, the control unit 160 also identifies device identification information D2 (ki) (here, the control target device 200 (ki) of the control target device 200 (ki) corresponding to the control target device 200 (ki) recognized from the second captured image D5 (ki). Name), control commands D3 (ki, 1) to D3 (ki, n) and input operation information D4 (ki, 1) to D4 (ki, n) corresponding to the device identification information D2 (ki). Input operation information D4 (ki, n) corresponding to device identification information D2 (ki) (see FIGS. 20 to 22) and control commands D3 (ki, 1) to D3 (ki, n) read from the control target device information database DB An operation screen Gr (ki) (see FIGS. 20 to 22) for displaying display keys 1) to D4 (ki, n) (here, n = 5) is generated.

  In step S215, the control unit 160 displays the operation screen Gr (ki) (see FIGS. 20 to 22) of the control target device 200 instead of the shooting screen Gp that has been displayed on the display unit 130 so far. 130 is displayed on the display screen G. 20 to 22, ki is indicated as 1 and 2.

  In step S215a, the control unit 160 determines whether or not ki is equal to k. When it is determined that ki is equal to k (step S215a: Yes), the process proceeds to step S216, while ki is When it is determined that it is not equal to k (step S215a: No), 1 is added to ki (ki ← ki + 1) (step S215b), and the process proceeds to step S214.

  Since a plurality of control target devices 200 (1) to 200 (k) (here k = 2) are recognized, the operation screens Gr (1) to Gr (k) are also a plurality of control target devices 200 (1 ) To 200 (k). For example, as the operation screens Gr (1) to Gr (k) generated in step S215, the operation screens Gr (1) to Gr (k) are displayed on the display screen G (for example, the operation screens Gr on the display screen G are displayed). When (1) to Gr (k) cannot be entered, the operation screens Gr (1) to Gr (k) are displayed in a reduced scale) (see FIG. 20), input operation information D4 (1, 1) to D4 (k, n) are prioritized, and one or a plurality of units are selected from the ones with the highest priority by the screen switching key CH (see the center right side of the display screen G in FIG. 21) on the display screen G. In the mode of performing switching display for sequentially switching the input operation information D4 (1,1) to D4 (k, n) (see FIG. 21), or the screen switching key CH (below the display screen G in FIG. Device identification information D (1) ~ D2 (k) operation per window Gr (1) switches the ～Gr (k) mode for switching the display (see FIG. 22), and the like.

  In step S217, if the control unit 160 determines that the input operation is a display key (“BACK”) BK (step S217: Yes), the process proceeds to step S202 shown in FIG. ("BACK") If it is determined that the input operation is not BK (step S217: Yes), is the received input operation an input operation of the input operation information D4 (1, 1) to D4 (k, n)? It is determined whether or not (step S218).

  In step S218, when the control unit 160 determines that the input operation information D4 (1, 1) to D4 (k, n) is not input (step S218: No), the process proceeds to step S216. If it is determined that the input operation information D4 (1,1) to D4 (k, n) is an input operation (step S218: Yes), the input operation information D4 (i, j) (i, j) ( Here, the control command D3 (i, j) corresponding to i = 1 or 2, j = 1-5 is transmitted from the transmission unit 151 to the control target device 200 (i) (step S219).

[About this embodiment]
As described above, in the present embodiment, prior to controlling the control target device 200, first, device identification information D2 (1) to D2 (ma) for identifying the control target devices 200 (1) to 200 (ma). ), And control commands D3 (1) to D3 (ma) for controlling the control target devices 200 (1) to 200 (ma), and further input operation information D4 (1,1) to D4 (ma, n) ) To get. Next, the first captured images D1 (1,1) to D1 (m, t) of the control target devices 200 (1) to 200 (m) photographed by the photographing unit 110 are used as control target devices 200 (1) to 200 (1) to 200 (1) to 200 (1). Device identification information D2 (1) to D2 (m) and control commands D3 (1,1) to D3 (m, n) corresponding to 200 (m), and input operation information D4 (1,1) to D4 ( m, n) is registered in the storage unit 120 in advance. In this way, device identification information D2 (1) to D2 (m), control commands D3 (1,1) to D3 (m, n), and input operation information D4 (1,1) to D4 (m, n) are stored. Matching with second captured images D5 (1) to D5 (k) of the control target devices 200 (1) to 200 (k) captured by the capturing unit 110 in a state registered in advance in the unit 120 An image is extracted from the first captured images D1 (1,1) to D1 (m, t) registered in the storage unit 120 to recognize the control target devices 200 (1) to 200 (k); Control commands D3 (1,1) to D3 (k, n) corresponding to the device identification information D2 (1) to D2 (k) of the recognized control target devices 200 (1) to 200 (k), and further input operation Information D4 (1,1) to D4 (k, n) is read from the storage unit 120 and a control command The operation screens Gr (1) to Gr (k) of the input operation information D4 (1,1) to D4 (k, n) corresponding to 3 (1,1) to D3 (k, n) are generated and generated. The operation screens Gr (1) to Gr (k) are displayed on the display unit 130. In this way, when the control target devices 200 (1) to 200 (k) are controlled, the photographing unit 110 does not have to be directed to the control target devices 200 (1) to 200 (k). Good. That is, after the control target devices 200 (1) to 200 (k) are recognized, it is not always necessary to place the photographing unit 110 in the direction of the control target devices 200 (1) to 200 (k). When an input operation for input operation information D4 (1,1) to D4 (k, n) on the operation screens Gr (1) to Gr (k) is received from the input unit 140, the input operation of the received input operation is received. A control command D3 (i, j) corresponding to the information D4 (i, j) is transmitted from the transmission unit 151 to the control target device 200 (i). Thereby, it becomes possible to control the control target device 200 (i).

  In addition, since the photographing unit 110 and the display unit 130 are provided in the remote control device 100, the device configuration can be reduced in size and simplified. Specifically, it can be performed if there is one composite device such as a smartphone, and the number of necessary devices can be reduced.

  In the present embodiment, the registration control step (registration control means P2) uses the same plurality of first photographed images D1 (1,1) to D1 (m, t) photographed a plurality of times by the photographing unit 110. Device identification information D2 (1) to D2 (m) and control commands D3 (1,1) to D3 (m, n) as control target devices 200 (1) to 200 (m), and further input operation information D4 (1 , 1) to D4 (m, n) and registered in advance in the storage unit 120, so that the plurality of first photographed images obtained by photographing the control target devices 200 (1) to 200 (m) from different angles. D1 (1, 1) to D1 (m, t) can be registered in advance in the storage unit 120 as the same control target device 200 (1) to 200 (m), and thereby the second captured image D5. Control target devices 200 (1) to 20 by (1) to D5 (k) Recognition accuracy can be improved (k).

  By the way, when the user controls the control target device 200 for which remote control is desired, another control target device 200 for which the user does not desire remote control may be reflected. In this case, if the control target device 200 is recognized immediately, another control target device 200 that is not desired by the user may be erroneously recognized.

  In this respect, in this embodiment, even if the control target device 200 is recognized, the control target device 200 is put on hold, and when the control target device 200 is continuously recognized for a predetermined time, the recognition of the control target device 200 is confirmed. Even if other undesired control target device 200 may be reflected, the probability of erroneously recognizing the other control target device 200 not desired by the user as the control target device 200 desired by the user is reduced. Can do.

  By the way, if there is no first notification step (first notification control means P8), even when the user is photographing the article, even if the recognition of the control target device 200 is put on hold by the second captured image D5. Until the operation screen Gr (see FIGS. 15 and 16 and FIGS. 20 to 22) is displayed on the display unit 130, the user understands whether or not the article is recognized as the control target device 200. I can't.

  In this regard, in the present embodiment, when recognition of the control target device 200 is put on hold, a notification is made that the recognition of the control target device 200 is put on hold by the first notification step (first notification control means P8). Thus, when the user is photographing the article, if the control target device 200 is recognized by the second photographed image D5, the user understands that the photographed article is recognized as the control target device 200. be able to.

  Further, if there is no second notification control step (second notification control means P9), even if the user recognizes the control target device 200 from the second captured image D5 when the user is shooting an article, Until the operation screen Gr (see FIGS. 15 and 16 and FIGS. 20 to 22) is displayed on the display unit 130, the user can understand whether or not the article is determined to be the control target device 200. Can not.

  In this regard, in the present embodiment, when the recognition of the control target device 200 is confirmed, the second notification control step (second notification control means P9) notifies that the recognition of the control target device 200 is confirmed. If the recognition of the control target device 200 is confirmed by the second photographed image D5 when the user is photographing the article, the user is made to understand that the photographed article is confirmed to be the control target device 200. Can do.

  In the present embodiment, a plurality of operation screens Gr (1) to Gr (k) (displayed on one display unit 130 by recognizing the plurality of control target devices 200 (1) to 200 (k) ( The input operation for the input operation information D4 (1,1) to D4 (k, n) in FIG. 20 to FIG. 22 can be received from the input unit 140. Accordingly, a plurality of operations displayed on one display unit 130 can be received. Operation for controlling a plurality of control target devices 200 (1) to 200 (k) with respect to the input operation information D4 (1,1) to D4 (k, n) on the screens Gr (1) to Gr (k). The convenience can be improved.

  In the present embodiment, the reading unit 170 that reads information from the external recording medium ME, and / or the device identification information D2 (1) to D2 (ma) and the control command D3 (from the server 300 via the network 400). 1, 1) to D3 (ma, n) are further provided, so that, for example, addition of models of control target devices 200 (1) to 200 (ma) or device identification information D2 (1) to Even when D2 (ma) and / or control commands D3 (1, 1) to D3 (ma, n) are changed, the external recording medium ME corresponds to the control target devices 200 (1) to 200 (ma). Device identification information D2 (1) to D2 (ma) and / or control commands D3 (1,1) to D3 (ma, n), and further input operation information D4 (1,1) to D4 (ma, n) get Therefore, the addition of models of various control target devices 200 (1) to 200 (ma) or device identification information D2 (1) to D2 (ma) and / or control commands D3 (1,1) to D3 It is possible to easily cope with the change of (ma, n).

  By the way, device identification information D2 (1) to D2 (ma) and control commands D3 (1,1) to D3 (ma, n), and input operation information D4 (1,1) are individually assigned to each external recording medium ME. ) To D4 (ma, n) are stored, the addition of the models of the control target devices 200 (1) to 200 (ma) or the device identification information D2 (1) to D2 (ma) and / or the control command D3 (1) , 1) to D3 (ma, n) when the device identification information D2 (1) to D2 (ma) and / or control commands D3 (1, 1) to D3 (ma, n) are accompanied. Furthermore, the workability of the storage operation of the input operation information D4 (1,1) to D4 (ma, n) becomes complicated.

  In this regard, in the present embodiment, the device identification information D2 (1) to D2 (ma) and the control commands D3 (1,1) to D3 (ma, n), and the input operation information D4 ( 1, 1) to D4 (ma, n) are stored, so that various control target devices 200 (1) to 200 (ma) are added or device identification information D2 (1) to D2 (ma) and Device identification information D2 (1) to D2 (ma) and / or control commands D3 (1,1) to D3 (ma, n) accompanying change of control commands D3 (1,1) to D3 (ma, n) In addition, it is possible to improve the workability of the storage operation of the input operation information D4 (1, 1) to D4 (ma, n).

  In the present embodiment, since the input unit 140 is a touch input unit, it is possible to directly input to the operation screen Gr displayed on the display screen G, and the operability can be improved accordingly. Thus, as in the present embodiment, it can be easily applied to the mobile terminal device 10 such as a smartphone.

DESCRIPTION OF SYMBOLS 10 Mobile terminal device 100 Remote control apparatus 100a Remote control apparatus main body 110 Image pick-up part 120 Storage part 130 Display part 140 Input part 150 1st communication part 151 Transmission part 160 Control part 161 Processing part 170 Reading part 180 2nd communication part 181 Reception part 200 Controlled Device 300 Server 400 Network 401 Public Line 402 Internet CH Screen Switch Key D1 First Captured Image D2 Device Identification Information D3 Control Command D4 Input Operation Information D5 Second Captured Image DB Controlled Device Information Database G Display Screen Gp Shooting Screen Gr operation screen Gs mode selection screen Gu reception screen M counter ME external recording medium P control program P1 acquisition control means P2 registration control means P3 recognition control means P4 generation control means P5 display control means P6 reception control means P7 transmission control means P8 first News Control means P9 second notification control unit W frame

Claims (8)

A remote control device for controlling the control target device from a position away from the control target device,
An imaging unit for imaging the device to be controlled;
A storage unit that stores a first captured image of the control target device imaged by the imaging unit, device identification information for identifying the control target device, and a control command for controlling the control target device;
A display unit for displaying an operation screen of input operation information corresponding to the device identification information and the control command;
An input unit for receiving an input operation for the input operation information on the operation screen;
A transmission unit for transmitting the control command to the device to be controlled;
Acquisition control means for acquiring the device identification information and the control command;
Registration control means for previously registering the first captured image of the control target device captured by the capturing unit in the storage unit in association with the device identification information and the control command corresponding to the control target device; ,
Recognition that recognizes the control target device by extracting from the first captured image registered in the storage unit a matching image that matches the second captured image of the control target device captured by the photographing unit. Control means;
Generation control means for reading the control command corresponding to the device identification information of the control target device recognized from the second photographed image from the storage unit and generating the operation screen of the input operation information corresponding to the control command When,
Display control means for displaying the operation screen on the display unit;
An acceptance control means for accepting the input operation for the input operation information on the operation screen from the input unit;
Transmission control means for transmitting the control command corresponding to the input operation information of the received input operation from the transmission unit to the control target device when the input operation is received from the input unit. Remote control device. The remote control device according to claim 1,
The registration control means registers a plurality of the first photographed images photographed a plurality of times by the photographing unit in the storage unit in association with the device identification information and the control command as the same control target device. A remote control device characterized by that. The remote control device according to claim 1 or 2,
The recognition control unit is in a suspended state even when the control target device is recognized, and when the control target device is continuously recognized for a predetermined time, the recognition control unit confirms the recognition of the control target device. Remote control device. The remote control device according to claim 3,
The remote control device further comprising first notification control means for notifying that the recognition of the device to be controlled has been put on hold. The remote control device according to claim 3 or claim 4,
The remote control device further comprising second notification control means for notifying that the recognition of the control target device is confirmed. The remote control device according to any one of claims 1 to 5,
The acquisition control means acquires the device identification information and the control command corresponding to a plurality of devices to be controlled,
The registration control unit associates the first captured images of the plurality of control target devices captured by the capturing unit with the device identification information and the control command corresponding to the control target device in the storage unit in advance. Register,
The recognition control means includes a plurality of the coincidence images that respectively coincide with the second photographed images of the plurality of control target devices photographed by the photographing unit, in the first photographed image registered in the storage unit. Recognize each of a plurality of control target devices extracted from the inside,
The generation control unit reads a plurality of the control commands respectively corresponding to the device identification information of the plurality of control target devices recognized from the plurality of second captured images from the storage unit, and outputs the control commands to the plurality of control commands. Generating a plurality of the operation screens of the corresponding input operation information,
The display control means displays a plurality of the operation screens on the display unit,
The remote control device, wherein the reception control unit receives the input operation for the input operation information on the plurality of operation screens from the input unit. The remote control device according to any one of claims 1 to 6,
A receiving unit that receives the device identification information and the control command from a server that stores the control command in association with the device identification information;
The acquisition control means acquires the device identification information and the control command by receiving the device identification information and the control command stored in the server from the server by the receiving unit via the network. Features remote control device. A remote control method of a remote control device for controlling the control target device from a position away from the control target device,
Obtaining device identification information for identifying the device to be controlled, and a control command for controlling the device to be controlled;
A first captured image of the control target device captured by a capturing unit included in the remote control device is provided in the remote control device in association with the device identification information and the control command corresponding to the control target device. Registering in advance in the storage unit;
Extracting a matching image that matches the second captured image of the control target device captured by the capturing unit from the first captured image registered in the storage unit and recognizing the control target device When,
Reading the control command corresponding to the device identification information of the control target device recognized from the second captured image from the storage unit and generating an operation screen of input operation information corresponding to the control command;
Displaying the operation screen on a display unit provided in the remote control device;
Receiving an input operation for the input operation information on the operation screen from an input unit provided in the remote control device;
Transmitting the control command corresponding to the input operation information of the received input operation from the transmission unit included in the remote control device to the control target device when the input operation is received from the input unit; A remote control method comprising:

Images