JP2014064115A - Terminal device, remote operation system, and remote operation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow for intuitive operation of a plurality of control target devices by one terminal device.SOLUTION: A terminal device comprises: a device management database in which device information of a control target device, operation screen information, and operation information are stored; image recognition means for comparing photographic image data of the control target device photographed by photographing means with device image data which is one of the device information stored in the device management database, and recognizing and identifying the control target device; information extraction means for referencing the device management database on the basis of device information of the identified control target device and extracting the operation information corresponding to the control target device; display means for displaying the photographed photographic image; operation input means for specifying any location on a display screen of the display means and performing operation input; and output means for acquiring an operation signal corresponding to an operation region from the operation information and outputting the signal to the control target device when an operation region defined by the operation information extracted by the operation input means is operated.

Description

  The present invention relates to a terminal device, a remote operation system, and a remote operation method for remotely operating various devices to be controlled with a single device.

  Conventionally, a technology for remotely operating a plurality of information appliances connected to a home network using augmented reality technology has been disclosed (see, for example, Patent Document 1).

  In addition, a technique is disclosed in which a plurality of electronic devices are distinguished from each other and selected as a target for remote operation (see, for example, Patent Document 2).

  The information home appliance operation system described in Patent Literature 1 includes a marker in the information home appliance, acquires status information and / or content information of the corresponding information home appliance when a marker is detected, and corresponds to the information home appliance corresponding to the image data in the frame memory. The operation interface image data including the status information and / or the content information is synthesized, and the operation parameter value and / or the content selection information input through the operation interface is transmitted to the corresponding information home appliance.

  In addition, the image processing apparatus described in Patent Document 2 includes a data storage unit that stores object identification data for identifying an object that can be operated by a user, and feature data that represents features of the appearance of each object, An environment map representing the position of one or more objects existing in space, the input image obtained by photographing the real space using an imaging device, and the feature stored in the data storage unit An environment map storage unit that stores the environment map generated based on the data, and an object included in the environment map stored in the environment map storage unit can be operated based on the object identification data And a selection unit that selects at least one object recognized as a candidate object that is a candidate for an operation target by the user.

JP 2012-37919 A JP 2011-146996 A

  In the information home appliance operation system described in Patent Document 1, a device (information home appliance) that can be connected to a network that complies with the Digital Living Network Alliance (DLNA) guidelines is an object of operation, and a device that cannot be connected to the network cannot be operated. There was a problem. That is, there is a problem that conventional remote control compatible devices cannot be operated.

  In addition, since the image processing apparatus described in Patent Document 2 identifies a device using an environment map that represents the position of an object, there is a problem that remote operation may be difficult if the device arrangement is changed. It was. In the image processing apparatus described in Patent Document 2, a command field for displaying a control command for operating a device is displayed in the center of the screen in the form of a balloon (balloon). By operating, the control command is transmitted to the operation target device. For this reason, there has been a problem that it is not possible to intuitively operate the device to be operated on the remote controller.

  The present invention was devised to solve such problems, and its purpose is to provide a terminal device, a remote operation system, and a remote operation method capable of operating an operation target device more intuitively and improving operability. There is to do.

  In order to solve the above problems, a terminal device of the present invention is a terminal device for remotely operating a control target device, and performs operations corresponding to various operations of the device information, operation screen information, and operation screen information of the control target device. One of the device management database in which information is stored, photographing means for photographing the control target device, data of the photographed image of the control target device, and the device information stored in the device management database. Image recognition means for recognizing and identifying the device by comparing data of a certain device image, and referring to the device management database based on the device information of the identified device to be controlled, corresponding to the device to be controlled Information extracting means for extracting operation information to be performed, display means for displaying the photographed image, and operation for instructing and inputting an arbitrary position on the display screen of the display means. When an operation area defined by the extracted operation information is operated by the input means and the operation input means, an operation signal corresponding to the operation area is acquired from the operation information, and the control target device And an output means for outputting the output.

  In the terminal device according to the present invention, the coordinate conversion is performed on the operation area on the operation input unit so as to coincide with the display position of the corresponding operation area of the control target device displayed on the display unit. It is said.

  According to the present invention, by directly operating an operation area including various operation function keys and the like of the control target device displayed on the display unit of the terminal device on the display screen, the control target device displayed on the display unit is Operation becomes more intuitive.

  In the terminal device of the present invention, the information extraction unit further extracts operation screen information corresponding to the operation information of the control target device, and the display unit extracts the operation extracted by the information extraction unit. The screen information may be combined with the captured image and displayed.

  Further, the terminal device according to the present invention is configured to include an emphasis processing means for emphasizing the display of the operation area displayed on the display means. In this case, the highlighting process can be a process of enlarging and displaying the operation area. Further, the enhancement processing may be processing for rearranging and displaying the operation areas. Further, the enhancement process is a process of surrounding the operation region itself or a peripheral part including the operation region with a frame, and the output unit operates the inside of the region surrounded by the frame, The operation signal corresponding to the operation region can be output. Still further, the enhancement process is a process of displaying a balloon corresponding to the operation area, and the output means operates the operation area corresponding to the balloon by operating the balloon. It can be configured to output a signal.

  In the terminal device of the present invention, when a plurality of control target devices are photographed by the photographing unit and the plurality of control target devices are recognized by the image recognition unit, the display unit displays all the recognized control items. The operation screen information of the target device can be combined with the captured image and displayed. By displaying a plurality of operation screen information on the display means, a plurality of operation screen information can be individually operated on the same screen.

  In the terminal device of the present invention, the display is switched from the display of the plurality of operation screen information to the display of any one operation screen information, and is switched from the display of the one operation screen information to the display of the plurality of operation screen information. It is good also as a structure provided with the switching means. That is, only arbitrary operation screen information desired by the user can be displayed from a state in which a plurality of operation screen information is displayed.

  In the terminal device of the present invention, the photographing direction of the photographing unit and the output direction of the output unit are arranged in the same direction on the apparatus housing.

  The remote operation system of the present invention is a remote operation system for remotely operating a control target device by communication between a device management server and a terminal device, wherein the device management server includes device information of the control target device and A device management database in which operation information is stored; a receiving unit for receiving specific information for identifying a control target device transmitted from the terminal device; and the device management database based on the specific information received by the receiving unit. And an information extracting unit that extracts the operation information corresponding to the control target device, and a transmission unit that transmits the extracted operation information to the terminal device. The terminal device includes the control target device. A known device database in which device information is stored, photographing means for photographing the device to be controlled, and data of a photographed image of the photographed control target device Image recognition means for recognizing and specifying the device by comparing data of the device image that is one of the device information stored in the known device database, and specifying information specified by the image recognition means. Transmitting means for transmitting to the device management server; receiving means for receiving the operation information transmitted from the device management server; and storing means for storing the operation information received by the receiving means in the known device database. Extracted by the display means for displaying the photographed image taken by the photographing means, the operation input means for instructing and inputting an arbitrary position on the display screen of the display means, and the operation input means When the operation area defined by the operation information is operated, an operation signal corresponding to the operation area is acquired from the operation information and output to the control target device. It is characterized and output means for, further comprising: a.

  The remote operation system of the present invention is a remote operation system for remotely operating a control target device by communication between a device management server and a terminal device, wherein the device management server includes device information of the control target device and Device management database storing operation information, receiving means for receiving captured image data of the control target device transmitted from the terminal device, received captured image data, and stored in the device management database Image recognition means for recognizing and specifying the device by comparing with device image data that is one of the device information, and referring to the device management database based on the device information of the specified control target device And information extracting means for extracting operation information corresponding to the device to be controlled, transmitting means for transmitting the extracted operation information to a terminal device, The terminal device includes a known device database in which device information of the control target device is stored, a photographing unit that photographs the control target device, and captured image data of the control target device captured by the photographing unit. The transmission means for transmitting to the database, the reception means for receiving the operation information transmitted from the device management server, the storage means for storing the received operation information in the known device database, and the imaging means Displayed by the operation information extracted by the display means for displaying the captured image, the operation input means for instructing and inputting an arbitrary position on the display screen of the display means, and the operation input means When an operation area is operated, an output that acquires an operation signal corresponding to the operation area from the operation information and outputs the operation signal to the control target device Is characterized by comprising: a stage, a.

  In the remote operation system of the present invention, the information extraction unit further extracts operation screen information corresponding to the operation information of the control target device, and the display unit extracts the information extracted by the information extraction unit. The operation screen information may be combined with the captured image and displayed.

  According to the present invention, the control target device displayed on the display unit is remotely operated by directly operating the control target device or the operation screen information of the control target device captured by the photographing unit and displayed on the display unit. . As a result, all the control target devices stored in the device management database can be operated with only one terminal device of the present invention. Further, since the device management database is stored in the device management server, it becomes easy to add or update data of a new control target device.

  A remote operation method of the present invention is a remote operation method for remotely operating a control target device using a terminal device provided with a photographing means, an image recognition means, an information extraction means, a display means, an operation input means, and an output means. A shooting step of shooting the control target device with the shooting unit using a device management database in which device information and operation information of the control target device are stored; and A step of recognizing and identifying the device by comparing the captured image data and the device image data, which is one of the device information stored in the device management database, and the information extracting means An extraction step of referring to the device management database based on device information of the device to be controlled and extracting operation information corresponding to the device to be controlled; A display step in which the display means combines and displays the extracted operation screen information with the data of the captured image, and an arbitrary position on the display screen of the display means is instructed by the operation input means to extract When the operation area specified by the operated information is operated, the output means acquires an operation signal corresponding to the operation area from the operation information and outputs the operation signal to the control target device; It is characterized by including.

  The remote operation method of the present invention is a remote operation method for remotely operating a control target device by communication between a device management server and a terminal device, wherein the device management server includes device information of the control target device and A device management database in which operation information is stored; a reception unit; an information extraction unit; and a transmission unit. The terminal device includes a photographing unit, a known device database, an image recognition unit, and a transmission unit. A receiving unit, a display unit, an operation input unit, and an output unit, wherein the imaging unit of the terminal device captures an image of the control target device, and the image recognition unit of the terminal device captures the image A specifying step for recognizing and specifying the device by comparing the captured image data of the device to be controlled with device image data that is one of device information stored in the known device database; A transmission step in which the transmission means of the terminal device transmits the specific information specified by the image recognition means to the device management server; a reception step in which the reception means of the device management server receives the specific information; An information extracting unit of the device management server refers to the device management database based on the received specific information and extracts operation information corresponding to the device to be controlled; and a transmitting unit of the device management server A transmitting step of transmitting the extracted operation information to the terminal device; a receiving step of receiving the operation information transmitted from the device management server by a receiving unit of the terminal device; and a storing unit of the terminal device A storage step of storing the received operation information in the known device database, and a display unit of the terminal device includes the imaging A display step for displaying the photographed image photographed at the stage, and an operation region defined by the operation information extracted by instructing an arbitrary position on the display screen of the display means by the operation input means Is operated, the output means acquires an operation signal corresponding to the operation area from the operation information stored in the known device database, and outputs the operation signal to the control target device. It is characterized by including.

  The remote operation method of the present invention is a remote operation method for remotely operating a control target device by communication between a device management server and a terminal device, wherein the device management server includes device information of the control target device and A device management database in which operation information is stored; a receiving unit; an image recognizing unit; an information extracting unit; and a transmitting unit. The terminal device includes a photographing unit, a transmitting unit, a receiving unit, and a storage unit. Means, a display means, an input operation means, and an output means, a photographing step in which the photographing means of the terminal device photographs the control target device, and the control object photographed by the transmitting means of the terminal device A transmission step of transmitting captured image data of the device to the device management server, a receiving step of receiving the captured image data transmitted by the receiving means of the device management server, and the device An identifying step in which the image recognition means of the management server recognizes and identifies the device by comparing the received captured image data with device image data that is one of device information stored in the device management database; An extraction step in which the information extraction unit of the device management server refers to the device management database based on the specified information and extracts operation information corresponding to the device to be controlled; and a transmission unit of the device management server A transmitting step of transmitting the extracted operation information to the terminal device; a receiving step of receiving the operation information transmitted from the device management server by a receiving unit of the terminal device; and a storing unit of the terminal device A storage step of storing the received operation information in the known device database, and a display means of the terminal device comprising: A display step for displaying the photographed image captured in step (2), and an operation region specified by the extracted operation information is indicated by an arbitrary position on the display screen of the display means being instructed by the operation input means. When operated, the output means includes an output step of acquiring an operation signal corresponding to the operation area from the operation information stored in the known device database and outputting the operation signal to the control target device. It is characterized by that.

  In the remote operation method of the present invention, in the extraction step, operation screen information corresponding to the operation information of the control target device is further extracted, and in the transmission step in the device management server, the extracted operation screen is extracted. Information is transmitted to the terminal device. In the receiving step of the terminal device, the operation screen information transmitted from the device management server is received, and in the storing step, the received operation screen information is received as the known information. In the output step, the operation area specified by the extracted operation information is operated by designating an arbitrary position on the display screen of the display means by the operation input means. And obtaining an operation signal corresponding to the operation area from the operation screen information stored in the known device database. It may be output to the control target device.

  According to the present invention, the control target device displayed on the display unit is remotely operated by directly operating the control target device or the operation screen information of the control target device captured by the photographing unit and displayed on the display unit. . As a result, all the control target devices stored in the device management database can be operated with only one terminal device of the present invention. Further, since the device management database is stored in the device management server, it becomes easy to add or update data of a new control target device.

  According to the present invention, the control target device displayed on the display unit can be remotely operated by directly operating the operation area of the control target device displayed on the display unit on the screen of the display unit. Thereby, since all the control target devices stored in the device management database can be operated by only one terminal device of the present invention, a remote control device for each control target device becomes unnecessary.

  Further, according to the present invention, the control target device displayed on the display means can be controlled by directly operating the operation area such as various operation function keys of the control target device displayed on the display means of the terminal device on the screen. Remote operation can be performed more intuitively (that is, as if an operation function key provided on an actual remote control device or device main body is directly operated).

It is a schematic block diagram of the terminal device which concerns on Embodiment 1 of this invention. It is explanatory drawing which shows the structural example of an apparatus management database. (A) is explanatory drawing which shows the structural example of the apparatus image database in the apparatus management database shown in FIG. 2, (b) is explanatory drawing which shows the structural example of the remote control image database in the apparatus management database shown in FIG. FIG. 4C is an explanatory diagram showing a configuration example of command definition data in the device management database shown in FIG. It is explanatory drawing which shows the other structural example of an apparatus management database. (A) is explanatory drawing which shows the structural example of the operation panel image database in the apparatus management database shown in FIG. 4, (b) shows the structural example of the command definition data in the machine management database shown in FIG. It is explanatory drawing. It is explanatory drawing which shows the further another structural example of an apparatus management database. (A) is explanatory drawing which shows the structural example of the apparatus image database in the apparatus management database shown in FIG. 6, (b) is explanatory drawing which shows the structural example of the mask image database in the apparatus management database shown in FIG. FIG. 6C is an explanatory diagram showing a configuration example of a guide image database in the device management database shown in FIG. It is explanatory drawing which shows the structural example of the command definition data in the apparatus management database shown in FIG. It is a flowchart which shows the process of the specific example 1 which operates a control object apparatus remotely. (A)-(d) is explanatory drawing of the screen display which concerns on the specific example 1. FIG. It is a flowchart which shows the process of the specific example 2 which operates a control object apparatus remotely. (A)-(e) is explanatory drawing of the screen display which concerns on the specific example 2. FIG. It is a flowchart which shows the process of the specific example 3 which operates a control object apparatus remotely. It is a display explanatory drawing of the screen which concerns on the specific example 3. FIG. It is a flowchart which shows the process of the specific example 4 which operates a control object apparatus remotely. It is a flowchart which shows the process of the specific example 4 which operates a control object apparatus remotely. (A), (b) is explanatory drawing of the screen display which concerns on the specific example 4. FIG. It is a schematic block diagram of the remote control system which concerns on Embodiment 2 of this invention. (A), (b) is explanatory drawing which shows the structural example of a known apparatus database. It is a flowchart which shows the process of the specific example 5 which operates a control object apparatus remotely. It is a flowchart which shows the process of the specific example 5 which operates a control object apparatus remotely. It is a schematic block diagram of the remote control system which concerns on Embodiment 3 of this invention. It is a flowchart which shows the process of the specific example 6 which operates a control object apparatus remotely. It is a flowchart which shows the process of the specific example 6 which operates a control object apparatus remotely.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiment is an example embodying the present invention, and is not intended to limit the technical scope of the present invention.

<Embodiment 1>
FIG. 1 shows a schematic configuration diagram of a terminal apparatus 1 according to Embodiment 1 of the present invention.

  The terminal device 1 according to Embodiment 1 is a terminal device that can remotely control various devices to be controlled with one device, and includes a device management database 11, an operation input unit 12, a display unit 13, and a photographing unit. 14, an image recognition unit 15, an information extraction unit 16, an image composition unit 17, and a control command transmission unit 18, and the image recognition unit 15, the information extraction unit 16, and the image composition unit 17 are configured by a control device 19. .

  Although not shown, the control device 19 includes a processing unit, a ROM (Read Only Memory), and a RAM (Random Access Memory). The processing unit includes a CPU (Central Processing Unit), executes a program stored in the ROM, controls each unit of the terminal device 1, and performs data processing. The ROM stores programs executed by the processing unit and data. The RAM is used as a program execution processing area, a temporary storage area for image data, and various work areas.

  The device management database 11 stores device information of various control target devices, operation screen information, and operation information corresponding to various operations of the operation screen information in association with each control target device.

  FIG. 2 shows a configuration example of the device management database 11. However, the device management database 11A shown in FIG. 2 is a configuration example corresponding to a remote control image when the control target device has a remote control device.

  The device management database 11A includes a device image item column 11a which is device information, a manufacturer name item column 11b, a model name item column 11c, a model number item column 11d, and a remote control image item column 11e which is operation screen information. It consists of an item column 11f of command definition data which is operation information.

  In the device image item column 11a, for example, as shown in FIG. 3A, an appearance image viewed from the front side of the device is stored. This device image is image data used for collation with a device image photographed by the photographing unit 14 described later, and has a unique internal coordinate system (Xi, Yi). As the image data, in addition to a method of holding the image data of the target device as a bitmap, data such as contour shape and dimensions, and feature information such as parameters after Fourier transform for each block may be held. Also, a three-dimensional three-dimensional image may be used instead of the two-dimensional image as shown in FIG.

  In FIG. 3A, the television images A and B and the BD (or DVD) recorder C are illustrated as device images. However, the device images are not limited to these models. Device images of various devices (home appliances) that can be controlled by a remote control device such as a component, an air conditioner, a fan, and a lighting device are stored. Accordingly, the data corresponding to these device images is also included in each item column of the maker name item column 11b, the model name item column 11c, the model number item column 11d, the remote control image item column 11e, and the command definition data item column 11f. Is stored.

  The manufacturer name item column 11b stores the name of the manufacturer of the device.

  The model name item column 11c stores the model name (for example, name or symbol) of the device.

  The model number field 11d stores the model number of the device.

  In the remote control image item column 11e, for example, as shown in FIG. 3B, an image (remote control image) of a dedicated remote control device which is operation screen information of the device is stored.

  The command definition data item column 11f stores operation information corresponding to various operation function keys (hereinafter also referred to as operation buttons) of the remote control device. That is, the relative position of each operation button is mapped to the remote control image, and the function equivalent to each button operation is pre-programmed corresponding to this position, and stored as command definition data in the item column 11f. .

  FIG. 3C shows a configuration example of command definition data stored in the item column 11f.

  Each line of the item column 11f stores command definition data of the corresponding remote controller. The command definition data of the microcomputer device stores control commands (operation signals: the same below) corresponding to each operation button, button positions, and input operation possible ranges of the buttons (operation areas: the same below) in association with each other. In this example, the button position of each operation button is defined by the center coordinates (Xb, Yb) of each button with the origin at the upper left of the remote control image and its effective operation radius r, and this defined button position is controlled. Associated with a command.

  More specifically, referring to FIG. 3C, for example, in the command definition data AAA of the remote control device AA, the power button on the first line corresponds to the button position (Xb = 12, Yb = 15) and the effective operation radius r (= 8) are associated with each other, and the same applies to the buttons in the second and subsequent lines. The command definition data (AAB, CCC) of other remote control devices AB, CC, etc. have the same configuration according to the respective devices.

  FIG. 4 shows another configuration example of the device management database 11. However, the device management database 11B illustrated in FIG. 4 is a configuration example corresponding to the operation panel image when the operation panel is in front of the control target device such as a stereo component. However, in this example, a music recorder or the like that can be controlled by an infrared remote control device is originally assumed.

  The device management database 11B includes a device image item column 11a which is device information, a manufacturer name item column 11b, a model name item column 11c, a model number item column 11d, and an operation panel image item column 11e which is operation screen information. ', It is composed of an item column 11f' for command definition data. Among these, the item columns 11a to 11d have the same configuration as the above-described device management database 11A, and thus description thereof is omitted.

  In the operation panel image item column 11e ′, for example, as shown in FIG. 5A, an operation panel image which is operation screen information of the device is stored.

  Operation information corresponding to various operation buttons on the operation panel is stored in the item column 11f ′ of the command definition data. That is, the relative position of each operation button is mapped to the operation panel image, and the function equivalent to each button operation is pre-programmed corresponding to this position, and stored in the item column 11f 'as command definition data. ing.

  FIG. 5B shows a configuration example of the command definition data stored in the item column 11f ′.

  Each row of the item column 11f ′ stores command definition data for various operation buttons of the corresponding operation panel. This command definition data stores a control command corresponding to each operation button, a button position, and an input operation possible range of the button in association with each other. In this example, the button position of each operation button is stored in the operation panel image. It is defined as a rectangular range consisting of the center coordinates (Xb, Yb) of each button with the upper left as the origin and effective distances in the respective directions (two-dimensional directions X, Y) from the center coordinates. It is associated. The item column 11f ′ is provided with sub-columns 11f′1 to 11f′3 such as a state (precondition) after operation, a state after operation, a display comment, and the like, and operation buttons. It is also possible to store balloon screen coordinates (X, Y) in the case where the operable range that substitutes for the function name and button is displayed as a balloon (balloon display). The balloon screen coordinates (X, Y) can also be set by the center coordinates (Xb, Yb) of the balloon and the effective distance in each direction (two-dimensional directions X, Y) from the center coordinates. Each balloon screen coordinate may be set in advance as a fixed value, or may be set dynamically according to the display state of the screen. For example, the balloons may be adjusted and set so that the display positions of the balloons and buttons on the screen do not overlap.

  Specifically, referring to FIG. 5B, for example, in the command definition data BBB of the operation panel BB, the operation button on the first line is the recording button described in the display comment field, and the recording button operation is valid. STOP is a precondition for the recording button control command (= B1) to be associated with internal coordinates (Xb = 28 ± 3, Yb = 40 ± 15) as the button position, After the operation, the transition to the REC state is set. The same applies to the second and subsequent lines. Balloon coordinates are also defined for the pause button. In this example, screen coordinates (Xb = 100 ± 50, Yb = 150 ± 30), which are balloon positions, are associated with each other. “Pause” and “Resume” are buttons that perform different displays and functions depending on the state at the same position. It is set to have a pause function when the precondition is PLAY and a resume function when it is PAUSE. doing.

  FIG. 6 shows still another configuration example of the device management database 11. The device management database 11C illustrated in FIG. 6 is a device management database corresponding to the electric fan. The fan described here is based on the premise that the neck can be moved up and down, left and right by one unit angle (for example, one unit = 5 degrees) by button operation of a dedicated remote controller. However, FIG. 6 shows only a configuration example of the device management database corresponding to the operation of changing the angle of the neck, “Power ON / OFF” arranged in front of the device, “Low”, “Medium” of the air volume. The operation buttons such as “strong” can have the same configuration as that of the device management database 11B shown in FIG. 4, and thus the description of the command definition data corresponding to these operation buttons is omitted here.

  The device management database 11C includes a device image item column 11a, a manufacturer name item column 11b, a model name item column 11c, a model number item column 11d, and a mask image item column 11g indicating an operable start area. , A guide image item column 11h representing an annotation such as an operable start region and an operable direction, and a command definition data item column 11f ″.

  In the device image item column 11a, as shown in FIG. 7A, an appearance image viewed from the front or slightly above the device is stored. FIG. 7A is an appearance image with the height of the viewpoint as the center of the blade and separated from the front to such an extent that the pedestal switch can be seen. It is relatively difficult to collate images with a device with such a three-dimensional structure with irregularities on its appearance. For example, a device may be given multiple marks or barcodes with specific colors or shapes as feature points. Thus, collation can be facilitated.

  As shown in FIG. 7B, the mask image item field 11g indicating the operable start area has the same internal coordinate system as the device image that is the image data used for collation. Is filled with “1” (indicated by a white area in FIG. 7B), and pixels in the non-operational area are filled with “0” (indicated by a dotted area in FIG. 7B). The pixel map information is retained. This area is a material for determining whether an effective flick operation described later has been performed.

  As shown in FIG. 7C, the item column 11h of the guide image representing annotations such as the operable start area and the operable direction has the same internal coordinate system as the device image that is the image data used for collation. Annotation such as an operation startable area and an operable direction is held as vector data. The guide image is converted into a coordinate system on the screen, and is displayed so as to be superimposed on an image of the target device so that the user can easily understand.

  The command definition data item column 11f ″ stores a control signal that is operation information corresponding to the swing direction of the electric fan. That is, each operation area (fan) in the swing direction is included in the operable area of the mask image. The relative positions of the four operation buttons on the top, bottom, left, and right of the main body are mapped, and the flick operation that is the operation method of each area is associated with this position (in a predetermined direction while touching) The function (control signal output) equivalent to the action of lightly flicking a finger is programmed in advance and stored as a control command in the item column 11f ″ of the command definition data.

  FIG. 8 shows an example of the configuration of command definition data (in this example, command definition data HHH) stored in the item column 11f ″.

  Each line of the item column 11f ″ stores command definition data of each operation area in the corresponding operation direction. This command definition data is controlled by a control command corresponding to each operation area and an operation of the operation area. The flick direction column defining the control direction to be associated with the pixel pitch / command defining the pixel width of the movement amount on the internal coordinates of the flick operation required to output the command once. , “/ Command” means “per command”, that is, “per command”.

  More specifically, referring to FIG. 8, for example, in the command definition data HHH, the operation area in the first row is a lower operation area, and the control command (= H1) for the lower operation area is used. The “+ Y direction” that is the flick direction is associated with “10” that is the above-described pixel pitch / command, and the same applies to the second and subsequent lines.

  The remote operation method using the device management databases 11A to 11C will be described later with specific examples.

  Returning to FIG. 1, the operation input unit 12 includes operation buttons provided on the apparatus main body, a touch panel (touch pad) arranged on the display unit 13, and the like. In the first embodiment, a multi-touch screen that can recognize that a plurality of fingers have touched is used as the touch panel.

  The display unit 13 can display an augmented reality image for displaying various information in the virtual space by combining it with an image in the real space, and is configured using display components such as a liquid crystal panel, an organic EL panel, and a plasma panel. ing.

  The imaging unit 14 is a camera 14a using, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor as an imaging device. The imaging unit 14 images a device to be controlled in a real space and uses the captured image data as A. The A / D converter 14b performs A / D conversion and outputs the result to the image recognition unit 15. As the camera 14a, a general-purpose imaging device such as a Web camera or an HD (High-Definition) resolution shooting camera can be used.

  The image recognizing unit 15 captures data of a captured image of the control target device captured by the capturing unit 14 and device image data (one of the device information stored in the device management database 11). Data or feature data) to identify and identify the device.

  The information extraction unit 16 searches (refers to) the device management database 11 using the device information (the above-described model name and the like) of the control target device specified by the image recognition unit 15 as a key, and the corresponding operation screen information (the above-described remote controller). Image, operation panel image, etc.) and operation information (the above-mentioned command definition data) are extracted.

  The image composition unit 17 composes the operation screen information extracted by the information extraction unit 16 (such as the above-described remote control image and operation panel image) with the first photographed image and displays it on the display unit 13.

  When any operation unit (operation button) of the operation screen information (the above-described remote control image or operation panel image) displayed on the display unit 13 is operated, the control command transmission unit 18 operates to extract the extracted operation information (the above-described operation information). From the command definition data), an operation signal (the above-described control command) corresponding to the operated operation unit is acquired and output to the control target device.

  According to the present invention, the control target device displayed on the display unit can be remotely operated by directly operating the operation screen information of the control target device captured by the photographing unit and displayed on the display unit. Thereby, since all the control target devices stored in the device management database can be operated by only one terminal device of the present invention, a remote control device for each control target device becomes unnecessary.

<Specific example 1 of remote control>
FIG. 9 shows a specific example 1 in which the device to be controlled is remotely operated using the device management database 11A shown in FIGS. 2 and 3A to 3C in the terminal device 1 according to the first embodiment. This will be described with reference to a flowchart and explanatory diagrams of screen display shown in FIGS.

  Here, when a user views a television, for example, at home, the remote operation is performed using a portable terminal device such as a smartphone, which is the terminal device 1 of the present invention, instead of a remote control device dedicated to the television. explain.

  At this time, the user uses the imaging unit 14 of the terminal device 1 to image the television that is the device to be controlled, clears the display screen of the terminal device 1, and displays the captured image (step S1). FIG. 10A shows the display state of the captured device image. Here, the displayed image may be a real-time moving image such as a camera function preview screen, or may be a still image that has been taken once.

  The image recognizing unit 15 captures data of a captured image captured by the camera 14a and A / D converted by the A / D conversion unit 14b, and a device image that is one of the device information stored in the device management database 11A. The device is recognized by comparing (collating) with the data (step S2), and it is confirmed whether or not the device has been successfully identified (step S3). In the first embodiment, the device is identified by recognizing the shape. In the case of a television, in addition to the color and width of the frame and the shape of the stand, the manufacturer name, model name, and model number may be used for recognition. The shape is collated with the feature data of the device image stored in the device management database 11A. As a matching method, conventionally known image matching techniques such as various correlation functions in images, Fourier transform, and feature point extraction can be applied.

  If the device is identified successfully (Yes in step S3), the information extraction unit 16 uses the device name of the device to be controlled identified by the image recognition unit 15 as a key, such as a model name or a model number. The management database 11A is searched, and a remote control image (for example, AA) that is corresponding operation screen information and command definition data (for example, AAA) that is operation information are read from the device management database 11A (step S4).

  The image composition unit 17 converts the remote control image (AA) read by the information extraction unit 16 into a screen coordinate system, composes it with the first photographed image, and displays it on the display unit 13 (step S5). The composition may be displayed on the display unit 13 by using, for example, a well-known video see-through method using augmented reality (AR) technology.

  FIG. 10B shows a state in which a remote control image is combined with a captured image and displayed.

  As a result, the terminal device 1 enters a state of waiting for an input event by operating the remote control image displayed on the screen (step S6), and then whether or not an effective command operation (such as an effective operation of the operation button) has been performed. Is monitored (step S7).

  At this time, in the display state shown in FIG. 10B, for example, a remote control image is moved on the screen by dragging (sliding a finger while touching) on the screen, or two fingers are moved on the screen. The remote control image is reduced or enlarged by pinching in (an operation for reducing the interval between two fingers on the screen) or by pinching out (an operation for increasing the interval between two fingers on the screen) (see FIG. 10C). Then, the operation button may be displayed (highlighted) at a position where it can be easily pressed. Further, by touching a location other than the operation buttons of the displayed remote control image, only the operation buttons of the remote control device are displayed side by side (highlighted) as shown in FIG. 10D. It may be. By highlighting and displaying the operation buttons in this way, the position of the operation button can be easily recognized, and the operability is improved.

  Further, in a remote control device having a structure in which an operation button comes out by opening or sliding the lid, a button for opening the lid may be displayed, or the operation button may be displayed by opening the lid by a flick operation.

  This remote control image is synthesized and displayed while the television, which is a captured image, is displayed on the display unit 13.

  Here, as a method of operating the remote control device displayed on the screen, the operation buttons on the remote control image may be touched with a finger as they are, or the operation buttons rearranged and displayed on the screen may be touched with a finger. .

  Whether or not the operation button has been touched is determined as follows.

When the user touches, for example, the power button of the remote control image, the center coordinates of the power button are (Xb = 12, Yb = 15) and the effective operation radius is r = 8 from FIG. Therefore, the screen coordinates (Xt, Yt) of the touched part are converted (primary conversion) into coordinates (Xc, Yc) on the remote control image, and r1 = √ ((Xc−Xb) ² + (Yc−Yb) ^2. ) And when the calculated value r1 is equal to or smaller than the effective operation radius r = 8 (r ≧ r1), the power button (that is, the operation area of the power button) is touched (pressed) ).

  When an arbitrary operation button is touched in this way (when it is determined Yes in step S7), the control command transmission unit 18 sends the control command of the operated operation button to the television that is the control target device. Output (step S8). More specifically, the image of the device image A (see FIG. 3A) and the remote control image AA (see FIG. 3B) of the device management database 11A shown in FIG. 2 are displayed on the display unit 13 of the terminal device 1, for example. When the “channel +” button in the remote control image AA is operated, for example, the control command transmission unit 18 uses the command definition data corresponding to the remote control image AA shown in FIG. The control command “1A” of the “channel +” button is acquired from the AAA, and is transmitted to the television A that is the control target device. Here, the control command “1A” is transmitted by a method (infrared, wireless LAN, Bluetooth (registered trademark), etc.) supported by the TV A that is the device to be controlled.

  Thereafter, if the device image displayed on the display unit 13 is a still image (if determined No in step S9), the process returns to step S6 to return to an input event reception waiting state.

  On the other hand, when the device image displayed on the display unit 13 is a real-time moving image such as a camera function preview screen (when determined Yes in step S9), the process returns to step S1 and the processing is continued. To do. In addition, when the device image displayed on the display unit 13 is a moving image, when the television as the control target device is out of the shooting range, device identification is not successful in step S3 (determined as No). Thus, the process returns to step S1, and as a result, the remote control image is not displayed and only the captured image is displayed.

  According to the first specific example, the remote control device displayed on the display unit 13 is directly operated by directly operating the operation button of the remote control image displayed on the display unit 13 of the terminal device 1 or the operation button displayed after being rearranged. It can be operated remotely as if you were operating directly. When the control command is transmitted by the infrared method, the camera 14a is arranged in the same direction on the device housing (not shown) of the terminal device 1 by arranging the photographing direction of the camera 14a and the transmission direction of the control command transmitter 18 in the camera 14a. The control command transmission unit 18 can be directed to the control target device when the captured control target device is displayed on the display unit, and a directional infrared signal can be transmitted to the control target device. The same applies to the description of specific examples 2 to 5 later.

  In the first specific example, the remote control image on the display screen is not displayed when the device to be controlled is out of the shooting range. However, when the remote control image is touched during display, the remote control image is continuously displayed. Also good. The same applies to the following specific examples 2 to 6.

<Specific example 2 of remote control>
FIG. 11 shows a specific example 2 in which the device to be controlled is remotely operated using the device management database 11A shown in FIG. 2 and FIGS. 3A to 3C in the terminal device 1 according to the first embodiment. This will be described with reference to a flowchart and explanatory diagrams of screen display shown in FIGS. However, in the flowchart shown in FIG. 11, the same step numbers are assigned to the same steps as those in the flowchart shown in FIG. 9, and each step will be described as necessary.

  Here, when a user plays a program recorded on, for example, a BD (or DVD) recorder at home or the like and views it on a television, the terminal device 1 of the present invention is used instead of the remote controller dedicated to the recorder or the television. A case will be described in which both devices are remotely operated using one portable terminal device such as a smartphone.

  In step S1, two TVs and BD recorders, which are devices to be controlled, are captured on one screen. FIG. 12A shows an example of a state in which the captured device image is displayed on the display unit 13.

  Accordingly, the processing in steps S2 to S4 is repeated until it is determined Yes in step S11 (that is, processing is performed for both the television and the BD recorder). Here, the following description will be made assuming that the remote control image (AA) and command definition data (AAA) of the television A and the remote control image (CC) and command definition data (CCC) of the BD recorder C have been read.

  As a result, in the composite display in step S5, the remote control images of the television A and the BD recorder C are displayed on the display unit 13 in the state shown in FIG.

  As a result, the terminal device 1 enters a state of waiting for an input event by operating the remote control image displayed on the screen (step S7), and then whether or not an effective command operation (such as an effective operation of the operation button) has been performed. Is monitored (step S8).

  At this time, in the display state shown in FIG. 12B, for example, when both remote controllers are touched simultaneously, the two remote controllers may be displayed side by side and enlarged as shown in FIG.

  In the display state shown in FIG. 12B, when the remote controller image AA for TV is touched, only the remote controller image AA for TV is enlarged and displayed (highlighted) at the center of the screen as shown in FIG. When the remote controller image CC for BD recorder is touched, only the remote controller image CC for BD recorder may be enlarged and displayed (highlighted) at the center of the screen as shown in FIG. Further, when the enlarged display is performed, a return button may be displayed on the screen, and the return to the original display may be performed by touching the return button.

  Further, for example, in the state shown in FIG. 12C in which the remote control image AA is enlarged and displayed in the center portion, a flick operation is performed on the screen, for example, in the left direction, so that the page as shown in FIG. The other remote control image CC is displayed in the center of the screen so as to turn leftward, and in the state shown in FIG. 12 (e), for example, by flicking on the screen in the right direction, FIG. As shown, the original remote control image AA may be displayed in the center of the screen by turning the page to the right.

  If, for example, a “channel +” button (that is, an operation area of the “channel +” button) in the remote control image AA is operated in an input event reception waiting state (when it is determined Yes in step S7), control is performed. The command transmission unit 18 acquires the control command “1A” for the “channel +” button from the command definition data AAA corresponding to the remote control image AA shown in FIG. 3C (see FIG. 3C). This is transmitted to the television A which is a control target device (step S8).

  Similarly, if, for example, a “play” button in the remote control image CC is operated (Yes in step S7), the control command transmission unit 18 corresponds to the remote control image CC shown in FIG. The control command “1E” of the “playback” button is acquired from the command definition data CCC (see FIG. 3C), and is transmitted to the BD recorder C that is the control target device.

  According to the second specific example, by directly operating the operation buttons of the plurality of remote control images displayed on the display unit 13 of the terminal device 1, the feeling that the remote control device displayed on the display unit 13 is directly operated. Can be operated remotely. In addition, a plurality of remote control devices can be individually remotely operated on the display screen of one terminal device 1.

<Specific example 3 of remote control>
In the terminal device 1 according to the first embodiment, the flowchart shown in FIG. 13 and FIG. 14 show a specific example 3 in which the device to be controlled is remotely operated using the device management database 11B shown in FIGS. 4 and 5B. This will be described with reference to the display explanatory diagram shown in FIG. However, in the flowchart shown in FIG. 13, the same step numbers are assigned to the same steps as those in the flowchart shown in FIG. 9, and each step will be described as necessary.

  Here, when a user listens to music by operating, for example, a stereo component at home or the like, the terminal device 1 of the present invention is not a remote operation by a remote controller dedicated to the component, but a portable terminal such as a smartphone. A case where remote control is performed using the apparatus will be described.

  The user captures a stereo component that is a device to be controlled using the capturing unit 14 of the terminal device 1, clears the display screen of the terminal device 1, and displays the captured image (step S1). The captured device image is displayed on the display unit 13.

  The image recognition unit 15 compares and recognizes the captured image data and the device image data that is one of the device information stored in the device management database 11B (step S2), and recognizes the captured image data. The conversion parameters for converting the screen coordinate system (Xo, Yo) into the original internal coordinate system (Xi, Yi) stored in the command definition data item field 11f ′ of the device management database 11B are extracted. The device is specified (step S21). If the device is identified successfully (Yes in step S3), the information extraction unit 16 searches the device management database 11B and operates the operation panel image (for example, BB) corresponding to the operation screen information. And command definition data (for example, BBB) is read (step S4).

  As a result, in the combining process in step S5, the operation panel image is displayed on the display unit 13 in the state shown in FIG. Since the operation panel is included in the captured image data, it is not always necessary to synthesize and display the operation panel image when the operation panel in the captured image is sufficiently clear.

  Thereafter, the terminal device 1 enters a state of waiting for an input event by operating the operation panel image displayed on the screen (step S6). After that, if there is an input operation, the coordinate information included in the input event is converted by the conversion parameter. By converting to internal coordinates and determining whether the converted internal coordinates fall within the area defined by the button position internal coordinates in the item field 11f ′ of the command definition data, an effective command operation (operation button It is monitored whether or not an effective operation has been performed (step S7).

  The example shown in FIG. 14 shows a case where the “state” of the stereo component that is the device to be controlled is “PLAY”. In the command definition data shown in FIG. Only the operation buttons (that is, the stop, fast forward, rewind, and pause operation buttons) are valid, and a guide image (a rectangular shape surrounded by a broken line in FIG. 14) is defined on the valid operation buttons. Frame) f1-f4 and display comments (stop, fast forward, rewind) are displayed. The pause button 244 is provided with a balloon (balloon) b1 and displays a display comment (pause) there.

  In other words, pressing the PLAY or REC operation buttons during PLAY is prohibited by providing the frames f1 to f4 and the balloon b1. The operation buttons that are prohibited may be distinguished from the operation buttons that are effective by display in a different color such as gray or shaded display in addition to the control of simply not displaying.

  Here, rectangular frames f1 to f4 which are guide images are displayed by converting “button position internal coordinates (X, Y)” in the command definition data shown in FIG. 5B to screen coordinates. is there. For example, for the stop button 241, the button position internal coordinates are defined as (46 ± 4, 40 ± 15) by the command definition data, which is ± 4 pixels in the horizontal direction centering on the internal coordinates (46, 40). The inside of a rectangle of ± 15 pixels in the vertical direction is an effective operation area. In the example shown in FIG. 14, there is no margin in the horizontal direction but there is margin in the vertical direction. Therefore, the effective operation area is lengthened in the vertical direction to increase the effective operation area of the button, and the button operation on the display screen is performed. Making it easy.

  The pause button 244 has a balloon screen coordinate (100 ± 50, 150 ± 30) defined by the command definition data, which is ± 50 in the horizontal direction around the screen coordinate (100, 150). The inside of a rectangular balloon b1 having a pixel and ± 30 pixels in the vertical direction is an effective operation area. By displaying this balloon b1, button operation on the display screen is facilitated.

  Here, as an operation method of the operation panel displayed on the screen, an operation button on the operation panel is directly touched with a finger.

  If, for example, the “stop” button 241 in the operation panel image BB is operated in the standby state for the input event (determined as “Yes” in step S7), the control command transmission unit 18 shown in FIG. The control command “B3” of the “stop” button is acquired from the command definition data BBB corresponding to the operation panel image BB, and is transmitted to the stereo component B that is the control target device (step S8).

  According to the third specific example, by directly operating the operation button of the operation panel image displayed on the display unit 13 of the terminal device 1, the operation panel displayed on the display unit 13 is directly operated. Can be operated remotely.

<Specific example 4 of remote control>
Specific example 4 in the terminal device 1 according to the first embodiment described above that remotely controls the electric fan that is the device to be controlled using the device management database 11C illustrated in FIGS. 6 and 8 (that is, changes the neck angle). This will be described with reference to the flowcharts shown in FIGS. 15A and 15B and the display explanatory views shown in FIGS.

  When a user operates a fan at home or the like, it is not remotely operated with a dedicated remote control device, but is remotely operated using a portable terminal device such as a smartphone, which is the terminal device 1 of the present invention. explain. However, since the control of the fan on / off, the adjustment of the air flow, etc. is the same as in the above-mentioned specific examples 1 to 3, etc., explanation of these button operations is omitted here, and in specific example 4, the angle of the neck of the fan is set. Only the changing operation will be described.

  The user uses the imaging unit 14 of the terminal device 1 to image the electric fan that is the device to be controlled, clears the display screen of the terminal device 1, and displays the captured image (step S61). The captured device image is displayed on the display unit 13 as shown in FIG. Here, the displayed image may be a real-time moving image such as a camera function preview screen, or may be a still image that has been taken once.

  The image recognition unit 15 compares (recognizes) and recognizes the captured image data and the device image data, which is one piece of device information stored in the device management database 11C (step S62). The parameters for converting the screen coordinate system (Xo, Yo) into the original internal coordinate system (Xi, Yi) stored in the item column 11f ″ of the command definition data in the device management database 11C are extracted. The device is identified (step S63), and it is confirmed whether the device has been successfully identified (step S64), and if the device is identified successfully (Yes in step S64), image recognition is performed. The unit 15 collates the captured image data with the feature data in the device image item column 11a of the device management database 11C, extracts a plurality of coordinates of the corresponding feature points, and Calculation of parameters for converting the screen coordinate system (Xo, Yo) of the image data into a unique internal coordinate system (Xi, Yi) stored in the command definition data item field 11f ″ of the device management database 11C (Step S65).

  The information extraction unit 16 searches the device management database 11C using, for example, the model name or model number, which is device information of the control target device specified by the image recognition unit 15, as a key, and a mask image (for example, corresponding operation screen information). , HH) and the guide image (for example, HH1), and the control command HHH are extracted (step S66).

  The image composition unit 17 converts the data of the guide image HH1 extracted by the information extraction unit 16 into a screen coordinate system using the parameters, and superimposes the captured image on the screen of the display unit 13 (step S67). . FIG. 16B shows the display screen of the display unit 13 at this time. A broken-line circle indicated by reference numeral 311 in FIG. 16B and the vertical and horizontal arrows 312 displayed in the circle 311 correspond to the guide image. The guide image is displayed in red, for example, so that the user can easily see it.

  The guide image is displayed while being superposed while the electric fan is recognized and specified in the captured image and displayed on the display unit 13.

  After that, the terminal device 1 enters an input event waiting state by operating the fan image displayed on the screen (step S68), and then monitors whether or not a flick operation has been performed (step S69).

  Here, when the flick operation is performed (Yes in step S69), the control device 19 uses the screen coordinates (Xos, Yos) of the start point of the flick input as the internal coordinate system (Xis, Yis) (step S70), the coordinate (Xis, Yis) after the conversion is compared with the data of the mask image HH stored in the device management database 11C (step S71), and the coordinate (Xis, Yis) is It is determined whether or not it is an operation startable region (operation region), that is, whether or not the position of the coordinates (Xis, Yis) is inside the mask image HH (inside the circle not masked in FIG. 7B). (Step S72).

  As a result, when the coordinate (Xis, Yis) is an operation startable area (when determined Yes in step S72), the flick input start point (Xos, Yos) on the display screen is changed to the end point (Xoe). , Yoe) (Xov = Xoe-Xos, Yov = Yoe-Yos) is converted into a vector (Xiv, Yiv) in the internal coordinate system (step S73), and the sign of each component of the vector (Xiv, Yiv) Based on (positive / negative), the control command and the pixel pitch / command are obtained by referring to the command definition data item field 11f ″ of the device management database 11C.

  For example, if the vector (Xiv, Yiv) of the internal coordinate system is (Xiv, Yiv) = (40, −20), Xiv> 0 (that is, the flick direction is the + X direction). The command H3 and the pixel pitch / command 10 are obtained. Since Yiv <0 (that is, the flick direction is the -Y direction), the corresponding control command H2 and pixel pitch / command 10 are obtained (step S74).

  As a result, in this example, the control command H3 for the X axis is four times (= Xiv / pixel pitch = 40/10), and the control command H2 for the Y axis is twice (= Yiv / pixel pitch = 20/10), respectively. It transmits toward the electric fan which is a control object apparatus (step S75).

  In response to this, the electric fan rotates the direction of the neck by 4 units (20 degrees) in the right direction and 2 units (10 degrees) in the upward direction from the current neck direction.

  In other words, by increasing the length of the flick operation on the screen, the swing angle (rotation angle) of the neck that can be controlled by one operation can be increased, and the flick operation on the screen can be performed. By shortening the length of playing on the screen, the swing angle (rotation angle) of the neck that can be controlled by one flick operation can be reduced.

  Thereafter, if the device image displayed on the display unit 13 is a still image (if determined No in step S76), the process returns to step S68 to return to an input event reception waiting state.

  On the other hand, when the device image displayed on the display unit 13 is a real-time moving image such as a preview screen (when it is determined Yes in step S76), the process returns to step S61 to continue the processing.

  According to the fourth specific example, the direction of the neck of the electric fan displayed on the display unit 13 can be directly flicked with the finger within the operable start area of the guide image displayed on the display unit 13 of the terminal device 1. The angle can be remotely controlled.

  In Specific Example 4, a fan is exemplified as the control target device, and control of the swing angle by a flick operation is illustrated as the control content, but the control target device and the control content are not limited to these. That is, an operation startable area is displayed on the screen according to various control target devices (or simply set on the screen without displaying it), and the displayed (or set) operable area is displayed as a flick operation or It is also a concept included in the present invention to remotely control a device to be controlled by operating by various operation methods including a slide operation or the like.

<Embodiment 2>
FIG. 17 shows a schematic configuration diagram of a remote control system according to Embodiment 2 of the present invention.

  The remote operation system according to the second embodiment is a system for remotely operating a device to be controlled by the device management server 50 and the terminal device 60 communicating with each other through a network N.

  Therefore, in the remote operation system 2 according to the second embodiment, the device management server 50 includes the database management unit 51, the information extraction unit 56, and the database side communication unit 52 that communicates with the terminal device 60. The terminal device 60 includes a known device database 61, an operation input unit 62, a display unit 63, an imaging unit 64 including a camera 64a and an A / D conversion unit 64b, an image recognition unit 65, an image composition unit 67, and a control command transmission unit. 68, an information storage unit 70, and a device-side communication unit 66 that communicates with the device management server 50. An image recognition unit 65, an image composition unit 67, an information storage unit 70, and a known device database 61 are included in the control device 69.

  Here, the image recognition unit 65 corresponds to the image recognition unit 15 in FIG. 1, the information extraction unit 56 corresponds to the information extraction unit 16 in FIG. 1, and the operation input unit 62 corresponds to the operation input unit 12 in FIG. The display unit 63 corresponds to the display unit 13 in FIG. 1, the imaging unit 64 corresponds to the imaging unit 14 in FIG. 1, the image synthesis unit 67 corresponds to the image synthesis unit 17 in FIG. 1, and the control command transmission unit 68. Corresponds to the control command transmitter 18 in FIG. 1, and the control device 69 corresponds to the control device 19 in FIG.

  In addition, the information storage unit 70 performs a process of storing the information transmitted from the device management server 50 in the known device database 61.

  FIG. 18A shows a configuration example of the known device database 61. However, the known device database 61 shown in FIG. 18A corresponds to the device management database 11A shown in FIG. 2, and is a configuration example corresponding to a remote control image when the control target device has a remote control device.

  The known device database 61 includes a device image item column 11 a 1 as device information, a remote control image item column 11 storing operation screen information received from the device management server 50, and operation information received from the device management server 50. It consists of an item column 11f for command definition data to be stored. That is, the item field 11e1 of the remote control image and the item field 11f1 of the command definition data are initially blank, and each time the operation screen information and the operation information of the control target device are received from the device management server 50 side, The operation screen information and the operation information are sequentially stored (accumulated) in the device image item column 11a1 of the device to be controlled. The terminal device 60 controls the operation of the control target device based on the operation screen information and the operation information stored in the known device database 61.

  According to the remote operation system 2 of the present invention, the control target device displayed on the display unit 63 by directly operating the operation screen information of the control target device captured by the imaging unit 64 and displayed on the display unit 63. Can be operated remotely. As a result, since all the control target devices stored in the device management database 51 can be operated by only one terminal device of the present invention, a remote control device for each control target device becomes unnecessary. In addition, since the device management database 51 is stored in the device management server 50, it is possible to easily add or update data of a new device to be controlled. Furthermore, since it is not necessary for the terminal device 60 to have all the data of the control target device, the burden on the terminal device 60 is reduced. Furthermore, since the data of the operation target device stored on the terminal device 60 side does not need to be acquired again from the device management server 50, the operation of the operation target device can be controlled by processing only on the terminal device 60 side.

  Here, the configuration example in which the terminal device 60 has the known device database 61 is shown. However, in order to further reduce the load on the terminal device 60, the operation received from the device management server 50 without having the known device database 61 in the terminal device 60. You may make it control operation of a control object apparatus using screen information and operation information as it is.

<Specific example 5 of remote control>
Next, specific example 5 of remotely operating a control target device using the remote operation system 2 having the above configuration will be described with reference to the flowcharts of FIGS. 19A and 19B. However, FIG. 19A is a flowchart of the terminal device 60, and FIG. 19B is a flowchart of the device management server 50. In the specific example 5, the captured image data captured on the terminal device 60 side and the operation screen information and operation information extracted on the device management server 50 side are transmitted and received between the device management server 50 and the terminal device 60. Since it substantially corresponds to the specific example 1, the device management database 11A used in the specific example 1 is also used in the specific example 5. In addition, the explanatory diagram of the screen display shown in FIG.

  The user takes a picture of a television as a control target device using the photographing unit 64 of the terminal device 60, clears the display screen of the terminal device 1, and displays the photographed image (step S71). Here, it is assumed that the television set corresponding to the device image A in the device management database 11A shown in FIG.

  The captured device image is displayed on the display unit 63 (see FIG. 10A).

  The image recognition unit 65 compares and recognizes the captured image data and the device image A data, which is device information stored in the known device database 61 (step S72), and succeeds in identifying the device. It is confirmed whether or not (step S73). The device can be specified in the same manner as in the first specific example.

  The device-side communication unit 66 transmits the specific information (characteristic data or data of the captured image of the device, etc.) specified in this way to the device management server 50 (step S74), and then from the device management server 50. The information reception wait state is entered (step S75).

  On the other hand, the device management server 50 is waiting to receive specific information from the terminal device 60 (step S91). When the specific information from the terminal device 60 is received in this state (when it is determined Yes in step S91), the device management database 51 is searched based on this specific information, and the data of the remote control image which is the corresponding operation screen information (For example, AAA) and command definition data (for example, AAA) as operation information are extracted (step S92), and the extracted remote controller image data and command definition data are transmitted to the terminal device 60 through the database side communication unit 52 and the network N. (Step S93).

  On the other hand, when the terminal device 60 receives the data of the remote control image AA and the data of the command definition data AAA from the device management server 50 (when it is determined Yes in step S75), the information storage unit 70 receives the received remote control image. The AA data and the command definition data AAA are stored in the known device database 61 in association with the device image A. FIG. 18B shows a state where the data of the computer image AA and the command definition data AAA are stored in the known device database 61.

  The image composition unit 67 converts the remote control image AA stored in the known device database 61 into a screen coordinate system, composes it with the first photographed image, and displays it on the display unit 63 (step S76).

  The subsequent processing in steps S77 to S80 is the same as the processing in steps S6 to S9 in FIG.

  According to the fifth specific example, by directly operating the operation button of the remote control image displayed on the display unit 63 of the terminal device 1, the remote control device displayed on the display unit 63 can be remotely operated as if the remote control device is directly operated. Can be operated.

<Embodiment 3>
FIG. 20 shows a schematic configuration diagram of a remote control system according to Embodiment 3 of the present invention.

  The difference between the remote operation system according to the second embodiment and the remote operation system according to the third embodiment is that the remote operation system according to the third embodiment includes an image recognition unit 55 on the device management server 50 side. Since other configurations are the same as those of the remote control system according to the second embodiment, detailed description of the same components is omitted here.

  According to the remote operation system 2 according to the third embodiment, since the image recognition is performed on the device management server 50 side, the burden on the terminal device 1 side can be reduced.

<Specific example 6 of remote control>
Next, a specific example 6 of remotely operating a control target device using the remote operation system 2 having the above configuration will be described with reference to the flowcharts of FIGS. 21A and 21B. However, FIG. 21A is a flowchart of the terminal device 60, and FIG. 21B is a flowchart of the device management server 50. In this specific example 6 as well, the device management server 50 and the terminal device 60 transmit / receive the captured image data captured on the terminal device 60 side and the operation screen information and operation information extracted on the device management server 50 side. Corresponds substantially to the first specific example, and therefore the device management database 11A used in the first specific example 1 is also used in the sixth specific example. Further, the explanatory diagram of the screen display shown in FIG.

  The user uses the imaging unit 64 of the terminal device 60 to image the television that is the control target device, clears the display screen of the terminal device 1, and displays the captured image (step S101). Here, it is assumed that the television set corresponding to the device image A in the device management database 11A shown in FIG.

  The captured device image is displayed on the display unit 63 (see FIG. 10A).

The apparatus-side communication unit 66 transmits the captured image data captured in this manner to the device management server 50 (step S102), and determines whether an error code indicating failure in device identification has been received from the device management server 50. (Step S103). If an error code is received (Yes in Step S103), the process returns to Step S101. If no error code is received (No in Step S103), the device management server 50 is used. (Step S104).
On the other hand, the device management server 50 is in a state of waiting to receive photographed image data from the terminal device 60 (step S111). When the captured image data is received from the terminal device 60 in this state (when it is determined Yes in step S111), the image recognition unit 65 receives the received captured image data and the device information stored in the device management database 51. The device image A is compared (verified) and recognized (step S112), and it is confirmed whether or not the device has been successfully identified (step S113). The device can be specified in the same manner as in the first specific example.

  If the device is identified successfully (Yes in step S113), the information extraction unit 56 searches the device management database 51 based on the identified specific information, and the remote control image that is the corresponding operation screen information. Data (for example, AAA) and command definition data (for example, AAA) as operation information are extracted (step S114), and the extracted remote control image data and command definition data are transmitted to the terminal device through the database side communication unit 52 and the network N. It transmits to 60 side (step S115). On the other hand, if the device identification has failed (No in step S113), an error code is transmitted to the terminal device 60 (step S116).

  The terminal device 60 confirms whether or not an error code indicating failure in device identification has been received from the device management server 50. If an error code has been received (if determined Yes in step S103), the terminal device 60 returns to step S101. On the other hand, if an error code has not been received (if determined No in step S103), it is next determined whether or not remote control image AA data and command definition data AAA data have been received from the device management server 50. Is confirmed (step S104), and when the remote control image AA data and the command definition data AAA data are received (Yes in step S104), the information storage unit 70 receives the data of the remote control image AA. The command definition data AAA is stored in the known device database 61 in association with the device image A.

  The image composition unit 67 converts the remote control image AA stored in the known device database 61 into a screen coordinate system, composes it with the first photographed image, and displays it on the display unit 63 (step S105).

  The subsequent processing in steps S106 to S109 is the same as the processing in steps S6 to S9 in FIG. 13 described in the first specific example, and thus description thereof is omitted here.

  According to the sixth specific example, by directly operating the operation button of the remote control image displayed on the display unit 63 of the terminal device 1, the remote control device displayed on the display unit 63 can be remotely operated as if the remote control device is directly operated. Can be operated.

  The above specific examples 2 to 4 can also be implemented in the remote control system according to the second and third embodiments in the same manner as the above specific examples 5 and 6 corresponding to the above specific example 1. Therefore, the description of the processing corresponding to the specific examples 2 to 4 by the remote control system according to the second and third embodiments is omitted here.

  The embodiments disclosed herein are illustrative in all respects and do not serve as a basis for limited interpretation. Therefore, the technical scope of the present invention is not interpreted only by the above-described embodiments, but is defined based on the description of the scope of claims. Moreover, all the changes within the meaning and range equivalent to a claim are included.

1,60 terminal device (portable terminal device)
2 Remote operation system 11 (11A to 11D), 51 Device management database 12, 62 Operation input unit (operation input means)
13, 63 Display section (display means)
14, 74 Shooting unit (shooting means)
14a, 64a Camera 14b, 64b A / D conversion unit 15, 55, 65 Image recognition unit (image recognition means)
16, 56 Information extraction unit (information extraction means)
17, 67 Image composition unit (image composition means)
18, 68 Control command transmitter (output means)
19, 69 Control device 50 Device management server 61 Known device database 70 Information storage unit (information storage means)

Claims (18)

A terminal device for remotely operating a control target device,
A device management database storing operation information corresponding to various operations of the device information, operation screen information, and operation screen information of the control target device;
Photographing means for photographing the device to be controlled;
Image recognition means for recognizing and identifying the device by comparing the captured image data of the device to be controlled and the device image data which is one of the device information stored in the device management database; ,
Information extraction means for referring to the device management database based on the device information of the specified control target device and extracting operation information corresponding to the control target device;
Display means for displaying the captured image;
An operation input means for instructing and inputting an arbitrary position on the display screen of the display means;
When the operation area defined by the extracted operation information is operated by the operation input means, an output that acquires an operation signal corresponding to the operation area from the operation information and outputs the operation signal to the control target device Means,
A terminal device comprising: The terminal device according to claim 1,
A terminal device that performs coordinate transformation on the operation area on the operation input means so as to coincide with a display position of a corresponding operation area of the control target device displayed on the display means. The terminal device according to claim 1 or 2, wherein
The information extraction means further extracts operation screen information corresponding to the operation information of the control target device,
The terminal device characterized in that the operation screen information extracted by the information extraction unit is combined with the captured image and displayed. The terminal device according to claim 3,
A terminal device comprising: an emphasis processing means for emphasizing the display of the operation area displayed on the display means. The terminal device according to claim 4,
The terminal device characterized in that the enhancement process is a process of enlarging and displaying the operation area. The terminal device according to claim 4 or 5, wherein
The terminal device characterized in that the enhancement processing is processing for rearranging and displaying the operation areas. The terminal device according to any one of claims 4 to 6,
The enhancement process is a process of surrounding the operation area itself or a peripheral part including the operation area with a frame,
The output device outputs an operation signal corresponding to the operation area by operating the area surrounded by the frame. The terminal device according to any one of claims 4 to 6,
The enhancement process is a process of displaying a speech balloon corresponding to the operation area,
The output device outputs an operation signal of the operation area corresponding to the balloon part by operating the balloon part. The terminal device according to any one of claims 3 to 8,
When a plurality of control target devices are photographed by the photographing unit and the plurality of control target devices are recognized by the image recognition unit, the display unit displays the operation screen information of all recognized control target devices. A terminal device characterized by being combined with a photographed image and displayed. The terminal device according to claim 9,
A switch means for switching from the display of the plurality of operation screen information to the display of any one operation screen information, while switching from the display of the one operation screen information to the display of the plurality of operation screen information is provided. Terminal device to do. The terminal device according to any one of claims 1 to 10, wherein
The terminal device characterized in that the photographing direction of the photographing means and the output direction of the output means are arranged in the same direction on the apparatus housing. A remote operation system for remotely operating a control target device by communication between a device management server and a terminal device,
The device management server
A device management database storing device information and operation information of the device to be controlled;
Receiving means for receiving specific information for specifying a control target device transmitted from the terminal device;
Information extracting means for referring to the device management database based on the specific information received by the receiving means and extracting the operation information corresponding to the control target device;
Transmitting means for transmitting the extracted operation information to the terminal device,
The terminal device
A known device database in which device information of the device to be controlled is stored;
Photographing means for photographing the device to be controlled;
Image recognition means for recognizing and identifying the device by comparing the data of the photographed image of the device to be controlled with the data of the device image that is one of the device information stored in the known device database;
Transmitting means for transmitting the specific information specified by the image recognition means to the device management server;
Receiving means for receiving the operation information transmitted from the device management server;
Storage means for storing the operation information received by the receiving means in the known device database;
Display means for displaying the photographed image photographed by the photographing means;
An operation input means for instructing and inputting an arbitrary position on the display screen of the display means;
When the operation area defined by the extracted operation information is operated by the operation input means, an output that acquires an operation signal corresponding to the operation area from the operation information and outputs the operation signal to the control target device Means,
A remote control system characterized by comprising: A remote operation system for remotely operating a control target device by communication between a device management server and a terminal device,
The device management server
A device management database storing device information and operation information of the device to be controlled;
Receiving means for receiving photographed image data of the control target device transmitted from the terminal device;
Image recognition means for recognizing and identifying the device by comparing the received image data and the device image data which is one of the device information stored in the device management database;
Information extraction means for referring to the device management database based on the device information of the specified control target device and extracting operation information corresponding to the control target device;
Transmitting means for transmitting the extracted operation information to a terminal device,
The terminal device
A known device database in which device information of the device to be controlled is stored;
Photographing means for photographing the device to be controlled;
Transmitting means for transmitting captured image data of the control target device imaged by the imaging means to the device management database;
Receiving means for receiving the operation information transmitted from the device management server;
Storage means for storing the received operation information in the known device database;
Display means for displaying the photographed image photographed by the photographing means;
An operation input means for instructing and inputting an arbitrary position on the display screen of the display means;
When the operation area defined by the extracted operation information is operated by the operation input means, an output that acquires an operation signal corresponding to the operation area from the operation information and outputs the operation signal to the control target device Means,
A remote control system characterized by comprising: The remote control system according to claim 12 or 13,
The information extraction means further extracts operation screen information corresponding to the operation information of the control target device,
The remote operation system characterized in that the display means combines and displays the operation screen information extracted by the information extraction means with the photographed image. A remote operation method for remotely operating a control target device using a terminal device provided with a photographing means, an image recognition means, an information extraction means, a display means, an operation input means, and an output means,
Using a device management database storing device information and operation information of the control target device,
A photographing step of photographing the device to be controlled by the photographing means;
The image recognition means recognizes the device by comparing the captured image data of the control target device with the device image data which is one of the device information stored in the device management database. Specific steps to identify;
The information extraction means refers to the device management database based on the device information of the specified control target device, and extracts the operation information corresponding to the control target device;
A display step in which the display means combines and displays the extracted operation screen information with the data of the captured image;
When the operation area specified by the extracted operation information is operated by designating an arbitrary position on the display screen of the display means by the operation input means, the output means An output step of acquiring an operation signal corresponding to the operation area from the inside and outputting the operation signal to the control target device;
A remote operation method comprising: A remote operation method for remotely operating a control target device by communication between a device management server and a terminal device,
The device management server includes a device management database storing device information and operation information of the control target device, a receiving unit, an information extracting unit, and a transmitting unit.
The terminal device includes a photographing unit, a known device database, an image recognition unit, a transmission unit, a reception unit, a display unit, an operation input unit, and an output unit.
A photographing step in which the photographing means of the terminal device photographs the control target device;
The image recognition unit of the terminal device recognizes the device by comparing the captured image data of the captured device to be controlled with device image data that is one of device information stored in the known device database. Specific steps to identify;
A transmission step in which the transmission unit of the terminal device transmits the specific information specified by the image recognition unit to the device management server;
A receiving step in which the receiving means of the device management server receives the specific information;
An extraction step in which the information extraction means of the device management server refers to the device management database based on the received specific information, and extracts operation information corresponding to the control target device;
A transmission step in which the transmission means of the device management server transmits the extracted operation information to the terminal device;
A receiving step in which the receiving means of the terminal device receives the operation information transmitted from the device management server;
A storage step in which the storage means of the terminal device stores the received operation information in the known device database;
A display step in which the display means of the terminal device displays the photographed image photographed by the photographing means;
When the operation area specified by the extracted operation information is operated by instructing an arbitrary position on the display screen of the display means by the operation input means, the output means is connected to the known device database. An output step of obtaining an operation signal corresponding to the operation area from the operation information stored in the operation information and outputting the operation signal to the control target device;
A remote operation method comprising: A remote operation method for remotely operating a control target device by communication between a device management server and a terminal device,
The device management server includes a device management database storing device information and operation information of the control target device, a receiving unit, an image recognition unit, an information extraction unit, and a transmission unit.
The terminal device includes a photographing unit, a transmission unit, a reception unit, a storage unit, a display unit, an input operation unit, and an output unit,
A photographing step in which the photographing means of the terminal device photographs the control target device;
A transmission step in which the transmission unit of the terminal device transmits the captured image data of the captured device to be controlled to the device management server;
A receiving step in which the receiving unit of the device management server receives the transmitted captured image data;
A specification in which the image recognition unit of the device management server recognizes and identifies the device by comparing the received captured image data with device image data that is one of device information stored in the device management database. Steps,
An extraction step in which the information extraction means of the device management server refers to the device management database based on the specified information, and extracts operation information corresponding to the control target device;
A transmission step in which the transmission means of the device management server transmits the extracted operation information to the terminal device;
A receiving step in which the receiving means of the terminal device receives the operation information transmitted from the device management server;
A storage step in which the storage means of the terminal device stores the received operation information in the known device database;
A display step in which the display means of the terminal device displays the photographed image photographed by the photographing means;
When the operation area specified by the extracted operation information is operated by instructing an arbitrary position on the display screen of the display means by the operation input means, the output means is connected to the known device database. An output step of obtaining an operation signal corresponding to the operation area from the operation information stored in the operation information and outputting the operation signal to the control target device;
A remote operation method comprising: The remote control method according to any one of claims 15 to 17, comprising:
In the extracting step, operation screen information corresponding to the operation information of the control target device is further extracted,
In the transmission step in the device management server, the extracted operation screen information and the terminal device are transmitted,
In the receiving step of the terminal device, the operation screen information transmitted from the device management server is received,
In the storing step, the received operation screen information is stored in the known device database,
In the output step, when the operation area defined by the extracted operation information is operated by designating an arbitrary position on the display screen of the display means by the operation input means, the known device database The remote operation method characterized by acquiring the operation signal corresponding to the said operation area | region from the said operation screen information stored in A, and outputting it to the said control object apparatus.

Images