JP2012156836A - Remote controller and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a remote controller that efficiently specifies an apparatus to be remotely controlled and allows the apparatus to be remotely controlled to be operated, and further to provide a program and the like.SOLUTION: A remote controller includes: an imaging section for imaging an apparatus; a recognizing section for performing recognition processing of the apparatus; and a remote control processing section for performing remote control processing for remote control operation of the apparatus. The recognizing section performs marker recognition processing of an identification marker of the apparatus captured in an imaged image by image recognition processing of the imaged image. The remote control processing section performs remote control processing for the remote control operation of the apparatus based on the result of the marker recognition processing.

Description

  The present invention relates to a remote control device, a program, and the like.

  As remote control devices for remotely controlling devices such as TVs and DVD players, there are remote control devices dedicated to devices, multi-remote control devices, learning-type multi-remote control devices, and the like.

  Here, in the remote control device dedicated to the device, it is necessary to replace the remote control device for each device, and there is a problem that it is complicated for the user. In addition, one remote control device is required for one device in the home, and the number of remote control devices increases.

  In general, the multi-remote control device corresponds to a control signal that differs depending on the manufacturer in the same product field, such as a multi-remote control device for TV and a multi-remote control device for air conditioner. However, this multi-remote control device still has a problem that a remote control device is required for each device in the home. Further, it is necessary to set the manufacturer name and model information of the operation target device in advance in the remote control device, which is troublesome for the user.

  In the learning type multi-remote control device, it is possible to operate a plurality of devices with one remote control device by storing the infrared signal transmitted from the above-described device-specific remote control device. However, in this learning type multi-remote control device, setting input for storing all remote control operations of a plurality of devices is complicated for the user. Further, in order to store many remote control operations in the remote control device, the number of operation buttons on the remote control device main body becomes enormous. In addition, since the arrangement of the operation buttons mounted on the remote control device is fixed, there is a problem that the meaning of each operation and the operation buttons is difficult to understand and is difficult for the user to use.

  Patent Document 1 discloses a remote control device on which a camera is mounted. In the prior art of Patent Document 1, a device is identified by blinking light emitted from the device side.

  However, in this prior art, an information transmission device such as an infrared LED is provided on the device side that is the target of remote control operation, and the remote control device receives individual individual information and performs remote control operation of the device. For this reason, there is a problem that it is difficult to apply this conventional technique to existing equipment that does not have such an information transmission device.

  That is, general devices such as home appliances do not always include such an information transmission device such as an infrared LED. Therefore, in the above-described conventional technology, it is difficult to acquire the device information from a device that does not have such an information transmission device or to perform a remote control operation.

JP 2007-28087 A

  According to some aspects of the present invention, it is possible to provide a remote control device, a program, and the like that can efficiently specify and operate a remote control target device.

  One aspect of the present invention includes an imaging unit that images a device, a recognition unit that performs device recognition processing, and a remote control processing unit that performs remote control processing for remote-controlling the device, and the recognition unit includes the imaging Image recognition processing of the image performs marker recognition processing of the identification marker of the device shown in the captured image, and the remote control processing unit performs remote control processing for remote-controlling the device based on the result of the marker recognition processing Related to remote control device.

  According to the aspect of the present invention, the shape recognition process of the identification marker of the device shown in the captured image is performed by the image recognition process of the captured image from the imaging unit. Based on the result of the marker recognition process, a remote control process for operating the device by remote control is executed. Accordingly, for example, a device can be specified by marker recognition processing of an identification marker of the device, and various remote control processes can be executed for the specified device. Therefore, it is possible to efficiently specify and operate a remote control target device. A remote control device can be provided.

  In the aspect of the invention, the remote control processing unit may register the device information of the device obtained by the marker recognition process as registration information in the device registration mode.

  This makes it possible to register the device information of the device obtained by the marker recognition process in the device registration mode as registration information, and use the device information of the registration information in other modes and the like.

  In one aspect of the present invention, the information processing apparatus includes a storage unit that stores the registration information including the device information. The remote control processing unit reads out the registration information from the storage unit in a remote control operation mode, thereby The device information of the device may be acquired.

  In this way, the device information of the device obtained by the marker recognition process in the device registration mode is registered as registration information, the device information of the registration information is acquired in the remote control operation mode, the remote control processing of the remote control operation target device, etc. Can be executed.

  In one aspect of the present invention, the recognition unit performs shape recognition processing of a device shown in the captured image by image recognition processing of the captured image in the device registration mode, and the remote control processing unit performs the device registration. In the mode, the device feature amount information obtained by the shape recognition process in the device registration mode is registered as the device registered feature amount information, and the storage unit is obtained by the marker recognition process. Information associated with device information and the registered feature amount information obtained by the shape recognition processing is stored as the registration information, and the recognition unit performs image recognition processing of the captured image in the remote control operation mode. The remote controller processing unit performs the remote controller operation in the remote controller operation mode. The device feature amount information, which is the feature amount information obtained by the shape recognition processing in the node, is compared with the registered feature amount information, and the device feature amount information matches the registered feature amount information. The device information of the remote control operation target device may be acquired by reading out the device information from the storage unit.

  According to this configuration, the registered feature amount information of the device obtained by the shape recognition process is registered in the device registration mode, and the remote controller operation is performed by comparing the registered feature amount information with the device feature amount information in the remote control operation mode. The target device can be selected. Therefore, an efficient device selection process using the feature amount information can be realized.

  In one aspect of the present invention, the remote control processing unit uses the device feature information obtained from the identification marker by the marker recognition process in the device registration mode in the device registration mode. A process of registering as information, and the storage unit stores information in which the device information obtained by the marker recognition process is associated with the registered feature amount information obtained by the marker recognition process. And the recognition unit performs shape recognition processing of the device shown in the captured image by image recognition processing of the captured image in the remote control operation mode, and the remote control processing unit performs the shape recognition processing in the remote control operation mode. Device feature amount information, which is feature amount information obtained by the shape recognition process in the remote control operation mode, and the registered feature The device information of the remote control operation target device may be acquired by performing comparison processing with information and reading out the device information of the device whose device feature amount information matches the registered feature amount information from the storage unit. .

  In this way, in the device registration mode, it is possible to register the device information acquired by the marker recognition processing in association with the registered feature amount information obtained by the marker recognition processing. Then, in the remote control operation mode, by comparing the registered feature value information and the device feature value information, it becomes possible to select a remote control operation target device, and an efficient device selection process using the feature value information can be realized.

  According to another aspect of the present invention, the apparatus includes a motion sensor that detects movement of a remote control device, and the remote control processing unit performs remote control operation of the device based on a result of the shape recognition process and sensor information from the motion sensor. Remote control processing may be performed.

  In this way, the remote control process can be executed using not only the result of the shape recognition process but also the sensor information from the motion sensor, and the remote control process using the motion sensor effectively can be realized.

  In one aspect of the present invention, the image processing apparatus includes a display unit that displays a plurality of icon images corresponding to a plurality of devices or a plurality of device images of the plurality of devices, and the remote control processing unit includes the plurality of icon images or the plurality of icons. A device corresponding to the icon image or device image selected by the user from among the device images may be selected as the remote control operation target device.

  In this way, it is possible to provide a remote control device that can efficiently specify and operate a remote control target device by selecting an icon image or a device image.

  Moreover, in one aspect of the present invention, the recognition unit may perform the marker recognition process for the identification marker attached to a device.

  In this way, it is possible to acquire various types of information such as device information simply by performing marker recognition processing for the identification marker attached to the device, and convenience can be improved.

  In the aspect of the invention, the recognition unit may perform the marker recognition process for the identification marker displayed on the display unit of the device.

  In this way, it is possible to acquire various types of information such as device information simply by performing marker recognition processing for the identification marker displayed on the display unit of the device, and to acquire various types of information about the device. Is possible.

  In one embodiment of the present invention, the remote control processing unit acquires at least one of device status information and device operation manual information based on a result of the marker recognition processing of the identification marker displayed on the device display unit. May be.

  In this way, it becomes possible to acquire the status information and the operation manual information of the device by effectively utilizing the identification marker displayed on the display unit of the device.

  In the aspect of the present invention, the remote control processing unit may perform a remote control signal generation information generation process for operating a remote control target device as a remote control as the remote control process.

  In this way, it becomes possible to specify the remote control operation target device using the result of the marker recognition processing of the identification marker, and to transmit a remote control signal to the specified device.

  In the aspect of the present invention, the remote control processing unit may perform control to display a remote control operation screen for performing remote control operation on a remote control operation target device on the display unit as the remote control processing.

  In this way, it becomes possible to specify a remote control operation target device using the marker recognition processing result of the identification marker and the like, and display a remote control operation screen for operating the specified device on the display unit.

  In the aspect of the invention, the remote control processing unit may control the m-th device when the remote control device is determined to be directed in a direction corresponding to the m-th device among a plurality of devices. Control to display the m-th remote control operation screen for operation on the display unit is performed, and the remote control device is directed in a direction corresponding to the n-th device (m and n are natural numbers) among the plurality of devices. If it is determined that the display has been performed, the display unit may be controlled to display an n-th remote control operation screen for operating the n-th device by remote control.

  In this way, the remote control operation screen of the device corresponding to the direction in which the remote control device is directed is displayed on the display unit of the remote control device, and a plurality of devices can be operated efficiently with one remote control device. A remote control device can be provided.

  According to another aspect of the present invention, a computer functions as an imaging unit that images a device, a recognition unit that performs device recognition processing, and a remote control processing unit that performs remote control processing for operating the device by remote control. The unit performs marker recognition processing of the identification marker of the device shown in the captured image by image recognition processing of the captured image, and the remote control processing unit performs remote control operation of the device based on the result of the marker recognition processing Related to the remote control processing program.

1A to 1C are external views of a remote control device according to the present embodiment. The block diagram which shows the structural example of the remote control apparatus of this embodiment. 3A and 3B are explanatory diagrams of the method of the present embodiment. FIG. 4A to FIG. 4C are also explanatory diagrams of the method of this embodiment. FIGS. 5A to 5D are also explanatory diagrams of the method of the present embodiment. 6A and 6B are explanatory diagrams of a method for recognizing a device by image recognition processing in the device registration mode and the remote control operation mode. FIG. 7A is an example of the data structure of the remote control database, and FIG. 7B is an explanatory diagram of a method for registering registration information in the remote control database. Explanatory drawing of the method of acquiring apparatus information from an identification marker. FIG. 9A and FIG. 9B are explanatory diagrams of a method for selecting a device on the device selection screen. FIG. 10A and FIG. 10B are also explanatory diagrams of a method for selecting a device on the device selection screen. FIG. 11A and FIG. 11B are explanatory diagrams of a technique for performing marker recognition processing of the identification marker displayed on the display unit of the device. Explanatory drawing of the method of performing the marker recognition process of the identification marker displayed on the display part of a portable information terminal. Explanatory drawing of the method of displaying the remote control operation screen of the apparatus of the direction to which the remote control device was pointed. 14A and 14B are explanatory diagrams of a sleep mode and a reference device selection mode. FIGS. 15A to 15C are explanatory diagrams of a device selection method using a motion sensor. FIG. 16A to FIG. 16C are examples of screens displayed on the display unit in device selection using a motion sensor. The example of the arrangement | positioning information of the apparatus used by this embodiment. 18A and 18B are explanatory diagrams of a method for specifying the position of the user using the reference device. The flowchart for demonstrating the detailed operation | movement of this embodiment. The flowchart for demonstrating the detailed operation | movement of this embodiment. The flowchart for demonstrating the detailed operation | movement of this embodiment.

  Hereinafter, preferred embodiments of the present invention will be described in detail. The present embodiment described below does not unduly limit the contents of the present invention described in the claims, and all the configurations described in the present embodiment are indispensable as means for solving the present invention. Not necessarily.

1. Configuration Example FIGS. 1A to 1C are external configuration diagrams of a remote control device (remote control device) of the present embodiment. 1B and 1C are side views as viewed from the A1 direction and the A2 direction of FIG. 1A, respectively. As shown in FIGS. 1A to 1C, the remote control device according to the present embodiment includes an imaging unit 10 (camera), a motion sensor 20, a display unit 30, and an infrared light emitting unit 40. Specifically, as shown in FIG. 1C, the imaging unit 10 and the infrared light emitting unit 40 are provided on the upper side surface of the remote control device. The motion sensor 20 is provided inside the remote control device. Display unit 30 is provided on the operation-side surface of the remote control device.

  FIG. 2 shows a detailed configuration block diagram of the remote control device of the present embodiment. The remote control device includes an imaging unit 10, a motion sensor 20, an operation unit 28, a display unit 30, an infrared light emitting unit 40, a storage unit 50, an information storage medium 60, an external I / F (interface) unit 62, and a processing unit 100. Note that the remote control device according to the present embodiment is not limited to the configuration shown in FIG. 2, and various modifications such as omitting some of these components or adding other components are possible.

  The imaging unit 10 images a device or the like, and includes an optical system 12 and an imaging element 14. The optical system 12 is realized by a lens or the like, and the imaging element 14 is realized by a CMOS sensor, a CCD sensor, or the like.

  The motion sensor 20 is a device that detects the movement of the remote control device, and includes, for example, an acceleration sensor 22 and a gyro sensor 24 (angular velocity sensor).

  As the acceleration sensor 22, a capacitance type sensor that detects a change in position at a movable part supported by a beam structure as a change in capacitance, or a piezoresistive type that detects a change in position of a diaphragm by a piezoresistive element. Sensors and gas temperature distribution type sensors can be used. As the gyro sensor 24, an angular velocity sensor such as a vibration type that detects an angular velocity from Coriolis force generated by the rotation of the vibrator, an angular acceleration sensor that detects angular acceleration from a change in capacitance or inertia force, and the like are adopted. it can.

  By using the acceleration sensor 22, the acceleration of the movement of the remote control device in the X-axis, Y-axis, and Z-axis directions can be detected. By using the gyro sensor 24, for example, the angular velocity of the movement of the remote control device around the X axis and the Y axis can be detected. For example, the X axis is a coordinate axis along the horizontal direction of the remote control device in FIG. 1A, the Z axis is a coordinate axis along the vertical direction of the remote control device, for example, and the Y axis is orthogonal to the X axis and the Z axis. It is a coordinate axis. The gyro sensor 24 may be a three-axis gyro sensor that can also detect an angular velocity about the Z axis.

  The operation unit 28 is used by the user to input operation information of the remote control device, and can be realized by, for example, an operation button or a power button.

  The display unit 30 is for displaying various images, and is realized by, for example, an LCD or an organic EL display. For example, in the present embodiment, the display unit 30 displays a remote control operation screen that is an operation screen for operating the device by remote control.

  The display unit 30 is preferably realized by a touch panel display, for example. As the touch panel display, various types such as a resistive film type, a capacitance type, and an electromagnetic induction type can be adopted. When a touch panel display is used as the display unit 30, some or all of the functions of the operation unit 28 are realized by this touch panel display.

  The infrared light emitting unit 40 (remote control signal output unit) outputs a remote control signal to the device by infrared rays, and is realized by, for example, an infrared light emitting diode (LED). For example, a remote control signal for operating a device can be realized by a pulse phase modulation signal (PPM signal) or the like in which a bit value is represented by a combination of the length of time for turning on and off the infrared light. In this case, the infrared light emitting unit 40 emits infrared light modulated at a specific carrier frequency (for example, 38 to 42 KHz) in order to distinguish between the naturally radiated infrared light and the infrared light of the remote control device.

  The storage unit 50 stores various information necessary for the operation of the remote control device, and is realized by a semiconductor memory such as a RAM. In the present embodiment, the storage unit 50 stores registration information and arrangement information described later.

  The information storage medium 60 is realized by various memory media such as a memory card, and can be freely attached to and detached from the remote control device, for example. A program for realizing each unit of the present embodiment as a computer is stored in the information storage medium 60.

  The external I / F unit 62 performs interface processing for communicating information with the outside (external device). Information communication with the outside may be wired or wireless.

  The processing unit 100 performs various processes necessary for the operation of the remote control device. The functions of the processing unit 100 can be realized by hardware such as various processors (CPU and the like), ASIC (gate array and the like), and programs. For example, the processing unit 100 performs various processes of the present embodiment based on a program stored in the information storage medium 60. That is, in the information storage medium 60, a program for causing a computer (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing a computer to execute processing of each unit). Is memorized.

  Note that the remote control device of this embodiment does not have to be a dedicated remote control device, and a portable information terminal such as a mobile phone may function as the remote control device. In this case, the portable information terminal can be caused to function as a remote control device by executing a program for executing each part of the present embodiment on the portable information terminal.

  The processing unit 100 includes a recognition unit 110, a sensor information acquisition unit 120, and a remote control processing unit 130. Various modifications may be made such as omitting some of these components or adding other components.

  The recognition unit (specification unit) 110 performs device recognition processing (specification processing). For example, recognition processing such as device shape recognition processing and marker recognition processing is performed. The shape recognition process is performed by the shape recognition unit 112, and the marker recognition process is performed by the marker recognition unit 114. Details of these recognition processes will be described later.

  The sensor information acquisition unit 120 acquires sensor information (acceleration information, angular velocity information, angular acceleration information, etc.) from the motion sensor 20. Then, various processes such as a sensor information detection process and a calculation process based on the sensor information are performed.

  The remote control processing unit 130 performs remote control processing for operating the device by remote control. The remote control processing unit 130 includes a device selection unit 132, a remote control signal generation unit 134, a display control unit 136, and a communication processing unit 138. The device selection unit 132 performs selection processing of a remote control operation target device. The remote control signal generation unit 134 performs processing for generating information for generating a remote control signal for operating the remote control target device by remote control. And the infrared light emission part 40 produces | generates the remote control signal by infrared rays based on this remote control signal production | generation information, and transmits to an apparatus. The remote control signal generation information may be transmitted to the remote control target device via a network. The display control unit 136 performs display control of the display unit 30. Specifically, control is performed to display on the display unit 30 a remote control operation screen for remote control operation of the remote control target device. The communication processing unit 138 performs processing for communicating various types of information with the outside (external device) via the external I / F unit 62, for example. Details of the processing of the remote control processing unit 130 will be described later.

2. A method of the present embodiment A multi-function remote control device that consolidates remote control functions for remote-controlling various devices into a single remote control device has been conventionally known. Such a remote control device is provided with functions for eliminating differences in hardware (physical layer), differences in operation commands and control data for each operation target device, and storing and recalling control data for a plurality of operation target devices. It is necessary to take measures such as If these measures are taken, the remote control functions required in the home can be integrated into a single remote control device.

  However, in order to realize such a response by the multi-function remote control device, it is necessary to match the physical compatibility of the remote control device with respect to a plurality of operation target devices. Also, it is necessary for the multi-function remote control device to learn the remote control functions that are to be combined into one. Further, it is necessary to perform complicated operations such as selecting an operation target device on the multi-function remote control device side, assigning a function to an operation key, and operating the target operation target device by remote control. Therefore, when the remote control operation is performed, many restrictions are imposed on the user.

  Therefore, in the present embodiment, such a complicated operation is omitted, and the remote control device selects and presents the operation target device so that the user can use it intuitively, thereby improving the user's convenience.

  Specifically, in the present embodiment, a plurality of devices (electronic devices) can be remotely operated with a single remote control device. This eliminates the need to replace the remote control device for each device to be operated, improving user convenience.

  In this embodiment, the operation target device is automatically recognized by image recognition or the like, and a remote control code (remote control signal) that matches the device is automatically set. In this way, complicated initial settings and function switching operations are not required, and user convenience can be further improved. Further, since it is not necessary to add a new function to the operation target device side, it is possible to deal with an existing device already on the market, and it is possible to suppress an increase in cost on the device side.

  In the present embodiment, an operation target device is specified by a captured image acquired by a camera (imaging unit), and a remote control operation screen of the operation target device is displayed on a display unit realized by a touch panel display. Thereby, it is possible to provide the user with an easy-to-understand operation interface environment in which the optimum number of operation buttons is associated with the operation function.

  Specifically, the remote control operation function of the existing device is mounted on the remote control device as much as possible. The new portion can be registered and called using an identification marker such as a two-dimensional barcode, wired / wireless communication, memory media, and the like. In addition, by using a motion sensor such as a mounted camera, a gyro sensor, or an acceleration sensor, the operation target device that the user desires to use from the current position of the remote control device is arranged in association with the remote control device and the operation target device ( By selecting and presenting by analogy based on matrix information), the convenience of user selection is improved.

  If the operation target device selected by analogy is not what the user desires, the target operation target device can be easily selected by, for example, a gesture operation of shaking the remote control device up, down, left, or right.

  The on-board camera is used to read an identification marker such as a two-dimensional barcode and to obtain position information of the remote control device. The acceleration sensor is used to detect the gravitational direction, the moving direction, and the up / down / left / right swing. The gyro sensor is used to obtain the vertical and horizontal swing angles from the position recognized by the on-board camera. The memory medium is used so that registration information, arrangement information, and the like registered in the remote control device can be easily registered, moved, and deleted on a PC (personal computer).

  Next, the method of this embodiment will be described in more detail with reference to FIGS. 3 (A) to 16 (C).

  In the present embodiment, remote control processing for remote-controlling the device is performed based on the captured image from the imaging unit 10. Specifically, in the present embodiment, the recognition unit 110 performs marker recognition processing of the identification marker of the device shown in the captured image by image recognition processing on the captured image from the imaging unit 10. Then, the remote control processing unit 130 performs remote control processing for operating the device by remote control based on the result of the marker recognition processing.

  That is, in this embodiment, an identification marker is attached to the device. For example, in FIG. 3A, the identification marker MK is attached to the device ED1 that is a TV, and in FIG. 3B, the identification marker MK is attached to the device ED2 that is a DVD player.

  Here, the identification marker MK is an identifier that can be read by a computer by converting information such as characters, numbers, or symbols for identifying and recognizing each device into a figure or a pattern. Specifically, for example, a bar code (two-dimensional bar code) which is an identifier representing a character, a number, or a symbol by the thickness of a striped line. In FIGS. 3A and 3B, the identification marker MK is attached to each device by bonding or the like. However, as will be described later, the identification marker MK is displayed on the display portion of the device. Also good.

  4A, the user's remote control device RM faces the device ED1, and the captured image captured by the imaging unit 10 includes an image of the device ED1 and an identification marker attached to the device ED1. The image of MK is reflected. In FIG. 4B, the remote control device RM that the user has faces the device ED2, and the captured image captured by the imaging unit 10 includes an image of the device ED2 and an identification marker attached to the device ED2. The image of MK is reflected.

  In this case, the recognition unit 110 performs marker recognition processing for the identification marker MK, and the remote control processing unit 130 performs remote control processing based on the result of the marker recognition processing. For example, recognition processing or selection processing of a remote control operation target device is performed.

  Specifically, as shown in FIG. 4C, remote control processing unit 130 registers the device information of the device obtained by the marker recognition process of identification marker MK as registration information in the device registration mode. The device information is, for example, device identification information or remote control command information, and these information are embedded in the identification marker MK. For example, the storage unit 50 (remote control database) in FIG. 2 stores registration information including device information. Then, the remote control processing unit 130 reads the registration information from the storage unit 50 in the remote control operation mode, thereby acquiring the device information of the remote control operation target device. Then, the remote control processing unit 130 performs remote control processing for operating the device by remote control based on the acquired device information.

  For example, as shown in FIG. 5A, the remote control processing unit 130 performs remote control signal generation information generation processing for remote control operation of a remote control target device as remote control processing. That is, the remote control processing unit 130 performs generation processing of remote control signal generation information for operating the device ED1 by remote control based on the device information of the device ED1 read from the storage unit 50. Based on the remote control signal generation information, the infrared light emitting unit 40 transmits a remote control signal for the device ED1 to the device ED1 by infrared rays. As a result, the device ED1 can be remotely controlled by the remote control device RM.

  Alternatively, as shown in FIG. 5B, the remote control processing unit 130 performs control to display a remote control operation screen for remote control operation of the remote control target device on the display unit 30 as remote control processing. That is, the remote control processing unit 130 performs display control of the remote control operation screen for the device ED1 based on the device information of the device ED1 read from the storage unit 50. For example, in FIG. 5B, the remote control operation screen for the device ED1 on which the operation icon for the device ED1 (operation button icon) is displayed is displayed on the display unit 30. The display unit 30 is configured by a touch panel display, for example. Therefore, for example, when the user touches the operation icon for the device ED1 displayed on the remote control operation screen, a remote control signal corresponding to the operation icon is generated and transmitted to the device ED1, and the remote control operation of the device ED1 is performed. It becomes possible.

  On the other hand, in FIG. 5C, the remote control processing unit 130 performs generation processing of remote control signal generation information for remote-controlling the device ED2 based on the device information of the device ED2 read from the storage unit 50. The infrared light emitting unit 40 transmits the remote control signal for the device ED2 to the device ED2 by infrared rays based on the remote control signal generation information. As a result, the device ED2 can be remotely controlled by the remote control device RM.

  5D, the remote control processing unit 130 performs display control of the remote control operation screen for the device ED2 based on the device information of the device ED2 read from the storage unit 50. For example, in FIG. 5D, a remote control operation screen for the device ED2 on which an operation icon for the device ED2 is displayed is displayed on the display unit 30. For example, when the user touches the operation icon for the device ED2 displayed on the remote control operation screen, a remote control signal corresponding to the operation icon is generated and transmitted to the device ED2, and the remote control operation of the device ED2 is performed. It becomes possible.

  As described above, in this embodiment, device information of each device is acquired in the device registration mode, for example, by performing marker recognition processing of the identification marker attached to the device. Alternatively, in the remote control operation mode, the remote control target device is selected by marker recognition processing of the identification marker.

  In this way, it is possible to realize device information acquisition processing and device selection processing without providing an information transmission device such as an infrared LED on the device side as in the prior art described above. For example, as shown in FIGS. 4A and 4B, an identification marker in which the device information is embedded is attached to each device. The identification marker may be attached by the manufacturer of the device or by the user who purchased the device. When the user attaches, the device to be attached is searched with a PC or the like, and the image data of the identification marker is downloaded from the site of the manufacturer of the device. Then, the identification marker may be printed based on the downloaded image data, and the printed identification marker may be attached to the device.

  In the present embodiment, the device information of the device is acquired by imaging the identification marker attached to the device with the imaging unit 10 and performing marker recognition processing. That is, device information embedded in the identification marker is acquired.

  Therefore, even when the device does not have an information transmission device for device identification, the device information can be transmitted to the remote control device by attaching the corresponding identification marker to the device. Accordingly, it is possible to cope with a device that does not have a device information transmission device, and the convenience of the user can be greatly improved.

  FIGS. 6A and 6B are explanatory diagrams of a method for performing device shape recognition processing by image recognition processing. For example, the recognition unit 110 extracts device feature amount information by shape recognition processing, and recognizes the device based on the feature amount information. Specifically, as shown in FIG. 6A, edge extraction processing of the device shown in the captured image is performed, and the edge information of the device is extracted as feature amount information. In FIG. 6A, the vertical and horizontal size information of the rectangle representing the edge of the device is the feature amount information.

  Then, as shown in FIG. 6B, the remote control processing unit 130 performs processing for registering the feature amount information of the device obtained by the shape recognition processing as the registered feature amount information of the device in the device registration mode.

  The recognition unit 110 also performs marker recognition processing for the identification marker attached to the device by image recognition processing of the captured image. Then, the remote control processing unit 130 registers the device information obtained by the marker recognition processing in association with the registered feature amount information. As a result, the storage unit 50 stores, as registration information, information in which the device information obtained by the marker recognition process is associated with the registered feature amount information obtained by the shape recognition process.

  Next, the recognizing unit 110 performs shape recognition processing of the device shown in the captured image by image recognition processing of the captured image in the remote control operation mode. Then, in the remote control operation mode, the remote control processing unit 130 performs a comparison process between the device feature amount information that is the feature amount information obtained by the shape recognition process in the remote control operation mode and the registered feature amount information. Then, by this comparison process, a remote control operation target device selection process is performed. For example, in the remote control operation mode, when the device ED1 is shown in the captured image from the imaging unit 10, and the device feature amount information of the device ED1 matches the registered feature amount information in the remote control database, the device ED1 is displayed by the user. Judge that selected.

  As shown in FIG. 6B, in the remote control database of the storage unit 50, the device information of the device acquired by the marker recognition process and the registered feature amount information are associated and stored as registration information. In the remote control operation mode, the remote control processing unit 130 reads out device information of a device whose device feature amount information matches the registered feature amount information from the storage unit 50, thereby acquiring the device information of the remote control operation target device. For example, when the device feature value information of the device ED1 shown in the captured image matches the registered feature value information in the remote control operation mode, the device information of the device ED1 associated with the registered feature value information is acquired. That is, the device information of the device ED1 obtained by the marker recognition process of the identification marker attached to the device ED1 is acquired by reading from the storage unit 50. Then, the remote control processing unit 130 executes remote control processing based on the device identification information and remote command information of the acquired device information. For example, remote control signal generation information generation processing for operating a remote control target device by remote control, control for displaying a remote control operation screen for remote control operation of the remote control operation target device, and the like are performed.

  FIG. 7A shows an example of the data structure of the remote control database. In the remote control database, device information of each device is registered as registration information. This remote control database is stored in the storage unit 50 of the remote control device.

  The device information is, for example, device identification information or remote control command information. The device identification information is information for identifying a device such as a model name or a manufacturer name. The remote control command information is command system information of a remote control command, and includes a command name, model-specific button definition information, remote control signal information, and the like. Each piece of device identification information is associated with registered feature amount information and registered in the remote control database as registration information.

  Various methods can be envisaged as a method for registering registration information in the remote control database of the remote control device. For example, in FIG. 7B, a two-dimensional barcode (identification marker) in which device information is embedded is displayed on the screen of a PC (personal computer) or a portable information terminal. Then, the displayed two-dimensional barcode is photographed by the imaging unit 10 of the remote control device, so that registration information such as device information is registered in the remote control database of the remote control device. Alternatively, in a PC, device information or the like may be downloaded from a device manufacturer or the like via a network and registered in a remote control database of the remote control device via a memory medium. At this time, arrangement information in a matrix form described later may be created by editing on a PC, and the created arrangement information may be registered in the remote control database of the remote control device via a memory medium or the like.

  In FIGS. 6A and 6B, device feature amount information is extracted by performing device shape recognition processing in the device registration mode, and the extracted feature amount information is stored as registered feature amount information. Registered in the section 50.

  However, the present embodiment is not limited to this, and the feature amount information of the device may be acquired by marker recognition processing of the identification marker in the device registration mode.

  Specifically, as shown in FIG. 8, in the device registration mode, the remote controller processing unit 130 uses the device feature amount information acquired from the identification marker by the marker recognition processing in the device registration mode as the device registered feature amount information. Perform the registration process. For example, the feature amount information of the device is acquired from the identification marker by imaging the identification marker attached to the device by the imaging unit 10. Alternatively, as shown in FIG. 12 to be described later, an identification marker is displayed on a display unit of a portable information terminal such as a cellular phone, and the displayed identification marker is imaged by the imaging unit 10 so that the feature of the device is determined from the identification marker. Quantity information may be acquired.

  As shown in FIG. 8, the storage unit 50 stores, as registration information, information in which the device information obtained by the marker recognition process is associated with the registered feature amount information obtained by the marker recognition process. Then, the recognition unit 110 performs shape recognition processing of the device shown in the captured image by image recognition processing of the captured image in the remote control operation mode. Further, in the remote control operation mode, the remote control processing unit 130 performs a comparison process between the device feature amount information, which is feature amount information obtained by the shape recognition process in the remote control operation mode, and the registered feature amount information. Then, the device information of the device whose device feature amount information matches the registered feature amount information is read from the storage unit 50, thereby acquiring the device information of the remote control operation target device.

  As described above, in the device registration mode, if the device feature amount information is acquired by the marker recognition processing of the identification marker instead of the device shape recognition processing, the device shape recognition processing is not performed in the device registration mode. The processing load can be reduced. In addition, by embedding device feature information in the identification marker, more accurate and accurate device feature information can be registered in the storage unit as registered feature information.

  In the present embodiment, as shown in FIG. 2, the motion sensor 20 for detecting the movement of the remote control device RM is built in the remote control device RM. Then, the remote control processing unit 130 performs remote control processing for operating the device by remote control based on the result of the shape recognition processing and the sensor information from the motion sensor 20.

  For example, as will be described later, when the next device is selected after the reference device or the like is selected by the image recognition process, the device is selected based on the sensor information from the motion sensor 20. For example, when the user moves the remote control device in order to select a desired device, the direction in which the remote control device is moved is detected by the motion sensor 20, and the device existing in the direction corresponding to the moved direction is selected as the remote control operation target device. Choose as.

  In the present embodiment, as illustrated in FIG. 9A, when the imaging unit 10 of the remote control device RM captures images of the devices ED1 to ED3, the display unit 30 displays a plurality of images based on the captured images from the imaging unit 10. An icon image may be displayed. That is, a device selection screen on which a plurality of icon images are displayed may be displayed.

  Then, the remote control processing unit 130 selects a device corresponding to the icon image selected by the user from the plurality of icon images as a remote control operation target device. That is, a device selection process is performed by a user selection operation on the device selection screen.

  For example, in FIG. 10A, a plurality of icon images corresponding to a plurality of devices are displayed. This icon image is an image representing each device with a picture or symbol representing it. For example, as shown in the table data of FIG. 9B, each icon image such as IC1 to IC3 is associated with each device such as ED1 to ED3. When the devices ED1 to ED3 are reflected in the captured image of the remote control device RM as shown in FIG. 9A, the icon images IC1 to IC3 corresponding to the devices ED1 to ED3 are selected, and the icons as shown in FIG. A device selection screen using the images IC <b> 1 to IC <b> 3 is displayed on the display unit 30.

  In FIG. 10B, the user selects an icon image of a desired device from a plurality of icon images by a touch operation or the like. Then, the remote control processing unit 130 selects a device corresponding to the icon image selected by the user from the plurality of icon images as a remote control operation target device. In this way, it is possible to specify and select a device desired by the user by a simple process without executing a heavy load process such as an image recognition process.

  Although a plurality of icon images are displayed in FIGS. 10A and 10B, a plurality of device images may be displayed instead of such icon images. Then, remote control processing unit 130 selects a device corresponding to the device image selected by the user from among a plurality of device images as a remote control operation target device. The device image is an image of the device imaged by the imaging unit 10, and for example, the captured image of the imaging unit 10 is displayed on the display unit 30 as it is. When the user selects a desired device image on the device selection screen, which is a captured image, by touch operation or the like, a device corresponding to the device image is selected, and display control or remote control of the remote control operation screen for the device is selected. Signal generation processing and the like are performed.

  In the device registration mode, the recognition unit 110 performs shape recognition processing of the device shown in the captured image by image recognition processing of the captured image. On the other hand, the remote control processing unit 130 associates a plurality of devices recognized based on the shape recognition process with a plurality of icon images or a plurality of device images in the device registration mode. For example, table data associating a device with an icon image or device image as shown in FIG. 9B is created. Then, in the remote control operation mode, the remote control processing unit 130 may select a device associated with the icon image or device image selected by the user as the remote control operation target device as shown in FIG. 10B.

  In this way, the icon image or device image is associated with the device in the device registration mode, and the device corresponding to the icon image or device image selected by the user based on the association information in the remote control operation mode. Can be identified. As a result, an efficient device selection process using the device selection screen as shown in FIGS. 10A and 10B can be realized.

  3A and 3B, the identification marker MK is attached to the devices ED1 and ED2 by adhesion or the like, but this embodiment is not limited to this. For example, as shown in FIG. 11A, the identification marker MK may be displayed on the display unit 400 included in the device ED2. In this case, the recognition unit 110 performs marker recognition processing of the identification marker MK displayed on the display unit 400 of the device ED2, and the remote control processing unit 130 performs remote control on the device ED2 based on the result of the marker recognition processing. For remote control.

  In this way, the identification marker MK can be used as a dynamic identification marker. That is, the display mode of the identification marker MK can be dynamically changed by the display control of the display unit 400 by the device ED2. As a result, it is possible to increase the amount of information transmitted by the identification marker MK, or to transmit appropriate information to the remote control device using the identification marker MK in a timely manner.

  For example, the device ED2 in FIG. 11A displays various types of identification markers MK on the display unit 400, so that various information such as device status information and operation manual information can be transmitted to the remote control device side. It becomes possible. That is, the remote control processing unit 130 can acquire at least one of device status information and device operation manual information based on the result of the marker recognition processing of the identification marker MK displayed on the display unit 400 of the device ED2. Then, as shown in FIG. 11B, the acquired device status information, operation manual information, and the like can be displayed on the display unit 30 of the remote control device RM. Therefore, the identification marker MK for device recognition can be used for acquiring various types of information such as device status information and operation manual information, and the user interface environment can be improved. .

  As described above, when device information or the like is acquired from the identification marker, instead of displaying the identification marker on the display unit of the device itself, as shown in FIG. An identification marker may be displayed on a display unit of a telephone, a portable game device, or the like. And the apparatus information etc. of an apparatus are acquired by performing the marker recognition process of the identification marker displayed on the display part of the portable information terminal in this way. In this way, the operation of attaching the identification marker to the device, the process of displaying the identification marker on the display unit of the device, and the like can be eliminated. In addition, the user downloads the information of the identification marker MK for acquiring the device information of the desired device via the network and displays the information on the display unit of the portable information terminal, and the device information is stored in the remote control database of the remote control device. It becomes possible to register, and the convenience of the user can be improved.

  As shown in FIG. 13, in the present embodiment, when the remote control device RM is directed to a device, the remote control operation screen of the device may be displayed on the display unit 30.

  Specifically, remote control processing unit 130 (display control unit 136) determines that remote control device RM is directed in the direction corresponding to the m-th device (for example, device ED1) among the plurality of devices. The display unit 30 is controlled to display the m-th remote control operation screen (remote control screen of the device ED1) for remote-controlling the m-th device. On the other hand, when it is determined that remote control device RM is directed in the direction corresponding to the n-th device (for example, device ED2, where m and n are natural numbers) among the plurality of devices, the n-th device is remote-controlled. Control to display the nth remote control operation screen (the remote control operation screen of the device ED2) for operation on the display unit 30 is performed.

  For example, in FIG. 13, when the remote control device RM is directed toward the device ED1 that is a TV, a TV remote control operation screen on which a TV operation icon (operation button icon) is displayed is displayed on the display unit 30. . When the remote controller RM is directed toward the device ED2 that is a DVD player, a DVD player remote control operation screen on which an operation icon for the DVD player is displayed is displayed on the display unit 30. The same applies to the case where the remote controller RM is directed to the device ED3 that is an air conditioner or the device ED4 that is a lighting fixture.

  The display unit 30 is configured by a touch panel display, for example. Therefore, for example, as shown in FIG. 13, when the user points the remote control device RM toward the device ED1 (TV) and touches the operation icon for the device ED1 displayed on the remote control operation screen, the remote control signal corresponding to the operation icon is displayed. Is generated and transmitted to the device ED1. Similarly, when the user points the remote controller RM toward the device ED2 (DVD player) and touches the operation icon for the device ED2 displayed on the remote control operation screen, a remote control signal corresponding to the operation icon is generated and the device ED2 Sent to. The same applies to the devices ED3 and ED4.

  Therefore, the user can efficiently operate the plurality of devices ED1 to ED4 by remote control using one remote control device RM. In addition, since the operation target device selection process is automatically performed simply by pointing the remote control device toward the device desired by the user, the convenience of the user can be significantly improved as compared with the conventional multi-function remote control device.

  In this embodiment, as shown in FIG. 2, the motion sensor 20 is built in the remote control device RM. Then, as shown in FIG. 14A, remote control device RM is set to the sleep mode in a stationary state (a state where it is not moved) in order to extend the battery life. As shown in FIG. 14B, for example, when the acceleration sensor 22 senses that the remote control device has been picked up by the user, the sleep mode is canceled. Thereby, the imaging unit 10, the recognition unit 110, the remote control processing unit 130, and the like are activated, and the system shifts to the normal operation mode. Then, based on the captured image from the imaging unit 10 and the sensor information from the motion sensor 20, a remote control operation target device selection process is performed.

  Specifically, when the remote control device RM is in the sleep mode and the movement of the remote control device is detected based on the sensor information from the motion sensor 20, the remote control processing unit 130 changes the operation mode from the sleep mode to the reference mode. Transition to the device selection mode and select the reference device. As will be described in detail later, the remote control processing unit 130 uses the i-th device (for example, the device ED1) in this reference device selection mode as a reference device (placement reference in the placement information) that serves as a selection reference for the operation target device. Device).

  Next, as shown in FIGS. 15A to 15C, the remote control processing unit 130 performs a remote control operation target device selection process based on the sensor information from the motion sensor 20.

  For example, it is assumed that the movement of the remote control device RM is detected based on the sensor information from the motion sensor 20 after the i-th device among the plurality of devices is selected as the remote control operation target device. For example, in FIG. 15A, after the device ED1 which is the i-th device is selected, the motion sensor 20 detects the movement of the remote control device RM as shown in FIG. 15B.

  In this case, the remote control processing unit 130 selects a j-th device (i and j are natural numbers) different from the i-th device as a remote control operation target device. Specifically, a device that exists in a direction corresponding to the direction in which remote control device RM is moved is selected as the j-th device. For example, in FIG. 15B, since the motion sensor 20 detects that the remote controller RM is moved in the right direction, the device ED2 existing in the direction corresponding to the right direction is selected as the j-th device. As a result, as described in FIG. 13, the j-th remote control operation screen for the j-th device can be displayed on the display unit 30. For example, when the i-th device is a TV and the j-th device is a DVD player, a TV remote control operation screen is displayed in FIG. 15A, and a DVD player remote control is displayed in FIG. The operation screen is displayed.

  In this case, the storage unit 50 in FIG. 2 stores device arrangement information. The remote control processing unit 130 selects the j-th device (device ED2) as a remote control operation target device based on the sensor information from the motion sensor 20 and the arrangement information stored in the storage unit 50.

  Specifically, the storage unit 50 stores the device of the device arranged in each partition area with respect to each section area of a plurality of partition areas obtained by dividing the device placement area, as in the matrix arrangement information described later. Information associated with the information is stored as arrangement information. Then, the remote control processing unit 130 selects, as the j-th device, a device associated with the partition area specified based on the sensor information from the motion sensor 20 among the plurality of partition areas.

  Further, the storage unit 50 stores device identification information and device remote control command information as device information. Here, the device identification information is information for identifying and specifying a device such as a model name or a manufacturer name. The remote control command information is command information necessary for remote control operation processing of the device, and is information used for generating a remote control signal for transmitting a command to the device.

  Further, it is assumed that the movement of the remote control device is detected based on the sensor information from the motion sensor 20 after the j-th device is selected as the remote control operation target device. In this case, the remote control processing unit 130 selects a kth device (k is a natural number) different from the jth device as a remote control operation target device.

  For example, in FIG. 15C, after the device ED2 that is the j-th device is selected, the motion sensor detects the movement of the remote control device RM, and thus the device ED3 that is the k-th device is selected. Then, as described in FIG. 13, the kth remote control operation screen for the kth device is displayed on the display unit 30. For example, when the k-th device is an air conditioner, a remote control operation screen for the air conditioner is displayed in FIG.

  FIG. 16A to FIG. 16C are examples of a device selection screen that informs the user of a device selection state. For example, when the device ED1 is selected as the operation target device (reference device) as shown in FIG. 15A, the device ED1 is in the selected state (selection candidate) as shown in FIG. A device selection screen for informing the user is displayed on the display unit 30. When the device ED2 is selected as the operation target device as shown in FIG. 15B, a device selection screen for notifying the user that the device ED2 is in the selected state is displayed as shown in FIG. 16B. Is done. Further, when the device ED3 is selected as the operation target device as shown in FIG. 15C, a device selection screen for notifying the user that the device ED3 is in the selected state is displayed as shown in FIG. Is done. In this way, the user can select which one of a plurality of devices is selected by the image recognition processing or the motion detection of the remote control device by selecting the device shown in FIGS. 16 (A) to 16 (C). It becomes possible to check on the screen, and the convenience of the user can be improved.

  As described above, in the present embodiment, for example, after the device ED1 (i-th device) serving as a reference device is selected by image recognition processing, the devices ED2 and ED3 are selected by the user moving the remote controller RM. Become so.

  For example, when a plurality of devices are shown in the captured image of the imaging unit 10, it is difficult to determine which device the user has selected by only the image recognition process. In this regard, in FIGS. 15A to 16C, when the user moves the remote controller RM, devices corresponding to the moved direction are sequentially selected. Therefore, after the reference device is selected by the image recognition process, the motion sensor 20 detects the movement of the remote controller RM reflecting the user's intention, and the final operation target device can be selected. Accordingly, it is possible to realize the device selection process with a small processing load as compared with the method of realizing all the device selections only by the image recognition processing.

3. Specific Processing Example Next, a specific processing example of the present embodiment will be described. First, a registration method of registration information such as operation commands and data of the remote control device, a selection / presentation method of an operation target device of the remote control device, and gesture operation correspondence will be described in order. In addition, since the carrier frequency and the infrared wavelength may be different in the above-described physical layer difference elimination, this can be dealt with by preparing a plurality of infrared LEDs as necessary. Alternatively, these can be optional installations.

  For the registration method of registration information such as operation commands and data of the remote control device, the model name (equipment identification information in a broad sense) and command system information (remote control command information in a broad sense) obtained via a memory medium are matrixed Registered as placement information (matrix information).

  The arrangement information in the form of a matrix is information obtained by matrix management of the vertical / horizontal arrangement positions of the remote control operation target devices based on the position of the reference point of the remote control device. The model name and command system information are obtained by reading a two-dimensional barcode (identification marker) by the imaging unit of the remote control device.

  That is, in the remote control device of this embodiment, the model name and command system information (operation information) are registered as a remote control database. By recognizing the position of the remote control device in the room at the time of use, each piece of device arrangement information of the remote control operation target device is registered as matrix arrangement information.

  For example, FIG. 17 shows an example of matrix arrangement information. As the arrangement information, the front surface, the right side surface, the back surface, the left side surface, the upper surface, and the lower surface are set on the assumption that the room is a quadrangular shape. The front surface is a reference surface on which the reference device is arranged. With respect to the upper surface and the lower surface, it is possible to select any one of the front surface, right side surface, back surface, and left side surface by swinging the remote control device up and down. The number of partition areas on one surface is 3 × 3 in FIG. However, it may be 5 × 5 or more.

  Set the reference device at the center of the reference surface, which is the front surface. In FIG. 17, the device ED1 is the reference device. For example, in FIG. 17, in addition to the device ED1, devices ED2 and ED3 are arranged on the front surface. On the left side, devices ED4 and ED5 are arranged. Similarly, the devices ED6 to ED10 are arranged on the right side, the back, the top, and the bottom. And the apparatus information is linked | related with the division area of each apparatus of these ED1-ED10.

  The current position and orientation of the remote control device are limited by selecting the reference device from a position approximately 1.5 m away from the center of the living room. In the subsequent selection of the target device for remote control operation, the vertical and horizontal swings of the remote control device are read from the acceleration sensor and the gyro sensor, and compared with the arrangement information shown in FIG. 17, the nearest device is selected and presented.

  The correspondence between the selection / presentation method of the remote control target device of the remote control device and the user's gesture motion is as follows. For example, a camera serving as an imaging unit is realized by a monochrome or color camera having 100,000 pixels or more. Then, image processing is performed on the captured image to detect a reference point. For example, image recognition processing such as shape recognition of the reference device is performed to detect a reference point where the reference device is arranged.

  Then, the relative position from the reference point is calculated by the detected reference point, the three-axis acceleration sensor built in the remote control device, and the two-axis gyro sensor, so that one or several remote control target devices are available. limit.

  Of the remote control operation target devices limited as described above on the display unit 30 of the remote control device whose direction has been changed by the user, the device most likely to be indicated by the user is shown in FIG. Displayed as shown in FIG. If the user performs remote control operation input as it is, the selected device is considered to be a device that the user wants to operate.

  If the selected device is different from the device that the user wants to operate, there should be a remote control target device on either the top, bottom, left, or right of the currently selected device. Therefore, the final remote control operation target device is specified by detecting, by the motion sensor, the user's gesture motion of shaking the remote control device in the vertical and horizontal directions.

  By performing image recognition processing on a captured image from the imaging unit, a reference rectangle is detected from a position about 1.5 m away from a reference wall surface of a room in which a plurality of devices are arranged. This rectangle is a reference point for determining the relative position of the operation target device. Then, the position of the remote control device in the room can be specified by the degree of deformation of the rectangle serving as a reference at the time of imaging by the imaging unit.

  For example, in FIG. 18A, ED1 to ED8 are devices arranged in a room. The face mark represents the position of the user. The center face mark shown in B1 indicates the position when the reference point is set, and when the reference device is the device ED1, the device ED1 is photographed approximately at the center at a position of about 1.5 m from the wall surface.

  Since the position of the face mark shown in FIG. 18A cannot be limited by the user who is the operator of the remote control device, the reference device ED1 (reference position) is first set from the current position of the user. Is specified.

  For example, when the user is at the face mark position indicated by B1 in FIG. 18A, the reference device ED1 looks like a rectangle as indicated by C1 in FIG. On the other hand, when the user is at the position of the face mark shown in B2, the reference device ED1 looks like a rectangle as shown in C2. Therefore, the position of the user can be specified by the rectangular shape of the reference device ED1. If the reference point can be specified, the relative position of each operation target device can be specified based on the reference point. Therefore, the operation target device can be estimated by detecting the direction in which the user is pointing the remote control device using a motion sensor. .

  As described above, in the multi-function remote control device of the present embodiment, first, the reference point is specified in the room to be used. Then, what kind of operation target device is arranged in the vertical and horizontal directions on the front is registered in the remote control device. For example, in FIG. 17, the devices ED1, ED2, and ED3 are registered as devices arranged on the front surface. Registration uses the touch panel display and keypad of the remote control device. Alternatively, it is possible to edit and register on a PC using a memory medium. In the same way, the devices are registered for the remaining five surfaces (right side, back, left side, top, and bottom).

  In this way, after the registration information registration process, the reference point specifying process performed by imaging the reference plane (front surface) is first performed, and after the position of the remote control apparatus is specified, the remote control apparatus holds in the user's hand. Remote control operation is possible as long as this is done.

  If the user does not perform an operation with the remote control device placed on a plane for a certain period of time or longer, the position information of the reference point is uncertain, and thus a reference point specifying process performed by imaging the reference plane is required. The reference point specifying process can be determined by the user picking up the remote control device, pointing it toward the reference plane, and inputting it with a confirmation button or the like. Alternatively, after the remote control device is directed to the reference plane, it may be determined after a certain period of time.

  As described above, in the present embodiment, the storage unit 50 stores device arrangement information as shown in FIG. The remote control processing unit 130 performs a remote control operation target device selection process based on the sensor information from the motion sensor 20 and the arrangement information stored in the storage unit 50. Here, as shown in FIG. 17, the arrangement information is information in which the device information of the devices arranged in each partition area is associated with each section area of the plurality of partition areas obtained by dividing the device placement area. ing. For example, the arrangement area corresponding to the front surface of FIG. 17 is divided into 3 × 3 partition areas. And the apparatus information of apparatus ED1 used as a reference | standard apparatus is linked | related with the center division area. In addition, device information of the devices ED2 and ED3 is associated with the right and upper right partition regions of the device ED1, respectively.

  Then, the remote control processing unit 130 selects a device associated with the partition area specified based on the sensor information from the motion sensor 20 as a remote control operation target device. For example, assume that after the device ED1 as the reference device is selected as shown in FIG. 15A, the user performs an operation of shaking the remote control device in the right direction as shown in FIG. Then, the motion of the remote control device at this time is detected by the motion sensor 20 (acceleration sensor), and the device ED2 on the right side of the device ED1 is selected based on the sensor information from the motion sensor 20 and the arrangement information in FIG. The

  Thereafter, for example, it is assumed that the user performs an operation of shaking the remote control device upward. Then, the motion of the remote control device at this time is detected by the motion sensor 20, and the device ED3 above the device ED2 is selected based on the sensor information from the motion sensor 20 and the arrangement information in FIG. . After the device ED3 is selected in this way, when the user performs a display instruction operation on the remote control operation screen, the remote control operation screen of the device ED3 is displayed on the display unit 30 of the remote control device.

4). Operation Next, the detailed operation of the present embodiment will be described with reference to the flowcharts of FIGS.

  FIG. 19 is an operation flowchart of a technique for identifying an operation target device by comparing registered feature value information obtained by marker recognition processing and device feature value information, and performing remote control processing.

  First, it is determined whether or not the device registration mode is set. If the device registration mode is set, a captured image of the device is acquired from the imaging unit 10 (steps S21 and S22). Then, as described with reference to FIG. 8, the marker recognition processing of the identification marker is performed based on the acquired captured image, and device information and device feature amount information are acquired (step S <b> 23). Then, the feature amount information acquired by the marker recognition process is stored as registered feature amount information in the storage unit in association with the device information acquired by the marker recognition process (step S24).

  On the other hand, when it is not the device registration mode but the remote control operation mode, a captured image of the device is acquired from the imaging unit 10 (steps S25 and S26). Then, device shape recognition processing is performed based on the captured image, and device feature amount information is extracted (step S27).

  Next, the registered feature value information stored in step S24 is read from the storage unit 50 and compared with the device feature value information extracted in step S27 (step S28). Then, the device information of the device whose device feature amount information matches the registered feature amount information is read from the storage unit 50 (step S29). That is, the device information obtained by the marker recognition process of the identification marker is read. Based on the read device information, remote control processing such as remote control signal generation processing and remote control operation screen display processing is executed (step S30).

  FIG. 20 is an operation flowchart of a technique for performing remote control processing by selecting a device using a device selection screen on which an icon image is displayed.

  First, it is determined whether or not the device registration mode is set. If the device registration mode is set, a captured image of the device is acquired from the imaging unit 10 (steps S41 and S42). Then, marker recognition processing is performed based on the captured image, and device information is acquired (step S43). And the apparatus information acquired by the marker recognition process is memorize | stored in the memory | storage part 50 as registration information (step S44).

  On the other hand, if the remote control operation mode is selected instead of the device registration mode, a device selection screen on which the device icon image or device image is displayed is displayed as shown in FIG. 10A (steps S45 and S46).

  Next, as shown in FIG. 10B, the device information of the device corresponding to the icon image or device image selected by the user is read from the storage unit 50 (step S47). That is, the device information obtained by the marker recognition process of the identification marker is read. Based on the read device information, remote control processing such as remote control signal generation processing and remote control operation screen display processing is executed (step S48).

  FIG. 21 is an operation flowchart of a technique for performing remote control processing by selecting a device using a motion sensor.

  First, in the sleep mode described with reference to FIG. 14A, it is determined whether or not the motion of the remote control device is detected by the motion sensor 20 (steps S1 and S2). That is, it is determined whether or not the movement of the remote control device from the stationary state is detected by the acceleration sensor 22. If a motion is detected, the mode shifts to the reference device selection mode as described with reference to FIG. 14B (step S3).

  Next, device registration information and arrangement information are read from the storage unit 50 (step S4). That is, the registration information as described in FIG. 7A and the arrangement information as described in FIG. 17 are read from the storage unit 50 (remote control database).

  Next, image recognition processing of the device based on the captured image from the imaging unit 10 is performed, and it is determined whether or not the reference device is image-recognized (steps S5 and S6). When the image of the reference device is recognized, for example, a remote control operation screen for the reference device is displayed (step S7).

  Next, it is determined whether or not the motion of the remote control device is detected by the motion sensor 20, and when the motion of the remote control device is detected, a device corresponding to the moved direction is selected (steps S8 and S9). . For example, after the reference device ED1 is selected in FIG. 15A, when the motion of the remote control device is detected by the motion sensor 20 as shown in FIG. 15B, the device ED2 corresponding to the moved direction is selected. .

  Next, it is determined whether or not a display instruction operation has been performed by the user (step S10). That is, it is determined whether or not an operation for instructing display of the remote control operation screen has been performed. When a display instruction operation is performed, a remote control operation screen for the selected device is displayed (step S11). For example, when the user performs a display instruction operation with the device ED2 selected as shown in FIGS. 15B and 16B, a remote control operation screen for operating the device ED2 is displayed. Further, when the user performs a display instruction operation in a state where the device ED3 is selected as shown in FIGS. 15C and 16C, a remote control operation screen for operating the device ED3 is displayed. By doing so, as shown in FIG. 13, it becomes possible to display the remote control operation screen of the device corresponding to the direction in which the user points the remote control device.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described at least once together with a different term having a broader meaning or the same meaning in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. The configuration and operation of the remote control device are not limited to those described in the present embodiment, and various modifications can be made.

RM remote control device, ED1-ED10 equipment, MK identification marker,
10 imaging unit, 12 optical system, 14 imaging element, 20 motion sensor,
22 acceleration sensor, 24 gyro sensor, 28 operation unit, 30 display unit,
40 infrared light emitting section, 50 storage section, 60 information storage medium, 62 external I / F section,
100 processing unit, 110 recognition unit, 112 shape recognition unit, 114 marker recognition unit,
120 sensor information acquisition unit, 130 remote control processing unit, 132 device selection unit,
134 Remote control signal generator, 136 Display controller, 138 Communication processor

Claims (14)

An imaging unit for imaging the device;
A recognition unit that performs device recognition processing;
A remote control processing unit for performing remote control processing for remote control of the device;
Including
The recognition unit
By performing image recognition processing of the captured image, marker recognition processing of an identification marker of a device reflected in the captured image is performed,
The remote control processing unit
A remote control device for performing a remote control process for remote-controlling a device based on a result of the marker recognition process. In claim 1,
The remote control processing unit
In the device registration mode, the device information of the device obtained by the marker recognition process is registered as registration information. In claim 2,
A storage unit for storing the registration information including the device information;
The remote control processing unit
In the remote control operation mode, the device information of a remote control operation target device is acquired by reading the registration information from the storage unit. In claim 3,
The recognition unit
In the device registration mode, by performing image recognition processing of the captured image, perform shape recognition processing of the device reflected in the captured image,
The remote control processing unit
In the device registration mode, the device feature amount information obtained by the shape recognition process in the device registration mode is registered as device feature information.
The storage unit
Information associating the device information obtained by the marker recognition process with the registered feature amount information obtained by the shape recognition process is stored as the registration information,
The recognition unit
In the remote control operation mode, by performing image recognition processing of the captured image, performing shape recognition processing of the device reflected in the captured image,
The remote control processing unit
In the remote control operation mode, a comparison process is performed between device feature amount information, which is feature amount information obtained by the shape recognition process in the remote control operation mode, and the registered feature amount information, and the device feature amount information is A remote control device, wherein the device information of a remote control operation target device is acquired by reading out the device information of a device that matches registered feature amount information from the storage unit. In claim 3,
The remote control processing unit
In the device registration mode, the device feature amount information acquired from the identification marker by the marker recognition process in the device registration mode is registered as device feature information.
The storage unit
Information associating the device information obtained by the marker recognition process with the registered feature amount information obtained by the marker recognition process is stored as the registration information,
The recognition unit
In the remote control operation mode, by performing image recognition processing of the captured image, performing shape recognition processing of the device reflected in the captured image,
The remote control processing unit
In the remote control operation mode, a comparison process is performed between device feature amount information, which is feature amount information obtained by the shape recognition process in the remote control operation mode, and the registered feature amount information, and the device feature amount information is A remote control device, wherein the device information of a remote control operation target device is acquired by reading out the device information of a device that matches registered feature amount information from the storage unit. In claim 4 or 5,
Including a motion sensor to detect the movement of the remote control device,
The remote control processing unit
A remote control device for performing remote control processing for remote-controlling a device based on a result of the shape recognition processing and sensor information from the motion sensor. In claim 3,
A display unit for displaying a plurality of icon images corresponding to a plurality of devices or a plurality of device images of the plurality of devices;
The remote control processing unit
A remote control device, wherein a device corresponding to an icon image or a device image selected by a user from the plurality of icon images or the plurality of device images is selected as a remote control operation target device. In any one of Claims 1 thru | or 7,
The recognition unit
A remote control device that performs the marker recognition processing of the identification marker attached to a device. In any one of Claims 1 thru | or 7,
The recognition unit
A remote control device that performs the marker recognition processing of the identification marker displayed on a display unit of a device. In claim 9,
The remote control processing unit
A remote control device that acquires at least one of device status information and device operation manual information based on a result of the marker recognition processing of the identification marker displayed by a display unit of the device. In any one of Claims 1 thru | or 10.
The remote control processing unit
As the remote control process, a remote control apparatus generates a remote control signal generation information for operating a remote control target device by remote control. In any one of Claims 1 thru | or 11,
The remote control processing unit
As the remote control process, a remote control apparatus is provided that performs control to display a remote control operation screen for performing remote control operation on a remote control target device on a display unit. In claim 12,
The remote control processing unit
When it is determined that the remote control device is directed in the direction corresponding to the m-th device among the plurality of devices, the display displays the m-th remote control operation screen for operating the m-th device by remote control. Control to display in the
When it is determined that the remote control device is directed in the direction corresponding to the n-th device (m and n are natural numbers) among the plurality of devices, the n-th device for remote-controlling the n-th device A remote control device for controlling the remote control operation screen to be displayed on the display unit. An imaging unit for imaging the device;
A recognition unit that performs device recognition processing;
As a remote control processing unit that performs remote control processing for remote control of equipment,
Make the computer work,
The recognition unit
By performing image recognition processing of the captured image, marker recognition processing of an identification marker of a device reflected in the captured image is performed,
The remote control processing unit
A program for performing remote control processing for remote-controlling a device based on a result of the marker recognition processing.

Images