JPWO2018011890A1 - Control system and device control method - Google Patents

Abstract

The terminal device detects a specific image from the image captured by the image sensor (step S101), and transmits image detection information based on the specific image to the control device (step S102). Upon receiving the image detection information transmitted from the terminal device, the control device selects a control candidate device from a plurality of devices based on the image detection information and generates operation information related to the control candidate device (step S103). Transmit to the device (step S104). When receiving the operation information transmitted from the control device, the terminal device notifies the user of the content of the operation information (step S105). The control device determines whether or not control over the control candidate device is permitted by the user (step S106), and if permitted, transmits a control command to the control candidate device (step S107).

Description

  The present invention relates to a control system and a device control method.

  A technique for operating a device connected to a network using information indicated by a barcode is known (for example, Patent Document 1).

  In the digital home appliance remote control system disclosed in Patent Document 1, a bar code is read by a bar code reader connected to a mobile phone, and the digital home appliance is operated according to the information code described in the read bar code.

JP 2004-110148 A

  Incidentally, in recent years, wearable computers (also referred to as wearable devices and wearable terminals) are becoming popular. Such wearable computers are mainly used for entertainment and health management, and further, researches are being conducted to operate devices using information measured by sensors provided in the wearable computers.

  However, since the sensor included in the wearable computer is assumed to be constantly activated, that is, constantly measured, when used for device operation as described above, an operation unintended by the user is automatically performed. There are concerns.

  SUMMARY An advantage of some aspects of the invention is that it provides a control system and the like that can avoid execution of an operation unintended by a user.

In order to achieve the above object, a control system according to the present invention provides:
A control system that controls a plurality of devices, and a control system including a terminal device,
The terminal device
An image sensor;
Specific image detection means for detecting a specific image that matches a predetermined condition from an image captured by the image sensor;
Detection information transmitting means for transmitting image detection information based on the specific image to the control device;
When receiving the operation information transmitted from the control device, a notification means for notifying the user of the content of the operation information,
The control device includes:
When the image detection information transmitted from the terminal device is received, based on the image detection information, a control candidate device is selected from the plurality of devices, and operation information related to the control candidate device is generated, and the terminal device Operation information transmitting means for transmitting;
Device control means for determining whether or not control of the control candidate device is permitted by the user and, if permitted, controlling the control candidate device based on the operation information.

  In the present invention, the terminal device transmits image detection information based on the detected specific image to the control device. The control device selects a control candidate device based on the image detection information, generates operation information related to the control candidate device, and transmits the operation information to the terminal device. The terminal device notifies the user of the content of the operation information, and the control device controls the control candidate device when the control is permitted by the user. Therefore, it is possible to avoid an operation unintended by the user on the device.

It is a figure which shows the structure of the control system which concerns on Embodiment 1 of this invention. It is a block diagram which shows the hardware constitutions of a control apparatus. It is a figure for demonstrating the table etc. which are memorize | stored in the secondary storage device with which a control apparatus is provided. 2 is a block diagram illustrating a hardware configuration of a terminal device according to Embodiment 1. FIG. It is a figure which shows the function structure of the terminal device which concerns on Embodiment 1. FIG. It is a figure which shows the function structure of a control apparatus. 3 is a flowchart illustrating a procedure of device control processing according to the first embodiment. It is a figure which shows a mode that the specific image is contained in the user's visual field. It is a figure which shows the example of a projection of the operation information image | video in Embodiment 1. FIG. FIG. 6 is a diagram for explaining a first modification of the first embodiment. FIG. 6 is a diagram (No. 1) for describing a second modification of the first embodiment; FIG. 6 is a diagram (No. 2) for describing the second modification of the first embodiment. FIG. 10 is a diagram (No. 1) for describing a third modification of the first embodiment; FIG. 10 is a second diagram for explaining the third modification of the first embodiment. FIG. 10 is a diagram for explaining a fourth modification of the first embodiment. It is a block diagram which shows the hardware constitutions of the terminal device which concerns on Embodiment 2 of this invention. FIG. 10 is a diagram for explaining a control permission instruction in the second embodiment. FIG. 10 is a diagram for explaining a first modification of the second embodiment. FIG. 10 is a diagram for explaining a second modification of the second embodiment. It is a figure for demonstrating other embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(Embodiment 1)
FIG. 1 is a diagram illustrating a configuration of a control system 1 according to the first embodiment of the present invention. The control system 1 is a so-called HEMS (Home Energy Management System) system that manages electric power used in a general home.

  As illustrated in FIG. 1, the control system 1 includes a control device 2, a terminal device 3, and a plurality of devices 4 (devices 4-1, 4-2,...). The control device 2 is installed at an appropriate location in the house H, monitors the power consumed in this home (demand area), and displays the power consumption status. In addition, the control device 2 performs control of each device 4 and monitoring of an operation state.

  The device 4 (devices 4-1, 4-2,...) Is an electrical device such as an air conditioner, an illuminator, a television, a water heater, or an IH (Induction Heating) cooker. Each device 4 is installed in a house H (including a site) and connected to a power line that supplies power from a commercial power source, a power generation facility, a power storage facility, or the like (all not shown). Each device 4 is communicably connected to the control device 2 via a wireless network (not shown). This wireless network is, for example, a network conforming to ECHONET Lite. Depending on the specifications of the device 4, it may be connected to this wireless network via an external communication adapter (not shown).

  In response to the request from the control device 2, each device 4 transmits data (operation state data) storing information indicating the current operation state to the control device 2.

  As shown in FIG. 2, the control device 2 includes a CPU (Central Processing Unit) 20, a communication interface 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, and a secondary storage device 24. . These components are connected to each other via a bus 25. The CPU 20 controls the control device 2 in an integrated manner. Details of functions realized by the CPU 20 will be described later.

  The communication interface 21 includes a network card for wirelessly communicating with each device 4 via the wireless network described above and an IC chip for wirelessly communicating with the terminal device 3.

  The ROM 22 stores a plurality of firmware, data used when executing these firmware, and the like. The RAM 23 is used as a work area for the CPU 20.

  The secondary storage device 24 includes an EEPROM (Electrically Erasable Programmable Read-Only Memory), a readable / writable nonvolatile semiconductor memory such as a flash memory, an HDD (Hard Disk Drive), or the like. As shown in FIG. 3, the secondary storage device 24 stores a remote operation program 240 and device information 241. In addition to this, the secondary storage device 24 includes various programs including a program for monitoring the operation state of the device 4, a program for monitoring the power consumed in the home, and the like. Stores data used during program execution.

  The remote operation program 240 is a computer program executed by the CPU 20. The remote operation program 240 describes processing for operating the device 4 via the terminal device 3.

  The device information 241 is information related to the device 4 and includes a connection information table 2410, a location information table 2411, an operation information table 2412, and an operation code table 2413. The connection information table 2410 is a data table that stores connection information of the device 4. The connection information is information for communicating with the device 4, for example, the address of the device 4.

  The location information table 2411 is a data table that stores location information of the device 4. The location information is information indicating a location where the device 4 is installed. More specifically, the location information is information for identifying a room in the house H. The information for identifying a room is, for example, an ID (identification) previously assigned to each room or a name of each room (“living room”, “kitchen”, etc.).

  The operation information table 2412 is a data table that stores the current operation state of the device 4. The operation state includes, for example, a power supply state (power on / off), operation presence / absence (during operation / stop), abnormality presence / absence (abnormal / normal), operation mode, setting information, and the like. The operation mode indicates an operation method. For example, in the case of an air conditioner, cooling, heating, ventilation, dehumidification, and the like correspond to the operation mode, and in the case of an illuminator, normal illumination, power saving illumination, and the like correspond to the operation mode. In the case of an air conditioner, the setting information corresponds to a set temperature (also referred to as a target temperature), an air volume, and the like, and in the case of an illuminator, a brightness level or the like. The CPU 20 updates the operation information table 2412 based on the operation state data acquired from each device 4.

  The operation code table 2413 is a data table in which operation codes are associated with information for identifying the device 4 (for example, device ID and address) and operation details of the device 4. The operation code is, for example, a four-byte character string combining alphabets and numbers. For example, “L001” indicates the power on / off operation of the illuminator installed in the living room, and “L002” Indicates the power on / off operation of the air conditioner installed in the living room, and “K013” indicates the power saving function on / off operation of the refrigerator installed in the kitchen.

  Returning to FIG. 1, the terminal device 3 is a so-called wearable computer (also referred to as a wearable device or a wearable terminal). .)

  As illustrated in FIG. 4, the terminal device 3 includes a CPU 30, a communication interface 31, an image sensor 32, a video projection unit 33, a ROM 34, a RAM 35, and a secondary storage device 36. These components are connected to each other via a bus 37.

  The CPU 30 comprehensively controls the terminal device 3. Details of the functions realized by the CPU 30 will be described later. The communication interface 31 includes an IC chip for wireless communication with the control device 2.

  The image sensor 32 is an image sensor such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and the terminal device 3 can capture an image of a subject in the field of view of the user wearing the terminal device 3. It is provided outside the frame. That is, the imaging range (field of view) of the image sensor 32 is equivalent to the field of view of the user wearing the terminal device 3.

  The image projection unit 33 projects an image on at least one of the right-eye lens and the left-eye lens in the terminal device 3 under the control of the CPU 30. These lenses in the terminal device 3 employ a half mirror made of a dielectric multilayer film, and have a structure that transmits light from the front and reflects an image by the image projection unit 33. As a result, the user can see the projected image while viewing the external environment while wearing the terminal device 3. That is, the user wearing the terminal device 3 can experience so-called augmented reality (AR).

  The ROM 34 stores a plurality of firmware, data used when executing these firmware, and the like. The RAM 35 is used as a work area for the CPU 30.

  The secondary storage device 36 is configured by a readable / writable nonvolatile semiconductor memory such as an EEPROM or a flash memory. The secondary storage device 36 stores various programs including a remote operation program describing processing for remotely operating the device 4 via the control device 2 and data used when executing these programs.

  Functionally, the terminal device 3 includes a specific image detection unit 300, a detection information transmission unit 301, and a notification unit 302 as illustrated in FIG. These functional units are realized by the CPU 30 executing a remote operation program stored in the secondary storage device 36.

  The specific image detection unit 300 detects a specific image that matches a predetermined condition from images captured by the image sensor 32 at regular time intervals (for example, at intervals of 0.5 seconds). In this embodiment, the specific image is a two-dimensional code, and more specifically, a matrix-type two-dimensional code. The specific image may be a stack type two-dimensional code or a one-dimensional code. Alternatively, the specific image is not limited to that based on a known standard, and may be defined for use in the control system 1.

  Each time the specific image is detected by the specific image detection unit 300, the detection information transmission unit 301 decodes the detected specific image and stores the decoded result (the operation code described above in the present embodiment). (Image detection information) is transmitted to the control device 2.

  When the notification unit 302 receives later-described operation information transmitted from the control device 2, the notification unit 302 notifies the user of the content of the operation information. More specifically, the notification unit 302 projects an image indicating the content of the operation information on a predetermined position of the lens of the terminal device 3.

  Next, the functional configuration of the control device 2 will be described. Functionally, the control device 2 includes an operation information transmission unit 200 and a device control unit 201 as shown in FIG. These functional units are realized by the CPU 20 executing the remote operation program 240 stored in the secondary storage device 24.

  The operation information transmission unit 200 includes a candidate determination unit 2000 and an operation information generation unit 2001. When the candidate determination unit 2000 receives the above-described image detection information from the terminal device 3, the candidate determination unit 2000 extracts an operation code from the received image detection information. Then, the candidate determining unit 2000 refers to the operation code table 2413 using the extracted operation code, and determines the device 4 (control candidate device) to be a control target candidate and the operation content of the device 4.

  For example, when the operation code is “L001”, the candidate determination unit 2000 determines an illuminator installed in the living room (hereinafter referred to as a living illuminator) as a control candidate device, and turns the power on / off. It is determined as the operation content of the lighting device in the living room.

  The operation information generation unit 2001 refers to the operation information table 2412 and acquires the current operation state corresponding to the determined operation content in the determined control candidate device. Then, the operation information generation unit 2001 generates operation information for notifying the terminal device 3 based on the determined control candidate device and its operation content and the acquired current operation state of the control candidate device.

  For example, if the living room illuminator is a control candidate device, the operation is turned on / off, and the living room illuminator is currently off, the operation information includes the living illuminator turned off to on. Information indicating that the control is to be performed is stored. The operation information generation unit 2001 transmits the operation information generated in this way to the terminal device 3.

  The device control unit 201 includes a permission determination unit 2010 and a control command transmission unit 2011. The permission determination unit 2010 determines whether or not the user is permitted to control the control candidate device based on the image detection information transmitted from the terminal device 3 after the operation information is transmitted.

  More specifically, the permission determination unit 2010 sets the image detection information having the same content (that is, the same operation code) as the image detection information corresponding to the operation information previously transmitted to the terminal device 3 for a predetermined period (first operation). 1 period), it is determined whether or not it has been continuously received from the terminal device 3. And when it receives continuously, it will discriminate | determine that control with respect to the said control candidate apparatus was permitted noting that permission conditions were satisfied. The first period is, for example, 10 seconds.

  Further, the permission determination unit 2010 can receive the image detection information having the same content as the image detection information corresponding to the operation information transmitted to the terminal device 3 once during a predetermined period (second period). If not, it is determined that the non-permission condition is satisfied, and control of the control candidate device is not permitted. The second period is, for example, 10 seconds.

  The permission determination unit 2010 determines whether or not permission has been given as described above after the operation information is transmitted to the terminal device 3 by the operation information generation unit 2001, that is, the content of the operation information is transmitted to the user in the terminal device 3. This is performed during a predetermined period (waiting for permission) after the notification. The permission waiting period is, for example, 30 seconds. If neither the permission condition nor the non-permission condition is satisfied during the permission waiting period, the permission determination unit 2010 determines that control of the control candidate device is not permitted.

  When the permission condition is satisfied, the control command transmission unit 2011 generates a control command for controlling the control candidate device and transmits the control command to the control candidate device. More specifically, the control command transmission unit 2011 generates a control command for controlling the control candidate device with the content indicated by the operation information.

  FIG. 7 is a flowchart illustrating a procedure of device control processing in remote operation, which is executed by the control system 1. When a specific image (that is, a two-dimensional code) printed on an object such as paper enters the visual field of the user wearing the terminal device 3 (user visual field through the lens) (see FIG. 8), the specific image of the terminal device 3 The detection unit 300 detects the specific image (step S101).

  The detection information transmission unit 301 of the terminal device 3 decodes the specific image detected by the specific image detection unit 300, and transmits the image detection information storing the decoded result (that is, the operation code) to the control device 2 (step). S102).

  When the operation information transmission unit 200 (candidate determination unit 2000, operation information generation unit 2001) of the control device 2 receives the image detection information from the terminal device 3, the operation information transmission unit 200 based on the operation code stored in the received image detection information. Then, control candidate devices are determined and operation information is generated (step S103). And the operation information transmission part 200 transmits the produced | generated operation information to the terminal device 3 (step S104).

  When the notification unit 302 of the terminal device 3 receives the operation information from the control device 2, the notification unit 302 notifies the user of the content of the operation information (step S105). More specifically, the notification unit 302 projects an image (operation information image) indicating the content of the operation information onto the lens of the terminal device 3. FIG. 9 shows an example in which the operation information video is projected.

  When permitting control on a control candidate device (here, when permitting control to turn on a living room illuminator from off), at least the first within the permission waiting period (for example, within 30 seconds) described above. Care is taken that the two-dimensional code is in the user's field of view (view) for one period (eg, 10 seconds). Specifically, the user does not move the head or move the object on which the two-dimensional code is printed so that the two-dimensional code is out of the user's field of view.

  On the other hand, if the control of the control candidate device is not permitted, the user moves the head or moves the head so that the two-dimensional code is out of the user's field of view for the second period (for example, 10 seconds). Just move the object with the two-dimensional code printed on it.

  The permission determination unit 2010 of the control device 2 transmits the operation information to the terminal device 3 (that is, after the content of the operation information is notified by the terminal device 3), and then the image detection information sent from the terminal device 3. Is monitored to determine whether or not control of the control candidate device is permitted (step S106).

  When it is determined that the control is permitted (step S106; YES), the control command transmission unit 2011 generates a control command for controlling the control candidate device with the content of the operation information generated by the operation information generation unit 2001. Then, it transmits to the control candidate device (step S107).

  As described above, the control system 1 according to the first embodiment of the present invention can control the device 4 using a specific image, that is, a two-dimensional code. Excellent in properties. Even if the specific image is detected by the terminal device 3, the control device 2 does not immediately control the corresponding device 4, but generates operation information corresponding to the specific image and transmits it to the terminal device 3. And the terminal device 3 alert | reports the content of the operation information from the control apparatus 2 to a user. The control device 2 executes the control of the device 4 (control candidate device) only when the user permits the execution of the control. Therefore, it is possible to avoid a situation in which an operation unintended by the user is performed on the device 4, and an effect of preventing an erroneous operation on the device 4 can be obtained.

  Further, since it is possible to instruct the control device 2 with simple operations without performing a special operation, permission or non-permission of the execution of control, the remote operation service of the device 4 using such a terminal device 3 Can be provided to a wide range of users.

  The first embodiment may be modified as follows.

(Modification 1)
In addition to the content of the operation information, the notification unit 302 of the terminal device 3 may notify the user of permission and non-permission instruction methods as shown in FIG.

(Modification 2)
In the first embodiment, when the operation information video is projected, the user can control the specific image (two-dimensional code) in the user's field of view for the first period (for example, 10 seconds). Instructed permission. However, the instruction method is not limited to this.

  For example, the user may be able to instruct permission of control by moving the position of the current specific image in the field of view to a predetermined region in the field of view. In this case, when receiving the operation information from the control device 2, the notification unit 302 of the terminal device 3 projects an operation information image on the lens of the terminal device 3 and uses a destination frame 1101 as shown in FIG. 11 as the lens. Project.

  When the user permits the execution of the control, the user moves the head or moves the position of the object on which the specific image is printed, so that the user's field of view (that is, within the imaging range of the image sensor 32). The position of the specific image in is moved to the area surrounded by the destination frame 1101 (see FIG. 12).

  In this modification, the detection information transmission unit 301 of the terminal device 3 uses the image detection information to be transmitted to the control device 2 during a predetermined period (same period as the permission waiting period) after the operation information video is projected. In addition to the result of decoding the specific image detected by the specific image detection unit 300, the position information of the specific image is stored. The specific image position information is information indicating the position of the specific image in the image (captured image) captured by the image sensor 32.

  Further, the permission determination unit 2010 of the control device 2 in this modification determines whether or not the control is permitted by the user based on the position information of the specific image included in the received image detection information. That is, when the specific image is located in the area corresponding to the projection position of the movement destination frame 1101 in the captured image, the permission determination unit 2010 determines that the permission condition is satisfied, and allows the control candidate device to control. It is determined that it has been done. It is assumed that data indicating the correspondence between the captured image area and the projection position of the movement destination frame 1101 is stored in advance in the secondary storage device 24 of the control device 2.

(Modification 3)
In the first embodiment, the control in the case where one specific image enters the user's field of view (the imaging range of the image sensor 32), that is, the control when there is one control candidate device has been described. However, as shown in FIG. 13, there may be a case where a plurality of specific images fall within the user's field of view. In such a case, when receiving the operation information transmitted from the control device 2, the notification unit 302 of the terminal device 3 in this modified example projects the operation information video in the vicinity of the corresponding specific image (see FIG. 14). ).

  In this way, even when a plurality of specific images are detected, the user can grasp the correspondence between each specific image and the content of the operation information.

  In this modification, as in Modification 2 described above, the user moves the position of any specific image in the user's field of view to the area surrounded by the movement destination frame 1101 projected onto the lens. Thus, the control device 2 can be instructed to permit control of the control candidate device corresponding to the specific image. In this modification, the permission waiting period is not fixed and may be changed according to the number of control candidate devices. For example, when there is one control candidate device, the permission waiting period may be 30 seconds, and when there are three control candidate devices, the permission waiting period may be 60 seconds.

(Modification 4)
In the control device 2, after the candidate determining unit 2000 determines the control candidate device and its operation content, before the operation information generating unit 2001 transmits the operation information to the terminal device 3, whether or not the control candidate device can be controlled is determined. You may make it judge. Specifically, the operation information transmission unit 200 of the control device 2 further includes a control availability determination unit (not shown) that determines whether control of the control candidate device is possible.

  For example, the control availability determination unit refers to the operation information table 2412 to determine whether an abnormality is detected in the control candidate device. When an abnormality is detected, the control availability determination unit determines that the control candidate device cannot be controlled. In addition, for example, when a predetermined operation prohibition period has not elapsed since the most recent operation performed on the control candidate device, the control availability determination unit determines that the control candidate device cannot be controlled. To do. The operation prohibition period may be common to all the devices 4, or may be set for each type of device 4 or individually.

  Further, for example, when the control candidate device is in the middle of a pre-scheduled operation or immediately before the start of a pre-scheduled operation (for example, within 10 minutes before the start time), the control availability determination unit It is determined that control of the control candidate device is not possible.

  If the control information determining unit 2001 determines that control is impossible, the operation information generating unit 2001 generates operation information indicating that the current operation state of the control candidate device and control are not possible, and transmits the operation information to the terminal device 3. Send. An example of the contents of the operation information notified by the terminal device 3 in this case is shown in FIG.

  The permission determination unit 2010 determines whether control is permitted or not even when the image detection information corresponding to the control candidate device determined to be uncontrollable by the control permission determination unit is received from the terminal device 3. The image detection information is discarded without performing the processing relating to the above.

  In this way, by determining whether or not the control candidate device can be controlled, the device 4 that is not preferably operated via the terminal device 3 from the viewpoint of safety, energy saving, comfort, and the like is excluded from the operation target. Is possible.

(Embodiment 2)
Subsequently, Embodiment 2 of the present invention will be described. In the following description, components and the like that are common to the first embodiment are denoted by the same reference numerals, and description thereof is omitted.

  FIG. 16 is a block diagram illustrating a hardware configuration of the terminal device 3A according to the second embodiment. In the terminal device 3A, a line-of-sight detection unit 38 is added in addition to the configuration of the terminal device 3 of the first embodiment. The line-of-sight detection unit 38 includes an electrooculogram sensor and detects the line of sight of the user wearing the terminal device 3A. Note that the line-of-sight detection unit 38 may include an image sensor that captures an image of the user's eye, and may detect the user's line of sight by analyzing the captured image of the eye.

  When the notification unit 302 of the terminal device 3A receives the operation information from the control device 2, the operation information video is projected onto the lens of the terminal device 3A and a mark 1701 as shown in FIG. 17 is projected onto the lens. When permitting execution of the control, the user looks at the mark 1701 for a predetermined period (third period). The third period is shorter than the first and second periods described above, for example, 3 seconds.

  The detection information transmission unit 301 of the terminal device 3A includes the specific image detection unit in the image detection information transmitted to the control device 2 during a predetermined period (same period as the permission waiting period) after the operation information video is projected. In addition to the result of decoding the specific image detected by 300, the line-of-sight information is stored. The line-of-sight information is information indicating the detection value of the line-of-sight detection unit 38.

  The permission determination unit 2010 of the control device 2 of the present embodiment determines whether or not the control is permitted by the user by monitoring the line-of-sight information included in the received image detection information. More specifically, the permission determination unit 2010 determines whether or not the user is looking at the mark 1701 from the line-of-sight information, and the state in which the user is looking at the mark 1701 is a predetermined period (the same period as the third period). ), It is determined that the control for the control candidate device is permitted, assuming that the permission condition is satisfied. It is assumed that data indicating the correspondence between the detection value of the line-of-sight detection unit 38 and the projection position of the mark 1701 is stored in advance in the secondary storage device 24 of the control device 2.

  On the other hand, when the permission condition is not satisfied during the permission waiting period, the permission determining unit 2010 determines that the control of the control candidate device is not permitted.

  When the above permission condition is satisfied, the control command transmission unit 2011 generates a control command for controlling the control candidate device with the content of the operation information generated by the operation information generation unit 2001, and transmits the control command to the control candidate device. To do.

  As described above, in the control system 1 according to Embodiment 2 of the present invention, the device 4 can be controlled using a specific image, that is, a two-dimensional code. Excellent in properties. Even if the specific image is detected by the terminal device 3 </ b> A, the control device 2 does not immediately control the corresponding device 4, generates operation information corresponding to the specific image, and transmits the operation information to the terminal device 3. Then, the terminal device 3A notifies the user of the contents of the operation information from the control device 2. The control device 2 executes the control of the device 4 (control candidate device) only when the user permits the execution of the control. A situation in which an operation unintended by the user is performed on the device 4 can be avoided, and an effect of preventing an erroneous operation on the device 4 can be obtained.

  Further, the user can instruct the control device 2 to permit or disallow the execution of the control only by changing the line of sight. Therefore, it is possible to provide a wider range of users with a remote operation service of the device 4 using such a terminal device 3.

  The second embodiment may be modified as follows.

(Modification 1)
In the second embodiment, when the control content is notified, the user instructs the permission of the control by staring at the mark 1701 projected on the lens of the terminal device 3 during the third period. However, the instruction method is not limited to this.

  For example, the notification unit 302 of the terminal device 3A projects not the mark 1701 but marks 1801A and 1801B indicating the movement destinations of the line of sight when allowing or not allowing onto the lens of the terminal device 3A as shown in FIG. May be.

  When the user permits the control, the user looks at the mark 1801A during the third period. On the other hand, when the user does not permit the control, the user looks at the mark 1801B.

  In this modification, the permission determination unit 2010 of the control device 2 determines whether the user is looking at the mark 1801A from the line-of-sight information included in the received image detection information. When the state where the user is looking at the mark 1801A continues for the third period, the permission determining unit 2010 determines that the control for the control candidate device is permitted, assuming that the permission condition is satisfied.

  Further, the permission determination unit 2010 determines whether the user is looking at the mark 1801B from the line-of-sight information. When the state where the user is looking at the mark 1801B continues during the third period, the permission determining unit 2010 determines that the control for the control candidate device is not permitted, assuming that the disallowed condition is satisfied.

  It is assumed that data indicating the correspondence between the detection value of the line-of-sight detection unit 38 and the projection positions of the marks 1801A and 1801B is stored in advance in the secondary storage device 24 of the control device 2.

  If neither the permission condition nor the non-permission condition is satisfied during the permission waiting period, the permission determination unit 2010 determines that control of the control candidate device is not permitted.

(Modification 2)
In the second embodiment, the control when one specific image enters the user's field of view (the imaging range of the image sensor 32), that is, when the number of control candidate devices is one has been described. However, as shown in FIG. 13, there may be a case where a plurality of specific images fall within the user's field of view. In such a case, the notification unit 302 of the terminal device 3A in this modified example receives the operation information from the control device 2 as in the case of the modified example 3 of the first embodiment, and closes it to the corresponding specific image. An operation information image is projected (see FIG. 19).

  In this modification, when permitting the control, the user looks at the operation information video corresponding to the control candidate device during the fourth period. The fourth period is shorter than the first and second periods described above and longer than the third period, for example, 5 seconds. In addition, the notification unit 302 changes the projection mode of the operation information video to which the user's line of sight is directed from the others. For example, in the example of FIG. 19, it is shown that the user's line of sight is directed to the operation information video corresponding to the kitchen illuminator that is the control candidate device. In this way, the user can easily and accurately recognize where his / her line of sight is directed.

  As described above, the detection information transmission unit 301 of the terminal device 3A uses the image detection information to be transmitted to the control device 2 during a predetermined period (same period as the permission waiting period) after the operation information video is projected. In addition to the result of decoding the specific image detected by the specific image detection unit 300, the line-of-sight information is stored. However, the line-of-sight information in this modification is information indicating whether or not the user's line of sight is directed to the operation information video corresponding to the specific image.

  The permission determination unit 2010 of the control device 2 in this modification monitors the line-of-sight information included in the received image detection information, and determines whether the user is viewing any one operation information video. If the state where any one of the operation information images is viewed continues for a predetermined period (fourth period), the permission determination unit 2010 determines that the permission condition is satisfied and responds to the image detection information. It is determined that control of the control candidate device to be permitted is permitted.

  As in the case of the third modification of the first embodiment, the permission waiting period is not fixed and may be changed according to the number of control candidate devices. For example, when there is one control candidate device, the permission waiting period may be 30 seconds, and when there are three control candidate devices, the permission waiting period may be 60 seconds.

(Modification 3)
As in the fourth modification of the first embodiment, in the control device 2 of the second embodiment, the operation information transmission unit 200 may further include a control availability determination unit (not shown) that determines whether control of the control candidate device is possible. Good. In this case, when the control information determining unit 2001 determines that the control is impossible, the operation information generating unit 2001 generates operation information indicating that the current operation state of the control candidate device and the control are not possible, and the terminal Transmit to device 3A.

  The permission determination unit 2010 determines whether control is permitted or not even when the image detection information corresponding to the control candidate device determined to be uncontrollable by the control permission determination unit is received from the terminal device 3A. The image detection information is discarded without performing the processing relating to the above.

  The present invention is not limited to the above embodiments, and various modifications can be made without departing from the scope of the present invention.

  For example, in each of the above embodiments, the terminal devices 3 and 3A are eyeglass-type head-mounted displays (also referred to as smart glasses), but as the terminal device according to the present invention, a non-transmissive head-mounted display or It is also possible to employ a wearable computer other than the head mounted display. In such a terminal device, instead of the video projection unit 33 in the terminal devices 3 and 3A, a display device such as a liquid crystal display that displays an image showing the state of the outside world (outside world image) captured by the image sensor 32 is provided. In this case, the operation information video is displayed on the display device so as to be superimposed on the external image.

  Further, in order to notify the user that the content of the operation information is notified, the terminal devices 3 and 3A may emit sound or light, or may vibrate.

  Further, the terminal devices 3 and 3A may notify the user that the control device 2 has already received a control permission or non-permission instruction. In this case, the control device 2 notifies the terminal devices 3 and 3A that an instruction to allow or disallow control by the user has been received. Receiving such notification, the terminal devices 3 and 3A change the projection mode (or display mode) of the corresponding operation information video according to permission or non-permission. An example in which such notification is performed by the terminal device 3 in the third modification of the first embodiment is shown in FIG. In the example of FIG. 20, it is shown that the control for the living room illuminator is permitted, the control for the living room air conditioner is not permitted, and the control for the kitchen illuminator is waiting for a permission or non-permission instruction. Has been.

  Moreover, the notification mode of the content of the operation information is not limited to the video only, and may be notified by a voice message, for example.

  Further, in each of the embodiments described above, the control device 2 determines whether or not the control of the control candidate device is permitted by the user. However, the determination may be performed by the terminal devices 3 and 3A. Specifically, the terminal devices 3 and 3A include a permission determination unit (not shown) that performs the same function as the permission determination unit 2010 of the control device 2, and the permission determination unit includes information ( Permission determination information) is transmitted to the control device 2. And the control apparatus 2 discriminate | determines whether the user permitted control with respect to a control candidate apparatus based on this permission discrimination | determination information.

  Moreover, you may provide separately the push button, switch, etc. for accepting the permission or non-permission of the control with respect to a control candidate apparatus from the user in the terminal devices 3 and 3A. Alternatively, a microphone may be provided in the terminal devices 3 and 3A so that the user can instruct permission or non-permission of control of the control candidate device by voice, or an acceleration sensor is provided in the terminal devices 3 and 3A. The user may be able to instruct permission or non-permission of control of the control candidate device by tilting or shaking the terminal devices 3 and 3A.

  In the above embodiment, each function unit (see FIG. 6) of the control device 2 is realized by executing the remote operation program 240 stored in the secondary storage device 24 by the CPU 20. Moreover, each function part (refer FIG. 5) of the terminal devices 3 and 3A was implement | achieved by running the remote operation program memorize | stored in the secondary storage device 36 by CPU30. However, all or part of the functional units of the control device 2 and the terminal devices 3 and 3A may be realized by dedicated hardware. The dedicated hardware is, for example, a single circuit, a composite circuit, a programmed processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination thereof.

  In the above embodiment, the program executed by the control device 2 includes a CD-ROM (Compact Disc Read Only Memory), a DVD (digital versatile disc), a magneto-optical disc (Magneto-Optical Disc), and a USB (Universal Serial). Bus) It is also possible to store and distribute in a computer-readable recording medium such as a memory, a memory card, or an HDD. Then, by installing such a program on a specific or general-purpose computer, it is possible to cause the computer to function as the control device 2 in each of the above embodiments.

  Further, the program may be stored in a disk device or the like of a server on a network such as the Internet, and the program may be downloaded from the server to a computer.

  Various embodiments and modifications can be made to the present invention without departing from the broad spirit and scope. Further, the above-described embodiment is for explaining the present invention, and does not limit the scope of the present invention. That is, the scope of the present invention is shown not by the embodiments but by the claims. Various modifications within the scope of the claims and within the scope of the equivalent invention are considered to be within the scope of the present invention.

  The present invention can be suitably employed in a system or the like that manages the operation of an electrical device installed in a home.

  1 control system, 2 control device, 3 terminal device, 4-1, 4-2 equipment, 20, 30 CPU, 21, 31 communication interface, 22, 34 ROM, 23, 35 RAM, 24, 36 secondary storage device, 25, 37 bus, 32 image sensor, 33 video projection unit, 38 eye gaze detection unit, 200 operation information transmission unit, 201 device control unit, 240 remote operation program, 241 device information, 300 specific image detection unit, 301 detection information transmission unit 302 notification unit 2000 candidate determination unit 2001 operation information generation unit 2010 permission determination unit 2011 control command transmission unit 2410 connection information table 2411 location information table 2412 operation information table 2413 operation code table

In order to achieve the above object, a control system according to the present invention provides:
A control system that controls a plurality of devices, and a control system including a terminal device,
The terminal device
An image sensor;
Specific image detection means for detecting a specific image that matches a predetermined condition from an image captured by the image sensor;
Detection information transmitting means for transmitting image detection information based on the specific image to the control device;
When receiving the operation information transmitted from the control device, a notification means for notifying the user of the content of the operation information,
The control device includes:
When the image detection information transmitted from the terminal device is received, based on the image detection information, a control candidate device is selected from the plurality of devices, and operation information related to the control candidate device is generated, and the terminal device Operation information transmitting means for transmitting;
It is determined whether or not control over the control candidate device is permitted by the user, and if permitted, device control means for controlling the control candidate device based on the operation information, and
The notification means notifies the user that a control permission or non-permission instruction has been given to the control candidate device .

Claims (8)

A control system that controls a plurality of devices, and a control system including a terminal device,
The terminal device
An image sensor;
Specific image detection means for detecting a specific image that matches a predetermined condition from an image captured by the image sensor;
Detection information transmitting means for transmitting image detection information based on the specific image to the control device;
When receiving the operation information transmitted from the control device, a notification means for notifying the user of the content of the operation information,
The control device includes:
When the image detection information transmitted from the terminal device is received, based on the image detection information, a control candidate device is selected from the plurality of devices, and operation information related to the control candidate device is generated, and the terminal device Operation information transmitting means for transmitting;
A control system comprising: device control means for determining whether or not control of the control candidate device is permitted by a user and, if permitted, controlling the control candidate device based on the operation information.   The device control unit continuously receives the image detection information corresponding to the control candidate device for a predetermined period from the terminal device after the operation information is transmitted by the operation information transmission unit. In this case, the control system according to claim 1, wherein it is determined that control of the control candidate device is permitted by a user. After the content of the operation information is notified by the notification means, the detection information transmission means stores position information regarding the position of the specific image in the image detection information,
The control system according to claim 1, wherein the device control unit determines whether or not control of the control candidate device is permitted by a user based on the position information. The terminal device further includes line-of-sight detection means for detecting the line of sight of the user,
After the content of the operation information is notified by the notification means, the detection information transmission means stores line-of-sight information indicating a result detected by the line-of-sight detection means in the image detection information,
2. The control system according to claim 1, wherein the device control unit determines whether or not control of the control candidate device is permitted by a user based on the line-of-sight information.   The operation information transmitting means determines whether or not the control candidate device is controllable, and if it is determined that control is impossible, the operation information transmitted to the terminal device cannot control the control candidate device. The control system of any one of Claim 1 to 4 which stores the information which shows.   The control system according to any one of claims 1 to 5, wherein the notification unit notifies a user that a control permission or non-permission has been instructed for the control candidate device. After the content of the operation information is notified by the notification means, the terminal device determines whether or not control over the control candidate device is permitted by a user, and permits determination information indicating the determined result is the control device Further comprising permission determination means for transmitting to
The control system according to claim 1, wherein the device control unit determines whether or not control of the control candidate device is permitted by a user based on the permission determination information. A control device that controls a plurality of devices, and a method that is executed by a control system including a terminal device,
The terminal device detects a specific image that matches a predetermined condition from the image captured by the image sensor, transmits image detection information based on the specific image to the control device,
When the control device receives the image detection information transmitted from the terminal device, the control device selects a control candidate device from the plurality of devices based on the image detection information and generates operation information related to the control candidate device, Transmitted to the terminal device,
When the terminal device receives the operation information transmitted from the control device, it informs the user of the content of the operation information,
A device control method, wherein the control device determines whether or not control of the control candidate device is permitted by a user, and controls the control candidate device based on the operation information when permitted.

Images