JP2019062271A - Device control apparatus, device control method, and device control system - Google Patents

Abstract

To collectively control a plurality of devices based on the user's operation.SOLUTION: A control server 1 according to an embodiment of the present invention includes an operation detection unit 111 that detects operation of a user from an image obtained by imaging the user, a device selection unit 113 that selects a plurality of devices 3 on the basis of the operation detected by the operation detection unit 111, and a device control unit 114 that controls the plurality of devices 3 selected by the device selection unit 113. The device 3 performs operations such as turning-on, turning-off, and adjustment by receiving a control signal from the control server 1.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an apparatus control apparatus, an apparatus control method, and an apparatus control system that control an apparatus based on a user's operation.

  There have been developed device control apparatuses that issue instructions to various devices installed in a house by detecting user's actions (gesture). Here, the operation of the user does not directly operate the operation unit of the device but refers to various operations performed by the user in the same space as or apart from the operation unit of the device. When the device control device detects that the user has performed a predetermined operation, it can turn on or off or adjust devices such as lighting, air conditioners, AV (Audio Visual) devices based on the operation.

  Patent Document 1 describes a technique for detecting an operation performed by a user in a user terminal and starting communication with a partner associated with the operation.

Patent No. 5820058

  In the house, a plurality of devices that can be controlled by the action of the user may be installed. However, in the technology described in Patent Document 1, since it is premised that the user and the user terminal are in a one-to-one relationship, it is not possible to control a plurality of devices collectively. It is troublesome for the user to perform an operation of instructing each of the plurality of devices to be on or off.

  The present invention has been made in view of the above-described points, and an object thereof is to provide a device control apparatus, a device control method, and a device control system capable of collectively controlling a plurality of devices based on a user's operation.

  A device control apparatus according to a first aspect of the present invention is a device control apparatus for detecting a user's operation from an image obtained by imaging a user, and a plurality of devices based on the operation detected by the detection unit. And a device control unit that controls the plurality of devices selected by the selection unit.

  The selection unit may select the plurality of devices corresponding to the shape indicated by the trajectory of the operation.

  The selection unit may select the plurality of devices located in a range in which the shape is extended starting from the position of the user.

  The selection unit may select the plurality of devices based on a position at which the operation has been performed.

  The selection unit may select the plurality of devices based on the operation and environment information indicating an environment in which the operation is performed.

  The selection unit is based on the operation, the time when the operation was performed, the temperature of the position where the operation is performed, and the environment information indicating at least one of brightness of the position where the operation is performed. The plurality of devices may be selected.

  The selection unit may select the plurality of devices based on the operation and user information indicating a user who performed the operation.

  The information processing apparatus further includes an acquisition unit that acquires setting information indicating setting of whether or not each of the plurality of devices can be selected, and the selection unit indicates that the setting information is selectable based on the operation. Multiple devices may be selected.

  After the device control unit performs the control of the plurality of devices, when the operation reverse to the control is performed on the plurality of devices within a predetermined time, the device control unit is configured to It may be set in the setting information that selection is not possible.

  The information processing apparatus further includes an acquisition unit that acquires setting information indicating setting of whether or not selection is possible for each of the plurality of types of devices, and the selection unit can select the setting information based on the operation. The plurality of devices may be selected.

  The apparatus control unit further includes an acquisition unit configured to acquire setting information indicating a group of the plurality of devices, and the device control unit is configured such that the plurality of devices selected by the selection unit are some devices of the group indicated by the setting information. When not included, the control may be performed on the part of the devices in addition to the plurality of devices.

  The device control unit may intermittently operate the some devices before performing the control of the some devices.

  The detection unit detects a first operation of the user and a second operation of the user performed after the first operation as the operation, and the selection unit is based on the first operation. The plurality of devices may be selected, and the device control unit may perform the control corresponding to the second operation on the plurality of devices selected by the selection unit.

  In the device control method according to the second aspect of the present invention, a plurality of operations are detected based on the steps of detecting an operation of the user from an image obtained by imaging the user and the operation detected in the detecting step. It has a step of selecting a device, and a device control unit that controls the plurality of devices selected in the selecting step.

  A device control system according to a third aspect of the present invention includes a plurality of devices and a device control device for controlling the plurality of devices, wherein the device control device is configured to use the image obtained by imaging a user. A detection unit that detects a user's operation, a selection unit that selects the plurality of devices based on the operation detected by the detection unit, and a device control unit that controls the plurality of devices selected by the selection unit And the plurality of devices operate according to the control of the device control unit.

  According to the present invention, it is possible to control a plurality of devices collectively based on the user's operation.

It is a schematic diagram of the apparatus control system which concerns on 1st Embodiment. It is a schematic diagram which shows the operation | movement of the user which the control server which concerns on 1st Embodiment detects. It is a schematic diagram which shows the range which specific operation designates in 1st Embodiment. It is a schematic diagram which shows the control database used by the control server which concerns on 1st Embodiment. FIG. 2 is a block diagram of a device control system according to the first embodiment. It is a figure which shows the flowchart of the apparatus control method which concerns on 1st Embodiment. It is a schematic diagram of the apparatus control system which concerns on 2nd Embodiment. It is a schematic diagram which shows the range which specific operation specifies in 3rd Embodiment. It is a schematic diagram which shows the operation | movement of the user which the control server which concerns on 3rd Embodiment detects. It is a figure which shows the flowchart of the apparatus control method which concerns on 4th Embodiment. It is a schematic diagram which shows the control database used with the control server which concerns on 5th Embodiment. It is a schematic diagram which shows the group database used by the control server which concerns on 6th Embodiment.

First Embodiment
FIG. 1 is a schematic view of a device control system S according to the present embodiment. The device control system S communicates with the control server 1 as a device control device, one or more sensor devices 2 for acquiring various information, and one or more devices 3 controlled by the control server 1. And an intermediary gateway 4. The device control system S may include devices such as other servers and terminals.

  The device 3 is a controlled device that is controlled to be turned on, off, adjusted, etc. by receiving a control signal from the control server 1. As the device 3, any device such as lighting, air conditioner, AV device, etc. can be used. The device 3 may operate not only by the control server 1 but also by an operation by the user. The device 3 has a communication module, and communicates with the control server 1 via the gateway 4 by wired communication or wireless communication.

  The sensor device 2 measures and acquires information such as an image, a sound, and a temperature. The sensor device 2 has various sensors necessary for acquiring information. The sensor device 2 may be incorporated in the device 3 or may be installed independently. The sensor device 2 has a communication module, and communicates with the control server 1 via the gateway 4 by wired communication or wireless communication.

  The gateway 4 is a relay device that performs processing such as protocol conversion on signals transmitted and received between the sensor device 2, the device 3, and the control server 1 to enable communication. The gateway 4 is connected to each of the sensor device 2 and the device 3 by wired connection or wireless connection, and is connected to the control server 1 via an arbitrary network N such as a local area network or the Internet.

  The control server 1 is connected to the network N by wire communication or wireless communication outside the building where the sensor device 2, the device 3 and the gateway 4 are installed. The control server 1 is configured by a single computer or a cloud that is a collection of computer resources. The control server 1 receives a signal indicating the information measured by the sensor device 2 via the gateway 4. Next, the control server 1 determines the device 3 to be controlled and the control content based on the information from the sensor device 2. Then, the control server 1 transmits a signal indicating the control content to the device 3 via the gateway 4.

[Description of user operation]
FIG. 2 is a schematic view showing the operation of the user detected by the control server 1 according to the present embodiment. The actions of the user include specific actions and adjustment actions. The specific operation (first operation) is an operation for specifying the device 3 to be controlled, and the adjustment operation (second operation) is an operation for specifying control content.

  The specific operation is an action of moving a finger in the air so as to draw a predetermined figure F specifying a range including one or more devices 3 to be controlled. That is, the figure F is a shape indicated by the trajectory of the user's action. The figure F is not limited to a circular shape as in the example of FIG. 2, and is a predetermined shape preset in the control server 1 such as an ellipse, a rectangle, a triangle, or another polygon.

  The adjustment operation is an action of changing the shape of the hand or moving the hand after the specific operation. The adjustment operation is, for example, an action of holding a hand after a specific operation (i.e., forming a hand). Thereby, the control server 1 selects one or more devices 3 located in a range within the extension of the figure F drawn by the specific operation as a control target, and controls the content corresponding to the action of grasping the hand (for example, Decide to turn off the device 3).

  The operation of the user illustrated in FIG. 2 is an example, and the control server 1 is configured to detect a predetermined operation that can be determined from an image obtained by imaging the user. The operation of the user is not limited to the two operations of the specific operation and the adjustment operation, but may be used with one operation of simultaneously specifying one or more control targets and control contents, or by combining three or more operations. You may use.

  FIG. 3 is a schematic view showing a range R designated by the specific operation in the present embodiment. FIG. 3 shows the interior of the building as viewed from above. In FIG. 3, the range R designated by the user's specific operation is represented by a broken line. When the control server 1 detects the figure F drawn by the user's specific operation, the control server 1 specifies a range R in which the figure F is three-dimensionally extended starting from the position of the user (for example, the user's finger) in the building. Then, the control server 1 selects one or more devices 3 located in the range R as a control target. The position of each device 3 is stored in advance in the control server 1. Although FIG. 3 is represented in two dimensions for visibility, the range R is actually three-dimensional, and the control server 1 selects the device 3 included in the range R in the three-dimensional space.

  When the user located at a predetermined place (for example, the entrance) in the building performs a specific operation, the control server 1 executes the selection of all the devices 3. The total selection of the device 3 is to select all the devices 3 registered in the control server 1 regardless of the range R and the position of the device 3 designated by the specific operation. A place for performing the full selection (for example, a predetermined coordinate range based on a predetermined point of a building) is stored in advance in the control server 1. For example, the control server 1 performs all selection when the user performs a specific operation at the entrance, thereby turning off all the devices 3 in the building when the user goes out, or in the building when the user returns home All devices 3 can be turned on. As described later with reference to FIG. 4, the control server 1 can be set to exclude the devices 3 that the user does not want to control by the full selection from the targets of the full selection.

[Description of control database D]
FIG. 4 is a schematic view showing a control database D used in the control server 1 according to the present embodiment. The control database D stores setting information for determining the device 3 to be controlled and the control content from the operation of the user. Specifically, the control database D associates and stores the device number D1, the position information D2, the adjusting operation D3 and the all selection D4.

  The device number D1 is a number for identifying the device 3. The device number D1 may be, for example, a serial number of the registered device 3, or may be a MAC (Media Access Control) address previously assigned to the device 3.

  The position information D2 is information indicating the position of the device 3, and is represented by, for example, three-dimensional coordinates X, Y, Z based on a predetermined point of a building. The position of the device 3 is used to specify the device 3 located within the range R when the range R is specified by the user's specific operation. For example, the control server 1 estimates the position of the device 3 from an image obtained by imaging the device 3 with the sensor device 2. Alternatively, the user may set the position of each device 3 in the control server 1 using an application of a portable terminal or the like. In this case, for example, the user inputs coordinate information on the application or marks the map in the building displayed on the application to set the position of the device 3 in the control server 1. Although three-dimensional coordinates are used in this embodiment, two-dimensional coordinates may be used except for the coordinate in the height direction. In this case, the device 3 is identified by the horizontal position without distinguishing the height.

  The adjustment operation D3 is information indicating an operation of specifying the control content. As the adjusting operation D3, for example, in addition to holding the hand shown in FIG. 2, opening the hand, moving the hand in a specific shape and moving in a specific direction, and the like can be used. As control contents designated by the adjustment operation D3, any operation that can be executed by the device 3, such as on, off, adjustment, can be used. A plurality of adjustment operations D3 may be set for one device 3 (device number D1).

  The full selection D4 sets whether or not the device 3 is the target of the full selection to be performed when a specific operation is performed at a predetermined place in a building. Specifically, when executing the all selection, the control server 1 selects the device 3 for which the all selection D4 is set to “enabled” regardless of the position information D2, and the all selection D4 is “not available”. Do not select the device 3 set to. Thereby, the user can exclude the apparatus 3 (for example, heating apparatuses, such as an electric stove and an electric stove) which he / she does not want to control by all selection from the targets of all selection.

  The full selection D4 may be set for each operation (on, off, etc.) of the device 3. In this case, in the all selection D4 of a certain device 3, "on" can be set to "disable" for the on operation, and "off" for the off operation.

  The adjustment operation D3 and the all selection D4 for each device 3 may be set in the control server 1 in advance. Alternatively, the user may set the adjustment operation D3 and the all selection D4 for each device 3 in the control server 1 using an application or the like of the portable terminal. In this case, the control server 1 determines the device 3 related to safety, such as a heating device, based on the appearance of the device 3 acquired from the image, and “not” the initial value of all selections D4 of the device 3 in advance. It is desirable to set to. For example, a known pattern matching technique can be used to determine the device 3 related to safety.

  The control database D shown in FIG. 4 is an example, and the contents and format of the control database D are arbitrarily set as long as they can be used in the device control method according to the present embodiment. In FIG. 4, the control database D is represented by a table of character strings for visibility, but may be represented by any data format (file format) interpretable by the control server 1, for example, binary data or It may be text data. The control database D may be stored inside the control server 1 or may be stored in a storage device outside the control server 1.

[Configuration of Device Control System S]
FIG. 5 is a block diagram of the device control system S according to the present embodiment. In FIG. 5, arrows indicate the flow of main data, and there may be flows of data other than those shown in FIG. In FIG. 5, each block is not a hardware (apparatus) unit configuration but a function unit configuration. As such, the blocks shown in FIG. 5 may be implemented in a single device or may be implemented separately in multiple devices. Transfer of data between the blocks may be performed via any means such as a data bus, a network, a portable storage medium, and the like.

  The sensor device 2 has an image sensor 21. The image sensor 21 is a camera having an imaging element that outputs information indicating a captured image. The image captured by the image sensor 21 is used by the control server 1 to detect the above-described specific operation and adjustment operation. The control server 1 also estimates the position of the user from the images captured from the multiple directions by the multiple image sensors 21. Alternatively, in order to detect the position of the user, a distance sensor may be provided which measures the distance by irradiating the subject (user) with infrared light or ultrasonic waves.

  The control server 1 includes a control unit 11, a communication unit 12, and a storage unit 13. The control unit 11 includes an operation detection unit 111, a setting acquisition unit 112, a device selection unit 113, and a device control unit 114.

  The communication unit 12 is a communication interface for communicating with the sensor device 2 and the device 3 via the gateway 4. The communication unit 12 performs predetermined processing on communication signals received from the sensor device 2 and the device 3 to acquire data, and inputs the acquired data to the control unit 11. Further, the communication unit 12 performs predetermined processing on the data input from the control unit 11 to generate a communication signal, and transmits the generated communication signal to the sensor device 2 and the device 3.

  The storage unit 13 is a storage medium including a read only memory (ROM), a random access memory (RAM), a hard disk drive, and the like. The storage unit 13 stores in advance a program to be executed by the control unit 11.

  The control unit 11 is a processor such as a central processing unit (CPU), for example, and executes the program stored in the storage unit 13 to obtain an operation detection unit 111, a setting acquisition unit 112, a device selection unit 113, and a device control unit. It functions as 114. At least a part of the functions of the control unit 11 may be performed by an electric circuit. In addition, at least a part of the functions of the control unit 11 may be executed by a program executed via a network.

  The device control system S according to the present embodiment is not limited to the specific configuration shown in FIG. For example, the control server 1 is not limited to one device, and may be configured by connecting two or more physically separated devices in a wired or wireless manner.

  The motion detection unit 111 detects the motion of the user indicated by the image captured by the sensor device 2. Specifically, first, the operation detection unit 111 receives information from the sensor device 2 via the communication unit 12. The motion detection unit 111 acquires an image captured by the image sensor 21 of the sensor device 2 from the received information. The motion detection unit 111 applies an image analysis technique of three-dimensional recognition to the acquired image to detect a motion and a position of the user. The motion detection unit 111 repeats the detection of the motion until the specific motion for drawing the graphic F is detected. Known techniques can be used as image analysis techniques for three-dimensional recognition.

  When the specific operation for drawing the graphic F is detected, the setting acquisition unit 112 acquires setting information of each device 3 from the control database D. The device selection unit 113 determines whether the position of the user detected by the operation detection unit 111 is within a predetermined coordinate range for performing full selection. When the position of the user is not within the predetermined coordinate range, the device selection unit 113 selects the device 3 corresponding to the graphic F of the specific operation as a control target. Specifically, from the position information D2 of each device 3 in the setting information of the control database D acquired by the setting acquisition unit 112, the device selection unit 113 is a figure starting from the position of the user detected by the operation detection unit 111 The device 3 (i.e., the device number D1) located in a three-dimensionally extended range R of F is selected as a control target.

  When the position of the user is within the predetermined coordinate range, the device selection unit 113 selects all devices for which all selection D4 is set to “OK” from the setting of the control database D acquired by the setting acquisition unit 112. 3 (ie, device number D1) is selected as a control target.

  After the control target is determined, the motion detection unit 111 acquires an image of the next frame (or after a predetermined time), applies the image analysis technology of three-dimensional recognition in the same manner, and detects the motion of the user. The operation detection unit 111 repeats the detection of the operation until an operation that matches one of the adjustment operations (that is, the adjustment operation D3) included in the setting information of the control database D acquired by the setting acquisition unit 112 is detected.

  After the specific operation is detected, if the adjustment operation is not detected even after a predetermined time has elapsed, the operation detection unit 111 returns to the detection of the specific operation. When the adjustment operation is detected, the operation detection unit 111 determines the content (on, off, adjustment, etc.) designated by the adjustment operation in the setting information of the control database D acquired by the setting acquisition unit 112 as the control content. Do.

  In order to suppress erroneous detection of an operation, the operation detection unit 111 may detect a specific operation or an adjustment operation that matches the operation when the same operation is continued for a predetermined time.

  When the operation detection unit 111, the setting acquisition unit 112, and the device selection unit 113 determine the control target and control content associated with the user's operation, the device control unit 114 performs control according to the determined control content. A control signal is generated and transmitted to the determined device 3 to be controlled via the communication unit 12.

  The device 3 performs predetermined operations such as on, off, and adjustment in accordance with the control signal received from the control server 1 via the gateway 4.

  As described above, the control server 1 according to the present embodiment puts together all the devices 3 within the extended range of the figure F drawn by the user's operation or all the devices 3 in the building where the user's operation has been performed. An instruction can be given to one or more selected and selected devices 3 simultaneously. Therefore, it is not necessary for the user to issue an operation instruction to each device 3.

  If an image is constantly transmitted from the sensor device 2 to the control server 1, the communication load is large. Therefore, the sensor device 2 only detects an operation close to a predetermined operation (specific operation and adjustment operation) from the image. The transmission of the image of may be started. In this case, the sensor device 2 is configured as a computer that executes the operation detection software, and registers in advance an operation close to the specific operation and the adjustment operation. The sensor device 2 starts transmission of an image to the control server 1 when detecting a previously registered operation from an image captured by the image sensor 21 using the operation detection software. Thereby, the amount of communication of data transmitted from the sensor device 2 to the control server 1 can be reduced.

[Flowchart of Device Control Method]
FIG. 6 is a diagram showing a flowchart of the device control method according to the present embodiment. The flowchart of FIG. 6 is started, for example, when the user performs a predetermined start operation on the control server 1.

  First, the motion detection unit 111 detects the motion of drawing the figure F by the user (that is, the above-described specific motion) and the position of the user from the image captured by the sensor device 2 (S11).

  When the operation detection unit 111 does not detect an operation in step S11 (NO in S12), the control server 1 returns to step S11 and repeats the process. When the operation detection unit 111 detects an operation in step S11 (YES in S12), the setting acquisition unit 112 acquires setting information of each device 3 from the control database D (S13).

  The device selection unit 113 determines whether the position of the user detected by the operation detection unit 111 is within a predetermined coordinate range for performing full selection. When the position of the user is within the predetermined coordinate range (YES in S14), the device selection unit 113 determines that all selections D4 are "OK" from the setting of the control database D acquired by the setting acquisition unit 112 in step S13. All the set devices 3 are selected as control targets (S15).

  When the position of the user is not within the predetermined coordinate range (NO in S14), the device selection unit 113 starts the operation detection unit 111 in step S11 based on the setting of the control database D acquired by the setting acquisition unit 112 in step S13. The apparatus 3 located in the range R which extended the figure F three-dimensionally from the detected user's position as a starting point is selected as a control object (S16).

  After the control target is determined, the operation detection unit 111 detects the user's operation (that is, the above-described adjustment operation) from the image of the next frame (or after a predetermined time) captured by the sensor device 2. The operation detection unit 111 may repeat the detection of the operation until an operation that matches any of the adjustment operations is detected, and if the adjustment operation can not be detected even after a predetermined time has elapsed, the control server 1 returns to step S11. Processing may be repeated. When the adjustment operation is detected, the operation detection unit 111 determines the content designated by the adjustment operation in the setting information of the control database D acquired by the setting acquisition unit 112 in step S13 as the control content (S17).

  The device control unit 114 generates a control signal for performing control in accordance with the control content determined by the operation detection unit 111 in step S17, and one or more control targets determined by the device selection unit 113 in step S15 or S16. It transmits to the apparatus 3 via the communication part 12 (S18).

  If a predetermined termination condition (for example, the user performs a predetermined termination operation on the control server 1) is not satisfied (NO in S19), the control server 1 returns to step S11 and repeats the processing. If the predetermined termination condition is satisfied (YES in S19), the control server 1 terminates the process.

[Effect of First Embodiment]
The control server 1 which concerns on this embodiment selects and controls the apparatus 3 in the range which extended the figure F drawn in the air by the user's operation | movement three-dimensionally. With such a configuration, the control server 1 can collectively control a plurality of devices based on the operation of the user, so that the convenience of the user can be improved. Furthermore, the control server 1 selects all selectable devices 3 in the building when the user operates at a predetermined place. Thereby, for example, when the user goes out at the entrance or returns home, it is possible to easily control the plurality of devices 3 such as turning on or off all the devices 3 in the building.

[Modification of First Embodiment]
In the above-described device control method, the control server 1 constantly detects the operation, but may start the detection of the operation triggered by the detection of the specific start voice. In this case, the sensor device 2 includes an audio sensor such as a microphone. First, the control server 1 does not detect an operation before the start voice is detected, and starts detection of the start voice. The control server 1 detects a predetermined start sound from the sound acquired from the sound sensor of the sensor device 2. The start voice is, for example, voice stored in the control server 1 such as “OK” or “Hi”. Also, the start voice may be voice pre-recorded on the control server 1 by the user.

  For example, the control server 1 applies speech recognition technology to the speech acquired from the speech sensor of the sensor device 2 and detects the speech even if the speech corresponding to the speech is included in the speech. Good. Known techniques can be used as the speech recognition technique. In addition, when the frequency spectrum of the voice acquired from the voice sensor of the sensor device 2 includes the predetermined ratio or more of the frequency spectrum of the start voice recorded in advance, the control server 1 detects the start voice. It is also good. The predetermined ratio is a ratio considered to be coincident with the start speech, and is, for example, 90%. In addition, the control server 1 starts the start voice when the degree of correlation between the pattern of the frequency spectrum of the voice acquired from the voice sensor of the sensor device 2 and the pattern of the frequency spectrum of the start voice recorded in advance is equal to or more than a predetermined value. May be detected.

  Then, when detecting the start voice, the control server 1 starts the detection of the operation and performs the above-described device control method. The control server 1 stops the detection of the operation when a predetermined time has elapsed from the time when the start voice is detected, and detects the start voice again.

  With such a configuration, the control server 1 detects an operation from an image only during a predetermined period after detecting the start voice. Therefore, the processing load and communication load of the control server 1 can be reduced.

Second Embodiment
In the first embodiment, the control server 1 is disposed outside the building and communicates with the sensor device 2 and the device 3 via the network N. Therefore, depending on the communication state of the network N, it may take time to execute the operation detection and the control of the device. Further, when an image is constantly transmitted from the sensor device 2 to the control server 1, the communication load is large. On the other hand, in the present embodiment, in addition to the control server 1-1 disposed outside the building, the control server 1-2 is provided in the building.

  FIG. 7 is a schematic view of the device control system S according to the present embodiment. The device control system S includes, in addition to the sensor device 2, the device 3 and the gateway 4 as in the first embodiment, the control server 1-1 connected to the network N outside the building, and the gateway 4 in the building It includes a control server 1-2 connected to the network N. The control server 1-1 and the control server 1-2 each have a configuration similar to that of the control server 1 of FIG.

  In the present embodiment, the speed and accuracy of the motion detection are balanced by adjusting processing and data between the control server 1-2 in the building and the control server 1-1 outside the building. For example, the control server 1-2 in the building performs operation detection processing with a small amount of calculation, and the control server 1-1 outside the building performs operation detection processing with a large amount of calculation. Alternatively, for example, a small amount of control database D is stored in the storage unit 13 of the control server 1-2 in the building, and a large amount of control database D is stored in the storage unit 13 of the control server 1-1 outside the building.

  First, the control server 1-2 in the building detects an operation based on the information from the sensor device 2. And when control server 1-2 can not detect operation, control server 1-2 transfers information from sensor device 2 to control server 1-1 outside a building, and control server 1-1 is a sensor device. The operation is detected based on the information from 2. One of the control server 1-2 and the control server 1-1 controls the device 3 based on the detected operation.

[Effect of Second Embodiment]
Generally, the control server 1-2 in the building does not have a communication load because it does not go through the network N, but there is a restriction on the device configuration due to the installation space and the installation cost. On the other hand, the control server 1-1 outside the building can perform high-speed processing because it can have a high-level equipment configuration, but it imposes a communication load. Therefore, the device control system S according to the present embodiment performs simple operation detection in the control server 1-2 in the building, and performs complicated operation detection in the control server 1-1 outside the building. Such a configuration can improve the speed of motion detection while maintaining the accuracy of motion detection.

Third Embodiment
In the present embodiment, the devices 3 that can be selected as the control target are changed according to the place where the user is present and the shape of the figure F drawn by the user. Thereby, fine device control can be realized.

[User location setting]
FIG. 8 is a schematic view showing a range R designated by the specific operation in the present embodiment. FIG. 8 shows the interior of the building as viewed from above. In FIG. 8, the range R designated by the user's specific operation is represented by a broken line. In the present embodiment, a range of selectable devices 3 is set for each place where a specific operation is performed (for example, a predetermined coordinate range based on a predetermined point of a building), and stored in advance in the storage unit 13 of the control server 1 Be done. For example, at the entrance, all the devices 3 in the building can be selected, and in the kitchen, only the devices 3 in the same room can be selected.

  When the operation detection unit 111 detects the figure F drawn by the user's specific operation, the setting acquisition unit 112 acquires setting information from the storage unit 13 of the control server 1, and the device selection unit 113 detects the position of the user in the building. A range R in which the figure F is three-dimensionally extended is specified as a starting point. Further, the device selection unit 113 extracts, from the setting information acquired by the setting acquisition unit 112, the device 3 that can be selected at the location to which the position of the user who has performed the specific operation belongs.

  Then, among the devices 3 located within the range R, the device selection unit 113 selects one or more devices 3 that can be selected at the location to which the position of the user who performed the specific operation belongs as a control target. Although FIG. 8 is expressed in two dimensions for visibility, the range R is actually three-dimensional in practice, and the device selection unit 113 is included in the range R in the three-dimensional space, and is specified The selectable device 3 is selected at the place where the operation was performed.

[Setting by shape of figure F]
Fig.9 (a) and FIG.9 (b) are schematic diagrams which show the operation | movement of the user which the control server 1 which concerns on this embodiment detects. FIG. 9A and FIG. 9B show a specific operation of drawing a figure F for specifying the device 3 to be controlled. The figure F in FIG. 9A is circular, and the figure F in FIG. 9B is rectangular. In the present embodiment, the range of selectable devices 3 is set for each shape of the figure F, and is stored in advance in the storage unit 13 of the control server 1. For example, a circular figure F can be selected for all devices 3 in a building, and a rectangular figure F can be selected so that only the devices 3 in the same room can be selected.

  When the operation detection unit 111 detects the figure F drawn by the user's specific operation, the setting acquisition unit 112 acquires setting information from the storage unit 13 of the control server 1, and the device selection unit 113 detects the position of the user in the building. A range R in which the figure F is three-dimensionally extended is specified as a starting point. Further, the device selection unit 113 extracts the devices 3 that can be selected according to the shape of the figure F from the setting information acquired by the setting acquisition unit 112. Then, the device selection unit 113 selects one or more devices 3 that can be selected according to the shape of the graphic F among the devices 3 located within the range R as a control target.

  The operation of the user shown in FIGS. 9A and 9B is an example, and the control server 1 detects the graphic F having a predetermined shape that can be determined from the image obtained by imaging the user. Configured The figure F is not limited to a circle or a rectangle as in the example of FIGS. 9A and 9B, but is a predetermined shape preset in the control server 1 such as an ellipse, a triangle, and other polygons. .

[Effect of the third embodiment]
The control server 1 according to this embodiment produces the same effect as that of the first embodiment, and changes the device 3 to be selected according to the place where the user performed an operation or the shape of the drawn figure F. For example, controlling all unintended devices 3 can be suppressed by enabling all devices 3 in the building to be selectable at the entrance and selecting only the devices 3 in the same room in the kitchen. In addition, the user can change the selection of the control target finely according to the user's intention by properly using the shape of the figure F drawn in the specific operation.

Fourth Embodiment
In this embodiment, in addition to the operation of the user, the device 3 selectable as the control target is changed based on additional information including user information and environment information such as time, temperature, brightness and the like. Thereby, fine device control can be realized.

  In the present embodiment, selectable devices 3 are set for each piece of environment information indicating an environment in which an operation such as a time zone, temperature, brightness, etc. is performed, and stored in advance in the storage unit 13 of the control server 1. The sensor device 2 includes a watch that measures time, a temperature sensor that measures temperature, and an illuminance sensor that measures brightness.

  Further, in the present embodiment, the selectable devices 3 are set for each piece of user information indicating the user who performs the operation, and are stored in advance in the storage unit 13 of the control server 1. The user information may be information identifying the user (individual) or information indicating the user's attributes (ie, height of the user, palm size, voice, body temperature, or a device possessed by the user).

  When the operation detection unit 111 detects the figure F drawn by the user's specific operation, the setting acquisition unit 112 acquires setting information from the storage unit 13 of the control server 1. Further, the setting acquisition unit 112 acquires, from the sensor device 2, environment information indicating the time when the user's operation was performed, the temperature of the position where the user's operation was performed, and the brightness of the position where the user's operation was performed. Do. Further, the setting acquisition unit 112 acquires information identifying a user or user information indicating an attribute of the user from the image captured by the sensor device 2 using a person recognition technique. A well-known technique can be used as a person recognition technique.

  The device selection unit 113 can select all of the devices 3 located within a range R where the figure F is three-dimensionally extended starting from the position of the user in the building, or when the specific operation is performed at a predetermined location All devices 3 are selected as control targets.

  Furthermore, the device selecting unit 113 extracts the selectable device 3 from the setting information acquired by the setting acquiring unit 112 based on the environment information and the user information acquired by the setting acquiring unit 112. The device selection unit 113 selects one or more devices 3 that can be selected according to the environment information and the user information acquired by the setting acquisition unit 112 among the devices 3 located within the range R or all the selected devices 3 Narrow down as a control target.

[Flowchart of Device Control Method]
FIG. 10 is a diagram showing a flowchart of the device control method according to the present embodiment. In the flowchart of FIG. 10, step S20 is performed after steps S15 and S16 and before step S17 in the flowchart of FIG. The other steps are the same as those in the flowchart of FIG.

  After the control target is selected in step S15 or step S16, the setting acquisition unit 112 acquires environment information indicating time, temperature, and brightness from the sensor device 2. Further, the setting acquisition unit 112 acquires information identifying a user or user information indicating an attribute of the user from the image captured by the sensor device 2 using a person recognition technique. A well-known technique can be used as a person recognition technique. The device selection unit 113 selects one or more of the devices 3 located within the range R or all selected devices 3 that can be selected by additional information including the environment information and the user information acquired by the setting acquisition unit 112. The device 3 is narrowed down as a control target (S20).

[Effect of Fourth Embodiment]
The control server 1 according to the present embodiment achieves the same effects as the first embodiment, and changes the devices 3 selectable as the control target based on the environment information such as time, temperature, brightness, etc. and the user information. Do. For example, it is possible to perform control such as removing lighting from the control target during the daytime, and removing the air conditioner from the control target when the air temperature is high or low. Further, for example, when the user is a child, control such as limitation of a control target can be performed. Thus, the control server 1 according to the present embodiment can perform device control in more detail.

Fifth Embodiment
In the present embodiment, it is set for each device 3 whether or not it is possible to select collectively according to a specific operation. Thereby, fine device control can be realized.

[Description of control database D]
FIG. 11 is a schematic view showing a control database D used by the control server 1 according to the present embodiment. Control database D of FIG. 11 further includes multiple selection D5 in control database D of FIG. The other elements are the same as those of the control database D of FIG.

  The multiple selection D5 sets whether to set the device 3 as the target of multiple selection by the specific operation. Specifically, in the range R designated by the specific operation, the device selection unit 113 selects the device 3 for which the multiple selection D5 is set to “enabled”, and the multiple selection D5 is set to “disabled”. Do not select device 3 Thereby, the user can exclude the apparatus 3 (for example, heating apparatuses, such as an electric stove and an electric stove) which you do not want to control by multiple selection from the object of multiple selection. When there is one device 3 included in the range R designated by the specific operation, the device selection unit 113 may select the device 3 on condition that the multiple selection D5 is “OK”, or The device 3 may be selected regardless of the selection D5.

  The multiple selection D5 may be set for each operation (on, off, etc.) of the device 3. In this case, in the multiple selection D5 of a certain device 3, "on" can be set to "impossible" and "off" can be set to "possible".

  The multiple selection D5 for each device 3 may be set in the control server 1 in advance. Alternatively, the user may set the multiple selection D5 for each device 3 in the control server 1 using an application or the like of the portable terminal. In this case, the control server 1 determines the device 3 related to safety, such as a heating device, based on the appearance of the device 3 acquired from the image, and “not” the initial value of the multiple selection D5 of the device 3 in advance. It is desirable to set to. For example, a known pattern matching technique can be used to determine the device 3 related to safety.

  Alternatively, after control by the operation of the user is performed on the device 3, the device control unit 114 may automatically set the multiple selection D5 on the device 3 according to the operation of the user on the device 3. . Specifically, after controlling the device 3 by the operation of the user, the device control unit 114 monitors a manual operation of the device 3 by the user. When the user performs an operation reverse to the control by the control server 1 in the device 3 (that is, the user manually performs the operation of turning off after the operation of turning on or vice versa) within a predetermined time The device control unit 114 sets the multiple selection D5 to “impossible”. As a result, the user can easily designate the device 3 that does not want to perform control by multiple selection and can set it as a target of control.

  When the specific operation for drawing the graphic F is detected, the setting acquisition unit 112 acquires setting information of each device 3 from the control database D. The device selection unit 113 three-dimensionally displays the figure F starting from the position of the user detected by the operation detection unit 111 from the position information D2 of each device 3 in the setting information of the control database D acquired by the setting acquisition unit 112. The apparatus 3 located in the range R extended to is extracted. Furthermore, the device selection unit 113 selects, as the control target, the devices 3 for which the multiple selection D5 is set to “OK” among the extracted devices 3.

  Although multiple selection D5 can be set for each individual device 3 in FIG. 11, multiple selection D5 may be set for each type of device 3. The type of the device 3 is, for example, a lighting device, an air conditioner, a heating device, and the like. In particular, with regard to the device 3 of the type related to the safety of the user, such as a heating device, control by multiple selection is not possible for the on operation, and control in advance is possible so that control by multiple selection is possible for the off operation. It is desirable to set in the server 1. Although it may be dangerous to turn on the heating device remotely, the risk of turning it off is small, so it is possible to operate only with the switch off while securing the convenience of multi-selection control. Can also improve safety.

[Effect of Fifth Embodiment]
The control server 1 according to the present embodiment achieves the same effect as that of the first embodiment, and sets whether or not control is possible collectively by a specific operation for each device 3 or for each type of device 3. Thereby, device control can be performed more finely. Furthermore, when the user performs the reverse operation on the device 3 after the control by the control server 1, the device 3 that the user wants to exclude from the plurality of selections can be easily set by automatically setting the plurality of devices 3 to be unselectable. It can be set to Furthermore, by making it possible to set whether or not a plurality of items can be selected for each type of device 3, it is possible to set a plurality of devices 3 of a type related to safety like heating devices, for example. .

Sixth Embodiment
In this embodiment, a group of devices 3 to be controlled collectively is set, and when the device 3 designated by the operation is missing from the group, the missing device 3 is complemented and controlled. As a result, even when the device 3 can not be correctly designated by the operation, the devices 3 normally treated as a group can be controlled collectively.

[Description of group database DD]
FIG. 12 is a schematic view showing a group database DD used by the control server 1 according to the present embodiment. The group database DD stores group information for grouping the devices 3 to be controlled collectively by a plurality of selections. Specifically, the group database DD stores the group number DD1 and the device number DD2 in association with each other.

  The group number DD1 is a number for identifying a group of two or more devices 3. The group number DD1 is, for example, a serial number of a registered group. The device number DD2 is a number for identifying the device 3. The device number DD2 corresponds to the device number D1 of the control database D.

  The device control unit 114 registers the plurality of devices 3 (device number DD2) as one group (group number DD1) in the group database DD when controlling the plurality of devices 3 specified by the user's operation. . That is, the device control unit 114 learns a group of devices 3 to be controlled collectively. The registration of the group in the group database DD may be performed when the plurality of devices 3 are controlled for the first time, or may be performed when the plurality of devices 3 are controlled a predetermined number of times or more.

  The device control unit 114 may register, as a group, a plurality of devices 3 controlled within a predetermined time, not simultaneously. Further, the device control unit 114 may detect an operation manually performed on the device 3 by the user, and may register the plurality of devices 3 operated within a predetermined time as a group.

  The group database DD may be set in advance in the control server 1. Alternatively, the user may set the group database DD in the control server 1 using an application of a mobile terminal or the like.

  The group database DD shown in FIG. 12 is an example, and the contents and format of the group database DD are arbitrarily set as long as they can be used in the device control method according to the present embodiment. In FIG. 12, the group database DD is represented by a table of character strings for visibility, but may be represented by any data format (file format) interpretable by the control server 1, for example, binary data or It may be text data. The group database DD may be stored inside the control server 1 or may be stored in a storage device outside the control server 1.

  When the specific operation for drawing the graphic F is detected, the setting acquisition unit 112 acquires setting information of each device 3 from the control database D, and acquires group information from the group database DD. From the position information D2 of each device 3 in the setting information of the control database D, the device selection unit 113 is positioned within a range R in which the figure F is three-dimensionally extended starting from the position of the user detected by the operation detection unit 111. Device 3 to be extracted. Furthermore, the device selection unit 113 determines whether the extracted device 3 belongs to any group registered in the group database DD.

  Specifically, the device selecting unit 113, for example, extracts a predetermined ratio or more (for example, 50% or more) of the plurality of devices 3 (device number DD2) associated with a certain group (group number DD1). If it matches, it is determined that the extracted device 3 belongs to the group. For example, when using the group database DD of FIG. 12, when the device numbers of the extracted device 3 are “1” and “2”, the device numbers “1”, “2” and the like associated with the group number “1” Since it corresponds to about 67% of “3”, the extracted device 3 is determined to belong to the group of the group number “1”. When it is determined that the extracted device 3 belongs to a plurality of groups, the group with the largest matching ratio of the device numbers is selected.

  In the extracted device 3, when part of the device number associated with the group to which the extracted device 3 belongs is missing (that is, the matching ratio of the device numbers is less than 100%), the device selection unit 113 In addition to the extracted device 3, the device 3 with the missing device number in the group to which the extracted device 3 belongs is determined as the control target. In other words, when a part of the devices 3 belonging to the group is selected by the specific operation, the device selection unit 113 selects all the devices 3 belonging to the group as control targets. With such a configuration, even if the operation detection unit 111 can not correctly detect the specific operation, the device 3 to be controlled can be complemented based on the group information.

  The device control unit 114 generates a control signal for performing control according to the control content determined by the operation detection unit 111, and transmits the control signal to the device 3 to be controlled determined by the device selection unit 113 via the communication unit 12.

  The device control unit 114 performs control according to the control content after causing the device 3 of the missing device number in the group to which the extracted device 3 belongs to perform preliminary operation such as blinking of a lamp for a predetermined time (that is, intermittent operation). Control signals may be transmitted. When the user performs an operation on the device 3 during the preliminary operation, the device control unit 114 stops the transmission of the control signal to the device 3. Furthermore, the device control unit 114 may set the device 3 whose transmission of the control signal has been canceled to be excluded from the group to which the extracted device 3 belongs. Such a configuration makes it possible to notify the user of the device 3 complemented by the group information and allows the user to reject the complement if the complement is not intended by the user.

[Effect of Sixth Embodiment]
The control server 1 according to the present embodiment achieves the same effects as those of the first embodiment, and all of the devices belonging to the group, even if some of the devices 3 belonging to the group are selected by the specific operation. The device 3 is selected as a control target. Thus, for example, even when the recognition of the operation from the image is not normally performed or the operation is shifted from the device 3 to be controlled, the group of the devices 3 to be controlled can be complemented and controlled. .

  As mentioned above, although the present invention was explained using an embodiment, the technical scope of the present invention is not limited to the range given in the above-mentioned embodiment, and various modification and change are possible within the range of the gist. is there. For example, a specific embodiment of device distribution and integration is not limited to the above embodiment, and all or a part thereof may be functionally or physically distributed and integrated in any unit. Can. In addition, new embodiments produced by any combination of a plurality of embodiments are also included in the embodiments of the present invention. The effects of the new embodiment generated by the combination combine the effects of the original embodiment.

  The control unit 11 (processor) of the control server 1 is a main body of each step (process) included in the device control method shown in FIGS. That is, the control unit 11 reads the device control program for executing the device control method shown in FIGS. 6 and 10 from the storage unit 13, and executes the device control program to control each portion of the control server 1. The device control method shown in FIGS. The steps included in the device control method shown in FIGS. 6 and 10 may be partially omitted, the order between the steps may be changed, and a plurality of steps may be performed in parallel.

S Device Control System 1 Control Server 111 Operation Detection Unit 112 Setting Acquisition Unit 113 Device Selection Unit 114 Device Control Unit 3 Device

Claims (15)

A detection unit configured to detect an operation of the user from an image obtained by imaging the user;
A selection unit that selects a plurality of devices based on the operation detected by the detection unit;
A device control unit that controls the plurality of devices selected by the selection unit;
A device control device having The selection unit selects the plurality of devices corresponding to the shape indicated by the trajectory of the movement.
The device control apparatus according to claim 1. The selection unit selects the plurality of devices located within a range in which the shape is extended starting from the position of the user.
The device control device according to claim 2. The selection unit selects the plurality of devices based on the position at which the operation is performed.
The device control apparatus according to any one of claims 1 to 3. The selection unit selects the plurality of devices based on the operation and environment information indicating an environment in which the operation is performed.
The device control apparatus according to any one of claims 1 to 4. The selection unit is based on the operation, the time when the operation was performed, the temperature of the position where the operation is performed, and the environment information indicating at least one of brightness of the position where the operation is performed. , Select the plurality of devices,
The device control device according to claim 5. The selection unit selects the plurality of devices based on the operation and user information indicating a user who has performed the operation.
The device control apparatus according to any one of claims 1 to 6. It further comprises an acquisition unit for acquiring setting information indicating setting of whether or not each of the plurality of devices can be selected,
The selection unit selects the plurality of devices indicating that the setting information is selectable based on the operation.
The device control apparatus according to any one of claims 1 to 7. After the device control unit performs the control of the plurality of devices, when the operation reverse to the control is performed on the plurality of devices within a predetermined time, the device control unit is configured to Setting the setting information that selection is impossible,
The device control apparatus according to claim 8. The system further includes an acquisition unit that acquires setting information indicating setting of whether or not selection is possible for each of the plurality of types of devices
The selection unit selects the plurality of devices indicating that the setting information is selectable based on the operation.
The device control apparatus according to any one of claims 1 to 7. The apparatus further comprises an acquisition unit that acquires setting information indicating a group of the plurality of devices.
When the plurality of devices selected by the selection unit do not include a part of the devices in the group indicated by the setting information, the device control unit may add the plurality of devices to the part of the devices. Perform the above control,
The device control apparatus according to any one of claims 1 to 7. The device control unit intermittently operates the some devices before performing the control of the some devices.
The device control apparatus according to claim 11. The detection unit detects, as the operation, a first operation of the user and a second operation of the user performed after the first operation.
The selection unit selects the plurality of devices based on the first operation,
The device control unit performs the control corresponding to the second operation on the plurality of devices selected by the selection unit.
The device control apparatus according to any one of claims 1 to 12. Detecting an action of the user from an image obtained by imaging the user;
Selecting a plurality of devices based on the operation detected in the detecting step;
A device control unit that controls the plurality of devices selected in the selecting step;
A device control method having: A plurality of devices and a device control device for controlling the plurality of devices;
The device control device
A detection unit configured to detect an operation of the user from an image obtained by imaging the user;
A selection unit that selects the plurality of devices based on the operation detected by the detection unit;
A device control unit that controls the plurality of devices selected by the selection unit;
Have
A device control system, wherein the plurality of devices operate according to the control by the device control unit.

Images