JP2019062270A - Device control apparatus, device control method, and device control system - Google Patents

Abstract

To suppress erroneous detection of operation when a device is controlled based on a user's operation.SOLUTION: A control server 1 according to an embodiment of the present invention includes an operation detection unit 111 that detects operation of a user from an image obtained by imaging the user, a feature information acquisition unit 112 that acquires user's feature information, a user determination unit 113 that determines whether to allow the user to control a device 3 on the basis of the feature information acquired by the feature information acquisition unit 112, and a device control unit 114 that performs control of the device 3 on the basis of the operation detected by the operation detection unit 111 when the user determination unit 113 permits the user to control the device 3.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an apparatus control apparatus, an apparatus control method, and an apparatus control system that control an apparatus based on a user's operation.

  There have been developed device control apparatuses that issue instructions to various devices installed in a house by detecting user's actions (gesture). Here, the operation of the user does not directly operate the operation unit of the device but refers to various operations performed by the user in the same space as or apart from the operation unit of the device. When the device control device detects that the user has performed a predetermined operation, it can turn on or off or adjust devices such as lighting, air conditioners, AV (Audio Visual) devices based on the operation.

  Patent Document 1 describes a technique for detecting an operation performed by a user in a user terminal and starting communication with a partner associated with the operation.

Patent No. 5820058

  In a house, various users may exist. In particular, children often perform various movements such as raising their hands, running and jumping. In such a situation where various users exist, the device control apparatus that controls the device based on the user's operation detects an operation not intended to instruct the device as an instruction to the device, or the like. There is a problem that false detection of

  However, in the technology described in Patent Document 1, it is assumed that the user and the user terminal are in a one-to-one relationship, and a situation in which various user actions can be detected is not assumed, and the operation is No measures have been taken against false positive detection.

  The present invention has been made in view of the above-described points, and provides an apparatus control apparatus, an apparatus control method, and an apparatus control system capable of suppressing erroneous detection of an operation when performing an apparatus control based on the user's operation. The purpose is to

  A device control apparatus according to a first aspect of the present invention includes: a detection unit that detects an operation of the user from an image obtained by imaging a user; an acquisition unit that acquires characteristic information of the user; A determination unit that determines whether to permit control of the device to the user based on the acquired characteristic information, and the detection unit determines that the determination unit permits control of the device based on the acquired characteristic information And a device control unit configured to control the device based on the detected operation.

  The acquisition unit acquires the size of at least a part of the user's body as the feature information, and the determination unit controls the device to the user when the size is within a predetermined range. May be permitted.

  The acquisition unit acquires information indicating a feature of the voice of the user as the feature information, and the determination unit controls the device when the information indicating the feature of the voice satisfies a predetermined condition. You may allow

  The acquisition unit acquires information indicating the feature of the face of the user as the feature information, and the determination unit controls the device when the information indicating the feature of the face satisfies a predetermined condition. You may allow

  The acquisition unit may acquire the temperature of the user as the feature information, and the determination unit may permit the user to control the device when the temperature is within a predetermined range.

  The acquisition unit acquires, as the characteristic information, whether or not the user holds a predetermined device, and the determination unit controls the device when the user holds the predetermined device. May be permitted to the user.

  The acquisition unit may acquire the position of the user as the feature information, and the determination unit may permit the user to control the device when the position is within a predetermined range.

  The acquisition unit acquires, as the feature information, a time zone in which the operation is performed by the user, and the determination unit instructs the user to control the device when the time zone is within a predetermined range. May be permitted.

  The acquisition unit may acquire the past operation of the user as the feature information, and the determination unit may permit the user to control the device when the past operation satisfies a predetermined condition. .

  When the detection unit detects the motion by a plurality of the users, the determination unit selects one of the plurality of the users based on the feature information of the plurality of users acquired by the acquisition unit. The selected one user may be allowed to control the device.

  When there are a plurality of the users in the image, the control of the device is controlled to a second user different from the first user based on the characteristic information of the first user among the plurality of users. May be permitted.

  The detection unit detects, as the operation, a first operation of the user and a second operation of the user performed after the first operation, and the device control unit detects the first operation by the first operation. The control corresponding to the second operation may be performed on the selected device.

  A device control method according to a second aspect of the present invention includes the steps of: detecting an operation of the user from an image obtained by imaging the user; acquiring characteristic information of the user; and acquiring the characteristic information of the user Based on the acquired characteristic information, determining whether to permit control of the device to the user, and detecting when control of the device is permitted to the user in the determining step And a device control unit configured to control the device based on the operation detected in the step.

  An apparatus control system according to a third aspect of the present invention includes an apparatus and an apparatus control apparatus for controlling the apparatus, wherein the apparatus control apparatus operates the user based on an image obtained by imaging the user. A detection unit that detects, an acquisition unit that acquires characteristic information of the user, and a determination unit that determines whether to permit the user to control the device based on the characteristic information acquired by the acquisition unit And a device control unit configured to control the device based on the operation detected by the detection unit when the determination unit grants control of the device to the user, the device including the device It operates according to the said control by a control part.

  According to the present invention, it is possible to suppress erroneous detection of an operation when controlling a device based on the user's operation.

It is a schematic diagram of the apparatus control system which concerns on 1st Embodiment. It is a schematic diagram which shows the operation | movement of the user which the control server which concerns on 1st Embodiment detects. It is a schematic diagram which shows the control database used by the control server which concerns on 1st Embodiment. FIG. 2 is a block diagram of a device control system according to the first embodiment. It is a figure which shows the flowchart of the apparatus control method which concerns on 1st Embodiment. It is a figure which shows the flowchart of the user determination processing which concerns on 1st Embodiment. It is a schematic diagram of the apparatus control system which concerns on 2nd Embodiment. It is a block diagram of the apparatus control system which concerns on 3rd Embodiment.

First Embodiment
FIG. 1 is a schematic view of a device control system S according to the present embodiment. The device control system S communicates with the control server 1 as a device control device, one or more sensor devices 2 for acquiring various information, and one or more devices 3 controlled by the control server 1. And an intermediary gateway 4. The device control system S may include devices such as other servers and terminals.

  The device 3 is a controlled device that is controlled to be turned on, off, adjusted, etc. by receiving a control signal from the control server 1. As the device 3, any device such as lighting, air conditioner, AV device, etc. can be used. The device 3 may operate not only by the control server 1 but also by an operation by the user. The device 3 has a communication module, and communicates with the control server 1 via the gateway 4 by wired communication or wireless communication.

  The sensor device 2 measures and acquires information such as an image, a sound, and a temperature. The sensor device 2 has various sensors necessary for acquiring information. The sensor device 2 may be incorporated in the device 3 or may be installed independently. The sensor device 2 has a communication module, and communicates with the control server 1 via the gateway 4 by wired communication or wireless communication.

  The gateway 4 is a relay device that performs processing such as protocol conversion on signals transmitted and received between the sensor device 2, the device 3, and the control server 1 to enable communication. The gateway 4 is connected to each of the sensor device 2 and the device 3 by wired connection or wireless connection, and is connected to the control server 1 via an arbitrary network N such as a local area network or the Internet.

  The control server 1 is connected to the network N by wire communication or wireless communication outside the building where the sensor device 2, the device 3 and the gateway 4 are installed. The control server 1 is configured by a single computer or a cloud that is a collection of computer resources. The control server 1 receives a signal indicating the information measured by the sensor device 2 via the gateway 4. Next, the control server 1 determines the device 3 to be controlled and the control content based on the information from the sensor device 2. Then, the control server 1 transmits a signal indicating the control content to the device 3 via the gateway 4.

[Description of user operation]
FIG. 2A and FIG. 2B are schematic views showing the operation of the user detected by the control server 1 according to the present embodiment. The actions of the user include specific actions and adjustment actions. The specific operation (first operation) is an operation for specifying the device 3 to be controlled, and the adjustment operation (second operation) is an operation for specifying control content.

  In the example of FIG. 2A, the specific action is an action of pointing a finger toward the device 3 to be controlled, and the adjustment action is an action of opening the hand after the specific action (that is, forming a hand shape). is there. Thereby, the control server 1 selects the device 3 located in the pointing direction (that is, the extension of the finger) as a control target, and the control content (for example, turning on the device 3) corresponding to the action of opening the hand. decide.

  The adjustment operation is an action of changing the shape of the hand or moving the hand after the specific operation. In the example of FIG. 2 (b), the specific action is an action of the user sleeping on a specific sofa, and the adjustment action is an action of holding a hand after the specific action (i.e., forming a hand). Thereby, the control server 1 selects the device 3 (for example, the illumination located above the sofa) associated with the sofa as the control target, and turns off the control content (for example, the device 3) corresponding to the action of holding the hand. To determine).

  The operation of the user shown in FIGS. 2A and 2B is an example, and the control server 1 is configured to detect a predetermined operation that can be determined from an image obtained by imaging the user. . The operation of the user is not limited to the two operations of the specific operation and the adjustment operation, but one operation of simultaneously specifying the control target and the control content may be used, or three or more operations may be used in combination.

[Description of control database D]
FIG. 3 is a schematic view showing a control database D used by the control server 1 according to the present embodiment. The control database D is information for determining the device 3 to be controlled and the control content from the operation of the user. The control database D stores the device number D1, the position information D2, the specific operation D3 and the adjustment operation D4 in association with each other.

  The device number D1 is a number for identifying the device 3. The device number D1 may be, for example, a serial number of the registered device 3, or may be a MAC (Media Access Control) address previously assigned to the device 3.

  The position information D2 is information indicating the position of the device 3, and is represented by, for example, three-dimensional coordinates X, Y, Z based on a predetermined point of a building. The position of the device 3 is used to identify the device 3 located in the pointing direction when the pointing operation is performed by the user. For example, the control server 1 estimates the position of the device 3 from an image obtained by imaging the device 3 with the sensor device 2. Alternatively, the user may set the position of each device 3 in the control server 1 using an application of a portable terminal or the like. In this case, for example, the user inputs coordinate information on the application or marks the map in the building displayed on the application to set the position of the device 3 in the control server 1. Although three-dimensional coordinates are used in this embodiment, two-dimensional coordinates may be used except for the coordinate in the height direction. In this case, the device 3 is identified by the horizontal position without distinguishing the height.

  The specific operation D3 is information indicating an operation of specifying the device 3 to be controlled. As the specific operation D3, it is possible to use, for example, pointing at the finger shown in FIG. 2A, or performing a specific operation at a specific place shown in FIG. 2B. A plurality of specific operations D3 may be set for one device 3 (device number D1).

  The adjustment operation D4 is information indicating an operation of specifying the control content. As the adjustment operation D4, for example, the hand shown in FIG. 2A, the hand held in FIG. 2B, the hand having a specific shape and moving in a specific direction, etc. are used. be able to. As control contents designated by the adjustment operation D4, any operation that can be executed by the device 3, such as on, off, adjustment, can be used. A plurality of adjustment operations D4 may be set for one device 3 (device number D1).

  The specific operation D3 and the adjustment operation D4 for each device 3 may be set in the control server 1 in advance. Alternatively, the user may set an arbitrary specific operation D3 and adjustment operation D4 for each device 3 in the control server 1 using an application or the like of the portable terminal.

  The control database D shown in FIG. 3 is an example, and the contents and format of the control database D are arbitrarily set as long as they can be used in the device control method according to the present embodiment. In FIG. 3, the control database D is represented by a table of character strings for visibility, but may be represented by any data format (file format) interpretable by the control server 1, for example, binary data or It may be text data. The control database D may be stored inside the control server 1 or may be stored in a storage device outside the control server 1.

[Configuration of Device Control System S]
FIG. 4 is a block diagram of the device control system S according to the present embodiment. In FIG. 4, arrows indicate the flow of main data, and there may be flows of data other than those shown in FIG. In FIG. 4, each block is not a hardware (apparatus) -based configuration but a function-based configuration. As such, the blocks shown in FIG. 4 may be implemented in a single device or may be implemented separately in multiple devices. Transfer of data between the blocks may be performed via any means such as a data bus, a network, a portable storage medium, and the like.

  The sensor device 2 includes an image sensor 21, a distance sensor 22, an audio sensor 23, and a temperature sensor 24. The image sensor 21 is a camera having an imaging element that outputs information indicating a captured image. The distance sensor 22 is a sensor that outputs information indicating a distance measured by irradiating, for example, an infrared ray or an ultrasonic wave to a subject (user). It is desirable that the distance sensor 22 be configured to be able to measure distances in a plurality of directions and to identify in which direction the distance is. The distance sensor 22 may be omitted, and the control server 1 may calculate the distance to the user from the image captured by the image sensor 21. In this case, the control server 1 estimates the position of the user from the images captured from a plurality of directions by the plurality of image sensors 21.

  The voice sensor 23 is a microphone that outputs information indicating voices obtained from the surroundings. It is desirable that the voice sensor 23 be equipped with, for example, a plurality of directional microphones and configured to be able to identify which direction the voice is from. The temperature sensor 24 is a sensor that outputs information indicating the temperature obtained from the surroundings. The temperature sensor 24 is preferably configured to be equipped with, for example, a plurality of radiation thermometers, and to identify in which direction the temperature is. The conversion of analog signals from the sensors 21 to 24 into digital data may be performed in each sensor or may be performed by the control server 1.

  The control server 1 includes a control unit 11, a communication unit 12, and a storage unit 13. The control unit 11 includes an operation detection unit 111, a feature information acquisition unit 112, a user determination unit 113, and a device control unit 114.

  The communication unit 12 is a communication interface for communicating with the sensor device 2 and the device 3 via the gateway 4. The communication unit 12 performs predetermined processing on communication signals received from the sensor device 2 and the device 3 to acquire data, and inputs the acquired data to the control unit 11. Further, the communication unit 12 performs predetermined processing on the data input from the control unit 11 to generate a communication signal, and transmits the generated communication signal to the sensor device 2 and the device 3.

  The storage unit 13 is a storage medium including a read only memory (ROM), a random access memory (RAM), a hard disk drive, and the like. The storage unit 13 stores in advance a program to be executed by the control unit 11.

  The control unit 11 is, for example, a processor such as a CPU (Central Processing Unit), and executes the program stored in the storage unit 13 to obtain an operation detection unit 111, a feature information acquisition unit 112, a user determination unit 113, and device control. It functions as the unit 114. At least a part of the functions of the control unit 11 may be performed by an electric circuit. In addition, at least a part of the functions of the control unit 11 may be executed by a program executed via a network.

  The device control system S according to the present embodiment is not limited to the specific configuration shown in FIG. For example, the control server 1 is not limited to one device, and may be configured by connecting two or more physically separated devices in a wired or wireless manner.

  The motion detection unit 111 determines the device 3 to be controlled and the control content based on the motion of the user indicated by the image captured by the sensor device 2. Specifically, first, the operation detection unit 111 receives information from the sensor device 2 via the communication unit 12. The motion detection unit 111 acquires an image captured by the image sensor 21 of the sensor device 2 from the received information. The motion detection unit 111 applies an image analysis technique of three-dimensional recognition to the acquired image to detect a motion and a position of the user. Known techniques can be used as image analysis techniques for three-dimensional recognition. The operation detection unit 111 repeats the detection of the operation until an operation that matches any of the specific operations (that is, the specific operation D3) included in the control database D is detected.

  When a finger pointing specific motion is detected, the motion detection unit 111 determines from the position information D2 of each device 3 in the control database D the device 3 positioned on the extension of the finger starting from the detected user position. (Ie, the device number D1) is determined as the control target. When another specific operation (such as sleeping on the sofa in FIG. 3) is detected, the operation detection unit 111 determines the device 3 associated with the specific operation in the control database D as a control target.

  After the control target is determined, the motion detection unit 111 acquires an image of the next frame (or after a predetermined time), applies the image analysis technology of three-dimensional recognition in the same manner, and detects the motion of the user. The operation detection unit 111 repeats the detection of the operation until an operation that matches any of the adjustment operations (that is, the adjustment operation D4) included in the control database D is detected. After the specific operation is detected, if the adjustment operation is not detected even after a predetermined time has elapsed, the operation detection unit 111 returns to the detection of the specific operation. When the adjustment operation is detected, the operation detection unit 111 determines the content (on, off, adjustment, etc.) designated by the adjustment operation in the control database D as the control content.

  In order to suppress erroneous detection of an operation, the operation detection unit 111 may detect a specific operation or an adjustment operation that matches the operation when the same operation is continued for a predetermined time.

  When the operation detection unit 111 determines the control target and control content associated with the user's operation, the feature information acquisition unit 112 transmits the image, the distance, the sound, and the temperature from the sensor device 2 via the communication unit 12. It receives and acquires it as user's feature information.

  Based on the feature information acquired by the feature information acquisition unit 112, the user determination unit 113 determines whether to permit the user who performed the operation to execute the control target and control content determined by the operation detection unit 111. Specifically, the user determination unit 113 determines permission or rejection of control using parameters of the user's face, voice, body size, body temperature, and past movement extracted from the feature information.

  In order to make the determination using the face of the user, the storage unit 13 stores in advance face information of the user who is permitted to control by the operation. The face information stored in the storage unit 13 may be a face image or a feature amount calculated from the face image. The user determination unit 113 specifies the face area of the user whose motion has been detected by the motion detection unit 111 from the image indicated by the feature information acquired by the feature information acquisition unit 112 using a person recognition technology, Compare with face information of each registered user. A well-known technique can be used as a person recognition technique. Then, the user determination unit 113 permits the control when the face area in the image matches the face information of any user registered in the storage unit 13 in advance, and rejects the control when not.

  As another method, the user determination unit 113 estimates the age from the face area in the image by person recognition technology, and permits control when the estimated age is within a predetermined range (for example, a predetermined value or more), Otherwise, control may be denied. A well-known technique can be used as a person recognition technique.

  In order to make a determination using the voice of the user, voice information of the user who permits control by the operation is stored in advance in the storage unit 13. The voice information stored in the storage unit 13 may be voice data, or may be a feature value (voiceprint) calculated from the voice. The user determination unit 113 estimates the voice in the direction in which the user whose motion is detected by the motion detection unit 111 among voices in a plurality of directions indicated by the feature information is the voice of the user. Next, the user determination unit 113 compares the estimated user's voice with the voice information of each user registered in advance in the storage unit 13 using voice analysis technology. Known techniques can be used as speech analysis techniques. Then, the user determination unit 113 permits the control when the estimated user's voice matches the voice information of any user registered in the storage unit 13 in advance, and rejects the control otherwise.

  As another method, the user determination unit 113 estimates the age from speech using speech analysis technology, and permits control when the estimated age is within a predetermined range (for example, a predetermined value or more), otherwise, You may deny control. Known techniques can be used as speech analysis techniques.

  In order to make a determination using the size of the user's body, the user determination unit 113 uses the human body detection technology to operate the motion detection unit 111 from the image indicated by the feature information acquired by the feature information acquisition unit 112. The detected human body area of the user is specified, and the distance from the sensor device 2 to the specified human body area is specified from the distance indicated by the feature information acquired by the feature information acquisition unit 112. Known techniques can be used as human body detection techniques.

  Specifically, the user determination unit 113 specifies the distance in the direction in which the user whose motion is detected by the motion detection unit 111 among the distances in the plurality of directions indicated by the feature information is the distance to the human body region. Then, the user determination unit 113 estimates the actual size of the user in the three-dimensional space from the size in the image of the specified human body region and the specified distance. As the size of the user, the size of the user's entire body (for example, height) or the size of a part of the user's body (for example, palm) can be used. The user determination unit 113 permits the control when the estimated user size (for example, the height or the palm size) is within a predetermined range (for example, a predetermined value or more), and rejects the control otherwise.

  From the temperature indicated by the feature information acquired by the feature information acquisition unit 112, the user determination unit 113 determines the body temperature (skin temperature) of the user whose operation has been detected by the operation detection unit 111, in order to perform determination using the user's body temperature. presume. Specifically, the user determination unit 113 estimates the temperature in the direction in which the user whose operation is detected by the operation detection unit 111 among the temperatures in the plurality of directions indicated by the feature information is the user's body temperature. Then, the user determination unit 113 permits control when the estimated user's body temperature is within a predetermined range (for example, less than or equal to a predetermined value), and rejects control otherwise.

  In order to perform determination using the past motion of the user, the storage unit 13 stores the motion (movement) of the user in the past predetermined time in the image indicated by the feature information acquired by the feature information acquisition unit 112 Ru. As the movement of the user, the time-series position of the human body detected using the above-described human body detection technique can be used. The user determination unit 113 acquires, from the storage unit 13, the motion of the user whose motion has been detected by the motion detection unit 111 within the past predetermined time, and calculates the speed and direction of the motion. Then, the user determination unit 113 permits control when the calculated speed and direction of movement of the user satisfy a predetermined condition (for example, the speed is equal to or less than a predetermined value and the direction is horizontal). Reject

  The apparent height of the user changes as the user sits and stands. The user determination unit 113 may use the past motion of the user to appropriately determine the height of the user. In this case, the user determination unit 113 acquires, from the storage unit 13, the motion within the past predetermined time of the user whose motion is detected by the motion detection unit 111, and the motion (for example, moving in the vertical direction at the same position) ) To determine that the user has sat or stood. Then, when it is determined that the user is sitting, the user determination unit 113 corrects the apparent height estimated from the image to a large size (for example, 1.5 times).

  The user determination unit 113 may determine permission of control when all of the parameters of the user's face, voice, body size, body temperature, and past motion satisfy the conditions described above, and one of the parameters may be selected. The permission of control may be determined when the unit satisfies the condition. It is preferable to use more conditions for determining permission of control, since the accuracy of suppressing false detection is improved.

  When the user determination unit 113 determines that the user's control is permitted, the device control unit 114 generates a control signal for performing control in accordance with the control content determined by the operation detection unit 111, and causes the communication unit 12 to It transmits to the apparatus 3 of the control object determined by the operation | movement detection part 111 via.

  The device 3 performs predetermined operations such as on, off, and adjustment in accordance with the control signal received from the control server 1 via the gateway 4.

  As described above, the control server 1 according to the present embodiment determines, based on the feature information of the user who performed the operation (gesture), whether or not to permit control of the device 3 by the operation, for example, by the operation of the child Unintended control can be suppressed.

  If an image is constantly transmitted from the sensor device 2 to the control server 1, the communication load is large. Therefore, the sensor device 2 only detects an operation close to a predetermined operation (specific operation and adjustment operation) from the image. The transmission of the image of may be started. In this case, the sensor device 2 is configured as a computer that executes the operation detection software, and registers in advance an operation close to the specific operation and the adjustment operation. The sensor device 2 starts transmission of an image to the control server 1 when detecting a previously registered operation from an image captured by the image sensor 21 using the operation detection software. Thereby, the amount of communication of data transmitted from the sensor device 2 to the control server 1 can be reduced.

[Flowchart of Device Control Method]
FIG. 5 is a diagram showing a flowchart of the device control method according to the present embodiment. The flowchart of FIG. 5 is started, for example, when the user performs a predetermined start operation on the control server 1.

  First, the operation detection unit 111 detects the user's operation (that is, the above-described specific operation and adjustment operation) from the image captured by the sensor device 2, and determines the device 3 to be controlled and the control content based on the detected operation. (S1).

  When the operation detection unit 111 does not detect an operation in step S1 (NO in S2), the control server 1 returns to step S1 and repeats the process. When the operation detection unit 111 detects an operation in step S1 (YES in S2), the control server 1 executes a user determination process described later with reference to FIG. 6 (S3).

  When control is permitted in the user determination process of step S3 (YES in S4), the device control unit 114 generates a control signal for performing control in accordance with the control content determined based on the operation in S1, and S1 It transmits via the communication part 12 to the apparatus 3 of the control object determined based on operation | movement by (S5). If the control is rejected in the user determination process of step S3 (NO in S4), the control server 1 proceeds to the next step without performing step S5.

  If a predetermined termination condition (for example, the user performs a predetermined termination operation on the control server 1) is not satisfied (NO in S6), the control server 1 returns to step S1 and repeats the processing. If the predetermined termination condition is satisfied (YES in S6), the control server 1 terminates the process.

[Flowchart of user determination processing]
FIG. 6 is a view showing a flowchart of user determination processing according to the present embodiment. The flowchart of FIG. 6 is executed in step S3 of the device control method of FIG.

  The feature information acquisition unit 112 receives the image, the distance, the sound, and the temperature from the sensor device 2 through the communication unit 12 and acquires the image as the user's feature information (S31). The user determination unit 113 determines whether the parameters of the user's face, voice, body size, body temperature, and past movement extracted from the feature information acquired by the feature information acquisition unit 112 in step S31 each satisfy predetermined conditions Is determined (S32).

  If the user determination unit 113 determines that at least one of the user's face, voice, body size, body temperature, and past motion parameters extracted from the feature information in step S32 satisfies the predetermined condition (YES in S33 And the user determination unit 113 determines that the user is permitted to control (S34).

  When the user determination unit 113 determines that the parameters of the user's face, voice, body size, body temperature, and past motion extracted from the feature information in step S32 do not satisfy predetermined conditions (NO in S33) The user determination unit 113 determines to deny the control to the user (S35).

  In step S32, the user determination unit 113 permits control when at least one of the face, voice, body size, body temperature, and past motion parameters extracted from the feature information of the user satisfies the condition, Control may be permitted if all of the parameters satisfy the condition. Not limited to these, conditions required for permission of control are arbitrarily set in the control server 1.

  After permission of control is determined in step S34 or after determination of refusal of control in step S35, the control server 1 returns to the flowchart of FIG.

[Effect of First Embodiment]
Children often perform various movements such as raising their hands, running and jumping. The control server 1 which concerns on this embodiment determines the decision | availability of control by a user's operation | movement using the characteristic information of the user which can distinguish a child and an adult.

  Since the face of the child and the face of the adult can be distinguished by wrinkles etc., the control server 1 distinguishes children and adults by the face of the user. Also, since the voice of the child tends to be higher than the voice of the adult, the control server 1 distinguishes the child and the adult by the voice of the user. Also, since the child's body tends to be smaller than the adult's body, the control server 1 distinguishes the child and the adult according to the size of the user's body (for example, the height and the size of the palm). In addition, since the temperature of the child (skin temperature) tends to be higher than that of the adult, the control server 1 distinguishes the child and the adult by the temperature of the user. Also, since the children tend to run and bounce, the control server 1 distinguishes children and adults by their past movements (speed and direction).

  Then, the control server 1 permits control of the device by the operation performed by the user estimated to be an adult, and rejects control of the device by the operation performed by the user estimated as a child. With such a configuration, it is possible to suppress false detection of various operations performed by a child as operations of control of the device.

  In addition, the control server 1 misdetects the operation performed by the unregistered user by determining whether the face or voice is a registered user, not limited to distinction between a child and an adult, and controls the device. It can suppress doing.

[Modification of First Embodiment]
In the above-described device control method, the control server 1 constantly detects the operation, but may start the detection of the operation triggered by the detection of the specific start voice. In this case, the sensor device 2 includes an audio sensor such as a microphone. First, the control server 1 does not detect an operation before the start voice is detected, and starts detection of the start voice. The control server 1 detects a predetermined start sound from the sound acquired from the sound sensor of the sensor device 2. The start voice is, for example, voice stored in the control server 1 such as “OK” or “Hi”. Also, the start voice may be voice pre-recorded on the control server 1 by the user.

  For example, the control server 1 applies speech recognition technology to the speech acquired from the speech sensor of the sensor device 2 and detects the speech even if the speech corresponding to the speech is included in the speech. Good. Known techniques can be used as the speech recognition technique. In addition, when the frequency spectrum of the voice acquired from the voice sensor of the sensor device 2 includes the predetermined ratio or more of the frequency spectrum of the start voice recorded in advance, the control server 1 detects the start voice. It is also good. The predetermined ratio is a ratio considered to be coincident with the start speech, and is, for example, 90%. In addition, the control server 1 starts the start voice when the degree of correlation between the pattern of the frequency spectrum of the voice acquired from the voice sensor of the sensor device 2 and the pattern of the frequency spectrum of the start voice recorded in advance is equal to or more than a predetermined value. May be detected.

  Then, when detecting the start voice, the control server 1 starts the detection of the operation and performs the above-described device control method. The control server 1 stops the detection of the operation when a predetermined time has elapsed from the time when the start voice is detected, and detects the start voice again.

  With such a configuration, the control server 1 detects an operation from an image only during a predetermined period after detecting the start voice. Therefore, the processing load and communication load of the control server 1 can be reduced.

Second Embodiment
In the first embodiment, the control server 1 is disposed outside the building and communicates with the sensor device 2 and the device 3 via the network N. Therefore, depending on the communication state of the network N, it may take time to execute the operation detection and the control of the device. Further, when an image is constantly transmitted from the sensor device 2 to the control server 1, the communication load is large. On the other hand, in the present embodiment, in addition to the control server 1-1 disposed outside the building, the control server 1-2 is provided in the building. By providing the control server 1-2 in the building, it is possible to reduce the time required to execute the operation detection and control of the device. As a result, compared with direct operation by touching the switch or the remote control of the device, time and effort can be reduced without spending time, and convenience for the user is improved.

  FIG. 7 is a schematic view of the device control system S according to the present embodiment. The device control system S includes, in addition to the sensor device 2, the device 3 and the gateway 4 as in the first embodiment, the control server 1-1 connected to the network N outside the building, and the gateway 4 in the building It includes a control server 1-2 connected to the network N. The control server 1-1 and the control server 1-2 each have the same configuration as that of the control server 1 of FIG.

  In the present embodiment, the speed and accuracy of the motion detection are balanced by adjusting processing and data between the control server 1-2 in the building and the control server 1-1 outside the building. For example, the control server 1-2 in the building performs operation detection processing with a small amount of calculation, and the control server 1-1 outside the building performs operation detection processing with a large amount of calculation. Alternatively, for example, a small amount of control database D is stored in the storage unit 13 of the control server 1-2 in the building, and a large amount of control database D is stored in the storage unit 13 of the control server 1-1 outside the building.

  First, the control server 1-2 in the building detects an operation based on the information from the sensor device 2. And when control server 1-2 can not detect operation, control server 1-2 transfers information from sensor device 2 to control server 1-1 outside a building, and control server 1-1 is a sensor device. The operation is detected based on the information from 2. One of the control server 1-2 and the control server 1-1 controls the device 3 based on the detected operation.

[Effect of Second Embodiment]
Generally, the control server 1-2 in the building does not have a communication load because it does not go through the network N, but there is a restriction on the device configuration due to the installation space and the installation cost. On the other hand, the control server 1-1 outside the building can perform high-speed processing because it can have a high-level equipment configuration, but it imposes a communication load. Therefore, the device control system S according to the present embodiment performs simple operation detection in the control server 1-2 in the building, and performs complicated operation detection in the control server 1-1 outside the building. Such a configuration can improve the speed of motion detection while maintaining the accuracy of motion detection.

Third Embodiment
In this embodiment, in addition to the user's own information such as the user's face, voice, body size, body temperature, and past movements, the user's location, the device possessed by the user, the time when the user performed an action, etc. The information on the environment surrounding the user is used to determine whether the user can control. The other configurations and processes are similar to those of the first embodiment.

  FIG. 8 is a block diagram of a device control system S according to the present embodiment. The sensor device 2 includes a position measurement unit 25 and a short distance communication unit 26 in addition to the image sensor 21, the distance sensor 22, the voice sensor 23, and the temperature sensor 24 similar to those of the first embodiment.

  The position measurement unit 25 measures the position of the user. The position of the user is, for example, information indicating a room in which the user is present. The position measurement unit 25 stores in advance a map indicating the position of the sensor device 2 itself and the room allocation of the building. The position measuring unit 25 estimates which room on the map the user has by estimating the position of the user based on the position of the sensor device 2 from the image captured by the image sensor 21 and the distance measured by the distance sensor 22. Identify who is in Alternatively, the position measurement unit 25 may estimate the position of the user from the images captured from a plurality of directions by the plurality of image sensors 21. The position measurement unit 25 outputs information indicating the position of the user.

  The short distance communication unit 26 is a communication interface having an antenna or the like for performing short distance communication with a device possessed by the user. The short distance communication is, for example, Bluetooth (registered trademark). The device possessed by the user is, for example, a watch having a near field communication function. The short distance communication unit 26 stores in advance information identifying a device possessed by the user. The short distance communication unit 26 tries short distance communication with the device possessed by the user, and outputs information indicating whether or not the short distance communication is possible.

  The feature information acquisition unit 112 acquires, from the sensor device 2, the position of the user and the possibility of near field communication as feature information in addition to the image, distance, sound, and temperature. Also, the feature information acquisition unit 112 acquires, as feature information, the time when the operation detection unit 111 detects an operation.

  The user determination unit 113 adds the user's face, voice, body size, body temperature, and past motion parameters as in the first embodiment, as well as the position of the user extracted from the feature information and whether near-field communication is possible. And the parameter of the time at which the user performed the operation determine permission or denial of control.

  In order to make a determination using the position of the user, the storage unit 13 stores in advance determination criteria for each position (for example, a room) in the building. Then, the user determination unit 113 sets a determination criterion corresponding to the position of the user indicated by the feature information acquired by the feature information acquisition unit 112 in accordance with the determination criterion for each position stored in the storage unit 13. For example, the user determination unit 113 permits control by a user whose height is small in the living room, and rejects control by a user whose height is small in the study. With such a configuration, for example, it is possible to set, for each room, whether or not control by the operation of a child is possible.

  In order to make a determination using the device possessed by the user, the user determination unit 113 determines that the user who performed the operation possesses the device based on the possibility of the short distance communication indicated by the feature information acquired by the feature information acquiring unit 112. Determine if there is. Specifically, the user determination unit 113 determines that the user holds the device when the short distance communication is possible, and determines that the device is not held otherwise. Then, the user determination unit 113 permits control when it is determined that the user holds the device, and otherwise rejects control. Such a configuration can allow a user wearing a particular device to gain operational control.

  In order to perform determination using the time when the user performed an operation, the storage unit 13 stores in advance determination criteria for each time zone. Then, the user determination unit 113 sets a determination criterion corresponding to the time when the user indicated by the feature information acquired by the feature information acquisition unit 112 performs in accordance with the determination criterion for each time zone stored in the storage unit 13. For example, the user determination unit 113 permits control by a user whose height is small in the daytime, and rejects control by a user whose height is small in the night. With such a configuration, it is possible to set, for example, whether or not the control by the operation of the child can be performed for each time zone.

  The user determination unit 113 may determine whether or not control is possible using only information on the environment surrounding the user, such as the location of the user, the device possessed by the user, and the time at which the user performed an operation. The control availability may be determined in combination with the user's own information such as the user's face, voice, body size, body temperature, and past movements.

[Effect of the third embodiment]
In this embodiment, the same effect as that of the first embodiment is obtained by determining whether or not control by operation is possible based on the environment surrounding the user. Furthermore, by combining the information of the user's own such as the face and the height with the information of the environment such as the position and the time, it is possible to determine whether or not the control is finer.

Fourth Embodiment
In the present embodiment, when the actions of a plurality of users are simultaneously detected, the priority is determined based on the user's feature information, and the control by any user's action is performed according to the priorities. Thereby, for example, when an adult and a child operate at the same time, control can be permitted by giving priority to the operation of the adult. Note that simultaneous here includes substantially simultaneous.

  When a plurality of user actions are performed simultaneously, the action detection unit 111 detects the action and position of each user. Then, the operation detection unit 111 determines the control target and the control content associated with the operation of each user.

  The feature information acquisition unit 112 receives information from the sensor device 2 via the communication unit 12 and acquires an image, a distance, a sound, and a temperature indicated by the information as feature information. The feature information includes one or more information of distances in a plurality of directions, voices, and temperatures, and it is possible to specify which user in the image the information relates to.

  The user determination unit 113 determines which user the execution of the control target and the control content determined by the operation detection unit 111 is permitted based on the feature information of the plurality of users acquired by the feature information acquisition unit 112. Specifically, the user determination unit 113 extracts face, voice, body size, body temperature, and past motion parameters from the feature information for each of a plurality of users, and based on whether they satisfy the conditions To determine the priority. The conditions applied to each parameter are the same as in the first embodiment. For example, the priority is set to be higher as the number of parameters satisfying the condition is larger, and to be lower as the number is smaller. Also, the contribution to the priority may be set to be different for each type of parameter (for example, when the condition of the face is satisfied, the priority is higher than when the condition of the body temperature is satisfied). The priority shown here is an example, and the user may arbitrarily set the priority.

  Then, the user determination unit 113 selects one of the plurality of users having the highest priority, permits control by the operation of the selected one user, and rejects control by the operation of the other user.

[Effect of Fourth Embodiment]
In the present embodiment, even when operations by a plurality of users are simultaneously performed, the device control can be smoothly performed by selecting any one of the users according to the priority determined based on the feature information. Can.

Fifth Embodiment
While the first embodiment determines whether or not control by an operation is possible based on the characteristic information of the user who performed the operation, the second embodiment is a second embodiment in the vicinity of the first user who performed the operation. Based on the user's feature information, it is determined whether or not the control by the operation is possible. Thus, for example, as in the case of so-called parental control, it is possible to deny control by the operation of the child alone and permit control by the operation of the child accompanied by an adult.

  The motion detection unit 111 detects the motion and position of the first user when the first user and the second user exist in the image captured by the sensor device 2 and associates the motion and the position with the first user's motion. Determine the control target and the control content. The feature information acquisition unit 112 receives the image, the distance, the sound, and the temperature from the sensor device 2 through the communication unit 12 and acquires it as feature information. The feature information includes one or more information on distances in a plurality of directions, voice, and temperature, and can specify information on a second user in the vicinity of the first user.

  The user determination unit 113 determines, among the feature information acquired by the feature information acquisition unit 112, the control target and the control content determined by the operation detection unit 111 based on the feature information of the second user who is near the first user. It is determined whether or not the first user is permitted to execute. That is, the user determination unit 113 extracts the parameters of the face, voice, body size, body temperature, and past movement of the second user who is not the first user who performed the operation but is located near the first user. Alternatively, it is determined whether a plurality of parameters satisfy the condition. The conditions applied to each parameter are the same as in the first embodiment. Then, when the parameter of the second user satisfies the condition, the user determination unit 113 permits control by the operation of the first user.

[Effect of Fifth Embodiment]
In this embodiment, since the controllability is determined based on the feature information of the second user located in the vicinity of the first user who performed the operation, the operation is performed even if the first user does not satisfy the condition. Control can be granted.

  As mentioned above, although the present invention was explained using an embodiment, the technical scope of the present invention is not limited to the range given in the above-mentioned embodiment, and various modification and change are possible within the range of the gist. is there. For example, a specific embodiment of device distribution and integration is not limited to the above embodiment, and all or a part thereof may be functionally or physically distributed and integrated in any unit. Can. In addition, new embodiments produced by any combination of a plurality of embodiments are also included in the embodiments of the present invention. The effects of the new embodiment generated by the combination combine the effects of the original embodiment.

  The control unit 11 (processor) of the control server 1 is a main body of each step (process) included in the device control method shown in FIGS. That is, the control unit 11 reads the device control program for executing the device control method shown in FIGS. 5 and 6 from the storage unit 13, and executes the device control program to control each portion of the control server 1. The device control method shown in FIGS. 5 and 6 is executed. The steps included in the device control method shown in FIGS. 5 and 6 may be partially omitted, the order of the steps may be changed, and a plurality of steps may be performed in parallel.

S Device Control System 1 Control Server 111 Operation Detection Unit 112 Feature Information Acquisition Unit 113 User Determination Unit 114 Device Control Unit 3 Device

Claims (14)

A detection unit configured to detect an operation of the user from an image obtained by imaging the user;
An acquisition unit for acquiring the characteristic information of the user;
A determination unit that determines whether to permit the user to control the device based on the feature information acquired by the acquisition unit;
A device control unit that controls the device based on the operation detected by the detection unit when the determination unit grants control of the device to the user;
A device control device having The acquisition unit acquires the size of at least a part of the user's body as the feature information.
The determination unit permits the user to control the device when the size is within a predetermined range.
The device control apparatus according to claim 1. The acquisition unit acquires information indicating a feature of the voice of the user as the feature information.
The determination unit permits the user to control the device when the information indicating the feature of the voice satisfies a predetermined condition.
The device control apparatus according to claim 1. The acquisition unit acquires information indicating the feature of the face of the user as the feature information.
The determination unit permits the user to control the device when the information indicating the feature of the face satisfies a predetermined condition.
The device control apparatus according to any one of claims 1 to 3. The acquisition unit acquires the temperature of the user as the feature information.
The determination unit permits the user to control the device when the body temperature is within a predetermined range.
The device control apparatus according to any one of claims 1 to 4. The acquisition unit acquires, as the feature information, whether the user holds a predetermined device.
The determination unit permits the control of the device to the user when the user possesses the predetermined device.
The device control apparatus according to any one of claims 1 to 5. The acquisition unit acquires the position of the user as the feature information,
The determination unit permits the user to control the device when the position is within a predetermined range.
The device control apparatus according to any one of claims 1 to 6. The acquisition unit acquires, as the feature information, a time zone in which the operation is performed by the user.
The determination unit permits the user to control the device when the time zone is within a predetermined range.
The device control apparatus according to any one of claims 1 to 7. The acquisition unit acquires a past operation of the user as the feature information.
The determination unit permits the user to control the device when the past operation satisfies a predetermined condition.
The device control apparatus according to any one of claims 1 to 8. When the detection unit detects the motion by a plurality of the users, the determination unit selects one of the plurality of the users based on the feature information of the plurality of users acquired by the acquisition unit. , Allowing the selected one of the users to control the device,
The device control apparatus according to any one of claims 1 to 9. When there are a plurality of the users in the image, the control of the device is controlled to a second user different from the first user based on the characteristic information of the first user among the plurality of users. To give permission,
The device control apparatus according to any one of claims 1 to 10. The detection unit detects, as the operation, a first operation of the user and a second operation of the user performed after the first operation.
The device control unit performs the control corresponding to the second operation on the device selected by the first operation.
The apparatus control apparatus as described in any one of Claims 1-11. Detecting an action of the user from an image obtained by imaging the user;
Acquiring the characteristic information of the user;
Determining whether to permit the user to control the device based on the feature information acquired in the acquiring step;
A device control unit configured to control the device based on the operation detected in the detecting step when control of the device is permitted to the user in the determining step;
A device control method having: An apparatus and an apparatus control device for controlling the apparatus;
The device control device
A detection unit configured to detect an operation of the user from an image obtained by imaging the user;
An acquisition unit for acquiring the characteristic information of the user;
A determination unit that determines whether to permit the user to control the device based on the feature information acquired by the acquisition unit;
A device control unit that controls the device based on the operation detected by the detection unit when the determination unit grants control of the device to the user;
Have
The device control system, wherein the device operates according to the control by the device control unit.

Images