JP2016162164A - Operation device and operation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an operation device capable of easily performing operation with respect to equipment, even in a situation where the equipment to be operated cannot be directly operated and to provide an operation method.SOLUTION: The operation device for controlling the operation in the equipment to be operated includes an imaging part which photographs an image including a face, a face recognition part which recognizes the face included in the image and outputs the information of a feature point indicating the position of the part of the recognized face as face recognition information, a face direction detection part which detects the direction of the face on the basis of the face recognition information and outputs face direction information indicating the detected face direction, a face distance detection part which detects a distance between the imaging part and the face on the basis of the face recognition information and outputs face distance information indicating the detected distance, and a control part which refers to operation information indicating an operation content for the equipment previously set according to the direction of the face and the distance between the imaging part and the face and outputs an operation signal indicating the operation with respect to corresponding equipment to the equipment on the basis of the face direction information and the face distance information.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an operating device and an operating method.

  For example, while driving the vehicle, the line of sight cannot be directed to the operation target device (hereinafter referred to as “operation target device”) mounted on the vehicle, and further, the operation cannot be performed by groping. There is a situation. For this reason, there is a need for a method that can perform a desired operation on an operation target device even in a situation where the operation target device cannot be directly operated. In order to realize such a demand, for example, a technique for performing an operation on an operation target device by voice as in Patent Document 1 has been proposed.

JP 2000-322078 A

  However, in the technique for performing an operation on the operation target device by voice as disclosed in Patent Document 1, the operation is instructed interactively for the purpose of preventing the operation target device from malfunctioning. . For this reason, there is a problem that it takes time for the operation target device to accept an operation instructed by voice. In addition, when the voice during the conversation is misrecognized, there is a problem that it is difficult to correct the correct operation, such as starting the conversation again from the beginning.

  The present invention has been made based on the above-described problems, and provides an operating device and an operating method that can easily operate an apparatus even in a situation where the apparatus to be operated cannot be directly operated. It is an object.

  In order to solve the above-described problem, an operation device according to one embodiment of the present invention is an operation device that controls an operation in a device to be operated, and is included in an imaging unit that captures an image including a face, and the image A face recognition unit for recognizing the face and outputting feature point information representing the position of the recognized face part as face recognition information; detecting the orientation of the face based on the face recognition information; A face direction detection unit that outputs face direction information indicating the orientation of the face, a distance between the imaging unit and the face is detected based on the face recognition information, and face distance information indicating the detected distance A face distance detection unit that outputs the operation information representing the operation content for the device determined in advance according to the orientation of the face and the distance between the imaging unit and the face, and the face direction information Based on the face distance information, the corresponding machine An operation signal representing the operation to, and a control unit for outputting to the device.

  Further, an operation method according to one embodiment of the present invention is an operation method in an operation device that controls an operation in a device to be operated, and recognizes the face included in an image including a face photographed by an imaging unit, A face recognition step of outputting information of feature points representing the position of the recognized face part as face recognition information, detecting the face direction based on the face recognition information, and expressing the detected face direction A face direction detecting step for outputting face direction information; a face distance detecting step for detecting a distance between the imaging unit and the face based on the face recognition information; and outputting face distance information representing the detected distance. Based on the face orientation information and the face distance information, referring to the operation information representing the operation contents for the device determined in advance according to the orientation of the face and the distance between the imaging unit and the face. The corresponding said An operation signal representing the operation to the vessel, and a control step of outputting to the device.

  According to the present invention, it is possible to easily perform an operation on a device even in a situation where the device to be operated cannot be directly operated.

It is the block diagram which showed schematic structure of the operating device in embodiment of this invention. It is the flowchart which showed the process sequence of the process in the operating device of embodiment of this invention. It is a figure explaining the process of face recognition in the face recognition part with which the operating device of embodiment of this invention was equipped. It is a figure explaining an example of the detection process of the face direction in the face direction detection part with which the operating device of embodiment of this invention was equipped. It is a figure explaining another example of the detection process of the face direction in the face direction detection part with which the operating device of embodiment of this invention was equipped. It is a figure explaining another example of the detection process of the face direction in the face direction detection part with which the operating device of embodiment of this invention was equipped. It is a figure explaining another example of the detection process of the face direction in the face direction detection part with which the operating device of embodiment of this invention was equipped. It is a figure explaining an example of the detection process of the face distance in the face distance detection part with which the operating device of embodiment of this invention was equipped. It is a figure explaining another example of the detection process of the face distance in the face distance detection part with which the operating device of embodiment of this invention was equipped. It is a figure explaining another example of the detection process of the face distance in the face distance detection part with which the operating device of embodiment of this invention was equipped. It is a figure explaining another example of the detection process of the face distance in the face distance detection part with which the operating device of embodiment of this invention was equipped. It is the figure which showed typically an example of the distance of the operating device of embodiment of this invention, and the face to detect. It is the figure which showed an example of the content operated according to the state of the face which the operating device of embodiment of this invention detected. It is the figure which showed typically an example of the positional relationship of the operating device of embodiment of this invention, and the face to detect.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of an operating device according to an embodiment of the present invention. The operating device 10 includes a control unit 100, an imaging unit 200, a storage unit 300, and a communication unit 400. The operation device 10 is a user interface device for outputting an instruction for performing an operation desired by a user (user) to a device to be operated (not shown) (hereinafter referred to as “operation target device”).

  The control unit 100 controls each component in the controller device 10, that is, the entire controller device 10. The control unit 100 includes a face recognition unit 110, a face direction detection unit 120, and a face distance detection unit 130, and is realized by a processing device such as a CPU (Central Processing Unit), for example.

  The imaging unit 200 includes a camera as a shooting function, and takes an image in accordance with control from the control unit 100. In the operation device 10, the imaging unit 200 captures an image including the user's face at a short predetermined time interval. The imaging unit 200 stores the captured images in the storage unit 300 sequentially.

  The storage unit 300 stores a program for the control unit 100 or components in the control unit 100 to execute various processes and data used for the processes. The storage unit 300 includes, for example, various memories such as a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory (Flash Memory), and a storage device such as an HDD (Hard Disk Drive). The storage unit 300 stores (writes) data and outputs (reads) data in accordance with control from the control unit 100.

  The communication unit 400 communicates with the operation target device via a wireless network, a wired network, or the like in accordance with control from the control unit 100. More specifically, for example, the communication unit 400 uses a short-range wireless communication such as Bluetooth (registered trademark) or a wireless communication such as WiFi (registered trademark) to indicate a signal (operation indicating a user's desired operation). Signal) is output (transmitted) to the operation target device. In addition, for example, the communication unit 400 transmits an operation signal to the operation target device using a communication cable connected to the operation target device.

  The face recognition unit 110 is a processing unit that performs face recognition processing for recognizing the entire face of the user included in the image captured by the imaging unit 200. The face recognition unit 110 sequentially performs face recognition processing on each image captured at a predetermined short time interval by the imaging unit 200, and the result of the face recognition processing is determined as the face direction detection unit 120 and the face distance detection. Sequentially output to the unit 130. That is, the face recognition unit 110 sequentially outputs the information of the recognized user's face (hereinafter referred to as “face recognition information”) to the face direction detection unit 120 and the face distance detection unit 130. Thereby, the face direction detection part 120 and the face distance detection part 130 can perform each process accurately. Note that the processing function of the face recognition process in the face recognition unit 110 may be realized by software that executes a program stored in the storage unit 300, for example. Detailed description regarding the processing of the face recognition unit 110 will be described later.

  The face orientation detection unit 120 is a processing unit that detects the orientation of the user's face based on the user's face recognition information input from the face recognition unit 110. The face orientation detection unit 120 sequentially performs detection processing (hereinafter referred to as “face orientation detection processing”) for detecting the orientation of the user's face based on each face recognition information sequentially input from the face recognition unit 110. Thereby, the face direction detection unit 120 can track changes in the user's face direction (for example, changes in the vertical direction and changes in the horizontal direction) from the results of the respective face direction detection processes, for example, The inclination of the user's face in the vertical direction and the horizontal direction can be detected. The face orientation detection unit 120 outputs information indicating the detected face orientation of the user (hereinafter referred to as “face orientation information”). Note that the processing function of the face orientation detection process in the face orientation detection unit 120 may be realized by software that executes a program stored in the storage unit 300, for example. A detailed description of the processing of the face orientation detection unit 120 will be described later.

  The face distance detection unit 130 is a processing unit that detects the distance between the user's face and the camera included in the imaging unit 200 based on the user's face recognition information input from the face recognition unit 110. The face distance detection unit 130 performs detection processing (hereinafter referred to as “face distance detection processing”) that detects the distance between the user's face and the camera based on each face recognition information sequentially input from the face recognition unit 110. Perform sequentially. Accordingly, the face distance detection unit 130 can track a change in the distance between the user's face and the camera (for example, a change in the front-rear (front and back) direction) from the result of each face distance detection process. For example, it is possible to detect the movement of the user's face toward the front or back. The face distance detection unit 130 outputs information indicating the distance between the detected user's face and the camera (hereinafter referred to as “face distance information”). Note that the processing function of the face distance detection process in the face distance detection unit 130 may be realized by software that executes a program stored in the storage unit 300, for example. Detailed description regarding the processing of the face distance detection unit 130 will be described later.

  In the operation device 10, operation information indicating the operation content for the operation target device according to the state of the user's face is set in advance. The operation information is information in which the operation target device is associated with the content of the operation that can be instructed to the operation target device, and is stored in the storage unit 300, for example. The control unit 100 refers to the operation information, and operates according to the face direction information output from the face direction detection unit 120 and the face distance information output from the face distance detection unit 130, that is, an operation signal representing an operation desired by the user. Is transmitted to the operation target device via the communication unit 400.

  Next, a processing procedure in which the operation device 10 operates the operation target device according to the state of the user's face will be described. FIG. 2 is a flowchart illustrating a processing procedure of processing (operation processing of the operation target device) in the operation device 10 according to the embodiment of the present invention. In the following description, the operating device 10 has already recognized an initial state that becomes a reference when detecting a change in the face state of the user, and the initial state is stored in the storage unit 300, for example. The explanation will be made assuming that

  When the controller device 10 starts operating, first, the control unit 100 activates the imaging unit 200 (step S101). Then, the control unit 100 causes the imaging unit 200 to capture an image with the connected camera (step S102). Thereby, the imaging unit 200 stores the captured image in the storage unit 300.

  Subsequently, the control unit 100 reads the image stored in the storage unit 300 and performs face recognition processing on the user's face included in the image read by the face recognition unit 110. Then, the control unit 100 determines whether or not the user's face included in the image read by the face recognition unit 110 has been recognized (step S104).

  As a result of the determination in step S104, when it is determined that the user's face cannot be recognized (“NO” in step S104), the control unit 100 returns to step S102 and takes the next image on the imaging unit 200. The process from step S103 to step S104 is repeated.

  On the other hand, as a result of the determination in step S104, the user's face can be recognized (at this time, the recognized face recognition information of the user is output from the face recognition unit 110 to the face direction detection unit 120). When it determines ("YES" of step S104), the control part 100 performs the face direction detection process which detects the direction of the user's face recognized by the face direction detection part 120 (step S105). In the face orientation detection process in step S105, for example, the initial state of the user's face stored in the storage unit 300 is also used. As a result, the face orientation detection unit 120 outputs the detected user face orientation information.

  Subsequently, the control unit 100 performs face distance detection processing for detecting the distance between the user's face recognized by the face distance detection unit 130 and the camera (step S106). At this time, the face recognition unit 110 outputs the recognized user face recognition information to the face distance detection unit 130. In the face distance detection process in step S106, for example, the initial state of the user's face stored in the storage unit 300 is also used. Thereby, the face distance detection unit 130 outputs the detected face distance information of the user.

  Subsequently, the control unit 100 sets in advance the face state of the user represented by the face direction information output from the face direction detection unit 120 and the face distance information output from the face distance detection unit 130. It is determined whether or not the operation content is for the operation target device (step S107). Note that the determination process in step S107 is performed by the control unit 100 comparing the detected face state of the user with, for example, operation information stored in the storage unit 300. Then, when the user's face state currently being compared is included in the operation information, the control unit 100 determines that the user's face state is the operation content for the operation target device, and is currently performing the comparison. When the user's face state is not included in the operation information, it is determined that the user's face state is not the operation content for the operation target device.

  As a result of the determination in step S107, when it is determined that the state of the user's face is the operation content for the operation target device (“YES” in step S107), the control unit 100 activates the communication unit 400 in step S108, and the user An operation signal representing an operation corresponding to the face state of the user is transmitted to the operation target device. Then, the control unit 100 proceeds to step S109.

  On the other hand, as a result of the determination in step S107, when it is determined that the state of the user's face is not the operation content for the operation target device (“NO” in step S107), the control unit 100 proceeds to step S109.

  Subsequently, the control unit 100 determines whether or not to continue the operation process for the operation target device (step S109). As a result of the determination in step S109, when it is determined that the operation processing of the operation target device is to be continued (“YES” in step S109), the control unit 100 returns to step S102 and causes the imaging unit 200 to capture the next image. The processes from step S103 to step S108 are repeated. On the other hand, as a result of the determination in step S109, when it is determined that the operation process for the operation target device is not continued ("NO" in step S109), the control unit 100 ends the operation process for the operation target device.

  With such a configuration and processing procedure, the controller device 10 detects a change in the state of the user's face from each image captured by the camera provided in the imaging unit 200, and based on the detected state of the user's face, The operation desired by the user is output to the operation target device.

  In the configuration of the controller device 10 illustrated in FIG. 1, the case where the face recognition unit 110, the face direction detection unit 120, and the face distance detection unit 130 are configured in the control unit 100 has been described. However, each of the face recognition unit 110, the face orientation detection unit 120, and the face distance detection unit 130 is not a component of the control unit 100, but a component of the controller device 10, that is, the face recognition unit 110 and the face The configuration may be such that each of the orientation detection unit 120 and the face distance detection unit 130 is provided in the controller device 10 as a component connected to the control unit 100.

  In the processing procedure of the controller device 10 illustrated in FIG. 2, the case where the controller device 10 has already recognized the initial state of the user's face has been described. However, when the controller device 10 has not already recognized the initial state of the user's face, in the first step S107, the face orientation information output from the face orientation detector 120 and the face distance detector 130 are output. For example, the initial state of the user's face can be stored in the storage unit 300 by setting the user's face state represented by the face distance information as the initial state of the user's face. In addition, when making the operating device 10 recognize the initial state of a face, the structure which a user performs predetermined operation and procedure may be sufficient. For example, a configuration in which the user inputs an instruction for recognizing the initial state of the face to the controller device 10 by voice may be used.

  Next, a process for detecting a change in the user's face in the controller device 10 will be described. First, face recognition processing in the face recognition unit 110 will be described. FIG. 3 is a diagram illustrating face recognition processing (face recognition processing) in the face recognition unit 110 provided in the operation device 10 according to the embodiment of the present invention. FIG. 3 shows an example of only the user's face included in the image taken by the imaging unit 200.

  As shown in FIG. 3, the face recognition unit 110 extracts the entire area of the user's face included in the image captured by the imaging unit 200. Then, the face recognition unit 110 detects each part of the face included in the extracted entire area of the user's face, and uses the feature points representing the position of the detected face part as face recognition information. And output to the face distance detection unit 130. In FIG. 3, the user's “brow”, “eyes” (“pupil”, “eye head”, and “eye corner”), “nose” (“nose apex” and “nose”), “mouth” )), “Mandible”, and “ear” are detected as feature points. Note that the method of extracting the entire face region and detecting the feature points of the facial part in the face recognition unit 110 is a well-known face detection and face method generally used in, for example, digital still cameras and digital video cameras. Detailed description is omitted because it can be performed using a recognition technique.

  Next, the face direction detection process in the face direction detection unit 120 will be described. Based on the user's face recognition information input from the face recognition unit 110, the face orientation detection unit 120 sets the user's face facing the front as an initial state, and the user's face is up and down and left and right from the initial state. A change in the orientation of the user's face is detected based on the inclination rate.

  FIG. 4 is a diagram illustrating an example of face orientation detection processing (face orientation detection processing) in the face orientation detection unit 120 provided in the operation device 10 according to the embodiment of the present invention. FIG. 4 illustrates an example in which the face direction detection unit 120 detects a state in which the user's face is tilted left and right. More specifically, as shown in FIG. 4, the face direction detection unit 120 includes feature points that represent the positions of the left and right “pupils” included in the user's face recognition information input from the face recognition unit 110. By using the feature points representing the positions of the left and right “ears”, the right / left inclination of the user's face is detected by determining the ratio of the user's face being displaced from the center.

  More specifically, as shown in FIG. 4A, the face direction detection unit 120 represents the positions of the left and right “ears” when the user's face is facing the front. A distance (length) between two feature points is calculated, and a straight line (hereinafter referred to as “reference line C”) is formed at a position that is half the length between the two feature points, that is, at the center position. ”). Then, as shown in FIG. 4A, the face direction detection unit 120 determines the distance (length) between the reference line C and the feature point of the “ear” on the left side of the user on the left side (hereinafter, L1 is referred to as “left distance”, and the distance (length) between the reference line C and the “ear” feature point on the right side of the user is referred to as right distance (hereinafter referred to as “right distance”) R1. The face direction detection unit 120 sets the position indicated by the reference line C and the length represented by each of the left distance L1 and the right distance R1 as an initial state representing a state where the user's face is facing the front.

  Thereafter, the face orientation detection unit 120 is based on feature points representing the positions of the left and right “pupils” and the left and right “ears” included in the respective face recognition information sequentially input from the face recognition unit 110. The lengths of the left distance L1 and the right distance R1 are sequentially calculated. Then, as shown in FIG. 4A, the face direction detection unit 120 has a length that is represented by the left distance L1 and a length that is represented by the right distance R1 within the same length or within a predetermined short length difference. In some cases, it is determined that the user's face is facing the front. Further, as shown in FIG. 4B, the face orientation detection unit 120 determines that the user's face is in the right direction when the length represented by the left distance L1 is longer than the length represented by the right distance R1. Is determined to be in a state of facing. Further, as shown in FIG. 4C, the face direction detection unit 120 determines that the user's face is leftward when the length represented by the left distance L1 is shorter than the length represented by the right distance R1. Is determined to be in a state of facing.

  Then, the face orientation detection unit 120 determines, from the relationship between the length represented by the left distance L1 and the length represented by the right distance R1, the angle θ when the user's face is facing right or left, that is, the user's face. Is detected (see FIG. 4B and FIG. 4C). The face orientation detection unit 120 sequentially outputs information representing the detected left and right tilt (direction and angle θ) of the user's face as face orientation information representing the user's face orientation.

  By such a method, the face direction detection unit 120 sequentially detects the deviation of the user's face from the reference line C based on the respective face recognition information sequentially input from the face recognition unit 110, and the detected result is detected as a face. Output sequentially as direction information.

  Note that the method of detecting the orientation of the user's face by the face orientation detection unit 120 is not limited to the method shown in FIG. 4, and various methods using each face recognition information sequentially input from the face recognition unit 110. Can be considered. 5-7 is a figure explaining another example of the face direction detection process (face direction detection process) in the face direction detection part 120 with which the operating device 10 of embodiment of this invention was equipped. Note that the example shown in FIGS. 5 to 7 also illustrates a case where the face orientation detection unit 120 detects a state in which the orientation of the user's face is tilted left and right.

  In the example illustrated in FIG. 5, the face orientation detection unit 120 uses the feature points representing the positions of the left and right “pupils” included in the user face recognition information input from the face recognition unit 110 to detect the face of the user. This is a method of detecting the right / left inclination of the user's face by determining the ratio of deviation from the center.

  More specifically, as shown in FIG. 5A, the face direction detection unit 120 represents the positions of the left and right “pupils” when the user's face is facing the front. The length between the two feature points is calculated, and the reference line C is assumed at the center position of the length between the two calculated feature points, as in the example shown in FIG. Then, as shown in FIG. 5A, the face orientation detection unit 120 sets the center point M1 at the center position of the calculated length between the two feature points. The face direction detection unit 120 sets a state where the position of the center point M1 is on the reference line C as an initial state indicating a state where the user's face is facing the front.

  Thereafter, the face direction detection unit 120 sequentially determines the position of the center point M1 based on the feature points representing the positions of the left and right “pupils” included in the respective face recognition information sequentially input from the face recognition unit 110. calculate. Then, as shown in FIG. 5A, the face orientation detection unit 120 detects the user's face when the position of the center point M1 is on the reference line C or is within a predetermined small angle. Is determined to be facing the front. Further, as shown in FIG. 5B, when the face direction detection unit 120 detects that the position of the center point M1 is shifted from the reference line C to the right by a predetermined angle θ or more, It is determined that the user's face is facing right. Further, as shown in FIG. 5C, the face direction detection unit 120 detects that the position of the center point M1 is shifted from the reference line C to the left by a predetermined angle θ or more. It is determined that the user's face is facing left. Then, the face orientation detection unit 120 sequentially outputs information on the determined user face orientation and the detected angle θ as face orientation information representing the user face orientation.

  As described above, the face direction detection unit 120 also detects the deviation of the center point M1 from the reference line C of the user's face based on each face recognition information sequentially input from the face recognition unit 110. , The tilt of the user's face to the left and right can be detected.

  In the example illustrated in FIG. 6, the face direction detection unit 120 uses a feature point that represents the position of one of the “pupils” included in the user's face recognition information input from the face recognition unit 110. This is a method of detecting the inclination of the user's face to the left and right by determining the ratio of the user's face being off-center. In the example illustrated in FIG. 6, an example in which the left / right inclination of the user's face is detected based on the feature point indicating the position of the left “pupil” is illustrated.

  More specifically, as shown in FIG. 6A, the face direction detection unit 120 is an example shown in FIG. 4A when the user's face is facing the front. Similarly, a reference line C is assumed based on two feature points representing the positions of the left and right “ears”. Then, as shown in FIG. 6A, the face direction detection unit 120 sets the length between the reference line C and the left “pupil” feature point as the left distance L2. The face direction detection unit 120 sets the length represented by the left distance L2 as an initial state representing a state where the user's face is facing the front.

  Thereafter, the face direction detection unit 120 determines the left distance L2 (hereinafter referred to as the initial state) based on the feature point representing the position of the left “pupil” included in each face recognition information sequentially input from the face recognition unit 110. In order to distinguish from the left distance L2, the length of “left distance L21”) is sequentially calculated. Then, as shown in FIG. 6A, the face direction detection unit 120 has the calculated length represented by the left distance L21 equal to the length represented by the initial left distance L2 or a predetermined short length. If it is within the difference in length, it is determined that the user's face is facing the front. Further, as shown in FIG. 6B, the face direction detection unit 120 determines that the user's face is in the right direction when the length represented by the left distance L21 is shorter than the length represented by the left distance L2. Is determined to be in a state of facing. In addition, as shown in FIG. 6C, the face orientation detection unit 120 determines that the user's face is leftward when the length represented by the left distance L21 is longer than the length represented by the left distance L2. Is determined to be in a state of facing.

  Then, the face direction detection unit 120 detects an angle θ (the inclination of the user's face to the left and right) in a state where the user's face is facing rightward or leftward from the length represented by the left distance L21 (FIG. 6 (b) and FIG. 6 (c)). As in the example shown in FIG. 4A, the face orientation detection unit 120 sequentially outputs information on the user's face orientation and angle θ as face orientation information.

  As described above, the face direction detection unit 120 is based on the information of one of the feature points even when the face recognition information sequentially input from the face recognition unit 110 does not include the information of the paired feature points. Thus, it is possible to detect the inclination of the user's face to the left and right by sequentially detecting the deviation of the user's face from the reference line C. In the example illustrated in FIG. 6, the case where the left / right inclination of the user's face is detected based on the feature point indicating the position of the left “pupil” has been described. The tilt of the user's face to the left and right can also be detected based on the feature points representing the position of the “pupil”.

  In the example shown in FIG. 7, for example, when the user wears glasses such as sunglasses, the left and right “pupil” positions are represented in the face recognition information of the user input from the face recognition unit 110. This is an example in which the face direction detection unit 120 detects the left / right inclination of the user's face when no point is included. In the example illustrated in FIG. 7, the face direction detection unit 120 determines the ratio of the user's face being deviated from the center using the feature points representing the positions of the left and right “mouth corners” included in the face recognition information. Detect the left / right tilt of the user's face.

  More specifically, as shown in FIG. 7A, the face orientation detection unit 120 represents the positions of the left and right “mouth corners” when the user's face is facing the front. By calculating the length between the two feature points, a reference line C is assumed at the center position of the length between the two calculated feature points, as in the example shown in FIG. Then, as shown in FIG. 7A, the face direction detection unit 120 is centered at the center position of the length between the calculated two feature points, as in the example shown in FIG. Set point M2. The face direction detection unit 120 sets a state where the position of the center point M2 is on the reference line C as an initial state indicating a state where the user's face is facing the front.

  After that, the face orientation detection unit 120 sequentially determines the position of the center point M2 based on the feature points representing the positions of the left and right “mouth corners” included in the respective face recognition information sequentially input from the face recognition unit 110. calculate. Then, as shown in FIG. 7A, the face direction detection unit 120 detects the user's face when the position of the center point M2 is on the reference line C or is within a predetermined small angle. Is determined to be facing the front. Further, as shown in FIG. 7B, when the face direction detection unit 120 detects that the position of the center point M2 is shifted from the reference line C to the right by a predetermined angle θ or more, It is determined that the user's face is facing right. Further, as shown in FIG. 7C, the face direction detection unit 120 detects that the position of the center point M2 is shifted from the reference line C to the left by a predetermined angle θ or more. It is determined that the user's face is facing left. Then, as in the example illustrated in FIG. 5, the face orientation detection unit 120 sequentially outputs information on the determined user face orientation and the detected angle θ as face orientation information representing the user face orientation. To do.

  As described above, even when the face recognition information sequentially input from the face recognition unit 110 does not include any feature point information, the face direction detection unit 120 is not included in the other face recognition information. By sequentially detecting the shift of the center point M2 from the reference line C of the user's face based on the feature point information, the tilt of the user's face to the left and right can also be detected. In the example shown in FIG. 7, the case where the left / right inclination of the user's face is detected using the feature points representing the positions of the left and right “mouth corners” has been described. For example, the glasses worn by the user If the face recognition information input from the face recognition unit 110 includes feature point information representing the position and shape of the lens edge, the feature points of the glasses are used to move the user's face to the left and right. Tilt can also be detected. For example, the face orientation detection unit 120 calculates the center point between two lenses from the feature point of the shape of the glasses, and the user's face is shifted from the center based on the positional relationship between the reference line and the calculated center point. It is also possible to detect the inclination of the user's face to the left and right by determining the ratio.

  As described above, the face orientation detection unit 120 detects a change in the orientation of the user's face based on the feature point information included in the face recognition information input from the face recognition unit 110. Note that some examples of the face direction detection processing in the face direction detection unit 120 have been described, but the method of detecting the face direction of the user by the face direction detection unit 120 is limited to the method illustrated in FIGS. Is not to be done. For example, even when the face orientation detection unit 120 uses information on other feature points included in the face recognition information input from the face recognition unit 110, the face orientation detection process can be performed by considering the same. In addition, for example, the face orientation detection unit 120 uses the feature points of the face area included in the face recognition information input from the face recognition unit 110, and the size and face of the user's face area included in the captured image. The face orientation detection process can also be performed by determining the distribution of the area. Further, for example, the face orientation detection process may be performed using an existing technique different from the example of the method illustrated in FIGS.

  4 to 7 describe the case of detecting the state in which the user's face is tilted to the left and right, but even when detecting the state in which the user's face is tilted up and down. The same can be considered by using appropriate feature points in the face orientation detection processing by the face orientation detector 120. For example, assuming a horizontal straight line connecting feature points representing the positions of the left and right “pupils” as a reference line, the distance (length) between the reference line and the feature point representing the position of the “nose apex” Based on the above, it is possible to detect the vertical inclination of the user's face. Therefore, description of an example of a method for detecting the upward / downward tilt of the user's face direction by the face direction detection unit 120 is omitted.

  Next, the face distance detection process in the face distance detection unit 130 will be described. Based on the user's face recognition information input from the face recognition unit 110, the face distance detection unit 130 sets the current position where the user's face is facing the front as an initial state, and the user's face before and after the initial state ( A change in the distance of the user's face from the camera is detected based on the rate of movement in the front and back directions.

  FIG. 8 is a diagram illustrating an example of face distance detection processing (face distance detection processing) in the face distance detection unit 130 included in the operation device 10 according to the embodiment of the present invention. FIG. 8 illustrates an example in which the face distance detection unit 130 detects a state in which the distance between the user's face and the camera changes in a direction away from the camera. More specifically, as shown in FIG. 8, the face distance detection unit 130 uses a facial region feature point included in the user face recognition information input from the face recognition unit 110 to capture a captured image. The distance between the position of the camera and the position of the user's face is detected by determining the size of the user's face included in.

  More specifically, as shown in FIG. 8A, the face distance detection unit 130 is in a state where the entire face of the user is included within the angle of view of the image captured by the imaging unit 200. The number of pixels (number of pixels) representing the size of the face is calculated from the feature points of the face area. The face distance detection unit 130 sets the number of pixels representing the size of the face calculated here as an initial state (initial value). The face distance detection unit 130 uses, for example, a well-known face detection technique that is generally used based on the number of pixels in the face area as an initial value, for example, the actual distance between the camera and the user's face. The absolute value of the distance, such as the distance, may be estimated.

  After that, the face distance detection unit 130 is included within the angle of view of the image captured this time by the imaging unit 200 based on the feature points of the face area included in the face recognition information sequentially input from the face recognition unit 110. The number of pixels representing the size of the user's face is sequentially calculated. Then, the face distance detection unit 130 determines a change in the distance between the user's face position and the camera position based on the difference between the initial number of pixels and the currently calculated number of pixels. For example, as shown in FIG. 8B, when the number of pixels of the user's face size calculated this time is smaller than the number of pixels of the face size set as the initial value, the face distance detection unit 130 determines that the distance between the user's face and the camera has changed in a direction away from the camera. When the face distance detection unit 130 estimates the absolute value of the distance, the amount of change in the distance between the camera and the user's face is calculated based on the difference between the initial number of pixels and the number of pixels calculated this time. It may be estimated as a relative distance. The face distance detection unit 130 sequentially outputs information indicating the distance between the detected camera position and the user's face position as face distance information indicating the distance of the user's face.

  By such a method, the face distance detection unit 130 sequentially detects a change in the distance between the camera position and the user's face position based on the face recognition information sequentially input from the face recognition unit 110, and the detection result Are sequentially output as face distance information.

  Note that the method of detecting a change in the distance of the user's face from the camera by the face distance detection unit 130 is not limited to the method shown in FIG. 8, and each face recognition sequentially input from the face recognition unit 110. Various methods using information can be considered. 9 to 11 illustrate another example of detection processing (face distance detection processing) of a change in distance from the camera of the user's face in the face distance detection unit 130 provided in the operation device 10 according to the embodiment of the present invention. It is a figure to do. 9 to 11 also show a case where the face distance detection unit 130 detects a state in which the distance between the user's face and the camera changes in the direction away from the camera. 9 to 11 also show the angle of view of the image captured by the imaging unit 200. FIG.

  In the example illustrated in FIG. 9, the face distance detection unit 130 uses the feature points representing the positions of the left and right “pupils” included in the user's face recognition information input from the face recognition unit 110, and uses the left and right “pupils”. This is a method for detecting a change in the distance between the user's face and the camera.

  More specifically, as shown in FIG. 9A, the face distance detection unit 130 represents the positions of the left and right “pupils” when the user's face is facing the front. The distance (length) between the two feature points is calculated, and the calculated distance (length) between the two feature points is defined as a distance D1. Then, the face distance detection unit 130 sets the distance D1 as an initial value. The face distance detection unit 130 may estimate the absolute value of the distance between the camera and the user's face based on the initial distance D1.

  Thereafter, the face distance detection unit 130 determines between the left and right “pupils” based on the feature points representing the positions of the left and right “pupils” included in the respective face recognition information sequentially input from the face recognition unit 110. Distance D1 (hereinafter, referred to as “distance D11” to be distinguished from the initial distance D1). Then, the face distance detection unit 130 determines a change in the distance between the user's face position and the camera position based on the difference between the initial distance D1 and the currently calculated distance D11. For example, as shown in FIG. 9B, when the user's face distance D11 calculated this time is shorter than the user's face distance D1 as the initial value, the face distance detection unit 130 It is determined that the distance between the face of the camera and the camera changes in a direction away from the camera. Then, the face distance detection unit 130 sequentially outputs information representing the distance between the determined camera position and the user's face position as face distance information representing the user's face distance.

  In this way, the face distance detection unit 130 is based on the feature points representing the positions of the left and right “pupils” included in the respective face recognition information sequentially input from the face recognition unit 110. The distance between the camera and the user's face can also be detected by calculating the distance (length).

  Further, in the example shown in FIG. 10, the face distance detection unit 130 positions the “eye” and “eye corner” of one of the “eyes” included in the user's face recognition information input from the face recognition unit 110. This is a method of determining the size of the “eye” using the feature point representing the change of the distance between the user's face and the camera. FIG. 10 shows an example in which a change in the distance of the user's face from the camera is detected based on the left “eye” feature point.

  More specifically, as shown in FIG. 10A, the face distance detection unit 130 is the same as the example shown in FIG. 9 when the user's face is facing the front. The distance (length) between the two feature points representing the positions of the “eye” and “eye corner” of the left “eye”, that is, the distance D2 representing the size of the “eye” on the left side is calculated and initialized. Value. The face distance detection unit 130 may estimate the absolute value of the distance between the camera and the user's face based on the distance D2 as an initial value, as in the example shown in FIG.

  After that, the face distance detection unit 130 determines the left side based on the feature points representing the positions of the “eye” and “eye corner” of the left “eye” included in each face recognition information sequentially input from the face recognition unit 110. A distance D2 representing the size of the “eyes” (hereinafter, referred to as “distance D21” to be distinguished from the distance D2 as an initial value) is sequentially calculated. Then, the face distance detection unit 130 determines a change in the distance between the user's face position and the camera position based on the difference between the initial distance D2 and the currently calculated distance D21. For example, as shown in FIG. 10B, the distance D21 representing the size of the “eye” on the left side of the user calculated this time is the distance D2 representing the size of the “eye” on the left side of the user as an initial value. If the distance is shorter, the face distance detection unit 130 determines that the distance between the user's face and the camera is changing in a direction away from the camera. Similarly to the example shown in FIG. 9, the face distance detection unit 130 sequentially outputs information representing the distance between the determined camera position and the user's face position as face distance information.

  As described above, the face distance detection unit 130 adds two “pupils” that are paired to calculate the distance (length) between the left and right “pupils” in the face recognition information sequentially input from the face recognition unit 110. The distance between the camera and the user's face can be detected based on the feature point information indicating the size of one of the eyes even when the feature point information indicating the position of it can. In the example illustrated in FIG. 10, the case where the distance between the camera and the user's face is detected based on the feature point for calculating the size of the “eye” on the left side has been described. The distance between the camera and the user's face can also be detected based on the feature point for calculating the size of the right “eye”.

  Further, in the example shown in FIG. 11, for example, when the user wears glasses such as sunglasses, the distance between the left and right “pupils” is added to the user's face recognition information input from the face recognition unit 110. Feature points that represent the positions of the left and right "pupils" for calculating (length), and feature points that represent the positions of the "eyes" and "eyes" for calculating the size of one of the "eyes" This shows a method in which the face distance detection unit 130 detects a change in the distance of the user's face from the camera when no is included. In FIG. 11, when the face distance detection unit 130 detects a change in the distance of the user's face from the camera using the feature point indicating the position of the “end” of the eyeglass lens included in the face recognition information. An example is shown.

  More specifically, as shown in FIG. 11A, the face distance detection unit 130 has the “end portions” of the left and right lenses of the glasses when the user's face is facing the front. The distance (length) between the two feature points representing the position “is calculated and set as an initial value. In FIG. 11A, there is a feature point that represents the position of the left “end” of the lens on the left side of the glasses and the feature point that represents the position of the “end” on the right side of the right lens of the glasses. An example in which the distance D3 is calculated and set as an initial value is shown. The face distance detection unit 130 may estimate the absolute value of the distance between the camera and the user's face based on the initial distance D3 as in the example illustrated in FIG.

  After that, the face distance detection unit 130 includes a feature point indicating the position of the “edge” on the left side of the left lens of the glasses included in each face recognition information sequentially input from the face recognition unit 110, and the right side of the glasses. A distance D3 between the two feature points (hereinafter, referred to as “distance D31” in order to distinguish from the initial distance D3) based on the feature point representing the position of the “end” on the right side of the lens. To do. Then, the face distance detection unit 130 determines a change in the distance between the user's face position and the camera position based on the difference between the initial distance D3 and the currently calculated distance D31. For example, as shown in FIG. 11B, the distance D31 that represents the distance between the ends of the glasses worn by the user calculated this time is shorter than the distance D3 that represents the distance between the ends of the glasses as the initial value. The face distance detection unit 130 determines that the distance between the user's face and the camera has changed in a direction away from the camera. Similarly to the example shown in FIG. 9, the face distance detection unit 130 sequentially outputs information representing the distance between the determined camera position and the user's face position as face distance information.

  As described above, the face distance detection unit 130 includes the face recognition information sequentially input from the face recognition unit 110 even when the feature point information of the part of the user's face is not included. The distance between the camera and the user's face can be detected based on the information of other feature points. In the example shown in FIG. 11, the feature point indicating the position of the “end” on the left side of the lens on the left side of the glasses worn by the user, and the position of the “end” on the right side of the lens on the right side of the glasses However, for example, the distance between two feature points representing the position of the “end” of one of the lenses of the glasses worn by the user is described. It can also be calculated. For example, as shown in FIG. 11 (a), the camera and the user can be similarly calculated by calculating the distance D4 between two feature points representing the positions of the left and right “ends” of the left lens of the glasses. It is possible to detect the distance from the face. Further, for example, the distance between the camera and the user's face can also be detected by calculating the distance between two feature points representing the positions of the left and right “mouth corners”.

  In this manner, the face distance detection unit 130 detects a change in the distance between the camera position and the user face position based on the feature point information included in the face recognition information input from the face recognition unit 110. . Although some examples of the face distance detection processing in the face distance detection unit 130 have been described, the method of detecting the distance between the camera and the user's face by the face distance detection unit 130 is shown in FIGS. The method is not limited. For example, even when the face distance detection unit 130 uses information on other feature points included in the face recognition information input from the face recognition unit 110, the face distance detection process can be performed by considering the same. Further, for example, the face distance detection process may be performed using an existing technique different from the example of the method illustrated in FIGS.

  8 to 11, the case has been described in which the state in which the distance between the user's face and the camera changes is detected. However, the distance between the user's face and the camera changes in a direction closer to the camera. Even in the case of detecting a state in which the user is performing, it can be considered in the same way by using an appropriate feature point in the face distance detection processing by the face distance detection unit 130. Therefore, description of an example of a method of detecting a state in which the distance between the user's face and the camera is changing in the direction in which the face distance detection unit 130 approaches is omitted.

<Specific example>
Next, a specific example in which the operation target device is operated by the operation device 10 will be described. In the following description, an example will be described in which the operation device 10 is mounted on a vehicle and a user of the operation device 10 operates the operation target device mounted on the vehicle when the vehicle is driven. Here, the operation target devices mounted on the vehicle are, for example, an air conditioner such as an air conditioner and an audio device such as a radio, and change of the set temperature of the air conditioner, change of the radio channel, and change of the volume of the radio. A case where the operation is performed by the operation device 10 will be described. In the following description, it is assumed that the controller device 10 has already recognized the initial state of the user's face.

  FIG. 12 is a diagram schematically illustrating an example of the distance between the operation device 10 according to the embodiment of the present invention and the detected (user's) face. Note that FIG. 12 shows the distance between the position of the camera 201 provided in the imaging unit 200 in the controller device 10 and the position of the user's face. In the following description, the value of the distance between the position of the camera 201 and the position of the user's face is a distance D1 (for example, 40 [cm]) as shown in FIG. Let the state which is facing the front be an initial state. When the distance value between the position of the camera 201 and the position of the user's face is a distance D11 (for example, 35 [cm]) as shown in FIG. It is assumed that an operation for changing the set temperature of the air conditioner is performed according to the change. Further, when the distance between the position of the camera 201 and the position of the user's face is a distance D12 (for example, 30 [cm]) as shown in FIG. An operation to change the radio channel according to the change is performed. Further, when the distance between the position of the camera 201 and the position of the user's face is a distance D13 (for example, 25 [cm]) as shown in FIG. It is assumed that an operation for changing the volume of the radio according to the change is performed.

  FIG. 13 is a diagram illustrating an example of contents (operation information) to be operated according to the (user's) face state detected by the operation device 10 according to the embodiment of the present invention. The operation information illustrated in FIG. 13 is stored in the storage unit 300, for example.

  In FIG. 13A, the value of the distance between the camera 201 and the user's face detected by the operation device 10 and the orientation of the user's face are set as variables (parameters), and the combination of the respective parameters is mounted on the vehicle. 3 shows an example of operation information when operating the designated operation target device. More specifically, the face distance detection unit 130 outputs face distance information indicating that the position of the camera 201 and the position of the user's face is the distance D11 (see FIG. 12B), and detects the face orientation. When the unit 120 outputs face direction information indicating that the user's face direction is the left direction, the controller device 10 sets the set temperature to 1 [° C.] based on the operation information illustrated in FIG. An operation signal representing the operation to be lowered is transmitted to the air conditioner via the communication unit 400. Similarly, the face distance detection unit 130 outputs face distance information indicating that the position of the camera 201 and the position of the user's face is the distance D11, and the face direction detection unit 120 determines that the user's face direction is right. When the face orientation information indicating the direction is output, the controller device 10 transmits an operation signal indicating an operation for raising the set temperature by 1 [° C.] to the air conditioner based on the operation information illustrated in FIG. . For example, the face distance detection unit 130 outputs face distance information indicating that the position of the camera 201 and the position of the user's face is a distance D12 (see FIG. 12C), and the face direction detection unit 120 However, when the face orientation information indicating that the face orientation of the user is the left direction is output, the controller device 10 sets the radio channel to the previous one based on the operation information shown in FIG. An operation signal representing an operation to be performed is transmitted to the radio via the communication unit 400. For example, the face distance detection unit 130 outputs face distance information indicating that the position of the camera 201 and the position of the user's face is a distance D13 (see FIG. 12D), and the face direction detection unit 120 However, when face orientation information indicating that the user's face orientation is rightward is output, the controller device 10 performs an operation to increase the radio volume by one based on the operation information shown in FIG. An operation signal to be represented is transmitted to the radio via the communication unit 400.

  In FIG. 13B, each of the combinations of the value of the distance between the camera 201 and the user's face detected by the controller 10 and the value of the user's face direction and angle θ is set as a parameter. An example of operation information in the case of operating an operation target device mounted on a vehicle with a combination of parameters is shown. In FIG. 13B, the face distance detection unit 130 outputs face distance information indicating that the position of the camera 201 and the position of the user's face are the distance D13 (see FIG. 12D). An example of operation information only in the case is shown. FIG. 13B shows an example of an operation for changing the volume of the radio performed when the position of the camera 201 and the position of the user's face in FIG. 13A are the distance D13. More specifically, when the face direction detection unit 120 outputs face direction information indicating that the user's face direction is the left direction and the value of the angle θ is 20 degrees, the controller device 10 Based on the operation information shown in 13 (b), an operation signal representing an operation of lowering the volume of the radio by two is transmitted to the radio via the communication unit 400. Similarly, when the face orientation detection unit 120 outputs face orientation information indicating that the orientation of the user's face is the left direction and the value of the angle θ is 10 degrees, the controller device 10 displays FIG. ), An operation signal representing an operation of lowering the volume of the radio by one is transmitted to the radio. Further, for example, when the face orientation detection unit 120 outputs face orientation information indicating that the orientation of the user's face is the right direction and the value of the angle θ is 20 degrees, the controller device 10 displays FIG. Based on the operation information shown in b), an operation signal representing an operation to increase the volume of the radio by two is transmitted to the radio via the communication unit 400.

  In addition, in the example of the operation of the operation target device by the operation device 10 described above, a case where each operation target device is operated using the horizontal direction of the user's face as a parameter is illustrated. However, when the vertical direction of the user's face (the value of angle θ may be adjusted) is used as a parameter, or the combination of the horizontal direction and the vertical direction (the value of angle θ may be adjusted) is used as a parameter However, operation information can be considered similarly.

  Further, in an example of the operation of the operation target device by the operation device 10 described above, the value of the user's face angle θ detected by the face direction detection unit 120, the position of the camera 201 detected by the face distance detection unit 130, and the user's The case where the value of the distance from the face position is one value has been described. However, the user cannot always make the face position the same. Therefore, the controller device 10 has a predetermined range for the value of the angle θ of the user's face and the value of the distance between the position of the camera 201 and the position of the user's face, and values in this range are set to the same value. You may make it the structure processed as that.

  In addition, the controller device 10 may notify the user of information indicating which operation target device the detected position of the user's face can perform, for example, by voice. For example, in the example shown in FIG. 13A, when the orientation of the user's face is the front (initial state), the distance between the position of the camera 201 and the position of the user's face is the distance D1 (initial state). When the absence is detected, the name of the operation target device may be notified by voice according to the detected distance value. In addition, the operation target device to which the operation signal is transmitted from the operation device 10 changes the setting according to the transmitted operation signal, and notifies the user of information indicating the content of the changed setting, for example, by voice. May be.

  As described above, according to the embodiment for carrying out the present invention, the state of the user's face is obtained from each image captured by the camera 201 included in the imaging unit 200 in the controller device 10 at a short predetermined time interval. And detect changes. More specifically, the face distance detection unit 130 detects the distance between the position of the camera 201 and the position of the user's face, and the face direction detection unit 120 detects the direction of the user's face and the angle θ. And according to the form for implementing this invention, the operation signal according to the detected state of the user's face is transmitted to the operation target device. Thus, even in a situation where the user cannot directly operate the operation target device, the user can perform a desired operation on the device by changing the position and orientation of the face without removing the line of sight.

  In the embodiment for carrying out the present invention, it has been described that the camera 201 can shoot a state where the user's face is facing the front. However, for example, depending on the position of the camera 201 installed in the vehicle, the camera 201 cannot capture the user's face from the front because the actual positional relationship between the camera 201 and the user's face, and the user's face is the front It is also conceivable that the initial state cannot be obtained by photographing the state facing the camera. In this case, the controller device 10 corrects the state in which the user's face, which is the initial state, is facing the front according to the position where the camera 201 is installed.

  More specifically, as shown in FIG. 14A, when the camera 201 installed in the vehicle is installed at a position where the user's face can be photographed from the front, the user's face The state in which is facing the front can be the initial state. However, as shown in FIG. 14 (b), the camera 201 is not installed at a position where the driver's seat of the vehicle is photographed from the front, and the camera 201 cannot be photographed from the front. In the case where the user's face is facing the front, the initial state cannot be set. Therefore, the face direction detection unit 120 and the face distance detection unit 130 may be slightly left when the actual positional relationship between the camera 201 and the user's face is as shown in FIG. The angle of the state tilted in the direction is corrected to be the angle of the face in the initial state where the user's face is facing the front. Then, the face direction detection unit 120 and the face distance detection unit 130 perform the face direction detection process and the face distance detection process by the above-described method, for example, with the corrected initial state as a reference. Accordingly, the controller device 10 detects a change in the orientation of the user's face and a change in the distance between the camera position and the user's face even when the camera 201 cannot capture the user's face from the front. Thus, an operation signal corresponding to the detected face state of the user can be transmitted to the operation target device. The correction of the orientation of the user's face can be performed using a commonly used technique of face detection and face recognition, and thus detailed description thereof is omitted.

  Further, in the embodiment for carrying out the present invention, a case has been described in which the operation target device to which the operation device 10 transmits an operation signal is an air conditioner such as an air conditioner mounted on a vehicle and an audio device such as a radio. . However, the operation target device to which the operation device 10 transmits the operation signal is not limited to the device shown in the embodiment for carrying out the present invention. For example, it may be a travel guidance device such as a navigation device mounted on a vehicle, or an audio device other than a radio. Further, the operation target device to which the operation device 10 transmits the operation signal is not limited to the device mounted on the vehicle. For example, even if the device is installed in a home such as a television set, The idea of the invention can be applied.

  For example, the control unit 100 illustrated in FIG. 1 and the components provided in the control unit 100 (for example, the face recognition unit 110, the face orientation detection unit 120, and the face distance detection unit 130) are implemented. The program is recorded on a computer-readable recording medium, the program recorded on the recording medium is read into a computer system, and executed, thereby performing the above-described various processes related to the operating device 10 of the present embodiment. May be. Here, the “computer system” may include an OS and hardware such as peripheral devices. Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

  Further, the “computer-readable recording medium” means a volatile memory (for example, DRAM (Dynamic) in a computer system which becomes a server or a client when a program is transmitted through a network such as the Internet or a communication line such as a telephone line. Random Access Memory)) that holds a program for a certain period of time is also included. The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what implement | achieves the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  In addition, a part of the operation device 10 according to the present embodiment (for example, the configuration of the control unit 100 illustrated in FIG. Elements) or all of them may be realized as an integrated circuit such as an LSI (Large Scale Integration). In addition, each component in the operating device 10 of the present embodiment may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to the advancement of semiconductor technology, an integrated circuit based on the technology may be used.

  The embodiment of the present invention has been described above with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes various modifications within the scope of the present invention. It is.

In addition, this invention can be implemented also in the following aspect.
An operating device according to a first aspect is an operating device that controls an operation on a device to be operated, an imaging unit that captures an image including a face, and the face included in the image is recognized and recognized. A face recognition unit that outputs information of feature points representing the position of the face part as face recognition information; and a face direction that detects the face direction based on the face recognition information and represents the detected face direction. A face direction detection unit that outputs information, a face distance detection unit that detects a distance between the imaging unit and the face based on the face recognition information, and outputs face distance information representing the detected distance; With reference to the operation information representing the operation content for the device predetermined according to the orientation of the face and the distance between the imaging unit and the face, based on the face orientation information and the face distance information, An operation signal indicating an operation on the corresponding device And a control unit for outputting to the device.

  Further, according to the second aspect, in the operating device according to the first aspect, the face direction detection unit is configured to determine a direction and an angle when the face direction changes from an initial state of a predetermined face direction. The face distance information is output as the face direction information, and the face distance detection unit outputs the distance information when the distance between the imaging unit and the face changes from an initial state of a predetermined distance. The information is output as information, and the control unit includes the face direction value represented by a combination of the face direction and angle included in the face direction information, and the imaging unit and the face included in the face distance information. The operation signal that represents a predetermined operation amount according to a combination with the value of the face distance represented by the distance between and the face may be output to the corresponding device.

  Further, according to a third aspect, in the operation device according to the second aspect, the operation information includes a predetermined device to be operated for each value of the face distance, and the control unit May output the operation signal representing the operation amount corresponding to the face orientation value to the device corresponding to the face distance value included in the face distance information.

  According to a fourth aspect, in the operation device according to the second aspect or the third aspect, the operation information is the same device for each value of the face angle included in the face orientation information. The operation amount is determined in advance, and the control unit represents the operation amount corresponding to the face angle value in the device corresponding to the face distance value included in the face distance information. The operation signal may be output.

  According to the fifth aspect, in the operating device according to any one of the second aspect to the fourth aspect, the face recognition information includes information on a feature point representing at least one eye position. The face direction detection unit detects the face direction based on a feature point representing the eye position, and the face distance detection unit detects the imaging based on the feature point representing the eye position. You may detect the distance between a part and the said face.

  According to the sixth aspect, in the operation device according to any one of the second aspect to the fifth aspect, the face recognition information does not include feature point information indicating an eye position, and glasses The face direction detection unit detects the face direction based on the feature point indicating the position of the glasses, and the face distance detection unit includes the position of the glasses. A distance between the imaging unit and the face may be detected based on a feature point representing.

  According to a seventh aspect, in the operating device according to any one of the second aspect to the sixth aspect, the face recognition information includes information on feature points representing the face area, The face direction detecting unit detects the face direction based on a feature point representing the face area, and the face distance detecting unit is based on the feature point representing the face area. You may detect the distance between the said faces.

  According to an eighth aspect, in the operating device according to any one of the second aspect to the seventh aspect, the face direction detection unit and the face distance detection unit are configured such that the imaging unit is the face. The direction of the face is corrected according to the angle at the time of shooting, the face direction detection unit detects the direction of the face based on the corrected direction of the face, the face distance detection unit, A distance between the imaging unit and the face may be detected based on the corrected orientation of the face.

  According to a ninth aspect, in the operating device according to any one of the second aspect to the eighth aspect, the communication device further includes a communication unit that communicates with the device, The operation signal may be output to the target device via a communication unit.

  An operation method according to a tenth aspect is an operation method in an operation device that controls an operation in a device to be operated, and recognizes the face included in an image including a face photographed by an imaging unit, and performs the recognition A face recognition step for outputting, as face recognition information, feature point information representing the position of the face part, and detecting the face direction based on the face recognition information and indicating the detected face direction A face orientation detecting step for outputting orientation information; a face distance detecting step for detecting a distance between the imaging unit and the face based on the face recognition information; and outputting face distance information representing the detected distance; , Referring to operation information representing operation details for the device determined in advance according to the orientation of the face and the distance between the imaging unit and the face, and based on the face orientation information and the face distance information The corresponding machine An operation signal representing the operation to, and a control step of outputting to the device.

  The program of the eleventh aspect is a program that is executed by a computer of an operating device that controls an operation on a device to be operated, and recognizes the face included in an image including a face photographed by the imaging unit, A face recognition step of outputting information of feature points representing the position of the recognized part of the face as face recognition information; detecting the face direction based on the face recognition information; and detecting the detected face direction. A face direction detecting step for outputting face direction information to be expressed, and a face distance detection for detecting a distance between the imaging unit and the face based on the face recognition information and outputting face distance information representing the detected distance The operation information representing the operation content for the device determined in advance according to the step, the orientation of the face and the distance between the imaging unit and the face is referred to, and the face orientation information and the face distance information are Base There, the operation signal representing the operation to the corresponding said device to execute a control step of outputting to the device, to the computer.

  The recording medium according to the twelfth aspect is a computer-readable recording medium that records a program to be executed by a computer of an operating device that controls an operation on a device to be operated, and includes a face taken by an imaging unit. A face recognition step of recognizing the face included in the image and outputting information of feature points representing the position of the recognized face portion as face recognition information; and detecting the face orientation based on the face recognition information A face direction detecting step for outputting face direction information representing the detected face direction; detecting a distance between the imaging unit and the face based on the face recognition information; and detecting the detected distance. A face distance detection step for outputting the face distance information to be expressed, and operation information representing the operation content for the device predetermined according to the orientation of the face and the distance between the imaging unit and the face. And a computer-readable recording of a program for causing a computer to execute a control step of outputting, to the device, an operation signal representing an operation for the corresponding device based on the face orientation information and the face distance information. It is a possible recording medium.

DESCRIPTION OF SYMBOLS 10 ... Operating device 100 ... Control part 110 ... Face recognition part 120 ... Face direction detection part 130 ... Face distance detection part 200 ... Imaging part 201 ... Camera (imaging part)
300 ... Storage unit 400 ... Communication unit C ... Reference lines L1, L2, L21 ... Left distance R1 ... Right distance θ ... Angles M1, M2 ... Center points D1, D11 , D2, D21, D3, D31, D4 ... distance

Claims (5)

An operation device that controls an operation on an operation target device,
An imaging unit that captures an image including a face;
A face recognition unit that recognizes the face included in the image and outputs information of feature points representing the position of the recognized face part as face recognition information;
A face orientation detection unit that detects the orientation of the face based on the face recognition information and outputs face orientation information representing the detected face orientation;
A face distance detection unit that detects a distance between the imaging unit and the face based on the face recognition information, and outputs face distance information representing the detected distance;
With reference to the operation information representing the operation content for the device predetermined according to the orientation of the face and the distance between the imaging unit and the face, based on the face orientation information and the face distance information, A control unit that outputs an operation signal representing an operation to the corresponding device to the device;
An operating device comprising: The face orientation detection unit
Outputs the direction and angle information as the face direction information when the face direction changes from the initial state of the predetermined face direction,
The face distance detector
Outputs the distance information when the distance between the imaging unit and the face changes from the initial state of a predetermined distance as the face distance information,
The controller is
The face direction value represented by a combination of the face direction and angle included in the face direction information, and the distance between the imaging unit and the face included in the face distance information. Outputting the operation signal representing a predetermined operation amount according to a combination with a face distance value to the corresponding device;
The operating device according to claim 1. The operation information is
For each value of the face distance, the device to be operated is determined in advance,
The controller is
Outputting the operation signal representing the operation amount corresponding to the face orientation value to the device corresponding to the face distance value included in the face distance information;
The operating device according to claim 2. The operation information is
The operation amount for the same device is predetermined for each value of the face angle included in the face orientation information,
The controller is
Outputting the operation signal representing the operation amount corresponding to the face angle value to the device corresponding to the face distance value included in the face distance information;
The operating device according to claim 2 or 3. An operation method in an operating device that controls an operation in an operation target device,
A face recognition step of recognizing the face included in the image including the face photographed by the imaging unit and outputting information of a feature point representing the position of the recognized face part as face recognition information;
Detecting a face orientation based on the face recognition information, and outputting face orientation information representing the detected face orientation; and
A face distance detection step of detecting a distance between the imaging unit and the face based on the face recognition information and outputting face distance information representing the detected distance;
With reference to the operation information representing the operation content for the device predetermined according to the orientation of the face and the distance between the imaging unit and the face, based on the face orientation information and the face distance information, A control step of outputting an operation signal representing an operation on the corresponding device to the device;
Operation method including.

Images