JP2020144551A - Control device and control method - Google Patents

Abstract

To improve recognition accuracy of a gesture operation.SOLUTION: A control device according to an embodiment comprises a detection unit, a setting unit, and a recognition unit. The detection unit detects a specific portion of a user from a captured image. The setting unit sets a recognition area on the basis of the specific portion detected by the detection unit. The recognition unit recognizes a gesture operation in the recognition area set by the setting unit.SELECTED DRAWING: Figure 2

Description

The present invention relates to a control device and a control method.

Conventionally, there is a control device that accepts a gesture operation for an in-vehicle device in a vehicle. In such a control device, it is possible to prohibit an unintended occupant's gesture operation by accepting only a gesture operation from a pre-registered occupant (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2012-121386

However, in the prior art, there is room for improvement in improving the recognition accuracy of the gesture operation.

The present invention has been made in view of the above, and an object of the present invention is to provide a control device and a control method capable of improving the recognition accuracy of a gesture operation.

In order to solve the above-mentioned problems and achieve the object, the control device according to the present invention includes a detection unit, a setting unit, and a recognition unit. The detection unit detects a specific part of the user from the captured image. The setting unit sets a recognition area based on the specific portion detected by the detection unit. The recognition unit recognizes a gesture operation in the recognition area set by the setting unit.

According to the present invention, the recognition accuracy of the gesture operation can be improved.

FIG. 1A is a diagram showing a mounting example of a control device. FIG. 1B is a diagram showing an outline of a control method. FIG. 2 is a block diagram of the control device. FIG. 3 is a diagram showing a specific example of command information. FIG. 4 is a diagram showing a specific example of arbitration information. FIG. 5 is a flowchart showing a processing procedure executed by the control device.

Hereinafter, embodiments of the control device and control method disclosed in the present application will be described in detail with reference to the accompanying drawings. The present invention is not limited to the embodiments shown below.

First, the outline of the control device and the control method according to the embodiment will be described with reference to FIGS. 1A and 1B. FIG. 1A is a diagram showing a mounting example of a control device. FIG. 1B is a diagram showing an outline of a control method.

As shown in FIG. 1A, the control device 1 according to the embodiment is mounted on the vehicle C and functions as a user interface of various in-vehicle devices 50. Specifically, the control device 1 recognizes the occupant's gesture operation from the captured image captured by the camera 10, and outputs a control command corresponding to the recognized gesture operation to the in-vehicle device 50.

By the way, in a control device that recognizes a gesture operation, it is common to recognize a preset gesture operation and output a control command associated with the recognized gesture operation. However, the control device may recognize an operation not intended by the occupant as a gesture operation. Further, in the control device, if the entire captured image is the recognition target of the gesture operation, the processing load when recognizing the gesture operation increases, which is not preferable.

Therefore, in the control method according to the embodiment, it is decided to recognize a specific part of the occupant and set a recognition area to be recognized for the gesture operation based on the recognized specific part. Specifically, in the control method according to the embodiment, as shown in FIG. 1B, a specific part of the user is detected from the captured image I (step S1).

In the example shown in FIG. 1B, a case where the specific part is each part (ears, eyes, mouth, etc.) of the user's face is shown. The specific part is not particularly limited, but for example, when recognizing a gesture operation using a hand, it is preferable that the part is a part other than the hand.

Subsequently, in the control method according to the embodiment, the recognition area Ra is set based on the detected specific portion (part) (step S2).

As shown in FIG. 1B, the recognition area Ra is set in the vicinity of each part detected in step S1 and is an area to be recognized for the gesture operation. In other words, in the control method according to the embodiment, in the captured image I, the area other than the recognition area Ra is excluded from the recognition target of the gesture operation.

Then, in the control method according to the embodiment, the user's gesture operation is recognized in the recognition area Ra set in step S2 (step S3). For example, in the control method according to the embodiment, the gesture operation is recognized from the movement of the user's hand in the recognition area Ra.

That is, in the control method according to the embodiment, the gesture operation is recognized only in the recognition area Ra. Thereby, in the control method according to the embodiment, it is possible to suppress erroneous recognition of the gesture operation based on the unintended movement of the user.

Therefore, according to the control method according to the embodiment, it is possible to improve the recognition accuracy of the gesture operation.

Further, in the control method according to the embodiment, by setting the recognition area Ra, it is possible to reduce the processing load spent on the recognition processing of the gesture operation as compared with the case of recognizing the gesture operation from the entire captured image I. ..

Further, in the control method according to the embodiment, as will be described later, the recognition area Ra can be set for each part of the user's face, and different control commands can be associated with each recognition area Ra.

In other words, in the control method according to the embodiment, even when the same operation of the user is recognized as a gesture operation, a different control command can be output for each recognition area Ra. This makes it possible to output various control commands.

Next, a configuration example of the control device according to the embodiment will be described with reference to FIG. FIG. 2 is a block diagram of the control device. Note that FIG. 2 also shows the camera 10 and the in-vehicle device 50.

The camera 10 is an in-vehicle camera including an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is provided in the vehicle C, for example, so as to image the driver's seat. The camera 10 images the position of the driver's face and outputs the captured image I to the control device 1 at any time.

The in-vehicle device 50 is various devices mounted on the vehicle C. Hereinafter, the case where the in-vehicle device 50 is a navigation device having an audio function, a hands-free call function, and the like will be described, but the in-vehicle device 50 may be another device such as an air conditioner.

As shown in FIG. 2, the control device 1 according to the embodiment includes a control unit 2 and a storage unit 3. The control unit 2 includes an acquisition unit 21, a detection unit 22, a setting unit 23, a recognition unit 24, and an output unit 25. The storage unit 3 stores model information 31, command information 32, and arbitration information 33.

The control device 1 includes, for example, a computer having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a data flash, an input / output port, and various circuits.

The CPU of the computer functions as the acquisition unit 21, the detection unit 22, the setting unit 23, the recognition unit 24, and the output unit 25 of the control unit 2, for example, by reading and executing the program stored in the ROM.

Further, at least one or all of the acquisition unit 21, the detection unit 22, the setting unit 23, the recognition unit 24, and the output unit 25 of the control unit 2 are ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or the like. It can also be configured with the hardware of.

Further, the storage unit 3 corresponds to, for example, a RAM or a data flash. The RAM or data flash can store model information 31, command information 32, arbitration information 33, information on various programs, and the like. The control device 1 may acquire the above-mentioned program and various information via another computer or a portable recording medium connected by a wired or wireless network.

The model information 31 is information about a feature model for identifying a specific part from the captured image I. In the present embodiment, the specific part is each part of the face, and the model information 31 is information about the feature model of each part of the face.

The command information 32 is information to which the gesture operation and the control command are associated. FIG. 3 is a diagram showing a specific example of the command information 32. As shown in FIG. 3, the command information 32 is information in which "parts", "gesture operations", and "control commands" are associated with each other.

“Part” indicates a facial part, and indicates a recognition area Ra set for each part. In the example shown in FIG. 3, a case where the parts are a mouth and an ear is shown. “Gesture operation” indicates a gesture operation to be recognized in the recognition area Ra of each part.

As shown in FIG. 3, in the command information 32, different gesture operations are associated with each part. For example, the "mouth" is associated with the action of raising the index finger at the mouth and the action of covering the mouth with the palm as a gesture operation. Further, the action of covering the ear with the palm from the back side is associated with the "ear" as a gesture operation.

In addition, each gesture operation is associated with "control commands" such as volume down, mute, volume up, and mute release. Specifically, the "mouth" gesture operation is associated with control commands for lowering the volume such as volume down and mute, and the "ear" gesture operation is associated with volume up and mute release. , Control commands for raising the volume are associated.

That is, the user can lower the volume by performing a commonly used gesture operation for promoting silence, while the user can raise the volume by performing a gesture operation indicating that it is difficult to hear.

That is, by associating the control command related to the volume with a gesture operation that directly associates the volume up / down, the user can easily remember the control command corresponding to the gesture operation.

In other words, the user can easily perform the intended operation by the gesture operation. The command information 32 shown in FIG. 3 is an example and can be arbitrarily changed. Further, in the example of FIG. 3, the hand performing the gesture operation may be the right hand, the left hand, or the right hand and the left hand may be associated with different control commands.

Further, even for the same gesture operation, different control commands may be associated with each "part", or for example, control commands for different devices may be associated with each "part". As a result, gesture operations corresponding to various control commands can be accepted.

Further, as shown in FIG. 3, in the present embodiment, the recognition area Ra is set for each part of the face. The facial parts are easy to recognize, and the user can easily perform gesture operations by hand.

That is, the face part is an area where the recognition area Ra can be easily set and the user can easily touch it. Therefore, it is possible to contribute to both reduction of erroneous setting of the recognition area Ra and improvement of operability.

Returning to the description of FIG. 2, the arbitration information 33 will be described. The arbitration information 33 is information on the priority of the voice output by the vehicle-mounted device 50, and is information on the voice to be controlled by the above-mentioned control command.

FIG. 4 is a diagram showing a specific example of the arbitration information 33. As shown in FIG. 4, the arbitration information 33 is information in which a "state", an "interrupt", a "control target", and a "control command" are associated with each other.

"Status" indicates the audio type currently being output, and "interrupt" indicates the audio type to be output by an interrupt with respect to the audio type currently being output. “Control target” indicates a voice type to be controlled by the control command, and “control command” indicates a control command that is valid for the control target.

For example, in the example shown in FIG. 4, when there is no interrupt during the output of the audio audio, the control target is the audio audio, and the control commands are mute, unmute, volume up, and volume down.

Further, when there is no interrupt while the ringtone of the HF (hands-free) call is being output, the control target is the HF call (incoming call), and only mute is valid as the control command. Further, when an interrupt of the HF telephone (incoming call) is input while the audio / voice is being output, the control target is the HF telephone (incoming call), and only mute is valid as the control command.

As will be described later, the output unit 25 refers to the arbitration information 33, confirms the control target and the control command, and then outputs the control command to the in-vehicle device 50. In the present embodiment, the guidance voice of the navigation device, the warning sound, and the like may be excluded from the above-mentioned control targets.

Returning to the description of FIG. 2, each configuration of the control unit 2 will be described. The acquisition unit 21 of the control unit 2 acquires the captured image I input from the camera 10. The acquisition unit 21 outputs to the detection unit 22 each time the captured image I is acquired.

The detection unit 22 detects a specific part of the user from the captured image I acquired by the acquisition unit 21. As described above, the camera 10 is provided at a position where the driver's face is imaged. Therefore, the detection unit 22 detects each part of the driver's face as a specific part of the user.

That is, in the present embodiment, the target person for the gesture operation is the driver. As a result, the driver can easily operate the in-vehicle device by gesture operation even while driving the vehicle.

Specifically, the detection unit 22 detects each part by referring to the model information 31 and extracting the feature points of each part from the captured image I. The detection unit 22 passes the information regarding the position of each detected part to the setting unit 23 in association with the captured image I. The specific portion detected by the detection unit 22 is not limited to the facial part, and may be a portion other than the face, or the entire face may be detected as one specific portion.

The setting unit 23 sets the recognition area Ra based on the specific portion detected by the detection unit 22. Specifically, the setting unit 23 sets the recognition area Ra as a specific part in a predetermined area centered on each part of the face, for example.

Then, the setting unit 23 assigns an identifier indicating a facial part to the set recognition area Ra, and passes it to the recognition unit 24 together with the captured image I.

The recognition unit 24 recognizes the gesture operation in the recognition area Ra set by the setting unit 23. Specifically, when the recognition unit 24 recognizes the gesture operation to be recognized in each recognition area Ra from the captured image I, the recognition unit 24 recognizes the gesture operation. On the other hand, the recognition unit 24 does not recognize gesture operations that are not the recognition target in each recognition area Ra.

That is, the recognition unit 24 does not recognize the gesture operation outside the recognition area Ra in the captured image I, or the gesture operation outside the recognition target in the recognition area Ra even if it is in the recognition area Ra.

That is, by recognizing different gesture operations for each recognition area Ra, the recognition unit 24 recognizes only the gesture operations in which both the recognition area Ra and the gesture operation associated with the recognition area Ra match.

As a result, the recognition unit 24 can suppress erroneous recognition of recognizing an unintended operation of the user as a gesture operation. Further, since the recognition unit 24 only needs to perform the recognition process only in each recognition area Ra, it is possible to reduce the processing load as compared with the case where the recognition process is performed on the entire captured image I.

Further, the recognition unit 24 may set only the gesture operation that recognizes the gesture operation continuously for a predetermined time (for example, 3 seconds) as the recognition result. This is to prevent erroneous recognition of the gesture operation for the operation accidentally performed by the user in the recognition area Ra.

As a result, the recognition unit 24 can recognize only the gesture operation intentionally performed by the user, so that the recognition accuracy of the gesture operation can be improved.

The output unit 25 outputs a control command corresponding to the gesture operation recognized by the recognition unit 24 to the in-vehicle device 50. Specifically, the output unit 25 refers to the command information 32 and selects a control command according to the gesture operation recognized by the recognition unit 24.

Subsequently, the output unit 25 acquires information on the voice type currently being output by the in-vehicle device 50, and confirms the control target and valid control commands based on the arbitration information 33. Then, the output unit 25 outputs a control command addressed to the control target when the control command selected from the command information 32 is valid.

That is, as shown in FIG. 4, in a situation where there is no interruption during playback of audio-voice, the audio-voice is the control target and all control commands are valid, but the ringtone of the HF telephone is interrupted during playback. In the absence situation, only the mute control command is valid.

Therefore, the output unit 25 outputs the control command to the in-vehicle device 50 only when the recognition unit 24 recognizes the gesture operation corresponding to the mute in the situation where there is no interruption while playing the ringtone of the HF telephone. ..

Further, as shown in FIG. 4, the output unit 25 issues a control command to the in-vehicle device 50 to mute the ringtone of the HF telephone when an incoming HF telephone is interrupted during playback of the audio / voice. It will be output.

In this way, the output unit 25 selects and outputs a control target or a control command according to the voice type currently being output by the in-vehicle device 50, so that the user can easily perform a plurality of control targets with the same gesture operation. It becomes possible to operate. Therefore, it is possible to improve usability.

Next, the processing procedure executed by the control device 1 according to the embodiment will be described with reference to FIG. FIG. 5 is a flowchart showing a processing procedure executed by the control device 1. The processing procedure is repeatedly executed by the control unit 2.

As shown in FIG. 5, when the control device 1 first starts acquiring the captured image I from the camera 10 (step S101), the control device 1 detects a specific part of the user from each captured image I (step S102).

Subsequently, the control device 1 sets the recognition area Ra based on the specific portion (step S103), and determines whether or not the recognition of the gesture operation is started (step S104). When the control device 1 starts recognizing the gesture operation (step S104, Yes), it determines whether or not a predetermined time has elapsed from the start of recognition (step S105).

When the control device 1 determines that the predetermined time has elapsed (steps S105, Yes), the control device 1 determines whether or not the gesture operation is continuing (step S106). When the gesture operation is ongoing (step S106, Yes), the control device 1 recognizes it as a gesture operation (step S107).

Subsequently, the control device 1 acquires information on the voice type currently being output by the in-vehicle device 50 (step S108), and selects a control command and a control target based on the gesture operation and arbitration information 33 recognized in step S107. (Step S109).

Then, the control device 1 outputs a control command to the in-vehicle device 50 (step S110), and ends the process. Further, when the control device 1 has not started recognizing the gesture operation in the determination in step S104 (step S104, No), or when the gesture operation is not continuing in the determination in step S106 (step S106, No), the same. The process ends at that point.

Further, in the determination of step S105, if the predetermined time has not elapsed (steps S105, No), the control device 1 continues the process of step S105.

As described above, the control device 1 according to the embodiment includes a detection unit 22, a setting unit 23, and a recognition unit 24. The detection unit 22 detects a specific part of the user from the captured image I. The setting unit 23 sets the recognition area Ra based on the specific portion detected by the detection unit 22.

The recognition unit 24 recognizes the gesture operation in the recognition area Ra set by the setting unit 23. Therefore, according to the control device 1 according to the embodiment, the recognition accuracy of the gesture operation can be improved.

By the way, in the above-described embodiment, the case where the control device 1 recognizes the gesture operation of the driver in the vehicle C has been described, but the present invention is not limited to this. That is, the control device 1 may be mounted on a vehicle other than the vehicle C, or may control a device other than the in-vehicle device 50.

Further effects and variations can be easily derived by those skilled in the art. For this reason, the broader aspects of the invention are not limited to the particular details and representative embodiments described and described above. Therefore, various changes are possible without departing from the spirit or scope of the overall concept of the invention as defined by the appended claims and their equivalents.

1 Control device 10 Camera 21 Acquisition unit 22 Detection unit 23 Setting unit 24 Recognition unit 25 Output unit I Captured image Ra Recognition area

Claims (6)

A detector that detects a specific part of the user from image data,
A setting unit that sets a recognition area based on a specific part detected by the detection unit, and a setting unit.
A control device including a recognition unit that recognizes a gesture operation in a recognition area set by the setting unit. The detection unit
Detects each part of the face as a specific part,
The setting unit
The control device according to claim 1, wherein a recognition area is set for each part. The recognition unit
The control device according to claim 2, wherein a different gesture operation is recognized for each recognition area based on each part of the face detected by the detection unit. The detection unit
The control device according to claim 1, 2, or 3, wherein the specific part of the driver who is the user is recognized from the image data captured in the vehicle. It is further provided with an output unit that outputs a control command for controlling an in-vehicle device in response to a gesture operation recognized by the recognition unit.
The output unit
The control device according to claim 4, wherein the volume of the voice output from the in-vehicle device is adjusted according to a gesture operation. A detection process that detects a specific part of the user from image data,
A setting step of setting a recognition area based on a specific part detected by the detection step, and
A control method including a recognition step of recognizing a gesture operation in the recognition area set by the setting step.

Images