WO2020137592A1

WO2020137592A1 - Gesture detection device and gesture detection method

Info

Publication number: WO2020137592A1
Application number: PCT/JP2019/048757
Authority: WO
Inventors: 佳行津田
Original assignee: 株式会社デンソー
Priority date: 2018-12-27
Filing date: 2019-12-12
Publication date: 2020-07-02
Also published as: JP7024702B2; JP2020107030A

Abstract

This gesture detection device, which performs inputting on a manipulation screen (11) away from an operator on the basis of an instruction gesture performed by the operator, comprises: a detection unit (110) that detects a visual line direction and the instruction gesture of the operator; and a control unit (120) that, when there is a gaze on the manipulation screen for a predetermined gazing time or longer and there is an instruction gesture for a predetermined instruction time or longer from the detection data detected by the detection unit, executes a fine adjustment mode in which a manipulation cursor (11a) for manipulation using a gaze direction of the gaze as an initial position on the manipulation screen is displayed, and the manipulation cursor moves according to motion by the instruction gesture.

Description

Gesture detection device and gesture detection method

Cross-reference of related applications

This application is based on Japanese Patent Application No. 2018-244368 filed on Dec. 27, 2018, the disclosure of which is incorporated herein by reference.

The present disclosure relates to a gesture detection device that detects a gesture used for an input operation, and a gesture detection method.

As a conventional gesture detection device, for example, one described in Patent Document 1 is known. In the gesture detection device (information input device) of Patent Document 1, the device controller specifies the device to be controlled based on the gesture (hand shape, state, change in state) of the operator, and further issues a control command. It can be determined and input (remote operation) to the control target device. For example, when the display (TV) screen is pointed by the pointing gesture with the index finger raised as the operator's gesture, the device controller recognizes that the controlled device is the TV, and the pointing gesture , You can move the cursor on the display located on the extension line of your finger.

In addition, in the gesture detection device (operation input device) of Patent Document 2, the expansion surface is set by the display screen and the extension surface that is the outer area of the display screen. Then, a virtual line connecting the position of the operator's eyes (reference point) and the fingertip (pointer) of the hand held in front of the deployment surface defines the position (point of interest) on the deployment surface by pointing. It is supposed to be done.

JP, 2013-205983, A JP, 2005-321870, A

However, in each of the above-mentioned Patent Documents 1 and 2, a predetermined pointing method is determined in advance, and the pointing direction is specified based on that method. Therefore, if the operator makes a pointing gesture different from the predetermined pointing method, an erroneous operation occurs. On the other hand, if the operator is restricted in which part and how to point the finger in order to avoid erroneous operation, the operator feels troublesome.

In view of the above problems, one of the objects of the present disclosure is to provide a gesture detection device and a gesture detection method that can reduce erroneous operations without causing the operator to feel bothered.

A gesture detection device according to an example of the present disclosure is a gesture detection device that performs an input on an operation screen away from the operator based on an instruction gesture performed by the operator,
A detection unit that detects an operator's line-of-sight direction and a pointing gesture,
From the detection data detected by the detection unit, if there is a gaze for a predetermined time or more on the operation screen, and if there is an instruction gesture for a predetermined time or more, the gaze direction by the gaze is the initial position on the operation screen. And a control unit that executes a fine adjustment mode in which the operation cursor is displayed for operation and the operation cursor is moved according to the movement by the instruction gesture.

A gesture detection method according to an example of the present disclosure is a gesture detection method of performing input on an operation screen remote from the operator based on an instruction gesture performed by the operator,
A detection step of detecting the operator's gaze direction and gesture,
From the detection data detected in the detection step, if there is a gaze for a predetermined gaze time or longer on the operation screen, and if there is a gesture for a predetermined instruction time or longer, the gaze direction by the gaze is set as the initial position on the operation screen. An operation step of displaying an operation cursor for operation and executing a fine adjustment mode in which the operation cursor is moved in accordance with the movement by the instruction gesture.

With this, the operator can move the operation cursor displayed at the initial position of the gaze by performing an instruction gesture. Therefore, it is not necessary to finely regulate the gesture for the operator, and the operator can move the operation cursor accurately. In general, it is possible to reduce erroneous operations without causing the operator to feel bothered.

The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description with reference to the accompanying drawings. In the drawing,
FIG. 1 is a block diagram showing the overall configuration of the gesture detection device according to the first embodiment. FIG. 2 is a flowchart showing the control contents performed by the control unit. FIG. 3 is an explanatory diagram showing a procedure for determining that there is a gaze and an instruction gesture when the gaze time=instruction time. FIG. 4 is an explanatory diagram showing a procedure for allowing a time difference between the gaze and the pointing gesture when the gaze time≠instruction time. FIG. 5 is an explanatory diagram showing a modified example 1 in which the operation cursor is moved according to the position of the finger. FIG. 6 shows a second modification and is an explanatory diagram showing a movable range of the operation cursor. FIG. 7 shows the third modification and is an explanatory diagram showing the procedure when the advance notice is displayed. FIG. 8 is a diagram illustrating a third modification and is an explanatory diagram illustrating a procedure of performing a notice display when there is a time difference between the gaze and the pointing gesture. FIG. 9 is a diagram illustrating a fourth modification and is an explanatory diagram illustrating a procedure for canceling an operation.

A plurality of embodiments will be described below with reference to the drawings. In each embodiment, the same reference numerals may be given to portions corresponding to the matters described in the preceding embodiments, and redundant description may be omitted. In the case where only a part of the configuration is described in each embodiment, the other embodiments described above can be applied to the other part of the configuration. Not only the combination of the parts clearly stating that each embodiment can be specifically combined, but also the combination of the embodiments is partially combined even if not explicitly stated unless there is a problem in the combination. It is also possible.

(First embodiment)
The gesture detection device 100 according to the first embodiment will be described with reference to FIGS. 1 to 4. The gesture detection device 100 according to the present embodiment is mounted in, for example, a vehicle, detects a line-of-sight direction of an operator (driver)'s gaze, and a pointing direction and a pointing position by a pointing gesture, and, for example, various vehicle devices (operation screens). ) Is a device for instructing input.

▽The gaze direction of the operator's gaze is mainly understood as the direction of the gaze to the vehicle device (operation screen) that the operator wants to operate.

Further, the pointing gesture is, for example, a finger, a hand, an arm, an eye (line of sight), a face, or the like that indicates a direction or position desired by the operator. Here, a finger pointing gesture is mainly used. It will be described as being used. The finger pointing gesture is a gesture that indicates a desired direction depending on the extending direction of the finger, and for example, the index finger is used.

Examples of various vehicle equipment include, for example, an air conditioner that air-conditions the interior of the vehicle, a car navigation device (hereinafter, car navigation device) that displays the current position of the vehicle or guidance to the destination, or TV broadcast, radio broadcast, There is an audio device for reproducing a CD/DVD or the like. Here, an air conditioner will be described as a typical example of the vehicle equipment.

The air conditioner has, for example, a display unit 11 using a liquid crystal display, an organic EL display, or the like, and the display unit 11 includes, for example, an operation cursor 11a for input operation, an icon 11b, and an operating state. Is displayed. The display unit 11 corresponds to the operation screen separated from the operator of the present disclosure.

As shown in FIG. 1, the gesture detection device 100 includes a detection unit 110, a control unit 120, and the like. The control unit 120 issues an input instruction (instruction for input operation) to the display unit 11 (air conditioner) remote from the operator based on the line-of-sight direction of the operator detected by the detection unit 110 and the pointing gesture. I am supposed to do it.

In the air conditioner, the set temperature and the amount of conditioned air are changed by pointing gestures. The car navigation device displays the current position, sets the destination, enlarges/reduces the map, etc., and the audio device changes the television station, the radio station, selects the song, and changes the volume.

The detection unit 110 continuously (with time) detects a gaze state (gaze direction) of the operator with respect to the display unit 11 and a finger pointing gesture (direction of the index finger), and gaze direction data by gaze, and a finger pointing. The gesture data is output to the control unit 120. The detection unit 110 is configured to acquire the position of the operator's eyes (the direction of the line of sight) and the posture of the finger.

As the detection unit 110, a camera that forms a luminance image of an object, a range image sensor that forms a range image, or a combination thereof can be used. As the camera, there is a near-infrared camera that captures near-infrared rays, a visible-light camera that captures visible light, or the like. Further, as the range image sensor, for example, a stereo camera that simultaneously captures images with a plurality of cameras and measures information in the depth direction from parallax, or the depth is determined by the time until the light from the light source is reflected by the object and returned. There is a ToF (Time of Flight) camera to measure. In the present embodiment, as the detection unit 110, the camera that forms the brightness image of the target object as described above is used.

The control unit 120 is, for example, a computer including a CPU, ROM, RAM and the like. The control unit 120 operates the operation cursor on the display unit 11 based on the data (gaze direction data, finger pointing gesture data) detected by the detection unit 110 and preset positional relationship data between the display unit 11 and the detection unit 110. 11a is displayed, and a fine adjustment mode in which the operation cursor 11a is moved according to the movement by the pointing gesture is executed. The operation cursor 11a has, for example, a ring-shaped design. The design of the operation cursor 11a may be an arrow, a triangle, a cross, or the like, instead of the ring-shaped one.

The gesture detection device 100 of the present embodiment is configured as described above, and the operation and effect will be described below with reference to the flowchart of FIG. 2 and FIGS. 3 and 4.

First, in step S100 of FIG. 2, the control unit 120 acquires the operator's line-of-sight direction data and gesture data from the detection unit 110. Step S100 corresponds to the detection step of the present disclosure.

Next, in step S110, the control unit 120 determines from the line-of-sight direction data whether the operator is gazing in the area of the display unit 11 and whether there is a pointing gesture. The control unit 120 determines that there is gaze when the line of sight has been focused on an arbitrary position in the display unit 11 for a predetermined predetermined gaze time or longer. Further, when the operator's finger is pointing in the direction of the display unit 11 side for a predetermined instruction time or longer, the control unit 120 determines that there is a pointing gesture. The gaze time and the instruction time are, for example, about 200 ms, and the gaze time=instruction time may be set, or the gaze time≠instruction time may be set.

In the case of gaze time=instruction time, the time during which both gaze and finger pointing (moving is allowed) are both t1, and the threshold for determining whether time t1 is established is T1,
When the time during which both the gaze and the pointing (still) are performed is t2, and the threshold for determining the establishment of the time t2 is T2,
As shown in FIG. 3, when the time t1≧threshold value T1 is satisfied and then the t2≧threshold value T2 is satisfied (when T1+T2 gazing and finger pointing are satisfied), the control unit 120 makes the determination in step S110 an affirmative determination. .. Alternatively, as shown in FIG. 4, when the pointing gesture is performed within a certain time after the gaze, the control unit 120 makes the determination in step S110 an affirmative determination.

In the case of gaze time≠instruction time In the case of gaze time≠instruction time, it is unlikely that gaze and pointing are performed simultaneously. As shown in FIG. 4, when the pointing gesture is performed within a certain period of time after gazing, the control unit 120 makes the determination in step S110 an affirmative determination. That is, even if the gaze and the pointing are not performed at the same time, if the time difference is within a certain time, the control unit 120 proceeds to steps S120 and S130 below to display the operation cursor 11a and execute the fine adjustment mode. Tolerate.

If it is determined to be no in step S110, the control unit 120 repeats steps S100 and S110. On the other hand, if an affirmative decision is made in step S110, the control unit 120 moves to step S120.

In step S120, the control unit 120 sets the gazed position as the initial position in the display unit 11, and displays the operation cursor 11a at the initial position.

Next, in step S130, the control unit 120 executes the fine adjustment mode regarding the movement of the operation cursor 11a. That is, the control unit 120 first performs an offset (correction) to match the direction initially pointed by the finger pointing gesture (the finger pointing direction at initialization) with the initial position. Then, the control unit 120 moves the operation cursor 11a according to the movement of the finger by the finger pointing gesture of the operator. That is, the operation cursor 11a is moved in the direction of the pointing gesture performed by the operator (execution of the fine adjustment mode).

When moving the operation cursor 11a, the control unit 120 may incorporate attenuation of the movement amount so that the movement amount of the operation cursor 11a becomes smaller than the movement amount of the finger.

here,
The yaw direction (left-right direction) after the offset is θn,
The pitch pointing direction (vertical direction) after offset is φn,
The yaw direction (left-right direction) before offset is θo,
The pointing direction in the pitch direction (vertical direction) before offset is φo,
The physical pointing direction (yaw direction) at initialization is θp,
The physical pointing direction (pitch direction) at initialization is φp,
The direction (yaw direction) pointing to the initial position of the operation cursor 11a is θg,
The direction (pitch direction) indicating the initial position of the operation cursor 11a is φg,
The yaw angle magnification (greater than 0 and less than or equal to 1) is α,
If the scaling factor for the pitch angle (greater than 0 and less than 1) is β,
θn=α(θo−θp)+θg
φn=β(φo-φp)+φg
In this way, θn and φn can be expressed as linear correction equations, and the respective magnifications α and β should be set to values of 1 or less (attenuation weaving amount of movement). Instead of the linear correction formula, a polynomial expression or a sigmoid curve may be used.

On the contrary, the control unit 120 sets the magnifications (movement magnifications) α and β of the movement of the operation cursor 11a during execution of the fine adjustment mode to be higher as the detection level of the movement by the gaze and the instruction gesture is higher. May be.

Returning to step S130, when the desired icon 11b is selected by the operation cursor 11a and a predetermined determination gesture (for example, tap gesture) is performed, the control unit 120 outputs an input instruction regarding the icon 11b to the air conditioner. .. That is, an input operation is performed on the air conditioner. The above steps S110 to S130 correspond to the execution steps of the present disclosure.

The continuation and termination of the fine adjustment mode are executed as follows. That is, in step S140, the control unit 120 determines whether or not there is movement in the line-of-sight direction of the operator, and if there is movement in the line-of-sight direction, in step S150 whether there is a predetermined gesture by the operator in advance. Determine whether or not.

When the negative determination is made in step S140, and when the positive determination is made in step S140 and the negative determination is made in step S150, the control unit 120 continues the fine adjustment mode (movement of the operation cursor 11a based on the pointing gesture) in step S160. Let That is, since the operator's line-of-sight direction moves rapidly, it may be re-initialized by an unexpected movement. However, even if the line-of-sight direction is moved after the operation cursor 11a is displayed, a predetermined predetermined value is set. The fine adjustment mode continues to be executed until there is a gesture. Step S160 is repeatedly executed according to the determinations of steps S140 to S150.

The predetermined gesture that is determined in advance is a gesture that means that the operator intentionally stops the pointing gesture, and includes, for example, a gesture of making a hand, a gesture of breaking the shape of a hand (shape of pointing), It may be a predetermined decision gesture for input, a gesture to shake the head, a gesture to redirect the finger in the direction of the moved line of sight, a gesture to tap another finger with the thumb, or a gesture to re-hold the finger other than the index finger. it can.

On the other hand, if a positive determination is made in both steps S140 and S150, the control unit 120 ends execution of the fine adjustment mode in step S170.

As described above, in the present embodiment, when the operator is gazing at the display unit 11 and there is a pointing gesture, the control unit 120 sets the gaze direction by the gaze as the initial position on the display unit 11 and sets the operation cursor. 11a is displayed, and the operation cursor 11a is moved according to the movement by the pointing gesture (fine adjustment mode execution).

With this, the operator can move the operation cursor 11a displayed at the gazed initial position by performing a pointing gesture. Therefore, the operator does not need to finely control the gesture, and the operator can move the operation cursor 11a accurately. In general, it is possible to reduce erroneous operations without causing the operator to feel bothered.

Further, after displaying the operation cursor 11a, the control unit 120 continues to execute the fine adjustment mode according to the movement by the instruction gesture, even if there is a movement in the line-of-sight direction until a predetermined gesture is given. I am trying. This prevents the fine adjustment mode from being interrupted by the re-initialization even if the operator makes an unexpected movement in the line-of-sight direction.

In step S130, the control unit 120 incorporates attenuation of the movement amount (reduces the respective magnifications α and β) so that the movement amount of the operation cursor 11a becomes smaller than the movement amount of the finger, or On the contrary, the higher the detection level of the gaze and the motion of the pointing gesture is, the higher the respective magnifications α and β of the movement of the operation cursor 11a during the execution of the fine adjustment mode are. Thereby, the operability of the operation cursor 11a can be improved.

Specifically, when the condition of gaze and pointing gesture detection is bad (strong ambient light, low confidence in detection processing, etc.), the respective magnifications α and β are made relatively low to detect noise. The influence of can be suppressed. If the condition is good, the magnifications α and β can be relatively increased to move the operation cursor 11a quickly.

Further, the control unit 120 allows the fine adjustment mode to be executed when a pointing gesture is performed within a certain time after gazing. As a result, the operating conditions can be eased, and the operator can easily use the device.

(Modification 1)
Modification 1 is shown in FIG. In the first embodiment, the control unit 120 executes the fine adjustment mode based on a change in the direction of the finger pointing as a pointing gesture, but is not limited to this, and based on a change in the position of the pointing. It may be one.

For example, a constant magnification is used to convert the movement amount (mm) of the finger from the physical coordinates at the time of initialization into the movement amount (pix (pixel)) of the operation cursor 11a, or the same as the first embodiment. Similarly, it is possible to use a polynomial, a sigmoid curve, or the like.

FIG. 5 shows how the operation cursor 11a is moved when the position of the finger (fingertip) is moved when executing the fine adjustment mode using the finger. The plane for moving the fingers may not be parallel to the display unit 11. For example, the fingers may be tilted 90 degrees so that the direction away from the body corresponds to the upward direction on the display unit 11.

With this, it is possible to increase the degree of freedom in operating the pointing gesture (pointing gesture).

(Modification 2)
Modification 2 is shown in FIG. Modification 2 is different from the first embodiment in that the control unit 120 displays the movable range of the operation cursor 11a. The movable range is displayed, for example, as a circle including the operation cursor 11a with the position of the operation cursor 11a as the center.

The size of the movable range may be increased or decreased according to the physique of the operator. The movable range may be displayed under a specific condition, for example, when the operation cursor 11a is near the boundary of the movable range for a certain time after initialization.

With this, the operator can know the possible range of finger operation and improve operability.

(Modification 3)
Modification 3 is shown in FIGS. Modification 3 is different from the first embodiment in that the control unit 120 performs a notice display indicating the initial position when the operator is gazing and in the middle of a predetermined instruction time. It is a thing.

As shown in FIG. 7, the control unit 120 displays the operation cursor 11a in advance at the initial position after the time t1≧threshold T1 is satisfied and before the time t2≧threshold T2 is satisfied, and after the time t2≧threshold T2 is satisfied. , The original operation cursor 11a is displayed.

Alternatively, as shown in FIG. 8, even when there is a time difference between the gaze and the pointing gesture, the control unit 120 operates after the time t1≧threshold T1 is satisfied and before the time t2≧threshold T2 is satisfied. The cursor 11a is displayed in advance at the initial position and the original operation cursor 11a is displayed after t2≧threshold value T2 is satisfied.

The design of the operation cursor 11a for notice display and the original design (look) of the operation cursor 11a may be different. When the design of the operation cursor 11a is different, the color, brightness, size, and transmittance are changed, a specific figure or an outer frame is displayed in an overlapping manner, or the movable range described in Modification 2 is announced. There is a method of displaying only at the time of display or only at the time of the original operation cursor 11a.

With this, since the initial position can be confirmed before moving the operation cursor 11a by pointing, it is easy to adjust the movement of the operation cursor 11a thereafter.

(Modification 4)
Modification 4 is shown in FIG. The modification 4 is different from the first embodiment in that the control unit 120 cancels the execution of the fine adjustment mode when the stationary state of the instruction gesture is not recognized after the gaze.

When the time until the time t2≧the threshold T2 is established is t3, and the threshold for determining the establishment of the time t3 is T3,
The control unit 120 displays the notice after the time t1≧threshold value T1 is satisfied, but when t3≧threshold value T3 is satisfied without the t2≧threshold value T2 being satisfied, the series of operations is canceled and the operation cursor 11a is moved. to erase. That is, the control unit 120 cancels the execution of the fine adjustment mode when the time t3 exceeds the threshold value T3 unless there is a still state with the pointing gesture after the gazing.

As a result, even if the operator is gazing, if the pointing gesture is not stationary, it is determined that there is no intention of a clear operation, the unnecessary fine adjustment mode is stopped, and the control load is reduced. be able to.

(Other embodiments)
In the first embodiment and each of the modified examples 1 to 4, the operation target is various vehicle devices, but the invention is not limited to this and may be applied to devices provided for homes and facilities. Examples of household equipment include televisions and audio equipment, and examples of equipment for facilities include ATMs at banks and automatic ticket vending machines at stations.

In addition, the target operator of the one installed in the vehicle is not limited to the driver, but may be the passenger seat. In this case, the passenger seat can also perform various gestures by the gesture detection device 100 by performing the instruction gesture, and can operate various vehicle devices.

The disclosure in this specification and drawings is not limited to the illustrated embodiment. The disclosure encompasses the illustrated embodiments and variations on them based on them. For example, the disclosure is not limited to the combination of parts and/or elements shown in the embodiments. The disclosure can be implemented in various combinations. The disclosure may have additional parts that may be added to the embodiments. The disclosure includes omissions of parts and/or elements of the embodiments. The disclosure includes replacements or combinations of parts and/or elements between one embodiment and another. The disclosed technical scope is not limited to the description of the embodiments. It is to be understood that some technical scopes disclosed are shown by the description of the claims, and further include meanings equivalent to the description of the claims and all modifications within the scope.

The control unit 120 and the method thereof described in the present disclosure may be realized by a dedicated computer configuring a processor programmed to execute one or a plurality of functions embodied by a computer program. Alternatively, the apparatus and method described in the present disclosure may be realized by a dedicated hardware logic circuit. Alternatively, the device and the method described in the present disclosure may be realized by one or more dedicated computers configured by a combination of a processor that executes a computer program and one or more hardware logic circuits. Further, the computer program may be stored in a computer-readable non-transition tangible recording medium as an instruction executed by the computer.

Claims

In a gesture detection device for performing input on an operation screen (11) separated from the operator based on an instruction gesture made by the operator,
A detection unit (110) for detecting the direction of the operator's line of sight and the pointing gesture;
From the detection data detected by the detection unit, if there is a gaze for a predetermined gaze time or more on the operation screen, and if there is the pointing gesture for a predetermined pointing time or more, the gaze direction by the gaze is operated. A gesture detection including: a control unit (120) that displays an operation cursor (11a) for operation as an initial position on the screen and executes a fine adjustment mode in which the operation cursor is moved according to the movement by the instruction gesture. apparatus.
After the operation cursor is displayed, the control unit continues to execute the fine adjustment mode according to the movement by the instruction gesture until a predetermined gesture is made, even if there is movement in the line-of-sight direction. The gesture detection device according to claim 1.
The gesture detection device according to claim 1 or 2, wherein the control unit executes the fine adjustment mode in accordance with at least one of a change in a direction indicated by the instruction gesture and a change in a position.
4. The control unit increases the moving magnification of the operation cursor when executing the fine adjustment mode, as the detection level of the gaze and the movement by the pointing gesture is higher. The gesture detection device according to 1.
The gesture detection device according to any one of claims 1 to 4, wherein the control unit displays a movable range of the operation cursor.
The gesture detection device according to any one of claims 1 to 5, wherein the control unit performs a notice display indicating the initial position in the middle of the predetermined instruction time with the gaze.
The gesture detection device according to claim 6, wherein the control unit displays the operation cursor and the notice display so that they look different.
The gesture detection device according to any one of claims 1 to 7, wherein the control unit cancels execution of the fine adjustment mode when the stationary state of the instruction gesture is not recognized after the gaze.
9. The gesture detection according to claim 1, wherein the control unit permits execution of the fine adjustment mode when the instruction gesture is performed within a certain time after the gaze. apparatus.
In the gesture detection method for inputting on the operation screen (11) away from the operator based on the instruction gesture made by the operator,
A detection step (S100) of detecting the operator's gaze direction and the pointing gesture,
From the detection data detected in the detection step, if there is a gaze for a predetermined gaze time or more on the operation screen, and if there is the pointing gesture for a predetermined pointing time or more, the gaze direction by the gaze is operated. An operation step (S110 to S130) for displaying an operation cursor (11a) for operation as an initial position on the screen and executing a fine adjustment mode for moving the operation cursor according to the movement by the instruction gesture. Gesture detection method.