JP2020107031A - Instruction gesture detection apparatus and detection method therefor - Google Patents

Abstract

To provide an instruction gesture detection apparatus which can properly correct even a large error in correcting an actually specified position for an operation object during ordinary operation, and a method therefor.SOLUTION: An instruction gesture detection apparatus which detects a specified operation object out of operation objects 11a, 11b displayed on an operation screen 11 located at a distance from an operator, on the basis of an instruction gesture made by the operator includes: a detection unit 110 which detects an instruction gesture of an operator; and a control unit 120 which executes position correction for aligning a position actually specified by the instruction gesture with a position of the operation object desired by the operator, on the basis of detection data detected by the detection unit. The control unit increases an original distance between the operation objects to be equal to or larger than a predetermined distance, in executing position correction.SELECTED DRAWING: Figure 1

Description

The present invention relates to an instruction gesture detection device that detects an operation target indicated by an instruction gesture used for an input operation, and a detection method thereof.

As a conventional pointing gesture detecting device, for example, one described in Patent Document 1 is known. In the pointing gesture detection device (eye-gaze input device) of Patent Document 1, at the first stage when the user uses the eye-gaze input system, an actual gazing point with respect to the operation frame and a detection point obtained by the sensor based on the eye-gaze of the user. Initial calibration is performed to correct the deviation (error) from the

In addition, when an event such as closing the eyes or selecting a menu with a line of sight occurs during use of the system, dynamic calibration is performed. In the dynamic calibration, the shift amount between the gazing point and the detection point with respect to the operation frame is corrected as in the initial calibration. The correction coefficient obtained by the dynamic calibration is used in the subsequent control.

Japanese Patent Laid-Open No. 2000-10723

In the above-mentioned Patent Document 1, error correction is performed on the assumption that the detection point falls within the operation frame that is focused on, from the viewpoint of performing a dedicated operation for calibration when performing the initial calibration. Has become. However, setting the initial calibration as a dedicated operation is troublesome for the user. Considering that the calibration is performed during the normal operation while omitting the trouble of the initial calibration, the following problems are conceivable. That is, when a plurality of operation frames are arranged adjacent to each other and the deviation (error) between the actual gazing point and the detection point is large, for example, when the detection point is located outside the operation frame, which operation frame is corrected. It may be difficult to determine whether to perform the correction, and correct correction may not be possible.

If the correction is not performed correctly, the user intentionally performs an operation that is aware of the difference between the gazing point and the detection point, which makes it impossible to obtain an accurate detection point and corrects even more correctly. I can not do it.

In view of the above problems, an object of the present invention is to provide a pointing gesture detection device and a method thereof that can correct the position correctly even in a state where a deviation (error) is large in correcting an actual pointing position with respect to an operation target in a normal operation. To provide.

The present invention adopts the following technical means in order to achieve the above object.

In the first invention, based on an instruction gesture made by the operator, the instructed operation object is detected from among the plurality of operation objects (11a, 11b) displayed on the operation screen (11) away from the operator. In the pointing gesture detection device,
A detection unit (110) for detecting an operator's instruction gesture;
Based on the detection data detected by the detection unit, a control unit (120) for performing a position correction for aligning the position actually pointed by the pointing gesture with the position of the operation target desired by the operator,
The control unit is characterized by widening the original intervals of the plurality of operation targets to be equal to or more than a predetermined interval when performing the position correction.

Further, in the second invention, based on the instruction gesture performed by the operator, the instructed operation object is selected from among the plurality of operation objects (11a, 11b) displayed on the operation screen (11) away from the operator. In the instruction gesture detection method for detecting,
A detection step of detecting the instruction gesture of the operator,
Based on the detection data detected in the detection step, the position actually designated by the pointing gesture, a correction step for performing a position correction to match the position of the operation target desired by the operator,
The correction step is characterized in that the original intervals of the plurality of operation targets are widened to be equal to or larger than a predetermined interval.

According to the present invention, when the position correction is performed during the normal operation, the original intervals of the plurality of operation targets (11a, 11b) are widened to be equal to or more than the predetermined interval, so that the respective operations are performed. It is possible to absorb the variation of the actual pointing position around the object (11a, 11b) and set a reaction area that reacts to the pointing operation to the operation target (11a, 11b). Become. Therefore, even if the actual pointing position is located outside the operation target (11a, 11b) and the deviation (error) is large, which operation target among the plurality of operation targets (11a, 11b) Since it is clearly understood whether the position should be corrected, it is possible to correct the position correctly.

The reference numerals in parentheses of the above-mentioned means indicate the correspondence with the specific means described in the embodiments described later.

It is a block diagram which shows the whole structure of the instruction|indication gesture detection apparatus in 1st Embodiment. It is explanatory drawing which shows the point at the time of performing a position correction (example of two icons). It is explanatory drawing which shows the point at the time of performing a position correction (example of six icons). It is explanatory drawing which shows the operation cursor (screen division part) at the time of position correction in 2nd Embodiment. It is explanatory drawing which shows the operation cursor (blurred part) at the time of position correction in 2nd Embodiment.

Hereinafter, a plurality of modes for carrying out the present invention will be described with reference to the drawings. In each form, parts corresponding to the items described in the preceding form may be designated by the same reference numerals, and redundant description may be omitted. In the case where only a part of the configuration is described in each mode, the other mode described above can be applied to the other part of the configuration. Not only the combination of the parts clearly showing that the respective embodiments can be specifically combined, but also the combination of the embodiments is partially combined even if the combination is not particularly specified, unless there is a problem in the combination. It is also possible.

(First embodiment)
The instruction gesture detection device 100 according to the first embodiment will be described with reference to FIGS. 1 to 3. The instruction gesture detection device 100 according to the present embodiment is mounted on, for example, a vehicle, detects a pointing direction and a pointing position by an instruction gesture of an operator (driver), and displays a plurality of items on the display unit 11 of various vehicle devices. The designated icons (11a to 11d) are detected from among the icons 11a to 11d. Then, based on the instructed icons (11a to 11d), an input instruction is given to various vehicle devices.

The instruction gesture by the operator is, for example, a finger, a hand, an arm, an eye (line of sight), a face, or the like that indicates a direction or position desired by the operator. Here, mainly, a finger pointing gesture is used. Will be described. The finger pointing gesture is a gesture that points in a desired direction depending on the extending direction of the finger, and for example, the index finger is used.

Examples of various vehicle devices include an air conditioner for air conditioning the interior of the vehicle, a car navigation device (hereinafter, car navigation device) for displaying the current position of the vehicle or guidance to a destination, television broadcasting, radio broadcasting, There is an audio device for reproducing a CD/DVD or the like. Here, an air conditioner will be described as a typical example of the vehicle equipment.

The air conditioner has, for example, a display unit 11 using a liquid crystal display, an organic EL display, or the like. On the display unit 11, for example, icons 11a to 11d for performing an input operation to the air conditioner, And the operating status is displayed. The display unit 11 corresponds to the operation screen away from the operator of the present invention, and the icons 11a to 11d correspond to the plurality of operation targets of the present invention.

The icons 11a to 11d correspond to operation buttons for input operation, and for example, each has a rectangular shape and a plurality (here, four) are provided. The icons 11a to 11d are arranged so as to be adjacent to each other in the left-right direction. The plurality of icons 11a to 11d include, for example, an icon 11a for increasing the air conditioning set temperature, an icon 11b for decreasing the air conditioning set temperature, an icon 11c for increasing the air conditioning air volume, an icon 11d for decreasing the air conditioning air volume, and the like. ..

The instruction gesture detection device 100 includes a detection unit 110, a control unit 120, and the like, as shown in FIG.

The detection unit 110 continuously detects a pointing gesture (direction of the index finger) with respect to the display unit 11 (the plurality of icons 11a to 11d) of the operator over time, and indicates the direction of the operator's finger with the pointing gesture data. Is output to the control unit 120. The control step in which the finger pointing gesture is detected by the detection unit 110 corresponds to the detection step of the present invention.

Further, the detection unit 110 captures the operator's body position or face to detect the operator's seating position data (mainly the seating position in the front-rear direction by seat slide) or the operator identification data (for example, Mr. A or Mr. B). Is output to the control unit 120.

As the detection unit 110, a camera that forms a brightness image of an object, a range image sensor that forms a range image, or a combination thereof can be used. As the camera, there is a near infrared camera that captures near infrared rays, a visible light camera that captures visible light, or the like. Further, as the range image sensor, for example, a stereo camera that simultaneously captures images with a plurality of cameras to measure information in the depth direction from parallax, or the depth from the time when light from a light source is reflected by an object and returned. There is a ToF (Time of Flight) camera for measuring. In the present embodiment, as the detection unit 110, the camera that forms the brightness image of the target object as described above is used.

The control unit 120 is, for example, a computer including a CPU, a ROM, a RAM, and the like. The control unit 120 is based on the data detected by the detection unit 110, that is, the finger pointing gesture data, the sitting position data, the operator identification data, and the like, and the positional relationship data between the display unit 11 and the detection unit 110 set in advance. Then, position correction (calibration) is performed to match the position actually designated by the operator with the position of the (target) icon (11a to 11d) desired by the operator (details will be described later). ). The positions of the aimed icons (11a to 11d) can be defined as the center positions of the icons (11a to 11d), for example.

The control step of the position correction performed by the control unit 120 corresponds to the correction step of the present invention. Then, the control unit 120 detects the instructed icons (11a to 11d) and gives an input instruction (instruction for input operation) to the air conditioner.

In the air conditioner, the set temperature is changed, the amount of conditioned air is changed, and the like according to the icons (11a to 11d) selected and determined based on the pointing gesture. The car navigation device displays the current position, sets the destination, enlarges/reduces the map, etc., and the audio device changes the television station, the radio station, selects the song, and changes the volume.

The pointing gesture detection device 100 according to the present embodiment has the above-described configuration, and the position correction performed by the control unit 120 will be described below with reference to FIGS. 2 and 3.

1. Position Correction Method In the position correction according to the present embodiment, a dedicated correction work for position correction is not set for the operator, and the position correction is performed in the normal operation to the vehicle device based on the pointing gesture. I am trying to keep going. The position correction is executed when the pointing gesture detecting device 100 is used for the first time, when the sitting position of the operator is changed, when the operator is changed, or the like. The position correction is executed stepwise using, for example, the following linear equations set and data obtained in order by the pointing gesture.

First, it is assumed that the finger pointing gesture for the operator's icon (FIG. 1, 11a to 11d) is performed in order of 1st, 2nd...ith.

Let yaw angle (horizontal angle) in the direction of aiming the icons (11a to 11d) from the operator be Θi, and pitch angle (vertical angle) be Φi,
If the yaw angle in the direction actually instructed by the operator is θi and the pitch angle is φi,
The deviation between the target direction and the actually designated direction is represented by the following linear expression.

_Θi = aθi + b + ε θi
Φi=cφi+d+ε _φi
However, a, b, c and d are correction parameters, and ε _θi and ε _φi are correction residuals.

Since two equations are obtained for one icon (any of 11a to 11d) and there are four correction parameters, at least two icons (two of 11a to 11d) should be used to correct all parameters. Need to tell.

At the start of the operation, the correction cannot be completely performed, and therefore a=1, b=0, c=1, d=0 are used as default values.

When the operator designates the first icon (one of 11a to 11d), only two parameters (for example, b and d) are calculated and the position is corrected. At this time,
b=Θ1-θ1
d=Φ1-Φ1.

When the operator designates the second icon (one of 11a to 11d), all parameters are calculated as follows and position correction is performed. At this time,
a=(Θ2-θ1)/(θ2-θ1)
b=(θ2θ1-θ1θ2)/(θ2-θ1)
c=(Φ2-Φ1)/(Φ2-Φ1)
d=(φ2Φ1-φ1Φ2)/(φ2-φ1).

When the operator designates the third and subsequent icons (one of 11a to 11d), the least squares method is used for updating. At this time, all the points designated in the past may be used, or the latest N points may be used.

Total error D=Σ{(aθi+b−Θi) ² +(cφi+d−φi) ² }
To minimize (solve ∂D/∂a=0, ∂D/∂b=0, ∂D/∂c=0, ∂D/∂d=0 for a, b, c, d), etc. In addition, it is possible to make a stepwise correction according to the amount of acquired data.

2. Enlargement of Icon Intervals FIG. 2 shows a case where two icons 11a and 11b are displayed for the sake of simplicity. Although FIG. 2 shows the case where the position correction is performed in three stages (when the correction is started, in the middle of the correction, and when the correction is completed), the present invention is not limited to this and may be performed in two stages (correction start, correction completion). ..

When performing the position correction, the control unit 120 widens (enlarges) the original interval between the plurality of icons (11a, 11b) to be a predetermined interval or more. In short, by separating the positions of the respective icons (11a, 11b), as shown in FIG. 2, it is possible to absorb the variation in the actually designated position, and perform an instruction operation for the icon (11a, 11b). The region that can react with (hereinafter, reaction region) is set to be wide.

The control unit 120 may change the predetermined interval stepwise so as to approach the original interval in accordance with the progress of the position correction (right in FIG. 2).

Further, as shown in FIG. 3, for example, when the number of icons is six (11a to 11f), the number of displayed icons is reduced (2, 11a, 11b) at the time of starting the position correction to match the progress of the position correction. The number of displayed icons may be increased in sequence (four, 11a to 11d). When the number of icons is two (11a, 11b) or four (11a to 11d), it is preferable that the remaining icons 11c to 11f can be displayed on the hierarchical display screen.

As described above, when the position correction is performed during the normal operation, the original intervals of the plurality of icons (11a, 11b) are widened to be equal to or more than the predetermined interval, so that the respective icons (11a, 11b, It is possible to set the reaction area around 11b) to absorb the variation of the actual pointing position and to react to the pointing operation to the icon (either 11a or 11b). Therefore, even if the actual pointing position is located outside the icons (11a, 11b) and the deviation (error) is large, the position of any of the plurality of icons (11a, 11b) can be corrected. Since it is clearly understood whether or not it should be performed, it is possible to correct the position correctly.

In addition, the intervals of the icons (11a, 11b) are mischievously changed by gradually changing (approaching) the intervals of the icons (11a, 11b) to the original intervals according to the progress of the position correction. It is possible to suppress the spread.

(Second embodiment)
The second embodiment is shown in FIGS. The second embodiment is different from the first embodiment in that a cursor 11g for indicating a designated position is displayed on the display unit 11 at a position actually designated by a pointing gesture. .. The cursor 11g has, for example, a design in which a cross is provided in a ring shape. The shape of the cursor 11g is not limited to this, and may be an arrow, a triangle, a polygon, or the like.

Then, the control unit 120 displays the cursor 11g as a rough area when the position correction is not completed. In other words, the display intentionally does not clearly show the position of the cursor 11g to the operator (display is such that the position of the cursor can be roughly understood).

FIG. 4 illustrates a case where a rectangular area (screen divided area) is set instead of the cursor 11g. The size of the rectangular area is made to approach (become smaller) the original size of the cursor 11g according to the progress of the position correction.

Further, FIG. 5 shows a case where the cursor 11g is displayed in a blurred state. In order to blur the cursor 11g, there are methods such as blurring the position of the cursor 11g, blurring the boundary, lowering the contrast (making the color unclear), and increasing the transparency (making the color lighter and transparent). is there. Similar to FIG. 4, the size of the blurring is made closer to the original size of the cursor 11g (becoming smaller) according to the progress of the position correction.

As a result, at the stage of position correction, the operator intentionally recognizes the deviation between the position based on his/her own sense pointed by the pointing gesture and the position of the cursor 11g actually displayed. It is possible to prevent the finger pointing gesture from being performed in accordance with the position shifted to. That is, when the operator intentionally makes a pointing gesture at a position that is displaced, an accurate detection point cannot be obtained, and correct position correction cannot be performed forever, but such a defect can be suppressed.

(Other embodiments)
In each of the above-described embodiments, the control unit 120 executes the position correction based on the direction of the finger pointing as the finger pointing gesture, but the present invention is not limited to this, and may be based on the position of the finger pointing.

Further, in each of the above-described embodiments, the operation target is various vehicle devices, but the present invention is not limited to this and may be applied to devices provided for homes and facilities. Examples of household equipment include televisions and audio equipment, and examples of equipment for facilities include ATMs at banks and automatic ticket vending machines at stations.

Further, in a vehicle-mounted type, the target operator is not limited to the driver, but may be a passenger. In this case, the passenger seat also performs the instruction gesture, whereby the gesture detection by the instruction gesture detection device 100 is performed, and various vehicle devices can be operated.

The disclosure in this specification and the drawings is not limited to the illustrated embodiments. The disclosure encompasses the illustrated embodiments and variations on them based on them. For example, the disclosure is not limited to the combination of parts and/or elements shown in the embodiments. The disclosure can be implemented in various combinations. The disclosure may have additional parts that may be added to the embodiments. The disclosure includes omissions of parts and/or elements of the embodiments. The disclosure includes replacements or combinations of parts and/or elements between one embodiment and another. The disclosed technical scope is not limited to the description of the embodiments. It is to be understood that some technical scopes disclosed are indicated by the description of the claims and further include meanings equivalent to the description of the claims and all modifications within the scope.

The control unit 120 and the method described in the present disclosure may be realized by a dedicated computer that configures a processor programmed to execute one or a plurality of functions embodied by a computer program. Alternatively, the apparatus and the method described in the present disclosure may be realized by a dedicated hardware logic circuit. Alternatively, the apparatus and the method thereof described in the present disclosure may be realized by one or more dedicated computers configured by a combination of a processor that executes a computer program and one or more hardware logic circuits. Further, the computer program may be stored in a computer-readable non-transition tangible recording medium as an instruction executed by the computer.

11 Display (operation screen)
11a to 11f icons (operation target)
11g Operation cursor (cursor)
100 instruction gesture detection device 110 detection unit 120 control unit

Claims (4)

An instruction gesture detection device that detects the instructed operation target from among the plurality of operation targets (11a, 11b) displayed on the operation screen (11) away from the operator based on the instruction gesture performed by the operator. At
A detection unit (110) for detecting the instruction gesture of the operator,
A control unit (120) for performing position correction for adjusting the position actually designated by the pointing gesture to the position of the operation target desired by the operator based on the detection data detected by the detection unit; ,,
The said control part is an instruction|indication gesture detection apparatus which widens the original space|interval of the said some operation target so that it may become more than a predetermined predetermined space|interval, when performing the said position correction. The instruction gesture detection device according to claim 1, wherein the control unit changes the predetermined interval stepwise so as to approach the original interval according to the progress of the position correction. On the operation screen, a cursor (11g) for indicating the designated position is displayed at the actually designated position.
The pointing gesture detection device according to claim 1, wherein the control unit displays the cursor as a rough area when the position correction is not completed. An instruction gesture detection method for detecting the instructed operation target among a plurality of operation targets (11a, 11b) displayed on the operation screen (11) away from the operator based on the instruction gesture performed by the operator. At
A detection step of detecting the pointing gesture of the operator,
A correction step of performing a position correction to match the position actually pointed by the pointing gesture with the position of the operation target desired by the operator based on the detection data detected in the detection step. ,
In the correction step, the instruction gesture detection method in which the original intervals of the plurality of operation targets are expanded to be a predetermined interval or more.

Images