JP5838061B2 - Driving assistance device - Google Patents

Description

  Embodiments described herein relate generally to a driving support apparatus that operates an operated device in a vehicle with an input interface that can ensure driving safety.

  2. Description of the Related Art Conventionally, as a driving assistance device, a device in which a touch pad is arranged in the middle of a handle and an operated device is operated is known. In this case, it is necessary to confirm the position with the line of sight once toward the touch pad and to release the hand from the handle.

  There is also known an input device that detects which part of an image displayed by a projector disposed in an automobile is touching. In this case, it is necessary to touch the image for the input operation, and in addition to confirming the position to touch the image, it is necessary to release the hand from the handle.

JP 2010-117748 A JP 2004-262363 A

  The conventional technique has a safety problem because it is necessary to check the line of sight once for input operation and to release the hand from the handle.

  In this embodiment, there is provided a driving support device that can operate an operated device with an input interface that can ensure driving safety.

According to the embodiment, an operation start determination unit that determines whether the operation of the operated device is started by a slide operation of the handle, an operation end determination unit that ends the slide operation started by the operation start determination unit, and the operation Ri by the start of the start determination unit, the relative movement of the hand on the handle from the movement and position of the hand on the front SL handles calculated, lateral manipulation of the hand on the handle wheel, the vertical wheel An operation processing unit that processes determination of the operation content on the handle based on at least one of the twisting operations, and drives the operated device based on the operation content determined by the operation processing unit A drive unit.

1 is a schematic system configuration diagram for explaining a first embodiment relating to a driving support device. FIG. It is a flowchart for demonstrating operation | movement. It is explanatory drawing for demonstrating the specific marker installation position of vibration component cancellation. It is explanatory drawing for demonstrating the position of a coordinate system, a camera, and a screen. It is explanatory drawing for demonstrating the position of a mark, the position of a handle | steering-wheel, and direction. It is explanatory drawing for demonstrating the vector acquisition position with respect to a handle | steering-wheel. It is explanatory drawing for demonstrating a handle top slide operation. It is explanatory drawing for demonstrating the detection result of a relative motion calculation part. It is explanatory drawing for demonstrating the mark attached on the handle circle. It is explanatory drawing for demonstrating the radius acquisition of a handle circle. It is explanatory drawing for demonstrating the operation example of palm operation. It is explanatory drawing for demonstrating the shadow of a handle | hand and a palm. It is explanatory drawing for demonstrating the handle movement by a palm. It is explanatory drawing for demonstrating the handle movement by a palm. It is explanatory drawing for demonstrating the handle movement by a palm. It is explanatory drawing for demonstrating the state before correct | amending in a directly upward direction. It is explanatory drawing for demonstrating the state after correct | amending in a directly upward direction.

  Hereinafter, embodiments will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a schematic system configuration diagram for explaining a first embodiment related to a driving support device. In this embodiment, a driver slides his / her hand on the handle and reads the sliding motion, thereby enabling operation of a device to be operated such as car navigation without operating buttons or a touch panel. is there.

  In FIG. 1, the driving support device displays an operation start determination unit 11 that determines whether the driver 10 has started a sliding operation of the steering wheel, an operation processing unit 12 that processes the sliding operation, a window raising / lowering or a navigation display based on the sliding operation. A drive unit 13 that performs cursor position movement of the device, brightness adjustment and position adjustment of the head-up display, and an operation end determination unit 14 that determines the end of device operation by the slide are provided.

  The operation processing unit 12 includes a hand movement measurement unit 121 that measures the movement of the hand on the handle, a handle position measurement unit 122 that measures the movement of the handle, and the movement of the hand on the handle moves relative to the handle. A relative motion calculation unit 123 for determining this, and an operation determination unit 124 for determining what slide operation has been performed.

  As a result, the driver does not need to look at the device to be operated, does not need to take his hand off the handle, and allows an analog operation in which the operation amount can be felt by hand. Since the slide operation is simple, an operation start determination unit 11 and an operation end determination unit 14 are provided to prevent malfunction. The basic operation is shown in the flowchart of FIG.

<Operation start determination unit 11>
The operation start determination unit 11 determines whether the operation of the operated device is started by the sliding operation of the handle. For example, when the driver utters “operating XX”, the voice recognition unit 21 recognizes that the slide operation for the operated device: XX is started.

  As a means for determining the start of the slide operation, in addition to the determination of the start of the slide operation by voice, an image of a meter or the like is taken from the handle, for example, a camera 22 installed in front of the driver is used to raise or lower a finger, gesture or mouth movement The start of the slide operation may be determined by reading. In addition, the operation may be re-confirmed by the display unit 23 such as a head-up display or the voice guidance unit 24, and the operation may be started for the first time when the driver re-operates by voice or gesture. . By doing so, it is possible to prevent erroneous operation.

<About the operation processing unit 12>
The operation processing unit 12 reads the movement of the hand on the handle from the start of the slide operation until the end of the operation, and determines whether the operation is to operate the operated device: XX. . The operation processing unit 12 includes a hand motion measurement unit 121, a handle position measurement unit 122, a relative motion calculation unit 123, and an operation determination unit 124.

  Hereinafter, each unit of the operation processing unit 12 will be described.

<About the manual motion measurement unit 121>
When the hand motion measuring unit 121 reads the relative motion directly with respect to the position of the handle, the handle position measuring unit 122 and the relative motion calculating unit 123 are not necessary. As a method, there are a method of reading the movement of the hand using the touch pad 25 on the handle, and a method of reading the movement of the hand by attaching a camera to the inside of the rotating object of the handle or the wheel of the handle. In that case, all of the wheels of the handle may be the touch pad 25, or a plurality of cameras may be installed inside the rotating portion of the handle or the wheel of the handle.

  When reading an absolute hand motion, a handle position measurement unit 122 and a relative motion calculation unit 123 are required. As a method therefor, there is a method of reading the movement of the hand by attaching a camera to the handle. In this case, a plurality of cameras may be installed facing the handle so that the operation can be recognized no matter where the handle is held.

  As a method of reading the movement of the hand with the camera, image processing such as a known gradient method or block matching is performed.

<About the handle position measurement unit 122>
In order to read the movement of the hand on the handle relative to the handle, it is necessary to read the absolute movement of the handle in addition to measuring the absolute movement of the hand by the hand movement measuring unit 121. In order to read the movement of the handle, for example, the angle information by the in-vehicle LAN 26 such as CAN or the shape of the handle is read by the camera 22 and the movement is measured.

  Alternatively, in order to cancel the vibration component of the vehicle body 15, a mark is attached to a portion that does not rotate due to the rotation of the handle HD, and in order to cancel the rotation component of the handle, a plurality of marks are attached to the portion that rotates the handle, and the position is read. The handle position may be measured. By doing so, it is possible to read the angle deviation from the camera due to the vibration of the vehicle body 15 and the rotation angle of the handle.

  Specifically, in order to cancel the vibration component, a mark M1 indicating a fixed position for canceling the vibration as shown in FIG. 3 and a mark M2 for measuring the rotation angle of the handle HD are respectively installed in the vehicle body. Hereinafter, cancellation of vibration components will be described based on the attachment positions of the marks M1 and M2 and the marks M1 and M2.

  First, the mark M1 for canceling the vibration component will be described. As shown in FIG. 4, the mark M1 is attached to a position where the camera 22 is not moved before the position of the camera 22 and the rotating portion of the handle HD or a characteristic point is found. The motion of the mark M1 is read relative to the camera 22 by a known image processing such as a gradient method or block matching. The positional deviation seen from the camera 22 due to vibration changes only in the direction in which the camera 22 faces, and the relative positional deviation between the camera 22 and the mark M1 does not occur, and the angular deviation of the camera 22 is calculated.

  When the focal distance to the acquired image position as shown in FIG. 5 is f, the position of the mark M1 at the acquired image position is (x1, y1, f), and the displacement position due to vibration is (x2, y2, f). When the coefficient of angular deviation is (Rx, Ry, Rz) and the angular deviation is very small, it is expressed as follows when the calculation is linearly approximated.

x2−x1 = − (x1 · y1 / f) · Rx + (f · f + x1 · x1) · (1 / f) · Ry−y1 · Rz
y2−y1 = − (f · f + y1 · y1) · (1 / f) · Rx + (x1 · y1 / f) · Ry + x1 · Rz
The above requires two marks, but in order to simplify the calculation, if the angle deviation of the z-axis, which is the direction to the screen, does not occur, the calculation is as follows. Can do.

x2−x1 = − (x1 · y1 / f) · Rx + (f · f + x1 · x1) · (1 / f) · Ry
y2−y1 = − (f · f + y1 · y1) · (1 / f) · Rx + (x1 · y1 / f) · Ry
The influence of vibration is canceled from the above calculation results.

  Next, the mark M2 for canceling the rotation component of the handle will be described. There are a method of acquiring the rotation angle of the handle HD from the in-vehicle LAN 26 and a method of acquiring from the video of the camera 22. When acquiring from the camera image, the mark M2 is attached to the rotating handle (not limited to the gripping position), and the operation of the mark is made relatively to the camera 22 by a known gradient method or block matching. read. In order to make the mark M2 visible even when the handle HD is rotated, the plurality of marks are attached on the rotating handle (not limited to the gripping position). The position of the mark M2 after canceling the influence of the vibration is assumed to be (x2, y2, f).

Next, in order to identify how the handle HD moves on the screen, the rotation amount of the handle HD is measured from the position of the mark M2. When the center position of the mark M2 by rotation of the handle is (Mpx, Mpy, Mpz) and the axis direction of the handle HD (perpendicular to the handle surface) is (Hdx, Hdy, Hdz), the formula of the surface is
Hdx · (x−Mpx) + Hdy · (y−Mpy) + Hdz · (z−Mpz) = 0
Since the vector to the position (Ex, Ey, Ez) with respect to the camera on the surface formed by rotating the handle of the mark is (x2, y2, f),
Ex = x2 · t
Ey = y2 · t
Ez = f · t
Then, the value is substituted into the formula of the surface formed by the handle rotation of the mark M2, and t is derived to obtain (Ex, Ey, Ez).

Next, in order to convert the plane of the handle circle into a plane of z = 0 with the center of the handle circle being (0, 0, 0), (Ex, Ey, Ez) is subtracted at the center of the handle (x3, y3 , Z3), x-axis rotation (x4, y4, z4), and y-axis rotation (x5, y5, z5). If the rotation angle of the x-axis rotation is Rhx and the rotation angle of the y-axis is Rhy,
x3 = Ex-Mpx
y3 = Ey-Mpy
z3 = Ez-Mpz
x4 = x3
y4 = y3.cos (Rhx) -z3.sin (Rhx)
z4 = z3 · cos (Rhx) + y3 · sin (Rhx)
x5 = x4 · cos (Rhy) + z4 · sin (Rhy)
y5 = y4
z5 = z4 · cos (Rhy) −x4 · sin (Rhy)
Thus, the respective rotation angles here are as follows. Note that the handle circle is a circle drawn at the center of the handle HD regardless of the thickness of the handle HD.

Rhx = atan (Hdy / Hdz)
Rhy = −atan (Hdx / (Hdz · cos (Rhx) + Hdy · sin (Rhx)))
Next, the z-axis rotation angle position of the mark M2 is acquired, and the value obtained by subtracting the angle when the mark M2 is not rotating becomes the rotation angle (Rhz) of the handle at that time.

  The rotation angle of the handle by the above conversion may be acquired by preparing a Look Up Table or a projection conversion formula for each mark position in advance.

<Regarding the relative motion calculation unit 123>
The relative motion calculation unit 123 detects the absolute hand motion acquired by the manual motion measurement unit 121 from the hand motion measurement unit 121 as a plurality of vectors A from the image. From the absolute handle position obtained by the handle position measuring unit 122 for the vector A group, the relative movement of the hand with respect to the handle is converted as a vector H group. One vector H is (Hx, Hy, Hu, Hv) (= (x position on the screen, y position on the screen, x-direction movement amount in one frame, y-direction movement amount in one frame)) 4 Consists of two parameters.

  Although a plurality of hand motion vectors that can be acquired on the screen can be acquired and the plurality of vectors are used, the conversion calculation will be described using only one vector.

One absolute hand motion vector A acquired on the screen is expressed as (Ax, Ay, Au, Av) (= (x position on the screen, y position on the screen, x direction movement amount in one frame, 1 (Y direction movement amount in the frame)) and the vector B canceled the influence of the vibration of the vehicle body is (Bx, By, Bu, Bv) (= (x position on the screen, y position on the screen, 1 frame) x-direction movement amount, y-direction movement amount in one frame)), and the current rotation amount of vibration (R1x, R1y, R1z) (= (x-axis rotation amount, y-axis rotation amount, z-axis rotation amount)) When the rotation amount one frame before is (R0x, R0y, R0z), Bx and By are
Bx = Ax − (− (Ax · Ay / f) · R1x + (f · f + Ax · Ax) · (1 / f) · R1y−Ay · R1z)
By = Ay − (− (f · f + Ay · Ay) · (1 / f) · R1x + (Ax · Ay / f) · R1y + Ax · R1z)
The position (Cx, Cy) before canceling the influence of vibration in the previous frame is
Cx = Ax-Au
Cy = Ay-Av
So, the position (Dx, Dy) where the influence of vibration in the previous frame was canceled is
Dx = Cx − (− (Cx · Cy / f) · R0x + (f · f + Cx · Cx) · (1 / f) · R0y−Cy · R0z)
Dy = Cy-(-(f.f + Cy.Cy). (1 / f) .R0x + (Cx.Cy/f) .R0y + Cx.R0z)
So (Bu, Bv) is
Bu = Bx-Dx
Bv = By-Dy
It becomes.

Next, the movement of the handle is specified on the screen. When the handle rotation center position is (Hpx, Hpy, Hpz) and the handle axis direction (perpendicular to the handle surface) is (Hdx, Hdy, Hdz), the formula of the surface is
Hdx · (x−Hpx) + Hdy · (y−Hpy) + Hdz · (z−Hpz) = 0
Thus, the vector amount to the position (Ex, Ey, Ez) with respect to the camera on the handle surface of the vector (Bx, By, Bu, Bv) that has canceled the influence due to the vibration of the vehicle body is (Bx, By, f). So
Ex = Bx · t
Ey = By · t
Ez = f · t
Then, by substituting that value into the formula of the handle surface, t is derived, and (Ex, Ey, Ez) is obtained.

Then, in order to convert the handle surface into a plane with z = 0 at the center of the handle circle at (0, 0, 0), subtract (Ex, Ey, Ez) at the center of the handle (x3, y3, z3) The x-axis is rotated (x4, y4, z4), and the y-axis is rotated (x5, y5, z5). If the rotation angle of the x-axis rotation is Rhx and the rotation angle of the y-axis is Rhy,
x3 = Ex-Hpx
y3 = Ey-Hpy
z3 = Ez-Hpz
x4 = x3
y4 = y3.cos (Rhx) -z3.sin (Rhx)
z4 = z3 · cos (Rhx) + y3 · sin (Rhx)
x5 = x4 · cos (Rhy) + z4 · sin (Rhy)
y5 = y4
z5 = z4 · cos (Rhy) −x4 · sin (Rhy)
The rotation angles of the x-axis and the y-axis here are as follows.

Rhx = atan (Hdy / Hdz)
Rhy = −atan (Hdx / (Hdz · cos (Rhx) + Hdy · sin (Rhx)))
When the current rotation angle of the handle HD is Rh1z and the rotation angle of the handle one frame before is Rh0z, the handle rotation angle Rhz in one frame is
Rhdz = Rh1z-Rh0z
Then, the coordinate of the vector set on the plane of z = 0 is rotated by z axis by −Rhdz (x6, y6, z6), and the coordinate one frame before is specified.

x6 = x5.cos (-Rhdz) -y5.sin (-Rhdz)
y6 = y5.cos (-Rhdz) + x5.sin (-Rhdz)
z6 = z5
The value is converted in reverse to z = 0. That is, it rotates by y-axis by −Rhy, rotates by x-axis by −Rhx, and adds the handle rotation center position (Hpx, Hpy, Hpz). If the value is (Fx, Fy, Fz) and the coordinates developed on the surface where the vector is detected are (Gx, Gy),
Gx = f · Fx / Fz
Gy = f · Fy / Fz
The vector H (Hx, Hy, Hu, Hv) in which the rotational component of the handle is canceled is
Hx = Bx
Hy = By
Hu = Bu− (Bx−Gx)
Hv = Bv- (By-Gy)
This vector H (Hx, Hy, Hu, Hv) is used.

  However, it is still difficult to handle as it is. Therefore, as shown by the thick lines in FIGS. 6B and 6C, the handle circle is straightened and processed. As described above, the vertex of the vector and the position one frame before the vector are converted into the tertiary position of the handle surface, the center coordinate of the handle is subtracted, the Rhx portion is rotated by the x axis, and the Rhy portion is rotated by the y axis. The radius and the z-axis rotation angle θ are acquired. The absolute amount Rhz of the rotation angle of the handle is canceled at the z-axis rotation angle θ, and the position of the hand relative to the position of the handle is obtained.

  Then, the radius is converted as the y axis and the angle as the x axis. Next, if the shape of the handle is an ellipse, an offset corresponding to the radius deviation of the handle is added at each z-axis rotation angle to convert the handle shape to look straight, so that the handle position becomes zero. To do. The above conversion makes the operation determination easy to understand. Assuming that the vector I at this time is (Ix, Iy, Iu, Iv), the meanings thereof are as follows, and this vector is sent to the operation determination unit 124 to determine the operation.

Ix: Position of handle wheel direction (angle with respect to the center of the wheel) Iy: Position of handle wheel direction with respect to the center of the wheel (0 is the position of the wheel of the handle)
Iu: Amount of movement in the direction of the wheel of the handle (angle with respect to the center of the wheel) Iv: Amount of movement in the direction of the handle in the direction of the center of the wheel Prepare a Look Up Table of radius and z-axis rotation angle, expand the vector by linear interpolation, etc., shift it by the rotation angle of the handle, and convert it by adding an offset if the handle shape is an ellipse You may let them.

  Alternatively, a Look Up Table may be prepared in consideration of the case where the handle has an elliptical shape for each z-axis rotation angle, an image may be developed with the Look Up Table, and a vector may be detected from the image. In that case, it develops with Look Up Table from the position where the vibration component was canceled from the original image.

<About Operation Determination Unit 124>
In the operation determination unit 124, a plurality of vectors I are obtained from the relative motion calculation unit 123. The content of the slide operation is determined from the vector I group, and an analog operation amount to the operated device according to the content, Outputs the direction or operation command code. This output is specified by the operated device specified by voice or the like.

  As shown in FIG. 7, a case where an analog movement operation command in the vertical direction and an analog movement operation command in the horizontal direction on the display unit 23 are performed will be described. The vertical operation of the display unit 23 is a direction 7a in which the handle is twisted like an accelerator of a motorcycle, and the horizontal operation of the display unit 23 is a direction 7b in which the handle 23 is moved in the direction of the wheel of the handle. In such a case, the detection result of the vector I group in the relative motion calculation unit 123 is as shown in FIG.

  First, specify the position of the hand on the handle. Since the ellipse of the wheel of the handle is expanded into a straight line with y = 0, if there are many vectors of a certain size in the region corresponding to the thickness of the wheel (near y = 0), the range of x The vector distribution is measured with a1 <x <a2. By doing so, the operation vector on the handle is separated from the non-operation vector. Moreover, you may identify with the color of a hand by color information.

  In the vertical operation, those at the same y position Iy have the same movement amount Iv in the y direction, which is the largest at the position of the handle (y = 0) and small at the top and bottom (y ≠ 0). The movement amount of the operation is calculated by the vector amount in the y direction at the handle position (y = 0). For example, the range near the wheel of the steering wheel is Range1 (-b1 <y <b1), the range outside the wheel of the steering wheel is Range2 (b1 <y <b2), and the range inside the wheel of the steering wheel is Range3 (-b2 < Let y <−b1). First, vectors whose vector magnitude Iv is almost zero are removed. If the remaining number is small, it is determined that the slide operation is not performed. Then, an average J1v of Iv of vectors in Range1 is obtained. If the variance of Iv in Range 1 (for example, a ratio obtained by dividing the number of deviations from the average (| Iv−J1v |) greater than a certain threshold value by the total number) is large, it is determined that the operation is not a slide operation because of variation. Similarly, the average J2v and variance of Range2 are calculated, and if the variance is large, it is determined that the operation is not a slide operation because of variation.

  Similarly, the average J3v and the variance of Range3 are calculated. If the variance is large, it is determined that the operation is not a slide operation because the variation is present. Next, when the three averages (J1v, J2v, and J3v) are not the same polarity, it is determined that the slide operation is not performed.

  Next, when the ratio of the size (| J1v |) in Range1 to the size (| J2v | and | J3v |) in other ranges is large in a certain range ((c1 <| J1v | / | J2v | <c2) and (c1 <| J1v | / | J3v | <c2), where c1 and c2 are 1 or more). The movement amount of the vertical operation is calculated by a value obtained by multiplying J1v by a certain coefficient.

  In the horizontal operation, all the vectors at the position where the hand is located have almost the same movement amount Iu in the x direction, and the movement amount of the operation is calculated by the vector amount in the x direction at the handle position (y = 0). . For example, first, a vector whose vector size Iu is almost zero is removed. If the remaining number is small, it is determined that the slide operation is not performed. Then, an average J1u of Iu of vectors in Range1 is obtained. When the dispersion of Iu in Range 1 (for example, a ratio obtained by dividing the number of deviations from the average (| Iu−J1u |) larger than a certain threshold value by the total number) is large, it is determined that the operation is not a slide operation because of variation. Similarly, the average J2u and variance of Range2 are calculated, and if the variance is large, it is determined that the operation is not a slide operation because of variation. Similarly, the average J3u and the variance of Range3 are calculated. If the variance is large, it is determined that the operation is not a slide operation because the variation is present.

  Next, when the three averages J1u, J2u, and J3u are not of the same polarity, it is determined that the operation is not a slide operation. Next, when the size | J1u | in the range 1 is within a certain range with respect to the sizes | J2u | and | J3u | in the other ranges, ((d1 <| J1u | / | J2u | <d2 ) And (d1 <| J1u | / | J3u | <d2), where d1 is 1 or less and d2 is 1 or more. The movement amount of the lateral operation is calculated by a value obtained by multiplying J1u by a certain coefficient. When the camera is in the reverse direction with respect to the driver, the horizontal operation is in the opposite direction.

  When the operation here is an analog movement amount and direction, the movement amount and direction are a combination of the vertical operation and the horizontal operation. For example, the movement amount and direction of the operation are converted into the movement amount and direction of the cursor on the display unit 23.

  In the case where the operation here is an operation command, the operation command is used when the cumulative movement amount in a plurality of frames exceeds the specified threshold value instead of the movement amount between one frame. For example, when the cumulative amount of vertical operation exceeds a specified threshold, the display on the display unit 23 is switched to a different image.

  If the operation range is insufficient, release the hand once to return the position, and grasp again to operate. By not operating when the hand is away from the handle, sliding can be performed indefinitely.

  Various operation method examples of this embodiment will be described with reference to (1) to (12).

(1) Not only an analog operation but also a vertical operation can be quickly performed up and down to make a Yes (decision, selection) determination, and a horizontal operation can be quickly performed to the left and right to make a No (cancel) determination.
(2) When it is assumed that the hand is sticky and does not slip on the handle, the twisting and movement in the direction of the ring are as small as the flesh of the hand is deformed. In that case (even when sliding), the movement speed of the operation may be operated based on the displacement amount of the hand from the reference position with the position of the first hand as a reference. Further, the lateral direction may be determined by the action of twisting the wrist of the gripped hand. Alternatively, the movement of the hand and the movement of returning it, such as up and down and left and right, may be a series of operations, and the direction of movement first may be the operation movement direction.
(3) When the handle is squeezed strongly (the size of the vector is small, the orientation is different (the direction of the center of the hand), and immediately returns to the original position after movement (image at the end of movement and at the start of movement) (The vector in the image of) does not come out)), and when the wrist is twisted (there are many vectors that are somewhat large, the orientation is scattered (the direction of rotation around the center of the hand), and immediately returns to its original position after movement) , Or when you twist your wrist quickly or when you open your hand lightly (there are many vectors that are somewhat large and their orientation is different (the direction away from the center of the hand) and immediately return to their original position after movement) You can cancel it.
(4) The determination may be made when the handle is strongly gripped on the thumb side, and may be canceled if the handle is strongly gripped on the little finger side. In addition, the operation may be changed for each finger that is strongly grasped.
(5) Using both hands, the operation may be extended as a rotation operation when each hand is upside down, a reduction operation when each hand is brought closer, and an enlargement operation when each hand is moved away.
(6) The operation target may be changed with the finger raised and operated. In addition, when the finger is raised, what the operation target is may be notified by voice. In this case, the operation target may be raised by raising the finger with one hand, and the analog operation may be performed with the other hand.
(7) Different operating systems may be used for the left and right hands.
(8) It is good also as operation by the relative distance on the basis of the position of one hand.
(9) Up and down and left and right with the right hand, and depth and twist directions (rotation of the axis in the depth direction) and time direction with the left hand, etc., the left and right hands may be used for three-dimensional and four-dimensional operations.
(10) With the position of the hand holding the handle or the position of the hand when the operation is started as a reference, the speed of movement operated by the relative distance from the position may be changed.
(11) The operating system may be changed by first grasping a predetermined position on the handle.
(12) The size of the hand may be detected, and the threshold of the cursor or movement or the amount of movement may be normalized according to the size of the hand. By doing so, the shift of the operation interval by a person is reduced.

<About Drive Unit 13>
The drive unit 13 is a device to be operated. For example, when operated, the window is raised and lowered via the in-vehicle LAN 26, the cursor position of a display device such as a navigation is moved, the brightness adjustment and the position adjustment of the head-up display are performed.

  In order to inform the driver of whether or not the operation has been performed, voice guidance may be provided. For example, in an analog operation, a sound is generated once for a predetermined amount of movement, and when the movement is several times that amount, the current state may be guided by voice. For example, when the in-vehicle temperature is operated, a sound of “pong” is sounded every time, and when the resolution is 5 degrees, guidance is made by voice such as “15 degrees”. Further, the sound quality of the guided sound may be changed depending on the operating direction (whether the temperature is raised or lowered). Moreover, what is necessary is just to alert | report that it operated by the voice, if it was only operated.

  The device to be operated may be a volume of music, a music selection list, navigation operation, an in-car environment such as a wiper speed and window opening / closing, or a portable device such as a smartphone. Moreover, you may perform operation outside vehicles, such as a house and a company, via the internet. Further, the driver may operate different states depending on the driving situation such as switching of the display of the camera attached to the vehicle and switching of the viewpoint of the overhead image.

<About calibration>
In order to realize the above operation, the rotation center position (Mpx, Mpy, Mpz) of the mark M2 on the handle HD and the angle (Rhz) when the handle HD is not rotated, the vertical direction (Hdx, Hdy, Hdz), the rotation center position (Hpx, Hpy, Hpz) of the handle surface, and the radius for each angle of the handle HD must be known. Therefore, in order to acquire each position, it is necessary to calibrate first. Although the mounting position and angle of the camera 22, the distance to the mark, the position and orientation of the handle may be actually measured, a method of simple calibration will be described here.

First, as shown in FIGS. 9 and 3, the mark M3 is attached on the handle HD, and the attachment position of the mark M3 is acquired in a state where the handle HD is not rotated. The state in which the handle HD is not rotated (there is no depression angle) may be determined by visual observation, but may be determined that the vehicle HD is not rotated in order to make the in-vehicle LAN 26. Using the position as a reference, the handle HD is rotated to obtain a plurality of attachment positions of the mark M3. If the attachment position of the mark M3 on the image is (M3x, M3y, f), and the position on the three-dimensional space is (Hmx, Hmy, Hmz),
Hmx = M3x · t
Hmy = M3y · t
Hmz = f · t
x3 = Hmx−Hpx
y3 = Hmy-Hpy
z3 = Hmz-Hpz
x4 = x3
y4 = y3.cos (Rhx) -z3.sin (Rhx)
z4 = z3 · cos (Rhx) + y3 · sin (Rhx)
x5 = x4 · cos (Rhy) + z4 · sin (Rhy)
y5 = y4
z5 = z4 · cos (Rhy) −x4 · sin (Rhy)
It becomes. Each rotation angle here is as follows.

Rhx = atan (Hdy / Hdz)
Rhy = −atan (Hdx / (Hdz · cos (Rhx) + Hdy · sin (Rhx)))
If Sqrt () is a √ (root) operation,
cos (Rhx) = Hdz / sqrt (Hdy · Hdy + Hdz · Hdz)
sin (Rhx) = Hdy / sqrt (Hdy · Hdy + Hdz · Hdz)
Substituting
Rhy = −atan (Hdx / sqrt (Hdy · Hdy + Hdz · Hdz))
cos (Rhy) = sqrt (Hdy · Hdy + Hdz · Hdz) / sqrt (Hdx · Hdx + Hdy · Hdy + Hdz · Hdz)
sin (Rhy) = − Hdx / sqrt (Hdx · Hdx + Hdy · Hdy + Hdz · Hdz)
It becomes. Since it is a point on the handle, the following conditions are satisfied.

z5 = 0
r.r = x5.x5 + y5.y5 (r (radius) is constant)
Hdx · (Hmx−Hpx) + Hdy · (Hmy−Hpy) + Hdz · (Hmz−Hpz) = 0 (calculate t)
Since the size of the object seen from the image is relative, the radius r of the rotation of the mark M3 on the handle HD and the z coordinate Hpz of the rotation center of the handle surface are calculated as constants (for example, 1), and the handle that satisfies the condition is calculated. The vertical direction (Hdx, Hdy, Hdz) of the surface and the rotation center position (Hpx, Hpy, Hpz) of the handle surface are derived.

  Next, as shown in FIG. 10, in a state where the handle is not rotated (a), x = at the conversion position where the handle surface is z = 0 and the rotation center of the handle surface is (0, 0, 0). Two points P1 and P2 close to the handle circle on the straight line of 0 and z = 0 are selected and converted to positions Q1 and Q2 on the image. By reading the position of the handle on the straight line formed by the two points on the image, the handle position can be canceled. As a method of reading the handle position, for example, first, a sheet 101 of a color different from that of the handle HD, such as green, is arranged so as to be behind the handle HD when viewed from the camera 22. Note that this is not particularly necessary when it is possible to determine without arranging the sheet 101.

  Then, the angle θ0 of the mark M3 on the handle circle is derived while the handle HD is not rotating. Next, the inner and outer positions Q3 and Q4 of the wheel of the handle HD on the straight lines Q1 and Q2 on the screen are read, and the middle position Q5 is calculated. The position P5 when the position is converted to the plane of z = 0 becomes the radius y5 of the handle, and the angle at that time becomes θ = 0. If the shape of the handle ring is not a perfect circle, only the intersection between the outside and the straight line of the handle ring is read from the next, and only the distance between the outside and the center of the handle ring acquired when the handle is not rotated The shifted position is used as the center of the handle wheel. The reason for doing so is that the boundary cannot be found when there is a portion that supports the wheel of the handle heading from the wheel of the handle toward the handle rotation center. Then, transformation P6 is performed so that the surface becomes z = 0, and the handle angle y5 and the angle θ1 of the mark on the handle circle canceled by the angle θ0 acquired when the handle is not rotated are set to θ. . Plot multiple of them to create a radius curve.

  Next, the rotation center position (Mpx, Mpy, Mpz) of the mark M2 on the rotating object of the handle and the angle (Rhz) when the handle is not rotated are also the same calculations as those for deriving the mark M3 on the handle circle. Derived by.

  The necessary parameters are obtained first by the above method. Since the mark M3 on the handle circle is placed on the handle circle touched by the driver, the mark M3 is removed when the calibration is completed.

<About the operation completion determination unit 14>
The determination of the end of the slide operation by the operation end determination unit 14 is performed, for example, when the voice recognition unit 21 recognizes that the driver has issued “end of operation”.

  In addition to the voice recognition by the voice recognition unit 21, the means for determining the end of the operation is determined when it is determined that the hand that is not the hand operated by the camera 22 or the touch pad 25 has moved away from the handle, or by the in-vehicle LAN 26. Even when the brake is depressed, the accelerator is depressed, the handle is suddenly turned, the handle is turned a lot, the gesture is completely different, or the voice is completely different It may be determined to end.

  Further, an operation time may be provided, and the operation end determination may be automatically made when the time has passed. Further, when the operation is performed with the cursor displayed on the display unit 23 or the like, the operation may be forcibly terminated because it is dangerous to look at the display unit 23 while keeping the eye line away from the traveling direction for a certain period of time. Further, the operation may be ended when the line of sight is moved to a place that is not related to the operated device and the traveling direction. The operation may be started only when a finger of a specific hand is raised, and the operation may be ended when the finger is lowered. Further, the operation may be ended when the operation is permitted only when the vehicle is stationary, when the vehicle moves, or when the brake is not stepped on. Further, the operation may be ended when the eyes are widened, the mouth is opened wide, or the expression such as a surprised expression is changed.

  Thus, since the position is not specified on the handle HD, it is not necessary to remove the line of sight, and it is safe because the hand is not released from the handle HD. Since it can be operated at a position where the handle HD is gripped, it can be operated with a minimum of hand movement. Further, since two-dimensional analog operation is possible, it can be said that there is a degree of freedom of operation. There is no need to reach for anything that is to be operated even when stopped.

  In this embodiment, it is possible to operate while safely driving without releasing the hand from the handle and keeping the line of sight away from the outside of the vehicle. In addition, the operation amount can be felt by hand. Also, malfunctions can be prevented by designating the start and end of the operation.

(Second Embodiment)
FIGS. 11-17 shows 2nd Embodiment regarding a parking assistance apparatus, respectively. In the first embodiment, the operation is performed in a state where the handle is held. In this embodiment, the operation is performed using the palm of the hand by spreading the finger without holding the handle.

  When the palm is in contact with the outside or inside of the wheel of the handle, a depth operation to move the hand to the back or to the front as shown by the dashed circle (a) in FIG. 11 and a side operation to move in the direction of the wheel of the handle are possible. It is. Further, a twisting operation in the direction of rotating the palm with respect to the center of the palm is also possible.

  As shown by a broken line circle (b) in FIG. 11, when the palm is brought into contact with the front side of the handle, a vertical operation for moving the hand up and down and a horizontal operation for moving the hand left and right are possible. Further, a twisting operation in the direction of rotating the palm with respect to the center of the palm is also possible.

  As described above, in this embodiment, it is possible to operate more intuitively by touching only the palm of the hand to the handle, and by maintaining the condition that the palm of the hand touches the handle, safe driving can be maintained. can do. Hereinafter, a different part from 1st Embodiment is demonstrated.

<Regarding the relative motion calculation unit 123>
In the relative motion calculation unit 123 in this embodiment, the rotation center of the handle surface is set to (0, 0, 0), and the hand motion with the handle surface set to z = 0 is used. At that time, the handle circle rotates the handle by the amount of rotation in one frame.

<About Operation Determination Unit 124>
In the operation determination unit 124 in this embodiment, first, whether the hand is in contact with the handle is determined by using the handle HD itself as a touch pad as shown in FIG. It is determined whether the hand side boundary becomes dark to some extent at the outer or inner boundaries 12a and 12b.

  This is because when the hand is away from the handle, it does not darken at the border on the hand side. The position of the camera may be shifted or a plurality of cameras may be attached so that the shadow can be easily captured. A light may be attached. When acquiring the shadow, as described in the first embodiment, it may be acquired after converting the curve of the wheel of the handle to the side, or may be acquired at the position of the handle in the original image. Good.

  When the palm is grounded to the outside of the wheel of the handle, as shown in FIGS. 13 to 15, the obtained vector is larger outward in the direction from the center of the wheel of the handle to the outside as shown in FIG. In this case, it is determined that the hand has been moved to the back. On the contrary, when the obtained vector is larger in the direction from the outside toward the vicinity of the center of the wheel of the handle as shown in FIG. 14, it is determined that the hand is moved forward.

In the case of only the movement in the depth direction as shown in FIG. 15, in the case of the operation direction based on the traveling direction of the car, at the position where the right rear direction of the own vehicle on the camera image is converted to the plane of z = 0. In the case of the operation direction with reference to the wheel surface of the handle, the convergence point is (FOEHx, FOEHy). When the convergence point (FOEWx, FOEWy) directly behind the host vehicle is a vector (VCx, VCy, VCz) from the camera in the direction behind the host vehicle, and the convergence point on the screen is (VCfx, VCfy, f),
VCfx = f · VCx / VCz
VCfy = f · VCy / VCz
It becomes. As in the first embodiment, the convergence point on the screen is set to (0, 0, 0) as the rotation center of the handle surface, the handle surface is set to z = 0, and the handle is rotated by the amount of rotation in one frame. . The x and y coordinates at that time are (FOEWx, FOEWy).

The vertical convergence point (FOEHx, FOEHy) with respect to the wheel surface of the handle is the convergence point (VHfx) on the screen because the vertical vector of the handle surface is (Hdx, Hdy, Hdz). , VHfy, f)
VHfx = f · Hdx / Hdz
VHfy = f · Hdy / Hdz
It becomes. As in the first embodiment, the convergence point on the screen is set to (0, 0, 0) as the rotation center of the handle surface, the handle surface is set to z = 0, and the handle is rotated by the amount of rotation in one frame. . The x coordinate and y coordinate at that time are (FOEHx, FOEHy).

Next, the operation direction is derived. If the vertical and horizontal movements are operated simultaneously in addition to the depth direction, the convergence point is shifted. Therefore, first, the operation direction (Tx, Ty, Tz) with respect to the screen is derived by the least squares solution using the following calculation formula. One of the obtained vector I groups is (Ix, Iy, Iu, Iv) (= (x position on the screen, y position on the screen, x direction movement amount in one frame, y direction movement amount in one frame) ))
Hf · Tx−Ix · Tz = Iu · Hf
Hf · Ty−Iy · Tz = Iv · Hf
Hf = sqrt (Hpx · Hpx + Hpy · Hpy + Hpz · Hpz) (Hf is the distance to the center of the wheel ring)
If the variance is large at this time, the operation is not performed.

When the obtained operation direction (Tx, Ty, Tz) is an operation direction based on the traveling direction of the vehicle, it is necessary to rotate the operation direction by a convergence point in the direction directly behind the vehicle. If the operation direction after rotation is (TAx, TAy, TAz), x3 = Tx
y3 = Ty
z3 = Tz
x4 = x3
y4 = y3 · cos (Rfx) −z3 · sin (Rfx)
z4 = z3 · cos (Rfx) + y3 · sin (Rfx)
TAx = x4 · cos (Rfy) + z4 · sin (Rfy)
TAy = y4
TAz = z4 · cos (Rfy) −x4 · sin (Rfy)
The respective rotation angles here are as follows.

Rfx = atan (FOEWy / Hf)
Rfy = −atan (FOEWx / (Hf · cos (Rfx) + FOEWy · sin (Rfx)))
In the case of the operation direction based on the wheel surface of the handle, the calculation may be performed by replacing (FOEWx, FOEWy) with (FOEHx, FOEHy). Since the operation direction (TAx, TAy, TAz) after the rotation is the direction from the camera, it becomes (−TAx, TAy, −TAz) when the camera is in the reverse direction with respect to the driver.

  When the horizontal direction is the horizontal direction with respect to the ground, as shown in FIG. 16, the operation direction is rotated by the z axis so that the vertical direction is the upward direction with respect to the ground. The vertical direction to the ground is obtained when the position directly above the rotation center of the wheel of the handle on the camera image is not rotated on the plane of the wheel of the handle or when it is not rotated on the plane of the wheel of the handle. This is the direction of the point when one point is selected and the vertex is converted so that it becomes a plane of z = 0.

  In addition, when the movement in a direction with a ring is set to the horizontal direction, as shown in FIG.

  Further, the operation direction may be calculated on the screen obtained without making the wheel of the handle a surface with z = 0. This is particularly effective when the camera position is not perpendicular to the handle surface from the center of the handle or when the camera is too close to the handle surface. In this case, in the case of the operation direction based on the traveling direction of the vehicle, the calculation may be performed by replacing (FOEWx, FOEWy, Hf) with the convergence point (VCfx, VCfy, f) on the screen. In the case of the operation direction based on the wheel surface of the handle, the calculation may be performed by replacing (FOEWx, FOEWy, Hf) with the convergence point (VHfx, VHfy, f) on the screen.

Various operation method examples of this embodiment will be described with reference to (1) to (4).
(1) When the vector around the wheel of the handle becomes larger with time and a shadow is formed on the boundary between the wheel of the handle and the hand, it may be determined that the handle has been hit and Yes may be determined.
(2) If the vector is larger toward the outer side of the wheel of the handle and the direction is the center of the wheel, and the vector is smaller in the vicinity of the wheel, it may be determined that the palm is tilted around the palm and Yes may be determined.
(3) If the vector that crosses the wheel of the handle in the horizontal direction is continuous in time and a shadow is once formed at the boundary between the wheel of the handle and the hand, and the shadow disappears, it is determined that the handle wheel is moved. , No (cancel) determination may be made.
(4) When the vector is distributed in the rotation direction around a part of the wheel of the handle, it may be judged that the hand is twisted around the palm and No (cancel) judgment may be made.

<About expansion mode>
Examples of expansion modes conceivable in the first and second embodiments will be described in the following (1) to (16).
(1) A camera may be installed on the rotating part of the handle to obtain an operation. In that case, in order to absorb the vibration, a mark is placed on the rotating part of the handle to absorb the vibration. Also, a mark is attached to a fixed portion that is not a rotating portion of the handle, and the rotation angle of the handle is acquired. By doing so, the operation acquisition range becomes narrow, but the operation can be acquired with high accuracy, and the amount of calculation is reduced. In this case, since the camera is installed on the rotating part, the mark attached to the fixed part that is not the rotating part of the handle is calculated as the rotating part.
(2) The detected vector is rotated or moved, but the image may be converted after canceling vibration and rotation at the acquired mark position, and the vector may be detected from the converted image.
(3) Since rotation and movement calculations are projective transformations, the following projection transformation formulas may be created by converting four points in advance, and the transformation formulas may be used for the transformation.
x = (A.x + B.y + C) / (G.x + H.y + 1)
y = (D.x + E.y + F) / (G.x + H.y + 1)
(4) Far infrared rays may be used so that the position of the hand can be easily detected. Further, only the red component of the visible light camera may be used.
(5) In order to be able to see all the wheel portions of the handle, the camera may be attached to a plurality of places, and the mark may be attached accordingly. By doing so, the operation can be performed anywhere on the handle.
(6) A characteristic mark such as a glove or a ring may be attached to the hand to increase detection accuracy.
(7) The movement of the hand may be detected more accurately by reading the tip of the toe, the base of the finger, the groove of the finger gap, the joint position of the finger, the wrinkle of the finger, and the like.
(8) Only a specific part of the handle (such as directly above or next to the handle) may be operated. By doing so, it can be distinguished whether it operated more or drove.
(9) A camera may be attached in the wheel of the handle, and the operation may be acquired by making the handle transparent.
(10) The operation may be determined based on temporal continuity and its variation without being determined by an operation between one frame.
(11) Processing may be performed using a camera such as a smartphone, voice recognition, or processing performance.
(12) If it is used only when the host vehicle is stopped, the process of absorbing the vibration component and the mark used at that time may be removed after calibration. Further, it is also possible to omit processing for canceling the rotation angle of the handle, assuming that the handle is not rotated. In that case, vibration is not absorbed, so a Look Up Table is created to convert the handle wheel so that it is directly in front, that is, the center is (0, 0, 0) and the handle surface is z = 0. A vector may be obtained from the image after conversion. If the shape of the handle is a perfect circle, the processing for obtaining the rotation angle of the handle and the mark used at that time may be removed after calibration. In that case, since the rotation angle of the handle is not required, the image may be converted with the Look Up Table extended from the beginning with the handle wheel extended to a line, and the vector may be obtained from the image.
(13) It may be time-limited when the process of confirming the mark and absorbing the vibration component is not performed in order to reduce the calculation. Further, it may be limited when the rotation angle of the handle is not rotating after confirming the mark, that is, when it is not necessary to calculate the relative movement amount.
(14) In order to prevent drowsiness or change mood during driving, the steering wheel may be regarded as a person or an animal, and the response of the steering wheel may be enjoyed by performing the described series of operations. For example, when the handle is regarded as a person's shoulder and the handle is moved firmly, a reaction that seems to be pleasant such as “feeling good” is made. In addition, it is assumed that the handle child is on the rotating car, and the left and right hands are twisted by moving them upside down, causing the child to react with a joyful voice. Alternatively, the child may be scolded with a rattling sound by twisting the wrist of the hand holding the handle. At that time, you may cry if you don't keep on ceasing. In addition, by making the handle look like an animal such as a cat and performing an operation of pointing the handle, a reaction that makes the cat or the like sound pleasantly is made.
(15) It may be used for bicycle or motorcycle grips.
(16) It may be used as an operation system instead of a personal computer or a game mouse or touchpad. In that case, it is good also as the use which pops out only when using it, storing the stick for grasping under the keyboard etc. In that case, the bar may be horizontal or vertical.

  In this embodiment, analog operation is possible with a simple hand movement, which is safe. In addition, it is safe because the handle can be gripped immediately. The twist direction is added and more multi-dimensional operation becomes possible.

  Although several embodiments have been described, these embodiments have been presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, combinations, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

11 Operation start determination unit 12 Operation processing unit 121 Manual motion measurement unit 122 Handle position measurement unit 123 Relative motion calculation unit 124 Operation determination unit 13 Drive unit 14 Operation end determination unit 21 Voice recognition unit 22 Camera 23 Display unit 24 Voice guide unit 25 Touchpad 26 in-house LAN
M1-M3 Mark HD handle

Claims (7)

An operation start determination unit that determines whether the operation of the operated device is started by the sliding operation of the handle;
An operation end determination unit for ending the slide operation started by the operation start determination unit;
Wherein Ri by the start of the operation start judgment part, the relative movement of the hand from the movement and position of the hand on the front Symbol handle on the handle to calculate the hand on the handle lateral manipulation of wheels, wheels An operation processing unit that processes determination of the operation content on the handle based on at least one of the longitudinal twisting operations ;
And a driving unit that drives the operated device based on the operation content determined by the operation processing unit. The operation processing unit determines whether at least one of a horizontal operation, a vertical operation, a depth operation, and a twist operation has been performed with a palm pressed against the handle. The driving assistance apparatus as described . The camera fixed to the non-rotating part measures the direction of the camera due to vibration based on the shape of the rotated part around the handle and the attached mark, and cancels the vibration or determines that the camera is not operated when it vibrates The driving support apparatus according to claim 1, further comprising a processing unit. The camera fixed to the non-rotating part measures the rotation movement of the handle due to the depression angle according to the shape of the handle and the mark attached on the handle, cancels the rotation of the handle, or the handle rotates. The driving support device according to claim 1, further comprising an operation processing unit that determines that no operation is performed. The driving support device according to claim 1, wherein the operation processing unit performs vector conversion into a dimension of a radius and an angle of the steering wheel, and adds an offset corresponding to the radius of the steering wheel to convert the steering wheel shape into a straight line. . The driving support device according to claim 1, wherein the operation processing unit makes a different operation determination with left and right hands. Furthermore, the said operation process part can drive the said to-be-operated apparatus different content according to the grip state on the said handle | steering-wheel .

Images