JP2016157457A - Operation input device, operation input method and operation input program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an operation input device capable of performing operation input based on the shape of user's hand by using a general camera.SOLUTION: With an operation input device, based on an imaged image near a driver seat of a moving body, an operation input of a user is recognized. Firstly, based on the imaged image obtained with a camera, a detection reference region is decided with an actual object near the driver seat, for example a handle, as a reference. Then, in the detection reference region, an operation object is detected based on a constraint in appearance of the operation object that is decided by a relation between an actual object and the operation object that is used for operation input such as, for example, a finger/hand. Then, a state of the detected operation object is estimated, and operation input is identified based on the state of the estimated operation object, for an operation signal corresponding to the operation input to be outputted to the device.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a method for inputting an operation to an apparatus using a change in the state of an operation article in an image photographed by a camera.

  Various methods have been proposed as an interface for performing operation input to the apparatus. In particular, the following has been proposed as a method for performing an operation input for a device using a user's hand shape in the vicinity of the handle of a driver's seat of a vehicle.

  Patent Document 1 describes an interface device for operating an in-vehicle device. This device captures a user's hand holding the handle from the back side of the handle and acquires a gesture. As a result, this apparatus photographs the hand in the state of pointing forward while grasping the handle and determines its shape.

  Patent Document 2 describes a technique in which a predetermined detection area including the hand of an occupant of a vehicle is photographed by a plurality of cameras, and the shape and movement of the hand are extracted based thereon to determine an operation command.

  In addition, Patent Documents 3 and 4 describe techniques for performing operation input based on hand shapes and hand gestures.

JP 2007-164814 A JP 2006-335112 A JP 2007-164814 A JP 2006-335112 A

  However, in the method described in the above patent document, it is necessary to attach a camera to the handle or to use a plurality of cameras or special cameras.

  Examples of the problems to be solved by the present invention include those described above. An object of this invention is to provide the operation input apparatus which can perform the operation input based on a user's hand shape using a general camera.

  The invention according to claim 1 is an operation input device for performing an operation input, and in a captured image captured by a camera that captures the vicinity of a driver's seat of a moving body, a part of an entity installed in the vicinity of the driver's seat. Determining means for determining the operation article based on an area that is not shielded by the operation article used for the operation input in the detection reference area in the captured image; And a specifying unit that specifies the operation input based on the detected operation article.

  The invention according to claim 7 is an operation input method that is executed by an operation input device that performs operation input. In a captured image that is captured by a camera that captures the vicinity of a driver's seat of a moving body, A determination step of determining a part of the installed entity as a detection reference area; and the operation article in the detection reference area in the captured image based on an area that is not shielded by the operation article used for the operation input. And a specifying step for specifying the operation input based on the detected operation article.

  The invention according to claim 8 is an operation input program executed by an operation input device that performs an operation input, and in a captured image captured by a camera that captures the vicinity of a driver's seat of a moving body, A determination step of determining a part of the installed entity as a detection reference area; and the operation article in the detection reference area in the captured image based on an area that is not shielded by the operation article used for the operation input. The operation input device is caused to execute a detection step of detecting the operation input and a specifying step of specifying the operation input based on the detected operation article.

It is a block diagram which shows the structure of the operation input apparatus which concerns on an Example. The installation state of a camera is shown typically. It is a flowchart of operation recognition processing. It is a figure explaining the determination method of a detection reference area. An example of a hand shape is shown. It is a flowchart of an operation article detection process. It is a figure explaining the detection method of an operation thing. It is a figure explaining the estimation method of an operation article state. It is a figure explaining the estimation method of an operation article state.

  In a preferred embodiment of the present invention, an operation input device that performs an operation input to a device mounted on a moving body includes: a camera that captures the vicinity of a driver's seat of the moving body; and a captured image that is captured by the camera. A detection reference area determining means for determining a detection reference area based on an entity installed in the vicinity of the driver's seat, and a relationship between the entity and an operation article used for operation input in the detection reference area; An operation object detection unit that detects the operation object, an operation object state estimation unit that estimates a state of the detected operation object, and an estimated state of the operation object based on a restriction on how the operation object is seen Operation determining means for specifying an operation input based on the above and outputting an operation signal corresponding to the operation input to the apparatus.

  The operation input device recognizes an operation input by a user based on a photographed image near the driver's seat of the moving body. First, a detection reference region is determined from a photographed image obtained by a camera with reference to an entity near the driver's seat, for example, a steering wheel. Next, in the detection reference area, the operation article is detected based on the restriction on the appearance of the operation article determined by the relationship between the entity and the operation article used for operation input such as a finger / hand. Then, the state of the detected operation article is estimated, an operation input is specified based on the estimated operation object state, and an operation signal corresponding to the operation input is output to the apparatus. Thereby, operation input can be easily performed when a user performs a gesture with a hand, a finger, etc. near a driver's seat.

  In one mode of the operation input device, the moving body is a car, the entity is a handle installed near a driver's seat, and the camera captures an area including the handle.

  In another aspect of the operation input device, the detection reference area determination unit extracts a predetermined shape of the entity from the photographed image, and determines a part thereof as the detection reference area. For example, the detection reference area determination unit extracts the shape of the handle as the entity and sets a part of the shape as the detection reference area.

  In another aspect of the operation input device, the restriction on the appearance of the operation article is a restriction caused by a geometric relationship between the entity and the operation article. For example, the operation object detection means is configured to detect the finger position caused by a geometrical relationship such as the position, thickness, and orientation of the finger with respect to the handle when the finger or hand as the operation object is positioned on the handle as the entity. A finger or a hand as an operation object is detected based on the appearance.

  In another aspect of the operation input device, the restriction on how the operation object is seen is that light reflection characteristics of each of the entity and the operation object in a state where the operation object is positioned on the entity. This is a restriction caused by the difference in appearance due to. For example, the operation object detection means detects a finger or a hand as an operation object based on a difference in brightness between a handle as an entity and a finger or hand as an operation object positioned thereon.

  In one preferable example, the operation article is a finger, and the state of the operation article is the number of fingers.

  In another aspect of the operation input device, the operation article state estimation unit estimates the shape of the operation article as the state of the operation article. For example, the operation article state estimation means estimates the hand shape as the state of the hand that is the operation article. Specifically, the operation article state estimation means estimates the shape of the operation article extending outside the detection reference area based on the detection result of the operation article in the detection reference area. Further, the operation object state estimation means extracts at least one feature of luminance or color of the detected operation object, and detects a region applicable to the feature outside the detection reference region, thereby detecting the detection reference. A region of the operation article existing outside the region is detected to estimate the shape of the operation article. In a preferred example, the operation article is a hand, and the shape of the operation article is the shape of the entire hand.

  Another preferred embodiment of the present invention is an operation input method executed by an operation input device that includes a camera and inputs an operation to an apparatus mounted on a moving body, and the image is captured by the camera. In the image, a detection reference area determination step for determining a detection reference area based on an entity installed near the driver's seat, and a relationship between the entity and the operation article used for operation input in the detection reference area An operation object detection step for detecting the operation object, an operation object state estimation step for estimating the state of the detected operation object, and the estimated operation object And an operation determining step of specifying an operation input based on the state and outputting an operation signal corresponding to the operation input to the device.

  Another preferred embodiment of the present invention is an operation input program that is executed by an operation input device that includes a camera and inputs an operation to an apparatus mounted on a moving body, the image being captured by the camera In the image, a detection reference area determination step for determining a detection reference area based on an entity installed near the driver's seat, and a relationship between the entity and the operation article used for operation input in the detection reference area An operation object detection step for detecting the operation object, an operation object state estimation step for estimating the state of the detected operation object, and the estimated operation object The operation input device is caused to execute an operation determination step of specifying an operation input based on the state of the operation and outputting an operation signal corresponding to the operation input to the device. This operation input program can be stored and handled in a storage medium.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[Device configuration]
FIG. 1 shows a configuration of an operation input device according to an embodiment. In this embodiment, the operation input device is used for operation input to the in-vehicle device. Here, the in-vehicle device includes various devices mounted on the vehicle such as a car air conditioner, a car AV device, and a navigation device mounted on the vehicle.

  As shown in FIG. 1, the operation input device includes a camera 5 and an operation recognition processing unit 10. The camera 5 is fixedly installed at a position where the user (car driver) takes a picture of the periphery of the steering wheel to be driven. FIG. 2 schematically shows the installation state of the camera 5. FIG. 2 is a side view of the driver sitting on the driver's seat and holding the handle 2.

  The installation location of the camera 5 is preferably near the sun visor in front of the driver's seat and in front of the front. Thus, the camera 5 captures an area including the steering wheel (steering) 2, the dashboard 3, the left and right levers 4 (see FIG. 4A), etc. near the driver's seat, and the operation recognition processing unit 10 is output. Since the driver operates the handle 2 while driving the vehicle, the driver's hand and fingers are included in the captured image. In the present embodiment, it is assumed that the handle 2 in the driver's seat has the same shape, size, and arrangement in many types of automobiles and is black or a color close to black.

  In this embodiment, the camera 5 is not a special camera such as a distance measuring camera, an infrared camera, or a stereo camera, but is a normal camera, and can capture, for example, 30 frames per second. However, the camera 5 is not provided with an infrared cut filter that is usually provided. In addition, an infrared irradiation device for auxiliary light is provided in the vicinity of the camera 5 to irradiate infrared rays in the vicinity of the handle 2 at night. Accordingly, the camera 5 can be photographed around the user's hand even at night.

  The operation recognition processing unit 10 receives and analyzes a video (a continuous frame image of 30 frames per second) taken by the camera 5 and recognizes an operation (gesture) performed around the handle 2. And if operation is recognized, the operation recognition process part 10 will output an operation signal with respect to the vehicle equipment of operation object. In this embodiment, the operation recognition processing unit 10 is realized by software processing using a computer.

  As illustrated in FIG. 1, the operation recognition processing unit 10 includes a detection reference region determination unit 11, an operation object detection unit 12, an operation object state estimation unit 13, and an operation determination unit 14.

  The detection reference area determination unit 11 analyzes a captured image input from the camera 5 and determines a detection reference area. This is called “detection reference region determination processing”. The detection reference area refers to an area that becomes a reference in recognizing an operation input by a user in a captured image.

  The operation article detection unit 12 detects an operation article (for example, a user's finger or hand) used by the user for operation input based on the detection reference area determined by the detection reference area determination process. This is called “operation object detection processing”.

  The operation article state estimation unit 13 estimates the state of the operation article detected by the operation article detection process. This is called “operation article state estimation processing”. Specifically, the operation article state estimation unit 13 estimates the shape of the operation article of the user's finger or hand.

  The operation determination unit 14 determines an operation corresponding to the state of the operation article estimated by the operation article state estimation process, and outputs an operation signal.

[Operation recognition processing]
Next, the operation recognition process executed by the operation recognition processing unit 10 will be described in detail. FIG. 3 is a flowchart of the operation recognition process. As illustrated in FIG. 3, the operation recognition processing unit 10 includes a detection reference region determination process (step S11), an operation object detection process (step S12), an operation object state estimation process (step S13), and an operation determination process (step S14) is executed. Hereinafter, it demonstrates in order.

(1. Detection reference area determination process)
The detection reference region determination unit 11 of the operation recognition processing unit 10 performs detection reference region determination processing (step S11). Specifically, the detection reference area determination unit 11 first detects the handle 2 as an entity from the captured image input from the camera 5, and then determines the detection reference area using the handle 2 as a starting point.

  First, detection of the handle 2 will be described. From the installation conditions of the camera installed so as to look down from the upper part of the driver's seat and the driver's seat structure in which the steering wheel is assumed to have the same position, size, and shape in many car models, The position, size, and shape of the handle 2 in the captured image can be generally predicted. An example of a photographed image including the handle 2 is shown in FIG. In this example, the captured image includes a handle 2, a dashboard 3, a lever 4, and the like.

  The detection reference region determination unit 11 obtains an edge (luminance gradient) from the captured image, detects an elliptical edge by matching an ellipse with various parameters, and interprets this as the outer peripheral portion of the handle 2. Specifically, as illustrated in FIG. 4B, the detection reference region determination unit 11 detects an ellipse 2x corresponding to the outer peripheral portion of the handle 2.

  Since both the camera 5 and the handle 2 are fixed, if the captured images taken over a certain period of time are overlapped, the surrounding image of the handle 2 can be obtained by removing dynamic factors such as sunlight, shadows and driver's hands. Can be generated. Here, “a certain amount of time” is a level at which it can be said that there is a sufficiently small change in the sun and a bias in the appearance of the driver's hand. The photographed image used for detecting the handle 2 includes photographed images such as a case where light from various directions hits or is cloudy in a few days, during driving or other than driving. Note that it is not always necessary to superimpose captured images of all times. For example, a frame image may be cut out under conditions such as when the sunlight changes, when driving is started, or when driving is stopped.

  In addition, when multiple ellipses are detected in the captured image, the same position in the ellipse appears to be hidden from the multiple ellipses for a relatively long time while the car is running. The ellipse 2x corresponding to the outer periphery of the handle 2 may be selected on the condition that This is a condition assuming that the driver's hand holds the same position of the handle 2 for a long time. For example, even if a semi-circular edge similar to the steering wheel 2 is detected from the boundary of the dashboard 3, it is possible for the driver to hold the hand over the dashboard 3 for a long time like the steering wheel 2. Therefore, the semicircular edge can be excluded from the handle candidates.

  Next, determination of the detection reference area will be described. The detection reference area is determined based on the ellipse 2x detected as described above and corresponding to the outer periphery of the handle 2 as an entity. Specifically, as shown in FIG. 4C, an arc shape having a predetermined width from an arc in a predetermined range on the upper side of the ellipse 2x corresponding to the detected outer peripheral portion of the handle 2 toward the inner side of the ellipse. This area is formed, and this is determined as the detection reference area DA. The “upper side” referred to here is a side of the ellipse 2x corresponding to the outer periphery of the handle 2 that the driver recognizes as the upper side of the handle 2. The “predetermined range” is, for example, a range of 45 degrees from the right above the ellipse 2x (the 12 o'clock position of the clock) to the left and right. The predetermined range corresponds to a range where the operator performs a gesture operation. The “predetermined width” is, for example, 0.2 times the radius of the minor axis of the ellipse 2x. The predetermined width corresponds to the width of the handle 2.

  The determination of the detection reference area needs to be performed when the positional relationship between the camera 5 and the handle 2 is changed by adjusting the position of the handle 2 when the camera 5 is installed. Specifically, the detection reference area may be determined when it is detected from the image of the camera 5 that the positional relationship has changed. Further, the detection reference area may be redetermined in accordance with a switch operation by the user.

(2. Operation object detection processing)
If the detection reference area is determined, the appearance when the user puts his / her finger so as to touch the handle 2 can be assumed based on the detection reference area. That is, it is possible to assume the direction of the finger and the width of the finger when the finger is placed on the handle 2. FIG. 5 schematically shows a state where three fingers are placed on the handle 2.

  Specifically, the direction of the finger is a direction across the detection reference area DA. The width of the finger can be determined by assuming the scale of the handle 2 from the apparent size of the handle 2 and estimating the apparent finger thickness when the finger is placed on the handle 2. Also, assuming that the handle surface is dark, the finger looks brighter than the surface of the handle 2 when the finger is placed, and a steep change in luminance value appears on the boundary between the handle 2 surface and the finger area. It can also be assumed that it exists in a shape. If the handle 2 is white, it can be assumed that the finger appears darker than the surface of the handle 2 when the finger is placed. Here, the description will be made assuming that the handle 2 is dark.

  The operation object detection unit 12 performs an operation object detection process (step S12). In the operation object detection process, an operation object (finger / hand) is detected in the detection reference region. Specifically, the operation object detection processing unit 12 detects the operation object based on the restriction on the appearance of the operation object determined by the relationship between the entity and the operation object. In the present embodiment, the finger / hand that is the operation object is detected based on the restriction of the appearance of the finger / hand determined by the relationship between the handle 2 that is the entity and the finger / hand that is the operation object.

  In one example, the restriction on the appearance of the operation article is a restriction caused by a geometric relationship between the entity and the operation article in a state where the operation article is located on the entity. “Geometric relationship” includes position, thickness, orientation, and the like. That is, the finger as the operation object is detected based on the position, thickness, orientation, and the like of the finger as the operation object with respect to the handle 2 as the entity.

  In another example, the restriction on the appearance of the operation object is a restriction caused by the difference in the appearance due to the reflection characteristics of the light of the object and the operation object when the operation object is located on the object. is there. In the present embodiment, as described above, the handle 2 has a dark color close to black, and a white (bright) region close to white located on the black handle 2 can be detected as a finger that is an operation object. .

  FIG. 6 is a flowchart of the operation article detection process. First, the operation article detection unit 12 extracts a luminance value of each pixel in the detection reference area DA from the photographed image, and obtains a straight line across the detection reference area DA by obtaining the intensity and direction of the luminance value gradient in each pixel. Extract (step S21). Note that the luminance value gradient in a certain pixel refers to a gradient formed by the luminance values of a plurality of pixels that are adjacent to the pixel with the pixel as the center. An example of a straight line crossing the detection reference area DA is shown in FIG.

  Next, the operation article detection unit 12 aggregates the intensity and direction of the luminance value gradient for a plurality of pixels constituting one extracted straight line (step S22). For a plurality of pixels constituting the straight line, the intensity of the luminance value gradient at the pixel is stronger than a predetermined value, and the direction of the luminance value gradient is substantially orthogonal to the straight line. When the predetermined ratio or more exists (step S23: Yes), the straight line is set as the boundary candidate of the operation article (step S24). The boundary of the operation article specifically corresponds to the left and right boundary lines of the operation article such as a finger or a hand crossing the handle 2.

  The boundary candidate of the operation article is classified into two types depending on which one of the left and right areas of the boundary candidate is bright and which is dark in the detection reference area DA. Specifically, in step S24, as shown in FIG. 7B, one straight line is selected from either the left boundary candidate 31 that is dark on the left side and bright on the right side and the right boundary candidate 32 that is bright on the left side and dark on the right side. Is detected. In this way, the processes in steps S22 to S24 are executed for all the crossing straight lines extracted in step S21, and boundary candidates are extracted.

  Next, the operation article detection unit 12 specifies a pair of the left boundary candidate 31 and the right boundary candidate 32 as an operation object candidate, and identifies all operation object candidates from all the boundary candidates extracted in steps S22 to S25 ( Step S26). Here, the operation object candidate specifies a pair in which the left boundary candidate 31 and the right boundary candidate are arranged in this order from the left side. In other words, a pair of the left boundary candidate 31 and the right boundary candidate 32 located on the right side thereof is specified. As shown in FIG. 7C, the operation object candidate is a combination of the two boundary candidates being bright inside and dark outside, and is assumed to be an operation object such as a finger placed on the dark handle 2. be able to.

  Next, if the width of the operation object candidate (that is, the width of the bright portion) is considered appropriate as the operation object (user's finger or hand), the operation object detection unit 12 sets the operation object candidate as the operation object. Detection is performed (step S28). In addition, as a condition for determining the operation object candidate as the operation object, it may be added that the two boundary candidates forming the operation object candidate are substantially parallel.

  In the example of FIG. 7C, there are operation object candidates 35 and 36. In this case, since the operation object candidate 35 corresponds to the width of a general human finger, it is detected as an operation object (finger). On the other hand, since the operation object candidate 36 is narrower than the width of a general human finger, it is not detected as an operation object (finger).

  In this way, when the processing of steps S27 to S29 is performed for all the operation object candidates (step S29: Yes), all the operation objects (finger / hand) included in the photographed image are detected.

(3. Operation object state estimation process)
Next, the operation article state estimation unit 13 performs an operation article state estimation process (step S13). In the operation object state estimation process, the state of the operation object detected by the operation object detection unit 12, specifically, the shape of a finger or a hand is estimated. The state of the operation article corresponds to the operation command input by the user. There are two methods for estimating the state of the operation article.

(1) First Method The first method uses information of a finger that is simply detected as the state of the operation article. For example, the operation article state estimation unit 13 estimates the number of detected fingers, that is, the “number of fingers” put out on the handle 2 for operation as the state of the operation article. In this case, the number of fingers corresponds to the operation command.

  In addition to this, the distance between the detected fingers (how to open the finger), the position of the finger with respect to the handle 2 (for example, how many positions on the arc the handle 2 has) An angle (for example, an angle formed by the boundary line of the finger with respect to a reference line connecting the center of the handle 2 and the outer periphery of the handle at the position where the finger is detected) may be used as the state of the operation object corresponding to the operation command. .

  The operation article state estimation unit 13 also performs a process of associating the position of the operation article such as a finger among a plurality of frames included in the captured image from the camera 5. Thereby, the hand movement between frames of the captured image can be used as the state of the operation article.

(2) Second Method The second method uses the shape of the entire hand (hereinafter referred to as “hand shape”) as the state of the operation article. For example, the hand shapes A to D shown in FIG. The operation object state estimation unit 13 determines whether the operation object (hand) detected by the operation object detection unit 12 indicates any of the hand shapes A to D.

  In the above-described operation object detection process, the operation object is detected only in the detection reference area DA (on the handle 2), so that it is difficult to determine the hand shape by itself. For example, the hand shape A and the hand shape B shown in FIG. 8 both have the detection results shown in FIG. 9B in the detection reference area DA, and it is difficult to distinguish them. Therefore, the following processing is performed to estimate the hand shape.

  First, the operation article state estimation unit 13 extracts only a hand shape that includes the finger shape currently detected on the detection reference area DA from the hand shape to be determined. Assuming that only the hand shapes A to D are used, if the shape of the operation article detected on the detection reference area DA is the shape shown in FIG. 9A, the detected hand shape of the operation article is the hand shape. C or hand shape D can be estimated, and hand shape A and hand shape B can be excluded.

  Next, the operation article state estimation unit 13 extracts features of the appearance of the operation article detected on the detection reference area DA. Specifically, the operation article state estimation unit 13 extracts color information (skin color). The method of detecting a hand by color may not always be used stably in various environments because the appearance of the skin color may change depending on the shooting environment such as outside light. However, in this case, the operation object detection unit 12 has another condition that is more appropriate than the color information, that is, the restriction on the appearance of the operation object (finger / hand) in the detection reference area DA (specifically, the operation object is Since the operation article is detected based on whether or not the handle 2 is overlaid, the operation article state estimation unit 13 does not need to detect a finger or a hand based on the color information (skin color). Conversely, if color information or luminance information is extracted from the region of the operation object (finger / hand) detected by the operation object detection unit 12, the operation object state estimation unit 13 determines the skin color of the person at that time. The appearance can be acquired accurately. Therefore, the operation article state estimation unit 13 acquires how the person's skin color looks from the color information or luminance information of the area detected as the operation article (finger / hand) by the operation article detection unit 12.

  Next, the operation article state estimation unit 13 detects an operation article area (hand area) other than the detection reference area DA from the captured image using the acquired color information or luminance information. Specifically, in the vicinity of the detection reference area DA in the photographed image, an area having a color or luminance that is the same as or close to the color of the previously obtained operation article (finger / hand) is defined as the operation article area (hand area). Extract. At this time, the operation object state estimation unit 13 detects an operation object region other than the detection reference region DA by using a correlation with a captured image in a state where the operation object (finger / hand) does not exist in the detection reference region DA. It is preferable.

  Next, the operation object state estimation unit 13 is formed by combining the operation object region (hand region) detected within the detection reference region and the operation object region (hand region) detected outside the detection reference region. It is determined whether or not the hand shape is any of the hand shapes A to D shown in FIG. In this case, the operation article state estimation unit 13 can determine the hand shape by a shape matching process by simple superposition. The operation article state estimation unit 13 may perform the shape matching process by assigning a higher weight to the hand region detected in the detection reference region than the hand region detected outside the detection reference region. Good.

  Thus, the operation article state estimation unit 13 estimates the state (number of fingers, hand shape, etc.) of the operation article corresponding to the operation command by executing the first method and the second method.

(4. Operation decision processing)
Next, the operation determination unit 14 performs an operation determination process (step S14). Specifically, the operation determination unit 14 specifies an operation command input by the user based on the estimation result of the operation article state estimation unit 13. As described above, the state of the operation article (number of fingers, hand shape, etc.) is associated with a predetermined operation command. Therefore, the operation determination unit 14 determines that the operation command corresponding to the operation object state estimated by the operation object state estimation unit 13 is the operation command input by the user, and outputs an operation signal corresponding to the operation command. .

  For example, when the operation article state estimation unit 13 detects a specific hand shape that has not been detected in a previous frame in a certain frame in the captured image, the operation determination unit 14 corresponds to the hand shape. The operation signal of the operation command is output once. Further, when the operation object state estimation unit 13 detects that the position of the finger as the operation object has changed with respect to the position in the previous frame, the operation determination unit 14 outputs the movement amount as an operation signal. For example, the operation article state estimation unit 13 estimates that one finger is at a position of 0 degree on the handle at a certain time and one finger is at a position of 3 degrees on the handle at the next time. In this case, the operation determination unit 14 determines that the finger has moved a distance corresponding to 3 degrees from the left to the right between the two time points, and outputs an operation signal corresponding to the movement amount. When the operation state estimation unit 13 detects any of the hand shapes A to D, the operation determination unit 14 outputs an operation command corresponding to the hand shape.

  As described above, the operation recognition processing unit 10 can interpret an operation input performed by the user using an operation object such as a finger / hand in the detection reference area, and can output an operation signal corresponding to the operation input. .

[Advantages of using a handle]
In the above embodiment, the operation input is performed by using the handle of the vehicle as the detection reference region and using the finger or hand as the operation object. This has the following advantages.

  (1) Since the handle has almost the same shape and position regardless of the type of automobile, and is fixedly arranged, it is easy to detect with a camera.

  (2) By setting the detection reference area on the handle, the operation article can be easily detected. This is because the way the fingers are visible can be limited. That is, by using the handle as a base, it is easy to limit the range in which the user puts out the finger, compared to the case where the finger pointing gesture is performed at a free position in the air.

  (3) By using the information of the operation article detected on the handle as highly reliable information, it is possible to detect the shape of the hand other than on the handle (region outside the handle) with high accuracy. Further, the appearance information (for example, skin color or brightness at that time) of the operation article is extracted from the information of the operation article detected on the handle, and can be used to detect the operation article region from the area other than the handle. . Thereby, the detection accuracy of the operation article region can be increased.

  (4) The method in which the user performs a gesture on the handle is a type of display in which an image is displayed forward, such as a head-up display (HUD), compared to a method in which the user performs a gesture near the handle or near the shift lever. Suitable for use in combination with equipment. This is because it is possible to realize an interface in which the user points in the direction while looking at the image displayed forward.

[Modification]
In addition to the steering wheel, the detection reference area can be set by using an in-vehicle structure that can be seen in the same manner as the steering wheel, such as an edge of the dashboard and a lever around the steering wheel.

2 Handle 5 Camera 10 Operation Recognition Processing Unit 11 Detection Reference Area Determination Unit 12 Operation Object Detection Unit 13 Operation Object State Estimation Unit 14 Operation Determination Unit

Claims (9)

An operation input device for performing operation input,
In a captured image captured by a camera that captures the vicinity of a driver's seat of a moving body, a determination unit that determines a part of an entity installed near the driver's seat as a detection reference region;
Detecting means for detecting the operation article based on an area not covered by the operation article used for the operation input in the detection reference area in the captured image;
Specifying means for specifying the operation input based on the detected operation article;
An operation input device comprising:   The detection means detects the operation object based on a geometrical relationship generated by the entity shielding the operation object in a state where the operation object is positioned on the object. The operation input device according to claim 1.   The detection means detects the operation object based on a difference in appearance due to light reflection characteristics of the operation object and the operation object in a state where the operation object is positioned on the operation object. The operation input device according to claim 1 or 2. An estimation means for estimating a state of the detected operation article;
4. The apparatus according to claim 1, wherein the specifying unit specifies the operation input based on the estimated state of the operation article, and outputs an operation signal corresponding to the operation input to the apparatus. The operation input device according to one item.   The operation input device according to claim 4, wherein the estimation unit estimates a shape of the operation article as a state of the operation article based on a detection result of the operation article.   The estimation means extracts at least one feature of the detected brightness or color of the operation article, and detects a region applicable to the feature, thereby detecting the region of the operation article and determining the shape of the operation article. The operation input device according to claim 5, wherein the operation input device is estimated. An operation input method executed by an operation input device for performing operation input,
In a captured image captured by a camera that captures the vicinity of a driver's seat of a moving object, a determination step of determining a part of an entity installed near the driver's seat as a detection reference region;
In the detection reference region in the captured image, a detection step of detecting the operation article based on an area that is not shielded by the operation article used for the operation input;
A specifying step of specifying the operation input based on the detected operation object;
An operation input method comprising: An operation input program executed by an operation input device that performs operation input,
In a captured image captured by a camera that captures the vicinity of a driver's seat of a moving object, a determination step of determining a part of an entity installed near the driver's seat as a detection reference region;
In the detection reference region in the captured image, a detection step of detecting the operation article based on an area that is not shielded by the operation article used for the operation input;
A specifying step of specifying the operation input based on the detected operation object;
Is executed by the operation input device.   A storage medium storing the operation input program according to claim 8.

Images