JP2011222031A - On-vehicle electronic equipment operation device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an on-vehicle electronic equipment operation device capable of substantially improving the operability even in the situation that the screen of the operation device cannot be viewed or the like by providing both of a touch input part and hand photographing means as input means and more effectively utilizing them.SOLUTION: An operation screen intrinsic to on-vehicle electronic equipment to be an operation object is displayed on an image display device, and relation of an instruction input state on the operation screen by one or both of a touch input point and a photographed fingertip point and operation command output contents to the on-vehicle electronic equipment is set as a predetermined interface specification by a user interface engine, and operation command information is outputted on the basis of the interface specification corresponding to the instruction input state on the operation screen. Then, since the interface specification is changed corresponding to the number of detected photographed fingertip points, by increasing the types of operation areas corresponding to the number of detected fingers, operation input by a plurality of fingers is diversified and made efficient.

Description

  The present invention relates to an on-vehicle electronic device operating device.

JP 2000-335330 A JP 2001-216069 A

  Conventionally, as an operation device for an in-vehicle device such as a car navigation device, an operator's hand is photographed with a camera, a finger image is extracted from the photographed data, and the extracted finger image area is displayed on a GUI input screen of the in-vehicle device (for example, A type of display that is superimposed on a navigation screen) has been proposed. For example, Patent Document 1 uses a camera attached to the ceiling of a vehicle body to photograph a hand operating a touch panel at the seat, and the photographed hand and the touch panel are disposed opposite to the front of the operator. A configuration for displaying on a liquid crystal panel is disclosed. Patent Document 2 discloses an operation input device that assigns different operation contents in accordance with the distance between a user's finger or the like and the display screen.

  In the above-described conventional operation system, the shooting information of the hand by the camera is only used for superimposed display on the screen of the hand outline image indicating the operation position, and the image information itself is effectively used as input information. It was difficult. In particular, during driving, the driver cannot gaze at the screen of the operating device, so that an erroneous operation is likely to occur and a complicated operation is difficult. Although the operation device of Patent Document 1 uses two types of input information of hand imaging information and touch detection information, it cannot be said that they were effectively used for the purpose of improving operability. .

  The problem of the present invention is that both the touch input unit and the hand photographing unit are provided as the input unit, and by using both more effectively, the operability is greatly improved even in a situation where the screen of the operation device cannot be watched. An object of the present invention is to provide an on-vehicle electronic device operating device that can be improved.

Means for solving the problems and effects of the invention

A configuration related to the present invention is an operation device that is used by being mounted on a vehicle to perform operation input of an in-vehicle electronic device, and in order to solve the above-described problem,
A touch input device that has an operation surface that accepts a touch operation by an operator's fingertip and detects a touch input point by the operation;
An imaging device that has an imaging range that has a unique coordinate correspondence with the operation surface, and that captures an operator's hand approaching the operation surface;
Photographing fingertip point detecting means for detecting a fingertip point of a hand as a photographing fingertip point based on hand image data acquired by the photographing device;
An image display apparatus having a display screen having a coordinate correspondence relationship unambiguously with an operation surface and a photographing range;
An indicator image display control means for displaying on the display screen an indicator image for instructing the fingertip point to the image display device;
On the image display device, an operation screen display control means for displaying an operation screen unique to the in-vehicle electronic device to be operated,
The relationship between the instruction input state on the operation screen by one or both of the touch input point and the shooting fingertip point and the content of the operation command output to the in-vehicle electronic device is set as a predetermined interface specification. A user interface engine that outputs operation command information based on the interface specifications according to the instruction input state of
Traveling state detecting means for detecting the traveling state of the vehicle;
It is assumed that there is provided interface specification changing means for changing the interface specification of the user interface engine according to the detected traveling state.

  According to the configuration related to the present invention, an operation screen unique to the in-vehicle electronic device to be operated is displayed on the image display device, and one of the touch input point and the imaging fingertip point is selected by the user interface engine. The relationship between the instruction input state on the operation screen by both and the operation command output content to the in-vehicle electronic device is set as a predetermined interface specification, and based on the interface specification according to the instruction input state on the operation screen Outputs operation command information. And since it is comprised so that an interface specification may be changed according to the detected driving state of a vehicle, the operativity of a vehicle-mounted electronic device operating device can be optimized at any time according to a vehicle driving state.

  The interface specification changing means can be configured to change the display content of the operation screen together with the operation command output content corresponding to the display content in accordance with the traveling state of the vehicle. That is, in a user interface (a kind of graphical user interface) using an operation screen with input information of touch operation and shooting fingertip point detection, the display content of the operation screen (that is, the operation screen) according to the running state of the vehicle In the case where an operation situation in which the screen of the operation device cannot be watched occurs according to the vehicle running state, appropriate operation support can be performed.

  Specifically, the vehicle speed detection means is provided in the running state detection means, and the interface specification is an interface specification for low speed running (including when the vehicle is stopped) used when the detected vehicle speed value is less than a threshold value, and the detected vehicle speed value. It is possible to prepare a high-speed running interface specification that is used when the value exceeds the threshold. The interface specification changing means can be configured to switch the interface specification between the low speed travel interface specification and the high speed travel interface specification according to the vehicle speed value. When the vehicle speed is low (especially when the vehicle is stopped), the driver can ensure a high gaze time and frequency of the operation screen. However, when the vehicle speed is high, the driver must watch the front, and it is difficult to view the operation screen for a long time. Become. Therefore, as described above, interface specifications for low-speed driving and interface specifications for high-speed driving are prepared in advance, and the operability that is suitable for each situation can be ensured by switching between the vehicle speed and the gaze possibility of the operation screen. Can be achieved appropriately.

  The operation screen display control means can be configured to display and form a plurality of operation areas (for example, operation buttons and input windows) for outputting different operation command on the operation screen. In this case, the interface specification changing means can be configured to set the number of display formations of the effective operation area in the interface specification during high-speed driving to be smaller than the display formation number of the effective operation area in the low-speed driving interface specification. Further, the display formation dimension of the effective operation area in the interface specification during high-speed traveling can be configured to be set larger than the display formation dimension of the effective operation area in the interface specification during low-speed traveling. When driving at high speeds where it is difficult to keep an eye on the operation screen, the operation tends to be frustrated, and if the number of operation areas is too large or the area of each operation area is too small, erroneous operations are likely to occur. Therefore, by configuring as described above, such a problem can be effectively solved.

  Further, the photographing fingertip point detecting means can be configured to simultaneously detect a plurality of fingertip points based on hand image data. The operation screen display control means can be configured to increase the types of operation areas to be displayed and formed on the operation screen as the number of detected fingertip points increases. By increasing the types of operation areas in accordance with the number of detected fingers, it is possible to diversify and increase the efficiency of operation inputs using a plurality of fingers. In particular, each time a fingertip point is detected, if an operation area is dynamically set and formed at a position corresponding to each fingertip point, a unique operation area is set for each finger. The function sharing of each finger is clarified, and complicated input can be made more intuitive.

  In this case, the interface specification changing means can be configured to set the maximum number of display formation types in the operation area in the interface specification during high-speed driving smaller than the maximum number of display formation types in the operation area in the interface specification during low-speed driving. By switching to an interface specification with a small maximum number of display formation types during low-speed traveling where it is difficult to look at the screen, it is possible to make it difficult for erroneous operations to occur.

  On the other hand, the operation screen display control means detects a drag operation along the operation surface of the touch input device, and switches and displays an operation area effective on the screen among a plurality of operation command output contents different from each other. It can also be configured as. The interface specification changing means switches the operation area continuously during one drag operation in the low speed running interface specification, while the interface area changes every time the drag operation occurs in the high speed running interface specification. Can be configured to be switched intermittently. If the operation area can be continuously switched by the drag operation, there is an advantage that the user can quickly reach the desired operation area while confirming the contents of the operation area to be switched one after another. However, when the screen is difficult to view directly (at high speed), a desired operation area may be skipped if the screen is continuously switched, and the target area may not be reached as intended. Therefore, in the high-speed running interface specification, it is possible to reliably reach the desired operation area by switching the operation area intermittently each time a drag operation occurs.

  As yet another aspect, there is provided operation operation pattern detection means for detecting a specific movement along the operation surface of the image fingertip point as an operation operation pattern, and the operation screen display control means outputs operation commands different from each other. It is also possible to adopt a configuration in which a plurality of operation areas are displayed on the operation screen. The user interface engine outputs first operation command information corresponding to the operation movement pattern as the operation movement pattern is detected, and when the operation area is selected by a touch input point or a shooting fingertip point, An operation command information output unit that outputs second operation command information associated with the operation region is configured. The interface specification changing means activates at least the second operation command information according to the selection of the operation area in the low speed running interface specification, and the first operation according to the detection of the operation action pattern in the high speed running interface specification. Only command information can be activated. According to this configuration, in the low-speed driving interface specification, it is possible to select the operation area while looking at the screen, while in the high-speed driving interface specification, the operation area selection operation is eliminated, and the screen does not have to be watched. Since only possible operation movement patterns (that is, gestures) are activated, the operability is improved.

Next, the first aspect of the present invention is an operating device that is mounted on a vehicle and used to perform an operation input of an in-vehicle electronic device.
A touch input device having an operation surface for receiving a touch operation by an operator's fingertip and detecting a touch input point by the touch operation;
An imaging device that has an imaging range that has a unique coordinate correspondence with the operation surface, and that captures an operator's hand approaching the operation surface;
An imaging fingertip point detection unit configured to be capable of simultaneously detecting a plurality of fingertip points of a hand based on hand image data acquired and acquired by the imaging device,
An image display apparatus having a display screen having a coordinate correspondence relationship unambiguously with an operation surface and a photographing range;
An indicator image display control means for displaying on the display screen an indicator image for instructing the fingertip point to the image display device;
On the image display device, an operation screen display control means for displaying an operation screen unique to the in-vehicle electronic device to be operated,
The relationship between the instruction input state on the operation screen by one or both of the touch input point and the shooting fingertip point and the output content in the operation command information to the in-vehicle electronic device is set as a predetermined interface specification, and the operation screen A user interface engine that outputs operation command information based on the interface specifications according to the instruction input state above,
And interface specification changing means for changing the interface specification of the user interface engine in accordance with the number of shooting fingertip points detected by the shooting fingertip point detecting means.

  By changing the interface specifications of the user interface engine according to the number of photographing fingertip points, it is possible to optimize the input specifications according to the number of detected fingers. For example, when the operation screen display control means displays a plurality of operation areas for performing different operation command outputs on the operation screen, the interface specification changing means is the photographing detected by the photographing fingertip point detecting means. The number of effective operation region display formations when the number of fingertip points is plural may be set to be larger than the number of effective operation region display formations when the number of photographing fingertip points is one. By increasing the types of operation areas in accordance with the number of detected fingers, it is possible to diversify and increase the efficiency of operation inputs using a plurality of fingers.

A second aspect of the present invention is an operating device that is used by being mounted on a vehicle in order to perform an operation input of an in-vehicle electronic device.
A touch input device having an operation surface for receiving a touch operation by an operator's fingertip and detecting a touch input point by the touch operation;
An imaging device that has an imaging range that has a unique coordinate correspondence with the operation surface, and that captures an operator's hand approaching the operation surface;
Photographing fingertip point detecting means for detecting a fingertip point of a hand as a photographing fingertip point based on hand image data acquired by the photographing device;
An image display apparatus having a display screen having a coordinate correspondence relationship unambiguously with an operation surface and a photographing range;
An indicator image display control means for displaying on the display screen an indicator image for instructing the fingertip point to the image display device;
On the image display device, an operation screen display control means for displaying an operation screen unique to the in-vehicle electronic device to be operated,
The relationship between the instruction input state on the operation screen by one or both of the touch input point and the shooting fingertip point and the output content in the operation command information to the in-vehicle electronic device is set as a predetermined interface specification, and the operation screen A user interface engine that outputs operation command information based on the interface specifications according to the instruction input state above,
Interface specification changing means for changing the interface specification of the user interface engine according to whether or not a touch operation on the touch input device is detected at the same time when a photographing fingertip point is detected, To do.

  According to the above configuration, when an imaging fingertip point is detected using the operation input system consisting of two systems of a touch input device and a hand imaging device, touch operations to the touch input device are simultaneously performed. Since the interface specification of the user interface engine is changed depending on whether or not it is detected, the input usage mode of the operating device can be greatly expanded.

  Specifically, the operation screen display control means detects a drag operation along the operation surface of the touch input device, and scrolls and displays the image information being displayed on the screen of the image display device in a direction corresponding to the drag operation. Can be configured to process. In this case, the interface specification changing means can be configured to set the scroll speed of the image information when the touch operation is detected together with the photographing fingertip point to be lower than the scroll speed when the touch operation is not detected. The screen scrolling speed can be easily and intuitively switched depending on whether the user drags while touching the operation surface or not.

  On the other hand, the operation screen display control means is configured to detect a drag operation along the operation surface of the touch input device and perform scroll display processing of the hierarchical information output menu being displayed on the screen of the image display device. You can also. The interface specification changing means can be configured to perform scroll display processing on different layers of the information output menu depending on whether a touch operation is detected or not detected. By switching between dragging while touching the operation surface or dragging without touching, the hierarchy of the information output menu to be scrolled is changed, so that the convenience of menu search can be greatly improved.

The perspective view which shows an example of the in-vehicle installation form of the operating device of this invention. The cross-sectional schematic diagram which shows an example of the internal structure of the operation part. The block diagram which shows an example of the electrical constitution of the operating device of FIG. Explanatory drawing which shows the flow of the image processing which determines a photography fingertip point. The flowchart which shows the flow of an imaging | photography fingertip point specific process. The flowchart which shows the detail of a fingertip determination process. The figure explaining the concept of the width | variety of a front-end | tip area | region. The figure explaining the concept of the labeling process for isolate | separating a some front end area | region. The figure which shows embodiment which superimposes and displays the hand image acquired with the hand imaging | photography camera. The figure which shows the example of a screen display which superimposed the indicator image. The figure which shows the 1st reference form relevant to this invention. The figure which shows a 2nd reference form similarly. The figure which shows a 3rd reference form similarly. The figure which similarly shows the 4th reference form. The figure which shows a 5th reference form similarly. The figure which similarly shows the 6th reference form. The figure which shows 1st embodiment which concerns on 1st of this invention. The figure which shows 1st embodiment which concerns on 2nd of this invention. Explanatory drawing following FIG. Explanatory drawing following FIG. The figure which shows 2nd embodiment which concerns on 2nd of this invention. Explanatory drawing following FIG. Explanatory drawing following FIG.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 shows an example of an operating device for an in-vehicle electronic device to which the image display device of the present invention is applied. The operation device 1 has a monitor (display device) 15 disposed in the center of the instrument panel in the interior of the automobile, and the operation unit 12 is located at a position where the center console C can be operated from either the driver seat 2D or the passenger seat 2P. Is arranged. Although the purpose of use is not particularly limited, for example, it is for performing functional operations of the car navigation device and the car audio device while looking at the screen of the monitor 15 provided in the center console.

  The operation unit 12 is attached so that the input operation surface faces upward. The input operation surface is formed by a known touch panel 12a, and any of a resistive film method, a surface acoustic wave method, and a capacitance method may be adopted. The base material of the touch panel 12a is a transparent input support plate made of a transparent resin plate, a glass plate, or the like, and accepts a touch operation by the operator's fingertip while supporting it on the upper surface side. Then, an input coordinate system corresponding to the screen area of the monitor 15 is set along the plate surface.

  FIG. 2 is a cross-sectional view schematically showing the internal structure of the input unit 12. The touch panel 12a is fitted on the upper surface of the housing 122e so that the input operation surface 102b side is up. In the housing 12d, a hand photographic camera (photographing device) 12b is housed together with an illumination light source 12c and a photographic optical system, and constitutes a photographic image acquisition means. The illumination light source 12c is a light emitting diode light source (ie, a monochromatic light source) having a luminance directivity directly above the element by a convex curved mold, and a plurality of illumination light sources 12c are arranged so as to surround the lower surface of the touch panel 12a. Each illumination light source 12c is attached by tilting the mold front end side with high brightness toward the inside of the lower surface of the touch panel 12a, and the primary imaging reflected light RB1 by the palm surface of the operator's hand H on the input operation surface 102b is It is generated in a downward direction so as to pass through the touch panel 12a.

  The photographing optical system has a first reflecting portion 12p and a second reflecting portion 12r. The first reflecting portion 12p is a prism plate (transparent material plate in which triangular prism-shaped microprisms are arranged in parallel in the plane: hereinafter also referred to as a prism plate 12p) and is disposed directly below the touch panel 12a. By reflecting the primary imaging reflected light RB1 from the hand H obliquely upward, it is guided as the secondary imaging reflected light RB2 to the lateral outside of the facing space 12f between the prism plate 12p and the touch panel 12a. The second reflecting portion 12r is a plane reflecting mirror (hereinafter also referred to as a plane reflecting mirror 12r) disposed on the lateral side outside the facing space, and reflects the secondary imaging reflected light RB2 to the side, thereby The light is guided as the tertiary imaging reflected light RB3 to the hand photographing camera 12b located on the opposite side of the second reflecting portion 12r across the space 12f. The hand photographing camera 12b is provided at a position corresponding to the focal point of the tertiary imaging reflected light RB3, and acquires a photographed image of the hand H including the operator's finger.

  As shown in an enlarged view in FIG. 2, the prism plate 12 p is formed of rib-shaped microprisms each having a reflecting surface inclined at an equal angle with respect to the mirror base surface MBP and parallel to each other along the mirror base surface MBP. Even if the mirror base surface MBP is not inclined, the light incident in the normal direction can be reflected to the (oblique) side. Accordingly, the first reflecting portion 12p for side reflection can be disposed in parallel with the lower side of the touch panel 12a, and the height direction dimension of the facing space 12f can be greatly reduced.

  In addition, the second reflecting portion 12r and the hand photographic camera 12b are disposed facing each other with the opposing space 12f sandwiched from the side, so that the tertiary imaging reflected light RB that directly enters the hand photographic camera 12b crosses the opposing space 12f. Can be guided in shape. As a result, the second reflecting portion 12r and the hand photographing camera 12b can be disposed close to the side edge of the touch panel 12a, and the incident path of the imaging reflected light from the hand H to the hand photographing camera 12b is within the facing space 12f. In other words, since it is folded into a three-fold shape, the entire imaging optical system is greatly reduced in size and the casing 12d is made thinner. In particular, by reducing the size of the touch panel 12a (that is, the vertical and horizontal dimensions of the input operation surface 102b), the entire input unit 12 can be dramatically reduced in size and thickness, and the width of the center console unit C in FIG. It can be mounted on a relatively small vehicle or a vehicle that can utilize only a limited installation space in front of the shift lever.

  The input operation surface 102b of the touch panel 12 corresponds to the shooting range 102b of the hand shooting camera 12b, and when assuming an average hand of an adult, only a part of the longitudinal direction including the tip of the middle finger is included. The vertical direction (Y direction) dimension is determined so as to enter the input operation surface 102b (for example, 60 to 90 mm (as a specific example, 75 mm)). As a result, only the portion ahead of the base of the finger is displayed in the screen area of the monitor 15, so that the shape of the palm portion other than the finger is not involved in the display, and will be described later using the indicator image. The display process can be greatly simplified. In addition, the horizontal dimension (X direction) of the input operation surface 102b is 110 to 130 mm (for example, 120 mm), and the index finger, middle finger, ring finger, and little finger are within the imaging range when a hand is put on and each finger is wide open. The thumb is out of the shooting range. However, it is also possible to keep all fingers within the imaging range if each finger is properly closed.

  FIG. 3 is a block diagram showing an electrical configuration of the controller device 1. It is the operation ECU 10 that controls the operation device 1. The operation ECU 10 is configured as a computer hardware board mainly composed of the CPU 101. Specifically, the CPU 101, the RAM 1102, the ROM 103, the graphic controller 110, the video interface 112, the touch panel interface 114, the general-purpose input / output unit 104, and the serial communication interface. 116 has a structure interconnected by an internal bus 105. The graphic controller 110 includes a display video RAM 111 and a monitor 15, the video interface 112 includes a shooting video RAM 113 and a hand shooting camera 12 b, the touch panel interface 114 includes a touch panel 12 a, and the general-purpose input / output unit 104 includes a driver ( The illumination light source 12c is connected via a drive circuit 115. The serial communication interface 116 is connected to an in-vehicle serial communication bus 30 such as a CAN communication bus, and other ECUs connected to the network, specifically, a navigation ECU 200 for controlling a car navigation device and a body system. Mutual communication with the ECU 301 is possible.

  The video interface 112 is continuously input with an analog or digital video signal acquired by the hand-shooting camera 12b, and is captured as image frame data in the shooting video RAM 113 at predetermined time intervals. The content stored in the video RAM 113 for shooting is updated whenever new image frame data is fetched.

  The graphic controller 110 acquires the input screen image frame data from the navigation ECU 200 via the communication interface 116, and also acquires the indicator image frame data in which the indicator image is pasted at a predetermined position from the CPU 101 side. Frames are synthesized on the video RAM 111 by a known alpha blending process or the like, and output to the monitor 15.

  The touch panel interface 114 has a unique drive circuit corresponding to the method of the touch panel 12a, and detects an input position (touch input point) by a touch operation on the input operation surface 102b based on a signal input state from the touch panel 12a. The detection result is output as position input coordinate information.

  It should be noted that the shooting range (shooting field of view; as a result, the shot image acquired by the hand shooting camera 12b) of the hand shooting camera 12b, the input operation surface of the touch panel 12b, and the screen area of the monitor 15 (and input for determining the display contents). The screen image frame data and the indicator image frame data) have a uniquely defined two-dimensional coordinate correspondence.

The ROM 103 stores the following software executed by the CPU 101.
Touch panel control software 103a: The coordinates of the touch input point are acquired from the touch panel interface 114, and operation reference content determination reference information (for example, soft button area specifying information or , Including information such as operation command contents to be output when the soft button is touched. Then, the current operation input content is specified based on the touch input point coordinates and the acquired determination reference information, and an execution command output of the corresponding operation command is performed to the navigation ECU 200.

Display control software 103b: Instructs the graphic controller 110 to input input screen image frame data, and transfers indicator image frame data created by a method described later to the graphic controller 110.
Fingertip point calculation software 103c (tip extraction means, image tip position specifying means, fingertip determination means): Operates using the fingertip point calculation processing memory 1102a of the RAM 1102 as a work area. An operation for binarizing a photographed image of the hand of the operator photographed by the hand photographing camera 12b and specifying the fingertip position of the actual finger image as a photographing fingertip point, specifically, in the insertion direction of the binarized photographed image A calculation is performed to specify a predetermined representative point (here, the geometric gravity center position) of the tip region ta as the imaging fingertip point tp. Further, based on at least one of the size and area of the tip region ta, it is determined whether or not the photographing fingertip point tp indicates the true fingertip point tp. It should be noted that the binarization processing of the hand-photographed image may be performed in advance by incorporating a pixel binarization circuit into the output stage of the video interface.
Indicator image composition software 103d (indicator image display means): Operates the indicator image composition memory 1102b of the RAM 1102 as a work area. As shown in FIG. 9, the photographed hand image SF ′ may be displayed as an indicator image on the screen so as to be superimposed, or as shown in FIG. 10, the indicator image having a finger width narrower than that of the hand image. F1 may be created based on the indicator image data 103e prepared separately from the photographed image, and may be superimposed on the screen. In this case, the indicator image F1 is processed to be pasted on the indicator image frame so that the fingertip position coincides with the photographing fingertip point.

User interface engine 103e: An instruction input state on the operation screen by one or both of the touch input point and the shooting fingertip point, and an operation command output to the in-vehicle electronic device (the car navigation system and the car audio system in the present embodiment) The processing is performed when the relationship with the contents is set as a predetermined interface specification and the operation command information is output based on the interface specification in accordance with the instruction input state on the operation screen. Further, depending on the traveling state of the vehicle (particularly the vehicle speed), the number of shooting fingertip points, whether or not the touch operation on the touch input device 12 is detected at the same time when the shooting fingertip points are detected, etc. It has a function (interface specification changing means) for changing the interface specifications as appropriate.

  Hereinafter, how the shooting fingertip point of the hand to be shot is specified in the controller device 1 will be described using a flowchart. The principal part of the processing is performed by the fingertip point calculation software 103c. First, on the screen (area) of the monitor 15, the keyboard input screen (excluding the hand image SF) of FIG. 32 is displayed by a preceding command input (for example, based on a touch input operation on another screen). (However, it may be another input screen such as a map screen).

  FIG. 5 shows the flow of fingertip point identification processing (repeatedly executed in a certain cycle). In the above state, as shown in FIG. 2, the hand H is placed on the input operation surface 102b of the touch panel 12a. When approaching, the hand photographing camera 12b photographs a hand image based on the reflected light of the illumination light from the illumination light source 12c by the hand. In S101 of FIG. 5, the captured image is captured. The pixel of the hand image appears brighter than the background area by receiving the reflected light. Therefore, as shown in FIG. 4A, if the luminance of a pixel is binarized with an appropriate threshold, an area showing a high luminance pixel value (here, “1”) is a subject area (shown in black in the figure). The region indicating the low luminance pixel value (here, “0”) can be separated from each other as a background region (shown in white in the figure). In S102, the binarized image data is stored as first image data (A).

In S103, the area ratio σ of the subject area in the first image data is calculated. If the object does not exist in the imaging range of the hand photography camera 12b, because less than the threshold sigma ₀ there is the area ratio sigma of the subject region, in this case, it skips the following process.

  Next, in S105 of FIG. 5, the first image data is moved a predetermined distance in the palm length direction (Y direction) of the hand image (for example, 20% to 80% (actually, with the tip of the middle finger first joint as the fingertip portion). The image data obtained by translating the distance by about 5 to 20 mm) will be referred to as second image data shown in FIG. In S106, as shown in FIG. 4C, a non-overlapping region that appears on the fingertip side when the two image data are superimposed is specified as a tip region (fingertip region) ta in the hand insertion direction. By translating and superimposing the acquired original image data in the palm length direction of the hand image, the fingertip region can be easily specified as a non-overlapping region. In addition, even when some fingers are closed and in close contact, a rounded fingertip region can be reliably separated and specified.

  In C of FIG. 4, the second image data B is created by moving the first image data A backward in the palm length direction (Y direction) toward the wrist, and the hand image fingertip of the first image data A A non-overlapping region appearing on the side is specified as a tip region (fingertip region). The fingertip region can be specified on the first image data A in which the coordinate correspondence relationship with the imaging range (and thus the screen region of the monitor 15) is stored, and the fingertip point described later, and the corresponding coordinate point specification process on the screen region, which will be described later, is specified. Can be performed easily.

  The first and second image data are both binarized, and the non-overlapping area is specified by calculating the image difference between the first image data and the second image data. Thereby, the process of specifying the pixels in the non-overlapping area can be converted into a logical operation between corresponding pixels of the first and second image data (specifically, the exclusive OR between the corresponding pixels can be zero). It can be identified as a pixel in a non-overlapping area). Note that the non-overlapping region of the first and second image data may be narrowly formed on the side surface portion of the finger. However, such a side surface portion can be easily removed by performing a process of inverting those pixel columns by “0” when the number of consecutive “1” pixels in the X direction is less than a predetermined value. it can.

  Next, in S107 of FIG. 5, a contraction process as shown in D of FIG. 4 is performed on each fingertip region extracted as described above. Specifically, for all pixels having a value of “1”, pixels that have a certain adjacent relationship with this (specifically, four pixels on the top, bottom, left, and right, and further, four pixels that are diagonally adjacent to each other) If one pixel whose value is inverted with respect to the target pixel is included in the 8 pixels added to the above, a process of inverting the value of the pixel by “0” is performed. This process may be repeated a plurality of times as necessary.

  Then, after the contraction process, a process of separating the individual tip regions on the image data is performed. That is, as shown in FIG. 8, the image is scanned in a predetermined direction (for example, the x direction), and the same value is determined depending on whether or not a certain number of pixels are interrupted by a certain number (for example, 3 bits). A labeling code (represented by numerals such as 1, 2, 3,... In this embodiment) is applied to each pixel while discriminating between the tip region and another tip region. In the second and subsequent scans, when the detection state of the “0” pixel shifts to the detection of the “1” pixel, the labeling state of, for example, eight pixels surrounding the “1” pixel is discriminated, and already recognized. If a pixel labeling code is detected, the same labeling code is applied, and if nothing is detected, a new labeling code is applied. Then, pixel sets with different labeling codes are recognized as different tip regions.

Next, the process proceeds to S108, and a determination process is performed as to whether or not each separated tip area is a true fingertip area (fingertip determination process). Basically, as shown in FIG. 7, the dimension L in the finger width direction (hereinafter also simply referred to as “width”) L of the specified tip region ta is determined based on the width dimension of a general adult. This is a process of determining a true fingertip region on the condition that it is within a specified range (upper limit value W _th1 , lower limit value W _th2 ). As shown in FIG. 1, while looking at a monitor (display device) 15 disposed on the instrument panel, a touch panel disposed on a center console C on the rear side (front side as viewed from a seated passenger) in the vehicle length direction. The operation input surface 102a of 12a, and by extension, the shooting range 102b of the hand shooting camera 12b for shooting the operator's hand on the operation input surface 102a from below is inserted forward from the rear edge side of the shooting range 102b. . Therefore, the insertion direction of the hand is assumed to be in a direction (that is, the Y direction) orthogonal to the long side of the horizontal rectangular imaging range 102b, and the finger width direction is orthogonal to the insertion direction of the hand in the plane of the operation input surface 102a. Direction, i.e., the direction that coincides with the long side of the imaging range 102b (that is, the X direction). That is, the width of the tip region ta is fixedly measured in the X direction, that is, the long side direction of the imaging range 102b.

FIG. 6 is a flowchart illustrating an example of details of the fingertip determination process.
In S <b> 1, the width W of each separated tip region ta is specified. Specifically, it can be calculated by W = X _max −X _min where X _{min is the} minimum value and X _max is the maximum value among the X coordinates of the individual constituent pixels of the tip region ta. In S1002, it is determined whether or not the width W of the tip region ta specified as described above is within the specified range. The touch panel 12a is arranged on the center console C, but a driver or a passenger seated on the side of the touch panel 12a often uses the center console C as a storage. At this time, if an article other than the hand is placed in the operation input surface 102a of the touch panel 12a, that is, the photographing range 102b, the article is naturally photographed instead of the hand of the passenger. If a determination method using the width W of the tip region ta extracted as a difference image between the first image captured and acquired and the second image obtained by translation in the Y direction is employed, documents, books, etc. If placed, the width of the tip area that is extracted and specified clearly exceeds the upper limit value of the specified range based on the general adult width dimension, so it can be reliably determined as a non-fingertip area. It is. In addition, when a mobile phone is placed, for example, the width of the first tip region derived from the protruding antenna part is below the lower limit of the specified range because the antenna is clearly narrower than the finger, and determined as a non-fingertip region Can do.

  In the case where a true hand is detected, one or two fingers (for example, only the index finger or the index finger and the middle finger) are stretched and the remaining fingers are bent. (For example, when gripped like a fist), the width of the tip region of the closed part exceeds the upper limit of the specified range, and the width of the extended finger is within the specified range It can happen. Therefore, for an article having a shape from which a plurality of tip regions can be extracted, if the width is within a specified range for at least one of the plurality of tip regions, the tip region is determined as a true fingertip region. It can be configured as follows.

  Next, there is a case where an article having a shape that can be mistaken as a true fingertip region because the width W is within a specified range although it is a non-fingertip region is placed on the input operation surface 102b of the touch panel. For example, when money is placed on the input operation surface, the size of the money is close to the finger width, so the width of the tip area after the contraction process is within the specified range, and it is erroneously determined as the fingertip area as it is. The The difference between the image of a finger and money is that, in the case of a finger, the image area on the base side of the finger extends to the rear edge (edge on the hand insertion side) of the shooting range, whereas in the case of money, the image is taken. This is a circular area isolated within the range, and the point between the rear end edge of the circular area and the rear end edge of the shooting range is the background area of the subject (that is, the “0” pixel area). Therefore, if the total area of the captured image is S, the total distance from each non-overlapping area to the rear edge of the imaging range is d, and the value of S / d is estimated as the finger width, the erroneous determination as described above Can be effectively avoided. That is, in the case of money, the total area S decreases because the background region is interposed on the rear edge side of the photographing range. Therefore, if the estimated finger width value calculated by S / d is less than the lower limit value of the specified range, this can be excluded as a non-fingertip region. In the flowchart of FIG. 6, the determination based on the above principle is performed in S1005 and S1006.

  Next, in S1007 of FIG. 6, a representative point is determined for the tip region ta that is not determined to be excluded in S1002 and S1006. In the present embodiment, the geometric gravity center position G of the region is used as a representative point. The calculation method is well known. For example, the sum of the X coordinate value and the Y coordinate value of all the pixels constituting the tip region is obtained. The coordinates of the gravity center position G can be calculated by dividing by the number of pixels. Note that, instead of the center-of-gravity position G, for example, the pixel having the maximum Y coordinate value may be used as the representative point.

  Note that since the touch panel input operation surface 102b actually comes in contact with the finger pad area slightly lowered in the X direction from the tip of the finger, in FIG. A predetermined distance is offset, and this is determined as the image fingertip point G. However, the barycentric position of E may be used as it is as the image fingertip point G, and in this case, the processing of F is unnecessary.

  Next, depending on the positional relationship of the hand with respect to the shooting range 102b, there may occur a case where the representative point determined by the above algorithm using the difference image does not correspond to the true fingertip point. Specifically, this is a case where the fingertip portion is located outside the imaging range. When the corresponding coordinate ranges of the shooting range 102b, the input operation surface 102b of the touch panel, and the screen of the monitor 15 coincide with each other, the true fingertip position is the shooting range 102b (and thus the input operation surface 102b and the monitor 15). The tip region identified from the difference image in both the case where the image is within the outer peripheral area of the screen) and the case where the fingertip part protrudes outside the imaging range 102b and the true fingertip part is imagewise interrupted. Is in a shape that fits in the outer peripheral region. And even when the fingertip part protrudes, it is still a part of the image of the finger, and it is highly possible that the area width falls within the specified range. This leads to misunderstanding as an area.

  Therefore, in the present embodiment, a configuration is adopted in which the outer peripheral area of the effective coordinate range of the imaging range 102b is used as an imaging area outside display that extends outside the effective coordinate range on the display screen of the monitor 15. Since the finger portion that protrudes outside the screen forms a subject area within this non-display shooting area, the tip area specified based on the difference image and the shooting fingertip point specified by the representative point also falls within this non-display shooting area. Occur. On the other hand, as shown on the right in FIG. 13, when the true fingertip does not enter the imaging area outside the display and stays in the outer peripheral area of the screen, the tip area and the imaging fingertip point also appear in the screen. Accordingly, if the tip area ta exists in the non-display shooting area 102e, it is not recognized as a true fingertip (invalid), and if it does not exist in the non-display shooting area 102e, it is recognized as a true fingertip. It is possible to perform processing so as to be (valid). For example, as shown in FIG. 14, even when a plurality of shooting fingertip points tp are specified, it is possible to individually determine validity / invalidity based on whether or not they exist in the non-display shooting area 102e. In the flowchart of FIG. 6, determination based on the above principle is performed in S1008 to S1010. Note that the rear end edge in the Y direction of the display screen on the hand insertion side may coincide with the rear end edge of the imaging range 102b, and the non-display imaging area 102e may not be formed.

In addition to the above, various methods can be adopted as an algorithm for determining whether or not the tip region ta is a true fingertip region. For example, when the Y-direction parallel movement distance of the first image for obtaining the second image is set to be smaller than a general adult finger width value, the tip region obtained as a difference image between the first image and the second image ta tends to be a horizontally long region in which the X-direction dimension W _X (that is, the width) is larger than the Y-direction dimension W _Y. Therefore, it is possible to determine whether or not it is a true fingertip region based on whether or not the X / Y aspect ratio φ (≡W _X / W _Y ) of the tip region ta is within a specified range. For example, in the case of the paper or document shown on the left in FIG. 10, φ is extremely large, and for the mobile phone antenna shown on the right (thinner than the finger), the value of φ is also reduced by the shortage of the X-direction dimension W _X. Since they become smaller, any of them can be excluded as a non-fingertip region.

In consideration of the case where the finger is inserted with an inclination with respect to the Y direction, as shown in FIG. 16, various parallel line pairs circumscribing the tip region ta are generated while changing the angle, and the maximum parallel line is obtained. The aspect ratio φ may be calculated by W _max / W _min , where the distance between the lines is W _max and the distance between the parallel lines that is the minimum is W _min .

  In addition, as shown in FIG. 17, the total area of the subject area (“1” pixel area) of the captured image is S, and the number of identified tip areas (non-overlapping areas) ta is N. Can be estimated as an average finger area, and it can be determined whether or not the tip region ta is a true fingertip region based on whether or not the S / N is within a specified range. This is particularly effective when the Y-direction dimension of the imaging region is set to a value that can include only a part of the finger tip side so that the subject region is mainly formed only by the finger region when the hand is photographed. It can be said that it is a technique.

  Returning to FIG. 5, when the fingertip determination process in S108 is completed as described above, it is determined in S109 which of the identified tip areas is determined as the true fingertip area, and in S110, the true fingertip area is determined. Only the coordinates of the representative point (centroid coordinates G) are stored as (true) imaging fingertip points, and those that are not true fingertip areas are discarded or invalidated.

  Next, various embodiments of the present invention realized by the user interface engine 103e will be described. As shown in FIG. 3, the operation ECU 10 acquires vehicle speed information detected by the vehicle speed sensor 301 from the body ECU 300 via the serial communication bus 30 and is set to a threshold vehicle speed (1 to 10 km / hr, for example, 5 km / hr). ) Is determined as a low-speed traveling state (stopped state), and if it exceeds the threshold vehicle speed, it is determined as a high-speed traveling state (traveling state).

  There are two types of interface specification data for the user interface engine 103e: low-speed driving interface specification (stop interface specification) data and high-speed driving interface specification (driving interface specification) data. However, in the running state, the latter is switched and used. FIG. 11 shows the first reference form. Specifically, the number of display operations in the effective operation areas SB and FHW in the running interface specification (right) is the effective number in the stopping interface specification (left). An example in which the number is set to be smaller than the number of display formations in the operation area SB is shown. The application target is a character input screen for inputting characters such as a destination of a car navigation system and a waypoint.

  In the stop-time interface specification, the operation area is an individual character input soft button SB constituting the character input keyboard KB. The image fingertip point is positioned on the soft button of the desired character, and the touch operation is performed at the corresponding position on the touch panel 12a (that is, the touch input point is generated in the soft button area). Character input is performed, and the code of the corresponding character is output as operation command information and transmitted to the car navigation system.

  On the other hand, in the running interface specification, a keyboard is not displayed, and a freehand input window FHW for handwritten input is displayed and formed as an operation area. The freehand input window FHW is formed to be larger than the character input soft button SB in the stop-time interface specification, and is formed to be approximately the same size as the character input keyboard KB, for example. A temporal change of either the touch input point or the image fingertip point is captured as a drawing trajectory, and the drawing trajectory is compared with individual character shapes on the character list, and a matching character code is output as operation command information. Sent to the car navigation system. Note that the soft button SB in the running interface specification only displays and forms a delete button for erasing a history of input during freehand and a confirm button for confirming the freehand input content.

  Next, FIG. 12 shows the second reference form. Specifically, the display formation dimensions of the effective operation area in the driving interface specification (right) are shown in FIG. An example in which the display area is set to be larger than the display formation size of the operation area is shown. The application target is a menu screen for setting the destination of the car navigation system.

  In the interface specification when the vehicle is stopped, the operation area is an individual menu button SB1. Here again, the image fingertip point is positioned on the soft button of the desired menu, and the touch operation is performed at the corresponding position on the touch panel 12a (that is, the touch input point is generated in the soft button region). The corresponding menu is selected, and the contents are output / transmitted as operation command information to the car navigation system.

  On the other hand, in the running interface specification, the number of menu buttons SB1 is reduced, and a lower menu button (here, a destination candidate) SB2 having a specific menu (here, destination history) as an upper layer is an interface when stopped. It is displayed and formed horizontally longer than the menu button SB1 in the specification. The image fingertip point is positioned on the soft button of the desired menu, and the touch operation is performed at the corresponding position on the touch panel 12a (that is, the touch input point is generated in the area of the soft button). The menu is selected, and the contents are output and transmitted to the car navigation system as operation command information.

  FIG. 13 shows a third reference form. Specifically, as the number of detected image fingertip points G increases, the types of operation areas to be displayed and formed on the operation screen are increased, and the running interface specifications are shown. An example is shown in which the maximum number of display formation types in the operation area is set smaller than the maximum number of display formation types in the operation area in the stop-time interface specification. The application target is a playback operation screen of a car audio system.

  In the stop-time interface specification, when the number of detected image fingertip points G is one as the operation area, the play / stop button P / S is displayed at a position corresponding to the image fingertip points G although not shown. . When the number of detected image fingertip points G is two, the fast forward button FF and the reverse button RF are displayed at positions corresponding to the image fingertip points G as shown in the upper right of FIG. Furthermore, when the number of detected image fingertip points G is 3, as shown in the upper left of FIG. 13, the fast forward button FF, the play / stop button P / S, and the reverse button RF are set to each image fingertip point G. Display at the corresponding position. In any case, by performing a touch operation on the touch panel 12a with any finger, the function of the corresponding button is selected, and the content is output as operation command information. Here, the maximum number of display formation types in the operation area is “3” when the number of detected image fingertip points G is three.

  On the other hand, in the running interface specification, regardless of the number of detected image fingertip points G, the effective operation area is only the play / stop button P / S (that is, the maximum number of display formation types is “1”). "). In the lower left of FIG. 13, the number of detected image fingertip points G is three, but only one play / stop button P / S forming the operation area is displayed at a position off the image fingertip points G. Is formed. In the lower center of FIG. 13, the play / stop button P / S is individually displayed on each image fingertip point G (that is, the number of image fingertip points G). That is, a plurality of operation areas are displayed corresponding to the image fingertip point G, but the operation area type is still “1”. In any case, the indicator image FI is individually displayed at each image fingertip point G. On the other hand, in the lower right of FIG. 13, even if a plurality of image fingertip points G are detected, only one indicator image FI is displayed, and only one operation region is displayed correspondingly. Yes. In either case, only the play / stop button P / S is operated regardless of which finger is used to perform the touch operation.

  FIG. 14 shows a fourth reference embodiment. The application target is the car audio system music selection screen, and the audio file group to be played has a hierarchical menu structure in which multiple recorded songs (second hierarchical menu title) are grouped for each album (first hierarchical menu title). Have. The title image of the currently selected album is displayed in the current album window AW (an operation area that is valid on the screen), and the corresponding album name is output as an operation command output, that is, album selection information. In this embodiment, a preparation album window group AW ′ indicating album groups arranged in front and back is displayed and formed in a row on the left and right sides of the album window AW (partially overlapping). The album name currently selected is displayed in the album name window ATW.

  On the screen, a drag operation along the operation surface 102b of the touch input device 12 is detected (here, the drag operation is detected by the time change of the touch input point TP, but the drag operation is detected by the change of the photographing fingertip point. May be detected). Then, when a drag operation is performed in the arrangement direction of the album window groups AW and AW ′, the album title images corresponding to the prepared album window group AW ′ are sequentially switched and displayed in the current album window AW (according to this, the album name is displayed). The album names displayed in the window ATW are also sequentially switched). If the direction of the drag operation changes, the album switching direction is also reversed.

  In the stop-time interface specification, as shown in the upper part of FIG. 14, the album title image (operation area) is continuously switched while one drag operation is continued. The lock mark URM is also displayed in the form of “unlocked” indicating that continuous switching is possible. On the other hand, in the running interface specification, only one title image (operation area) of the album is switched (that is, intermittently) every time a drag operation occurs. The lock mark URM is displayed in the form of “locking” indicating that continuous switching is impossible.

  FIG. 15 shows a fifth reference embodiment. In FIG. 14, when the current album window AW is touched, the recorded song list window LW of the currently selected album is displayed. Each song name display area on the list forms a soft button for song selection, and is output as an operation command output, that is, music selection information by a touch operation. When a drag operation is performed in the arrangement direction (in this case, the vertical direction) of the song title display area on the recorded song list window LW, the song title display area in the recorded song list window LW is scrolled sequentially in the corresponding direction. In the stop-time interface specification, as shown in the upper part of FIG. 15, the scroll is performed so that the music title display area is continuously switched while one drag operation is continued. On the other hand, in the running interface specification, as shown in the lower part of FIG. 15, every time a touch operation occurs, the direction mark BA on the screen closer to the touch input point is selected in the direction indicated. Is moved by one (that is, reproduction processing is performed by switching to the next song of the currently played song on the corresponding direction mark BA side).

  FIG. 16 shows a sixth reference embodiment. The electronic device to be applied here is an air conditioner. The left side of FIG. 16 shows an interface specification when the vehicle is stopped, and a mode switching button SB for performing various mode switching operations of the air conditioner is displayed as a plurality of operation areas. If a touch input point is detected in any of the mode switching buttons SB, a corresponding mode switching signal is output as second operation command information. On the other hand, a set temperature switching operation area GA1 and an air flow switching operation area GA2 are formed in the screen. In any of these cases, by performing a drag operation (operation operation pattern: gesture) in a specified direction within the region, the set temperature or the air volume can be operated according to the position change amount of the touch input point or the image fingertip point. It is continuously changed to the corresponding direction, and information indicating the change contents is output as the first operation command information (that is, both the first operation command information and the second operation command information are activated). . Further, the right side of FIG. 16 shows the travel interface specifications, the mode switching button SB is omitted (or invalidated), and only the set temperature switching operation area GA1 and the air volume switching operation area GA2 are formed (that is, Only the first operation command information is activated).

  Next, FIG. 17 shows the first embodiment of the present invention, and the application target is a menu screen for setting the destination of the car navigation system. The operation area is an individual menu button SB1, and the image fingertip point is positioned on the soft button of the desired menu, and the corresponding menu is selected by touching the corresponding position on the touch panel 12a. The contents are output and transmitted to the car navigation system as operation command information. The left side of FIG. 17 shows the use of the interface when a plurality of shooting fingertip points are detected simultaneously, and the right side shows the use of the interface when only one shooting fingertip point is detected. The number of display formations of the menu button SBS is set to be larger than the number of display formations of the menu button SBL when only one is detected. In the latter case, as the number of display formations of the menu button SB1 decreases, the formation size of each menu button SB1 is enlarged.

  18 to 20 show a first embodiment according to the second aspect of the present invention, and whether or not a touch operation on the touch input device 12 is detected at the same time when a photographing fingertip point is detected. Accordingly, the interface specification of the user interface engine 103e is switched. The application target is a map display screen of a car navigation system, and a map display area MA and scroll buttons SB are formed. As shown in FIG. 18, a map scroll operation in the map display area MA can be performed by touching the scroll button SB.

FIG. 19 shows an interface specification when a drag operation with a touch is applied to the map display area MA with any one finger. In this case, the map is dragged in the map display area MA. It is scrolled at a first speed V ₁ in the direction. On the other hand, FIG. 20 shows the interface specifications when a drag operation is performed without touching the map display area MA (that is, in a state where only the photographing fingertip point G is specified). In this drag operation, taking into account that the shooting fingertip points G of a plurality of fingers are detected, tracking of the first detected shooting fingertip point G or temporal change in the number of detected shooting fingertip points (0 → Detection can be performed in any of 1 → 2 → 3 → 2 → 1 → 1). In the specification, the map is scrolled in the drag operation direction in the map display area MA at the second speed V ₁ larger than the first speed V ₁ described above.

  FIGS. 21 to 23 show a second embodiment according to the second aspect of the present invention, and the application target is a music selection screen of the car audio system similar to FIGS. 14 and 15. As shown in FIG. 21, the current album window AW and the prepared album window group AW ′ are displayed and formed in the same manner as in FIG. 14, and the recorded song list window LW of the album is aligned with the lower side of the current album window AW. Is displayed. FIG. 22 shows an interface specification when a drag operation is performed without a touch on the map display area MA (that is, in a state where only the photographing fingertip point G is specified). By performing the drag operation in the arrangement direction of the album window groups AW and AW ′, the title images of the albums corresponding to the prepared album window group AW ′ are sequentially switched and displayed on the current album window AW (according to this, the recorded music is recorded). The recorded songs displayed in the list window LW are also switched sequentially). If the direction of the drag operation changes, the album switching direction is also reversed. FIG. 23 shows interface specifications when a touch operation is performed. If the song name display area in the recorded song list window LW is touched, the performance of the touched song name is switched. Note that when the current album window AW is touched, the performance may be switched to the next song of the currently selected one.

1 operation device 10 operation ECU
12 Input unit 12a Touch input device 12b Hand-held camera (shooting device)
15 Image display device 101 CPU (shooting fingertip point detection means, pointer image display control means, screen creation display control means)
103e User interface engine 301 Vehicle speed sensor (traveling state detecting means)
SF, SI Indicator image tp Touch input point G Shooting fingertip point

Claims (5)

An operating device that is mounted on a vehicle and used to input operation of an in-vehicle electronic device,
A touch input device that has an operation surface that accepts a touch operation by a fingertip of an operator and detects a touch input point by the touch operation;
An imaging device that has an imaging range having a unique coordinate correspondence with the operation surface and that captures the operator's hand approaching the operation surface;
An imaging fingertip point detection unit configured to be capable of simultaneously detecting a plurality of fingertip points of the hand as imaging fingertip points based on hand image data acquired and acquired by the imaging device;
An image display device having a display screen having a unique coordinate correspondence with the operation surface and the imaging range;
A pointer image display control means for displaying a pointer image indicating the fingertip point on the display screen with respect to the image display device;
On the image display device, an operation screen display control means for displaying an operation screen unique to the in-vehicle electronic device to be operated,
A relationship between an instruction input state on the operation screen by one or both of the touch input point and the photographing fingertip point and an output content in the operation command information to the in-vehicle electronic device is set as a predetermined interface specification A user interface engine that outputs the operation command information based on the interface specifications according to the instruction input state on the operation screen;
Interface specification changing means for changing the interface specification of the user interface engine according to the number of the shooting fingertip points detected by the shooting fingertip point detection means;
An in-vehicle electronic device operating device comprising: The operation screen display control means displays a plurality of operation areas for performing different operation command outputs on the operation screen,
The interface specification changing unit is configured to display the number of effective display areas of the operation area when the number of the shooting fingertip points detected by the shooting fingertip point detection unit is plural, and the number of the shooting fingertip points is also one. The in-vehicle electronic device operating device according to claim 1, wherein the operating device is set to be larger than the effective number of display formations in the operation area. An operating device that is mounted on a vehicle and used to input operation of an in-vehicle electronic device,
A touch input device that has an operation surface that accepts a touch operation by a fingertip of an operator and detects a touch input point by the touch operation;
An imaging device that has an imaging range having a unique coordinate correspondence with the operation surface and that captures the operator's hand approaching the operation surface;
Photographing fingertip point detecting means for detecting the fingertip point of the hand as a photographing fingertip point based on hand image data acquired by the photographing device;
An image display device having a display screen having a unique coordinate correspondence with the operation surface and the imaging range;
A pointer image display control means for displaying a pointer image indicating the fingertip point on the display screen with respect to the image display device;
On the image display device, an operation screen display control means for displaying an operation screen unique to the in-vehicle electronic device to be operated,
A relationship between an instruction input state on the operation screen by one or both of the touch input point and the photographing fingertip point and an output content in the operation command information to the in-vehicle electronic device is set as a predetermined interface specification A user interface engine that outputs the operation command information based on the interface specifications according to the instruction input state on the operation screen;
Interface specification changing means for changing the interface specification of the user interface engine according to whether or not a touch operation on the touch input device is simultaneously detected when the photographing fingertip point is detected;
An in-vehicle electronic device operating device comprising: The operation screen display control means detects a drag operation along the operation surface of the touch input device, and scrolls display processing of image information being displayed on the screen of the image display device in a direction corresponding to the drag operation. Is what
The interface specification changing means sets a scroll speed of the image information when the touch operation is detected together with the photographing fingertip point to be smaller than a scroll speed when the touch operation is not detected. The vehicle-mounted electronic device operating device as described. The operation screen display control means detects a drag operation along the operation surface of the touch input device, and scrolls and displays a hierarchical information output menu being displayed on the screen of the image display device. ,
4. The interface specification changing means performs scroll display processing on different levels of the information output menu depending on whether the touch operation is detected or not detected. In-vehicle electronic device operating device.