JP2010083206A - Device for operating on-vehicle electronic appliance - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To effectively use photographed and obtained image information also as input information in a device for operating an on-vehicle electronic appliance with both of an operation input part and a hand photographing means. <P>SOLUTION: In the device 1 for operating the on-vehicle electronic appliance with both of a touch panel 12a and a hand photographing camera 12b, an image including the hand of an occupant approximating to the touch panel 12a is photographed in a specific photographing area in anticipation of an operation input face, and the hand image area F1 of the operator is specified on the photographed image. Then, the rate of the area of the specified hand image area F1 relative to an operation input area is calculated as a ratio of a hand image area, and based on the operation condition to the touch panel 12a and the calculated ratio of the hand image area, operation input information to the on-vehicle electronic appliance is generated and output. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an on-vehicle electronic device operating device.

JP 2000-335330 A Japanese Unexamined Patent Publication No. 2000-6687

  Conventionally, as an operation device for an in-vehicle device such as a car navigation device, an operator's hand is photographed with a camera, a finger image is extracted from the photographed data, and the extracted finger image area is displayed on a GUI input screen of the in-vehicle device (for example, A type of display that is superimposed on a navigation screen) has been proposed. For example, in Patent Document 1, a camera attached to the ceiling of a vehicle body is used to photograph the state of a hand operating a switch panel at the seat, and the photographed hand and the switch panel are placed opposite to the front of the operator. A configuration for displaying on a liquid crystal panel is disclosed. Patent Document 2 discloses a configuration in which a driver's hand is photographed using a video camera arranged on the roof of a vehicle to extract a contour, and the contour image is superimposed on a soft button on a display. Yes.

  In the above-described conventional operation system, the shooting information of the hand by the camera is only used for superimposed display on the screen of the hand outline image indicating the operation position, and the image information itself is effectively used as input information. It was difficult.

  An object of the present invention is to make it possible to effectively use image information acquired and acquired as input information in an in-vehicle electronic device operating device provided with both an operation input unit and hand-photographing means. It is to plan the expansion.

Means for solving the problems and effects of the invention

In order to solve the above-described problems, an in-vehicle electronic device operating device according to the present invention includes:
An operation input unit provided at a position operable by an occupant seated in a vehicle seat, and having an operation input area of a certain area;
Hand photographing means for photographing an image including an occupant's hand approaching the operation input unit in a certain photographing range for expecting the operation input region;
Hand image area specifying means for specifying the operator's hand image area on the captured image;
A hand image area ratio calculating means for calculating an area ratio of the specified hand image area to the operation input area as a hand image area ratio;
Based on the operation state to the operation input unit and the calculated hand image area ratio, operation input information generation output means for generating and outputting operation input information for the in-vehicle electronic device,
It is provided with.

  According to the configuration of the present invention, in the vehicle-mounted electronic device operating device including both the operation input unit and the hand photographing unit, the vehicle is mounted on the basis of the operation state to the operation input unit and the calculated hand image area ratio. Since the operation input information for the electronic device is generated and output, in addition to the input information from the operation input unit, the image information acquired and acquired can be used effectively as the input information. Significant expansion can be achieved.

  The operation input information generation / output means can handle the calculated hand image area ratio value by establishing a unique relationship between the value of the hand image area ratio and the content of the operation input information to be generated. The operation input information to be specified can be specified based on the relationship and output. By determining the contents of the operation input information in advance according to the value of the hand image area ratio, the operation input information to be output can be easily specified.

  The operation input information generating / outputting unit is a specification for requesting activation of a specific function of the in-vehicle electronic device as the operation input information when the calculated hand image area ratio value exceeds a predetermined threshold area ratio. The function activation request information can be output. As a result, a characteristic input operation in which the hand image area ratio is equal to or greater than the threshold area ratio can be determined as an operation for calling a specific function of the in-vehicle electronic device, and the specific function is intuitively activated by a simple operation. It becomes possible. For example, if the threshold area ratio is set so high that it can be clearly distinguished from general-purpose input operations that are normally used for operation input, such as simple finger touch operations, a specific function is activated at an undesired timing. It is possible to make it difficult for erroneous operations to occur. As an example, the operation input information generating / outputting unit includes a hand for an operation input area whose hand image area ratio exceeds a threshold area ratio (preferably 70% or more: for example, 80%) set to a predetermined value of 60% or more. When the blocking state is specified, the specific function activation request information can be output.

  Further, the operation input information generation / output means activates the specific function in the in-vehicle electronic device when a predetermined operation input is applied to the operation input unit after the specific function is activated in the in-vehicle electronic device. Operation change request information for changing the state can be output. Since the operation state of the specific function activated based on the photographed image based on the hand image area ratio can be changed based on the input from the operation input unit, it is possible to further increase the control variation related to the operation of the specific function.

  For example, when the operation input is applied, the operation input information generation / output unit interrupts the operation of the specific function in the in-vehicle electronic device and outputs the operation return request information for returning to the state before starting the specific function. It can be constituted as follows. The operation of the specific function activated by the hand image area ratio can be interrupted smoothly and easily using the operation input to the operation input unit as a trigger.

  In addition, the vehicle electronic device operating device according to the present invention specifies a series of time-varying behavior corresponding to a predetermined input hand movement on the operation input area of the hand image area ratio calculated by the hand image area ratio calculating means. It is possible to provide a hand image area ratio time change behavior specifying means. In this case, the operation input information generation / output means is configured to generate and output corresponding operation input information when the time-change behavior of the specified hand image area ratio matches a predetermined change behavior. be able to. By specifying the hand image area ratio time-changing behavior and generating the corresponding operation input information when the specified content matches the predetermined changing behavior, more characteristic input hand movement can be performed. It can be associated with the operation input information, and a more intuitive input operation is possible. In particular, it also specifies the temporal change in the center of gravity position of the hand image area, and responds when both the temporal change in the area ratio and the temporal change in the center of gravity position match a predetermined change behavior. By configuring so as to generate the operation input information to be performed, it is possible to more accurately specify the movement of the hand defined as the specific input operation. In addition, it is highly discriminable from the general-purpose input operation, and it is possible to prevent an erroneous operation that activates the specific function at an undesired timing.

  On the other hand, the operation input area can be divided into predetermined sub areas, and the hand image area ratio calculating means can be configured to calculate the area ratio of the hand image area for each sub area. In this case, the hand image area ratio time change behavior specifying means can be configured to specify a series of time change behaviors of the hand image area ratio in each sub-region. By specifying a series of time-varying behavior of the hand image area ratio for each of the plurality of sub-regions, it is possible to specify the above-described input hand movement more finely.

  Further, the hand image area ratio time change behavior specifying means may be configured to specify the transition behavior of the number or position in the operation input area of the sub area where the area ratio of the hand image area exceeds the threshold value. In this case, the operation input information generation / output unit can be configured to generate and output the corresponding operation input information when the specified transition behavior matches a predetermined transition behavior. According to this configuration, the movement of the operator's hand moving on the operation input area without specifying the center-of-gravity change of the hand image area, the transition of the number or position of the sub areas where the area ratio of the hand image area exceeds the threshold value It can be specified by the behavior, and the hand movement related to the operation input can be specified more finely and easily.

  As a specific example, the operation input area can be divided into a plurality of sub areas arranged adjacent to each other in a predetermined direction on the operation input area. Then, the sub-region hand image area ratio time change behavior specifying means, the sub-region where the area ratio of the hand image region exceeds the threshold value is the first state sub-region, and the sub-region where the area ratio is less than the threshold value is the second state sub-region The time change behavior related to the distribution state on the operation input region of the first state subregion and the second state subregion is specified as the subregion state distribution transition behavior, and the operation input information generation / output means is specified. When the sub-region state distribution transition behavior matches a predetermined behavior, the corresponding operation input information can be generated and output. By coding the state of the sub-region based on whether or not the area ratio of the hand image region exceeds a threshold value, the state distribution state of the sub-region can be simply expressed by macro bitmap information with each sub-region as a unit.

  In this case, the hand image area ratio temporal change behavior specifying means specifies the temporal change of the appearance position of the first state subregion or the second state subregion on the operation input region as the subregion state distribution transition behavior. Can be configured. By specifying the temporal change in the appearance position of the first state subregion or the second state subregion, the user's input hand movement can be easily specified. In this case, if the configuration is such that the temporal change in the number of first state subregions or second state subregions occupying a plurality of subregions is specified as the subregion state distribution transition behavior, the user's input manual operation is further finely defined. Can be identified. For example, the hand image area ratio time change behavior specifying means operates as the sub-region state distribution transition behavior corresponding to the input hand motion crossing the operation input region from the first end edge side to the second end edge side in a predetermined direction. In the sub-region group arranged in the direction corresponding to the input hand action on the input region, the movement behavior of the appearance position of the first state sub-region in the sub-region arrangement direction can be specified. According to this method, the input hand movement across the operation input area can be easily specified by the movement behavior of the appearance position of the two-state sub area in the sub area arrangement direction.

  The operation input unit has a transparent input support plate that accepts a touch operation by the operator's fingertip while supporting one of the plate surfaces as an operation input region on the operation input region side, and corresponds to the screen region. An input coordinate system is set along the plate surface of the transparent input support plate, and it can be configured as a position input device that detects an input position by a touch operation and outputs the coordinate information. The hand photographing means can be configured to photograph an image including a hand on the transparent input support plate from the other plate surface side of the transparent input support plate. Further, a display device that displays an input screen for reference when performing an input operation by the position input device, and a position corresponding to the hand image area of the image photographed by the hand photographing means with respect to the screen area of the input screen. In addition, a pointer image display unit that superimposes and displays a pointer image based on the image information of the hand image area can be provided.

  Thus, the operator can grasp the finger position on the operation input surface from the indicator image on the screen, and can accurately perform the input operation according to the screen display content. In particular, a configuration in which the screen area (screen) of the display device is arranged with respect to the operator so as not to face the operator directly viewing the finger on the operation input surface (for example, an in-vehicle electronic device such as a car navigation system) In the case of the operation device, the operation input surface is arranged on the left and right sides (or diagonally forward) of the seat on which the operator is seated, and the screen of the display device is above the operation input surface and forward or diagonally of the operator. In the configuration facing the front), both the hand performing the operation and the screen cannot be seen directly, so the operator can use the indicator image on the screen as the only information source for grasping the hand operation position. .

  In this case, illumination means for projecting illumination light from the other plate surface side of the transparent input support plate to the imaging side surface of the hand can be provided. The hand photographing means can be configured to photograph a hand image based on the reflected light of the illumination light by the hand. In the present invention, based on the area ratio of the hand image area to the operation input area, image information acquired by photographing is also used as input information. And, by providing the illumination means as described above, when the hand is relatively close to the transparent input support plate, the amount of reflected light reaching the hand photographing means increases, and the hand image can be detected effectively. When the hand moves away from the transparent input support plate, the amount of reflected light decreases, and a hand that is more than a certain distance away is not recognized as a hand image. Therefore, for example, when a hand crosses over the transparent input support plate in order to operate another operation unit (for example, shift lever) near the operation device, the distance from the hand to the transparent input support plate is sufficiently large. A hand image is not detected with an effective area ratio, and erroneous input is less likely to occur.

  The display device can be determined as one of in-vehicle electronic devices (for example, a car navigation system) together with the display control means, and the operation input information generation / output means performs display control according to the calculated hand image area ratio. The screen display content change command information for instructing the means to change the screen display content for the display device can be configured to be output as operation input information. Thereby, change control of the screen display content according to the hand image area ratio calculated based on the photographed image becomes possible, and the degree of freedom of the screen display change control mode can be greatly increased.

  As an example, the operation input information generation / output unit displays the first screen displayed before the transition when the hand image area ratio transitions from a state where the hand image area ratio is less than a predetermined threshold to a state where the hand image area ratio exceeds the threshold. The screen switching command information for switching to a second screen different from the first screen can be configured to be output as screen display content change command information. As a result, the switching operation to the specific screen can be performed by a characteristic operation mode corresponding to the hand image area ratio, and the intuitive and easy-to-understand screen switching operation (and other electronic devices (for example, an audio device or an air conditioner) linked thereto). Operation switching operations)), and is highly discriminable from general-purpose input operations such as simple touch operations, and erroneous operations that cause screen switching (that is, activation of a specific function) at an undesired timing. Can be made difficult. On the other hand, the operation input information generation / output means is a screen return request for returning the display screen of the display device to the first screen when a touch operation to the position input device occurs in a state where the display device is switched to the second screen. Information can also be configured to be output as operation input information. Accordingly, the second screen that is temporarily activated as the specific function can be easily returned to the first screen again with a simple touch operation on the position input device as a trigger.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 shows an example of a vehicle-mounted electronic device operating device (hereinafter also simply referred to as an operating device) according to the present invention. The operation device 1 has a monitor (display device) 15 disposed in the center of the instrument panel in the interior of the automobile, and the operation unit 12 is located at a position where the center console C can be operated from either the driver seat 2D or the passenger seat 2P. Is arranged. Although the purpose of use is not particularly limited, for example, it is for performing functional operations of in-vehicle electronic devices such as a car navigation device and a car audio device while looking at the screen of the monitor 15 provided in the center console (also, the monitor 15 itself is a part of the in-vehicle electronic device).

  The operation unit 12 is attached so that the operation input surface (operation input area) faces upward. The operation input surface is formed by a well-known touch panel 12a, and any of a resistance film method, a surface acoustic wave method, and a capacitance method may be adopted. The base material of the touch panel 12a is a transparent input support plate made of a transparent resin plate, a glass plate, or the like, and accepts a touch operation by the operator's fingertip while supporting it on the upper surface side. Then, an input coordinate system corresponding to the screen area of the monitor 15 is set along the plate surface.

  FIG. 2 is a cross-sectional view schematically showing the internal structure of the input unit 12. The above-described touch panel 12a is fitted on the top surface of the housing 122e so that the operation input surface 102a side is up. In the housing 12d, a hand photographic camera 12b is housed together with an illumination light source 12c and a photographic optical system, and constitutes an image data acquisition unit. The illumination light source 12c is a light emitting diode light source (ie, a monochromatic light source) having a luminance directivity directly above the element by a convex curved mold, and a plurality of illumination light sources 12c are arranged so as to surround the lower surface of the touch panel 12a. Each illumination light source 12c is attached by tilting the mold front end side with high brightness toward the inside of the lower surface of the touch panel 12a, and the primary imaging reflected light RB1 by the palm surface of the operator's hand H on the operation input surface 102a is It is generated in a downward direction so as to pass through the touch panel 12a.

  The photographing optical system has a first reflecting portion 12p and a second reflecting portion 12r. The first reflecting portion 12p is a prism plate (transparent material plate in which triangular prism-shaped microprisms are arranged in parallel in the plane: hereinafter also referred to as a prism plate 12p) and is disposed directly below the touch panel 12a. By reflecting the primary imaging reflected light RB1 from the hand H obliquely upward, it is guided as the secondary imaging reflected light RB2 to the lateral outside of the facing space 12f between the prism plate 12p and the touch panel 12a. The second reflecting portion 12r is a plane reflecting mirror (hereinafter also referred to as a plane reflecting mirror 12r) disposed on the lateral side outside the facing space, and reflects the secondary imaging reflected light RB2 to the side, thereby The light is guided as the tertiary imaging reflected light RB3 to the hand photographing camera 12b located on the opposite side of the second reflecting portion 12r across the space 12f. The hand photographing camera 12b is provided at a position corresponding to the focal point of the tertiary imaging reflected light RB3, and captures and acquires image data of the hand H including the operator's finger.

  As shown in an enlarged view in FIG. 2, the prism plate 12 p is formed of rib-shaped microprisms each having a reflecting surface inclined at an equal angle with respect to the mirror base surface MBP and parallel to each other along the mirror base surface MBP. Even if the mirror base surface MBP is not inclined, the light incident in the normal direction can be reflected to the (oblique) side. Accordingly, the first reflecting portion 12p for side reflection can be disposed in parallel with the lower side of the touch panel 12a, and the height direction dimension of the facing space 12f can be greatly reduced.

  In addition, the second reflecting portion 12r and the hand photographic camera 12b are disposed facing each other with the opposing space 12f sandwiched from the side, so that the tertiary imaging reflected light RB that directly enters the hand photographic camera 12b crosses the opposing space 12f. Can be guided in shape. As a result, the second reflecting portion 12r and the hand photographing camera 12b can be disposed close to the side edge of the touch panel 12a, and the incident path of the imaging reflected light from the hand H to the hand photographing camera 12b is within the facing space 12f. In other words, since it is folded into a three-fold shape, the entire imaging optical system is greatly reduced in size and the casing 12d is made thinner. In particular, by reducing the size of the touch panel 12a (that is, the vertical and horizontal dimensions of the operation input surface 102a), the entire input unit 12 can be dramatically reduced in size and thickness, and the width of the center console unit C in FIG. It can be mounted on a relatively small vehicle or a vehicle that can utilize only a limited mounting space in front of the shift lever without any problem. Further, when the hand is relatively close to the touch panel 12a, the amount of reflected light reaching the hand photographing camera 12b is increased and the hand image can be detected effectively, but the amount of reflected light decreases when the hand is separated from the touch panel 12a. A hand that is more than a certain distance away will not be recognized as a hand image. Therefore, for example, when a hand crosses over the touch panel 12a to operate another operation unit (for example, a shift lever) in the vicinity of the operation device, the effective area ratio is sufficient if the distance from the hand to the touch panel 12a is sufficiently large. Thus, the hand image is not detected, and malfunctions are less likely to occur in the information input process described later using hand image recognition.

  The operation input surface 102a of the touch panel 12 corresponds to the photographing range 102b of the hand photographing camera 12b, and includes the tip of the middle finger when assuming an average hand of an adult as shown in FIG. The vertical direction (Y direction) dimension is determined so that only a part in the longitudinal direction falls within the operation input surface 102a (for example, 60 to 90 mm (as a specific example, 75 mm)). As a result, only the portion ahead of the base of the finger is displayed in the screen area of the monitor 15, so that the shape of the palm portion other than the finger is not involved in the display, and will be described later using the indicator image. The display process can be greatly simplified. In addition, the left-right direction (X direction) dimension of the operation input surface 102a is 110 to 130 mm (for example, 120 mm), and the index finger, middle finger, ring finger, and little finger are within the imaging range when a hand is put on and each finger is wide open. The thumb is out of the shooting range. However, it is also possible to keep all fingers within the imaging range if each finger is properly closed. Further, if the palm portion of the hand H is placed on the operation input surface 102a, a hand-capped state that covers 60% to 100% (for example, 80%) of the operation input surface 102a can be created as shown in FIG. 5A. It is.

  FIG. 3 is a block diagram showing an electrical configuration of the controller device 1. It is the operation ECU 10 that controls the operation device 1. The operation ECU 10 is configured as a computer hardware board mainly composed of the CPU 101. Specifically, the CPU 101, the RAM 102, the ROM 103, the graphic controller 110, the video interface 112, the touch panel interface 114, the general-purpose input / output unit 104, and the serial communication interface. 116 has a structure interconnected by an internal bus 105. The graphic controller 110 includes a display video RAM 111 and a monitor 15, the video interface 112 includes a shooting video RAM 113 and a hand shooting camera 12 b, the touch panel interface 114 includes a touch panel 12 a, and the general-purpose input / output unit 104 includes a driver ( The illumination light source 12c is connected via a drive circuit 115. The serial communication interface 116 is connected to an in-vehicle serial communication bus 30 such as a CAN communication bus, and communicates with other ECUs connected to the network, specifically, a navigation ECU 200 that controls the car navigation device. It is possible.

  The video interface 112 continuously receives an analog or digital video signal captured and acquired by the hand photographic camera 12b, and is captured as image frame data in the photographic video RAM 113 at predetermined time intervals. The content stored in the video RAM 113 for shooting is updated whenever new image frame data is fetched.

  The graphic controller 110 acquires the input screen image frame data from the navigation ECU 200 via the communication interface 116, and also acquires the indicator image frame data in which the indicator image is pasted at a predetermined position from the CPU 101 side. Frame synthesis is performed on the video RAM 111 by a known alpha blending process or the like, and the result is output to the monitor 15.

  The touch panel interface 114 has a unique drive circuit corresponding to the method of the touch panel 12a, detects an input position by a touch operation on the operation input surface 102a based on a signal input state from the touch panel 12a, and displays the detection result. Output as position input coordinate information.

  It should be noted that the shooting range of the hand photographic camera 12b (shooting field of view; as a result, image data captured and acquired by the hand shoot camera 12b), the operation input surface (operation input area) of the touch panel 12a, and the screen area of the monitor 15 (and its The input screen image frame data and the indicator image frame data for determining the display contents are uniquely defined in a two-dimensional coordinate correspondence.

The ROM 103 stores the following software executed by the CPU 101. Thereby, the CPU 101 functions as a hand image area specifying unit, a hand image area ratio calculating unit, and an operation input information generating / outputting unit.
Touch panel control software 103a: The input position coordinates are acquired from the touch panel interface 114, and the operation reference content determination reference information (for example, soft button region specifying information or the related information sent from the navigation ECU 200 together with the input screen image frame data) Information including operation command contents to be output when the soft button is touched. Then, the current operation input content is specified based on the input position coordinates and the acquired determination reference information, and a corresponding operation command execution command is output to the navigation ECU 200.

Display control software 103b: Instructs the graphic controller 110 to input input screen image frame data, creates indicator image frame data by a method described later, and further transfers this to the graphic controller 110.
Hand image area ratio calculation software 103c: The operator's hand image area FI is specified on the image captured as shown in FIG. 4B, and the area of the specified hand image area FI with respect to the operation input area of the touch panel 12a The rate is calculated as the hand image area rate. For the hand image area ratio, specifically, the total area of the operation input area (which can be substituted by the total number of pixels of the corresponding screen) is S0, and the area of the hand image area FI (the number of pixels in the hand image area FI is substituted) S) can be calculated as S / S0, but if the number of pixels in the screen area displaying the hand image area FI is assumed to be constant, the area (number of pixels) of the hand image area FI itself is the hand image. Since the value reflects the area ratio, it is of course possible to use this as the hand image area ratio.

Operation input information creation software 103d: In cooperation with the touch panel control software 103a, operation input information for the in-vehicle electronic device is generated and output based on the operation state on the touch panel 12a and the calculated hand image area ratio. Examples of the operation input information include the following.
(1) A unique relationship is defined between the value of the hand image area ratio and the content of the operation input information to be generated, and the operation input information corresponding to the calculated hand image area ratio value is Identify based on relationships. Specifically, when the calculated hand image area ratio value exceeds a predetermined threshold area ratio (particularly, a hand occlusion state with respect to the operation input region in which the hand image area ratio exceeds 80% is specified. In the case), as the operation input information, specific function activation request information for requesting activation of the specific function of the in-vehicle electronic device is output. In this case, the “specific function of the in-vehicle electronic device” means, for example, the first screen 301 on the monitor 15 in a state where the calculated hand image area ratio is less than the threshold area ratio, as shown in FIG. Is displayed, the second screen 302 shown by C2 in FIG. 5 is displayed in such a manner that the first screen 301 is switched as the hand image area ratio exceeds the threshold area ratio. That is, when a transition is made from a state where the hand image area ratio is less than a predetermined threshold to a state where the hand image area ratio exceeds the threshold, the first screen displayed before the transition is different from the first screen. Screen switching command information for switching to the second screen is output as screen display content change command information.

(2) Operation change for changing the operation state of the specific function in the in-vehicle electronic device when a predetermined operation input is applied to the touch panel 12a after the specific function is activated in the in-vehicle electronic device. Output request information. For example, when an operation input is applied, the operation of the specific function in the in-vehicle electronic device is interrupted, and the operation return request information for returning to the state before starting the specific function is output. As a specific example, a screen return request for returning the display screen of the monitor 15 to the first screen 301 when a touch operation on the touch panel 12a occurs in a state where the screen of the monitor is switched to the second screen 302 shown in FIG. 5C. Information is output as operation input information.

Hereinafter, the operation of the controller device 1 will be described.
On the screen (area) of the monitor 15, an input screen as shown in FIG. 4C (for example, the indicator image SF is displayed at this stage) by a preceding command input (for example, based on a touch input operation on another screen). (Not displayed) is displayed. Although FIG. 4 shows keyboard input, other input screens such as a map screen shown in FIG. 5B may be used. In this state, as shown in FIG. 2, when the hand H is brought close to the operation input surface 102a of the touch panel 12a, as shown in FIG. 4A, the hand photographing camera 12b is moved by the hand of the illumination light from the illumination light source 12c. A hand image is taken based on the reflected light. The pixel of the hand image appears brighter than the background area by receiving the reflected light. Therefore, as shown in FIG. 4B, if the luminance of a pixel is binarized with an appropriate threshold, an area showing a high luminance pixel value (here, “1”) is changed to a hand image area (black in the figure). An area indicating FI and a low luminance pixel value (here, “0”) can be separated from each other as a background area (represented by white in the figure).

Contour lines are extracted from the hand image area that is separated from the image, and the pixels of each part are set to different values so that the contour line part and the inner part of the contour can be visually identified, corresponding to the hand image shape. A pointer image frame in which the pointer image SF thus obtained is pasted at the photographing corresponding position is created. The indicator image frame is transferred to the graphic controller 110, combined with input screen image frame data acquired separately, and displayed on the monitor 15. As a method of combining the input screen image frame data and the indicator image frame data, the following method can be adopted depending on the data form of the indicator image SF.
(1) When the indicator image data is described by bitmap data from the beginning as described above, the indicator image is displayed in a watermark form on the input screen by alpha blending processing between corresponding pixels. can do.
(2) By converting the outline of the indicator image data into vector outline data, it is also possible to use an indicator image frame in which the handling points are mapped on the frame. In this case, the graphic controller 110 generates a contour line of the indicator image using the data on the frame, and further rasterizes the inside of the pointer image to convert it into a bitmap, and thereafter performs the alpha blending process similar to (1). Do.
(3) Similarly to (2), the outline is drawn on the input screen image frame by using the vector outline data constituting the indicator image data, and the pixels of the input screen image located inside thereof are extracted. The set values of the extracted pixels are shifted uniformly.

  In any of (1) to (3), for the pixels forming the contour line of the indicator image data, the indicator image with the contour line emphasized is superimposed by increasing the blend ratio on the indicator image data side. Can be displayed. It is also possible to display the indicator image data as image data of only the contour line described by the bitmap data or the vector outline data, and display only the contour line in an overlapping manner.

  As shown in FIG. 14, the user of the seat is displayed on the screen 15 while watching the input screen (hereinafter also referred to as screen 15) on the monitor 15 as shown in FIG. 4C (or FIG. 5B). A virtual finger operation input to the soft button SB is performed on the touch panel 12a. As shown in FIG. 2, when the hand H is extended on the touch panel 12a, this is photographed by the hand photographing camera 12b, and the indicator image (hand image) SF is formed at the corresponding position on the screen 15 by the image processing described above. It is displayed superimposed. Therefore, the positional relationship between the soft button SB on the screen 15 and the indicator image SF can grasp the positional relationship between the soft button area set on the touch panel 12a and the actual hand, and the operation on the soft button SB. Input is supported.

  In the above processing, the operation input information by the user is generated by the touch panel 12a alone, and the shooting information of the hand shooting camera 12b is not involved in the generation of the operation input information. However, in this embodiment of the present invention, the generation processing of input information related to the shooting information of the hand shooting camera 12b is performed by the hand image area ratio calculation software 103c and the operation input information creation software 103d according to the flowchart shown in FIG. Run in parallel.

In S1, according to the method described above, the area ratio (or the absolute value S of the area or the number of corresponding pixels) of the hand image area FI shown in FIG. 4B with respect to the operation input surface 102a of the touch panel 12a is calculated as the hand image area ratio. In S2, the hand image area ratio is compared with a threshold value (for example, 80%: or an absolute area threshold value S _th (or the number of pixels) on the operation input region). 4B, for example, the hand image area ratio is less than 40%, and normal touch input processing with reference to the input screen is performed as shown in FIG.

  On the other hand, as shown in FIG. 5A, a state where the hand image area ratio exceeds 80% is recognized as the above-mentioned hand-clogged state. At this time, when the input screen shown in FIG. 5B is displayed as the first screen 301, screen display content change command information (specific function activation request information) for switching to the second screen 302 as shown in FIG. It is transferred to the software 103b side. The display control software 103b receives this and performs screen switching processing (S3).

At this time, along with the screen switching to the second screen 302, the following accompanying function can be activated as a specific function of another in-vehicle electronic device (however, the present invention is not limited thereto).
(1) Mute, mute, or temporarily stop the function of the audio device.
(2) Switching the air volume of the car air conditioner (for example, temporarily increasing the air volume).
At this time, the switching itself of the second screen 302 is utilized as visual notification information indicating activation of these specific functions. The second screen 302 can be a simple blackout screen (screen off), but it is more convenient to display information 302a indicating the contents of the accompanying function together.

  Returning to FIG. 6, in this state, even if the hand image area ratio returns to a state below the threshold (that is, even if the hand closing operation is released), the display state of the second screen 302 (and the operation state of the accompanying function). Will continue. On the other hand, when a predetermined operation input, specifically, a touch operation is applied to the touch panel 12a in S4 / S5, the screen return request for returning the display screen of the monitor 15 to the first screen 301 of FIG. 5B. Information is output as operation input information. The display control software 103b receives this and performs screen switching processing (S6). In addition, the activation state of the accompanying function is also canceled, and the operation state before the activation (mute (mute) of the audio device, sound reduction, release of the function stop / return of the air volume setting of the car air conditioner, etc.).

Hereinafter, various modifications will be described.
The operation input information creation software 103d identifies a series of time-varying behavior of the hand image area ratio, and when the time-changing behavior of the identified hand image area ratio matches a predetermined change behavior, the corresponding operation is performed. It can be configured to generate and output input information. By specifying the time-varying behavior of the hand image area ratio and generating the corresponding operation input information when the specified content matches the predetermined changing behavior, more characteristic input hand motion Can be associated with the operation input information, and a more intuitive input operation can be performed.

  FIG. 7 shows a state where a known book viewer BV is displayed on the monitor 15. The information display area of the book viewer BV constitutes a spread screen area of the left page LP and the right page RP, and from the left or right page (here, the left page LP) side by executing the page turning command. While displaying the page turning animation on the other page (here, right page RP) as a moving image, the page RP 'on the back side of the turning side and the page LP' below the turning side are the next spread. The display is switched as page information.

  For example, the present invention can be applied as follows. That is, the user's input hand movement imitating this page turning is photographed as a moving image by the hand photographing camera 12b, and the shape change sequence of the hand image area obtained thereby is detected as the time change behavior of the hand image area ratio, The configuration is such that the page turning command is output as operation input information when it matches a predetermined change behavior. As a result, the user can experience a realistic page turning operation while performing a manual operation of turning the page above the touch panel 12a, while being a book viewer displayed on the screen. As shown in FIG. 8, in the page turning input manual operation, as shown in (a), the palm FI is placed on the area corresponding to the turning side page of the input operation surface 102a, as shown in (b) and (c). In addition, a series of operations are performed to turn the palm FI upside down as shown in FIG.

  The upper part of FIG. 9 shows the change in the area of the hand image region at this time. After the area is reduced from (a) → (b) → (c), the area increases again toward (d). . Therefore, the time change behavior profile of the hand image area (rate) has a downwardly convex shape in which the area is minimized in the middle. Therefore, a judgment window having a certain allowable width in the area axis direction corresponding to the shape on the time-area plane is set, and the time-varying behavior profile of the hand image area actually measured from the photographed image is this. If it enters the judgment window, it is processed to output a page turning command.

  In this case, when the temporal change in the center of gravity position of the hand image area FI is also specified, and both the temporal change in the area ratio and the temporal change in the center of gravity position match a predetermined change behavior. If it is configured to generate a page turning command, it is possible to specify the page turning input manual operation more precisely. Since the area of the input operation surface 102a is relatively small and the palm is returned on the spot without moving the wrist position in the X direction, the gravity center position of the palm itself does not change so much. Therefore, a determination window having a certain allowable width in the coordinate axis direction on the time-centroid coordinate plane is set for each coordinate value of X and Y, and the time change behavior profile of the centroid coordinates actually measured from the photographed image is obtained. If the judgment window is entered, processing is performed to output a page turning command.

As another method, as shown in FIG. 10, the operation input area 102a is divided into predetermined sub-areas A ₁ , A ₂ , A ₃ and these sub-areas A ₁ , A ₂ , A ₃ are divided. The area ratio of the hand image area FI can be calculated every time. In this case, a series of time-varying behavior of the hand image area ratio is specified in each of the sub areas A ₁ , A ₂ , and A ₃ , and the operation input area of the sub area in which the area ratio of the hand image area FI exceeds the threshold value. By grasping the temporal transition behavior related to the position or number of the input, it is possible to specify the intended input hand movement. For example, a sub-region where the area ratio of the hand image region exceeds a threshold (for example, 80%) is set as a first state sub-region, and a sub-region where the area ratio is less than the threshold is set as a second state sub-region. The temporal change behavior related to the distribution state on the operation input region 102a with the subregion is specified as the subregion state distribution transition behavior. Then, when the specified sub-region state distribution transition behavior matches a predetermined behavior, the corresponding operation input information can be generated and output.

Specifically, let us consider a hand operation in which the hand is moved from the right direction on the touch panel 12a and moved while the hand is lifted to the left direction (in order of FIG. 10 (a) → (b) → (c)).
The manual operation can be used as, for example, a command for calling a function of selecting (or starting to play) the next album in the operation mode (and input screen) of the car audio device, but is not limited thereto. Specifically, the appearance position and appearance of the first state sub-region (hereinafter referred to as “1”) and the second state sub-region (hereinafter referred to as “0”) on the operation input area 102a. Specify the change in number over time. In the case of page turning, as shown in FIG. 10, the operation input area 102a is traversed from the first end edge (right end edge in the figure) side to the second end edge (left end edge in the figure) side in the left-right direction (X direction). An input manual operation is performed. Correspondingly, the operation input area 102a is formed by arranging rectangular sub-areas A ₁ , A ₂ and A ₃ in the X direction corresponding to the input hand movement on the operation input area 102a. Yes. In FIG. 10, the movement behavior of the appearance position (and the number) of the first state sub-regions in the sub-region A ₁ , A ₂ , A ₃ arrangement direction is specified.

In the state where the hand is not brought close to the touch panel 12a, each of the sub-regions A ₁ , A ₂ , A ₃ is in the second state. Next, when Yuku close hand from the right on the touch panel 12a, increases Tezo area ratio area A _1, eventually sub-region A ₁ is the first state, the remaining two A _2, A ₃ is the second state (Hereinafter, the state distribution of each of the sub-regions A ₁ , A ₂ , A ₃ is expressed as {A ₁ , A ₂ , A ₃ } = {1, 0, 0} etc. according to the above coding definition)] . Further, as shown in FIGS. 10B and 10C, {A ₁ , A ₂ , A ₃ } = {0, 1, 1} as the hand is moved from right to left, and {1, 1, 1} → { ₁ , ₁ , 0}, and in FIG. 10C, {A ₁ , A ₂ , A ₃ } = {1, 0, 0}, and finally {0, 0 , 0}. Thus, by observing the change of {A ₁ , A ₂ , A ₃ }, the movement of the hand from right to left can be detected.

Note that the movement of the hand from left to right can be similarly identified by detecting changes in {A ₁ , A ₂ , A ₃ } in the reverse order. By detecting the above-described hand movement, the left-right movement with the hand floating is distinguished from the operation of moving the finger while touching the touch panel 12a (= touching or pressing the finger on the touch panel 12a). Is possible. For example, in the operation mode (and input screen) of a car audio device, if you move your finger to the left or right while touching your finger, you can use the command to select the next song (or previous song) and raise your hand. When left and right are operated, an intuitive and natural operation method is realized such as corresponding to a command for selecting the next album (or the previous album).

The shape of each sub-region is not limited to a quadrilateral (rectangular or square), and a polygon other than the quadrilateral (for example, a triangular shape) or any other arbitrary shape can be employed. . For example, in FIG. 13, an example in which the left and right subregions A ₁ and A ₃ are formed in a triangular shape is indicated by a one-dot chain line. In FIG. 13, triangular subregions A ₁ ′ and A ₃ ′ are set so as to correspond to the left and right subregions A ₁ and A ₃ of FIG. Although the left and right two sub-regions A ₁ and A ₃ in FIG. 10A are rectangular, in FIG. 13, these sub-regions A ₁ and A ₃ are respectively located above the side adjacent to the central sub-region A ₂ . The triangle is divided into two triangles by a diagonal line connecting the vertex and the lower vertex on the opposite side, and only the triangular area adjacent to the sub-area A ₂ is adopted as the sub-areas A ₁ ′ and A ₃ ′ (the remaining triangles) The region does not contribute to the detection of the hand image area ratio). For example, when the left and right operations are performed with the hand lifted, it is conceivable to actually draw an arc-shaped hand trajectory like Or shown in the figure. The sub-regions A ₁ ′ and A ₃ ′ correspond to the regions swept by the manual operation, and by setting the sub-regions A ₁ ′ and A ₃ ′ in a triangular shape as described above, As a result of eliminating the triangular area portion from the hand image area ratio, even if the hand moves like the locus Or, the area area in the front direction is large, so that it is easy to detect a finger entering from the front direction. Will occur.

  FIG. 11 shows an example of an input manual operation when a disc eject command to the CD / DVD drive 201 (connected to the navigation ECU 200) shown in FIG. 3 is output as operation input information as another embodiment. Indicates. The hand image area changes in the form of (a) → (b) → (c) → (d) and pans out so that the hand image FI retracts from the center of the screen to the near side (Y direction). Therefore, the time-change behavior profile of the hand image area (rate) decreases monotonously in the order of (a) → (b) → (c) → (d). Therefore, a judgment window having a certain allowable width in the area axis direction corresponding to the shape on the time-area plane is set, and the time-varying behavior profile of the hand image area actually measured from the photographed image is this. If it enters the judgment window, it will process to output a disc eject command.

  In this case, the gravity center position G of the hand image area FI does not change much in the X direction, but changes significantly in the Y direction. Therefore, if the time-varying behavior profile of the barycentric coordinates enters the determination window as shown in the lower part of FIG.

The perspective view which shows an example of the vehicle interior installation form of the vehicle-mounted electronic device operating device used as the application object of this invention. The cross-sectional schematic diagram which shows an example of the internal structure of an operation part. The block diagram which shows an example of the electrical constitution of the vehicle-mounted electronic device operating device of FIG. The figure which shows the positional relationship of a finger image and an operation input surface, and the example of the conventional screen display which superimposed this finger image. The schematic diagram which shows 1st embodiment which concerns on the characteristic part of this invention. 6 is a flowchart showing a processing flow corresponding to FIG. The schematic diagram which shows 2nd embodiment which concerns on the characteristic part of this invention. The schematic diagram which shows the time change of the hand image corresponding to the 1st example of the input hand operation | movement employable in embodiment in FIG. The figure which shows the time change of the area and center-of-gravity coordinate of the hand image of FIG. The figure which shows the 2nd example of the input hand operation | movement employable in embodiment of FIG. 7 with the example of a sub-region division | segmentation. The schematic diagram which shows the time change of the hand image corresponding to the 3rd example of the input hand operation | movement employable in embodiment of FIG. The figure which shows the time change of the area and center-of-gravity coordinate of the hand image of FIG. FIG. 8 is a time-varying image of a hand corresponding to a fourth example of an input hand motion that can be employed in the embodiment of FIG.

Explanation of symbols

1 In-vehicle electronic device operation device (image display device)
10 Operation ECU
12 Input unit 12a Touch panel (position input device)
102a Operation input screen (operation input area)
12b Hand-held camera (hand-photographing means)
15 Monitor (display device)
101 CPU (hand image area specifying means, hand image area ratio calculating means, and operation input information generating / outputting means)
SF indicator image

Claims (18)

An operation input unit provided at a position operable by an occupant seated in a vehicle seat, and having an operation input area of a certain area;
Hand photographing means for photographing an image including the occupant's hand approaching the operation input unit in a certain photographing range for expecting the operation input region;
Hand image area specifying means for specifying the operator's hand image area on the captured image;
Hand image area ratio calculating means for calculating an area ratio of the identified hand image area to the operation input area as a hand image area ratio;
Based on the operation state to the operation input unit and the calculated hand image area ratio, operation input information generation and output means for generating and outputting operation input information for the in-vehicle electronic device,
An in-vehicle electronic device operating device comprising:   An unambiguous relationship is defined between the value of the hand image area ratio and the content of the operation input information to be generated, and the operation input information generation / output means sets the calculated hand image area ratio to the value of the calculated hand image area ratio. The vehicle-mounted electronic device operating device according to claim 1, wherein the corresponding operation input information is specified and output based on the relationship.   The operation input information generation / output unit requests activation of the specific function of the in-vehicle electronic device as the operation input information when the calculated hand image area ratio value exceeds a predetermined threshold area ratio. The vehicle-mounted electronic device operating device according to claim 2, which outputs specific function activation request information for performing the operation.   The operation input information generating / outputting unit activates the specific function when a hand occlusion state is specified for the operation input area in which the hand image area ratio exceeds a threshold area ratio set to a predetermined value of 60% or more. 4. The on-vehicle electronic device operating device according to claim 3, wherein the request information is output.   The operation input information generating / outputting unit is configured to specify the identification in the in-vehicle electronic device when a predetermined operation input is applied to the operation input unit after the specific function is activated in the in-vehicle electronic device. The in-vehicle electronic device operating device according to claim 4, which outputs operation change request information for changing an operation state of the function.   The operation input information generation / output means outputs operation return request information for interrupting the operation of the specific function in the in-vehicle electronic device and returning to the state before starting the specific function when the operation input is applied. The in-vehicle electronic device operating device according to claim 5. Hand image area ratio time change behavior specifying means for specifying a series of time change behaviors corresponding to a predetermined input hand motion on the operation input region of the hand image area ratio by the hand image area ratio calculation means; Have
The operation input information generation / output means generates and outputs corresponding operation input information when the time change behavior of the identified hand image area ratio matches a predetermined change behavior. The vehicle-mounted electronic device operating device according to 1.   The operation input area is divided into a plurality of sub-areas, and the hand image area ratio calculating means calculates the area ratio of the hand image area for each sub-area, and the hand image area ratio time-varying behavior 8. The vehicle-mounted electronic device operating device according to claim 7, wherein the specifying means specifies a series of time-varying behaviors of the hand image area ratio in each sub-region.   The hand image area ratio time change behavior specifying means is for specifying the transition behavior of the number or position of the sub-regions in the operation input area where the area ratio of the hand image area exceeds a threshold, and the operation input 9. The in-vehicle electronic device operating device according to claim 8, wherein the information generating / outputting unit generates and outputs corresponding operation input information when the specified transition behavior matches a predetermined transition behavior. .   The operation input area is divided into a plurality of sub areas arranged adjacent to each other in a predetermined direction on the operation input area, and a sub area in which the area ratio of the handwriting image area exceeds a threshold The hand image area ratio time change behavior specifying means is a first state subregion, a subregion that is less than the threshold is a second state subregion, and the operation input region of the first state subregion and the second state subregion The time change behavior related to the above distribution state is specified as a sub-region state distribution transition behavior, and the operation input information generation / output unit is configured such that the specified sub-region state distribution transition behavior is a behavior determined in advance. The vehicle-mounted electronic device operating device according to claim 9, which generates and outputs corresponding operation input information when they match.   The hand image area ratio time change behavior specifying means specifies a temporal change in the appearance position of the first state subregion or the second state subregion on the operation input region as the subregion state distribution transition behavior. The vehicle-mounted electronic device operating device according to claim 10, which is a device.   The hand image area ratio time change behavior specifying means specifies a time change of the number of the first state subregion or the second state subregion in the plurality of subregions as the subregion state distribution transition behavior. The vehicle-mounted electronic device operating device according to claim 10 or 11, wherein   The hand image area ratio time change behavior specifying means, as the sub-region state distribution transition behavior corresponding to the input hand movement crossing the operation input region from the first end edge side to the second end side in a predetermined direction, In the sub-region group formed in the corresponding direction of the input hand movement on the operation input region, the movement behavior of the appearance position of the first state sub-region in the sub-region arrangement direction is specified. The vehicle-mounted electronic device operating device according to claim 12. The operation input unit includes a transparent input support plate that receives one of the plate surfaces as the operation input region and supports a touch operation by an operator's fingertip while supporting the operation input region. Is configured along the plate surface of the transparent input support plate, and is configured as a position input device that detects the input position by the touch operation and outputs the coordinate information,
The hand photographing means is for photographing an image including the hand on the transparent input support plate from the other plate surface side of the transparent input support plate,
Furthermore, a display device for displaying an input screen for reference when performing an input operation by the position input device is provided,
Indicator image display means for superimposing and displaying an indicator image based on image information of the hand image area at a position corresponding to the hand image area of the image photographed by the hand photographing means on the screen area of the input screen. The vehicle-mounted electronic device operating device according to any one of claims 1 to 13, which is provided.   Illumination means for projecting illumination light from the other plate surface side of the transparent input support plate to the imaging side surface of the hand is provided, and the hand photographing means is a hand image based on light reflected by the hand of the illumination light. The vehicle-mounted electronic device operating device according to claim 14,   The display device is determined as one of the in-vehicle electronic devices together with the display control means, and the operation input information generation / output means is configured to display the display on the display control means according to the calculated hand image area ratio. The in-vehicle electronic device operating device according to claim 15, wherein screen display content change command information for instructing a device to change screen display content is output as the operation input information.   When the operation input information generation / output unit transitions from a state where the hand image area ratio is less than a predetermined threshold to a state exceeding the threshold, the first screen displayed before the transition is displayed, The in-vehicle electronic device operating device according to claim 16, wherein screen switching command information for switching to a second screen different from the first screen is output as the screen display content change command information.   The operation input information generation / output unit returns the display screen of the display device to the first screen when a touch operation to the position input device occurs in a state where the display device is switched to the second screen. The in-vehicle electronic device operating device according to claim 17, wherein screen return request information to be output is output as the operation input information.