JP2009143373A - Vehicular operation input device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a vehicular operation input device having excellent operability for a driver while suppressing cost increase. <P>SOLUTION: In the prompter type vehicular operation input device, input operation to an operation icon 201 displayed on a display screen can be operated by remote control from a touch operation screen 12a1 provided in a steering wheel located in the position different from that of the display screen. On the display screen, in addition to the operation icon 201, a hand image of an operator DR taken by a camera 12b is displayed. When no hand image is taken, a face image of the driver DR is taken to identify the living body condition of the driver, and control corresponding to the condition is executed. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a prompter-type vehicle operation input device.

JP 2007-152984 A

  In recent years, as an operation input device (Patent Document 1) that employs an operation method different from a method of directly operating a touch panel, an operation input device that remotely operates a device such as a prompter method has been developed. Prompter-type operation input devices can be obtained by providing an operation unit at the operator's hand separately from the display device that displays the operation screen (main screen), and photographing and extracting the hand that operates the operation unit. The hand image and the manual operation are reflected on the operation screen (main screen) of the display device, and display control is performed as if the touch panel is directly operated. Among such prompter-type operation input devices, a touch panel may be arranged around the seat portion of the seat, which is provided to the user who is seated on the seat in order to increase operability. Many.

However, for a driver whose basic posture is to hold the handle with both hands, if the touch panel is placed around the seat, the hand in the handle must be moved to the vicinity of the seat. The burden of operation becomes larger than the person who is.
On the other hand, the application of a prompter-type operation input device to a vehicle has a problem that the cost increases due to the mounting of components such as a camera.

  The subject of this invention is providing the operation input apparatus for vehicles implement | achieved in the form with favorable operativity for a driver | operator, suppressing the cost increase as the whole vehicle.

Means for Solving the Problems and Effects of the Invention

In order to solve the above-described problem, the vehicle operation input device of the present invention includes:
A translucent operation panel provided on a steering wheel of a vehicle, the surface of which forms a touch operation surface;
A display means provided at a position different from the steering wheel so as to be visible from the driver, and capable of displaying an operation icon on the display screen;
A hand image photographing means for photographing a hand image of the operator of the touch operation surface close to the touch operation surface side of the operation panel;
Hand image composition display control means for compositing and displaying the photographed hand image at a corresponding position on the display screen;
Input accepting means for accepting an operation input having a content corresponding to the operation icon based on an operation to an operation input position on the operation unit corresponding to the display position of the operation icon displayed on the display screen When,
It is characterized by providing.

  According to the above configuration of the present invention, in the prompter type operation input device, the operation panel is provided on the steering wheel, so that the driver can easily operate. In addition, since the hand image to be operated is displayed on the display device, the operation becomes easier. In addition, with recent multifunctionalization of vehicles, various devices have been mounted on vehicles, and prompters have been installed in a space where various devices and their operation units are arranged. Since the operation input device of the method can change the displayed operation icon to an operation icon of another control content by display switching, it is possible to realize a large number of operation input units that are larger than the area occupied by the touch operation surface. There is also an advantage in terms.

  It should be noted that a correspondence relationship between the display screen of the display means and the touch operation surface of the operation panel is determined in advance so that the other in-plane position corresponding to one in-plane position can be uniquely specified. it can. Thereby, it can be set as the operation input device whose operability was more excellent.

Furthermore, the vehicle operation input device of the operation panel of the present invention includes:
A face image photographing means for photographing a face image including at least a part of the face of the driver seated on the driver's seat from the back side through the operation panel;
Biological state specifying means for specifying that the driver is in a predetermined biological state based on the captured face image;
Control content storage means for storing biological state correspondence control information in which the control content of the in-vehicle electronic device corresponding to various biological states is associated;
When the biological state is specified, based on the biological state correspondence control information, biological state correspondence control execution means for executing control content corresponding to the identified biological state;
The face image photographing means is also used as the hand image photographing means, and the photographing range is fixedly determined and arranged so that the driver's face image can be photographed from the back side of the operation panel via the operation panel. It can be set as the structure made.

  According to the above configuration, the control according to the biological state of the driver can be performed, and the face image photographing unit for specifying the biological information is also used as the above-described hand image photographing unit, and the photographing unit is configured as one. be able to. Thereby, in a vehicle equipped with both a prompter-type operation input device and a function for performing control in accordance with the biological state of the driver, the overall cost can be effectively suppressed. In addition, with recent multifunctionalization of vehicles, various devices have been mounted on vehicles, and the space for arranging various devices and their operation units has become narrower. It is a great advantage to make space for one means. The face image in the present invention is not limited to an image including the entire face, and may be an image including at least a part of the face (for example, an eye image or a nose image).

  The driving state specifying means specifies the driving attention reduced state in which the driver's attention to the vehicle is reduced as the biological state based on the face image taken by the photographing means. It can be configured with means. In this case, in the biological state correspondence control information, the control content for causing the alerting means that performs a predetermined alerting activation work to alert the driver to the driver's attention reduction state is executed. Are associated with each other, and the biological state corresponding control execution means is configured to execute the alerting action based on the biological state corresponding control information when the driving attention reduced state is specified as the biological state. Can do. Thereby, it becomes possible to give a warning to the driver whose attention is reduced. The alert activation work execution means for actually executing the alert activation work can be a voice output means for voice notification of a driver's attention reduction, a buzzer for notifying the driver's attention reduction by a buzzer sound, or the like. . In addition, as a warning activation work, a warning awakening operation that performs a warning in a form that encourages awakening from a state of reduced driving attention can be performed. Specifically, the air conditioning output described below can be executed as a warning wake-up operation.

  Further, the biological state specifying unit can be configured to include a sweating state specifying unit that specifies the sweating state of the driver as the biological state based on the face image taken by the photographing unit. In this case, the biological state correspondence control information is associated with the control state that causes the in-vehicle air conditioning output means to execute the air conditioning output of the strong wind or low temperature or the strong wind and low temperature than the current state for the sweating state. The execution means can be configured to execute the air conditioning output based on the biological state correspondence control information when the sweating state is specified as the biological state. As a result, it is possible to automatically give a cool feeling to the sweating driver.

Furthermore, the vehicle operation input device of the operation panel of the present invention includes:
An approaching object detection means for detecting an approaching object approaching the touch operation surface of the operation panel;
When an approaching object is detected, a biological state correspondence control prohibiting mode for prohibiting execution of the control content corresponding to the biological state by the biological state correspondence control execution unit is set, and the approaching object is not detected. Is a mode setting means for switching and setting a biological state correspondence control execution mode for executing control contents corresponding to the biological state by the biological state correspondence control execution means,
It can be set as the structure provided with.

  When equipped with both a prompter-type operation input device and a function that implements control according to the driver's biological condition, and sharing the necessary photographing means for both, if the hand is closer to the face The hand is blocked and the face is not reflected, and the driver's biological state cannot be detected accurately. In this case, the biological state of the driver may be specified by mistake, and accordingly, the wrong control content may be executed. According to the above configuration, when a hand close to the touch operation surface is detected, that is, when a hand that interferes with the photographing of the driver's face is detected, control according to the driver's biological state is executed. Is prohibited, so that erroneous control contents are not executed.

  Further, since it is meaningless to continue the detection process in a situation where the detection of the biological state of the driver can no longer be detected, the biological state identification means is set to the biological state correspondence control prohibit mode. In this case, it is configured to prohibit the execution of the biological state specifying process for specifying the biological state of the driver based on the face image photographed by the face image photographing unit, so that useless processing is omitted and the control load is reduced. Can be reduced.

  Further, the approaching object detection unit can be configured to be used also as the hand image specifying unit. Specifically, the hand image photographing means is configured to be able to photograph an approaching object including the operator's hand that is close to the touch operation surface side of the operation panel, and the approaching object detection means is based on the hand image specifying means. The approaching object can be detected in such a manner that an image of the approaching object is specified from the captured image. Thereby, since the approaching object detection means can be provided in a form using the imaging means, it is not necessary to provide the approaching object detection means as a separate device, leading to cost reduction.

  By the way, in the present invention, it is possible to provide a photographing light source that is fixedly arranged in the passenger compartment and that irradiates light of a predetermined wavelength band to a photographing target that enters the photographing range. The object to be photographed can be photographed in such a way that the reflected light of the light emitted from the light source is captured. By photographing with a light source, it is possible to extract a face image and a hand image from the photographed image with high accuracy. The accuracy of the hand display and the identification accuracy of the driver's biological state are also increased. The imaging light source can be arranged on the back side of the operation panel and hidden from the operator's eyes. Further, when the light source is, for example, an infrared light source, photographing in a dark place is also good. In this case, the operation panel can transmit only infrared light.

  In the configuration having the mode switching means, the luminance of the photographing light source is set to the first level in the biological state correspondence control execution mode, while the luminance of the photographing light source is set to the first level in the biological state correspondence control prohibit mode. Can also be set to a lower second level. The captured image used for the combined display of the hand image does not have to be so accurate, and at least the contour of the hand can be specified, so that the brightness of the light source is not so necessary. Rather, by suppressing the brightness of the light source to the minimum level, an unnecessary background image that is not close to the touch operation surface becomes difficult to be reflected, so that an advantage of facilitating hand contour specification can be obtained. On the other hand, since the driver's face image is for detecting a biological state, a face that is relatively far away must be photographed in detail, so the brighter the light source, the better. According to the above configuration, the brightness of the light source is set to be increased when the face image is captured than when the hand image is captured, and the advantage of the light source brightness in both modes is effectively used. be able to.

  Further, in the present invention, there can be provided a light reflection area specifying means for specifying a light reflection area in which the intensity of the reflected light exceeds a predetermined threshold intensity on the image photographed by the hand image photographing means. The accepting unit can be configured to accept a touch input to the light reflection area based on the identified light reflection area. According to the above configuration, since the position of the touch operation can be detected by image processing on the captured image captured by the imaging unit such as a camera, an operation input device such as a touch panel is not required as the operation panel, and the translucent The board | plate material etc. which have can be sufficient, and can suppress cost.

  Further, in this case, the input receiving means accepts a touch operation input to the light reflection area when the light reflection area specified by the light reflection area specifying means exceeds a predetermined threshold area. be able to. When an operation is input to a predetermined position on the touch operation surface, the operator may move the finger on the panel surface to a target position. At this time, in some cases, the panel surface may be lightly touched and traced. However, this contact is accidentally performed when searching for a position where an operation input is to be made, and is not intended for operation input to all areas touched by tracing. According to the above configuration, when the contact area to the operation panel is small, that is, when the operation panel surface is only touched lightly, it is not accepted as an operation input, so the operation while tracing as described above is possible. It becomes. In addition, even if the operation panel surface is lightly touched by mistake, the operation is not accepted as an operation input.

  The display means of the present invention can be provided in a form that is also used as a display device that displays a map image or the like in the in-vehicle navigation device. The cost can be suppressed by using the dual purpose.

  The display means may be a vehicle head-up display device including a display that uses a windshield glass driver front area as a display screen and radiates display light toward the windshield glass. Since the display screen is located in front of the driver (front), that is, on the back side of the front of the steering wheel, the display screen on which the operation icons are displayed can be easily viewed even while driving, and the touch operation surface is displayed. Blind operability is increased.

  The operation panel of the present invention can be provided on the boss portion of the steering wheel. A wide touch operation surface can be provided on the steering wheel. Further, the touch operation surface can be configured to have a shape similar to that of the display screen. Thereby, it becomes easy to specify the position of the operation icon on the touch operation surface, and the operation becomes easy. The operation panel is fixed on the vehicle body side so as not to rotate with respect to the rotation of the ring portion of the steering wheel, so that the operation panel does not rotate with the steering rotation operation, so that the operation panel is easy to operate.

  Hereinafter, an embodiment of a touch operation input device of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of a vehicle navigation device using the touch operation input device of the present invention. In this embodiment, as the touch-type operation input device 1, a display unit 15 that displays an operation switch image (operation icon) and an operation panel 12 that performs operation input on the switch image are provided at different positions. Prompter type operation device is adopted. Note that the operation target of the touch-type operation input device of the present embodiment is not limited to the navigation function, but can include the device functions of other in-vehicle electronic devices connected via (via) the in-vehicle LAN 50. In FIG. 1, the navigation device 10 is connected to the air conditioner ECU 101 via the in-vehicle LAN 50.

  The air conditioner ECU 101 serves as a main control unit of the in-vehicle air conditioner 100. As shown in FIG. 4, the air conditioning sensor 120, the air conditioning drive unit 130, and the air conditioning operation unit (air conditioning output state setting operation unit) 140 are provided. Are connected to the in-vehicle LAN 50 (see FIG. 1) via the LAN I / F 190. FIG. 4 is a diagram schematically showing the overall configuration of the in-vehicle air conditioner 100 including the air conditioner ECU 101 and the air conditioner unit U controlled by the air conditioner ECU 101. The air conditioner unit U is a so-called HVAC (Heating, Ventilating and Air-Conditioning) unit, and the air-conditioning state in the vehicle interior is independent between the left seat side and the right seat side (in the present embodiment, the driver seat side and the passenger seat side). It is configured to be adjustable.

  The duct 28 of the air conditioner unit U is formed with an inside air suction port 42 for circulating the air inside the vehicle and an outside air suction port 41 for taking in air outside the vehicle, and is switched to either by the inside / outside air switching damper 24. used. The air from the inside air inlet 42 or the outside air inlet 41 is sucked into the duct 28 by the blower 21. In the duct 28, an evaporator 22 is provided for cooling the sucked air to generate cool air. Further, the downstream side (air outlet side) of the evaporator 22 is branched into a route leading to the driver seat side air outlets 43 to 45 and a route reaching the passenger seat side air outlets 46 and 47.

  As shown in FIG. 2, the air conditioner unit U has a defroster outlet 43 for preventing windshield fogging in the upper rear of the instrument panel corresponding to the lower edge of the inner surface of the windshield. Is located on the right front corner of the instrument panel and on the right corner, the passenger side face outlet 46 is located on the left center and left corner of the front panel of the instrument panel, and the driver side foot outlet 44 is located on the driver's side on the driver seat side on the lower right side of the instrument panel. Foot outlets 47 are opened at the passenger seat side feet at the back left of the instrument panel lower surface, and the opening and closing states are respectively switched by the outlet switching dampers 32 to 36 in FIG. 4.

  The air conditioning drive unit 130 (FIG. 4) connected to the air conditioner ECU 101 includes the damper switching dampers 32 to 36, the damper driving gear mechanism 31 for switching the open / closed state of the dampers 32 to 36, the air mix dampers 25 and 26, and the inside and outside. These are the air switching damper 24 and servo motors 71 to 74 for driving them. These servo motors (actuators) 71 to 74 are controlled to rotate by the air conditioner ECU 101, and detect information such as the rotation position and rotation speed of the rotor and feed back to the air conditioner ECU 101. Specifically, the drive circuits 131 to 134 receive the drive command signal from the air conditioner ECU 101 and drive the corresponding servo motors 71 to 74.

  In addition to the servo motors 71 to 74, the blower 21 and the compressor can also be referred to as an air conditioning drive unit 130 connected to the air conditioner ECU 101. However, the direct drive source of the compressor is an in-vehicle engine, and the air conditioner ECU 101 executes the drive control.

  An air-conditioning sensor 120 (FIG. 4) connected to the air-conditioner ECU 101 includes an internal air temperature sensor (in-vehicle temperature detecting means) 121 that detects the in-vehicle temperature, an external air temperature sensor (in-vehicle temperature detecting means) 122 that detects the outside temperature, and an evaporator. It consists of well-known air conditioning sensors such as a post-evaporator sensor 123 that detects the temperature of the air immediately after passing, and a solar radiation sensor 124 that detects the amount of solar radiation.

  The air-conditioning operation unit 140 (FIG. 4) connected to the air-conditioner ECU 101 is provided on the air-conditioner panel 20 provided in the center front of the instrument panel that can be operated by the driver and the passenger on the front passenger seat. 142, a blow-out port changeover switch (MODE switch) 143, an inside / outside air changeover switch 144, a defroster switch 147, an A / C switch 148, and an independent / collective control changeover switch (DUAL switch) 149. Each of these switches is configured as a well-known pressing operation unit or dial operation unit. On the other hand, in the present embodiment, the air volume changeover switch 145 and the temperature setting switch 146 are configured to appear as touch switches assigned to the touch panel, as will be described later.

  The air conditioner ECU 101 has a known configuration including a CPU, a ROM, a RAM, and the like, and drives and controls the air conditioning drive unit 130 based on the operation states of the various air conditioning operation units 140 and the detection results of the various air conditioning sensors 120. Thus, well-known air conditioning control such as blowout temperature control, blowout air volume control, inside air intake / outside air intake switching control, and outlet switching control is executed. These air conditioning controls are executed in such a manner that the CPU of the air conditioner ECU 101 executes an air conditioning control program stored in its own ROM.

  Returning to FIG. 1 includes a position detector 11 that detects the current position of a vehicle, an operation information input unit 12 for inputting various instructions from an operator such as a driver, map data, and various types of information. A map data input device 14 for inputting map data and the like from a recorded external recording medium, a display unit 15 for performing various displays such as a map display screen and a TV (Television) screen, and various guide voices are output. A voice input / output device 16 for inputting voice of an operator such as a driver, a hard disk storage device 17 for storing various data such as vehicle information, and a vehicle I for exchanging vehicle information. An / F (Interface) unit 19, a wireless communication device 13 for performing short-range wireless communication with another communication device 13 a, and a control unit 18 connected to these 11 to 19 are provided.

  The position detector 11 receives a radio wave transmitted from an artificial satellite for GPS (Global Positioning System) via a GPS antenna and is added to the vehicle and a GPS receiver 11a that detects the position, direction, speed, and the like of the vehicle. A gyroscope 11b that detects the magnitude of the rotational motion, a distance sensor 11c that detects the distance traveled from the longitudinal acceleration of the vehicle, and the like, and a geomagnetic sensor 11d that detects the traveling direction from the geomagnetism. Each of the sensors 11a to 11d has an error of a different property, and is configured to be used while complementing each other. Depending on the accuracy, some of the sensors described above may be used, or a steering rotation sensor, a wheel sensor for each rolling wheel, or the like (not shown) may be used.

  As shown in FIG. 5, the operation information input unit 12 includes a translucent operation panel 12a whose front surface 12a1 forms a touch operation surface, and a rear surface of the operation panel 12a in a form including the operation panel 12a in a shooting range. The camera 12b shoots from the 12a2 side, and a light source (shooting light source) 12c that irradiates light in a predetermined wavelength band toward a shooting target that enters the shooting range.

  The operation panel 12a is made of a plate material having translucency with respect to light emitted from at least the light source 12c. The light source 12c is disposed in such a manner that light emitted from the light source 12c is transmitted from the back surface 12a2 of the operation panel 12a toward the front surface 12a1. The camera 12b is fixed to the vehicle body side so as to capture a predetermined shooting range from the back surface 12a2 side of the operation panel 12a in a form that captures the reflected light of the light emitted from the light source 12c.

  As shown in FIG. 3, the operation panel 12 a is disposed at the center of the boss portion 61 of the steering wheel 6 and can be touched by the driver. Specifically, the steering wheel 6 includes a ring part 62 that serves as a grip part for a driver to perform a handling operation, a boss part 61 located in the center of the ring part 62, a ring part 62, and a boss part 61. The operation information input unit 12 in which the operation panel 12a is fitted into the opening of the front end portion 12f of the housing 12e is fitted into the central opening of the boss portion 61. It is arranged in the form.

  Further, as shown in FIG. 6, the camera 12b is fixed with a shooting range determined so that the face F of the driver DR seated in the driver's seat 2D can be shot. Further, as shown in FIG. 5, the camera 12b of the present embodiment is configured to capture the reflected light of the light emitted from the light source 12c so as to capture the light desired to be reflected by the reflecting device 12r. By capturing and capturing the light reflected by the reflecting device 12r, it is possible to omit image processing for horizontally reversing the captured image. If left-right reversal processing is performed, photographing may be performed by directly capturing reflected light of light emitted from the light source 12c.

  Moreover, the light source 12c of this embodiment is an infrared light source. The camera 12b of the present embodiment is an infrared camera suitable for shooting in a dark place (including an infrared filter (not shown) that shuts out visible light and performs shooting), and an infrared light source Photographing is performed in such a manner that the light (infrared rays) irradiated from 12c captures the reflected light reflected by the photographing object existing within the photographing range. Note that if the operation panel 12a is made of a material having translucency only with respect to the light from the light source, the infrared filter in the camera 12b is not necessary. In the present invention, an infrared camera is used in consideration of the ease of data processing. However, the present invention is not limited to the infrared camera. In this case, the light source 12c does not need to be an infrared light source.

  By configuring the operation information input unit 12 in this way, the camera 12b normally captures an area where the face F of the driver DR is located as an imaging area, and at least captures the face image of the driving driver. While functioning as a face image photographing means for photographing, if there is some approaching object on the touch operation surface 12a1 side of the operation panel 12a, it functions as an approaching object photographing means for photographing the approaching object (approaching) If the object is a hand, it functions as a hand image taking means). That is, as long as there is no approaching object (such as a hand) H positioned so as to cover the touch operation surface 12a1, light (irradiation light) from the light source 12c passes outside the touch operation surface 12a1, The light is reflected by the object to be photographed within the photographing range, and part of the light (reflected light) returns to the back surface 12a2 side and is captured by the camera. On the other hand, when there is an approaching object H that is approaching the touch operation surface 12a1, it is reflected by the approaching object H and part of the light (reflected light) returns to the back surface 12a2 side. Captured by the camera 12b. If the approaching object H is in contact with the touch operation surface 12a1, most light is reflected in the contact region (contact surface) and returns to the back surface 12a2. Although details will be described later, the touch operation on the operation panel 12a in this embodiment is performed by detecting this contact area (light reflection area).

  FIG. 7 is an example of an image 150 photographed by the camera 12b. In FIG. 7, reference numeral 150H is an area where the approaching object H is reflected, reference numeral 150HA is a light reflection area where the contact area between the approaching object H and the touch operation surface 12a1 is reflected, and reference numeral 150e is the rear surface of the front end portion 12f of the housing. This is the area where Reference numeral 150a denotes an area in which the background (distant view) of the approaching object H on the operation panel 12a is projected. In FIG. 7, the face image 150F of the driver DR is shown in the background area 150a, but a part thereof is hidden by the approaching object H (hand image). When the approaching object is not moved, a driver DR face image 150F is photographed as shown in FIG.

  Returning to FIG. 1, the display unit 15 is a display device installed in front of the driver's seat, and any of a liquid crystal display, a plasma display, an organic EL display, and the like may be used. The display screen of the display unit 15 includes a mark indicating the current location identified from the current position of the vehicle detected by the position detector 11 and the map data input from the map data input device 14, a guidance route to the destination, a name, Additional data such as landmarks and marks of various facilities can be displayed in an overlapping manner. Also, facility guides can be displayed. The display unit 15 corresponds to display means of the present invention.

  Further, as described above, the display unit 15 has a display coordinate system that has an unambiguous correspondence with the operation system coordinates for indicating the input position on the touch operation surface 12a1 of the operation panel 12a. A switch image (operation icon) 201 is displayed at a predetermined position on the display coordinate system. When a touch input is made at the corresponding operation input position on the touch operation surface 12a1, the switch image 201 is displayed. The corresponding control content is executed. That is, in the present embodiment, the display screen of the display unit 15 has a shape similar to the touch operation surface 12a1 of the operation panel 12a, and the two-dimensional coordinate system defined on the display screen and the operation panel 12a. A unique correspondence with the two-dimensional coordinate system defined on the touch operation surface 12a1 is determined in advance, and the other in-plane position corresponding to the one in-plane position can be uniquely specified. Therefore, based on the correspondence, it is possible to perform a touch operation on the switch image 201 displayed on the display screen by a touch input on the touch operation surface 12a1. Note that the shape of the display screen of the display unit 15 and the touch operation surface 12a1 of the operation panel 12a is not limited to the display shape as described above, and at least the correspondence between the in-plane positions is defined as described above. It only has to be.

  The display unit 15 is not limited to the one disposed in the center console C as shown in FIG. 2, and may be installed above the lower edge of the windshield FG, for example. Specifically, a head-up display (HUD) or the like for displaying the above-mentioned image / data can be exemplified on the windshield FG of the vehicle.

  Further, the display unit 15 is not limited to the one arranged in the center console C as shown in FIG. For example, the visibility can be increased by providing the display screen of the display unit 15 at the driver front position. For example, the display unit 15 may be a meter display device, and the display screen may be provided in a meter display region located in the back of the steering wheel 6. Further, the display screen of the display unit 15 is installed above the lower edge of the windshield FG, so that the visibility is further increased. For example, as shown in FIGS. 18 and 19, a vehicle head-up display (see FIG. 18) is provided with a display screen 15 a that is a driver front area of the windshield glass FG and a display 15 b that emits display light toward the windshield glass. HUD) device may be the display unit 15.

  Returning to FIG. 1, the map data input device 14 is road data as network data, map data such as so-called map matching data for improving the accuracy of position specification, mark data indicating facilities, guidance image and audio data. It is a device for inputting various data including the like. A CD-ROM, DVD-ROM, hard disk, memory, memory card, or the like can be used as a recording medium for these data.

  The voice input / output device 16 can output facility guides and various guidance voices input from the map data input unit 14, and read-out voices of information acquired via the I / F 19. In addition, the voice input / output device 16 includes a microphone (not shown) and a known voice recognition unit, and can input voice of an operator such as a driver to the control unit 18 as a command or the like.

  The wireless communication device 13 is for performing narrowband communication with other communication devices 13a. For example, DSRC (Dedicated Short Range Communications), Bluetooth (registered trademark), wireless LAN (Local Area Network), UWB (Ultra Wideband). ) Etc. are used.

  The LAN I / F 19 is an interface circuit for exchanging data with other in-vehicle electronic devices and sensors via the in-vehicle LAN 50. Further, for example, data from another ECU may be taken in via the LAN I / F 19.

  The control unit 18 is configured around a well-known microcomputer including a well-known CPU, ROM, RAM, I / O (Input / Output) (not shown) and a bus line connecting these components. Based on each detection signal from the position detector 11 based on a program stored in the storage unit, the current position of the vehicle is calculated as a set of coordinates and traveling directions, and the vicinity of the current position read via the map data input device 14 A map display process for displaying on the display unit 15 a main image (including a moving image and a still image) 200B as a main image (including moving images and still images) 200B, and a map data input unit 14 Based on the stored point data, the facility as the destination is selected according to the operation of the operation information input unit 12, and the optimum route from the current position to the destination is automatically selected. A route guidance process of performing route guidance performs route calculation to determine. A technique such as the Dijkstra method is known as a technique for automatically setting an optimum route in this way.

  The control unit 18 also includes a switch image (operation icon) 201 and a processed image 200H based on the approaching object image 150H acquired from the operation information input unit 12 based on a program stored in a storage unit such as a ROM. Both are sub-images, and an image composition unit 18a for superimposing these images on a main image 200B (map operation image or the like) is provided. As shown in FIG. On the display screen. That is, the control unit 18 functions as a hand image composition display control unit of the present invention.

  Note that the processed image 200H based on the approaching object image 150H is the approaching object outline reflecting image 200H that has been processed and generated so that the outer shape of the approaching object image 150H is visually recognized. By synthesizing this image 200H, when an operator such as a driver moves a finger or the like of the hand so as to face the touch operation surface 12a1 of the operation panel 12a, for example, the display unit The processed image 200H reflecting the hand shape is moved and displayed on the display screen 15, and the operation can be performed as if the operation panel 12a is present on the display screen of the display unit 15. .

  Further, switch image data for displaying the switch image 201 is stored in a predetermined storage area of the storage device 17 and is used for the composite display of the switch image 201. On the other hand, similarly, in a predetermined storage area of the storage device 17, control performed by operation input to each switch image (touch input to a corresponding area on the touch operation surface 12 a 1) in a form associated with each switch image 201. The contents are also stored. The control unit 18 displays a composite video 200 in which the switch image 201 is synthesized on the main image 200B on the display screen, and accordingly, the touch operation surface 12a1 of the operation panel 12a displays the switch image 201 on the display screen. An operation input reception range (operation input position) corresponding to the switch image 201 is set in an area corresponding to the display position (operation input position setting means). When a touch operation is performed on the operation input reception range corresponding to the switch image 201 and the input is received (input reception unit), the control content corresponding to the switch image 201 is performed. . Specifically, a control signal based on the corresponding control content is output (control means). The presence / absence of touch input is determined by detecting a contact area by a touch operation performed on the touch operation surface 12a1 of the operation panel 12a having a correspondence relationship with the display screen of the display unit 15 from the captured image 150 of the camera 12b. The

  The control unit 18 includes a hand image composition display control unit, an input reception unit, a biological state identification unit (a driving attention reduction state identification unit, a sweating state identification unit), a biological state correspondence control execution unit, a mode setting according to the present invention. Function as a light reflection region specifying means. Note that the image composition unit 18a also has a part of the function as the hand image composition display control means.

  By the way, in the touch-type operation input device of the present invention, the operation panel 12a is provided on the steering wheel 6 of the vehicle, while the display screen of the display unit 15 is provided at a position different from the steering wheel 6, thereby improving operability. Therefore, the operator's hand facing the operation panel 12a can be synthesized and displayed at a corresponding position on the display screen (hand image synthesis display control means). When a touch operation is performed on the operation input position on the operation panel 12a corresponding to the operation icon on the display screen, the operation input corresponding to the position is received (input reception unit). Furthermore, in the present embodiment, the image includes only the face image of the driver sitting on the driver's seat by the camera 12b that captures the hand image (the entire face, the outline of the face, and a part of the face (eyes, nose, etc.): In the present embodiment, it is possible to capture the entire face), and it is possible to specify that the driver is in a predetermined biological state based on the captured facial image (biological state specifying means). The predetermined storage area (control content storage means) of the storage device 17 stores biological state correspondence control information in which the control contents of the in-vehicle electronic device corresponding to various biological states are associated with each other. Is specified, the control content corresponding to the specified biological state can be executed based on the biological state correspondence control information (biological state correspondence control execution means).

  And in this embodiment, based on the presence or absence of the detection of the proximity | contact target object to the touch operation surface of an operation panel, it switches and sets a biological condition corresponding | compatible control prohibit mode and a biological condition corresponding | compatible control execution mode (mode setting). means). This mode setting process is started by the control unit 18 in such a manner that the CPU executes a program stored in the ROM. Hereinafter, a description will be given using the flowchart of FIG.

  As the engine start operation such as turning on the ignition switch (IG) is performed, photographing by the camera 12b is started. In this state, the control unit 18 detects the approaching object H in S101. The approaching object H can be detected by separately providing a known proximity sensor such as an infrared sensor, but in the present embodiment, detection is performed based on the captured image 150 of the camera 12b.

  That is, since the camera 12b captures the reflected light of the light source 12c, the reflected light intensity is reflected on the gradation of the color appearing in the captured image 150, and the high-intensity reflected light is reflected. The more you capture it, the higher the gradation level appears. In particular, an object closer to the camera 12b captures reflected light having a higher intensity, so that when an object close to the operation panel 12a is projected, the area has a higher gradation level than other areas. Appears as an area. For this reason, in the present embodiment, a threshold is provided in advance for the gradation of the color appearing in the captured image 150, and it is detected that the approaching object H is in an area exceeding the threshold. That is, the captured video 150 captured by the camera 12b of the present embodiment is a multi-gradation captured image (monochrome captured image in the present embodiment), and this is used using a predetermined first gradation threshold. Then, the gradation level of each pixel is binarized, and an area exceeding the first gradation threshold is specified as the approaching object image 150H. The control unit 18 detects the approaching object in a form that specifies the approaching object image 150H, and functions as an approaching object detection unit together with the camera 12b.

  If an approaching object is detected, the process proceeds to S103. In S103, the control unit 18 sets a biological state correspondence control prohibiting mode in which execution of the control content corresponding to the biological state is prohibited, and starts the above-described hand image composition display processing by the image composition unit 18a. However, when the biological condition correspondence control prohibition mode is set in S103, the execution of the process of specifying the driver's biological condition based on the face image captured by the camera 12b is also prohibited. Not performed. On the other hand, if no approaching object is detected, the process proceeds to S102. In S <b> 102, the control unit 18 sets a biological state corresponding control execution mode in which the control content corresponding to the biological state is executed, and the process related to the above-described hand image composite display process is stopped. That is, the control unit 18 functions as a mode setting unit by executing S102 and S103. The above processing is repeatedly executed at a predetermined cycle.

  Next, the biological state correspondence control execution process executed by the control unit 18 when the biological state correspondence control execution mode is set will be described with reference to the flowchart of FIG. This biological state correspondence control execution process is started in the form that the CPU executes a program stored in the ROM in the control unit 18. Note that S111 is omitted in this embodiment.

  First, in S112, the control unit 112 acquires a captured image 50 by the camera 12b, and identifies a driver DR face image 150F from the captured image 150 in S113. Note that the face image is specified by performing image processing such as binarization processing or pattern matching based on grayscale information based on the captured image 150. In the present embodiment, the identification is performed by pattern matching with various basic face shape data (a plurality of data reflecting face shapes of various face orientations) stored in advance in the storage device 17. In S114 with the pattern, if the face image 150F is not specified in S113, the process proceeds to S119. On the other hand, if the face image 150F is specified in S113, the process proceeds to S115.

  In S115, the control unit 112 specifies whether or not the driver DR that has captured the face image 150F is in a predetermined biological state. In the present embodiment, the case where the driver DR is in a state where the driver's attention is reduced and the case where the driver DR is in a sweating state are specified as a predetermined biological state. Specifically, based on the driver DR face image 150F captured by the camera 12b, image processing such as binarization processing and pattern matching using density information is performed, and the driver DR eye position and blinking are performed. By recognizing the degree of opening, sweating, and the like, the biological state such as the driver DR looking aside, falling asleep and sweating is specified. In this embodiment, the control part 18 is performing the process as shown in FIGS. 13-16 as a process which detects a specific biological state separately from this process, and the predetermined | prescribed storage part (( Since the detection result information in each of these processes is stored in the driver biological state storage unit), the current biological state of the driver is specified by reading the detection result information from the storage unit. That is, the control unit 18 executes processes as shown in FIGS. 13 to 16 in such a manner that the CPU executes various programs stored in the ROM, and performs biological state specifying means (driving attention deficit state specifying means, sweating). It functions as a state specifying means).

  FIG. 13 is an example of a process for detecting a driving attention reduction state. Here, the case where the number of side-by-side detection as shown in FIG. 9D exceeds a predetermined number of times within a predetermined time period is specified as a state where the driving attention is reduced. The detection result information of the driving attention reduced state by the processing is stored in the first driver biological state storage unit provided in the storage device 17.

  In S201, the control unit 18 resets a timer 18b and a counter 18c, which will be described later, provided in the controller 18 as an initialization process, and a decrease in driving attentiveness stored in the storage device 17 (first driver biological state storage unit). The state detection result information is initialized (the memory state indicating that the driver's ecological state is in the normal state). In subsequent S202, the timer 18b is activated.

  In S203 to S206, it is determined from the face image 150F captured by the camera 12b whether or not the pupil of the eye is at the center of the eye, thereby identifying the driver DR. In S203, the face image 150F is extracted and acquired from the captured image 150 of the camera 12b. In S204, the position of the eye is specified from the acquired face image 150F. As shown in FIG. 9D, the position of the eye is identified by extracting the contour shape of the eye image 150E from the face image 150F by image processing such as pattern matching, and specifying the inner corner (150C) of the eye as the eye position. To do. In S205, the pupil position is specified from the acquired face image 150F by image processing such as pattern matching. As shown in FIG. 9 (d), the pupil position is specified in such a way that the position of the center of gravity of the black portion (150P) inside the eye (eye image 150E) in the face image 150F is specified. In S206, the driver DR looks aside based on the information acquired in S203 and S204. In this embodiment, the relative position of the pupil with respect to the entire eye is specified. Specifically, the distance from the eye position to the center of gravity of the pupil is calculated, and the deviation of the center of gravity of the pupil to the left and right is calculated. This is done in a way that identifies Note that the left-right bias of the pupil position may be specified by the distance between the center of gravity of the eye (the center of gravity of the contour of the eye) and the center of gravity of the pupil. If a predetermined bias or more as shown in FIG. 9D is detected, in S207, this is determined to be a side look and the process proceeds to S208. On the other hand, as shown in FIG. 9 (a), if a predetermined bias or more is not detected, the process proceeds to S211.

  In S208, the count value of the counter 18c provided in the storage unit of the control unit 18 is increased. This count value is the number of times that the driver DR looks aside during a predetermined time (from when the timer 18b is started in S202 until it is determined that the predetermined threshold time has elapsed in S211). It is a value that reflects That is, when this value is large, it is a situation where there is a lot of aside time, and it can be said that the attention to driving is reduced. In S209, it is determined whether or not the count value exceeds a predetermined value, and a predetermined number of times If it has exceeded, it progresses to S210 and memorize | stores a driving attention reduced state in the memory | storage device 17 (1st driver biological state memory | storage part) as detection result information of a driver | operator's ecological state. If the predetermined number of times is not exceeded in S209, the process proceeds to S211.

  In S211, the elapsed time is read from the timer 18b, and it is determined whether or not a predetermined time has elapsed since starting in S202. If the predetermined time has not elapsed, the process returns to S203, and the detection of an aside is started again. If the predetermined time has elapsed, the process proceeds to S212, and similarly to S209, it is determined whether or not the count value exceeds a predetermined value. If the predetermined number of times has not been exceeded, the process proceeds to S213, the normal state is stored in the storage device 17 (first driver biological state storage unit) as detection result information of the driver's ecological state, and the process proceeds to S214. If it exceeds the predetermined number of times, the process skips S213 and proceeds to S214. In S214, it is determined whether or not an end condition for the detection process is satisfied. Here, the IG is turned off as an end condition. If the end condition is satisfied, the present process is ended. If the end condition is not satisfied, the process proceeds to S215, the timer 18b is reset, the process returns to S202, and the timer 18b is again set. To resume the detection of aside.

  FIG. 14 is also an example of a process for detecting a driving attention reduction state. Again, the case where the number of detected side effects as shown in FIG. 9 (e) exceeds the predetermined number of times within a predetermined time period is specified as a state where the driving attention is reduced. The detection result information of the driving attention reduced state due to the processing is stored in the second driver biological state storage unit provided in the storage device 17.

  In S221, as the initialization process, the control unit 18 resets a timer 18b and a counter 18c, which will be described later, and a decrease in driving attentiveness stored in the storage device 17 (second driver biological state storage unit). The state detection result information is initialized (the memory state indicating that the driver's ecological state is in the normal state). In subsequent S222, the timer 18b is started.

  In S223 to S226, the driver's side aside is specified by determining whether or not the face is facing the front from the face image 150F captured by the camera 12b. In S223, the face image 150F is extracted and acquired from the captured image 150 of the camera 12b. In S224, the position of the center of gravity of the nose is specified from the acquired face image 150F. As shown in FIG. 9E, the nose centroid position is identified by extracting the contour shape of the nose image 150N from the face image 150F by image processing such as pattern matching and specifying the centroid position with respect to the contour shape. Do. In S225, the pupil positions of both eyes are specified from the acquired face image 150F by image processing such as pattern matching. As shown in FIG. 9E, the pupil position is specified by specifying the position of the center of gravity of the black portion 150P inside the eye (eye image 150E) in the face image 150F. In S226, the driver DR looks aside based on the information acquired in S223 and S224. In this embodiment, the distance between both eyes and the nose is specified, and the relative position of the pupil with respect to the entire eye is specified. Specifically, from the center of gravity position of each of the left and right eyes (eye image 150E). The distance to the position of the center of gravity of the nose (nasal image 150N) is calculated, and the difference between both distances is calculated. As shown in FIG. 9 (e), if a difference in distance greater than or equal to a predetermined distance is detected, in S227, this is determined to be an aside and the process proceeds to S228. On the other hand, as shown in FIG. 9A, when a distance difference of a predetermined distance or more is not detected and the distances substantially coincide, the process proceeds to S231.

  Note that S228 to S235 are the same processes as S208 to S215 in FIG. However, the detection result information of the driving attention reduced state by the processing is stored in a second driver biological state storage unit provided in the storage device 17, which is different from the first driver biological state storage unit.

  FIG. 15 is also an example of processing for detecting a driving attention reduced state. Here, when the number of times the eyes are closed as shown in FIG. 9 (a) exceeds a predetermined number of times within a predetermined time (when the number of blinks is large or when the doze is continuously closing eyes) Etc.) is identified as a state where the driver's DR has a reduced level of driving attention. The detection result information of the driving attention reduced state by the processing is stored in a third driver biological state storage unit (a storage area different from the first and second driver biological state storage units) provided in the storage device 17. Remembered.

  In S241, as the initialization process, the control unit 18 resets a timer 18b and a counter 18c, which will be described later, and a decrease in driving attentiveness stored in the storage device 17 (third driver biological state storage unit). The state detection result information is initialized (the memory state indicating that the driver's ecological state is in the normal state). In subsequent S242, the timer 18b is activated.

  In S243 to S246, the number of blinks of the driver and dozing are specified by determining the open / closed state of the eyes from the face image 150F captured by the camera 12b. In S243, the face image 150F is acquired from the captured image 150 of the camera 12b. In S244, the eye image 150E of both eyes is extracted from the acquired face image 150F by image processing such as pattern matching. As shown in FIG. 9A, the eye image 150E is extracted by specifying the upper eyelid image 150LU and the lower eyelid image 150LD by image processing such as pattern matching, and specifying the region between them as the eye image 150E. To do. In S245, the opening degree of the eye is specified from the acquired eye image 150E. The opening degree of the eye is specified by calculating the area of the eye image 150E. The distance in the vertical direction between the lowermost positions of the upper eyelid image 150LU and the lower eyelid image 150LD may be specified as the eye opening. In S246, it is determined based on the information acquired in S244 whether or not the eyes of the driver DR are closed. In the present embodiment, the area of the eye image 150E is calculated, and is performed based on whether the area exceeds a predetermined threshold area. If it exceeds the threshold area, it is determined that the eyes are closed in S247, and the process proceeds to S248. On the other hand, if the threshold area is not exceeded, the process proceeds to S250.

  In S248, the count value of the counter 18c provided in the storage unit of the control unit 18 is increased. This count value indicates a state in which the eyes of the driver DR are closed within a predetermined time (from the time when the timer 18b is activated in S242 until it is determined that a predetermined threshold time has elapsed in S250). This value reflects how many times it was detected. That is, when this value is large, it means that the time when the eyes are closed is large, and it can be said that the attention level to driving is low and the level of consciousness is low. In S249, the count value exceeds the predetermined value. If the predetermined number of times has been exceeded, the process proceeds to S250, and the driving attention reduced state is stored in the storage device 17 (third driver biological state storage unit) as the detection result information of the ecological state of the driving DR person. To remember. On the other hand, if the predetermined number of times is not exceeded in S249, the process proceeds to S251.

  In S251, the elapsed time is read from the timer 18b, and it is determined whether or not a predetermined time has elapsed since the activation in S242. If the predetermined time has not elapsed, the process returns to S243, and detection of the state where the eyes are closed is started again. If the predetermined time has elapsed, the process proceeds to S252, and similarly to S248, it is determined whether or not the count value exceeds a predetermined value. If the predetermined number of times has not been exceeded, the process proceeds to S253, the normal state is stored in the storage device 17 (third driver biological state storage unit) as the detection result information of the driver's ecological state, and the process proceeds to S254. On the other hand, if the predetermined number of times is exceeded in S252, S253 is skipped and the process proceeds to S254. In S254, it is determined whether or not an end condition for the detection process is satisfied. Here, IG is turned off as an end condition, and if the end condition is satisfied, the present process is ended. If the end condition is not satisfied, the process proceeds to S255, the timer 18b is reset, the process returns to S242, and the timer 18b is again set. To resume detection of the driver's eyes closed.

  FIG. 16 is also an example of a process for detecting a driving attention reduction state. Here, it is specified that the driver DR is sweating as shown in FIG. The detection result information of the driving attention reduced state by the processing is stored in a fourth driver biological state storage unit (a storage area different from the first to third driver biological state storage units) provided in the storage device 17. Remembered.

  In S <b> 281, the control unit 18 initializes detection result information of the driving attention reduced state stored in the storage device 17 (fourth driver biological state storage unit) provided therein as the initialization process (driving). Memory state indicating that the biological state of the person is in the normal state). In S282 to S286, the sweating state is specified from the face image 150F photographed by the camera 12b. In S282, the face image 150F is acquired from the captured image 150 of the camera 12b. In S283, the sweat image 150K is extracted from the acquired face image 150F by image processing such as pattern matching. The extraction of the sweat image 150K here is a liquid whose tone is higher than that of normal skin in a region located outside the eyes (eye image 150E) in the face image 150F shown in FIG. 9B. A reflection area is specified, and this is specified as a sweat image 150K. In S284, based on the information acquired in S283, whether or not the driver DR is in a sweating state is determined based on the presence or absence of the sweat image 150K. If the sweat image 150K is extracted, this is determined as the sweating state of the driver DR, and the process proceeds to S285, where the sweating state (fourth driver biological state) is detected as the biological state detection result information of the driver DR. Stored in the storage unit), and the process proceeds to S287. On the other hand, if the sweat image 150K is not extracted, the driver DR is determined to be in the normal state, and the process proceeds to S286, where the normal state is set as the detection result information of the biological state of the driver DR (fourth driver). Stored in the biological state storage unit), and the process proceeds to S287.

  In S287, it is determined whether or not the end condition of the detection process is satisfied. Here, the IG is turned off as an end condition. If the end condition is satisfied, the present process is terminated. If the end condition is not satisfied, the process returns to S282 to restart the sweat detection process.

  Returning to FIG. In S115, the detection result information of the processes of FIGS. 13 to 16 is read from each driver biological state storage unit of the storage device 17, and whether the current driver DR is in a state of reduced driving attention or sweating. Is identified. In addition, since each detection result information is memorize | stored in the separate memory | storage area | region (1st-4th driver | operator biological state memory | storage part), what kind of biological state a driver | operator is in each storage area | region As will be apparent, individual control details may be implemented accordingly. However, in the present embodiment, only two of the driving attention reduced state and the sweating state are recognized and the control content corresponding to both is performed by the following processing.

  In subsequent S116, if any of the driving attention reduced state or the perspiration state is detected in S115, the process proceeds to S117, and if none is detected, the process proceeds to S119. In S119, when the control corresponding to the biological state has already been performed, a process for ending the control is performed. And it returns to the process of FIG.

  In S117, the control content corresponding to the biological state specified in S115 is specified, and the control content specified in S118 is executed (biological state corresponding control execution means living body). In the present embodiment, when any of the driving attention reduction states is detected, a sound alert activation operation is executed, and when a sweating state is detected, the air conditioning output according to a predetermined air conditioning condition Is executed. Specifically, in the predetermined storage area (control content storage means) of the storage device 17, the biological state correspondence control information in which the control content of the in-vehicle electronic device is associated with the predetermined biological state of the driver DR. When the biological state is specified, the control content corresponding to the specified biological state is executed based on the biological state correspondence control information.

  The alert activation work of the present embodiment is performed by outputting a voice to the driver DR that the attention is reduced. In the storage device 17, alert voice data is stored in association with these reduced driving attention levels, and voice output is performed from the voice output device 16 based on this data. Note that the alert activation work may be another operation different from the sound output described above, and is a notification with a buzzer sound or a warning alerting action (for example, driving) that promotes awakening from a state of reduced attention For example, opening a seat-side window).

  On the other hand, the air conditioning output corresponding to the sweating state of the present embodiment is executed in such a manner that it is changed to an air conditioning condition of strong wind or low temperature, or a strong wind and low temperature than the current air conditioning condition. The storage device 17 stores air conditioning condition data in association with the sweating state, and instructs the air conditioning ECU 101 to change the air conditioning conditions based on this data. Specifically, the target blowing temperature set value currently set manually or automatically is reduced by a predetermined value, and the setting value of the target blowing air volume currently set manually or automatically is set. Is increased by a predetermined value. Then, the air conditioning output corresponding to the above-mentioned sweating state is executed by executing the blowout temperature control and the blowout airflow control based on the set values of the target blowout temperature and the target blowout airflow after the setting. If the value after setting is lower than the set lower limit value or higher than the set upper limit value, the set lower limit value or the set upper limit value is set.

  When the process of S118 ends, the process returns to FIG. Since the mode setting process of FIG. 10 is repeatedly executed, this biological state correspondence control execution process is repeated unless an approaching object is detected in S101, and the corresponding control content is continued as long as the above-described biological state is continuously detected. Will continue to run.

  Next, display control processing (hand image composition display processing) by the control unit 18 will be described with reference to the flowchart of FIG. Note that S121 is omitted in this embodiment. In S122, the control unit 112 acquires a captured image by the camera 12b, and extracts and specifies a hand image from the captured image. Whether or not the image is a hand image is determined based on the shape of the captured approaching object image 150H. As a technique for extracting a human hand image, any known technique (see, for example, JP-A-2003-346162) can be used. If a hand image is extracted and specified in S123, the process proceeds to S124, and if an image different from the hand is specified, the process is terminated.

  In S124, based on the extracted hand image, a composite video 200 as shown in FIG. 5 is displayed on the screen of the display unit 15. Specifically, the extracted hand image 150H is processed to be in a semi-transparent state, and then synthesized by the image synthesis unit 18a on the main image 200B (here, the map operation image). 5 is displayed. The processing / compositing display of the hand image 150H on the main image 200B is not limited to the above-described transflective composite display, and various modifications can be considered.

  By performing the display control process described above, it functions as a hand image composition display unit that composites and displays the photographed hand image 200H at a corresponding position on the display screen.

  Next, touch input reception processing by the control unit 18 will be described. FIG. 17 is a flowchart showing the flow.

  First, in S1, a hand contact area where the hand H is in contact with the touch operation surface 12a1 of the operation panel 12a is specified from the captured image 150 captured by the camera 12b (light reflection area specifying means). In the area where the hand H is physically in contact with the touch operation surface 12a1, the irradiation light from the light source 12c is reflected more than in the other areas. It appears as a light reflection region 150HA having a gradation level sufficiently higher than. Furthermore, although the gradation level of the light reflection area 150HA varies depending on the approaching object H, it appears at a certain level in the captured image 150. In the present embodiment, the second gradation threshold is set as a gradation level higher than the first gradation threshold so that only such a high gradation level can be specified, and this is used to determine the level of each pixel. By binarizing the tone level, the light reflection area 150HA exceeding the second gradation threshold is specified as the contact area on the photographed image 150.

  In S2, the area of the light reflection region 150HA is calculated, and it is determined whether or not the calculated area exceeds a threshold area obtained by a predetermined method. If the area of the light reflection area 150HA does not exceed the threshold area, it is determined that there is no touch input to the light reflection area (contact area), and no operation input to the area is accepted, and this process ends. On the other hand, if the area of the light reflection area 150HA exceeds the threshold area, it is determined that there is a touch input in the light reflection area (contact area), and an operation input to the area is accepted in S3. Advance (input acceptance means). Note that when a contact area is specified, an operation input to the area may be accepted regardless of the area. However, since the possibility of erroneous input increases, in the present embodiment, as described above. In addition, a threshold area for receiving an operation input is defined in advance.

  In S <b> 4, it is determined whether or not the contact area that has received the operation input is in an area corresponding to the switch image (operation icon) 201 displayed on the screen of the display unit 15. If the contact area is not an area corresponding to the switch image 201, the present process is terminated. On the other hand, if the contact area is an area corresponding to the switch image 201, the process proceeds to S5.

  In S5, the control content to be executed is specified according to the touched touch area, and the specified control content is executed in S6. In the present embodiment, since the control content is stored in association with each switch image 201, the corresponding control content is specified based on the stored information and executed (control means). When S6 ends, the input acceptance process ends, but the process is repeatedly performed at a predetermined cycle.

  Although one embodiment of the present invention has been described above, this is merely an example, and the present invention is not limited to this, and the knowledge of those skilled in the art can be used without departing from the spirit of the claims. Various modifications based on this are possible.

  For example, in the above-described embodiment, the luminance of the imaging light source 12c is fixedly set to a level at which the driver's face image can specify the biological state, but the luminance of the imaging light source in the biological state correspondence control execution mode. Can be set to the first level, and on the other hand, the luminance of the imaging light source can be set to a second level lower than the first level in the biological state correspondence control prohibition mode. In this case, the processing of S111 and S121 is executed in FIGS. Then, regarding the detection of the approaching object in S101 of FIG. 10, if the mode at that time is the biological state corresponding control execution mode, the image is taken by the camera 12b under the light source 12c that emits light at the first level of luminance. Detection is performed based on whether or not an approaching object can be extracted from the captured image, and in the biological state correspondence control prohibition mode, the camera 12b emits light with a second level of luminance. Detection is performed based on whether or not an approaching object can be extracted from the captured image. However, the gradation threshold used for the extraction of the approaching object in S101 is switched to a threshold corresponding to the mode set at that time, and then the approaching object extraction determination is performed.

  Further, in the present embodiment, the conventional touch panel is not used, and the hand contact area on the operation panel is specified by analyzing the captured image captured by the camera 12b. However, a conventional touch panel may be adopted. As the touch panel, various types of touch panels such as a resistance film method, a capacitance method, and a vibration detection method can be applied.

  Moreover, in the said embodiment, as shown in FIG. 20, the said contact area is shown on the main image 200B so that the contact area with the touch operation surface 12a1 by touch operation can be specified on the screen of the display part 15. As shown in FIG. You may superimpose and display (code | symbol 200HA). Thereby, it becomes possible to visually recognize a touch input position, and operability increases. Specifically, the captured image of the camera 12b is binarized using the second gradation threshold value, and a contact area exceeding the second gradation threshold value is extracted. Then, the extracted image of the contact area image 150HA is processed by the image combining unit 18a, combined with the approaching object image 200H, and further combined with the main image 200B (semi-transparent combining here). A composite image as shown in FIG.

  Further, the present invention is characterized in that a specific biological state of the driver is detected from a face image. As the biological state, another biological state such as the driving attention reduction state or the sweating state may be specified, and a predetermined control corresponding to the biological state may be executed. Also, in the above embodiment, when a driving attention reduced state is identified, a warning activation work is performed for this, but not only a warning but also an action to urge awakening to return to the normal state at the same time May be.

The block diagram which shows the structure of the navigation apparatus for vehicles using the prompter system operation device for vehicles which is an example of the operation input device of this invention. BRIEF DESCRIPTION OF THE DRAWINGS The principal part perspective view in a vehicle interior for demonstrating the outline | summary of the vehicle prompter type operating device with which this invention was applied. The figure which shows the front of the driver's seat for demonstrating the arrangement | positioning state of an operation panel. The block diagram which shows the structure of the vehicle-mounted air conditioner operated with the operation input apparatus of this invention. Sectional drawing which shows the structure of an operation information input part. The driver's seat side view which shows the imaging | photography range of the camera provided in the operation information input part. The figure which illustrates the picked-up image of a camera. The figure which illustrates the translucent composition of the hand image with respect to the main image. The figure which illustrates the face image which shows a driver | operator's various biological states. The flowchart which shows a mode setting process. The flowchart which shows a biological condition corresponding | compatible control execution process. The flowchart which shows a hand image synthetic | combination display process. The flowchart which shows an example of a biological condition detection process. The flowchart which shows an example of a biological condition detection process. The flowchart which shows an example of a biological condition detection process. The flowchart which shows an example of a biological condition detection process. The flowchart which shows a touch input process. The driver's seat side view which shows embodiment which provided the head-up display as a display part. The figure which shows the front of a driver's seat in embodiment which provided the head-up display as a display part. The figure different from FIG. 8 which illustrates the translucent synthesis | combination of the hand image with respect to a main image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Vehicle navigation apparatus 12 Operation information input part (operation part)
12a Operation panel 12b Camera (approaching object photographing means, hand image photographing means, face image photographing means)
12c Light source 15 Display section (display means)
18. Control unit (light reflection area specifying means, input receiving means, hand image composition display control means, biological state specifying means (driving attention reduction state specifying means, sweating state specifying means), biological state correspondence control executing means, approaching object Detection means, mode setting means)
18a Image composition unit (hand image composition display control means)
50 In-vehicle LAN
101 ECU
150 Captured image 200B Main image 201 Switch image (operation icon)
200H approaching object image H approaching object

Claims (15)

A translucent operation panel provided on a steering wheel of a vehicle, the surface of which forms a touch operation surface;
Display means provided at a position different from the steering wheel so as to be visible from the driver, and capable of displaying an operation icon on the display screen;
Hand image photographing means for photographing a hand image of an operator of the touch operation surface, which is close to the touch operation surface side of the operation panel;
Hand image composition display control means for compositing and displaying the photographed hand image at a corresponding position on the display screen;
Based on an operation to the operation input position on the touch operation surface corresponding to the display position of the operation icon displayed on the display screen, an operation input having contents determined corresponding to the operation icon Input receiving means for receiving,
A vehicle operation input device comprising: The hand image photographing means is disposed with a fixed photographing range so that the face of the driver sitting on the driver's seat can be photographed from the back side of the operation panel via the operation panel. And also configured to function as a face image photographing means for photographing the driver's face image,
Biological state specifying means for specifying that the driver is in a predetermined biological state based on the photographed face image;
Control content storage means for storing biological state correspondence control information in which control contents of the in-vehicle electronic device corresponding to various biological states are associated;
When the biological state is identified, based on the biological state correspondence control information, biological state correspondence control execution means for executing the control content corresponding to the identified biological state;
A vehicle operation input device according to claim 1. An approaching object detection means for detecting an approaching object approaching the touch operation surface of the operation panel;
When the approaching object is detected, a biological state correspondence control prohibiting mode for prohibiting execution of the control content corresponding to the biological state by the biological state correspondence control execution unit is set, and the approaching object is If not detected, a mode setting unit that switches and sets a biological state correspondence control execution mode for executing the control content corresponding to the biological state by the biological state correspondence control execution unit;
The vehicle operation input device according to claim 2, further comprising:   The biological state specifying unit executes a biological state specifying process of specifying the driver's biological state based on the face image captured by the face image capturing unit when the biological state correspondence control prohibit mode is set. The vehicle operation input device according to claim 3, wherein The hand image photographing means is capable of photographing an approaching object including the operator's hand, which is close to the touch operation surface side of the operation panel.
The vehicle according to claim 3 or 4, wherein the approaching object detection means detects the approaching object in a form that specifies an image of the approaching object from a photographed image by the hand image specifying means. Operation input device. The driving state specifying means specifies a driving attention reduced state in which the driver's attention to the vehicle is reduced as the biological state based on the face image photographed by the face image photographing means. Having a power drop state identifying means,
The biological state correspondence control information includes a control content that causes the alerting means that performs a preset alerting action to alert the driver to the driver's attention reduction state. Associated with,
The biological state correspondence control execution means executes the alerting action based on the biological state correspondence control information when the driving attention reduction state is specified as the biological state. The vehicle operation input device according to claim 5. The biological state specifying means includes a sweating state specifying means for specifying a driver's sweating state as the biological state based on the face image photographed by the face image photographing means,
The biological state correspondence control information is associated with the perspiration state, the control content for causing the in-vehicle air conditioning output means to execute an air conditioning output of stronger wind or lower temperature than the current, or stronger wind and lower temperature,
The said biological condition corresponding | compatible control execution means performs the said air-conditioning output based on the said biological condition corresponding | compatible control information, when the said perspiration state is specified as the said biological condition. The vehicle operation input device according to claim 1. A light source for photographing that is fixedly arranged in a vehicle interior and irradiates light of a predetermined wavelength band with respect to a photographing object that enters the photographing range;
The vehicle operation input according to any one of claims 1 to 7, wherein the hand image photographing means photographs a subject to be photographed by capturing reflected light of light emitted from the photographing light source. apparatus. Comprising the requirements of claim 3;
In the biological state correspondence control execution mode, the luminance of the imaging light source is set to a first level. On the other hand, in the biological state correspondence control prohibition mode, the luminance of the imaging light source is set to a second level lower than the first level. The vehicle operation input device according to claim 8, which is set. On the image photographed by the hand image photographing means, comprising a light reflection area specifying means for specifying a light reflection area where the intensity of the reflected light exceeds a predetermined threshold intensity,
The vehicle operation input device according to claim 8 or 9, wherein the input receiving unit is configured to receive a touch input to the light reflection area based on the identified light reflection area.   The input receiving unit is configured to receive a touch operation input to the light reflection region when the light reflection region specified by the light reflection region specifying unit exceeds a predetermined threshold area. The vehicle operation input device according to claim 10.   The vehicle operation input device according to any one of claims 1 to 11, wherein the operation panel is provided on a boss portion of the steering wheel.   The vehicle operation input device according to claim 1, wherein the touch operation surface of the operation panel has a shape similar to the display screen.   The vehicle operation input device according to any one of claims 1 to 13, wherein the display means is provided in a form that is also used as a display device of an in-vehicle navigation device.   15. The vehicle head-up display device, wherein the display means includes a display that uses a windshield glass driver front area as the display screen and emits display light toward the windshield glass. The vehicle operation input device according to any one of the above.

Images