JP2009255904A - Operation input device for vehicle - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an operation input device for a vehicle in which any mistaken operation is hardly executed in a remotely operating unit. <P>SOLUTION: The operation input device for the vehicle is provided with an operation panel 12a and a display unit 15 for displaying a position indication image 200H to display an indication position on the operation panel 12a, and specifies whether or not the state of fingers facing the operation panel 12a is a finger state for displaying the intention to operate. When the vehicle is running, the display of the position indication image 200H is permitted if the finger state is specified, and the display of the position indication image 200H is prohibited if the finger state is not specified. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a vehicle operation input device.

  In recent years, an operation input device that remotely operates a device has been developed as an operation input device (Patent Document 1). Such an operation input device is provided with a remote operation unit at the operator's hand separately from a display device that displays an operation screen (main screen), and further, photographs and extracts a hand that operates the remote operation unit, The obtained hand image and manual operation are reflected on the operation screen (main screen) of the display device, and display control is performed as if the operation screen is directly touch-operated. Among such operation input devices, those provided in the vehicle are provided with the remote operation unit at the user's seat on the seat in order to increase the operability. The remote operation unit is often configured as a touch operation type operation unit having a touch operation surface.

JP 2000-6687 A

  In the operation input device as described above, since the remote operation unit is provided at the user's hand, there is a problem that the operation unit is unintentionally touched and erroneous operation input is performed. In particular, in the case of a touch operation type operation unit such as a touch panel, there is a case where an input is made only by touching with some sort of margin.

  Furthermore, in the operation input device as described above, when the hand operating the remote operation unit is actually displayed on the operation screen, when the hand is close to the remote operation unit, (c) in FIGS. As shown, only the finger is displayed. However, some users perform touch operation input with five fingers of one hand in order to input faster. In this case, since many fingers are displayed side by side on the operation screen, the correspondence between the displayed finger and the finger that the operator actually wants to use for position indication cannot be understood, which may lead to an erroneous operation. There was sex. In particular, since the driver who is driving the vehicle cannot stare at the operation screen, it is difficult to input an operation with a large number of fingers while driving, and an erroneous operation is likely to occur.

  It is an object of the present invention to provide a vehicle operation input device that is unlikely to be erroneously operated in a remote control unit.

Means for Solving the Problems and Effects of the Invention

In order to solve the above-described problem, the vehicle operation input device of the present invention includes:
A display means having a display screen for operating a device provided in the vehicle, and a remote having a remote operation surface provided at a position different from the display screen for remotely performing operation input on the display screen An operation unit, an input receiving unit that receives an input corresponding to the pressing operation position based on a pressing operation on the remote operation surface, a hand image capturing unit that captures an image of a hand facing the remote operation surface, and a position by the hand A vehicle operation input device comprising: a position instruction image display means for displaying a position instruction image generated based on a photographed hand image at a position on an indicated display screen,
A finger state specifying means for simple operation for specifying whether or not the finger state of the hand facing the remote operation surface is a predetermined finger state for simple operation;
The display mode set for the position instruction image display means is set to a display permission mode that permits display of the position instruction image when the simple operation finger state is specified, and the simple operation finger state is specified. Display mode setting means for executing display mode switching control for setting to a display prohibition mode for prohibiting display of the position indication image if not,
It is characterized by providing.

  According to the configuration of the present invention described above, when an operation is performed with a finger state having a high degree of difficulty using five fingers as an operation on the remote operation unit, the operation display (the operation state at the remote operation unit is displayed on the display device). (Displaying with a position instruction image on the operation screen)), it is possible to prompt the user to input an operation in a finger state (simple operation finger state) that is simpler and easier to operate. As a result, it is possible to prevent an erroneous operation to the remote control unit that is unintentionally, and it is not possible to understand the correspondence relationship between the displayed finger and the finger that the operator actually wants to use for position indication to the remote control unit. There is nothing lost.

  In the present invention, vehicle speed detecting means for detecting the vehicle speed of the vehicle can be provided. In this case, the display mode setting means executes display mode switching control when the vehicle is in a vehicle running state exceeding a predetermined vehicle speed level, and permits display as a display mode when the vehicle is not in the vehicle running state. It can be configured to set the mode. According to this configuration, for example, when the vehicle is stopped, operation input by a finger state having a high degree of difficulty is permitted as an operation for the remote operation unit, and thus it is possible to perform operation input in each easy form. On the other hand, while the vehicle is traveling at a certain speed or higher, the operation display is restricted, so the operation with a specific finger state that is simpler and easier to operate is forced as an operation on the remote control unit, and in particular, an erroneous operation by the driver occurs. It becomes difficult. Further, the correspondence between the displayed finger and the actually used finger is not lost.

  In the present invention, an operator specifying means for specifying whether or not the operator on the remote operation surface is a driver can be provided. In this case, when the driver is specified as the operator on the remote operation surface, the display mode setting means executes display mode switching control, and the driver is not specified as the operator on the remote operation surface. In this case, the display permission mode can be set as the display mode. According to this configuration, for example, if the operator on the remote operation surface is a person other than the driver such as a passenger on the passenger seat, operation input with a highly difficult finger state is permitted as an operation on the remote operation unit. It is possible to input operations in a form that is easy to do. On the other hand, when the operator on the remote operation surface is a driver, the operation display is restricted, so that the operation with a specific finger state that is simpler and easier to operate as the operation on the remote operation unit is forced, and by the driver Misoperations are less likely to occur. The correspondence between the displayed finger and the actually used finger is not lost.

  In the present invention, when the display permission mode is set by the display mode setting means, the input acceptance by the input acceptance means is permitted, and when the display prohibition mode is set by the input acceptance means. Operation input reception control means for performing operation input reception control for prohibiting input reception can be provided. In the operation input device of the present invention, if the operation display is not performed, the position instruction operation from the remote operation unit becomes extremely difficult. Therefore, only in the display permission mode in which the position instruction image is displayed as in the above configuration. By allowing the input to be accepted, it is possible to reliably prevent an erroneous operation on the remote operation unit. In other words, the position indication image can be displayed only when the input to the remote control unit can be accepted, so that it can be said that the useless display of the position indication image is reduced and power saving is effective.

  In the present invention, when the simple operation finger state is specified by the simple operation finger state specifying unit, the input receiving unit is allowed to accept the input, and the simple operation finger state is specified. If there is no operation input acceptance control means for executing an operation input acceptance control that prohibits input acceptance by the input acceptance means. According to this configuration, since the acceptance of the input to the remote operation unit is permitted only when the simple operation finger state is specified regardless of the display mode, the erroneous operation to the remote operation unit is surely prevented. be able to.

  Further, the simple operation finger state specifying means in the present invention can be configured to specify the simple operation finger state based on the hand image taken by the hand image photographing means. More specifically, the finger state identification means for simple operation in the present invention is a finger shape recognition means for recognizing the finger shape of the hand facing the remote operation surface based on the hand image photographed by the hand image photographing means. And the case where the recognized finger shape is a predetermined finger shape for simple operation can be specified as the finger state for simple operation. According to this configuration, the hand image photographed by the hand image photographing means can be used not only for the indication position display but also for specifying the finger state for simple operation. That is, since the photographing unit is also used as a configuration, useless additional configurations can be eliminated, and the cost can be reduced.

  The finger state specifying means for simple operation in the present invention specifies the number of fingers that are substantially extended along the remote operation surface when the finger shape recognition means recognizes a finger extending substantially along the remote operation surface. Means for specifying the number of fingers in a state extending substantially along the specified remote operation surface as a simple operation finger state when the number of fingers extending substantially along the specified remote operation surface is a predetermined number for simple operation And can. Counting the fingers in the stretched state can be easily performed by image analysis, and there are few errors in counting. Further, since the finger extended state is a finger state that can be used for the operation as it is, the operator can start the operation in the simple operation finger state, which is excellent in operability.

  The simple operation finger state specifying means according to the present invention is an operation number specifying means for specifying the number of inputs made by pressing operations performed at different positions on the remote operation surface, received by the input receiving means during a predetermined period. And the case where the number of inputs specified within the period is a predetermined number for simple operation can be specified as the finger state for simple operation. According to this configuration, the finger state for simple operation can be easily identified from a predetermined number of touch operations by the user.

  The number for the simple operation can be one. In the operation input device of the present invention in which manual operation is displayed, when operating the remote operation unit, the fewer the number of fingers used for the position instruction, the lower the operation difficulty. Therefore, the one-finger state with the smallest number of fingers is the simplest and easy-to-understand operation finger state. In addition, since the one-finger state has a very simple hand shape, the process of specifying the one-finger state can be facilitated.

  The number for the simple operation can be four, more preferably three, and even more preferably two. Since the two-finger state is also a relatively simple hand shape, the process for specifying the finger state for simple operation is easy. Three or four are relatively simple finger states. In addition, these finger states are not hand-shaped that a user accidentally forms, so that erroneous operation can be reliably prevented.

  The simple operation finger state specifying means in the present invention, when the finger shape recognition means recognizes a finger extending substantially along the remote operation surface, the finger type in a state extending substantially along the remote operation surface When the specified finger type specifying means is provided, and the finger type extending substantially along the specified remote operation surface is only a predetermined simple finger type for simple operation, It can also be set as the structure specified as a finger state. Since the hand shape with only a specific finger extended is not a hand shape that is accidentally formed by the user, it is possible to reliably prevent erroneous operation. Further, since the finger extended state is a finger state that can be used for the operation as it is, the operator can start the operation in the simple operation finger state, which is excellent in operability.

  The finger type for the simple operation can be the index finger only. Since the single-finger state of the index finger has a very simple hand shape, the process of specifying the single-finger state can be facilitated. Further, since the finger is often used for the operation, it is convenient to start the operation as it is.

  The finger type for the simple operation can be only the middle finger. Since the single finger state of the middle finger has a very simple hand shape, the process of specifying the single finger state can be facilitated. In addition, since it has the feature of the longest finger, it is easy to recognize the type. Further, since the finger is also used for the operation, it is convenient to start the operation as it is.

  The finger types for the simple operation can be only two of the index finger and the middle finger. Since the two-finger state of the index finger and the middle finger has a very simple hand shape, the process of specifying the two-finger state can be facilitated. In addition, since the middle finger, which is the longest finger, is included, finger type recognition is easy. Also, since both fingers are used for operation, it is convenient to start the operation as it is.

  The position indication image display means in the present invention generates a real finger image that enables at least the contour of the finger of the hand to be specified based on the hand image taken by the hand image photographing means, and the generated real finger image is displayed. It can be displayed as the position indicating image. Since the position of the hand can be grasped from the contour of the hand on the screen, the position can be easily recognized and the operability becomes easier.

  The position indication image display means in the present invention uses, as a position indication image, either a pseudo finger image in which at least the finger extending direction and the fingertip position can be specified, or a fingertip position image in which at least the fingertip position of the finger can be specified. It can be displayed. The processing load can be reduced by using an image simplified to such an extent that the finger pointing position can be understood as the position pointing image.

  The display mode setting means of the present invention executes display mode switching control when the vehicle is in a vehicle running state that exceeds a predetermined vehicle speed level (vehicle speed threshold), and permits display as a display mode when the vehicle is not in the vehicle running state. In the above-described configuration for setting the mode, the predetermined vehicle speed level can be set to 0 km / h. When the vehicle is stopped, the driver can concentrate on the operation, so there is no need to provide an operation restriction. According to the above configuration, since the position indication image is always displayed when the vehicle is stopped, the operator can freely operate.

  Further, the display mode setting means of the present invention executes display mode switching control when the vehicle is in a vehicle traveling state exceeding a predetermined vehicle speed level (vehicle speed threshold), and is set as a display mode when not in the vehicle traveling state. In the above-described configuration in which the display permission mode is set, the predetermined vehicle speed level is set within a vehicle speed range (slow driving state) that is faster than 0 km / h and slower than 5 km / h, or 5 km / h or more and 10 km / h. It can be determined within a vehicle speed range (deceleration state) of less than. Further, a predetermined vehicle speed level can be set within a vehicle speed range of 10 km / h or more and less than 30 km / h (low speed running state), or within a vehicle speed range of 30 km / h or more and less than 60 km / h (medium-side running). State) or within a vehicle speed range of 60 km / h or higher (high-speed running state). For these settings, an appropriate vehicle speed threshold value may be set according to the operation difficulty level (operation burden level) of each operation screen that is switched and displayed on the screen.

  In the present invention, an operation screen display means for displaying a plurality of operation screens, which are received by the input receiving means by a pressing operation on the remote operation surface, on the display screen, and a display mode among the plurality of operation screens. Display mode execution enable / disable information storage means for storing in advance display mode execution enable / disable information that enables specification of an operation screen to be subjected to switching control can be configured. In this case, the display mode setting means specifies whether or not the operation screen displayed by the operation screen display means is an execution target of the display mode switching control based on the display mode execution availability information. The display mode switching control can be executed on the specified operation screen only when it is specified as the execution target. According to this configuration, since the display mode switching control is executed only when a predetermined operation screen is displayed, the display mode switching control is performed, for example, on an operation screen that is easy to operate even while traveling. The display permission mode can be fixedly set without setting, and on the contrary, the display prohibit mode can be fixedly set without setting the display mode switching control on the operation screen that is difficult to operate while driving. Useless display mode switching control can be omitted.

  Display mode execution enable / disable information indicates that, among the operation screens, the display mode switching is performed for a complicated operation screen including an operation in which a pressing operation on a remote operation surface that is accepted by the input receiving unit exceeds a predetermined operation burden level. It is possible to determine a control operation target and a simple operation screen that includes only operations that do not exceed a predetermined operation burden level as a display permission mode fixed setting target. According to this configuration, since the display permission mode is fixedly set on the simple operation screen, the display mode switching control can be omitted.

  The simple operation screen can include at least one of a menu selection screen, a volume adjustment screen, and an air volume adjustment screen. Since the menu screen has a relatively small number of operation input images (operation icons) displayed on the screen, and the distance between adjacent operation input images has a certain width, the operation is relatively easy. Volume adjustment screens such as volume adjustment and air volume adjustment are also displayed on a relatively simple operation screen. Therefore, these operation screens are suitable to be determined as simple operation screens. On the other hand, the complex operation screen can include at least a character input screen. For character input of 50 sounds, the number of operation input images displayed on the screen is large, and as a result, the distance between adjacent operation input images is often short, so that an intended operation input image may be operated. difficult. Therefore, such an operation screen is suitable to be determined as a complex operation screen.

The block diagram which shows the structure of the navigation apparatus for vehicles using the operation input device for vehicles which is an example of the touch-type operation input device of this invention. The principal part perspective view in a vehicle interior for demonstrating the outline | summary of the operation input apparatus for vehicles. Sectional drawing which shows the 1st structure of an operation information input part. The 1st figure explaining photography and display of a hand which counters an operation panel in a first embodiment of the present invention. The 2nd figure explaining photography and display of a hand which opposes an operation panel in a first embodiment of the present invention. The 3rd figure explaining photography and a display of the hand which opposes an operation panel in 1st embodiment of this invention. FIG. 9 is a fourth diagram illustrating photographing and display of a hand facing the operation panel in the first embodiment of the present invention. The figure which shows an example of the image image | photographed in order to identify an operator. The flowchart which shows the position instruction | indication image display process in 1st embodiment of this invention. The flowchart which shows the touch operation input process in 1st embodiment of this invention. The flowchart which shows the position instruction | indication image display process of 2nd embodiment of this invention. The flowchart which shows the position instruction | indication image display process of 3rd embodiment of this invention. The flowchart which shows the position instruction | indication image display process of 4th embodiment of this invention. The flowchart which shows the touch operation input process in 5th embodiment of this invention. The perspective view of the principal part in a vehicle interior for demonstrating the outline | summary (6th embodiment) of the operation input device for vehicles different from FIG. The setting example of the vehicle speed threshold value in 2nd embodiment thru | or 4th embodiment. The flowchart which shows the position instruction | indication image display process of 7th embodiment of this invention. The figure which shows an example of the display permission period which can be set in this invention. The figure which shows the display permission period in 7th embodiment of this invention. The figure which shows the display permission period in 8th embodiment of this invention. The flowchart which shows the flow of the highlight display process of the position instruction | indication image in 9th embodiment of this invention. An example of the operation screen in this invention. An example of the operation screen in this invention. An example of the operation screen in this invention. The flowchart which shows the flow of the highlight display process of the position instruction | indication image in 10th embodiment of this invention. The flowchart which shows the 1st example of the specific process of the operation target finger in 10th embodiment of this invention. The 1st figure explaining imaging | photography and a display of the hand which opposes an operation panel in 10th embodiment of this invention. The 2nd figure explaining photography and display of a hand which opposes an operation panel in a 10th embodiment of the present invention. The figure explaining the specific method of a fingertip. FIG. 27 is a diagram illustrating a first example of a method for specifying an operation target finger in the embodiment of FIG. 26. The figure explaining the 2nd example of the method of specifying an operation target finger in embodiment of FIG. The figure explaining the 3rd example of the method of specifying an operation target finger in the embodiment of FIG. The figure which shows an example of the time change of the highlight display of the position instruction | indication image in 10th Embodiment of this invention. The flowchart which shows the 2nd example of the specific process of the operation target finger in 10th Embodiment of this invention. The figure explaining the 1st example of the method of specifying an operation target finger in the 2nd example of 10th embodiment of the present invention. The figure explaining the 2nd example of the method of specifying an operation target finger in the 2nd example of 10th Embodiment of this invention. The flowchart which shows the 3rd example of the specific process of the operation target finger in 10th embodiment of this invention. The figure explaining an example of the method of specifying an operation target finger in the third example of the tenth embodiment of the present invention. The flowchart which shows the 4th example of the specific process of the operation target finger in 10th embodiment of this invention. The flowchart which shows the 5th example of the specific process of the operation target finger in 10th Embodiment of this invention. The figure explaining an example of the method of specifying an operation target finger in the fifth example of the tenth embodiment of the present invention. The flowchart which shows the 6th example of the specific process of the operation target finger in 10th embodiment of this invention. The figure explaining an example of the method of specifying an operation target finger in the sixth example of the tenth embodiment of the present invention. The flowchart which shows the 7th example of the specific process of the operation target finger in the tenth embodiment of the present invention. The 1st figure explaining the method of specifying an operation target finger in the 7th example of a 10th embodiment of the present invention. The 2nd figure explaining the method of specifying an operation target finger in the 7th example of a 10th embodiment of the present invention. The figure which shows the example of 1 display of the highlight display of the position instruction | indication image in 10th embodiment of this invention. The figure which shows the example of 1 display of the highlight display of the position instruction | indication image in 10th embodiment of this invention. The perspective view which shows simply the operation part which presupposes knob operation in the 11th example of 10th Embodiment of this invention. An example of the picked-up image from the x direction, y direction, and z direction shown in FIG. An example of the display image based on FIG. An example of the display image based on FIG. 50 different from FIG. The figure explaining an example of the method of specifying an operation target finger in the eleventh example of the tenth embodiment of the present invention. The flowchart which shows the flow of a finger tracking process. The flowchart following FIG. The figure explaining the tracking information memorize | stored in the control circuit which estimates a fingertip position. The figure explaining the basic algorithm which tracks a fingertip between moving image frames. The figure which shows the calculation method of the appearance predicted position of a fingertip (finger). The figure explaining the calculation method of the appearance predicted position in the next frame of the fingertip which newly appeared in the previous frame. The figure explaining the new appearance of a fingertip and the loss of a fingertip. The figure which shows an example of the time change of the highlight display of the position instruction | indication image in 10th Embodiment of this invention. The figure which shows an example of the time change of the highlight display of the position instruction | indication image in 10th Embodiment of this invention. The figure which shows the example of 1 display of the highlight display of the position instruction | indication image in 10th embodiment of this invention. The figure which shows the example of 1 display of the highlight display of the position instruction | indication image in 10th embodiment of this invention. The figure which shows the example of 1 display of the highlight display of the position instruction | indication image in 10th embodiment of this invention. The figure which shows one example of a display which displayed the position instruction | indication image in this invention with the pseudo finger image. The figure which shows the example of a display which displayed the position instruction | indication image in this invention with the fingertip position image. The figure explaining the 2nd example of the finger state for simple operations. The principal part perspective view in a vehicle interior for demonstrating the outline | summary (6th embodiment) of the operation input device for vehicles different from FIG.2 and FIG.15. The figure explaining the 3rd example of the finger state for simple operations. The figure explaining the 4th of the finger state for simple operations. The figure explaining the 5th example of the finger state for simple operations. The figure which shows the example of the screen conditions used as the implementation target of display mode switching control.

  Hereinafter, a first embodiment of the operation input device of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of a vehicle navigation device using the operation input device of the present invention. In the present embodiment, as the operation input device 1, a display device 15 capable of displaying an operation input image such as a switch image (operation icon) for touch operation on the display screen, and a touch operation on the display screen are remotely performed. An operation input device is employed in which an operation panel 12a for performing the operation is provided at a different position. Note that the operation target of the touch-type operation input device of the present embodiment is not limited to the navigation function, but is a device function (audio, air conditioner, power window, etc.) of other in-vehicle electronic devices controlled by the ECU 100 connected via the in-vehicle LAN 50. Sheet adjustment function, etc.).

  The navigation device 10 includes a position detector 11 that detects the current position of the vehicle, an operation information input unit 12 for inputting various instructions from an operator such as a driver, and an external that records map data and various information. A map data input device 14 for inputting map data and the like from a recording medium, a display device 15 for performing various displays such as a map display screen and a TV (Television) screen, and outputting various guide voices, etc. A voice input / output device 16 for inputting the voice of the operator, a hard disk storage device 17 for storing various data such as vehicle information, and a vehicle I / F (for transmitting and receiving vehicle information) Interface) unit 19, a wireless communication device 13 for performing short-range wireless communication with another communication device 13 a, and a control circuit 18 connected to these 11 to 19. The control circuit 18 can also be connected to the cameras 20 and 40 and the vehicle speed sensor 30. In the present embodiment, as shown in FIG. 69, the camera may be configured only as a camera 12b described later.

  The position detector 11 receives a radio wave transmitted from an artificial satellite for GPS (Global Positioning System) via a GPS antenna and is added to the vehicle and a GPS receiver 11a that detects the position, direction, speed, and the like of the vehicle. A gyroscope 11b that detects the magnitude of the rotational motion, a distance sensor 11c that detects the distance traveled from the longitudinal acceleration of the vehicle, and the like, and a geomagnetic sensor 11d that detects the traveling direction from the geomagnetism. Each of the sensors 11a to 11d has an error of a different property, and is configured to be used while complementing each other. Depending on the accuracy, some of the sensors described above may be used, or a steering rotation sensor, a wheel sensor for each rolling wheel, or the like (not shown) may be used.

  As shown in FIG. 3, the operation information input unit 12 irradiates light of a predetermined wavelength band, and transmits the light emitted from the light source 12c from the surface 12a2 side to the surface 12a1 side. A translucent operation panel 12a is provided, and a camera (approaching object photographing means) 12b that photographs from the back surface 12a2 side of the operation panel 12a so as to include the operation panel 12a in the photographing range. .

  As shown in FIG. 2, the operation information input unit 12 of the present embodiment is operated from both right and left seats below the lower edge of the windshield of the vehicle, such as the center console unit C of the vehicle. It is installed in a possible position. More specifically, it is disposed below the display device 15 and on the front side between both seats of the adjacent left and right seats. Note that the adjacent left and right seats in the present embodiment are the driver's seat 2D and the passenger seat 2P in FIG. 2, but may be, for example, the left and right seats in the rear row. Furthermore, the position is not limited to a position that can be operated from both the left and right seats, and may be installed at a position where the operator can easily operate, for example, a steering section or a door section.

  The operation panel 12a is a panel having translucency with respect to light emitted from at least the light source 12c. In this embodiment, it is configured as a well-known resistive film type touch panel provided with a vertical transparent electrode and a horizontal transparent electrode, and its output is input to the control circuit 18.

  As shown in FIG. 3, the operation panel 12a is fitted into the front end opening of the hollow casing 12f, and the surface 12a1 is a touch operation surface (remote operation surface). A cylindrical wall portion 122e is provided from the outer edge of the housing front end portion 121e forming the front end opening toward the rear, and the rear end portion is fixed to the circuit board 12k. The housing 12f is fixedly disposed at a predetermined position in the vehicle interior, and a camera 12b and a light source 12c are disposed inside.

  As shown in FIG. 3, the light source 12c irradiates light in such a manner that the operation panel 12a is transmitted from the back surface 12a2 to the front surface 12a1. Light (irradiation light) from the light source 12c passes outside from the touch operation surface 12a1 unless there is an approaching object (such as a hand) positioned so as to cover the touch operation surface 12a1, When there is an approaching object H that is approaching the touch operation surface 12a1, it is reflected by the approaching object H and part of the light (reflected light) returns to the back surface 12a2 side, and the camera 12b. To be captured.

  The camera 12b captures the reflected object reflected from the approaching object H by the light emitted from the light source 12c, and photographs the approaching object H from the back surface 12a2 side of the operation panel 12a. It is fixed on the vehicle body side to shoot the shooting range. Specifically, a reflecting member 12r is provided so that the photographing range can be photographed by the camera 12b, and the camera 12b photographs the reflected image. When there is no reflecting member 12r, the photographing range is directly photographed by the camera 12b. Therefore, it is necessary to perform left-right reversal processing on the photographed image.

  In this embodiment, the light source 12c is an infrared light source, and the camera 12b includes an infrared camera (not shown) that shuts out visible light and is suitable for shooting in a dark place. Do). Photographing is performed in such a manner that the camera 12b captures the reflected light reflected from the approaching object H by the light (infrared rays) emitted from the infrared light source 12c. The camera 12b is fixedly arranged on a substrate fixed to the vehicle body side, and a range including the touch operation surface 12a1 is set as a shooting range. The infrared light source 12 is an LED located on the rear side of the operation panel 12a. In addition, since the operation panel 12a according to the present embodiment is made of a material having translucency only with respect to the light from the light source 12c, an infrared filter in the camera 12b can be eliminated, and The inside of the housing 12f is not visible from the outside.

  Note that the camera 12b in this embodiment functions as a photographing unit. This camera 12b functions as an approaching object photographing means for photographing an approaching object approaching the operation panel 12a, and considering that the approaching object includes a hand, the hand It can be said that it functions as a hand image photographing means for photographing an image, and further as a finger image photographing means for photographing an image of a finger of a hand.

  4 to 7 are diagrams for describing photographing of the hand (finger) H facing the operation panel 12a. (A) of each figure has shown the state (finger state) of the operation panel 12a and the hand which opposes the surface 12a1, and (b) of each figure shows the hand of the state of (a) with the camera 12b. The image 150 is taken in the above, and is an image taken using the entire operation panel 12a as a shooting area. However, the video 150 of (b) in this embodiment is a binarized version of the video after shooting. In (b), reference numeral 150H is an area where the approaching object H is reflected, and reference numeral 150B is an area where nothing is reflected. The region denoted by reference numeral 150B is originally a region where the operation panel 12a is reflected, but is not reflected because the operation panel 12a has translucency with respect to the light from the light source 12c. Even in such a case, it is not strange that the background (the ceiling surface of the vehicle in the present embodiment) is reflected through the operation panel 12a, but such background is removed. Background removal is possible by using images with different illumination conditions, as described in Japanese Patent Application Laid-Open No. 2007-272596. Incidentally, (c) of each figure is an example of a screen display on the display device 15 when the video 150 of (b) is taken. However, FIG. 4C is an example of the screen display in the conventional prompter type operation input device. In this embodiment, when the image 150 as shown in FIG. 4B is displayed, FIG. ) Is displayed on the screen.

  Returning to FIG. 1, the display device 15 is a color display device installed on the front side of the driver's seat 2 </ b> D in FIG. 2 so as to be visible by the driver, and the display screen is farther from the driver than the touch operation surface 12 a 1. In the position. As the display device 15, any of a liquid crystal display, a plasma display, an organic EL display, or the like may be used. The display screen of the display device 15 includes a mark indicating the current location identified from the current position of the vehicle detected by the position detector 11 and the map data input from the map data input device 14, a guidance route to the destination, a name, Additional data such as landmarks and marks of various facilities can be displayed in an overlapping manner. Also, facility guides can be displayed. The display device 15 corresponds to the display means of the present invention.

  In addition, the display device 15 has a display screen for performing operations of devices included in the vehicle, and in the present embodiment, the display device 15 includes a touch panel 15a on the display screen. The touch panel 12a of the operation information input unit 12 already described is a remote operation unit provided so that the operation of the touch panel 15a can be remotely operated from a position closer to the driver or passenger on the passenger seat. Based on the touch operation performed on the touch operation surface 12a1, an input to the screen position of the display device 15 (touch operation surface position of the touch panel 15a) corresponding to the touch operation position is accepted. More specifically, a two-dimensional coordinate system (two-dimensional coordinate system defined on the touch operation surface of the touch panel 15a) defined on the display screen of the display device 15 and a touch operation surface 12a1 of the operation panel 12a are defined. An unambiguous correspondence between the two-dimensional coordinate system is determined in advance, and the other in-plane position corresponding to the one in-plane position can be uniquely identified, and the touch operation surface 12a1. If there is a touch operation, an operation input to a position coordinate on the display screen of the display device 15 corresponding to the position coordinate where the touch operation is performed is received based on the correspondence.

  That is, when there is a touch operation on the touch operation surface 12a1, the control circuit 18 specifies the position coordinates on the touch operation surface 12a1 where the touch operation is performed, and further, the position on the specified touch operation surface 12a1. A position coordinate on the display screen of the display device 15 corresponding to the coordinate is specified, and then a control signal for executing control content corresponding to the specified position coordinate on the display screen is output. For example, when a switch image (operation icon or the like) 200I is displayed on the display screen of the display device 15, if the touch operation is performed at a position corresponding to the switch image 200I on the touch operation surface 12a1, the control circuit 18 Then, an input is accepted on the assumption that there has been a touch operation on the switch image 200I, and accordingly, the control content corresponding to the switch image 200I is executed. If a map screen (map operation image) that can be displayed on the display screen of the display device 15 is displayed, the control circuit 18 can be operated by touching the position on the map on the touch operation surface 12a1. Then, an input to the touch operation position is accepted, and accordingly, display feed control for displaying a map with the touch operation position as a new screen center is executed.

  Note that the display device 15 is not limited to the display device 15 disposed in the center console C as shown in FIG. 2, and may be installed above the lower edge of the windshield FG, for example. Specifically, a head-up display (HUD) for displaying the above-described image / data on the windshield FG of the vehicle, a meter display device located behind the steering wheel, and the like can be exemplified.

  Returning to FIG. 1, the map data input device 14 is road data as network data, map data such as so-called map matching data for improving the accuracy of position specification, mark data indicating facilities, guidance image and audio data. It is a device for inputting various data including the like. A CD-ROM, DVD-ROM, hard disk, memory, memory card, or the like can be used as a recording medium for these data.

  The voice input / output device 16 can output facility guides and various guidance voices input from the map data input unit 14, and read-out voices of information acquired via the I / F 19. The voice input / output device 16 includes a microphone (not shown) and a known voice recognition unit, and can input voice of an operator such as a driver to the control circuit 18 as a command or the like.

  The wireless communication device 13 is for performing narrowband communication with other communication devices 13a. For example, DSRC (Dedicated Short Range Communications), Bluetooth (registered trademark), wireless LAN (Local Area Network), UWB (Ultra Wideband). ) Etc. are used.

  The LAN I / F 19 is an interface circuit for exchanging data with other in-vehicle electronic devices and sensors via the in-vehicle LAN 50. Further, for example, data from another ECU (ECU 100 in FIG. 1 or the like) may be taken in via the LAN I / F 19.

  The camera 20 is provided in the rear mirror peripheral part at the front upper part of the vehicle, and photographs the peripheral region including the touch operation surface 12a of the operation panel 12a. More specifically, for example, a photographed image 300 as shown in FIG. 8 is photographed, and the photographing range is determined so that not only the operator's hand of the operation panel 12a but also the arm following that hand is reflected. In the control circuit 18 that acquires the photographed peripheral image of the operation panel, the direction of the arm extending from the hand is specified by the image processing unit 18a to determine whether or not the operator is a driver. In the photographed image 300 in FIG. 8, reference numeral 300H is a hand image, reference numeral 300A is an image of an arm that continues from the hand, reference numeral 312a is an area on which the operation panel 12a is reflected, and the operator's arm of the operation panel 12a is the captured image. Since it extends from the right side, that is, from the driver's seat side, the operator is identified as the driver. In the present embodiment, the camera 20 functions as an operator specifying unit together with the control circuit 18 (image processing unit 18a). It should be noted that another method may be used for specifying whether or not the operator of the operation panel 12a is a driver. For example, an approaching object detection unit (for example, a reflective optical sensor such as an infrared reflection sensor) that detects an approaching object may be provided on each of the left and right sides of the touch operation surface 12a to identify the operator.

  The vehicle speed sensor 30 includes a rotation detection unit such as a known rotary encoder, and is installed near the wheel mounting unit, for example, to detect the rotation of the wheel and send it to the control circuit 18 as a pulse signal. In addition to converting the number of rotations of the wheel to the vehicle speed (vehicle speed), the control circuit 18 calculates an estimated arrival time from the current position of the vehicle to a predetermined location, and calculates an average vehicle speed for each travel section of the vehicle. Or calculate. In the present embodiment, the vehicle speed sensor 23 is vehicle speed detection means, and the control circuit 18 receives the detection result and determines whether or not the vehicle is in a predetermined low-speed driving state. Function.

  The control circuit 18 is configured around a well-known microcomputer including a well-known CPU, ROM, RAM, I / O (Input / Output) (not shown) and a bus line connecting these components. Based on each detection signal from the position detector 11 based on a program stored in the storage unit, the current position of the vehicle is calculated as a set of coordinates and traveling directions, and the vicinity of the current position read via the map data input device 14 A map display process for displaying a map of the range indicated by the operation of the operation information input unit 12 on the display device 15 as a main image (including moving images and still images) 200B as shown in FIG. Based on the point data stored in the map data input device 14, the facility as the destination is selected according to the operation of the operation information input unit 12, and the current position to the destination are selected. A route guidance process of performing route guidance performed automatically route calculation to find the optimal path. A technique such as the Dijkstra method is known as a technique for automatically setting an optimum route in this way.

  In addition, the control circuit 18 sub-switches a switch image (operation icon) 200I, a processed image 200H based on the approaching object image 150H acquired from the operation information input unit 12, and the like based on a program stored in a storage unit such as a ROM. The image processing unit 18a is used to display the image as a main image 200B (a map operation image or the like), or to display the image as an image, or to display it superimposed. In the present embodiment, when the approaching object image 150H is an image of a hand such as a finger, the processed image 200H generated based on the photographed hand image 150H is used as a position instruction image, and the position is determined by the hand. Overlays at the indicated position on the display screen. That is, the processed image 200H is overlaid and displayed as a position indicating image 200H indicating the position on the display screen where the position is specified by the finger of the hand.

  In the present embodiment, the processed image 200H based on the approaching object image 150H is an approaching object contour reflecting image that is processed and generated so that at least the outer shape (contour) of the approaching object image 150H is reflected. 200H. Then, this image 200H is superimposed on the main image 200B. Instead of the superimposed display, the image 200H may be combined with the main image 200B and displayed. As a result, when an operator such as a driver moves his / her finger or the like so as to face the touch operation surface 12a1 of the operation panel 12a, for example, a hand is displayed on the display screen of the display device 15. The processed image 200H reflecting the shape (finger shape) is moved and displayed, and the operation can be performed as if the operation panel 12a exists on the display screen of the display device 15.

  Further, the position instruction image 200H does not necessarily reflect the photographed hand shape / finger shape outline, and may be an image that indicates at least the position indicated by the photographed hand finger. It may be only a portion, only a fingertip, or a symbol image such as a pointer image. Moreover, there is no problem if the hand image (or finger image) 150H photographed by the camera 12b is displayed as it is without being processed (superimposed display or composite display) as the position indication image 200H, but the position overlapping image In view of the fact that a part of the main image 200B that should be displayed on the screen becomes invisible, it is more desirable to perform processing (for example, translucent synthesis) that makes the operation easier.

  Further, switch image data for displaying the switch image 200I is stored in a storage unit such as a ROM of the control circuit 18, and is used for overlapping display or composite display of the switch image 200I. On the other hand, in a storage unit such as a ROM of the control circuit 18, an operation input (touch input to a corresponding area on the touch operation surface 12 a 1) is performed in association with each switch image 200 I. The control content is also stored. When the control circuit 18 displays the image 200 in which the switch image 200I is superimposed on the main image 200B displayed on the display screen, the switch image 200I is displayed at the display position on both touch operation surfaces of the touch panels 12a and 15a. An operation input reception range (operation input position) is set in the corresponding area (operation input position setting means). When a map screen (map operation image) 200B2 that can be displayed and sent as the main image 200B is displayed on the display screen, an operation input reception range (operation input position) is set for the entire displayed map area. (Operation input position setting means).

  Further, the control circuit 18 can perform time counting by starting its own timer 18b and counting up the counter 18c. These are used when measuring time in the processing described later.

  The control circuit 18 functions as input receiving means, position indication image display means, operator specifying means, running state specifying means, simple operation finger state specifying means, and operation input control means of the present invention. Note that the image processing unit 18a also has a part of functions as a position instruction image display unit and an operator identification unit.

  Next, display processing of the position indication image displayed on the display device 15 will be described with reference to FIG. This process is performed in such a manner that the CPU executes a program stored in the storage unit of the control circuit 18.

  In S1, the control circuit 18 determines whether an image of the operator's hand has been acquired based on the captured image captured by the camera 12b. The camera 12b always shoots an approaching object (for example, an operator's hand such as a driver) H that approaches the touch operation surface (front surface) 12a1 side from the back surface 12a2 side of the operation panel 12a via the operation panel 12a. The captured image is always input to the image processing unit 18a of the control circuit 18. When the approaching object H enters the position facing the touch operation surface 12a1, the captured image 150 including the approaching object H is naturally input to the image processing unit 18a. The input captured image 150 is analyzed by a known image analysis method in the image processing unit 18a, and an approaching object image 150H is extracted based on a color difference or the like. Then, from the shape of the extracted approaching object image 150H, it is determined whether or not the image 150H is an image of a human hand. The determination result is output to the control circuit 18, and if it is determined that the hand image has been recognized, the process proceeds to S2, and if not determined, the present process is terminated.

  In the present embodiment, the light source 12c is arranged, and the camera 12b performs imaging in a manner that captures the reflected light of the light source 12c. Therefore, the reflected light intensity is reflected in the gradation of the color appearing in the captured image 150, and the higher the reflected light intensity, the higher the gradation level. That is, since the captured video 150 captured by the camera 12b of the present embodiment is a multi-gradation captured image (monochrome captured image in the present embodiment), the image processing unit 18a converts the captured image 150 into a predetermined gradation. The gradation level of each pixel is binarized using the threshold value, and the area exceeding the gradation threshold value is extracted as the approaching object image 150H as shown in FIGS. 4 to 7B. Further, in the image processing unit 18a, in order to specify an image of a human hand from the extracted approaching object image 150H, the shape of the approaching object image 150H and the hand shape stored in the storage unit of the control circuit 18 are used. Compared with patterns, the approaching object image 150H that matches these patterns is recognized as a hand image (hand image recognition means: hand shape recognition means).

  In S2, based on the acquired hand image, it is specified whether or not the hand state is a predetermined simple operation finger state (simple operation finger state specifying means). The simple operation finger state is, for example, a state where the number of fingers that can be an operation target, that is, the number of fingers extending substantially along the touch operation surface 12a is about one or two. In other words, an operation using five fingers to speed up input is a highly difficult finger state, and the simple operation finger state here is different from such a finger state. In the present embodiment, only when the acquired hand image is in a one-finger operation state in which only one finger appears in the surface direction (the direction in which the surface expands) of the touch operation surface 12a. It is specified as the finger state for simple operation. More specifically, in S1, the image processing unit 18a that has extracted the hand image identifies a substantially linear rod-shaped image pattern having an axial line longer than a predetermined length from the overall shape of the hand image. When there is the stick-like image pattern, it is recognized and specified as a finger image 150F (finger specifying means). The specified result is output to the control circuit 18, and the control circuit 18 specifies the number of the finger images 150F (number of finger specifying means: hand shape recognition means), and the case where the number is one is predetermined. It is specified that the state is a simple operation finger state (simple operation finger shape).

  In subsequent S3, based on the output result, it is determined whether or not the current finger state is the simple operation finger state. If it is determined that the current state is the simple operation finger state, the process proceeds to S4. In S4, the display permission mode in which the display of the position indication image is permitted is set as the display mode, and the process proceeds to S5. The set display mode is stored in a predetermined storage area (display mode storage unit) provided in the storage unit of the control circuit 18. In S4, the display permission mode is stored in the storage area.

  In S5, the control circuit 18 specifies the positional relationship between the display screen of the display device 15 and the photographed hand. That is, the position on the display screen where the photographed hand image H faces is specified. In the present embodiment, the operation panel image area where the touch operation surface 12a is located is specified on the captured image captured by the camera 12b, and the operation coordinate system of the touch operation surface 12a is set on the operation panel image area. Then, the position of the captured hand image H is specified on the set operation coordinate system. Then, in the display coordinate system defined on the display screen of the display device 15, a position corresponding to the position of the hand image H specified on the operation coordinate system is specified. In the present embodiment, the captured image (binarized image) 150 shown in FIG. 5 to FIG. 7B is an operation panel image area obtained by capturing the entire touch operation surface 12a1.

  Further, in S6, as shown in FIG. 6 and FIG. 7C, the position indicating image is displayed at the position corresponding to the hand image H specified on the operation panel image area on the display screen of the display device 15. 200H is displayed (position indication image display means). In the present embodiment, a processed image 200H in which the outer shape of the hand image 150H is clearly displayed and the inside of the contour is translucent is generated, and the main image 200B displayed on the screen of the display device 15 is generated. Are overlaid at the corresponding coordinate positions.

  On the other hand, if it is determined in S3 that the current finger state is not the simple operation finger state, the process proceeds to S7. In S7, the display mode is set to a display prohibition mode in which the display of the position indication image is prohibited. Thereby, the display of the position instruction image performed in S5 and S6 is not executed, and the position instruction image 200H is not displayed on the display screen of the display device 15, for example, as shown in FIG. The set display prohibition mode is stored in a predetermined storage area (display mode storage unit) provided in the storage unit of the control circuit 18.

  This process is ended by the end of S6 and S7. Note that this processing is repeatedly performed at a predetermined cycle after the end, and when the hand in the simple operation finger state that faces the operation panel 12a moves, the position instruction image displayed on the display screen of the display device 15 in accordance with this movement. (Processed image) The display position of 200H also moves.

  In the position indication image display process, as shown in FIG. 18, a period until the finger state for simple operation is unspecified is defined as the display permission period. That is, the processing is such that the display permission mode is set when the simple operation finger state is specified, and the display prohibition mode is set when the simple operation finger state is not specified. The control circuit 18 functions as the display mode setting means of the present invention by executing the display process of the position indication image.

  Finally, a touch operation input process on the operation panel 12a of the operation information input unit 12 will be described with reference to FIG. This processing is also performed in such a manner that the CPU executes a program stored in the storage unit of the control circuit 18.

  In the touch operation input process here, when the display permission mode is set, the acceptance of the touch operation input to the operation panel 12a is permitted (the control circuit 18 sets the input permission mode), and the display prohibit mode is set. If it is set, the process includes operation input reception control for prohibiting reception of the touch operation input (the control circuit 18 sets the input prohibit mode). The control circuit 18 functions as an operation input control means of the present invention by executing this touch operation input process.

  In S101, the control circuit 18 determines whether or not there is a touch operation on the operation panel 12a. Since the operation panel 12a is configured as a touch panel, when a touch operation is performed, an operation signal is input to the control circuit 18, and thus determination is made based on the presence or absence of the input. If there is a touch operation on the operation panel 12a, the process proceeds to S102, and if there is no touch operation, this process ends.

  In S102, the control circuit 18 determines whether or not the current display mode is the display permission mode. Since the current display mode is stored in a predetermined storage area (display mode storage unit) provided in the storage unit of the control circuit 18, the current display mode is determined based on the stored display mode information. When it is determined that the current display mode is the display permission mode, the process proceeds to S103, and when it is determined that the current display mode is not the display permission mode, that is, when it is determined that the display prohibition mode is set, this process ends.

  In S103, the touch operation position on the operation panel 12a, that is, the touch operation position on the touch operation surface 12a1 is specified. Since the operation panel 12 a is configured as a touch panel, an operation signal is input to the control circuit 18 when a touch operation is performed. Since the touch operation position is reflected in the operation signal as coordinate information on the operation coordinate system defined on the operation surface 12a1, the control circuit 18 specifies the touch operation position based on the operation signal. .

  In subsequent S104, the control circuit 18 performs control corresponding to the touch operation position on the operation panel 12a. Specifically, since each position on the touch operation surface 12a1 of the operation panel 12a corresponds to a position on the display screen of the display screen 15, the position on the touch operation surface 12a1 specified in S103. The corresponding control content determined at the position on the display screen of the display screen 15 is executed. Specifically, the control circuit 18 outputs a control signal for performing the control content. For example, if the position on the touch operation surface 12a1 specified in S103 corresponds to the position of the switch image 200I displayed on the display screen of the display screen 15, the control circuit 18 is assigned to the switch image. A control signal for executing the control content is output.

  The process ends when S104 ends. Note that this process is repeatedly performed at a predetermined cycle even after the completion of the process, and the control circuit 18 performs the control content corresponding to the position touched at that time or is performed by another control unit (for example, the ECU 100 or the like). Let

  In this touch operation input process, when the display permission mode is set, the acceptance of the touch operation input to the operation panel 12a is permitted (input permission mode), and when the display prohibit mode is set. The reception of the touch operation input is prohibited (input prohibition mode). However, in the position instruction image display process of FIG. 9, the display permission mode is set when the simple operation finger state is specified, and the display prohibition mode is set when the simple operation finger state is not specified. Therefore, as a result, the touch operation input to the operation panel 12a is not accepted unless the user's finger state is the simple operation finger state. In the above embodiment, the touch operation input to the operation panel 12a is not accepted unless the user's finger state is the simple operation finger state.

  Although one embodiment of the present invention has been described above, this is merely an example, and the present invention is not limited to this, and the knowledge of those skilled in the art can be used without departing from the spirit of the claims. Various modifications based on this are possible. Hereinafter, an embodiment different from the above embodiment will be described as the first embodiment. In addition, about a common structure, description is abbreviate | omitted by using the same code | symbol.

  Hereinafter, a second embodiment of the present invention will be described.

  In the first embodiment, the display permission mode is set to permit display of the position instruction image when the simple operation finger state is specified, and the position instruction is displayed when the simple operation finger state is not specified. Although it was configured to set to a display prohibition mode that prohibits image display, such display mode switching control is performed when the vehicle is in a predetermined traveling state, and when the vehicle is not in a predetermined traveling state, The display permission mode can be set as the display mode. In this case, the vehicle running state detecting means for detecting the running state of the vehicle is provided. FIG. 11 shows an example of such processing. Here, a vehicle speed sensor (vehicle speed detecting means) 30 for detecting the vehicle speed of the vehicle is connected to the control circuit 18, and the control circuit 18 is predetermined by the vehicle. The display mode switching control is executed when the vehicle is in a vehicle running state that exceeds the vehicle speed level (vehicle speed threshold), and when the vehicle is not in the vehicle running state, the display permission mode is set as the display mode regardless of the finger state. It is supposed to be set.

  Specifically, in S21, the control circuit 18 determines whether an image of the operator's hand has been acquired. This is the same as the processing of S1 in FIG. When the image of the operator's hand is acquired, the process proceeds to S22, and when the image of the operator's hand is not acquired, this process ends.

  In S22, the current running state of the vehicle is specified. That is, it is determined whether or not the vehicle is in a traveling state exceeding a predetermined vehicle speed level. The control circuit 18 stores a vehicle speed threshold value in a predetermined storage unit such as a ROM in advance, determines whether the vehicle speed detected by the vehicle speed sensor 30 exceeds the vehicle speed threshold value, specifies the vehicle running state as the vehicle speed threshold, The case where it does not exceed is specified as a low-speed driving state (including a stop state). Here, 5 km / h or the like is fixedly determined as the vehicle speed threshold. Then, in S23, it is determined whether or not the current vehicle is in the vehicle running state described above. If it is determined that the vehicle is in the vehicle running state, the process proceeds to S24. On the other hand, if it is determined in S23 that the vehicle is not in a traveling state (when a low-speed traveling state including a stop state is specified), the process proceeds to S26.

  In S24, it is specified whether or not the operator's hand (operating hand) is in the simple operation finger state. The finger state for simple operation is specified by a process similar to the process of S2 in FIG. In S25, if it is determined that the current finger state is the simple operation finger state, the process proceeds to S26. If it is determined that the current finger state is not the simple operation finger state, the process proceeds to S29.

  In S26, the display permission mode in which the display of the position indication image is permitted is set as the display mode. The set display permission mode is stored in a predetermined storage area (display mode storage unit) of the control circuit 18. In subsequent S27, the control circuit 18 specifies the positional relationship between the display screen of the display device 15 and the photographed hand. This process is the same as the process of S5 of FIG. In subsequent S28, the position instruction image 200H is displayed in an overlapping manner on the main image 200B displayed on the display screen of the display device 15 (position instruction image display means). This process is the same as S6 in FIG.

  Specifically, when it is determined in S23 that the current vehicle is not in the predetermined vehicle running state (when the low-speed running state including the stop state is specified), the processing of S27 and S28 is performed. Regardless of the finger state of the operator, the position indication image 200H based on the finger state is displayed (see (c) of FIGS. 4, 6, and 7), but in S22, the vehicle is in a traveling state. If it is determined in S25 that the current finger state of the operator is the simple operation finger state, the process of S27 and S28 results in the simple operation finger state being performed by the operator. A base position indication image 200H is displayed (see FIG. 6 and FIG. 7C).

  On the other hand, if it is determined in S25 that the current finger state is not the simple operation finger state, the process proceeds to S29. In S29, the display mode is set to a display prohibition mode in which the display of the position indication image is prohibited, and this is stored in a predetermined storage area (display mode storage unit) of the control circuit 18. Thereby, the display of the position instruction image 200H performed in S27 and S28 is not executed, and the position instruction image 200H is not displayed on the display screen of the display device 15, for example, as shown in FIG.

  Although the present process is terminated by the end of S28 and S29, the present process is repeatedly performed at a predetermined cycle after the termination.

  In the present embodiment, the predetermined vehicle speed level (vehicle speed threshold) is set to 5 km / h, and the display permission mode is always set in a slow running state of the vehicle that does not exceed the vehicle speed level. It has come to be. However, this vehicle speed level is set within a vehicle speed range that is higher than 0 km / h and lower than 5 km, or within a vehicle speed range that is greater than or equal to 5 km / h and less than 10 km / h, as shown in FIG. Alternatively, it may be set to 0 km / h so that the display permission mode is always set when the vehicle is stopped. In addition, this vehicle speed level may be set within a vehicle speed range of 10 km / h or more and less than 30 km (low-speed driving state of the vehicle), or within a vehicle speed range of 30 km / h or more and less than 60 km / h (in the vehicle). (High speed running state), or may be set within a vehicle speed range of 60 km / h or higher (high speed running state of the vehicle).

  In the process of FIG. 11, the display mode switching control is performed when the vehicle is in a predetermined vehicle running state (higher speed than the vehicle speed threshold), and when the vehicle is not in the predetermined vehicle running state (lower speed than the vehicle speed threshold). ), The display permission mode is fixedly set. However, when the vehicle is in a predetermined vehicle traveling state (higher speed than the vehicle speed threshold), the display prohibition mode is fixedly set and the predetermined vehicle traveling is performed. An embodiment in which the display mode switching control is performed when the vehicle is not in a state (lower speed than the vehicle speed threshold) may be employed.

  Hereinafter, a third embodiment of the present invention will be described.

  In the above embodiment, the display permission mode is set to permit the display of the position instruction image when the simple operation finger state is specified, and when the simple operation finger state is not specified, the position instruction image is displayed. The display prohibition mode is set to prohibit the display, but when the driver is specified as the operator of the touch operation surface 12a1, the display mode switching control is executed. However, when the vehicle is not in the traveling state, the display permission mode can be set as the display mode regardless of the finger state. The third embodiment is configured to perform the position instruction image display process shown in FIG. 12 in place of the position instruction image display process of FIG. 9 in the first embodiment. Hereinafter, the specific processing flow will be described with reference to FIG.

  In S31, the control circuit 18 determines whether an image of the operator's hand has been acquired. This is the same as the processing of S1 in FIG. When the image of the operator's hand is acquired, the process proceeds to S32, and when the image of the operator's hand is not acquired, this process is terminated.

  In S32, it is specified whether or not the operator who makes his / her hand face the operation panel 12a is a driver (operator specifying means). The operator can be specified based on, for example, a captured image (FIG. 8) of the camera 20 shown in FIG. Specifically, by identifying the direction in which the arm extends from the captured image of the camera 20 and determining whether or not the direction extends from the driver's seat 2D side, the operator Whether or not is a driver. In S33, it is determined whether or not the operator of the operation panel 12a is a driver. If it is determined that the operator is a driver, the process proceeds to S34. On the other hand, if it is determined in S33 that the operator of the operation panel 12a is a person other than the driver, the process proceeds to S36.

  In S34, it is specified whether or not the driver's hand (operator) as an operator is in the simple operation finger state. The finger state for simple operation is specified by a process similar to the process of S2 in FIG. In S35, it is determined whether or not the current finger state is the simple operation finger state. If it is determined that the current finger state is the simple operation finger state, the process proceeds to S36, and it is determined that the current state is not the simple operation finger state. If so, the process proceeds to S39.

  In S36, the display permission mode in which the display of the position indication image is permitted is set as the display mode. The set display permission mode is stored in a predetermined storage area (display mode storage unit) of the control circuit 18. In subsequent S37, the control circuit 18 specifies the positional relationship between the display screen of the display device 15 and the photographed hand. This process is the same as the process of S5 of FIG. In subsequent S38, the position instruction image 200H is displayed in an overlapping manner on the main image 200B displayed on the display screen of the display device 15 (position instruction image display means). This process is the same as S6 in FIG.

  Specifically, when a person who is different from the driver is specified as an operator of the operation panel 12a in S33, the finger of the operator is determined regardless of the finger state of the operator by the processes of S37 and S38. While the position indication image 200H based on the state is displayed (see (c) of FIGS. 4, 6, and 7), the driver is specified as the operator of the operation panel 12a in S32, and in S35. When it is determined that the finger state is the simple operation finger state, the position instruction image 200H based on the simple operation finger state performed by the driver is displayed by the processing of S37 and S38 (FIG. 6). And (c) of FIG. 7).

  On the other hand, if it is determined in S35 that the current finger state is the simple operation finger state, the process proceeds to S39. In S39, the display mode is set to a display prohibition mode in which the display of the position indication image is prohibited, and this is stored in a predetermined storage area (display mode storage unit) of the control circuit 18. Thereby, the display of the position instruction image 200H performed in S37 and S38 is not executed, and the position instruction image 200H is not displayed on the display screen of the display device 15 as shown in FIG. 5C, for example.

  This process is ended by the end of S38 and S39. This process is repeatedly performed at a predetermined cycle even after the process is completed.

  Hereinafter, a fourth embodiment of the present invention will be described.

  In the fourth embodiment, the display processing of the position indication image shown in FIG. 9 is performed as a combination of both FIG. 11 and FIG. That is, the traveling state of the vehicle (for example, the vehicle speed of the vehicle) is detected, and whether or not the operator of the remote touch operation surface 12a is a driver is determined. In a low-speed driving state that is lower than a predetermined vehicle speed level) and the operator of the specified remote touch operation surface 12a is a driver, the simple operation finger state is specified. The display of the position instruction image 200H is permitted only when it is present, but the display of the position instruction image 200H is prohibited when the simple operation finger state is not specified. When the vehicle is not in the predetermined traveling state, or when the operator of the specified remote touch operation surface 12a is not a driver, or when both of them occur at the same time, the specified finger state is set. Regardless, it is possible to perform processing that permits display of the position indicating image 200H. Specifically, the touch operation input process of FIG. 9 described above can be realized as a process as shown in FIG.

  That is, when the operator's hand image on the remote touch operation surface 12a is acquired in S41 (the same processing as S1 in FIG. 9), it is determined in S42 and S43 whether or not the vehicle is in a vehicle speed traveling state. (Similar to S22 and S23 in FIG. 11) If the vehicle is traveling at a high speed, the process proceeds to S44 and S45 to determine whether or not the operator of the remote touch operation surface 12a is a driver (FIG. 12). Processing similar to S32 and S33).

  If it is determined in S42 to S45 that the vehicle is in a vehicle speed traveling state and the operator of the remote touch operation surface 12a is a driver, the process proceeds to S46 and S47, and the operator's hand (operator) is Whether or not the finger state is the simple operation finger state is specified (the same processing as S2 and S3 in FIG. 9), and if it is determined that the current finger state is the simple operation finger state, the process proceeds to S48 and the simple operation is performed. If it is determined that the finger state is not established, the process proceeds to S51.

  In S48, the display permission mode is set as the display mode, and this is stored in a predetermined storage area (display mode storage unit). In subsequent S49, the control circuit 18 specifies the positional relationship between the display screen of the display device 15 and the photographed hand, and in S50, on the main image 200B displayed on the display screen of the display device 15, The position instruction image 200H is displayed in an overlapping manner (position instruction image display means). These processes are the same as the processes of S4 to S6 in FIG.

  More specifically, if it is determined in S42 that the vehicle is not in a vehicle running state, if a person different from the driver is specified as an operator of the operation panel 12a in S44, S48 to S50 are performed. Through the processing, the position instruction image 200H based on the finger state is displayed regardless of the finger state of the operator (see FIGS. 4, 6, and 7C). On the other hand, when it is determined that the vehicle is in a vehicle running state in S42, and the driver is specified as the operator of the operation panel 12a in S44, and then the simple operation finger state is specified in S46. In S48 to S50, the position instruction image 200H based on the finger state for simple operation performed by the driver is displayed (see FIGS. 6 and 7 (c)).

  On the other hand, if it is determined in S47 that the current finger state is not the simple operation finger state, the process proceeds to S51. In S51, the display prohibit mode is set as the display mode, and this is stored in a predetermined storage area (display mode storage unit). Thereby, the display of the position instruction image 200H performed in S49 and S50 is not executed, and the position instruction image 200H is not displayed on the display screen of the display device 15, for example, as shown in FIG.

  This process is terminated by the end of S50 and S51. This process is repeatedly performed at a predetermined cycle even after the process is completed.

  In the processing of FIG. 13, the display mode switching control is performed when the vehicle is in a predetermined vehicle running state (higher speed than the vehicle speed threshold) and the operator is a driver. In the embodiment, the display permission mode is fixedly set when the vehicle is not in a predetermined vehicle running state (lower speed than the vehicle speed threshold) or when the operator is not a driver. The display prohibit mode is fixedly set when the operator is a driver and the display mode switching control is performed when the vehicle is not in a predetermined vehicle running state and the operator is a driver. In other cases (when the operator is not a driver), the display permission mode may be fixedly set.

  Hereinafter, a fifth embodiment of the present invention will be described.

  In the above embodiment, when the display permission mode is set in the touch operation input process shown in FIG. 10, the control content corresponding to the touch operation input to the remote touch operation surface 12a is executed, that is, the corresponding control. When the execution of the content is permitted and the display prohibit mode is set, the control content corresponding to the touch operation input to the remote touch operation surface 12a is not executed, that is, the execution of the corresponding control content is prohibited. It was. However, when the simple operation finger state is specified rather than when the display permission mode is set, execution of the control content corresponding to the touch operation input to the remote touch operation surface 12a is permitted, and the display prohibit mode When the simple operation finger state is not specified, the execution of the control content corresponding to the touch operation input to the remote touch operation surface 12a can be prohibited.

  Specifically, when the finger state for simple operation is specified, acceptance of the touch operation input to the remote touch operation surface 12a is permitted, and when the finger state for simple operation is not specified, Operation input reception control for prohibiting reception of touch operation input to the remote touch operation surface 12a can be executed. Hereinafter, an example of the specific processing flow will be described with reference to FIG.

  In S111, the control circuit 18 determines whether or not there is a touch operation on the operation panel 12a. The process of S111 is the same as S101 of FIG. In S112, the control circuit 18 determines whether or not the current finger state is the simple operation finger state. In the present embodiment, the process of S2 of FIG. 9, S24 of FIG. 11, S34 of FIG. 12, or S46 of FIG. As the state is specified, a process of storing the specified current finger state in a predetermined storage area (finger state storage unit) provided in the storage unit of the control circuit 18 is performed.

  In S112, it is determined whether or not the current finger state is the simple operation finger state based on the finger state information stored in the storage area (finger state storage unit). When it is determined that the finger state is for the simple operation, the process proceeds to S113, and when it is determined that the finger state is not for the simple operation, the process is terminated.

  In S113, the touch operation position on the operation panel 12a, that is, the touch operation position on the touch operation surface 12a1 is specified (the same processing as S103 in FIG. 10). In subsequent S114, control corresponding to the touch operation position on the operation panel 12a is performed (the same processing as S114 in FIG. 9). Although the present process ends upon completion of S114, the present process is repeatedly performed at a predetermined cycle even after the completion.

  Note that the position instruction image display process in the embodiment for executing the touch operation input process shown in FIG. 14 may be any of the position instruction image display process described above and the position instruction image display process described later. .

  Hereinafter, a sixth embodiment of the present invention will be described.

  In the above embodiment, the camera 12b serving as the hand image photographing means is a camera that photographs a fixed area in the vehicle interior facing the touch operation surface (front surface) 12a1 of the touch panel 12a. It has the structure image | photographed from the back surface 12a2 side of 12a. However, the present invention is not limited to such a configuration. For example, as shown in FIG. 15, the touch operation surface 12 a 1 of the touch panel 12 a is attached to the center console portion C of the vehicle so as to photograph from an obliquely upper side, and an approaching object approaching the touch operation surface 12 a 1 is displayed on the touch operation surface 12 a 1. You may comprise so that it may image | photograph from the side. In this case, unlike the above-described embodiment, when a specific image (for example, a hand image) in the photographed image is extracted and displayed on the display unit 15, the process of reversing the image horizontally can be omitted. Further, the camera 12b serving as the hand image photographing means may be used as the camera 20 for photographing the peripheral area including the touch operation surface 12a1 of the operation panel 12a.

  The seventh embodiment of the present invention will be described below.

  In the above embodiment, as the display permission period, as shown in FIG. 18, a period until the simple operation finger state becomes unspecified is determined. A predetermined time can be determined as shown in FIG. That is, a period from when switching to the display permission mode until a predetermined time (prohibition mode return time) elapses is defined as a display permission period, and the display prohibition mode is restored as the prohibition mode return time elapses. Can be configured. However, the touch operation input process is the process of FIG. According to this configuration, a period during which the position indication image can be displayed (display permission mode) is triggered when the user indicates a specific finger state, and the touch operation surface 12a1 while viewing the position indication image is displayed. While the position instruction input (touch operation) is possible, the position instruction image is not displayed unless the user indicates a specific finger state, and the touch operation for performing the position instruction input cannot be performed smoothly. In other words, the display of the position indication image is permitted only when the user displays a clear intention to input the position instruction, and the intention display is confirmed by specifying the user's finger state. ing. Here, when the display permission mode is set, the image display as shown in FIG. 4C is also possible. Specifically, the position indication image display process (FIG. 8) of the above embodiment can be realized by the process as shown in FIG.

  In S10, the control circuit 18 determines whether or not an image of the operator's hand has been acquired based on the captured image captured by the camera 12b. This process is the same as the process of S1 in FIG.

  In S11, the control circuit 18 determines whether or not the current display mode is the display permission mode. Since the current display mode is stored in a predetermined storage area (display mode storage unit) provided in the storage unit of the control circuit 18, the current display mode is determined based on the stored display mode information. If it is determined that the current display mode is the display prohibition mode, the process proceeds to S12. If it is determined that the current display mode is not the display permission mode, that is, if it is determined to be the display prohibition mode, the process proceeds to S19.

  In S12, based on the acquired hand image, it is specified whether or not the hand state is a predetermined simple operation finger state (simple operation finger state specifying means). This process is the same as the process of S2 in FIG. In subsequent S13, when it is determined that the current finger state is the simple operation finger state, the process proceeds to S14.

  In S14, the display mode is set to a display permission mode in which the display of the position indication image is permitted, and the process proceeds to S15. The set display permission mode is stored in a predetermined storage area (display mode storage unit) of the control circuit 18.

  In continuing S15, the timer 18b provided in the control circuit 18 is started, time count is started, and it progresses to S16. The time is stored in a form counted up by the counter 18c. Before the timer is started, the count value of the counter 18c is reset.

  In S16, the control circuit 18 specifies the positional relationship between the display screen of the display device 15 and the photographed hand. The process of S16 is the same process as S5 of FIG. In subsequent S17, as shown in FIG. 6 and FIG. 7C, the position indication image 200H is superimposed on the main image 200B displayed on the display screen of the display device 15 (position indication image display means). . The process of S17 is the same process as S6 of FIG. However, in the second embodiment mobile phone, the position instruction image 200H as shown in FIG. 4C is also displayed in S16 and S17. That is, in the second embodiment, as long as the display permission mode is set, overlapping display (or combined display) of hand images (finger images) is permitted, and at least indicates the finger pointing position based on the hand images. The position indicating image 200H may be displayed in any form as long as it is an image. Therefore, when the display permission mode is set, there can be a multiple-finger-shaped overlapping display as shown in FIG.

  On the other hand, if it is determined in S13 that the current finger state is not the simple operation finger state, the process proceeds to S18. In S18, the display prohibition mode is set and stored in a predetermined storage area (display mode storage unit) provided in the storage unit of the control circuit 18.

  This process is ended by the end of S17 and S18, but this process is repeatedly performed at a predetermined cycle after the end, as long as the hand image is acquired and the display permission mode continues, the position indication image of S16 and S17 is displayed. The display permission state continues. That is, during this period, the position indication image is displayed even if the finger state is not the simple operation finger state. However, in the display permission mode, when it is determined in S19 that the predetermined time has elapsed by the timer 18b started in S15, the display prohibition mode is forcibly set in S20, and this processing ends. Become. The process of S20 is the same as the process of S18.

  In the second embodiment, the user indicates the intention of operation by a hand shape (simple operation finger state), and when this is indicated, the display permission mode is set for a certain period of time. ing. In this case, the simple operation finger state is not limited to the one-finger state (see, for example, FIGS. 6 and 7), but the two-finger state with two fingers extended (see FIG. 70), three A three-finger state with one finger extended (see FIG. 71), a four-finger state with four fingers extended (see FIG. 72), and the like can be variously determined. In addition, even in the single finger state, only the case where the finger is an index finger (see FIGS. 6 and 7) may be set as the simple operation finger state, or only the case where the finger is the middle finger (see FIG. 68). It is good also as a finger state for operation. Even in the two-finger state, the simple operation finger state may be used only when the fingers are the index finger and the middle finger (see FIG. 70). One or more of these may be defined as the simple operation finger state. From the viewpoint of processing, it is desirable to determine a hand state (finger state) that is easier to identify.

  Hereinafter, an eighth embodiment of the present invention will be described.

  Note that the identification of the simple operation finger state in S2 of FIG. 9 may not be performed based on the captured image as described above. For example, among the inputs by the touch operation performed on the touch operation surface 12a1 (remote operation surface) by the control circuit 18 during a predetermined period, the number of inputs by the touch operation performed at different positions on the touch operation surface 12a1. (Number of operations specifying means), and the case where the number of inputs specified in the period is a predetermined number for simple operation may be specified as the finger state for simple operation. In this case, for example, when three touch operations are performed at different positions within one second (or simultaneously), it is specified that there are three fingers, and the number of simple operations is determined to be three. In this case, it can be specified as a finger state for simple operation. Specifically, as shown in FIG. 20, when a touch operation is performed on the touch operation surface 12a1 in a state where the display prohibition mode (and the input prohibition mode) is set, the control circuit 18 The timer 18b is started and the time counting is started by counting up the counter 18c, and the number of inputs by the touch operation including the touch operation at the start of the time counting is counted within the predetermined period t0. However, when a touch operation is performed on the touch operation surface 12a1, touch position information (for example, position coordinates in the touch operation surface 12a1) is acquired and a retouch operation to the same position in the touch operation surface 12a1 is performed. Do not count that input. If the number of inputs when the predetermined period t0 has elapsed is the same as the number of fingers that define the simple operation finger state, the finger state of the current operator's hand is the simple operation finger state. A display permission mode (and an input permission mode) is set only for a predetermined display permission period from the time of the identification. When the display permission period ends, the display prohibition mode (and the input prohibition mode) is set again.

  The ninth embodiment of the present invention will be described below.

  In the above-described embodiment, the display mode switching control as described above is performed on all operation screens displayed on the display device 15. However, the display mode switching control is performed only on a predetermined operation screen. In this operation screen, the display permission mode or the display prohibition mode may be fixedly set.

  Based on the control command from the control circuit 18, the display device 15 can switch a plurality of operation screens 200 on which input by a pressing operation on the touch operation surface 12a1 is accepted on the display screen (operation screen display means). . For example, a menu selection screen 2003 as shown in FIG. 22, a car air conditioner air volume setting screen 2004 as shown in FIG. 23, a car audio volume setting screen 2005 as shown in FIG. The map display screen 2002 (FIG. 7C), the character input screen 2001 (FIG. 6C), and the like can be switched and displayed according to various inputs. Among these operation screens, display mode execution enable / disable information that makes it possible to specify the operation screen that is the execution target of the display mode switching control is stored in a predetermined storage area (display mode execution enable / disable information storage means) of the control circuit 18. The control circuit 18 refers to the display mode execution availability information when performing the position indication image display processing, and based on this, the operation screen displayed by the display device 15 is as described above. The display mode switching control is configured to be executed on the specified operation screen only when the display mode switching control is specified as an execution target. Hereinafter, the specific processing flow will be described with reference to FIG.

  In S61, the control circuit 18 determines whether an image of the operator's hand has been acquired. This is the same as the processing of S1 in FIG. When the image of the operator's hand is acquired, the process proceeds to S62, and when the image of the operator's hand is not acquired, this process is terminated.

  In S62, the control circuit 18 specifies the type of the operation screen 200 that is currently displayed, and specifies whether or not the display mode switching control is a predetermined execution target. Here, the display mode execution availability information stored in the predetermined storage area of the control circuit 18 includes a complicated operation in which the pressing operation on the touch operation surface 12a1 exceeds the predetermined operation burden level in the operation screen. The operation screen is determined as an execution target of the display mode switching control, and a simple operation screen including only an operation in which the pressing operation on the touch operation surface 12a1 does not exceed the operation load level is determined as a fixed setting target of the display permission mode. . The control circuit 18 refers to the display mode execution enable / disable information to specify whether or not the currently displayed operation screen 200 is an execution target of the display mode switching control.

  Note that the display mode execution availability information can be stored as information obtained by classifying screen types as follows. That is, as a complex operation screen, one or more operation input images displayed on the display device 15 (or all pressing operation areas defined on the touch operation surface 12a1 corresponding to the operation input image (operation icon) 200I). An operation screen in which the ratio SI / SA occupying the entire screen (or the entire area of the touch operation surface 12a1) exceeds a predetermined ratio S0 can be determined, and an operation screen not exceeding can be determined as a simple operation screen. (Example 1 in FIG. 73). Further, the number KI of one or more operation input images (or pressing operation areas determined on the touch operation surface 12a1 corresponding to the operation input images) 200I displayed on the display device 15 as the complex operation screen is determined in advance. An operation screen exceeding the given number K0 can be determined, and an operation screen not exceeding can be determined as a simple operation screen (example 2 in FIG. 73). Further, as a complex operation screen, an interval between one or more operation input images 200I displayed on the display device 15 (or between adjacent pressing operation areas defined on the touch operation surface 12a1 corresponding to the operation input image). An operation screen whose interval (LI) is narrower than a predetermined interval L0 can be determined, and an operation screen that does not exceed the predetermined interval L0 can be determined as a simple operation screen (example 3 in FIG. 73). Here, screens 2002, 2003, 3004, and 2005 as shown in FIGS. 22 to 24 and FIG. 7C are simplified operation screens, and a character input screen 2001 as shown in FIG. Name input screen) is defined as a complex operation screen.

  In S64, the control circuit 18 refers to the display mode execution availability information that can specify the complex operation screen and the simple operation screen, and the currently displayed operation screen 200 is the display mode switching control execution target (here, If it is determined whether or not it is a complicated operation screen, the process proceeds to S63. If it is determined in S63 that the currently displayed operation screen 200 is an execution target of display mode switching control, the process proceeds to S69. If it is determined that the operation screen 200 is not an execution target of display mode switching control, the process proceeds to S66. .

  In S64, it is specified whether or not the operator's hand (operating hand) is in the simple operation finger state (the same processing as S2 in FIG. 9). In S65, if it is determined that the current finger state is the simple operation finger state, the process proceeds to S66. If it is determined that the current finger state is not the simple operation finger state, the process proceeds to S69.

  In S66, the display permission mode in which the display of the position indication image is permitted is set as the display mode. The set display permission mode is stored in a predetermined storage area (display mode storage unit) of the control circuit 18. In subsequent S67, the control circuit 18 specifies the positional relationship between the display screen 200 of the display device 15 and the photographed hand (similar to S5 in FIG. 9). In subsequent S68, the position instruction image 200H is displayed on the main image 200B displayed on the display screen 200 of the display device 15 (position instruction image display means: same as S6 in FIG. 9).

  More specifically, if it is determined in S62 that the currently displayed operation screen 200 is not an execution target of the display mode switching control (here, it is specified that it is a simple operation screen), S67 and While the position instruction image 200H based on the finger state is displayed by the process of S68 regardless of the finger state of the operator (see (c) of FIGS. 4, 6, and 7), in S62, When it is determined that the currently displayed operation screen 200 is an execution target of the display mode switching control (in this case, when the operation screen 200 is identified as a complex operation screen), the operation is performed by the processing of S67 and S68. A position indication image 200H based on the simple operation finger state is displayed (see FIG. 6 and FIG. 7C).

  On the other hand, if it is determined in S65 that the current finger state is not the simple operation finger state, the process proceeds to S69. In S69, the display mode is set to a display prohibition mode in which the display of the position indication image 200H is prohibited, and this is stored in a predetermined storage area (display mode storage unit) of the control circuit 18. Thereby, the display of the position instruction image 200H performed in S67 and S68 is not executed, and the position instruction image 200H is not displayed on the display screen 200 of the display device 15 as shown in FIG. 5C, for example. .

  Although the present process is terminated by the end of S68 and S69, the present process is repeatedly performed at a predetermined cycle after the termination.

  In the processing of FIG. 21, the display mode switching control is performed on the complex operation screen, and the display permission mode is fixedly set on the simple operation screen. May be fixedly set, and the display mode switching control may be performed on the simple operation screen.

  The tenth embodiment of the present invention will be described below.

(First example of tenth embodiment)
In the operation input device as in the above embodiment, when the hand operating the remote operation unit (touch panel 12a) is actually displayed on the operation screen in the display device, when the hand is close to the remote operation unit, As shown in FIG. 4C, only the finger is displayed. However, if many fingers are displayed side by side at the same time, it is confusing which finger to operate, and the correspondence between the displayed finger and the finger that the operator actually wants to use for position indication The relationship may not be understood, and it may lead to erroneous operations. In particular, when the operation input device is for a vehicle, the driver who is driving the vehicle cannot stare at the operation screen, so that it is difficult to input an operation with a large number of fingers while driving, and an erroneous operation is likely to occur.

  In order to solve such a problem, the finger that specifies the finger in the opposite hand on the front side of the remote operation unit (touch panel 12a) is subjected to the position indication image display process so that the actual finger used for the position indication can be easily specified. The specified step, the operation target finger specifying step for specifying the operation target finger used for the operation from the specified finger to the remote operation unit, or the estimated operation target finger, and the background image 200B on the display screen are combined or superimposed. The position indicating image 200P corresponding to the specified operation target finger is displayed among the position indicating images 200H displayed in the display step of displaying the position indicating image 200H indicating the specified position of the specified finger. A position indication image emphasis display step performed by highlighting that emphasizes relatively more than the position indication image 200R corresponding to a finger different from the operation target finger; It can be carried way to. As a result, as shown in FIGS. 27 and 28, the designated operation target finger (here, one index finger) is displayed with emphasis on the remaining fingers, and the operability is improved. The control circuit 18 functions as a finger specifying unit, an operation target finger specifying unit, and a position instruction image emphasizing display unit by performing these steps.

  Specifically, in each of the above-described embodiments, the process corresponding to S6 of FIG. 8 is performed as a process as shown in FIG. At T1, the control circuit 18 specifies the finger of the hand facing the touch operation surface 12a1 on the front side of the touch operation surface (remote operation surface) 12a1 (here, the opposite side of the camera 12b) (finger specifying step). ). Here, as in S1 of FIG. 8, it is determined whether an image of the operator's finger has been acquired based on the captured image 150H captured by the camera 12b. The determination result is output to the control circuit 18, and if it is determined that the image of the finger has been recognized, the process proceeds to T2, and if not determined, this process ends.

  At T2, the control circuit 18 detects the fingertip from the acquired finger image 150F. The detection of the fingertip is performed by specifying the fingertip Q from the specified finger image 150F (fingertip specifying step). The fingertip Q in the present embodiment is specified in the form as shown in FIG. That is, when the finger image 150F is specified as shown in FIG. 29A, the image processing unit 18a displays the hand image 150H including the specified finger image 150F and the finger image 150H of the finger image 150F. The moving hand images 160H shifted by a predetermined amount in the length direction are overlapped with each other, the non-overlapping region 170 where the finger images 150H and 160H do not overlap each other is specified, and the barycentric position of the non-overlapping region 170 is calculated. Then, the center of gravity position is specified as a fingertip (fingertip position) Q. Note that another method may be used for specifying the fingertip Q. For example, the contour of the fingertip that appears curved may be extracted and smoothed, and then determined as the center of curvature. In addition, the fingertip Q may be specified by specifying the area up to the first joint of the finger as the fingertip (fingertip area) Q instead of specifying it as one point of the finger as described above.

  T1 and T2 are steps for performing finger specifying processing and fingertip specifying processing, respectively, but they may be executed simultaneously. That is, instead of specifying the fingertip from the finger image, the fingertip may be specified directly from the captured image captured by the camera 12b, and the finger may be specified by specifying the fingertip. This simplifies the process.

  In subsequent T3, the control circuit 18 designates an operation target finger (position instruction finger) P to be used or estimated to be used for an operation on the remote operation unit (operation target finger designation step). Specifically, the positional relationship reflecting the positional relationship between the specified fingertips Q in the touch operation surface direction (the direction in which the touch operation surface 12a1 spreads, that is, the direction perpendicular to the normal line of the touch operation surface 12a1). Information is acquired as designated finger information (designated finger information obtaining step), and on the basis of the obtained positional relationship information, a predetermined selection condition relating to the positional relationship among the fingers identified by T1 is obtained. The finger having the positional relationship information most suitable for is designated as the operation target finger P with priority.

  A plurality of fingers can be specified as the operation target finger P, but from the viewpoint of giving a position instruction so that the operation on the contact-type operation unit 12a is easy to understand, one or two relatively It is desirable to specify a small number of fingers. In the present embodiment, there is one. Furthermore, as the types of fingers to be designated, the index finger (see FIG. 27 and FIG. 28C) suitable for position indication, the middle finger (see FIG. 47), the index finger and the middle finger (see FIG. 27). 48), and the index finger is assumed in this embodiment.

  The process of T3 of this embodiment is executed as the process shown in the flowchart of FIG. That is, at T11, the fingertip Q specified as shown in FIG. 29C is specified on the surface 180 of the virtual two-dimensional coordinate system, and one axis X determined on the surface 180 is used. Then, another one-dimensional coordinate system is defined in the two-dimensional coordinate system. Here, the specified fingertip Q is projected toward the touch operation surface 12a1 (in the direction facing the remote operation unit), and the projection surface is set as the above-described surface 180 at a predetermined position on the projection surface. One axis X is set to define a one-dimensional coordinate system. At T12, the position coordinate (coordinate value) XQ of the fingertip Q specified on the surface 180 is specified in the one-dimensional coordinate system, and the axis coordinate information reflecting the position coordinate XQ is acquired as the positional relationship information. Then, based on the acquired axis coordinate information at T13, priority is given to a finger suitable for a predetermined selection condition regarding the axis coordinate information among the fingers specified at T1 in FIG. And designated as the operation target finger P.

  Further, in the present embodiment, an operation with the right hand of the passenger on the passenger seat is assumed, and the axis X is set so that the index finger of the operator is designated as the operation target finger P. Therefore, at T11, the axis X is set at the position shown in FIG. At T12, the fingertip Q is projected toward the set axis X, and the coordinate values X1 to X3 of the projected positions are calculated. At T13, the coordinate value of the acquired coordinate values X1 to X3 is the largest. A large fingertip Q is specified, the finger having this fingertip Q is estimated to be an index finger, and this finger is designated as the operation target finger P.

  In addition, as shown in FIG. 30, the axis X in the present embodiment is determined so as to go from the lower right of the captured image in which the right index finger is difficult to be positioned to the upper left where the right index finger is easily positioned. 31 and FIG. 32 may be set. In the case of FIG. 31, it is suitable to estimate the finger with the largest coordinate value as the middle finger, and if it can be estimated or specified whether the operating hand is left or right, the finger adjacent to the middle finger is estimated as the index finger. You can also. In the case of FIG. 32, if it can be estimated or specified whether the operating hand is left or right, it is suitable to estimate the leftmost finger as the index finger, and the finger adjacent to it can be estimated as the middle finger.

  In the present embodiment, the photographed image photographed by the camera 12b can be referred to as a projection surface 180 obtained by projecting the fingertip Q toward the touch operation surface 12a1. Therefore, the operation panel image area (the area where the touch operation surface 12a is reflected: here the entire area of the captured image 150) where the touch operation surface 12a is located in the captured image captured by the camera 12b and the projection surface 180 are the same. The coordinate position of the fingertip Q on the operation panel image area can be used as the projected coordinate position of the fingertip Q on the projection plane 180 as it is. At T3, the operation target finger P is designated by calculation on the coordinate system on the virtually determined projection plane 180, and image processing is not required. That is, since the image data is not used for the operation target finger P specifying process, the calculation cost (calculation amount / memory amount) can be reduced.

  Returning to FIG. At T4, as shown in FIG. 33 (c), the control circuit 18 synthesizes or superimposes on the background image 200B on the display screen of the display device 15 on the display screen corresponding to the specified fingertip Q. A position indication image 200H is displayed at the position. However, this display corresponds to the remaining finger R different from the operation target finger P in the display region 200P of the position instruction image 200H corresponding to the designated operation target finger P in the displayed position instruction image 200H. The position indication image 200H to be displayed is displayed in a highlighted manner in which it is emphasized relatively than the display area 200R. Thereby, the visibility of the operation target finger P is improved, the finger to be operated is easily transmitted to the user, and the operability is also improved.

  In the present embodiment, the control circuit 18 first specifies the positional relationship between the display screen of the display device 15 and the photographed finger. That is, the position on the display screen where the photographed finger faces is specified. In the present embodiment, an operation panel image area where the touch operation surface 12a is located (an area where the touch operation surface 12a is reflected: here, the entire area of the captured image 150) is specified on the captured image 150 captured by the camera 12b. At the same time, a two-dimensional operation coordinate system is set on the operation panel image area, and the position of the photographed finger is specified on the operation coordinate system. Then, the position on the display screen of the display device 15 corresponding to the position of the finger specified on the operation coordinate system is specified in the two-dimensional display coordinate system defined on the display screen.

  Then, as shown in FIG. 33 (a), the position indication image 200H is superimposed on the corresponding position coordinates of the main image 200B at the position specified on the display screen of the display device 15. Then, the above-described emphasis display of the position instruction image 200H (200P, 200R) is performed in such a manner that the emphasis level (enhancement degree) is gradually changed over a predetermined time t1, as shown in FIG. That is, in the process of highlight change, it has a state in the middle of change as shown in FIG. 33 (b), and the change in emphasis as shown in FIG. 33 (c) is completed through such a state in the middle of highlight change. To do. If fingers that are not specified by highlighting are made difficult to see, it may be difficult to recognize which fingers are highlighted if the fingers are made difficult to see suddenly. Therefore, at first, if all the fingers are displayed in the same manner as shown in FIG. 33A and the emphasis level is changed step by step, the finger displayed on the screen and his / her own are displayed. Matching with fingers is easy.

  Further, the highlighting is performed on the entire finger display areas 200P to 200R of the position indication image 200H, the outline, and only the inside of the outline, with a change in color (see FIG. 63) and a change in shading (see FIG. Fingers may be erased), shape change (FIG. 64: the operation target finger P is relatively thick here as an example), switching to another image, change in transparency (see FIG. 27), It blinks (see FIG. 65) or a combination thereof. In the present embodiment, the processing is executed in such a manner that the transparency inside the contour of the processed image forming the position indicating image 200H is changed. Further, the highlighting here holds the display area 200P of the position indicating image 200H corresponding to the operation target finger P in a constant state, and emphasizes the display area 200R of the position indicating image 200H corresponding to the remaining fingers R. It is done in a form that reduces the level. The position indicating image region 200R corresponding to the remaining finger R may be erased. However, in consideration of operability, the emphasis level is lower than that at the time of erasure, but it may be relatively easily visible.

  Further, the highlighting at T4 may be started immediately as the operation target finger P is designated at T3 as shown in FIG. 61, or the operation at T3 as shown in FIG. It may be started after a predetermined time has elapsed since the target finger P is designated. In particular, in the case of a configuration in which highlighting is not performed in stages, it is desirable to select the latter.

  This process ends when T4 ends. This process is repeatedly performed at a predetermined cycle after the end, and when the hand facing the operation panel 12a moves, the display position of the position instruction image 200H displayed on the display screen of the display device 15 also moves accordingly. To do. When the hand moves in the middle of the emphasis change, the display position of the position indicating image 200H (200P and 200R) moves while continuing the emphasis change, and when the hand moves in a state where the emphasis change is completed, The display position of the position indicating image 200H (200P and 200R) moves while maintaining the emphasized state.

  The display process of the position instruction image includes a finger specifying step, a fingertip specifying step, an operation target finger specifying step, a position instruction image emphasizing display step, and a specified finger information acquiring step. By executing this processing, the function functions as means for executing these steps (finger specifying means, fingertip specifying means, operation target finger specifying means, position indication image highlighting display means, and specified finger information acquiring means).

(Second example of tenth embodiment)
The method for designating the operation target finger P can also be as shown in FIGS. Here, as shown in the flowchart of FIG. 34, at T21, the fingertip Q specified at T2 of FIG. 25 is projected toward the touch operation surface 12a1, and is projected onto a predetermined region on the projection surface 180. A reference area Y is set (see FIG. 35 or FIG. 36), and at T22, distance information reflecting the distance between the set reference area Y and the projected fingertip Q is acquired as positional relationship information (specified) Finger information acquisition step) Based on the distance information acquired at T23, priority is given to a finger suitable for a predetermined selection condition regarding the distance among the fingers specified at T1 in FIG. The operation target finger P is designated (operation target finger designation step).

  Also in the present embodiment, the fingertip Q is specified on the virtually defined surface 180 as in the above embodiment, and the reference region Y is further defined (T21). Then, each distance is calculated (T22), and the operation target finger P is designated based on the calculated distance (T23). Here, the middle finger of the operator is assumed as the operation target finger P, and the finger with the longest standard distance is designated as the operation target finger P. Note that the reference region Y may be defined as a region having an area, or may be defined as a point or line having no area.

  Further, as shown in FIG. 35, for example, the reference region Y can be arranged at a predetermined position where the back of the hand is estimated to be located in a state where a finger is photographed by the camera 12b. The distance information in this case is information that does not directly reflect the length of the finger specified in T1, but is related to it, and a specific finger can be designated from the length of the distance. For example, the finger with the longest distance can be estimated as the middle finger, and if it can be estimated or specified whether the operating hand is left or right, the finger adjacent to the middle finger can be estimated as the index finger. When an index finger or the like is assumed as the operation target finger P, the second longest finger may be estimated as the index finger and specified.

  In addition, as shown in FIG. 36, for example, the reference region Y can be arranged at a position where the fingertip of the finger assumed as the operation target finger P is most easily located in a state where the finger is photographed by the camera 12b. In this case, the finger having the fingertip Q having the shortest distance from the reference area Y can be designated as the operation target finger P. In the case of FIG. 36, it is assumed that the operating hand is the right hand and is located at the upper left.

(Third example of tenth embodiment)
The operation target finger P can be designated as shown in FIGS. Here, as shown in the flowchart of FIG. 37, at T31, finger shape information relating to the shape of each finger specified at T1 of FIG. 25 is acquired as designated finger information (designated finger information acquisition step), and T32 Based on the obtained finger shape information, among the fingers specified at T1, the finger having the approach distance information most suitable for the predetermined selection condition related to the finger shape is preferentially operated. Designate as a target finger (operation target finger designation step).

  As the finger shape information, the finger length information related to the length relationship in the touch operation surface direction between the fingers specified at T1 in FIG. 25 can be determined. Specifically, the lengths d1 to d3 (see FIG. 38) from the fingertip Q specified at T2 in FIG. 25 to the position corresponding to the midpoint of the line segment of the finger edge appearing on the captured image 150. These can be calculated and used as the finger length information. In this case, since the finger length information is information reflecting the length of the finger specified in T1, a specific finger can be designated from the length. For example, the finger with the longest distance can be estimated as the middle finger, and if it can be estimated or specified whether the operating hand is left or right, the finger adjacent to the middle finger can be estimated as the index finger. When an index finger or the like is assumed as the operation target finger P, the second longest finger may be estimated as the index finger and specified. According to this configuration, it is possible to reliably eliminate the possibility that a halfway extended finger that is not intended for operation is designated for position indication. Note that the finger length information d1 to d3 may be information reflecting at least the length of each finger. For example, the finger length from the tip position of the curved fingertip to the finger edge appearing on the captured image 150 is sufficient. It may be a distance along the direction.

  Further, as finger shape information, finger width information related to the width relationship in the touch operation surface direction between the fingers specified in T1 of FIG. 25 can be determined. Specifically, lengths w1 to w3 (see FIG. 38) in the width direction that pass through the fingertip Q specified at T2 in FIG. 25 and are perpendicular to the finger length direction are set as the finger width information. Can do. In this case, since the finger width information appears larger as the finger approaches the touch operation surface 12a1, it can also be referred to as approach distance information that reflects the approach distance relationship to the touch operation surface 12a1. A large finger can be designated as the operation target finger P. Note that the finger width information w1 to w3 may be information reflecting at least the width of each finger, and may be a finger width calculated at a position different from the above.

(Fourth example of tenth embodiment)
The method of specifying the operation target finger P can be as shown in FIG. At T41, the touch operation position T on the touch operation surface 12a1 is detected by the finger specified at T1 in FIG. 25, and at T42, the detected finger that has performed the first touch operation is the operation target finger. Designate as P (operation target finger designation step). That is, the touch operation position T on the touch operation surface 12a1 is acquired as designated finger information, and the operation target finger P is designated based on the acquired finger information. According to this configuration, since the highlighting start trigger is determined as a touch operation by the user, it is possible to avoid missing a display change due to the highlighting. In this case, the designated operation target finger P is tracked when it is moved by a manual operation, and continues to be designated until the finger P disappears from the screen or a new touch operation is performed. Alternatively, conditions such as positional relationship information with other fingers R and finger shape information of the finger P so that the same finger is set as the operation target finger P when it appears on the screen again after disappearing from the screen May be stored in a storage unit in the control circuit 18 and the operation target finger P may be designated based on the stored information.

(Fifth example of tenth embodiment)
The operation target finger P can be designated as shown in FIGS. Here, as shown in the flowchart of FIG. 40, at T51, the touch operation position T on the touch operation surface 12a1 is detected by the finger specified at T1 of FIG. 25, and the touch operation position T is detected. Accordingly, at T52, as shown in FIG. 41, the finger closest to the detected touch operation position T is specified as the operation target finger P (operation target finger specifying step). In this case, the highlighting at T4 in FIG. 25 may be started (immediately) as the touch operation on the touch operation surface 12a1 is performed, or the touch operation on the touch operation surface 12a1 may be started. It may be started after a lapse of a predetermined time from being made. In particular, in the case of a configuration in which highlighting is not performed in stages, it is desirable to select the latter.

  In the fourth example and the fifth example of the tenth embodiment, when the touch operation surface 12a1 is touched by mistake and the touch operation is performed, the highlighting can be prevented from being started. Specifically, as a touch operation at T41 or T51, a touch operation that has an operation burden larger than that of a normal touch operation (a touch operation in which an operation input is valid) can be determined as the highlight display start operation. For example, as a touch operation for starting highlighting (highlighting start operation), a touch operation in which a pressing operation duration time until an input becomes valid is longer than a normal touch operation, or a press stronger than a normal touch operation It is possible to determine a touch operation by an operation force. In this case, at T41 or T51, the process proceeds at T42 or T52 only when a predetermined highlight display start operation is performed. When there is no highlight display start operation, the highlight display is not started until the highlight display start operation is performed. It can be set to a standby state.

(Sixth example of tenth embodiment)
The method of designating the operation target finger P can be as shown in FIGS. Here, as shown in the flowchart of FIG. 42, the luminance information of the image of the photographed finger is acquired as designated finger information at T61 (designated finger information obtaining step), and the obtained luminance information is obtained at T62. Based on the photographed finger image, a finger suitable for a predetermined selection condition related to the luminance information is preferentially designated as the manipulation target finger P (operation target finger designation step). Since the camera 12b captures the reflected light of the hand operating the touch operation surface 12a1, the fingertip closest to the touch operation surface 12a1 appears with higher brightness. That is, it can be said that the luminance information is the approach distance information (designated finger information) already described. On the other hand, since the fingertip that operates the touch operation surface 12a1 has a higher possibility of a position instruction as the finger is closer to the touch operation surface 12a1, for example, the finger with the highest brightness is designated as the operation target finger P with priority. Can be configured.

  As shown in FIG. 3, the camera 12b is shooting in the form of capturing the reflected light of the light source 12c. Therefore, the reflected light intensity is reflected in the gradation of the color appearing in the captured image 150, and the higher the reflected light intensity, the higher the gradation level. That is, since the captured video 150 captured by the camera 12b of the present embodiment is a multi-gradation captured image (monochrome captured image in the present embodiment), the image processing unit 18a converts the captured image 150 into a predetermined gradation. The gradation level of each pixel is binarized using the threshold value, and a region exceeding the gradation threshold value is extracted as the approaching object image 150H as shown in FIG. Further, in order to specify the image of the human hand / finger from the extracted approaching object image 150H, the image processing unit 18a first stores the shape of the approaching object image 150H and the storage unit of the control circuit 18. Compared to the hand and finger shape patterns, the approaching object image 150H that matches those patterns is recognized as a finger image. In the present embodiment, the gradation threshold value for specifying an image of a hand or finger is set as a first gradation threshold value, and a second gradation threshold value higher than that is defined, and the second gradation threshold value is determined. The fingertip finger having the region L1 (see FIG. 43) exceeding the threshold value can be designated as the operation target finger P. As another method, the operation target finger P may be designated by comparing the brightness of the fingertip Q or the average (or maximum / minimum) brightness of a predetermined region with the fingertip Q as a reference. Note that regions L1 to L3 in FIG. 43 indicate regions with different luminances in the captured video 150, and the luminances decrease in the order of L1, L2, and L3.

  That is, here, T1 in FIG. 25 is a step of specifying a finger image based on the luminance of the image captured by the camera 12b and specifying the finger by the finger image, and T2 in FIG. This is a step of specifying the fingertip using the hand image taken by the above. Furthermore, T3 in FIG. 25 is a step of specifying the operation target finger P based on the luminance of the hand image captured by the camera 12b. Therefore, in the present embodiment, it is possible to specify a finger, a fingertip, and even an operation target finger P from a captured image of the camera 12b.

  Here, it is assumed that the operation target finger P is designated by using image information of the fingertip portion instead of the entire finger. As a result, it is possible to limit the data that is the target of the operation target finger P specifying process, and the calculation cost (calculation amount / memory amount) can be reduced. Specifically, as shown in FIG. 43, a region S having a predetermined shape (in this case, a square) including the entire fingertip is set around the fingertip Q specified in T2, and all of the regions specified in T1 are specified. With respect to the finger, the luminance information in the region S is specified and acquired at T61, and the finger of the region S having the highest luminance among the acquired luminance information is preferentially designated as the operation target finger P at T62. ing.

(Seventh example of tenth embodiment)
The operation target finger P can be specified as shown in FIGS. 44 to 46. Here, as shown in the flowchart of FIG. 44, at T61, the approach distance information reflecting the approach distance relationship of the fingertip specified at T1 of FIG. 25 to the touch operation surface 12a1 is acquired as designated finger information. (Designated finger information acquisition step) At T62, based on the acquired approach distance information, priority is given to a finger suitable for a predetermined selection condition regarding the approach distance among the fingers specified by the finger specifying step. To designate as the operation target finger (operation target finger designation step). However, in the case of this configuration, means for detecting the distance between the touch operation surface 12a1 and the fingertip of the operating hand is required. As in the fifth example of the tenth embodiment, the luminance can be acquired as the approach distance information. For example, as shown in FIG. 45, the camera (photographing means) 40 is a means for acquiring the approach distance information. It can be. The camera 40 is arranged so that the distance between the touch operation surface 12a1 and the fingertip of the operator is reflected. From the captured image (for example, FIG. 46), the control circuit 18 causes the fingertip of each finger of the operator to touch the touch operation surface 12a1. The distances d1 to d4 are calculated, and for example, the finger with the closest fingertip can be designated as the operation target finger P. FIG. 46 is an example of a video image taken by the camera 40 of FIG.

(Eighth example of tenth embodiment)
Note that the operation target finger may be specified by combining a plurality of types of designated finger information such as the positional relationship information and the finger shape information. As a result, it is possible to more accurately specify a predetermined type of finger such as an index finger or a middle finger.

(Ninth example of tenth embodiment)
The camera 12b serving as the hand image photographing means is not photographed from the back surface 12a2 side of the touch panel 12a, and may be configured to be attached as shown in FIG. 12, for example. Further, the camera 12b serving as the hand image photographing means may be used as the camera 20 in FIG.

  As in the case of the camera 20 in FIG. 12, when the shooting range is determined so as to reflect not only the hand of the operator of the operation panel 12a but also the arm following that hand (for example, the shot image 300 in FIG. 13), In the control circuit 18 that acquires an image of the operation panel surrounding image to be shot, the image processing unit 18a can specify the direction of the arm extending from the hand, and can specify whether the operator is the left or right passenger. Accordingly, if the operator of the touch operation surface 12a1 is a person located on the left side of the touch operation surface 12a1, the operator can be identified as the right hand, and is located on the right side of the touch operation surface 12a1. If it is a person, the operation hand can be specified as the left hand.

  Then, the control circuit 18 can switch the designation condition of the operation target finger P based on the identification result of the operator. Specifically, when the index finger is specified as the operation target finger P, when the operating hand is the left hand, it is specified as the finger appearing on the right most, or as the finger adjacent to the right of the longest middle finger On the other hand, when the operating hand is the right hand, it can be specified as the finger that appears to the left most, or the finger that is adjacent to the left of the middle finger that appears the longest. Therefore, when selecting the operation target finger P, if the selection condition is switched depending on whether the operating hand is the left hand or the right hand, the switching is performed based on the identification result of the operating hand. be able to. Further, when the operator's hand is identified as the driver's hand, the display may not be performed so that the operation cannot be performed. In this way, since only the passenger on the left passenger seat can be designated as the operator, the right hand can be constantly determined as the operator, and this can be reflected in advance in the selection conditions for the operation target finger P.

  In this case, the camera 20 functions as an operator specifying unit together with the control circuit 18 (image processing unit 18a). It should be noted that another method may be used for specifying whether or not the operator of the operation panel 12a is a driver. For example, an approaching object detection unit (for example, a reflective optical sensor such as an infrared reflection sensor) that detects an approaching object may be provided on each of the left and right sides of the touch operation surface 12a to identify the operator.

(Tenth example of tenth embodiment)
In all the tenth embodiments described above, the highlighting as described above may be performed only when two or more fingers are specified. Since the control circuit 18 functioning as a finger specifying unit can specify the number of fingers, the position indicating image display process according to the tenth embodiment is executed when the specified number of fingers is 2 or more. And in the case of one, it can comprise so that the simple display process which displays only the position indication image corresponding to the finger | toe may be performed.

(Eleventh example of tenth embodiment)
In all the tenth embodiments described above, since the remote operation unit is premised on a contact operation (push operation or touch operation), one finger is designated for position indication. Further, it is also described that in some cases, two fingers can be designated for position indication as shown in FIG. However, the remote operation unit may be an operation unit such as a dial operation unit, which is premised on the operation of pinching the operation knob. In this case, the number of fingers such as two or three is operated. It can also be designated as the target finger P. Note that the finger specification or fingertip specification in the pinch-type operation can be performed using the above-described approach distance information (information reflecting the approach distance relationship to the remote operation surface).

  For example, as shown in FIG. 49, a dial operation unit 12a ′ (the front end portion 121e around the dial knob is also part of the dial operation unit 12a ′) is provided and operated as a remote operation unit of the present invention. Therefore, the approaching hand is configured to photograph with the cameras 41 and 42 (configured to be connected to the control circuit 18). The cameras 41 and 42 have different shooting ranges, and are set so that both shooting directions are orthogonal to each other on a plane perpendicular to the rotational axis of the dial operation unit 12a '. An example of images taken by the cameras 41 and 42 (binarized images similar to those shown in FIGS. 27 and 28) 150a and 150b are shown in FIGS. 50 (a) and 50 (b). On the other hand, a camera 12b is also installed in the operation information input unit 12 in the form shown in FIG. 3 (may be outside as shown in FIG. 12). An example of an image captured by the camera 12b (an image after binarization similar to (b) in FIGS. 27 and 28) 150c is shown in FIG. 50 (c). The image area 150S of these images 150a, 150b, and 150c is an area where the operation unit 12a 'is reflected.

  Here, the processing flow of the highlighting as described above implemented in this configuration will be described.

  First, the images 150a and 150b in FIGS. 50A and 50B are acquired (processing corresponding to T1 in FIG. 25), and the fingertip Q is specified from the hand image 150H extracted from the images 150a and 150b. Then, the specific position of the fingertip Q is specified (processing corresponding to T2 in FIG. 25). Here, a three-dimensional coordinate system (vehicle interior coordinate system) having an x-axis, y-axis, and z-axis as shown in FIG. 49 is defined in the vehicle interior and specified as position coordinates in the coordinate system. Specifically, an xz coordinate system and a yz coordinate system are defined for the images 150a and 150b, and any one of those already described in the tip region of the protruding image 150F extending from these to the housing front end 121e side. The fingertip Q is specified by the method, the position coordinates of the fingertip Q are specified, and the three-dimensional position coordinates in the vehicle interior coordinate system are specified from these.

  Subsequently, the operation target finger P is specified. Here, the distance from each fingertip Q to the front end surface of the housing front end portion 121e is calculated, and the fingertip Q that is closest to the front end surface of the housing front end portion 121e, and Q that is next closest to the fingertip Q are calculated. Both are specified as the operation target finger P. Note that the calculation of the distance here may be performed using the image 150a, 150b because the front end surface of the housing front end 121e is reflected (reference numeral 150E), or may be calculated from the luminance of the image 150c. Also good.

  Then, the position indicating area 200Q (200P) of the position indicating image 200H corresponding to the specified fingertip Q of the operation target finger P is more than the position indicating area 200Q (200R) of the position indicating image 200H corresponding to the other fingertip Q. In an emphasized manner, the image is superimposed on the background image 200B as shown in FIG. Note that the background image 200B (200B3) in FIG. 51 is a dial switch image displayed as the switch image 200I, and is rotated according to the rotation operation of the dial operation unit 12a ′ serving as the remote operation unit, and the rotation position is displayed. The control content corresponding to is executed.

  The operation target finger P is specified here by pointing the specified fingertip Q toward the dial operation unit 12a ′ serving as a remote operation unit as in the second example of the tenth embodiment already described ( Distance information reflecting the distance between the reference area Y set in a predetermined area on the projection plane 180 and the projected fingertip Q is projected (in the direction facing the dial operation unit 12a ′) It is acquired as relationship information, and using the acquired distance information, a finger suitable for a predetermined selection condition regarding the distance is preferentially designated as the operation target finger P. The reference area Y set here can be set as the center position of the projection area 120 obtained by projecting the dial operation unit 12a 'onto the projection plane 180, as shown in FIG.

  In addition, the method of designating the operation target finger P here may be a method that is determined so that at least the thumb is included. Specifically, it can be identified from various factors such as a difference in thickness with other fingers, a difference in positional relationship with other fingers, and a difference in extension direction of the fingers. When there are two operation target fingers P, the remaining one finger can be the finger closest to the front end surface of the housing front end 121e other than the thumb. However, it is desirable that the index finger or the middle finger is designated for ease of knob operation. For example, the index finger or the middle finger can be designated based on the thumb (for example, based on the positional relationship from the thumb). In the case of a remote control unit that assumes the above-described knob operation, the fingertips of three fingers may be emphasized as shown in FIG. In this case, it is desirable that the thumb, index finger, and middle finger are designated as the operation target finger P.

  In this embodiment, the fingertip area 200Q in the position indicating image 200H is used as the position indicating area. However, as in the other embodiments described above, the entire finger included in the position indicating image 200H may be used as the position indicating area. . On the contrary, in other embodiments already described, the fingertip area 200Q in the position indicating image 200H may be used as the position indicating area.

(Finger or fingertip tracking process)
Processing that can be applied to all the above-described embodiments will be described.
In the tenth embodiment described above, the operation target finger P is specified for each frame of the captured video, but in this case, the positional relationship between the operation panel 12a (12a ′) and the operating hand has changed. In this case, there is a possibility that the operation target finger P is automatically switched to another finger. In particular, as in the fourth and fifth examples of the tenth embodiment, when the operation target finger P is specified based on the contact position (user operation) to the remote operation unit, the touch operation is a reference. Since the operation target finger P is identified, it is not desirable that another finger is easily set as the operation target finger P because the position of the hand or the finger is moved. Also, in a case other than the tenth embodiment, a state in which a predetermined simple finger type for simple operation is specified as the simple operation finger state (for example, either the index finger or the middle finger or In the finger state in which both are specified), after the finger of that type is specified, the position of the hand or finger is moved, so another finger is easily specified as the finger for the type for simple operation. It is not desirable to end up. For this reason, after a finger such as the operation target finger P is designated or specified, a process for tracking the movement of the designated or specified finger is added, and the same finger is continuously designated based on the tracking result. By continuing identification, the above problem can be solved.

  54 and 55 show a process for tracking all fingers specified by T1 in FIG. 25, a process for tracking all fingertips specified by T2, or S2 (and S2 in FIG. 8). Is a process for tracking all fingers (and fingertips) specified by Here, in order to operate the operation panel 12a (12a ′) forming the remote operation unit, the hand (finger) facing the front side of the operation panel 12a (12a ′) is photographed by the camera 12b as the photographing means. It is configured to specify based on a moving image. In this case, the finger identifiable range (fingertip identifiable range) of the hand (finger) facing the front side of the operation panel 12a (12a ') is a photographing range of the camera 12b that is fixedly arranged. The camera 12b captures a moving image within the range, and the control circuit 18 acquires moving image frames that are sequentially captured at predetermined time intervals.

  54 and 55, the finger tracking process (finger tracking step) of FIG. 54 and FIG. 55 calculates the position of each fingertip identified from the sequentially acquired moving image frames within the frame, and uses the calculated position information as tracking information. By storing and accumulating in a predetermined tracking information storage unit such as a ROM of the control circuit 18, each movement is tracked while identifying each fingertip identified from the sequentially acquired moving image frames.

The tracking information is stored in a form as indicated by reference numeral 500 in FIG. In the tracking information 500 shown in FIG. 56, a fingertip ID (identification information) is assigned to each fingertip specified in the acquired moving image frame, and the previous moving image of each fingertip is associated with the fingertip ID. Position information indicating the position in the frame (hereinafter referred to as the previous frame) and the previous moving image frame (hereinafter referred to as the previous frame) can be stored. The position information here is the position coordinates of each fingertip on a predetermined two-dimensional coordinate system defined in the moving image frame. The upper diagram of FIG. 56 illustrates a moving image frame (also referred to as a finger identifiable range) in which the two-dimensional coordinate system is defined. In the previous frame stored as tracking information 500 in FIG. The specified fingertip position coordinates P _i (-1), the fingertip position coordinates P _i (-2) specified in the previous frame, and the fingertip R _j (0) specified in the current frame are displayed. Has been. As shown in FIG. 56, the position coordinates of each fingertip are sequentially stored and updated as new moving image frames are acquired. The position coordinates stored and held here are the position coordinates of the previous frame and the previous frame, and the position coordinates of the frames before the previous frame are sequentially deleted.

  The tracking information 500 includes position information indicating the position of each fingertip on the frame specified from the latest moving image frame (hereinafter referred to as the current frame). The position information is stored for each fingertip ID. Furthermore, the tracking information includes a predicted position (next position) where each fingertip will be positioned in the current frame, and the tracking information storage unit stores predicted position information indicating the predicted position. Further, the correspondence relationship between the fingertip position information in the current frame and the predicted fingertip position information predicted for the current frame is specified, and the corresponding fingertip tracking at the time of acquiring the current frame is determined for each fingertip ID from the corresponding result. The situation is specified, and this is also stored as tracking information in the tracking information storage unit. The tracking status includes the tracking continuation state in which tracking of the fingertip continues from the time of the previous frame to the current frame (tracking), and the tracking of the fingertip that has been tracked up to the previous frame is lost (disappears) in the current frame. It has at least three states: a state (lost) and a new appearance state (new appearance) in which a fingertip that has not been tracked up to the previous frame has newly appeared.

As a calculation method of the predicted position of the fingertip, for example, the following calculation method can be exemplified. That is, it is a method of calculating based on position information indicating the position of each fingertip on a plurality of past moving image frames. Here, as the tracking information of each finger, the position information of each fingertip on the previous frame times and the previous frame is included. Therefore, based on the position information, the movement of each fingertip is estimated, and on the next frame. The estimated movement destination of each fingertip is calculated as the predicted position. As a specific example, as shown in FIG. 58 (a), in the two-dimensional coordinate system on the moving image frame 400 defined in common on each moving image frame, the position coordinates P of the finger specified on the previous frame are determined. When moving from _i (-2) to the position coordinate P _i (-1) specified on the previous frame, the finger velocity vector V _i (-1) at the position coordinate P _i (-1) is Calculation is performed based on the moving direction and moving distance of the movement. In the next frame, it is estimated that the finger located at the position coordinate P _i (−1) moves by the velocity vector V _i (−1), and the position coordinate Q _i (−0) of the estimated movement destination is calculated (Q _i (−0) = P _i (−1) + V _i (−1)). The calculated position coordinate Q _i (−0) is the predicted position information, and is stored in the tracking information storage unit of the control circuit 18 as one piece of tracking information.

However, predicting the appearance of the next frame with respect to a fingertip that has newly appeared in the previous frame cannot be performed by the prediction method using both the previous frame and the previous frame as described above. In such a case, here, based on the position information of the fingertip specified on the previous frame, the position of the fingertip in the previous frame is determined as a temporary position, the position information indicating the temporary position, and the position information of the previous frame Based on the above, the appearance position of the fingertip on the next frame is specified. The method for determining the temporary position is as follows. That is, as shown in FIG. 59, the area in the moving image frame (fingertip identifiable range) 400 is partitioned in advance in association with each edge (here, the top, bottom, left, and right edges) (partition lines are An edge corresponding area 410 corresponding to each edge 401 (401a to 401d) is set, and then, first, the fingertip position P _i (−1) specified on the previous frame is set. The edge corresponding region 410 (401A to 401D) in which is present is specified. When the edge corresponding region 410 where P _i (−1) exists is specified, the position on the edge 401 of the edge corresponding region 410 closest to the P _i (−1) Is determined as a temporary position. That is, a perpendicular line that passes through the fingertip position P _i (−1) specified on the previous frame and goes to the edge 401 of the edge corresponding region 410 where the fingertip position P _i (−1) exists, and the edge The intersection with the part 401 is defined as a temporary position P _i ′ (−2).

The calculation method of the predicted position of the fingertip is a method of calculating based on the position information of each fingertip on the previous frame times and the previous frame, but in order to calculate with higher accuracy, FIG. Calculate as shown in b). That is, in the two-dimensional coordinate system on the moving image frame 400 defined in common on each moving image frame, the finger velocity vector on the position coordinates P _i (−1) specified on the previous frame once from the current frame. V _i (-1) and acceleration vector A _i (-1) are calculated, respectively, and in the next frame, the finger located at the position coordinate P _i (-1) is the velocity vector V _i (-1) and acceleration vector. It is estimated that the movement corresponding to A _i (-1) occurs, and the position coordinate Q _i (-0) of the estimated movement destination is calculated (Q _i (-0) = P _i (-1) + V _i (0) = P _i (−1) + V _i (−1) + A _i (−1)). First, the position coordinate P _i (−3) specified on the frame three times before the current frame is moved to the position coordinate P _i (−2) specified on the second frame from the current frame. based on the moving direction and moving distance of time, it calculates the position coordinates P _i of the finger in (-2) velocity vector V _i (-2). Then, this finger is specified on the previous frame once from the current frame from the previous position coordinate moved from the position coordinate P _i (−2) on the moving image frame 400 by the velocity vector V _i (−2). The acceleration vector A _i (−1) is calculated based on the moving direction and moving distance when moving to the position coordinate P _i (−1) (A _i (−1) = V _i (−1) −V _i (−2) = P _i (−1) −P _i (−2) −V _i (−2)). Also, the velocity vector V _i (−1) is obtained from the position coordinates P _i (−2) specified on the previous frame twice from the current frame, and the position coordinates P _i specified on the previous frame once from the current frame. Calculate based on the moving direction and moving distance when moving to (-1).

As a method for specifying the fingertip tracking status, for example, the following specifying method can be exemplified. That is, the fingertip position information specified on the current frame is compared with the predicted position information of the fingertip on the current frame, and based on whether or not these positional relationships satisfy a predetermined reference condition. It is a method to specify. For example, as shown in FIG. 56, the position coordinates Q ₁ (-0), Q ₂ (-0), Q ₃ (-0) of the fingertip indicated by each predicted position information calculated for the current frame, A pair corresponding to a pair closest to the position in the moving image frame from among the position coordinates R _a (0), R _b (0), R _c (0),. Are recognized as a pair satisfying the reference condition. However, in the case where the distance between both positions forming a pair is not more than a predetermined reference distance, the pair is not recognized as a corresponding pair. For the corresponding pair, it is determined that the fingertip actually identified in the current frame and the fingertip corresponding to the position information of the previous frame that is the basis for calculating the predicted position information are the same fingertip, The tracking of the fingertip is identified as a tracking continuation state (tracking) that continues from the previous frame to the current frame. Of the fingertips actually specified in the current frame, the fingertip for which no corresponding pair is found is specified as a new appearance state (new appearance). Of the fingertips corresponding to the predicted position information calculated for the current frame (fingertips identified as being tracked in the previous frame), the fingertips for which no corresponding pair was found are identified as a non-trackable state (lost). The identified tracking status of each fingertip is stored in the tracking information storage unit of the control circuit 18 in a form associated with each fingertip ID.

  In addition, when the untraceable state (lost) is specified as the tracking state, together with the untrackable state (tracking state) and the position information of the fingertip on the frame immediately before the untrackable state is specified, The untrackable recognition order in which the context of the untrackable specific time with other untrackable fingers can be specified is stored as tracking return determination information in the tracking information storage unit (tracking return determination information storage unit). Here, the tracking return determination information is also handled as one of the tracking information.

  Here, the flow of the finger tracking process (finger tracking step) shown in FIGS. 54 and 55 will be described. This finger tracking process is a process that is executed each time the control circuit 18 acquires a moving image frame from the camera 12b, whereby the control circuit 18 functions as a finger tracking unit.

  First, in R201 (FIG. 54), the control circuit 18 determines whether or not the fingertip is specified in a new moving image frame (current frame) acquired by moving image shooting by the camera 12b, and the fingertip is specified. If YES, the process proceeds to R202. If the fingertip is not specified, the process proceeds to R215. The fingertip specifying method is the same as T1 and T2 in FIG.

  In R202, the control circuit 18 specifies the correspondence between each fingertip specified on the current frame and each fingertip specified on the previous frame. Here, the correspondence between the fingertip position information in the current frame and the predicted fingertip position information predicted for the current frame is identified, and the tracking status of each fingertip at the time of acquiring the current frame is tracked from the corresponding result. For each fingertip ID, the state (tracking), untrackable state (lost), new appearance state (new appearance), or unspecified (not appearing) is specified. The identified tracking status is stored in the tracking information storage unit of the control circuit 18 for each corresponding fingertip ID in a form as indicated by reference numeral 500 in FIG. Here, when the fingertip specified on the previous frame and the fingertip specified on the current frame satisfying the above criteria are recognized as a tracking continuation state (tracking), it is specified on the previous frame. If any of the fingertips does not satisfy the above-mentioned standard condition, the fingertip specified on the current frame is recognized as a new appearance state (new appearance), and any of the fingertips specified on the current frame If there is a fingertip specified on the previous frame that does not satisfy the above, it is recognized as a non-trackable state (lost).

  In R203, whether or not the same finger exists on the fingertip specified on the previous frame and the fingertip specified on the current frame, in other words, the fingertip in the tracking continuation state (tracking) exists on the current frame. It is determined whether or not to do. If there is a fingertip in the tracking continuation state (tracking), the process proceeds to R204. If there is no fingertip in the tracking continuation state (tracking), the process proceeds to R205 in FIG.

  In R204, the tracking information corresponding to the fingertip ID of the fingertip in the tracking continuation state (tracking) is updated. The fingertip in the tracking continuation state (tracking) is the fingertip whose tracking information has been updated when the previous frame is acquired, and the position information of the current frame corresponding to the fingertip ID of the fingertip indicates the position of the fingertip in the current frame. Update with location information. Further, the position information of the current frame stored so far is updated as the position information of the previous frame, and the position information of the previous frame stored so far is updated as the position information of the previous frame.

In R205, whether there is a fingertip not specified on the current frame among the fingertips specified on the previous frame, in other words, tracking in the current frame, such as P ₁ (0) shown in FIG. It is determined whether there is a disabled fingertip. If there is a fingertip in the tracking disabled state (tracking), the process proceeds to R206, and if there is no fingertip in the tracking continuation state (tracking), the process proceeds to R207.

  In R206, information related to the finger when it is recognized that tracking is impossible is stored as tracking return determination information in association with the fingertip ID of the fingertip in the tracking disabled state (lost). This tracking return determination information is used to determine whether or not a finger that is not tracked here is recognized when a new fingertip that is not being tracked appears in a moving image frame acquired later. Information. As a result, when a fingertip once out of the frame enters the frame again, both can be recognized as the same finger, and the finger can be tracked again. Here, the fingertip ID of the fingertip that has become untrackable is stored and retained without being erased, and is associated with the fingertip ID along with the position information (position immediately before disappearance) of the previous frame corresponding to the fingertip ID. The untraceable recognition order (disappearance order) in which the context of the untrackable recognition time with the untrackable finger can be specified is stored as tracking return determination information. In the case of FIG. 56, “1” is assigned to the fingertip ID “2” as the order of disappearance, and when a new fingertip cannot be tracked in a moving image frame acquired later, the fingertip ID is “2” is assigned as the order of disappearance, and it can be determined that the higher the number assigned, the newer the untraceable recognition time.

  In R207, it is determined whether or not there is a fingertip not specified on the previous frame among the fingertips specified on the current frame, in other words, whether or not there is a fingertip newly appearing on the current frame. . If there is a fingertip in a new appearance state (new appearance), the process proceeds to R208. If there is no fingertip in a tracking continuation state, the process proceeds to R209.

  In R208, when it is specified in R207 that there is a fingertip in a new appearance state (new appearance), it is determined whether or not there is a fingertip that is currently in a tracking impossible state (lost). This determination is made based on whether or not the fingertip ID storing the tracking return determination information exists in the tracking information shown in FIG. At this stage, if it is determined that there is no fingertip that continues in the untrackable state (lost), the process proceeds to R213, and a new appearance state that has not been assigned to the fingertip in the new appearance state specified in R207. The position information indicating the position specified on the current frame of the fingertip in the newly appearing state is stored in a form in which the fingertip ID is assigned and associated with the fingertip ID. Further, the tracking status corresponding to the fingertip ID is changed from the non-trackable state to the tracking continuation state (tracking restart), and the tracking return determination information that has been continuously stored is erased. When R213 is terminated, the present process is terminated. On the other hand, if it is determined in R208 that there is a fingertip that continues to be untrackable (lost), the process proceeds to R209.

  In R209, it is specified whether or not there is a finger that is the same as the fingertip that is currently in the untrackable state (lost) among the newly appearing fingertips. Specifically, the position information in the current frame of the newly appearing fingertip actually identified in R207 and the position information (tracking return determination information) indicating the last position of the untrackable fingertip included in the tracking information 500 Based on the above, it is determined whether or not there is a fingertip that is closest to the position of the newly identified fingertip that is actually identified among the fingertips that are in an untraceable state within a certain range. Here, the distance between the fingertip position of each fingertip that is actually specified in the current frame and the fingertip position of each fingertip that is in an untrackable state is calculated, and the shortest of these distances is calculated. In addition, by identifying a distance that is less than a predetermined reference distance, each fingertip that is actually identified in the current frame corresponding to the distance and each fingertip that is in a non-trackable state are the same fingertip judge. However, if there are multiple fingertips that are in a non-trackable state within a certain range, the fingertip that has been in the most untrackable state based on the untrackable recognition order (tracking return judgment information) Identify the same as the fingertip identified as the new application in the current frame.

  In the subsequent R210, based on the result of R209, it is determined whether or not there is the same fingertip that has been in a trackable state (lost) among the fingertips identified as new applications in the current frame. If it exists, the process proceeds to R211. Otherwise, the process proceeds to R213.

  In R211, based on the result of R209, it is determined whether or not there are a plurality of fingertips that are determined to be the same as the fingertips that continue to be untrackable (lost) among the fingertips identified as new applications in the current frame To do. If it is determined that there is one instead of a plurality, the process proceeds to R212. In R212, the position information of the fingertip of the new application identified on the current frame, determined to be the same as the fingertip, is stored in association with the fingertip ID in the non-trackable state (lost) of the one fingertip, Update the tracking information of the fingertip ID. As a result, the tracking of the position of the fingertip that cannot be tracked is resumed. At this time, the tracking status corresponding to the fingertip ID is changed from the non-trackable state to the tracking continuation state (tracking restart), and the tracking return determination information that has been continuously stored is erased. When R212 is terminated, this process is terminated.

  On the other hand, in R211, based on the result of R209, among the fingertips identified as new applications in the current frame, it is determined that there are a plurality of fingertips that are determined to be the same as the fingertips that are in a non-trackable state (lost) If yes, go to R214. In R214, among the plurality of fingertips, based on the untrackable recognition order (tracking return determination information), the fingertip that has been most recently untrackable is identified and associated with the fingertip ID of the identified fingertip, The position information of the fingertip of the new application identified on the current frame determined to be the same as the fingertip is stored, and the tracking information of the fingertip ID is updated. As a result, the tracking of the position of the fingertip that cannot be tracked is resumed. At this time, the tracking status corresponding to the fingertip ID is changed from the non-trackable state to the tracking continuation state (tracking restart), and the tracking return determination information that has been continuously stored is erased. When R214 ends, this process ends.

  Note that in R201, when the fingertip is not specified in the acquired new moving image frame, the tracking information 500 as shown in FIG. 56 is reset, and the information corresponding to each ID is deleted. .

  In this way, by continuously tracking all the fingertips specified from the moving image photographed by the camera 12b in consideration of frame out, the operation target finger P can be fixed to the same finger. . Specifically, when the operation target finger P is designated by the process of specifying the operation target finger P, the operation target finger information indicating the operation target finger P is stored in a form associated with the fingertip ID as shown in FIG. By referring to this, the operation target finger P can be specified at any timing. That is, by adding a finger tracking process as shown in FIGS. 54 and 55, in the process of FIG. 25 that is repeatedly performed at a predetermined cycle, the process of specifying the operation target finger P by T3 is performed separately. The operation target finger P may be specified based on the operation target finger information stored in association with the fingertip ID.

  The fingertip tracking process is applied to the operation contents of the operation target finger designation operation performed by the user as in the fourth example of the tenth embodiment of FIG. 39 and the fifth example of the tenth embodiment of FIG. This is suitable for the embodiment in which the operation target finger P is designated. In the fourth example of the tenth embodiment of FIG. 39 and the fifth example of the tenth embodiment of FIG. 40, the touch operation on the touch panel 12a is an operation target finger designation operation, and based on the operation content of the touch operation, The operation target finger P is designated. That is, in the embodiment in which the operation target finger P is not changed unless the operation target finger designation operation is performed by the user, it is not necessary to perform an operation for specifying the operation target finger P every time a moving image frame is acquired. By simply picking up a finger to be designated from the tracking information, the operation target finger P can be easily identified.

  Further, the operation target finger P in the tenth embodiment may be configured to match the finger specified in the already described simple operation finger state.

  Hereinafter, embodiments of other embodiments will be described.

  In the above-described embodiment, the processed image (real finger image) 200H in which the contour shape of the photographed finger is reflected as it is is displayed as the position instruction image 200H, but the fingertip Q ( That is, an image completely different from the finger shape may be displayed, such as a fingertip position image (pointer or the like: see FIG. 67) 200H2 indicating a photographed finger pointing position). If highlighting is to be performed, only the fingertip region including the fingertip Q (for example, up to the first joint of the finger or an area within a predetermined radius centered on the fingertip position Q) may be displayed with emphasis. In this case, it is specified which position on the display screen the fingertip Q of the photographed finger faces, and a predetermined position indication image such as a pointer is displayed at the specified screen position. Further, as the position instruction image 200H, a pseudo finger image (here, a bar-shaped image: see FIG. 66) 200H1 or the like that simply shows the photographed finger length direction F and the fingertip Q may be displayed.

  Further, the operation target finger P in the tenth embodiment may be configured to match the finger specified as the already described simple operation finger state.

  Moreover, in the said embodiment, although the touch panel is provided and comprised as a remote operation part of this invention, an operation part provided with switches with pressing operation, such as a push switch and a seesaw switch, may be sufficient. In the above embodiment, a resistance film type touch panel is used as the operation panel 12a. However, another type of touch panel may be used.

  Further, as the operation panel 12a, an operation panel that detects the position of the touch operation by performing image processing on a photographed image photographed by photographing means such as a camera may be employed instead of the touch panel. Specifically, the operation panel 12a is a translucent panel whose surface 12a1 forms a touch operation surface, and a wavelength predetermined for an approaching object that approaches the operation panel 12a from the touch operation surface 12a1 side. A light source 12c for irradiating the band light through the operation panel 12a is provided. Then, the camera 12b serving as the hand image photographing means captures the approaching object from the back surface 12a2 side of the operation panel 12a1 in such a manner that the light emitted from the light source 12c captures the reflected light reflected by the approaching object. It is set as a photographing means. Further, the control circuit 18 functions as a light reflection area specifying means for specifying a light reflection area in which the intensity of the reflected light exceeds a predetermined threshold intensity on the image photographed by the camera 12b. Based on this light reflection area (coordinate information), it is configured to function also as an input receiving means for receiving a touch input to the light reflection area.

  According to this configuration, when a light reflection area exceeding a predetermined threshold intensity is specified in the captured image of the camera 12b, it is determined that a touch operation input has been made in the specified area, and the input is accepted. That is, the captured image of the camera 12b is configured to be used not only for displaying the position indication image as in the above-described embodiment, but also for specifying the light reflection area. Further, in this case, when the area of the specified light reflection area exceeds a predetermined threshold area, a touch operation input to the light reflection area can be received.

  Of course, each of the above-described embodiments can be implemented in combination with each element.

10 Vehicle Navigation Device 12 Operation Information Input Unit (Remote Operation Unit)
12a Operation panel 12b Camera (hand image photographing means)
12c Light source 15 Display device (display means)
15a touch panel 18 control circuit (input receiving means, position instruction image display means, simple operation finger state specifying means, operation input control means, display mode setting means, operator specifying means, running state specifying means, position instruction image emphasizing display means , Finger identification means, operation target finger designation means, fingertip identification means, designated finger information acquisition means)
18a Image processing unit 20 Camera 30 Vehicle speed sensor 40 Camera 41, 42 Camera 50 In-vehicle LAN
100 ECU
150 Captured image 200B Main image 200I Switch image (operation icon)
200H Position indication image (hand image)
H Approaching object (hand / finger)

Claims (28)

Display means having a display screen for operating devices provided in the vehicle, and a remote operation surface provided at a position different from the display screen for remotely performing operation input on the display screen A remote operation unit; an input receiving unit that receives an input corresponding to a pressing operation position based on a pressing operation on the remote operation surface; a hand image capturing unit that captures an image of a hand facing the remote operation surface; A position instruction image display means for displaying a position instruction image generated based on the image of the photographed hand at a position on the display screen instructed by a hand.
Finger state specifying means for simple operation for specifying whether or not the finger state of the hand facing the remote operation surface is a predetermined finger state for simple operation;
The display mode set for the position instruction image display means is set to a display permission mode that permits display of the position instruction image when the simple operation finger state is specified, and the simple operation finger state is set. Display mode setting means for executing display mode switching control for setting to a display prohibition mode for prohibiting display of the position indicating image when
A vehicle operation input device comprising: Vehicle speed detection means for detecting the vehicle speed of the vehicle,
The display mode setting means executes the display mode switching control when the vehicle is in a vehicle traveling state exceeding a predetermined vehicle speed level, and as the display mode when not in the vehicle traveling state. The vehicle operation input device according to claim 1, wherein the display permission mode is set. An operator specifying means for specifying whether the operator of the remote operation surface is a driver;
The display mode setting means executes the display mode switching control when the driver is specified as the operator of the remote operation surface, and the driver is specified as the operator of the remote operation surface. The vehicle operation input device according to claim 1 or 2, wherein the display permission mode is set as the display mode when the display is not performed.   When the display permission mode is set by the display mode setting means, the input acceptance by the input acceptance means is permitted, and when the display prohibition mode is set, by the input acceptance means. The vehicle operation input device according to any one of claims 1 to 3, further comprising operation input reception control means for executing operation input reception control that prohibits input reception.   In the case where the finger state for simple operation is specified by the finger state specifying means for simple operation, the input reception by the input receiving means is permitted, and the finger state for simple operation is not specified The operation input device for a vehicle according to any one of claims 1 to 3, further comprising operation input reception control means for executing operation input reception control for prohibiting input reception by the input reception means.   The simple operation finger state specifying means includes finger shape recognition means for recognizing a finger shape of a hand facing the remote operation surface based on the image of the hand photographed by the hand image photographing means. The vehicle operation input according to any one of claims 1 to 5, wherein the finger shape for the simple operation is specified as the finger state for the simple operation. apparatus.   When the finger shape recognizing unit recognizes a finger extending substantially along the remote operation surface, the simple operation finger state specifying unit specifies the number of fingers in a state extending substantially along the remote operation surface. A finger number specifying means for specifying the simple operation finger state when the number of fingers extending substantially along the specified remote operation surface is a predetermined simple operation number The vehicular operation input device according to claim 6 which is a thing.   The simple operation finger state specifying means includes an operation number specifying means for specifying the number of inputs made by pressing operations performed at different positions on the remote operation surface, received by the input receiving means during a predetermined period. 6. The device according to claim 1, wherein when the number of inputs specified in the period is a predetermined number for simple operation, the finger state for simple operation is specified. The vehicle operation input device according to claim 1.   The vehicle operation input device according to claim 7 or 8, wherein the number for the simple operation is one.   The vehicle operation input device according to claim 7 or 8, wherein the number for the simple operation is two.   The vehicle operation input device according to claim 7 or 8, wherein the number for the simple operation is three.   The vehicle operation input device according to claim 7 or 8, wherein the number for simple operation is four.   When the finger shape recognizing unit recognizes a finger extending substantially along the remote operation surface, the simple operation finger state specifying unit specifies a type of finger in a state extending substantially along the remote operation surface. The simple operation finger when the finger type in the state extending substantially along the specified remote operation surface is only a predetermined simple finger type for simple operation. The vehicle operation input device according to claim 6, which is specified as a state.   The vehicle operation input device according to claim 13, wherein the simple operation finger type is an index finger.   The vehicle operation input device according to claim 13, wherein the finger type for the simple operation is a middle finger.   The vehicle operation input device according to claim 13, wherein the finger type for simple operation is both an index finger and a middle finger.   The position instruction image display means generates a real finger image that enables at least the contour of the finger of the hand to be specified based on the hand image taken by the hand image photographing means, and the generated real finger image is used as the real finger image. The vehicle operation input device according to any one of claims 1 to 16, wherein the operation input device is displayed as a position indication image.   The position instruction image display means displays, as the position instruction image, either a pseudo finger image in which at least the finger extending direction and the fingertip position can be specified, or a fingertip position image in which at least the fingertip position of the finger can be specified. The vehicle operation input device according to any one of claims 1 to 16, wherein: Comprising the requirements of claim 2;
The vehicle operation input device according to any one of claims 1 to 18, wherein the predetermined vehicle speed level is set to 0 km / h. Comprising the requirements of claim 2;
The vehicle operation input device according to any one of claims 1 to 18, wherein the predetermined vehicle speed level is set in a vehicle speed range that is faster than 0 km / h and slower than 5 km / h. Comprising the requirements of claim 2;
The vehicle operation input device according to any one of claims 1 to 18, wherein the predetermined vehicle speed level is set within a vehicle speed range of 5 km / h or more and less than 10 km / h. Comprising the requirements of claim 2;
The vehicle operation input device according to any one of claims 1 to 18, wherein the predetermined vehicle speed level is set within a vehicle speed range of 10 km / h or more and less than 30 km / h. Comprising the requirements of claim 2;
The vehicle operation input device according to any one of claims 1 to 18, wherein the predetermined vehicle speed level is set within a vehicle speed range of 30 km / h or more and less than 60 km / h. Comprising the requirements of claim 2;
The vehicle operation input device according to any one of claims 1 to 18, wherein the predetermined vehicle speed level is set within a vehicle speed range of 60 km / h or more. An operation screen display means for displaying a plurality of operation screens that are received by the input receiving means by a pressing operation on the remote operation surface in a form of switching on the display screen;
Display mode execution availability information storage means for preliminarily storing display mode execution availability information that enables identification of an operation screen to be executed by the display mode switching control among the plurality of operation screens;
The display mode setting means is provided only when the operation screen displayed by the operation screen display means is specified as the execution target of the display mode switching control based on the display mode execution availability information. The vehicle operation input device according to any one of claims 1 to 24, wherein the display mode switching control is executed on the specified operation screen.   The display mode execution propriety information includes a complex operation screen including an operation in which a pressing operation on the remote operation surface, which is accepted by the input accepting unit, of the operation screen exceeds a predetermined operation burden level. 26. The vehicle operation input device according to claim 25, wherein a display mode switching control is determined as an execution target, and a simple operation screen that is only an operation that does not exceed a predetermined operation load level is determined as a fixed setting target of the display permission mode. .   27. The vehicle operation input device according to claim 26, wherein the simple operation screen includes at least one of a menu selection screen, a volume adjustment screen, and a wind speed adjustment screen.   The vehicle operation input device according to claim 26 or claim 27, wherein the complex operation screen includes at least a character input screen.

Images