JP5257779B2 - Display method, display device, and operation input device - Google Patents

Description

  The present invention relates to a display method for displaying an image corresponding to a finger in a form superimposed or synthesized on another image, a display device adopting the display method, and an operation input device including the display device About.

  In recent years, an operation input device that remotely operates a device has been developed as an operation input device (Patent Document 1). Such an operation input device is provided with a remote operation unit at the operator's hand separately from a display device that displays an operation screen (main screen), and further, photographs and extracts a hand that operates the remote operation unit, The obtained hand image and manual operation are reflected on the operation screen (main screen) of the display device, and display control is performed as if the operation screen is directly touch-operated. Among such operation input devices, those provided in the vehicle are provided with the remote operation unit at the user's seat on the seat in order to increase the operability. The remote operation unit is often configured as a touch operation type operation unit having a touch operation surface.

JP 2000-6687 A

  In the operation input device as described above, in the display device, when the hand operating the remote operation unit is actually displayed on the operation screen, when the hand is close to the remote operation unit, as shown in FIG. In this state, only the finger is displayed. However, if many fingers are displayed side by side at the same time, it is confusing which finger to operate, and the correspondence between the displayed finger and the finger that the operator actually wants to use for position indication The relationship could not be understood, which could lead to misoperation. In particular, when the operation input device is for a vehicle, the driver who is driving the vehicle cannot stare at the operation screen, so that it is difficult to input operations with multiple fingers while driving, and erroneous operations are likely to occur.

  An object of the present invention is to provide a display method of a position fingertip image that can easily identify an actual finger used for position indication in a display in which a position indication image corresponding to a finger is superimposed or synthesized on another image. And a manipulation input device including the display device.

Means for Solving the Problems and Effects of the Invention

In order to solve the above problems, the first display method of the present invention is:
In a display device comprising a display means having a display screen and a remote operation unit having a remote operation surface for remotely performing operations on the display screen, fingers are opposed to the front side of the remote operation unit A display method for displaying a position indicating image in a form superimposed or superimposed on an image forming a background at a screen position on the display screen corresponding to the finger position.
A finger specifying step of specifying a finger in the opposite hand on the front side of the remote control unit;
A designated finger information obtaining step for obtaining, as designated finger information, approaching distance information reflecting the approaching distance relationship between the fingers identified by the finger identifying step to the remote operation surface;
An operation target finger that is used for the operation from the specified finger to the remote operation unit or is estimated to be used is selected from the fingers specified by the finger specifying step based on the obtained approach distance information , An operation target finger designating step that preferentially designates the finger of the fingertip closer to the remote operation surface ;
A display step of displaying the position indicating image indicating the specified pointing position of the finger in a form synthesized or superimposed on a background image on the display screen, wherein the position indicating image is designated among the displayed position indicating images; A position indication image emphasis display step performed by highlighting the position indication image corresponding to the operation target finger relative to a position indication image corresponding to a finger different from the operation target finger;
It is characterized by having.
The second of the display methods of the present invention is
In a display device comprising a display means having a display screen and a remote operation unit having a remote operation surface for remotely performing operations on the display screen, fingers are opposed to the front side of the remote operation unit A display method for displaying a position indicating image in a form superimposed or superimposed on an image forming a background at a screen position on the display screen corresponding to the finger position.
The remote operation unit is a touch operation unit that detects the presence or absence of a touch operation and a touch position on the touch operation surface while the remote operation surface forms a touch operation surface,
A finger specifying step of specifying a finger in the opposite hand on the front side of the remote control unit;
An operation target finger specifying step of using a finger that has performed a touch operation on the touch operation surface among the specified fingers to be used for an operation to the remote operation unit, or to be specified as an operation target finger that is estimated to be used,
A display step of displaying the position indicating image indicating the specified pointing position of the finger in a form synthesized or superimposed on a background image on the display screen, wherein the position indicating image is designated among the displayed position indicating images; A position indication image emphasis display step performed by highlighting the position indication image corresponding to the operation target finger relative to a position indication image corresponding to a finger different from the operation target finger;
It is characterized by having.
The third of the display methods of the present invention is
In a display device comprising a display means having a display screen and a remote operation unit having a remote operation surface for remotely performing operations on the display screen, fingers are opposed to the front side of the remote operation unit A display method for displaying a position indicating image in a form superimposed or superimposed on an image forming a background at a screen position on the display screen corresponding to the finger position.
A finger specifying step of specifying a finger in the opposite hand on the front side of the remote control unit;
An operation target finger designation step for designating an operation target finger to be used for the operation from the identified finger to the remote operation unit or estimated to be used;
A display step of displaying the position indicating image indicating the specified pointing position of the finger in a form synthesized or superimposed on a background image on the display screen, wherein the position indicating image is designated among the displayed position indicating images; A position indication image emphasis display step performed by highlighting the position indication image corresponding to the operation target finger relative to a position indication image corresponding to a finger different from the operation target finger;
And the finger identifying step sequentially identifies the fingers of the hand that are within a finger identifiable range predetermined on the near side of the remote control unit at predetermined time intervals,
By acquiring the position information for each finger sequentially specified by the finger specifying step and storing the position information as tracking information in a predetermined tracking information storage unit, the finger sequentially specified by the finger specifying step is stored. Finger tracking step to track the movement of each finger,
After the operation target finger is specified in the operation target finger specifying step, the finger stored in the tracking information storage unit by the finger tracking step is used to track the movement of the finger specified as the operation target finger. An operation target finger tracking step of identifying the tracking information of the operation target finger from each tracking information, and tracking the movement of the finger designated as the operation target finger based on the tracking information of the operation target finger;
And the operation target finger designation step continuously designates the same finger as the finger designated as the operation target finger as the operation target finger based on the tracking result of the operation target finger tracking step. And
The finger tracking step includes
Among each finger being tracked that stores the tracking information for each finger specifying step, the finger is determined in advance with the finger actually specified when the finger is specified by the finger specifying step. If there is a finger that does not meet the reference criteria, the recognition is switched to the finger being tracked as an untrackable finger, and other untrackable fingers along with information about the finger when it is recognized as untrackable. The tracking unrecognized order in which the context of the non-trackable recognition time can be specified is stored as tracking return determination information in a predetermined tracking return determination information storage unit,
The finger tracking step is stored when there is an actually specified finger that does not satisfy the reference condition with the finger being tracked when the finger is specified by the finger specifying step. On the basis of the tracking return determination information, it is determined whether the finger actually identified is the same as the finger that was previously recognized as untraceable. Recognize that the finger that was not trackable corresponding to the tracking return determination information has reappeared at the position of the finger that was actually identified, and resume the movement tracking of the finger. If the finger has been identified, the finger actually identified is recognized as a newly appearing finger, and movement tracking of the finger is newly started. The tracking information includes at least the finger identifying step. The latest position of the finger specified by Are included, the even track return determination information includes last position information before the corresponding finger is recognized as the tracking impossible,
Further, the finger tracking step includes an actually specified finger that does not satisfy the reference condition with the finger being tracked when the finger is specified in the finger specifying step, and the tracking return is performed. If the determination information storage unit stores the tracking return determination information of a plurality of fingers, the position information of the finger actually identified and the last of the untrackable fingers included in the tracking return determination information When the tracking return determination information including the position information indicating the position closest to the finger that is actually specified is specified based on the position information, and one piece of the tracking return determination information is specified When the finger that is not traceable corresponding to the tracking return determination information is determined to be the same finger as the actually specified finger, while two or more tracking return determination information is specified, Add them Referring to the untraceable recognition order included in the return determination information, the finger that is not trackable corresponding to the latest untrackable recognition time among the tracking return determination information is actually identified. The finger is determined to be the same finger as the finger.

  According to the present invention, even when a large number of fingers face the remote operation surface and all of them are displayed on the screen, the position corresponding to the position of the operation target finger is determined according to a predetermined method and a predetermined rule. An instruction image is specified, and the position instruction image is emphasized relative to other fingers. Thus, the user can easily recognize from the screen which finger should be used for the operation. In addition, the correspondence between the displayed finger and the actual finger can be easily recognized as long as the user knows how to determine the operation target finger. Accordingly, it is possible to display a position indication image suitable for remote position indication.

  In the present invention, a fingertip specifying step for specifying a fingertip in the hand facing the remote operation surface can be provided. Specifically, the finger specifying step is a fingertip specifying step for specifying a fingertip from a hand facing the remote operation surface, and may be a step for specifying the finger in a form for specifying the fingertip. In this case, the position instruction image highlighting step is a display step based on the premise that the position instruction image is displayed at a position corresponding to the specified fingertip in a form synthesized or superimposed on the background image on the display screen. As described above, the highlighting can be performed.

  In the present invention, the fingertip refers to a predetermined region on the tip side of the finger (for example, up to the first indirect) or a specific position (for example, the region from the tip of the finger to the first indirect). Centroid position, etc.).

  Further, the operation target finger designation step can designate one finger as the operation target finger. In particular, if the number is one, it is easy to clearly recognize the operation target finger, and it is particularly suitable for application to a remote operation unit in which a contact operation including a push operation is performed. In addition, each processing can be facilitated because the one-finger state is a very simple hand shape.

  The operation target finger designation step may designate two or three fingers as the operation target fingers. If two or three are used, it is suitable for application to a remote control unit that performs pull operation, rotation operation, and the like.

  Further, in the position instruction image highlighting step, the position instruction image corresponding to a finger different from the operation target finger can be not hidden. Thereby, it becomes possible to grasp the correspondence between the displayed operation target finger and the actual operation target finger while referring to the display states of the other fingers.

  Moreover, in this invention, the designated finger information acquisition step which acquires the finger shape information which concerns on the shape of the finger specified by the finger specification step as specified finger information can be provided. In this case, the above-described operation target finger designating step is based on the obtained finger shape information, and among the fingers identified by the finger identifying step, the finger shape most suitable for a predetermined selection condition related to the shape of the finger It can be a step of preferentially specifying a finger having information as an operation target finger. When only one operation target finger is specified, the finger most suitable for the selection condition can be designated as the operation target finger.

  According to this configuration, for example, a predetermined type of finger can be specified or estimated from information related to the shape of the finger, and the finger can be designated as the operation target finger. Since each finger has a different shape for each type, it is possible to identify or estimate the finger from these, and therefore it is possible to identify or estimate an index finger or middle finger that is easy to use for position indication.

  Specifically, the designated finger information acquisition step may be a step of acquiring finger length information reflecting the length relationship between the fingers specified in the finger specification step in the remote operation surface direction as finger shape information. In this case, the operation target finger designating step includes a finger suitable for a predetermined selection condition related to the finger length information among the fingers identified by the finger identifying step based on the obtained finger length information. Can be designated as an operation target finger with priority. According to this configuration, since the finger length relationship is clarified, a finger of a predetermined type can be specified or estimated from the comparison of the lengths, and the finger can be designated as the operation target finger. For example, a middle finger that is easy to use for position indication can be identified as the finger that is identified (or estimated) as being the longest. If it is an index finger, as long as the operating hand is the left or right hand, it can be specified as one finger adjacent to the finger that is specified or estimated to be the longest. In addition, according to this configuration, since the longest appearing finger is generally highly likely to be used for the operation, the opposite halfway finger that is not intended to be used for the operation is surely excluded. It becomes possible to do.

  In addition, the designated finger information acquisition step acquires finger width information reflecting the relationship between the widths of the fingers specified in the finger specification step in the remote operation surface direction as finger shape information, and may be a step. In this case, the operation target finger designating step includes a finger suitable for a predetermined selection condition related to the finger width information among the fingers identified by the finger identifying step based on the obtained finger width information. Can be designated as an operation target finger with priority. According to this configuration, since the relationship between the widths (thicknesses) of the fingers is clarified, it is possible to specify or estimate a predetermined type of finger from the comparison of the widths, and specify the finger as an operation target finger. it can.

  Further, according to the present invention, the positional relationship between the fingers specified in the finger specifying step in the remote operation surface direction (the direction in which the remote operation surface spreads, that is, the direction perpendicular to the normal line of the remote operation surface). A designated finger information acquisition step for acquiring the positional relationship information as the specified finger information can be provided, and the above-described operation target finger specification step is performed based on the acquired positional relationship information. Of these steps, the finger having the positional relationship information most suitable for the predetermined selection condition related to the positional relationship can be preferentially designated as the operation target finger. When only one operation target finger is specified, the finger most suitable for the selection condition can be designated as the operation target finger.

  According to this configuration, for example, it is possible to specify or estimate a predetermined type of finger from the positional relationship of the specified plurality of fingers in the remote operation surface direction, and specify the finger as an operation target finger. Become. Since the positional relationship of each finger is determined for each type, the finger can be specified using this. For example, if the index finger is the easiest to use for position indication, the thumb is removed from the finger shape etc., and if the hand to be operated is the left hand, it can be specified as the rightmost finger and the hand to be operated is the right hand Can be identified as the leftmost finger. If the index finger can be specified, the adjacent finger can be specified as the middle finger. Further, in this configuration, since the operation target finger is selected according to the positional relationship of the fingers, it is easy for the user to understand which finger is selected.

  Specifically, in the designated finger information acquisition step, the fingertip specified in the fingertip specifying step is projected in a direction facing the remote control unit, and one axis is set at a predetermined position on the projection plane. A one-dimensional coordinate system, specify the coordinate value of the projected fingertip in the coordinate system, and acquire the axis coordinate information reflecting the coordinate value as positional relationship information, in this case, The operation target finger designation step gives priority to a finger suitable for a predetermined selection condition related to the axis coordinate information among the fingers identified by the finger identification step based on the acquired axis coordinate information. The step can be designated as an operation target finger. According to this configuration, the operation target finger can be easily specified only by the coordinates at which each fingertip is positioned in the position-dimensional coordinate system. In addition, the axis line in this case can be defined as, for example, a straight line from the region where the operation target finger is most likely to be located to the region where it is difficult to locate, or vice versa, on the remote operation surface.

  In the designated finger information acquisition step, the fingertip specified in the fingertip specifying step is projected in a direction facing the remote control unit, and the projected fingertip is projected with a reference area set in a predetermined area on the projection plane. In this case, the operation target finger designating step is based on the obtained distance information by the finger specifying step. Among the identified fingers, the step may be a step of preferentially designating a finger suitable for a predetermined selection condition related to the distance information as an operation target finger. According to this configuration, the operation target finger can be easily identified only by calculating the distance from the reference region. In this case, the reference area can be determined, for example, as an area where the operation target finger is most likely to be located on the remote operation surface, or an opposite area.

  Further, in the present invention, there can be provided a designated finger information obtaining step for obtaining, as designated finger information, approaching distance information reflecting the approaching distance relationship between the fingers identified by the finger identifying step and the remote operation surface, In this case, the operation target finger designating step can be a step of preferentially designating a finger suitable for a predetermined selection condition regarding the approach distance as the operation target finger. According to this configuration, for example, it becomes possible to specify the operation target finger from the positional relationship of the specified plurality of fingers in the approaching direction to the remote operation surface. Since the operation on the remote operation surface is accompanied by contact with the fingertip of the hand, the finger whose fingertip is closest to the remote operation surface is a finger having a higher possibility of position designation. Accordingly, in the operation target finger designation step, the finger closer to the remote operation surface among the fingers identified in the finger identification step is preferentially designated as the operation target finger based on the acquired approach distance information. If it is set as a step, the finger most suitable for a position instruction | indication can be designated as an operation target finger.

  The designated finger information acquisition step can be a step of acquiring the brightness of the hand image taken by the photographing means as the approach distance information. In this case, the operation target finger designating step can be a step of preferentially designating a finger suitable for a predetermined selection condition regarding the luminance as the operation target finger. In order to achieve this, in the present invention, the hand approaching the remote operation surface is irradiated with light from the light source and the reflected light is captured by the hand from the back side of the remote operation surface by the photographing means. A shooting step for shooting is required. Since the imaging means captures the reflected light of the hand operating the remote operation surface, the fingertip closest to the remote operation surface appears with higher brightness, so the proximity distance information can be easily obtained using the brightness. It is possible to omit means and processing for actually measuring the approach distance. In addition, since the finger closest to the remote operation surface is a finger having a higher possibility of position designation, the operation target finger designation step here is performed by the fingertip identification means among the fingers identified in the finger identification step. The specified fingertip with the highest luminance can be a step of giving priority to the operation target finger.

  By the way, all the above-mentioned designated finger information acquisition steps can be steps for acquiring designated finger information based on the photographed image obtained by the photographing step. According to this configuration, the finger designation information can be easily obtained using information obtained from the photographed image. In order to realize this, as a photographing step, light is emitted from the light source to the hand approaching the remote operation surface and the reflected light is captured so that the hand is photographed from the back side of the remote operation surface. Therefore, a step for photographing is required.

  In the present invention, the hand approaching the remote operation surface is irradiated with light from the light source, and a photographing step is provided in which the hand is photographed from the back side of the remote operation surface by the photographing means so as to capture the reflected light. In this case, the finger specifying step can be a step of specifying a finger from the photographed hand image. Furthermore, the fingertip specifying step can be a step of specifying the fingertip from the captured hand image. Furthermore, the display step of displaying the position instruction image may be a step of displaying a position instruction image generated based on the photographed hand image. According to these configurations, it is possible to specify a finger, a fingertip, and further specify an operation target finger and a position indication display by using a captured image of one imaging unit such as a camera. Compared with the provided configuration, the configuration can be simplified.

  By the way, the remote operation part in this invention is comprised as a touch operation part which detects the presence or absence of touch operation to the said touch operation surface, and a touch position, for example like a touch panel etc., and a remote operation surface makes a touch operation surface. Can do. In this case, the operation target finger designation step may be a step of designating, as the operation target finger, a finger that has performed a touch operation on the touch operation surface among the fingers identified in the finger identification step. According to this configuration, after the position instruction is actually made, the finger used for the position instruction is specified as the operation target finger. Since a general operator tends to use the operation target finger fixedly, the operation target finger can be accurately determined and displayed in an emphasized manner. In addition, by highlighting the finger that has been touched, it becomes possible to clearly recognize the correspondence between a large number of fingers displayed on the screen and the actual finger by using this touch operation.

  Note that when a touch operation is accidentally performed by touching the remote operation unit, the operation can be prevented from being reflected in the input, and further, it can be prevented from flourishing in the highlighted display. Specifically, the operation target finger designation step is a step of designating the finger that has performed the touch operation as the operation target finger when the touch operation is a touch operation that exceeds a predetermined pressing operation time. That's fine. Further, the operation target finger designation step may be a step of designating the finger that has performed the touch operation as the operation target finger when the first touch operation is a touch operation exceeding a predetermined pressing operation force. According to these configurations, an operation with a load of a certain level or more is not effective as a touch operation, so that an erroneous operation can be prevented.

  By the way, the position indication image emphasis display step in the present invention can be a step of performing emphasis display in such a manner that the emphasis level is changed stepwise over a predetermined time. That is, in the highlight display, the display is such that one or more transitional states different from both the state before the start of the highlight and the state after the completion of the highlight are passed between the start of the highlight display and the completion of the highlight display. be able to. According to this configuration, since the highlight change process is displayed, it is easy to recognize the correspondence between the displayed finger (especially the operation target finger) and the actual finger by seeing the process.

  Further, in the position indication image highlighting step, as the highlighting, at the start of the highlighting, a position indication image corresponding to the operation target finger and a position indication image corresponding to a finger different from the position indication finger. Are changed in such a manner that the relative enhancement level of the position pointing image corresponding to the operation target finger increases step by step as time elapses from the start of the highlight display. It can also be set as the step which performs the display to perform. In this case, the enhancement level of the position indicating image corresponding to the operation target finger is gradually (relatively) increased. Immediately after the completion of the emphasis state, immediately after that, it becomes confusing to recognize which finger that the highlighted operation target finger actually hits. You can eliminate confusion.

  In this way, if the importance of displaying the state before highlighting is considered, the position indication image highlighting step is performed after the predetermined time has elapsed after the operation target finger is specified by the operation target finger specifying step. It is good to set it as the step which starts a display. However, in the configuration in which the emphasis level gradually increases as described above, since the process of the change can be visually recognized over time, the position target image emphasis display step is designated by the operation target finger designation step. Accordingly, relative highlighting may be started.

  In the present invention, a hand image photographing step for photographing an image of a hand facing the remote operation surface can be provided. In this case, the finger specifying step can specify the finger of the hand facing the remote operation surface based on the photographed hand image. Further, in this case, the position indicating image highlighting step generates a real finger image that enables at least the contour of the finger of the identified hand to be identified based on the captured hand image, and the generated real finger image Can be displayed as a position indicating image. Since the contour of the actual hand is displayed on the operation screen, the position of the hand can be grasped from the contour. Since it is an actual hand contour, the position can be easily recognized, and the operation becomes easier.

  Further, in the present invention, the position instruction image highlighting step includes either the pseudo finger image in which the finger extending direction can be specified at least or the fingertip position image in which at least the fingertip position of the finger can be specified as the position instruction. It can also be configured to display as an image. The processing load can be reduced by using an image simplified to such an extent that the finger pointing position can be understood as the position pointing image.

  By the way, the highlighting in the present invention can be made on the contour line of the position indicating image. By emphasizing the contour line, the designated position of a specific finger becomes easier to understand.

  The highlighting in the present invention is performed by changing the display color of the position indication image, changing the shape of the position indication image, changing the transparency of the position indication image, or blinking the position indication image. There may be. By emphasizing with these simple image processes, the processing load can be reduced. Further, it is possible to sufficiently emphasize a specific finger even with these highlighted displays.

  By the way, in the present invention, a finger that is used or estimated to be used for an operation to the remote operation unit is designated as an operation target finger, and a position indication image displayed in correspondence with the designated operation target finger is obtained. It is characterized by being displayed with a relatively higher emphasis than the position indication image displayed corresponding to other fingers. In this case, it is better from the viewpoint of operability that a constant finger is always designated as the operation target finger. That is, it is desirable that the same finger is continuously designated as the operation target finger.

  As a configuration for this, the display method of the present invention includes an operation target finger tracking step for tracking the movement of the finger specified as the operation target finger after the operation target finger is specified in the operation target finger specifying step. The operation target finger designation step continuously designates the same finger as the operation target finger as the operation target finger based on the tracking result of the operation target finger tracking step. Can do. According to this configuration, since the movement of the finger designated as the operation target finger is tracked, the same finger can always be designated as the operation target finger. Compared to the case where the operation target finger changes as the finger moves, the operation is easier.

  Further, the finger specifying step of the present invention may sequentially specify fingers of a hand existing within a finger specifying range predetermined on the near side of the remote control unit at predetermined time intervals. For example, when the finger specifying step is to specify a finger from an image of a hand that is shot with a moving image by the shooting unit, for each moving image frame that is sequentially acquired at predetermined time intervals by shooting, from the hand image displayed in the frame The finger will be specified. In such a case, position information for each finger sequentially specified by the finger specifying step is acquired, and the position information is sequentially specified by the finger specifying step by storing the position information as tracking information in a predetermined tracking information storage unit. By providing a finger tracking step for tracking finger movement for each finger, the operation target finger tracking step is based on the tracking information of the operation target finger among the tracking information for each finger stored in the finger tracking step. The movement of the finger specified as the operation target finger can be tracked. According to this configuration, since the movement of all the fingers specified by the finger specifying step is tracked, it is possible to cope with any finger designated as the operation target finger. In addition, since the movement of the operation target finger occurs in conjunction with other fingers, if tracking information of not only the operation target finger but also other fingers is collected, this can be used for tracking the operation target finger. More accurate tracking is also possible.

  In the finger tracking step, each tracking target storing the tracking information for each finger specifying step is set to the corresponding tracking information, with the next position estimated to be positioned in the next finger specifying step. It is possible to have a next position estimation step that is calculated on the basis of this. In this case, the finger tracking step includes the next position of each finger being tracked calculated in the next position estimating step and the actual position of each finger actually specified in the next finger specifying step. The positional relationship specifies a pair that satisfies a predetermined standard condition, and the finger being tracked and the finger actually specified in the finger specifying step as one of the pair are mutually the same finger The movement tracking of the finger can be continued assuming that the finger being determined has moved to the position of the actually specified finger determined to be the same. Specifically, the position information of the actually identified finger that is recognized as the same is stored in the tracking information storage unit in the form of being stored in the tracking information storage unit as the latest position information of the finger being tracked. The movement tracking of the finger can be continued by updating the stored tracking information. According to this configuration, the tracking information is stored, so that the next movement destination of the finger being tracked can be estimated from the past tracking information. Furthermore, the positional relationship between the estimated position and the position where the finger is actually specified is compared. If both are close to each other, for example, a predetermined level or more, the finger being tracked moves as estimated. It can be judged that In other words, the movement destination being tracked can be estimated with high accuracy by using past tracking information, and the movement tracking of the finger is performed using the estimation result, so that high tracking accuracy can be obtained. .

  The tracking information may include each position information of the corresponding finger being tracked identified by the finger identifying step a plurality of times. In this case, the next position estimation step can calculate the next position of the corresponding finger being tracked based on the position information. According to this configuration, since the next position is estimated from the past positions of the finger being tracked, high estimation accuracy can be obtained.

  Further, in the next position estimation step of the present invention, when the tracking information includes each position information of the corresponding finger being tracked specified by the finger specifying step of the previous time and the previous time, both positions of the tracking information are included. While the next position of the finger being tracked can be calculated based on the information, the tracking information includes the corresponding position information of the finger being tracked identified by the previous finger identifying step. If the position information of the corresponding finger being tracked specified by the finger specifying step last time is not included, the position information specified by the previous finger specifying step and the position Based on the position of the edge of the finger identifiable range close to the position indicated by the information, the next position of the finger being tracked can be calculated. When the previous and previous positions of the finger being tracked are known, the next position can be estimated with high accuracy using both of them, but the position of the finger being tracked is known only for the previous time. In this case, that is, when the finger being tracked is a finger that has newly appeared last time, since only one position is known, the next position cannot be estimated with high accuracy alone. However, according to the above configuration, the position of the edge of the finger identifiable range close to the previous position is used instead of the previous position. In this case, since the finger that has just newly appeared last time is likely to have appeared from the nearest edge of the finger identifiable range, there is no previous position by using that edge as the previous position. However, the next position can be estimated with sufficient accuracy.

  In the finger tracking step of the present invention, when there is a finger being tracked that does not satisfy the reference condition with the finger actually specified when the finger is specified in the finger specifying step, the tracking is performed. It is possible to switch the recognition as a finger that cannot be tracked and to store information related to the finger when it is recognized as untraceable as tracking return determination information in a predetermined tracking return determination information storage unit. . In this case, the finger tracking step is further performed when there is an actually specified finger that does not satisfy the reference condition with the finger being tracked when the finger is specified by the finger specifying step. , Based on the tracking return determination information stored in the tracking return determination information storage unit, it is determined whether the finger actually identified is the same as a finger that has been made untrackable in the past, the same If it is determined that the finger that was untrackable corresponding to the tracking return determination information has reappeared at the position of the finger that was actually identified, the movement tracking of the finger On the other hand, if it is determined that the fingers are not the same, the finger actually identified is recognized as a newly appearing finger, and movement tracking of the finger can be newly started. Specifically, when it is determined that they are the same, the position information of the actually identified finger is the tracking information as the latest position information of the finger that is recognized as unidentifiable and that cannot be tracked. By renewing the tracking information stored in the tracking information storage unit in the form stored in the storage unit, the movement tracking of the finger is resumed. By storing the position information of the finger in the tracking information storage unit as the tracking information, the movement tracking of the finger can be newly started. The operation target finger is not specified when it moves outside the range that can be specified in the finger specification step, but it is a good idea to use a finger other than that finger as the operation target finger just because it is no longer specified. Is not good. For example, when the index finger has been moved as an operation target finger and the index finger has moved out of a specific range, it is less desirable to change the ring finger to the operation target finger simply because the ring finger has been specified. Rather, it is better not to designate the operation target finger until the index finger is identified again. That is, it is easier to operate if a certain finger is always designated as the operation target finger. According to the above configuration, when the finger being tracked is lost (tracking is impossible), the situation at the time of the loss is stored as tracking return determination information, and when a finger that has not been tracked is newly specified, Based on the accumulated tracking return determination information, it is determined whether the finger is a finger that has been lost in the past. If a finger that has been lost once appears again, it is recognized as the same finger. Can do. If the lost finger is the operation target finger, it can be immediately designated again as the operation target finger when it reappears, and movement tracking can be resumed as it is.

  The tracking information in this case may include at least the latest position information of the finger specified by the finger specifying step, and the tracking return determination information is also the last information before the corresponding finger is recognized as untrackable. It can include location information. According to this configuration, in the finger tracking step, the position immediately before being lost (not trackable) is compared with the reappearing position, and a pair in a close positional relationship can be determined as the same finger. That is, it is possible to easily determine whether or not a newly specified finger is a re-appearing finger simply by storing the previous position at the time of loss and comparing the position at the time of re-appearance.

  In the finger tracking step of the present invention, when the finger is specified by the finger specifying step, an actually specified finger that does not satisfy the reference condition with the finger being tracked is specified, and When tracking return determination information of a plurality of fingers is stored in the tracking return determination information storage unit, the position information of the finger actually identified and the untrackable finger included in the tracking return determination information Based on the last position information of the first finger, the finger that is untraceable corresponding to the tracking return determination information including the position information indicating the position closest to the finger actually identified is actually identified. It can be determined that the finger is the same finger. According to this configuration, even when there are a plurality of lost (untrackable) fingers, the last specified position of the finger in the lost is compared with the position of the newly specified finger, and the lost The middle finger is determined to be the same as the newly specified finger. Since a lost finger is likely to appear from the same position as the lost part, by using this, a newly identified finger or a lost finger reappears. Can be accurately identified.

  The finger tracking step of the present invention is performed when an actually specified finger that does not satisfy the reference condition with the finger being tracked when the finger is specified by the finger specifying step is specified. When the distance between the positions indicated by the actually specified finger position information and the last position information of the non-trackable finger included in the tracking return determination information is less than a predetermined reference distance Is allowed to recognize that the finger actually identified and the finger that is not traceable corresponding to the tracking return determination information are the same, but if it is not less than the reference distance, It can be prohibited to recognize that the finger specified in (2) is identical to the finger that cannot be traced corresponding to the tracking return determination information. When comparing the last specified position of a finger that is lost (not trackable) with the position of a newly specified finger, if the distance between them is too far away, both fingers may be different fingers High nature. With the above-described configuration, it is possible to more accurately identify whether a newly specified finger has newly appeared or a lost finger has reappeared.

  In the finger tracking step of the present invention, the finger that is actually specified when the finger is specified by the finger specifying step includes the finger being tracked that does not satisfy the reference condition. If it is recognized that tracking is not possible, the information related to the finger when it is recognized that tracking is not possible, and the unrecognizable order in which the tracking relationship with other non-tracking fingers can be identified. Can be stored in the tracking return determination information storage unit as tracking return determination information. In this case, the finger tracking step further specifies an actually specified finger that does not satisfy the reference condition with the finger being tracked when the finger is specified in the finger specifying step. In addition, when the tracking return determination information storage unit stores tracking return determination information of a plurality of fingers, it is included in the actually specified finger position information and the tracking return determination information. The tracking return determination information including the position information indicating the position closest to the actually specified finger can be specified based on the last position information of the finger that cannot be tracked. However, in the case where one tracking return determination information including the position information indicating the position closest to the actually specified finger is specified, the finger that cannot be tracked corresponding to the tracking return determination information is specified. Is determined to be the same finger as the actually specified finger, while two or more tracking return determination information including the position information indicating the position closest to the actually specified finger is specified. (That is, when the position indicated by the position information included in the plurality of tracking return determination information is in a positional relationship satisfying a predetermined proximity condition), the untrackable recognition order included in the tracking return determination information is Referring to the tracking return determination information, the finger that cannot be tracked corresponding to the latest tracking-unrecognized recognition time can be determined to be the same finger as the actually specified finger. According to the arrangement of fingers in the human hand, when multiple fingers are lost (not traceable) at the same position last, when they appear again, there is a high possibility that they will appear in order from the last lost finger. . With the above configuration, even if there are a plurality of fingers in the lost state, it is possible to accurately specify which finger in the lost state is the same as the newly specified finger.

  Note that the tracking return determination information of the present invention can be the tracking information of the finger stored in the tracking information storage unit when the corresponding finger is recognized as untrackable. According to this configuration, when the finger being tracked is lost (not trackable), the tracking information of the finger stored in the tracking information storage unit may be stored in the tracking return determination information as it is. If the tracking information storage unit and the tracking return determination information storage unit are the same, it is only necessary to leave the stored tracking information as it is.

  In the finger tracking step of the present invention, when the finger is specified by the finger specifying step and the specified finger is recognized as a newly appearing finger, new identification information is given to the new finger. The tracking information of each finger can be stored in the tracking information storage unit in association with the identification information. Thereby, identification information is attached to the finger specified in the finger specifying step, and information management corresponding to the finger is facilitated.

  Further, in the configuration of the present invention having the tracking return determination information storage unit, when the finger tracking step recognizes the finger being tracked to which the identification information is given as a finger that cannot be tracked, The tracking return determination information storage unit may store the identification information in association with the tracking return determination information. In this case, even if the finger being tracked is lost (tracking is impossible), the identification information of the finger remains as it is, so that information corresponding to the finger can be easily managed even after the finger is lost.

  The finger tracking step of the present invention includes the tracking information stored in the tracking information storage unit and the tracking recovery determination stored in the tracking recovery determination information storage unit when the finger is specified in the finger specifying step. Based on the information, whether the finger actually specified by the finger specifying step is the finger being tracked, or whether the finger is not trackable or not. The determination result can be stored in a predetermined tracking state storage unit in a form associated with the identification information as a tracking state. According to this configuration, it is possible to easily identify the tracking state (during tracking, lost, etc.) of the finger with identification information.

  In the finger tracking step of the present invention, if the finger is not specified when the finger is specified in the finger specifying step, the identification information and various information stored in association with the identification information are stored. It can be erased and initialized. While various information of the finger specified sequentially is stored and updated in association with the identification information, all the information can be initialized once at the timing when the finger is not specified.

  In addition, the display device of the present invention realizes each step (finger identification step, operation target finger designation step, fingertip identification step, position indication image emphasis display step, designation finger information acquisition step) in the display method of the present invention described above. Means (finger identification means, operation target finger designation means, fingertip identification means, position indication image emphasis display means, designation finger information acquisition means) can be provided. Thereby, the said effect is realizable in a display apparatus.

  Moreover, the operation input device of the present invention can be configured to include the display device. Specifically, the display screen is configured as a screen for performing device operations and the like, and the remote operation surface is provided at a position different from the display screen to remotely perform operation input on the display screen. And an input for accepting an input corresponding to a pressing operation position based on a pressing operation on the remote operation surface, a display means having the display screen, a remote operation unit having the remote operation surface A receiving unit, a hand image capturing unit that captures an image of a hand facing the remote operation surface, and a position indication image generated based on a hand image taken at a position on the display screen instructed by the hand. And a position indication image emphasis display means for displaying. Thereby, the operation input device for remote operation which has the effect of the said display method is realizable.

  The operation input device of the present invention can be a vehicle operation input device. Specifically, the device is a vehicle-mounted device, the display screen is arranged to be visible by the driver on the vehicle front side of the driver, and the remote operation surface is closer to the driver than the display screen. Can be provided in position. Since the operation target finger is displayed in an easy-to-understand manner, the operation becomes easy. In particular, it can contribute to simplification of the driver's operation.

The block diagram which shows the structure of the navigation apparatus for vehicles using the operation input device for vehicles which is an example of the touch-type operation input device of this invention. The principal part perspective view in a vehicle interior for demonstrating the outline | summary of the operation input apparatus for vehicles. Sectional drawing which shows the 1st structure of an operation information input part. The 1st figure explaining photography and display of the hand which opposes an operation panel. The 2nd figure explaining photography and a display of the hand which opposes an operation panel. The flowchart which shows the flow of the highlight display process of a position instruction | indication image. The flowchart which shows the flow of the specific process of the operation target finger in 1st embodiment of this invention. The figure explaining the specific method of a fingertip. The figure explaining the 1st example of the method of specifying an operation target finger in 1st embodiment of this invention. The figure explaining the 2nd example of the method of specifying an operation target finger in 1st embodiment of this invention. The figure explaining the 3rd example of the method of specifying an operation target finger in 1st embodiment of this invention. The figure which shows an example of the highlight display of a position instruction | indication image. The flowchart which shows the flow of a touch operation input process. The flowchart which shows the flow of the specific process of the operation target finger in the 1st modification of 1st embodiment of this invention. The figure explaining the 1st example of the method of specifying an operation target finger in the 1st modification of 1st embodiment of this invention. The figure explaining the 2nd example of the method of specifying an operation target finger in the 1st modification of 1st embodiment of this invention. The flowchart which shows the flow of the specific process of the operation target finger in the 2nd modification of 1st embodiment of this invention. The figure explaining an example of the method of specifying an operation target finger in the 2nd modification of 1st embodiment of this invention. The flowchart which shows the flow of the specific process of the operation target finger in the 3rd modification of 1st embodiment of this invention. The flowchart which shows the flow of the specific process of the operation target finger in the 3rd modification of 1st embodiment of this invention. The figure explaining an example of the method of specifying an operation target finger in the 4th modification of 1st embodiment of this invention. The figure explaining an example of the method of specifying an operation target finger in the 5th modification of 1st embodiment of this invention. The figure explaining an example of the method of specifying an operation target finger in the 5th modification of 1st embodiment of this invention. The figure explaining an example of the method of specifying an operation target finger in the sixth modification of the first embodiment of the present invention. The 1st figure explaining the method of specifying an operation target finger in the 6th modification of a first embodiment of the present invention. The 1st figure explaining the method of specifying an operation target finger in the 6th modification of a first embodiment of the present invention. The perspective view of the principal part in a vehicle interior for demonstrating the outline | summary of the operation input device for vehicles different from FIG. The figure which shows an example of the image image | photographed in order to identify an operator. An example of the display image different from FIG.4 or FIG.5 (c). The perspective view which shows simply the operation part on the premise of knob operation in the 7th modification of 1st embodiment of this invention. An example of the picked-up image from the x direction, y direction, and z direction shown in FIG. An example of the display image based on FIG. An example of the display image based on FIG. 31 different from FIG. The figure explaining the 2nd example of the method of specifying an operation target finger in the 7th modification of a first embodiment of the present invention. The figure explaining the imaging | photography and display of the hand which opposes an operation panel in the conventional vehicle operation input device. The flowchart which shows the flow of a finger tracking process. The flowchart following FIG. The figure explaining the tracking information memorize | stored in the control circuit which estimates a fingertip position. The figure explaining the basic algorithm which tracks a fingertip between moving image frames. The figure which shows the calculation method of the appearance predicted position of a fingertip (finger). The figure explaining the calculation method of the appearance predicted position in the next frame of the fingertip which newly appeared in the previous frame. The figure explaining the new appearance of a fingertip and the loss of a fingertip. The figure which shows an example of the time change of the highlight display of the position instruction | indication image in this invention. The figure which shows an example of the time change of the highlight display of the position instruction | indication image in this invention. The figure which shows the example of 1 display of the highlight display of the position instruction | indication image in 10th embodiment of this invention. The figure which shows the example of 1 display of the highlight display of the position instruction | indication image in 10th embodiment of this invention. The figure which shows the example of 1 display of the highlight display of the position instruction | indication image in 10th embodiment of this invention. The figure which shows the example of 1 display of the highlight display of the position instruction | indication image in 10th embodiment of this invention. The figure which shows one example of a display which displayed the position instruction | indication image in this invention with the pseudo finger image. The figure which shows the example of a display which displayed the position instruction | indication image in this invention with the fingertip position image. The 3rd figure explaining imaging | photography and display of the hand which opposes an operation panel. The flowchart which shows the position instruction | indication image display process of 2nd embodiment of this invention. The flowchart which shows the touch operation input process of 3rd embodiment of this invention. FIG. 10 is a fourth diagram illustrating photographing and display of a hand facing the operation panel. The flowchart which shows the position instruction | indication image display process of 4th embodiment of this invention. The flowchart which shows the touch operation input process of 6th embodiment of this invention. The figure which shows the 1st example of the display permission period which can be set in this invention. The figure which shows the display permission period in 4th embodiment of this invention. The figure which shows the display permission period in 5th embodiment of this invention. The flowchart which shows the position instruction | indication image display process of 7th embodiment of this invention. The flowchart which shows the position instruction | indication image display process of 8th embodiment of this invention. The flowchart which shows the position instruction | indication image display process of 9th embodiment of this invention. The figure which shows the example of a setting of a vehicle speed threshold value. The flowchart which shows the position instruction | indication image display process of 10th Embodiment of this invention. An example of the operation screen in this invention. An example of the operation screen in this invention. An example of the operation screen in this invention. The figure explaining an example of a predetermined finger state. The figure explaining an example of a predetermined finger state. The figure explaining an example of a predetermined finger state. The figure explaining an example of a predetermined finger state. The figure which shows the example of the screen conditions used as the implementation target of display mode switching control. An example of a predetermined finger state.

  Hereinafter, an embodiment of an operation input device of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing a configuration of a vehicle navigation device using the operation input device of the present invention. In the present embodiment, as the operation input device 1, a display device 15 capable of displaying an operation input image such as a switch image (operation icon) for touch operation on the display screen, and a touch operation on the display screen are remotely performed. An operation input device is employed in which an operation panel 12a for performing the operation is provided at a different position. Note that the operation target of the touch-type operation input device of the present embodiment is not limited to the navigation function, but is a device function (audio, air conditioner, power window, etc.) of other in-vehicle electronic devices controlled by the ECU 100 connected via the in-vehicle LAN 50. Sheet adjustment function, etc.).

  The navigation device 10 includes a position detector 11 that detects the current position of the vehicle, an operation information input unit 12 for inputting various instructions from an operator such as a driver, and an external that records map data and various information. A map data input device 14 for inputting map data and the like from a recording medium, a display device 15 for performing various displays such as a map display screen and a TV (Television) screen, and outputting various guide voices, etc. A voice input / output device 16 for inputting the voice of the operator, a hard disk storage device 17 for storing various data such as vehicle information, and a vehicle I / F (for transmitting and receiving vehicle information) Interface) unit 19, a wireless communication device 13 for performing short-range wireless communication with another communication device 13 a, and a control circuit 18 connected to these 11 to 19. The control circuit 18 can also be connected to the cameras 20 and 40 and the vehicle speed sensor 30.

  The position detector 11 receives a radio wave transmitted from an artificial satellite for GPS (Global Positioning System) via a GPS antenna and is added to the vehicle and a GPS receiver 11a that detects the position, direction, speed, and the like of the vehicle. A gyroscope 11b that detects the magnitude of the rotational motion, a distance sensor 11c that detects the distance traveled from the longitudinal acceleration of the vehicle, and the like, and a geomagnetic sensor 11d that detects the traveling direction from the geomagnetism. Each of the sensors 11a to 11d has an error of a different property, and is configured to be used while complementing each other. Depending on the accuracy, some of the sensors described above may be used, or a steering rotation sensor, a wheel sensor for each rolling wheel, or the like (not shown) may be used.

  As shown in FIG. 3, the operation information input unit 12 irradiates light of a predetermined wavelength band, and transmits the light emitted from the light source 12c from the surface 12a2 side to the surface 12a1 side. A translucent operation panel 12a is provided, and a camera (approaching object photographing means) 12b that photographs from the back surface 12a2 side of the operation panel 12a so as to include the operation panel 12a in the photographing range. .

  In addition, as shown in FIG. 2, the operation information input unit 12 of the present embodiment is operated from both the right and left seats below the lower edge of the windshield of the vehicle, such as the center console unit C of the vehicle. It is installed in a possible position. Furthermore, it is arranged below the display screen of the display device 15 and on the front side between both seats of the adjacent left and right seats. Note that the adjacent left and right seats in the present embodiment are the driver's seat 2D and the passenger seat 2P in FIG. 2, but may be, for example, the left and right seats in the rear row. Furthermore, the position is not limited to a position that can be operated from both the left and right seats, and may be installed at a position where the operator can easily operate, for example, a steering section or a door section.

  The operation panel 12a is a panel having translucency with respect to light emitted from at least the light source 12c. In this embodiment, it is configured as a well-known resistive film type touch panel provided with a vertical transparent electrode and a horizontal transparent electrode, and its output is input to the control circuit 18.

  As shown in FIG. 3, the operation panel 12a is fitted into the front end opening of the hollow casing 12f, and the surface 12a1 is a touch operation surface (remote operation surface). From the outer edge of the housing front end 121e forming the front end opening, a cylindrical wall 122e is provided rearward, and the rear end is fixed to the circuit board. The housing 12f is fixedly disposed at a predetermined position in the vehicle interior, and a camera 12b and a light source 12c are disposed inside.

  As shown in FIG. 3, the light source 12c irradiates light in such a manner that the operation panel 12a is transmitted from the back surface 12a2 to the front surface 12a1. Light (irradiation light) from the light source 12c passes outside from the touch operation surface 12a1 unless there is an approaching object (such as a hand) positioned so as to cover the touch operation surface 12a1, When there is an approaching object H that is approaching the touch operation surface 12a1, it is reflected by the approaching object H and part of the light (reflected light) returns to the back surface 12a2 side, and the camera 12b. To be captured.

  The camera 12b captures the reflected object reflected from the approaching object H by the light emitted from the light source 12c, and photographs the approaching object H from the back surface 12a2 side of the operation panel 12a. It is fixed on the vehicle body side to shoot the shooting range. Specifically, a reflecting member 12r is provided so that the photographing range can be photographed by the camera 12b, and the camera 12b photographs the reflected image. When there is no reflecting member 12r, the photographing range is directly photographed by the camera 12b. Therefore, it is necessary to perform left-right reversal processing on the photographed image.

  In this embodiment, the light source 12c is an infrared light source, and the camera 12b includes an infrared camera (not shown) that shuts out visible light and is suitable for shooting in a dark place. Do). Photographing is performed in such a manner that the camera 12b captures the reflected light reflected from the approaching object H by the light (infrared rays) emitted from the infrared light source 12c. The camera 12b is fixedly arranged on a substrate fixed to the vehicle body side, and a range including the touch operation surface 12a1 is set as a shooting range. The infrared light source 12 is an LED located on the rear side of the operation panel 12a. In addition, since the operation panel 12a according to the present embodiment is made of a material having translucency only with respect to the light from the light source 12c, an infrared filter in the camera 12b can be eliminated, and The inside of the housing 12f is not visible from the outside.

  Note that the camera 12b in the present embodiment functions as a photographing unit of the present invention. This camera 12b functions as an approaching object photographing means for photographing an approaching object approaching the operation panel 12a, and considering that the approaching object includes a hand, the hand It can be said that it functions as a hand image photographing means for photographing an image, and further as a finger image photographing means for photographing an image of a finger of a hand.

  4, 5, and 51 are diagrams for explaining photographing of a hand (finger) facing the operation panel 12a. (A) of each figure has shown the state (finger state) of the operation panel 12a and the hand which opposes the surface 12a1, and (b) of each figure shows the hand of the state of (a) with the camera 12b. The image 150 is taken in the above, and is an image taken using the entire operation panel 12a as a shooting area. However, the video 150 of (b) in this embodiment is a binarized version of the video after shooting. In (b), reference numeral 150H is an area where the approaching object H is reflected, and reference numeral 150B is an area where nothing is reflected. The region denoted by reference numeral 150B is originally a region where the operation panel 12a is reflected, but is not reflected because the operation panel 12a has translucency with respect to the light from the light source 12c. Even in such a case, it is not strange that the background (the ceiling surface of the vehicle in the present embodiment) is reflected through the operation panel 12a, but such background is removed. Background removal is possible by using images with different illumination conditions, as described in Japanese Patent Application Laid-Open No. 2007-272596. Incidentally, (c) of each figure is an example of a screen display on the display device 15 when the video 150 of (b) is taken. In the present invention, when a plurality of fingers are opposed to the operation panel 12a, the display area of the position indication image 200H corresponding to a specific finger corresponds to the remaining fingers as shown in FIG. The position indication image 200H to be displayed is combined with or superimposed on the background image 200B in a relatively emphasized form.

  Returning to FIG. 1, the display device 15 is a color display device installed on the front side of the driver's seat 2 </ b> D in FIG. 2 so as to be visible by the driver, and the display screen is farther from the driver than the touch operation surface 12 a 1. In the position. As the display device 15, for example, any of a liquid crystal display, a plasma display, an organic EL display, or the like may be used. The display screen also shows a mark indicating the current location identified from the current position of the vehicle detected by the position detector 11 and the map data input from the map data input device 14, a guide route to the destination, a name, and a mark. , Additional data such as marks of various facilities can be displayed in an overlapping manner. Also, facility guides can be displayed. The display device 15 corresponds to the display means of the present invention.

  In addition, the display device 15 has a display screen for performing operations of devices included in the vehicle, and in the present embodiment, the display device 15 includes a touch panel 15a on the display screen. The touch panel 12a of the operation information input unit 12 already described is a remote operation unit provided so that the operation of the touch panel 15a can be remotely operated from a position closer to the driver or passenger on the passenger seat. Based on the touch operation performed on the touch operation surface 12a1, an input to the screen position of the display device 15 (touch operation surface position of the touch panel 15a) corresponding to the touch operation position is accepted. More specifically, a two-dimensional coordinate system (two-dimensional coordinate system defined on the touch operation surface of the touch panel 15a) defined on the display screen of the display device 15 and a touch operation surface 12a1 of the operation panel 12a are defined. An unambiguous correspondence between the two-dimensional coordinate system is determined in advance, and the other in-plane position corresponding to the one in-plane position can be uniquely identified, and the touch operation surface 12a1. If there is a touch operation, an operation input to a position coordinate on the display screen of the display device 15 corresponding to the position coordinate where the touch operation is performed is received based on the correspondence.

  That is, when there is a touch operation on the touch operation surface 12a1, the control circuit 18 specifies the position coordinates on the touch operation surface 12a1 where the touch operation is performed, and further, the position on the specified touch operation surface 12a1. A position coordinate on the display screen of the display device 15 corresponding to the coordinate is specified, and then a control signal for executing control content corresponding to the specified position coordinate on the display screen is output. For example, when a switch image (operation icon or the like) 200I is displayed on the display screen of the display device 15, if the touch operation is performed at a position corresponding to the switch image 200I on the touch operation surface 12a1, the control circuit 18 Then, an input is accepted on the assumption that there has been a touch operation on the switch image 200I, and accordingly, the control content corresponding to the switch image 200I is executed. If a map screen (map operation image) that can be displayed on the display screen of the display device 15 is displayed, the control circuit 18 can be operated by touching the position on the map on the touch operation surface 12a1. Then, an input to the touch operation position is accepted, and accordingly, display feed control for displaying a map with the touch operation position as a new screen center is executed.

  The display screen of the display device 15 is arranged so as to be visible by the driver on the vehicle front side of the driver, and is provided at a position farther from the driver than the touch operation surface 12a1. Note that the display device 15 is not limited to the display device 15 disposed in the center console C as shown in FIG. 2, and may be installed above the lower edge of the windshield FG, for example. Specifically, a head-up display (HUD) for displaying the above-described image / data on the windshield FG of the vehicle, a meter display device located behind the steering wheel, and the like can be exemplified.

  Returning to FIG. 1, the map data input device 14 is road data as network data, map data such as so-called map matching data for improving the accuracy of position specification, mark data indicating facilities, guidance image and audio data. It is a device for inputting various data including the like. A CD-ROM, DVD-ROM, hard disk, memory, memory card, or the like can be used as a recording medium for these data.

  The voice input / output device 16 can output facility guides and various guidance voices input from the map data input unit 14, and read-out voices of information acquired via the I / F 19. The voice input / output device 16 includes a microphone (not shown) and a known voice recognition unit, and can input voice of an operator such as a driver to the control circuit 18 as a command or the like.

  The wireless communication device 13 is for performing narrowband communication with other communication devices 13a. For example, DSRC (Dedicated Short Range Communications), Bluetooth (registered trademark), wireless LAN (Local Area Network), UWB (Ultra Wideband). ) Etc. are used.

  The LAN I / F 19 is an interface circuit for exchanging data with other in-vehicle electronic devices and sensors via the in-vehicle LAN 50. Further, for example, data from another ECU (ECU 100 in FIG. 1 or the like) may be taken in via the LAN I / F 19.

  The control circuit 18 is configured around a well-known microcomputer including a well-known CPU, ROM, RAM, I / O (Input / Output) (not shown) and a bus line connecting these components. Based on each detection signal from the position detector 11 based on a program stored in the storage unit, the current position of the vehicle is calculated as a set of coordinates and traveling directions, and the vicinity of the current position read via the map data input device 14 A map display process for displaying a map of the range indicated by the operation information input unit 12 on the display device 15 as a main image (including moving images and still images) 200B as shown in FIG. Based on the point data stored in the map data input device 14, the facility as the destination is selected according to the operation of the operation information input unit 12, and the current position to the destination are selected. A route guidance process of performing route guidance performed automatically route calculation to find the optimal path. A technique such as the Dijkstra method is known as a technique for automatically setting an optimum route in this way.

  In addition, the control circuit 18 sub-switches a switch image (operation icon) 200I, a processed image 200H based on the approaching object image 150H acquired from the operation information input unit 12, and the like based on a program stored in a storage unit such as a ROM. The image processing unit 18a is used to display the image as a main image 200B (a map operation image or the like), or to display the image as an image, or to display it superimposed. In the present embodiment, when the approaching object image 150H is an image of a hand such as a finger, the processed image 200 generated based on the photographed hand image 150H is pointed by the finger of the hand. The position indication image 200H for indicating the position on the display screen is displayed in an overlapping manner. That is, the processed image 200H is overlaid and displayed as a position indicating image 200H indicating the position on the display screen where the position is specified by the finger of the hand.

  In the present embodiment, the processed image 200H based on the approaching object image 150H is an approaching object contour reflecting image that is processed and generated so that at least the outer shape (contour) of the approaching object image 150H is reflected. 200H. Then, this image 200H is superimposed on the main image 200B. Instead of the superimposed display, the image 200H may be combined with the main image 200B and displayed. As a result, when an operator such as a driver moves his / her finger or the like so as to face the touch operation surface 12a1 of the operation panel 12a, for example, a hand is displayed on the display screen of the display device 15. The processed image 200H reflecting the shape (finger shape) is moved and displayed, and the operation can be performed as if the operation panel 12a exists on the display screen of the display device 15.

  Further, the position instruction image 200H does not necessarily reflect the photographed hand shape / finger shape outline, and may be an image that indicates at least the position indicated by the photographed hand finger. It may be only a portion, only a fingertip, or an image such as a pointer. Moreover, there is no problem if the hand image (or finger image) 150H photographed by the camera 12b is displayed as it is without being processed (superimposed display or composite display) as the position indication image 200H, but the position overlapping image In view of the fact that a part of the main image 200B that should be displayed on the screen becomes invisible, it is more desirable to perform processing (for example, translucent synthesis) that makes the operation easier.

  Further, switch image data for displaying the switch image 200I is stored in a storage unit such as a ROM of the control circuit 18, and is used for overlapping display or composite display of the switch image 200I. On the other hand, in a storage unit such as a ROM of the control circuit 18, an operation input (touch input to a corresponding area on the touch operation surface 12 a 1) is performed in association with each switch image 200 I. The control content is also stored. When the control circuit 18 displays the image 200 in which the switch image 200I is superimposed on the main image 200B displayed on the display screen, the switch image 200I is displayed at the display position on both touch operation surfaces of the touch panels 12a and 15a. An operation input reception range (operation input position) is set in the corresponding area (operation input position setting means). When a map screen (map operation image) 200B2 that can be displayed and sent as the main image 200B is displayed on the display screen, an operation input reception range (operation input position) is set for the entire displayed map area. (Operation input position setting means).

  Further, the control circuit 18 can perform time counting by starting its own timer 18b and counting up the counter 18c. These are used when measuring time in the processing described later.

  Next, display processing of the position instruction image displayed on the display device 15 will be described with reference to FIG. This process is performed in such a manner that the CPU executes a program stored in the storage unit of the control circuit 18.

  At T1, the control circuit 18 specifies the finger of the hand facing the touch operation surface 12a1 on the front side of the touch operation surface (remote operation surface) 12a1 (here, the opposite side of the camera 12b) (finger specifying step). ). Here, based on the captured image captured by the camera 12b, it is determined whether an image of the finger of the operator has been acquired. The camera 12b always shoots an approaching object (for example, an operator's hand such as a driver) H that approaches the touch operation surface (front surface) 12a1 side from the back surface 12a2 side of the operation panel 12a via the operation panel 12a. The captured image is always input to the image processing unit 18 a of the control circuit 18. When the approaching object H enters the position facing the touch operation surface 12a1, the captured image 150 including the approaching object H is naturally input to the image processing unit 18a. The input captured image 150 is analyzed by a known image analysis method in the image processing unit 18a, and an approaching object image 150H is extracted based on a color difference or the like. Then, based on the shape of the extracted approaching object image 150H, it is determined whether or not the image 150H includes an image 150F of a human finger. Note that the finger image is recognized by identifying a substantially linear bar-shaped image pattern having an axial line longer than a predetermined length from the entire shape of the extracted hand image 150H, and there is the bar-shaped image pattern. This is recognized as a finger image 150F. The determination result is output to the control circuit 18, and if it is determined that the image of the finger has been recognized, the process proceeds to T2, and if not determined, this process ends.

  At T2, the control circuit 18 detects the fingertip from the acquired finger image 150F. The detection of the fingertip is performed by specifying the fingertip Q from the specified finger image 150F (fingertip specifying step). The fingertip Q in the present embodiment is specified in the form shown in FIG. That is, when the finger image 150F is specified as shown in FIG. 8A, the image processing unit 18a displays the hand image 150H including the specified finger image 150F and the hand image 150H of the finger image 150F. The moving hand images 160H shifted by a predetermined amount in the length direction are overlapped with each other, the non-overlapping region 170 where the finger images 150H and 160H do not overlap each other is specified, and the barycentric position of the non-overlapping region 170 is calculated. Then, the center of gravity position is specified as a fingertip (fingertip position) Q. Note that another method may be used for specifying the fingertip Q. For example, the contour of the fingertip that appears curved may be extracted and smoothed, and then determined as the center of curvature. In addition, the fingertip Q may be specified by specifying the area up to the first joint of the finger as the fingertip (fingertip area) Q instead of specifying it as one point of the finger as described above.

  T1 and T2 are steps for performing finger specifying processing and fingertip specifying processing, respectively, but they may be executed simultaneously. That is, instead of specifying the fingertip from the finger image, the fingertip may be specified directly from the captured image captured by the camera 12b, and the finger may be specified by specifying the fingertip. This simplifies the process.

  In subsequent T3, the control circuit 18 designates an operation target finger (position instruction finger) P to be used or estimated to be used for an operation on the remote operation unit (operation target finger designation step). Specifically, the positional relationship reflecting the positional relationship between the specified fingertips Q in the touch operation surface direction (the direction in which the touch operation surface 12a1 spreads, that is, the direction perpendicular to the normal line of the touch operation surface 12a1). Information is acquired as designated finger information (designated finger information obtaining step), and on the basis of the obtained positional relationship information, a predetermined selection condition relating to the positional relationship among the fingers identified by T1 is obtained. The finger having the positional relationship information most suitable for is designated as the operation target finger P with priority.

  A plurality of fingers can be specified as the operation target finger P, but from the viewpoint of giving a position instruction so that the operation on the contact-type operation unit 12a is easy to understand, one or two relatively It is desirable to specify a small number of fingers. In the present embodiment, there is one. Further, as the types of fingers to be designated, the index finger (see FIG. 4C), the middle finger (see FIG. 45), or the index finger and the middle finger (see FIG. 29), which are suitable for position indication, are suitable. In this embodiment, an index finger is assumed.

  The process of T3 of this embodiment is executed as the process shown in the flowchart of FIG. That is, at T11, the fingertip Q specified as shown in FIG. 8C is specified on the surface 180 of the virtual two-dimensional coordinate system, and one axis X determined on the surface 180 is used. Then, another one-dimensional coordinate system is defined in the two-dimensional coordinate system. Here, the specified fingertip Q is projected toward the touch operation surface 12a1 (in the direction facing the remote operation unit), and the projection surface is set as the above-described surface 180 at a predetermined position on the projection surface. One axis X is set to define a one-dimensional coordinate system. At T12, the position coordinate (coordinate value) XQ of the fingertip Q specified on the surface 180 is specified in the one-dimensional coordinate system, and the axis coordinate information reflecting the position coordinate XQ is acquired as the positional relationship information. Then, based on the acquired axis coordinate information at T13, priority is given to a finger suitable for a predetermined selection condition regarding the axis coordinate information among the fingers specified at T1 in FIG. And designated as the operation target finger P.

  Further, in the present embodiment, an operation with the right hand of the passenger on the passenger seat is assumed, and the axis X is set so that the index finger of the operator is designated as the operation target finger P. Therefore, at T11, the axis X is set at the position shown in FIG. At T12, the fingertip Q is projected toward the set axis X, and the coordinate values X1 to X3 of the projected positions are calculated. At T13, the coordinate value of the acquired coordinate values X1 to X3 is the largest. A large fingertip Q is specified, the finger having this fingertip Q is estimated to be an index finger, and this finger is designated as the operation target finger P.

  In addition, as shown in FIG. 9, the axis X in the present embodiment is determined so as to go from the lower right of the captured image in which the right index finger is difficult to be positioned to the upper left where the right index finger is easily positioned. 10 and FIG. 11 may be set. In the case of FIG. 10, it is suitable to estimate the finger with the largest coordinate value as the middle finger, and if it can be estimated or specified whether the operating hand is left or right, the finger adjacent to the middle finger is estimated as the index finger. You can also. In the case of FIG. 11, if it can be estimated or specified whether the operating hand is left or right, it is suitable to estimate the leftmost finger as the index finger, and the finger adjacent to it can be estimated as the middle finger.

  In the present embodiment, the photographed image photographed by the camera 12b can be referred to as a projection surface 180 obtained by projecting the fingertip Q toward the touch operation surface 12a1. Therefore, the operation panel image area (the area where the touch operation surface 12a is reflected: here the entire area of the captured image 150) where the touch operation surface 12a is located in the captured image captured by the camera 12b and the projection surface 180 are the same. The coordinate position of the fingertip Q on the operation panel image area can be used as the projected coordinate position of the fingertip Q on the projection plane 180 as it is. At T3, the operation target finger P is designated by calculation on the coordinate system on the virtually determined projection plane 180, and image processing is not required. That is, since the image data is not used for the operation target finger P specifying process, the calculation cost (calculation amount / memory amount) can be reduced.

  Returning to FIG. At T4, as shown in FIG. 12C, the control circuit 18 synthesizes or superimposes on the background image 200B on the display screen of the display device 15 on the display screen corresponding to the specified fingertip Q. An image 200H for position indication is displayed at the position. However, this display corresponds to the remaining finger R different from the operation target finger P in the display region 200P of the position instruction image 200H corresponding to the designated operation target finger P in the displayed position instruction image 200H. The position indication image 200H to be displayed is displayed in a highlighted manner in which it is emphasized relatively than the display area 200R. Thereby, the visibility of the operation target finger P is improved, the finger to be operated is easily transmitted to the user, and the operability is also improved.

  In the present embodiment, the control circuit 18 first specifies the positional relationship between the display screen of the display device 15 and the photographed finger. That is, the position on the display screen where the photographed finger faces is specified. In the present embodiment, an operation panel image area where the touch operation surface 12a is located (an area where the touch operation surface 12a is reflected: here, the entire area of the captured image 150) is specified on the captured image 150 captured by the camera 12b. At the same time, a two-dimensional operation coordinate system is set on the operation panel image area, and the position of the photographed finger is specified on the operation coordinate system. Then, the position on the display screen of the display device 15 corresponding to the position of the finger specified on the operation coordinate system is specified in the two-dimensional display coordinate system defined on the display screen.

  Then, as shown in FIG. 12A, the position indication image 200H is superimposed on the corresponding position coordinates of the main image 200B at the position specified on the display screen of the display device 15. Then, the above-described highlight display of the position indicating image 200H (200P, 200R) is performed in such a manner that the enhancement level (enhancement degree) is gradually changed over a predetermined time t1, as shown in FIG. That is, in the process of changing the highlighting, it has a state in the middle of change as shown in FIG. 12B, and the change in emphasis as shown in FIG. To do. If fingers that are not specified by highlighting are made difficult to see, it may be difficult to recognize which fingers are highlighted if the fingers are made difficult to see suddenly. For this reason, first, as shown in FIG. 12A, all fingers are displayed in the same manner, and if the emphasis level is changed step by step, the finger displayed on the screen and his / her own are displayed. Matching with fingers is easy.

  Further, the highlighting is performed on the entire finger display areas 200P to 200R of the position indication image 200H, the outline, and only the inside of the outline, with a color change (see FIG. 46) and a shading change (see FIG. Fingers may be erased), shape change (FIG. 47: the operation target finger P is relatively thick here as an example), switching to another image, change in transparency (see FIG. 4), It blinks (see FIG. 48) or a combination thereof. In the present embodiment, the processing is executed in such a manner that the transparency inside the contour of the processed image forming the position indicating image 200H is changed. Further, the highlighting here holds the highlighting state of the display area 200P of the position indicating image 200H corresponding to the operation target finger P constant, and emphasizes the display area 200R of the position indicating image 200H corresponding to the remaining fingers R. It is done in a form that reduces the level. The position indicating image region 200R corresponding to the remaining finger R may be erased. However, in consideration of operability, the emphasis level is lower than that at the time of erasure, but it may be relatively easily visible.

  Further, the highlighting at T4 may be started immediately as the operation target finger P is designated at T3 as shown in FIG. 43, or the operation at T3 as shown in FIG. It may be started after a predetermined time has elapsed since the target finger P is designated. In particular, in the case of a configuration in which highlighting is not performed in stages, it is desirable to select the latter.

  This process ends when T4 ends. This process is repeatedly performed at a predetermined cycle after the end, and when the hand facing the operation panel 12a moves, the display position of the position instruction image 200H displayed on the display screen of the display device 15 also moves accordingly. To do. When the hand moves in the middle of the emphasis change, the display position of the position indicating image 200H (200P and 200R) moves while continuing the emphasis change, and when the hand moves in a state where the emphasis change is completed The display position of the position indicating image 200H (200P and 200R) moves while maintaining the emphasized state.

  The display process of the position instruction image includes a finger specifying step, a fingertip specifying step, an operation target finger specifying step, a position instruction image emphasizing display step, and a specified finger information acquiring step. By executing this processing, the function functions as means for executing these steps (finger specifying means, fingertip specifying means, operation target finger specifying means, position indication image highlighting display means, and specified finger information acquiring means).

  Finally, touch operation input processing to the operation panel 12a of the operation information input unit 12 will be described with reference to FIG. This processing is also performed in such a manner that the CPU executes a program stored in the storage unit of the control circuit 18.

  At T101, the control circuit 18 determines whether or not there is a touch operation on the operation panel 12a. Since the operation panel 12a is configured as a touch panel, when a touch operation is performed, an operation signal is input to the control circuit 18, and thus determination is made based on the presence or absence of the input. If there is a touch operation on the operation panel 12a, the process proceeds to T102, and if there is no touch operation, this process ends.

  In subsequent T102, the touch operation position on the operation panel 12a, that is, the touch operation position on the touch operation surface 12a1 is specified. Since the operation panel 12 a is configured as a touch panel, an operation signal is input to the control circuit 18 when a touch operation is performed. Since the touch operation position is reflected in the operation signal as coordinate information on the operation coordinate system defined on the operation surface 12a1, the control circuit 18 specifies the touch operation position based on the operation signal. .

  In subsequent T103, the control circuit 18 performs control corresponding to the touch operation position on the operation panel 12a. Specifically, since each position on the touch operation surface 12a1 of the operation panel 12a corresponds to a position on the display screen of the display screen 15, the position on the touch operation surface 12a1 specified in T103 is set. The corresponding control content determined at the position on the display screen of the display screen 15 is executed. Specifically, the control circuit 18 outputs a control signal for performing the control content. For example, if the position on the touch operation surface 12a1 specified in T103 corresponds to the position of the switch image 200I displayed on the display screen of the display screen 15, the control circuit 18 is assigned to the switch image. A control signal for executing the control content is output.

  This process ends when T103 ends. Note that this process is repeatedly performed at a predetermined cycle even after the completion of the process, and the control circuit 18 performs the control content corresponding to the position touched at that time or is performed by another control unit (for example, the ECU 100 or the like). Let

  The control circuit 18 functions as an input receiving unit and an operation input control unit by executing the touch operation input process by the CPU.

  Although one embodiment of the present invention has been described above, this is merely an example, and the present invention is not limited to this, and the knowledge of those skilled in the art can be used without departing from the spirit of the claims. Various modifications based on this are possible.

  For example, in the above-described embodiment, the display method of the present invention is realized by a vehicle operation input device. However, the operation input device is not limited to a vehicle, for example, an operation input of a general household appliance. It may be a device. Furthermore, instead of an operation input device for performing device operations, a general display device having a screen display as a main function may be used.

  Hereinafter, an embodiment different from the above embodiment will be described as the first embodiment. In addition, about a common structure, description is abbreviate | omitted by using the same code | symbol.

  A first modification of the first embodiment will be described.

  As shown in FIGS. 14 to 16, the first modification of the first embodiment is different from the first embodiment in the method of specifying the operation target finger P. In the first modified example, as shown in the flowchart of FIG. 14, at T21, the fingertip Q specified at T2 of FIG. 6 is projected toward the touch operation surface 12a1, and predetermined on the projection surface 180 is predetermined. The reference area Y is set in the set area (see FIG. 15 or FIG. 16), and distance information reflecting the distance between the set reference area Y and the projected fingertip Q is used as positional relationship information in T22. A finger that is obtained (designated finger information obtaining step) and that is suitable for a predetermined selection condition related to the distance among the fingers identified at T1 in FIG. 6 based on the distance information obtained at T23. Is designated as the operation target finger P (operation target finger designation step).

  In the present modification as well, the fingertip Q is specified on the virtually defined surface 180 as in the first embodiment, and the reference region Y is further defined (T21). Then, each distance is calculated (T22), and the operation target finger P is designated based on the calculated distance (T23). Here, the middle finger of the operator is assumed as the operation target finger P, and the finger with the longest standard distance is designated as the operation target finger P. Note that the reference region Y may be defined as a region having an area, or may be defined as a point or line having no area.

  Further, as shown in FIG. 15, for example, the reference region Y can be arranged at a predetermined position where it is estimated that the back of the hand is located in a state where a finger is photographed by the camera 12b. The distance information in this case is information that does not directly reflect the length of the finger specified in T1, but is related to it, and a specific finger can be designated from the length of the distance. For example, the finger with the longest distance can be estimated as the middle finger, and if it can be estimated or specified whether the operating hand is left or right, the finger adjacent to the middle finger can be estimated as the index finger. When an index finger or the like is assumed as the operation target finger P, the second longest finger may be estimated as the index finger and specified.

  Further, for example, as shown in FIG. 16, the reference region Y can be arranged at a position where the fingertip of the finger assumed as the operation target finger P is most likely to be located in a state where the finger is photographed by the camera 12b. In this case, the finger having the fingertip Q having the shortest distance from the reference area Y can be designated as the operation target finger P. In the case of FIG. 16, it is assumed that the operating hand is the right hand and is located at the upper left.

  A second modification of the first embodiment will be described.

  As shown in FIGS. 17 and 18, the second modification of the first embodiment is different from the above embodiment in the method of specifying the operation target finger P. In the second modification, as shown in the flowchart of FIG. 17, at T31, finger shape information related to the shape of each finger specified at T1 of FIG. 6 is acquired as specified finger information (specified finger information acquisition). Step) Based on the finger shape information acquired at T32, the finger having the approach distance information most suitable for the predetermined selection condition related to the shape of the finger among the fingers specified at T1 is selected. Prioritize and specify as operation target finger (operation target finger specifying step).

  As the finger shape information, the finger length information related to the length relationship in the touch operation surface direction between the fingers specified at T1 in FIG. 6 can be determined. Specifically, the lengths d1 to d3 (see FIG. 18) from the fingertip Q specified at T2 in FIG. 6 to the position corresponding to the midpoint of the line segment of the finger edge appearing on the photographed image 150. These can be calculated and used as the finger length information. In this case, since the finger length information is information reflecting the length of the finger specified in T1, a specific finger can be designated from the length. For example, the finger with the longest distance can be estimated as the middle finger, and if it can be estimated or specified whether the operating hand is left or right, the finger adjacent to the middle finger can be estimated as the index finger. When an index finger or the like is assumed as the operation target finger P, the second longest finger may be estimated as the index finger and specified. According to this configuration, it is possible to reliably eliminate the possibility that a halfway extended finger that is not intended for operation is designated for position indication. Note that the finger length information d1 to d3 may be information reflecting at least the length of each finger. For example, the finger length from the tip position of the curved fingertip to the finger edge appearing on the captured image 150 is sufficient. It may be a distance along the direction.

  As the finger shape information, finger width information related to the width relationship in the touch operation surface direction between the fingers specified at T1 in FIG. 6 can also be determined. Specifically, lengths w1 to w3 (see FIG. 18) in the width direction that pass through the fingertip Q specified in T2 of FIG. 6 and are perpendicular to the length direction of the finger are used as the finger width information. Can do. In this case, since the finger width information appears larger as the finger approaches the touch operation surface 12a1, it can also be referred to as approach distance information that reflects the approach distance relationship to the touch operation surface 12a1. A large finger can be designated as the operation target finger P. Note that the finger width information w1 to w3 may be information reflecting at least the width of each finger, and may be a finger width calculated at a position different from the above.

  A third modification of the first embodiment of the present invention will be described.

  As shown in FIG. 19, the third modification of the first embodiment is different from the above embodiment in the method of specifying the operation target finger P. In the third modified example, as shown in the flowchart of FIG. 19, at T41, the touch operation position T on the touch operation surface 12a1 is detected by the finger specified at T1 of FIG. The detected finger that has performed the first touch operation is specified as the operation target finger P (operation target finger specifying step). That is, the touch operation position T on the touch operation surface 12a1 is acquired as designated finger information, and the operation target finger P is designated based on the acquired finger information. According to this configuration, since the highlighting start trigger is determined as a touch operation by the user, it is possible to avoid missing a display change due to the highlighting. In this case, the designated operation target finger P is tracked when it is moved by a manual operation, and continues to be designated until the finger P disappears from the screen or a new touch operation is performed. Alternatively, conditions such as positional relationship information with other fingers R and finger shape information of the finger P so that the same finger is set as the operation target finger P when it appears on the screen again after disappearing from the screen May be stored in a storage unit in the control circuit 18 and the operation target finger P may be designated based on the stored information.

  A fourth modification of the first embodiment of the present invention will be described.

  As shown in FIGS. 20 and 21, the fourth modification of the first embodiment is different from the above embodiment in the method of specifying the operation target finger P. In the fourth modified example, as shown in the flowchart of FIG. 20, at T51, the touch operation position T on the touch operation surface 12a1 is detected by the finger specified at T1 of FIG. As shown in FIG. 21, at T52, the finger closest to the detected touch operation position T is specified as the operation target finger P (operation target finger specifying step). In this case, the highlighting at T4 in FIG. 6 may be started (immediately) as the touch operation on the touch operation surface 12a1 is performed, or the touch operation on the touch operation surface 12a1 may be performed. It may be started after a lapse of a predetermined time from being made. In particular, in the case of a configuration in which highlighting is not performed in stages, it is desirable to select the latter.

  In the third and fourth modified examples of the first embodiment, when the touch operation surface 12a1 is touched by mistake and the touch operation is performed, the highlight display can be prevented from being started. Specifically, as a touch operation in T41 to T51, a touch operation that has an operation burden larger than that of a normal touch operation (a touch operation in which an operation input is valid) can be determined as the highlight start operation. For example, as a touch operation for starting highlighting (highlighting start operation), a touch operation in which a pressing operation duration time until an input becomes valid is longer than a normal touch operation, or a press stronger than a normal touch operation It is possible to determine a touch operation by an operation force. In this case, in T41 to T51, the process proceeds from T42 to T52 only when a predetermined highlight display start operation is performed. When there is no highlight display start operation, the highlight display is not started until the highlight display start operation is performed. It can be set to a standby state.

  A fifth modification of the first embodiment of the present invention will be described.

  As shown in FIGS. 22 and 23, the fifth modification of the first embodiment is different from the above embodiment in the method of specifying the operation target finger P. In the fifth modification, as shown in the flowchart of FIG. 22, the luminance information of the photographed finger image is obtained as designated finger information at T61 (designated finger information obtaining step), and obtained at T62. On the basis of the brightness information, a finger suitable for a predetermined selection condition regarding the brightness information is preferentially specified as the operation target finger P among the captured finger images (operation target finger specifying step). Since the camera 12b captures the reflected light of the hand operating the touch operation surface 12a1, the fingertip closest to the touch operation surface 12a1 appears with higher brightness. That is, it can be said that the luminance information is the approach distance information (designated finger information) already described. On the other hand, since the fingertip that operates the touch operation surface 12a1 has a higher possibility of a position instruction as the finger is closer to the touch operation surface 12a1, for example, the finger with the highest brightness is designated as the operation target finger P with priority. Can be configured.

  As shown in FIG. 3, the camera 12b is shooting in the form of capturing the reflected light of the light source 12c. Therefore, the reflected light intensity is reflected in the gradation of the color appearing in the captured image 150, and the higher the reflected light intensity, the higher the gradation level. That is, since the captured video 150 captured by the camera 12b of this modification is a multi-gradation captured image (in this case, a monochrome captured image), the image processing unit 18a sets a predetermined gradation threshold value. Using this, the gradation level of each pixel is binarized, and the area exceeding the gradation threshold is extracted as the approaching object image 150H as shown in FIG. Further, in order to specify the image of the human hand / finger from the extracted approaching object image 150H, the image processing unit 18a first stores the shape of the approaching object image 150H and the storage unit of the control circuit 18. Compared to the hand and finger shape patterns, the approaching object image 150H that matches those patterns is recognized as a finger image. In this modification, the gradation threshold value for specifying an image of a hand or finger is set as a first gradation threshold value, and a second gradation threshold value higher than that is defined, and the second gradation value is determined. The fingertip finger having the region L1 (see FIG. 23) exceeding the threshold value can be designated as the operation target finger P. As another method, the operation target finger P may be designated by comparing the brightness of the fingertip Q or the average (or maximum / minimum) brightness of a predetermined region with the fingertip Q as a reference. Note that regions L1 to L3 in FIG. 23 indicate regions with different luminances in the captured video 150, and the luminances decrease in the order of L1, L2, and L3.

  That is, in the fifth modification of the first embodiment, T1 in FIG. 6 is a step of specifying a finger image based on the luminance of the image captured by the camera 12b and specifying the finger from the finger image. T2 in FIG. 6 is a step of specifying the fingertip using the hand image taken by the camera 12b. Furthermore, T3 in FIG. 6 is a step of specifying the operation target finger P based on the luminance of the hand image captured by the camera 12b. Therefore, in the fifth modified example of the first embodiment, it is possible to specify a finger, a fingertip, and even an operation target finger P from a captured image of the camera 12b.

  Here, it is assumed that the operation target finger P is designated by using image information of the fingertip portion instead of the entire finger. As a result, it is possible to limit the data that is the target of the operation target finger P specifying process, and the calculation cost (calculation amount / memory amount) can be reduced. Specifically, as shown in FIG. 23, a region S having a predetermined shape (in this case, a square) including the entire fingertip is set around the fingertip Q specified in T2, and all of the regions specified in T1 are specified. With respect to the finger, the luminance information in the region S is specified and acquired at T61, and the finger of the region S having the highest luminance among the acquired luminance information is preferentially designated as the operation target finger P at T62. ing.

  A sixth modification of the first embodiment of the present invention will be described.

  As shown in FIGS. 24 to 26, the sixth modification of the first embodiment is different from the above embodiment in the method of specifying the operation target finger P. In the sixth modification, as shown in the flowchart of FIG. 24, at T61, the approach distance information reflecting the approach distance relationship of the fingertip specified at T1 of FIG. 6 to the touch operation surface 12a1 is designated. It is acquired as information (designated finger information acquisition step), and is suitable for a predetermined selection condition regarding the approach distance among the fingers specified by the finger specifying step based on the acquired approach distance information at T62. A finger is preferentially designated as an operation target finger (operation target finger designation step). However, in the case of this configuration, means for detecting the distance between the touch operation surface 12a1 and the fingertip of the operating hand is required. Although the luminance can be acquired as the approach distance information as in the sixth modification, for example, as shown in FIG. 25, the camera (photographing means) 40 can be used as a means for acquiring the approach distance information. . The camera 40 is arranged so that the distance between the touch operation surface 12a1 and the fingertip of the operator's finger is reflected. From the captured image (for example, FIG. 26), the control circuit 18 causes the fingertip of each finger of the operator to touch the touch operation surface 12a1. The distances d1 to d4 are calculated, and for example, the finger with the closest fingertip can be designated as the operation target finger P. FIG. 26 is an example of a video image taken by the camera 40 of FIG.

A seventh modification of the first embodiment of the present invention will be described.
The operation target finger may be designated in a combination of a plurality of types of designated finger information such as the positional relationship information and finger shape information. As a result, it is possible to more accurately specify a predetermined type of finger such as an index finger or a middle finger.

An eighth modification of the first embodiment of the present invention will be described.
In all the embodiments and modifications described above, the camera 12b that forms the hand image photographing unit is a camera that photographs a fixed area in the vehicle interior facing the touch operation surface (front surface) 12a1 of the touch panel 12a. As in the embodiment and the modification, the touch panel 12a has a configuration for photographing from the back surface 12a2 side. However, the present invention is not limited to such a configuration. For example, as shown in FIG. 27, the touch operation surface 12a1 of the touch panel 12a is attached to the center console portion C of the vehicle so as to be photographed obliquely from above, and an approaching object approaching the touch operation surface 12a1 is displayed on the touch operation surface 12a1. You may comprise so that it may image | photograph from the side. In this case, unlike the above-described embodiment and the deformed collar, when a specific image (for example, a hand image) in the captured image is extracted and displayed on the display unit 15, it is possible to omit the process of horizontally inverting the image.

  Further, the camera 12b serving as the hand image photographing means may be used as the camera 20 for photographing the peripheral area including the touch operation surface 12a1 of the operation panel 12a.

  The camera 20 is connected to the control circuit 18 and can be provided, for example, in the periphery of the rear-view mirror at the upper front of the vehicle as shown in FIG. More specifically, for example, a photographed image 300 as shown in FIG. 28 is photographed, and the photographing range is determined so that not only the operator's hand of the operation panel 12a but also the arm following that hand is reflected. In the control circuit 18 that acquires the image of the operation panel surroundings to be photographed, the image processing unit 18a specifies the direction of the arm extending from the hand, and specifies whether the operator is the left or right passenger. . Accordingly, if the operator of the touch operation surface 12a1 is a person located on the left side of the touch operation surface 12a1, the operator can be identified as the right hand, and is located on the right side of the touch operation surface 12a1. If it is a person, the operation hand can be specified as the left hand.

  Then, the control circuit 18 can switch the designation condition of the operation target finger P based on the identification result of the operator. Specifically, when specifying the index finger, if the operating hand is the left hand, it can be specified as the finger that appears rightmost, or as the finger adjacent to the right of the longest middle finger, On the other hand, when the operating hand is the right hand, it can be specified as the finger that appears to the left most, or the finger that is adjacent to the left of the middle finger that appears the longest. Therefore, when selecting the operation target finger P, if the selection condition is switched depending on whether the operating hand is the left hand or the right hand, the switching is performed based on the identification result of the operating hand. be able to. Further, when the operator's hand is identified as the driver's hand, the display may not be performed so that the operation cannot be performed. In this way, since only the passenger on the left passenger seat can be designated as the operator, the right hand can be constantly determined as the operator, and this can be reflected in advance in the selection conditions for the operation target finger P.

  In the captured image 300 of FIG. 28, reference numeral 300H is a hand image, reference numeral 300A is an image of an arm that continues from the hand, reference numeral 312a is an area where the operation panel 12a is reflected, and the operator's arm of the operation panel 12a is the captured image. Since it extends from the right side, that is, from the driver's seat side, the operator is identified as the driver. In this modification, the camera 20 functions as an operator specifying unit together with the control circuit 18 (image processing unit 18a). It should be noted that another method may be used for specifying whether or not the operator of the operation panel 12a is a driver. For example, an approaching object detection unit (for example, a reflective optical sensor such as an infrared reflection sensor) that detects an approaching object may be provided on each of the left and right sides of the touch operation surface 12a to identify the operator.

A ninth modification of the first embodiment of the present invention will be described.
In all the above-described embodiments and modifications, the highlighting as described above may be performed only when two or more fingers are specified. Since the control circuit 18 functioning as the finger specifying means can specify the number of fingers, the position indicating image display process is executed when the specified number of fingers is two or more, and the number of fingers is one. Can be configured to execute a simple display process for displaying only the position indicating image corresponding to the finger.

A tenth modification of the first embodiment of the present invention will be described.
In all of the above-described embodiments and modifications, the remote operation unit is premised on a contact operation (push operation or touch operation), and thus designates one finger for position indication. Further, it is also described that in some cases, two fingers can be designated for position indication as shown in FIG. However, the remote operation unit may be an operation unit such as a dial operation unit, which is premised on the operation of pinching the operation knob. In this case, the number of fingers such as two or three is operated. It can also be designated as the target finger P. Note that the finger specification or fingertip specification in the pinch-type operation can be performed using the above-described approach distance information (information reflecting the approach distance relationship to the remote operation surface).

  For example, as shown in FIG. 30, a dial operation unit 12a ′ (the front end portion 121e around the dial knob is also part of the dial operation unit 12a ′) is provided and operated as a remote operation unit of the present invention. Therefore, the approaching hand is configured to photograph with the cameras 41 and 42 (configured to be connected to the control circuit 18). The cameras 41 and 42 have different shooting ranges, and are set so that both shooting directions are orthogonal to each other on a plane perpendicular to the rotational axis of the dial operation unit 12a '. An example of images taken by the cameras 41 and 42 (binarized images similar to FIGS. 4 and 5B) 150a and 150b are shown in FIGS. 31A and 31B. On the other hand, the camera 12b is also installed inside the operation information input unit 12 in the form shown in FIG. 3 (may be outside as shown in FIG. 27). An example of an image captured by the camera 12b (an image after binarization similar to (b) of FIGS. 4 and 5) 150c is shown in FIG. 31 (c). The image area 150S of these images 150a, 150b, and 150c is an area where the operation unit 12a 'is reflected.

  Here, the processing flow of the above-described highlighting performed in this configuration will be described.

  First, the images 150a and 150b shown in FIGS. 31A and 31B are acquired (processing corresponding to T1 in FIG. 6), and the fingertip Q is specified from the hand image 150H extracted from the images 150a and 150b. Then, the specific position of the fingertip Q is specified (a process corresponding to T2 in FIG. 6). Here, a three-dimensional coordinate system (vehicle interior coordinate system) having an x-axis, a y-axis, and a z-axis as shown in FIG. Specifically, an xz coordinate system and a yz coordinate system are defined for the images 150a and 150b, and any one of those already described in the tip region of the protruding image 150F extending from these to the housing front end 121e side. The fingertip Q is specified by the method, the position coordinates of the fingertip Q are specified, and the three-dimensional position coordinates in the vehicle interior coordinate system are specified from these.

  Subsequently, the operation target finger P is specified. Here, the distance from each fingertip Q to the front end surface of the housing front end portion 121e is calculated, and the fingertip Q that is closest to the front end surface of the housing front end portion 121e, and Q that is next closest to the fingertip Q are calculated. Both are specified as the operation target finger P. Note that the calculation of the distance here may be performed using the image 150a, 150b because the front end surface of the housing front end 121e is reflected (reference numeral 150E), or may be calculated from the luminance of the image 150c. Also good.

  Then, the position indicating area 200Q (200P) of the position indicating image 200H corresponding to the specified fingertip Q of the operation target finger P is more than the position indicating area 200Q (200R) of the position indicating image 200H corresponding to the other fingertip Q. In an emphasized form, it is displayed in a superimposed manner on the background image 200B as shown in FIG. Note that the background image 200B (200B3) in FIG. 32 is a dial switch image displayed as the switch image 200I, and is rotated according to the rotation operation of the dial operation unit 12a ′ forming the remote operation unit, and the rotation position is displayed. The control content corresponding to is executed.

  Here, the operation target finger P is specified by pointing the identified fingertip Q toward the dial operation unit 12a ′ serving as a remote operation unit as in the first modification of the first embodiment described above. Distance information reflecting the distance between the reference area Y projected to the predetermined area on the projection plane 180 and the projected fingertip Q is projected (in the direction facing the dial operation unit 12a ′). It is acquired as positional relationship information, and using the acquired distance information, a finger suitable for a predetermined selection condition regarding the distance is preferentially designated as the operation target finger P. Note that the reference area Y set here can be set as the center position of the projection area 120 obtained by projecting the dial operation unit 12a 'onto the projection plane 180, as shown in FIG.

  In addition, the method of designating the operation target finger P here may be a method that is determined so that at least the thumb is included. Specifically, it can be identified from various factors such as a difference in thickness with other fingers, a difference in positional relationship with other fingers, and a difference in extension direction of the fingers. When there are two operation target fingers P, the remaining one finger can be the finger closest to the front end surface of the housing front end 121e other than the thumb. However, it is desirable that the index finger or the middle finger is designated for ease of knob operation. For example, the index finger or the middle finger can be designated based on the thumb (for example, based on the positional relationship from the thumb). In the case of a remote control unit that assumes the above-described knob operation, the fingertips of three fingers may be emphasized as shown in FIG. In this case, it is desirable that the thumb, index finger, and middle finger are designated as the operation target finger P.

  In the embodiment and the modification described above, the fingertip area 200Q in the position instruction image 200H is used as the position instruction area. However, as in the other embodiments described above, the entire finger included in the position instruction image 200H is position-indicated. It may be an area. On the contrary, in other embodiments already described, the fingertip area 200Q in the position indicating image 200H may be used as the position indicating area.

  A finger or fingertip tracking process that can be applied to all of the above-described embodiments and modifications will be described.

  In the embodiment and the modification described above, the operation target finger P is specified for each frame of the captured video. In this case, the positional relationship between the operation panel 12a (12a ′) and the operating hand is changed. The operation target finger P may automatically switch to another finger. In particular, as in the third and fourth modifications, when the operation target finger P is specified based on the contact position (user operation) to the remote operation unit, the operation target finger P is based on the touch operation. Therefore, it is not desirable that another finger is easily set as the operation target finger P simply because the position of the hand or finger is moved. In addition to the first embodiment, a finger state in which a predetermined finger type is specified as a predetermined finger state (such as an operation intention display finger state or a simple operation finger state to be described later). (For example, a finger state in which one or both of the index finger and the middle finger is specified) is also specified because the position of the hand or finger is moved after the finger of that type is specified. It is not desirable that a finger that has been easily identified as another type of finger. For this reason, after a finger such as the operation target finger P is designated or specified, a process for tracking the movement of the designated or specified finger is added, and the same finger is continuously designated based on the tracking result. By continuing identification, the above problem can be solved.

  36 and 37 are processes for tracking all the fingers specified by T1 in FIG. 6, more specifically, a process for tracking all the fingertips specified by T2. Here, in order to operate the operation panel 12a (12a ′) forming the remote operation unit, the hand (finger) facing the front side of the operation panel 12a (12a ′) is photographed by the camera 12b as the photographing means. It is configured to specify based on a moving image. In this case, the finger identifiable range (fingertip identifiable range) of the hand (finger) facing the front side of the operation panel 12a (12a ') is a photographing range of the camera 12b that is fixedly arranged. The camera 12b captures a moving image within the range, and the control circuit 18 acquires moving image frames that are sequentially captured at predetermined time intervals.

  In the finger tracking process (finger tracking step) in FIGS. 36 and 37, the position of each fingertip identified from the sequentially acquired moving image frames is calculated in the frame, and the calculated position information is used as tracking information. By storing and accumulating in a predetermined tracking information storage unit such as a ROM of the control circuit 18, each movement is tracked while identifying each fingertip identified from the sequentially acquired moving image frames.

The tracking information is stored in a form as indicated by reference numeral 500 in FIG. In the tracking information 500 shown in FIG. 38, a fingertip ID (identification information) is assigned to each fingertip specified in the acquired moving image frame, and the previous moving image of each fingertip is associated with the fingertip ID. Position information indicating the position in the frame (hereinafter referred to as the previous frame) and the previous moving image frame (hereinafter referred to as the previous frame) can be stored. The position information here is the position coordinates of each fingertip on a predetermined two-dimensional coordinate system defined in the moving image frame. The upper diagram of FIG. 38 illustrates a moving image frame (also referred to as a finger identifiable range) in which the two-dimensional coordinate system is defined. In the previous frame stored as tracking information 500 in FIG. The specified fingertip position coordinates P _i (-1), the fingertip position coordinates P _i (-2) specified in the previous frame, and the fingertip R _j (0) specified in the current frame are displayed. Has been. As shown in FIG. 38, the position coordinates of each fingertip are sequentially stored and updated as new moving image frames are acquired. The position coordinates stored and held here are the position coordinates of the previous frame and the previous frame, and the position coordinates of the frames before the previous frame are sequentially deleted.

  The tracking information 500 includes position information indicating the position of each fingertip on the frame specified from the latest moving image frame (hereinafter referred to as the current frame). The position information is stored for each fingertip ID. Furthermore, the tracking information includes a predicted position (next position) where each fingertip will be positioned in the current frame, and the tracking information storage unit stores predicted position information indicating the predicted position. Further, the correspondence relationship between the fingertip position information in the current frame and the predicted fingertip position information predicted for the current frame is specified, and the corresponding fingertip tracking at the time of acquiring the current frame is determined for each fingertip ID from the corresponding result. The situation is specified, and this is also stored as tracking information in the tracking information storage unit. The tracking status includes the tracking continuation state in which tracking of the fingertip continues from the time of the previous frame to the current frame (tracking), and the tracking of the fingertip that has been tracked up to the previous frame is lost (disappears) in the current frame. It has at least three states: a state (lost) and a new appearance state (new appearance) in which a fingertip that has not been tracked up to the previous frame has newly appeared.

As a calculation method of the predicted position of the fingertip, for example, the following calculation method can be exemplified. That is, it is a method of calculating based on position information indicating the position of each fingertip on a plurality of past moving image frames. Here, as the tracking information of each finger, the position information of each fingertip on the previous frame times and the previous frame is included. Therefore, based on the position information, the movement of each fingertip is estimated, and on the next frame. The estimated movement destination of each fingertip is calculated as the predicted position. As a specific example, as shown in FIG. 40 (a), in the two-dimensional coordinate system on the moving image frame 400 defined in common on each moving image frame, the finger position coordinates P specified on the previous frame are determined. When moving from _i (-2) to the position coordinate P _i (-1) specified on the previous frame, the finger velocity vector V _i (-1) at the position coordinate P _i (-1) is Calculation is performed based on the moving direction and moving distance of the movement. In the next frame, it is estimated that the finger located at the position coordinate P _i (−1) moves by the velocity vector V _i (−1), and the position coordinate Q _i (−0) of the estimated movement destination is calculated (Q _i (−0) = P _i (−1) + V _i (−1)). The calculated position coordinate Q _i (−0) is the predicted position information, and is stored in the tracking information storage unit of the control circuit 18 as one piece of tracking information.

However, predicting the appearance of the next frame with respect to a fingertip that has newly appeared in the previous frame cannot be performed by the prediction method using both the previous frame and the previous frame as described above. In such a case, here, based on the position information of the fingertip specified on the previous frame, the position of the fingertip in the previous frame is determined as a temporary position, the position information indicating the temporary position, and the position information of the previous frame Based on the above, the appearance position of the fingertip on the next frame is specified. The method for determining the temporary position is as follows. That is, as shown in FIG. 41, areas in the moving image frame (fingertip identifiable range) 400 are partitioned in advance in association with each edge (here, the top, bottom, left, and right edges) (partition lines are An edge corresponding area 410 corresponding to each edge 401 (401a to 401d) is set, and then, first, the fingertip position P _i (−1) specified on the previous frame is set. The edge corresponding region 410 (401A to 401D) in which is present is specified. When the edge corresponding region 410 where P _i (−1) exists is specified, the position on the edge 401 of the edge corresponding region 410 closest to the P _i (−1) Is determined as a temporary position. That is, a perpendicular line that passes through the fingertip position P _i (−1) specified on the previous frame and goes to the edge 401 of the edge corresponding region 410 where the fingertip position P _i (−1) exists, and the edge The intersection with the part 401 is defined as a temporary position P _i ′ (−2).

The calculation method of the predicted position of the fingertip is a method of calculating based on the position information of each fingertip on the previous frame times and the previous frame, but in order to calculate with higher accuracy, FIG. Calculate as shown in b). That is, in the two-dimensional coordinate system on the moving image frame 400 defined in common on each moving image frame, the finger velocity vector on the position coordinates P _i (−1) specified on the previous frame once from the current frame. V _i (-1) and acceleration vector A _i (-1) are calculated, respectively, and in the next frame, the finger located at the position coordinate P _i (-1) is the velocity vector V _i (-1) and acceleration vector. It is estimated that the movement corresponding to A _i (-1) occurs, and the position coordinate Q _i (-0) of the estimated movement destination is calculated (Q _i (-0) = P _i (-1) + V _i (0) = P _i (−1) + V _i (−1) + A _i (−1)). First, the position coordinate P _i (−3) specified on the frame three times before the current frame is moved to the position coordinate P _i (−2) specified on the second frame from the current frame. based on the moving direction and moving distance of time, it calculates the position coordinates P _i of the finger in (-2) velocity vector V _i (-2). Then, this finger is specified on the previous frame once from the current frame from the previous position coordinate moved from the position coordinate P _i (−2) on the moving image frame 400 by the velocity vector V _i (−2). The acceleration vector A _i (−1) is calculated based on the moving direction and moving distance when moving to the position coordinate P _i (−1) (A _i (−1) = V _i (−1) −V _i (−2) = P _i (−1) −P _i (−2) −V _i (−2)). Also, the velocity vector V _i (−1) is obtained from the position coordinates P _i (−2) specified on the previous frame twice from the current frame, and the position coordinates P _i specified on the previous frame once from the current frame. Calculate based on the moving direction and moving distance when moving to (-1).

As a method for specifying the fingertip tracking status, for example, the following specifying method can be exemplified. That is, the fingertip position information specified on the current frame is compared with the predicted position information of the fingertip on the current frame, and based on whether or not these positional relationships satisfy a predetermined reference condition. It is a method to specify. For example, as shown in FIG. 38, the position coordinates Q ₁ (-0), Q ₂ (-0), Q ₃ (-0) of the fingertip indicated by each predicted position information calculated for the current frame, A pair corresponding to a pair closest to the position in the moving image frame from among the position coordinates R _a (0), R _b (0), R _c (0),. Are recognized as a pair satisfying the reference condition. However, in the case where the distance between both positions forming a pair is not more than a predetermined reference distance, the pair is not recognized as a corresponding pair. For the corresponding pair, it is determined that the fingertip actually identified in the current frame and the fingertip corresponding to the position information of the previous frame that is the basis for calculating the predicted position information are the same fingertip, The tracking of the fingertip is identified as a tracking continuation state (tracking) that continues from the previous frame to the current frame. Of the fingertips actually specified in the current frame, the fingertip for which no corresponding pair is found is specified as a new appearance state (new appearance). Of the fingertips corresponding to the predicted position information calculated for the current frame (fingertips identified as being tracked in the previous frame), the fingertips for which no corresponding pair was found are identified as a non-trackable state (lost). The identified tracking status of each fingertip is stored in the tracking information storage unit of the control circuit 18 in a form associated with each fingertip ID.

  In addition, when the untraceable state (lost) is specified as the tracking state, together with the untrackable state (tracking state) and the position information of the fingertip on the frame immediately before the untrackable state is specified, The untrackable recognition order in which the context of the untrackable specific time with other untrackable fingers can be specified is stored as tracking return determination information in the tracking information storage unit (tracking return determination information storage unit). Here, the tracking return determination information is also handled as one of the tracking information.

  Here, the flow of the finger tracking process (finger tracking step) shown in FIGS. 36 and 37 will be described. This finger tracking process is a process that is executed each time the control circuit 18 acquires a moving image frame from the camera 12b, whereby the control circuit 18 functions as a finger tracking unit.

  First, in R201 (FIG. 36), the control circuit 18 determines whether or not the fingertip is specified in a new moving image frame (current frame) acquired by moving image shooting by the camera 12b, and the fingertip is specified. If YES, the process proceeds to R202. If the fingertip is not specified, the process proceeds to R215. The fingertip specifying method is the same as T1 and T2 in FIG.

  In R202, the control circuit 18 specifies the correspondence between each fingertip specified on the current frame and each fingertip specified on the previous frame. Here, the correspondence between the fingertip position information in the current frame and the predicted fingertip position information predicted for the current frame is identified, and the tracking status of each fingertip at the time of acquiring the current frame is tracked from the corresponding result. For each fingertip ID, the state (tracking), untrackable state (lost), new appearance state (new appearance), or unspecified (not appearing) is specified. The identified tracking status is stored in the tracking information storage unit of the control circuit 18 for each corresponding fingertip ID in a form as indicated by reference numeral 500 in FIG. Here, when the fingertip specified on the previous frame and the fingertip specified on the current frame satisfying the above criteria are recognized as a tracking continuation state (tracking), it is specified on the previous frame. If any of the fingertips does not satisfy the above-mentioned standard condition, the fingertip specified on the current frame is recognized as a new appearance state (new appearance), and any of the fingertips specified on the current frame If there is a fingertip specified on the previous frame that does not satisfy the above, it is recognized as a non-trackable state (lost).

  In R203, whether or not the same finger exists on the fingertip specified on the previous frame and the fingertip specified on the current frame, in other words, the fingertip in the tracking continuation state (tracking) exists on the current frame. It is determined whether or not to do. When the fingertip in the tracking continuation state (tracking) exists, the process proceeds to R204, and when the fingertip in the tracking continuation state (tracking) does not exist, the process proceeds to R205 in FIG.

  In R204, the tracking information corresponding to the fingertip ID of the fingertip in the tracking continuation state (tracking) is updated. The fingertip in the tracking continuation state (tracking) is the fingertip whose tracking information has been updated when the previous frame is acquired, and the position information of the current frame corresponding to the fingertip ID of the fingertip indicates the position of the fingertip in the current frame. Update with location information. Further, the position information of the current frame stored so far is updated as the position information of the previous frame, and the position information of the previous frame stored so far is updated as the position information of the previous frame.

In R205, whether there is a fingertip not specified on the current frame among the fingertips specified on the previous frame, in other words, tracking in the current frame, such as P ₁ (0) shown in FIG. It is determined whether there is a disabled fingertip. If there is a fingertip in the tracking disabled state (tracking), the process proceeds to R206, and if there is no fingertip in the tracking continuation state (tracking), the process proceeds to R207.

  In R206, information related to the finger when it is recognized that tracking is impossible is stored as tracking return determination information in association with the fingertip ID of the fingertip in the tracking disabled state (lost). This tracking return determination information is used to determine whether or not a finger that is not tracked here is recognized when a new fingertip that is not being tracked appears in a moving image frame acquired later. Information. As a result, when a fingertip once out of the frame enters the frame again, both can be recognized as the same finger, and the finger can be tracked again. Here, the fingertip ID of the fingertip that has become untrackable is stored and retained without being erased, and is associated with the fingertip ID along with the position information (position immediately before disappearance) of the previous frame corresponding to the fingertip ID. The untraceable recognition order (disappearance order) in which the context of the untrackable recognition time with the untrackable finger can be specified is stored as tracking return determination information. In the case of FIG. 38, “1” is assigned to the fingertip ID “2” as the order of disappearance, and when a new fingertip cannot be tracked in a moving image frame acquired later, “2” is assigned as the order of disappearance, and it can be determined that the higher the number assigned, the newer the untraceable recognition time.

  In R207, it is determined whether there is a fingertip not specified on the previous frame among the fingertips specified on the current frame, in other words, whether there is a fingertip newly appearing on the current frame. . If there is a fingertip in a new appearance state (new appearance), the process proceeds to R208. If there is no fingertip in a tracking continuation state, the process proceeds to R209.

  In R208, when it is specified in R207 that there is a fingertip in a new appearance state (new appearance), it is determined whether or not there is a fingertip that is currently in a tracking impossible state (lost). This determination is performed based on whether or not the fingertip ID storing the tracking return determination information exists in the tracking information shown in FIG. At this stage, if it is determined that there is no fingertip that continues in the untrackable state (lost), the process proceeds to R213, and a new appearance state that has not been assigned to the fingertip in the new appearance state specified in R207. The position information indicating the position specified on the current frame of the fingertip in the newly appearing state is stored in a form in which the fingertip ID is assigned and associated with the fingertip ID. Further, the tracking status corresponding to the fingertip ID is changed from the non-trackable state to the tracking continuation state (tracking restart), and the tracking return determination information that has been continuously stored is erased. When R213 is terminated, the present process is terminated. On the other hand, if it is determined in R208 that there is a fingertip that continues to be untrackable (lost), the process proceeds to R209.

  In R209, it is specified whether or not there is a finger that is the same as the fingertip that is currently in the untrackable state (lost) among the newly appearing fingertips. Specifically, the position information in the current frame of the newly appearing fingertip actually identified in R207 and the position information (tracking return determination information) indicating the last position of the untrackable fingertip included in the tracking information 500 Based on the above, it is determined whether or not there is a fingertip that is closest to the position of the newly identified fingertip that is actually identified among the fingertips that are in an untraceable state within a certain range. Here, the distance between the fingertip position of each fingertip that is actually specified in the current frame and the fingertip position of each fingertip that is in an untrackable state is calculated, and the shortest of these distances is calculated. In addition, by identifying a distance that is less than a predetermined reference distance, each fingertip that is actually identified in the current frame corresponding to the distance and each fingertip that is in a non-trackable state are the same fingertip judge. However, if there are multiple fingertips that are in a non-trackable state within a certain range, the fingertip that has been in the most untrackable state based on the untrackable recognition order (tracking return judgment information) Identify the same as the fingertip identified as the new application in the current frame.

  In the subsequent R210, based on the result of R209, it is determined whether or not there is the same fingertip that has been in a trackable state (lost) among the fingertips identified as new applications in the current frame. If it exists, the process proceeds to R211. Otherwise, the process proceeds to R213.

  In R211, based on the result of R209, it is determined whether or not there are a plurality of fingertips that are determined to be the same as the fingertips that continue to be untrackable (lost) among the fingertips identified as new applications in the current frame. To do. If it is determined that there is one instead of a plurality, the process proceeds to R212. In R212, the position information of the fingertip of the new application identified on the current frame, determined to be the same as the fingertip, is stored in association with the fingertip ID in the non-trackable state (lost) of the one fingertip, Update the tracking information of the fingertip ID. As a result, the tracking of the position of the fingertip that cannot be tracked is resumed. At this time, the tracking status corresponding to the fingertip ID is changed from the non-trackable state to the tracking continuation state (tracking restart), and the tracking return determination information that has been continuously stored is erased. When R212 is terminated, this process is terminated.

  On the other hand, in R211, based on the result of R209, among the fingertips identified as new applications in the current frame, it is determined that there are a plurality of fingertips that are determined to be the same as the fingertips that are in a non-trackable state (lost). If yes, go to R214. In R214, among the plurality of fingertips, based on the untrackable recognition order (tracking return determination information), the fingertip that has been most recently untrackable is identified and associated with the fingertip ID of the identified fingertip, The position information of the fingertip of the new application identified on the current frame determined to be the same as the fingertip is stored, and the tracking information of the fingertip ID is updated. As a result, the tracking of the position of the fingertip that cannot be tracked is resumed. At this time, the tracking status corresponding to the fingertip ID is changed from the non-trackable state to the tracking continuation state (tracking restart), and the tracking return determination information that has been continuously stored is erased. When R214 ends, this process ends.

  Note that in R201, when the fingertip is not specified in the acquired new moving image frame, the tracking information 500 as shown in FIG. 38 is reset, and the information corresponding to each ID is deleted. .

  In this way, by continuously tracking all the fingertips specified from the moving image photographed by the camera 12b in consideration of frame out, the operation target finger P can be fixed to the same finger. . Specifically, when the operation target finger P is specified by the process of specifying the operation target finger P, as shown in FIG. 38, the operation target finger information indicating the operation target finger P is stored in association with the fingertip ID. By referring to this, the operation target finger P can be specified at any timing. That is, by adding the finger tracking process as shown in FIG. 36 and FIG. 37, the process of specifying the operation target finger P by T3 is performed separately in the position instruction image display process of FIG. In T3, the operation target finger P may be specified based on the operation target finger information stored in association with the fingertip ID.

  Further, the application of the fingertip tracking process is performed when the operation target finger P is operated based on the operation content of the operation target finger designation operation performed by the user as in the third modification example in FIG. 19 or the fourth modification example in FIG. Suitable for the specified embodiment. In the third modification of FIG. 19 and the fourth modification of FIG. 20, the touch operation on the touch panel 12a is an operation target finger designation operation, and the operation target finger P is designated based on the operation content of the touch operation. . That is, in the embodiment in which the operation target finger P is not changed unless the operation target finger designation operation is performed by the user, it is not necessary to perform an operation for specifying the operation target finger P every time a moving image frame is acquired. By simply picking up a finger to be designated from the tracking information, the operation target finger P can be easily identified.

  Hereinafter, a second embodiment of the present invention will be described.

  In the above embodiment, since the remote operation unit is provided at the user's hand, the operation unit is unintentionally touched and an erroneous operation input is a problem. In particular, in the case of a touch operation type operation unit, there is a case where an input is made even if a contact is made with some force. In order to solve this problem, when a predetermined finger state that is predetermined as an operator's finger state is specified, the position indication image 200H in the first embodiment is highlighted only during a predetermined display permission period. On the other hand, in other cases, as shown in FIG. 54, even if the hand is in a state of facing the touch operation surface 12a1, the position instruction image 200H is not displayed. Specifically, a position instruction image display process as shown in FIG. 52 is performed. This processing is also performed in such a manner that the CPU executes a program stored in the storage unit of the control circuit 18.

  In S1, the control circuit 18 determines whether an image of the operator's hand has been acquired based on the captured image captured by the camera 12b. The camera 12b always shoots an approaching object (for example, an operator's hand such as a driver) H that approaches the touch operation surface (front surface) 12a1 side from the back surface 12a2 side of the operation panel 12a via the operation panel 12a. The captured image is always input to the image processing unit 18a of the control circuit 18. When the approaching object H enters the position facing the touch operation surface 12a1, the captured image 150 including the approaching object H is naturally input to the image processing unit 18a. The input captured image 150 is analyzed by a known image analysis method in the image processing unit 18a, and an approaching object image 150H is extracted based on a color difference or the like. Then, from the shape of the extracted approaching object image 150H, it is determined whether or not the image 150H is an image of a human hand. The determination result is output to the control circuit 18, and if it is determined that the hand image has been recognized, the process proceeds to S2, and if not determined, the present process is terminated.

  In the present embodiment, the light source 12c is arranged, and the camera 12b performs imaging in a manner that captures the reflected light of the light source 12c. Therefore, the reflected light intensity is reflected in the gradation of the color appearing in the captured image 150, and the higher the reflected light intensity, the higher the gradation level. That is, since the captured video 150 captured by the camera 12b of the present embodiment is a multi-gradation captured image (monochrome captured image in the present embodiment), the image processing unit 18a converts the captured image 150 into a predetermined gradation. The gradation level of each pixel is binarized using the threshold value, and the area exceeding the gradation threshold value is extracted as the approaching object image 150H as shown in FIGS. 4, 5, 51, and 54B. To do. Further, in the image processing unit 18a, in order to specify an image of a human hand from the extracted approaching object image 150H, the shape of the approaching object image 150H and the hand shape stored in the storage unit of the control circuit 18 are used. Compared with patterns, the approaching object image 150H that matches these patterns is recognized as a hand image (hand image recognition means).

  In S2, based on the acquired hand image, it is specified whether or not the hand is in a predetermined finger state (predetermined finger state specifying means). Here, it is specified whether or not it is a predetermined operation intention display finger state (operation intention display finger state specifying means). The operation intention display finger state is a simple operation finger state in the present embodiment. For example, one or two fingers that can be an operation target, that is, fingers that extend substantially along the touch operation surface 12a. The state is a minority of the degree. In other words, an operation using five fingers to speed up input is a highly difficult finger state, and the simple operation finger state here is different from such a finger state. In the present embodiment, only when the acquired hand image is in a one-finger operation state in which only one finger appears in the surface direction (the direction in which the surface expands) of the touch operation surface 12a. It is specified as an operation intention display finger state. More specifically, in S1, the image processing unit 18a that has extracted the hand image identifies a substantially linear rod-shaped image pattern having an axial line longer than a predetermined length from the overall shape of the hand image. When there is the stick-like image pattern, it is recognized and specified as a finger image 150F (finger specifying means). The specified result is output to the control circuit 18, and the control circuit 18 specifies the number of the finger images 150F (number of finger specifying means: hand shape recognition means), and the case where the number is one is predetermined. The operation state display finger state (operation intention display finger shape) is specified.

  In subsequent S3, based on the output result, it is determined whether or not the current finger state is a predetermined finger state (operation intention display finger state here), and the predetermined finger state is determined in advance. If it is determined that there is, the process proceeds to S4. In S4, the display permission mode in which the display of the position indication image is permitted is set as the display mode, and the process proceeds to S5. The set display mode is stored in a predetermined storage area (display mode storage unit) provided in the storage unit of the control circuit 18. In S4, the display permission mode is stored in the storage area.

  In S5, the control circuit 18 specifies the positional relationship between the display screen of the display device 15 and the photographed hand. That is, the position on the display screen where the photographed hand image H faces is specified. In the present embodiment, the operation panel image area where the touch operation surface 12a is located is specified on the captured image captured by the camera 12b, and the operation coordinate system of the touch operation surface 12a is set on the operation panel image area. Then, the position of the captured hand image H is specified on the set operation coordinate system. Then, in the display coordinate system defined on the display screen of the display device 15, a position corresponding to the position of the hand image H specified on the operation coordinate system is specified. In the present embodiment, the captured image (binarized image) 150 shown in FIG. 4, FIG. 5, FIG. 51, and FIG. 54B is an operation panel image region obtained by capturing the entire touch operation surface 12 a 1. Yes.

  Further, in S6, as shown in FIGS. 5 and 51 (c), a position indicating image is displayed at a position corresponding to the hand image H specified on the operation panel image area on the display screen of the display device 15. 200H is displayed (position indication image display means). In the present embodiment, a processed image 200H in which the outer shape of the hand image 150H is clearly displayed and the inside of the contour is translucent is generated, and the main image 200B displayed on the screen of the display device 15 is generated. Are overlaid at the corresponding coordinate positions. Further, in S6, the position indication image highlight display process (FIG. 6) in the first embodiment is performed, and the finger position indication image 200H specified as the operation target finger P is emphasized relatively to the remaining fingers. Is displayed.

  Here, it is assumed that the operation target finger P to be emphasized is defined as an index finger. Therefore, even if the finger is in a single finger state, if the finger extending at that time is not an index finger, the position pointing image corresponding to this finger is displayed in a non-emphasized state. On the other hand, if the index finger is the index finger, the index finger is displayed in the emphasized state as shown in FIG. 5 and FIG. 51C, but the non-emphasized finger is not displayed.

  On the other hand, if it is determined in S3 that the current finger state is not a predetermined finger state (in this case, an operation intention display finger state), the process proceeds to S7. In S7, the display mode is set to a display prohibition mode in which the display of the position indication image is prohibited. Accordingly, the position instruction image 200H is not displayed in S5 and S6, and the position instruction image 200H is not displayed on the display screen of the display device 15, for example, as shown in FIG. 54 (c). The set display prohibition mode is stored in a predetermined storage area (display mode storage unit) provided in the storage unit of the control circuit 18.

  This process is ended by the end of S6 and S7. This process is repeatedly performed at a predetermined cycle after the end, and when a hand in a predetermined finger state that faces the operation panel 12a moves, the position displayed on the display screen of the display device 15 is adjusted accordingly. The display position of the instruction image (processed image) 200H is also moved.

  In this position instruction image display process, as shown in FIG. 57, a period until a predetermined predetermined finger state becomes unspecified is determined as a display permission period. That is, the processing is such that the display permission mode is set when a predetermined finger state is specified in advance, and the display prohibition mode is set when a predetermined finger state is not specified. ing. The control circuit 18 functions as the display mode setting means of the present invention by executing the display process of the position indication image.

  In the operation input device of the conventional method, when the hand to be remotely operated is actually displayed on the operation screen, when the hand is close to the touch panel 12a (remote operation unit), only the finger is displayed. Some users perform touch operation input with five fingers of one hand for quick input. In this case, many fingers are displayed side by side on the operation screen. There is a problem in that the correspondence between the finger that is made and the finger that the operator actually wants to use for the instruction cannot be understood, and an erroneous operation is likely to occur. On the other hand, in the above embodiment, the touch operation input to the operation panel 12a is not accepted unless the predetermined finger state is determined in advance, so that the above-described problems can be solved. .

  Hereinafter, a third embodiment of the present invention will be described.

  In the second embodiment described above, the touch operation input process to the operation panel (remote operation unit) 12a of the operation information input unit 12 can be a process as shown in FIG. 53 instead of the process as shown in FIG. . This processing is also performed in such a manner that the CPU executes a program stored in the storage unit of the control circuit 18.

  In the touch operation input process of FIG. 53, when the display permission mode is set, the acceptance of the touch operation input to the operation panel 12a is permitted (the control circuit 18 sets the input permission mode), and the display prohibit mode is set. If it is set, the process includes operation input reception control for prohibiting reception of the touch operation input (the control circuit 18 sets the input prohibit mode). The control circuit 18 functions as an operation input control means of the present invention by executing this touch operation input process.

  In S101, the control circuit 18 determines whether or not there is a touch operation on the operation panel 12a. Since the operation panel 12a is configured as a touch panel, when a touch operation is performed, an operation signal is input to the control circuit 18, and thus determination is made based on the presence or absence of the input. If there is a touch operation on the operation panel 12a, the process proceeds to S102, and if there is no touch operation, this process ends.

  In S102, the control circuit 18 determines whether or not the current display mode is the display permission mode. Since the current display mode is stored in a predetermined storage area (display mode storage unit) provided in the storage unit of the control circuit 18, the current display mode is determined based on the stored display mode information. When it is determined that the current display mode is the display permission mode, the process proceeds to S103, and when it is determined that the current display mode is not the display permission mode, that is, when it is determined that the display prohibition mode is set, this process ends.

  In S103, the touch operation position on the operation panel 12a, that is, the touch operation position on the touch operation surface 12a1 is specified. Since the operation panel 12 a is configured as a touch panel, an operation signal is input to the control circuit 18 when a touch operation is performed. Since the touch operation position is reflected in the operation signal as coordinate information on the operation coordinate system defined on the operation surface 12a1, the control circuit 18 specifies the touch operation position based on the operation signal. .

  In subsequent S104, the control circuit 18 performs control corresponding to the touch operation position on the operation panel 12a. Specifically, since each position on the touch operation surface 12a1 of the operation panel 12a corresponds to a position on the display screen of the display screen 15, the position on the touch operation surface 12a1 specified in S103. The corresponding control content determined at the position on the display screen of the display screen 15 is executed. Specifically, the control circuit 18 outputs a control signal for performing the control content. For example, if the position on the touch operation surface 12a1 specified in S103 corresponds to the position of the switch image 200I displayed on the display screen of the display screen 15, the control circuit 18 is assigned to the switch image. A control signal for executing the control content is output.

  The process ends when S104 ends. Note that this process is repeatedly performed at a predetermined cycle even after the completion of the process, and the control circuit 18 performs the control content corresponding to the position touched at that time or is performed by another control unit (for example, the ECU 100 or the like). Let

  In this touch operation input process, when the display permission mode is set, the acceptance of the touch operation input to the operation panel 12a is permitted (input permission mode), and when the display prohibit mode is set. The reception of the touch operation input is prohibited (input prohibition mode). However, in the position instruction image display process of FIG. 52, the display permission mode is set when a predetermined finger state is specified in advance, and the display is prohibited when the predetermined finger state is not specified. Since the mode is set, as a result, the touch operation input to the operation panel 12a cannot be accepted unless the user's finger state is a predetermined finger state. Here, a simple finger state for simple operation, which is a single finger state, is a predetermined finger state (in this case, a finger state for operation intention display).

  Hereinafter, a fourth embodiment of the present invention will be described.

  In the second and third embodiments, as the display permission period, as shown in FIG. 57, a period until a predetermined predetermined finger state is unspecified is determined. As the display permission period, a predetermined time can be determined as shown in FIG. That is, a period from when switching to the display permission mode until a predetermined time (prohibition mode return time) elapses is defined as a display permission period, and the display prohibition mode is restored as the prohibition mode return time elapses. Can be configured. According to this configuration, a period during which the position indication image can be displayed (display permission mode) is triggered when the user indicates a specific finger state, and the touch operation surface 12a1 while viewing the position indication image is displayed. While the position instruction input (touch operation) is possible, the position instruction image is not displayed unless the user indicates a specific finger state, and the touch operation for performing the position instruction input cannot be performed smoothly. In other words, the display of the position indication image is permitted only when the user displays a clear intention to input the position instruction, and the intention display is confirmed by specifying the user's finger state. ing. Here, when the display permission mode is set, it is also possible to display an image of a finger state different from a predetermined finger state determined in advance. More specifically, the position instruction image display process shown in FIG. 52 can be realized by the process shown in FIG.

  In S10, the control circuit 18 determines whether or not an image of the operator's hand has been acquired based on the captured image captured by the camera 12b. This process is the same as the process of S1 in FIG.

  In S11, the control circuit 18 determines whether or not the current display mode is the display permission mode. Since the current display mode is stored in a predetermined storage area (display mode storage unit) provided in the storage unit of the control circuit 18, the current display mode is determined based on the stored display mode information. If it is determined that the current display mode is the display prohibition mode, the process proceeds to S12. If it is determined that the current display mode is not the display permission mode, that is, if it is determined to be the display prohibition mode, the process proceeds to S19.

  In S12, based on the acquired image of the hand, it is specified whether or not the state of the hand is a predetermined finger state (in this case, a finger state for operation intention display) (predetermined finger state specifying means: operation). Intention indication finger state identification means). This process is the same as the process of S2 in FIG. In subsequent S13, if it is determined that the current finger state is a predetermined predetermined finger state, the process proceeds to S14.

  In S14, the display mode is set to a display permission mode in which the display of the position indication image is permitted, and the process proceeds to S15. The set display permission mode is stored in a predetermined storage area (display mode storage unit) of the control circuit 18.

  In continuing S15, the timer 18b provided in the control circuit 18 is started, time count is started, and it progresses to S16. The time is stored in a form counted up by the counter 18c. Before the timer is started, the count value of the counter 18c is reset.

  In S16, the control circuit 18 specifies the positional relationship between the display screen of the display device 15 and the photographed hand. The process of S16 is the same process as S5 of FIG. In subsequent S17, as shown in FIG. 4, FIG. 5, and FIG. 51 (c), the position indication image 200H is superimposed on the main image 200B displayed on the display screen of the display device 15 (position indication). Image highlighting means). The process of S17 is the highlighting process of the position instruction image shown in FIG. 6 as in S6 of FIG. However, in the second embodiment mobile phone, the position instruction image 200H as shown in FIG. 4C is also displayed in S16 and S17. That is, in the fourth embodiment, as long as the display permission mode is set, overlapping display (or combined display) of hand images (finger images) is permitted, and at least the finger pointing position based on the hand images is indicated. The position indicating image 200H may be displayed in any form as long as it is an image. Therefore, when the display permission mode is set, there can be an overlapping display of a large number of fingers as shown in FIG. However, the operation target finger P is displayed more emphasized than the remaining fingers.

  On the other hand, if it is determined in S13 that the current finger state is not a predetermined finger state determined in advance (in this case, an operation intention display finger state), the process proceeds to S18. In S18, the display prohibition mode is set and stored in a predetermined storage area (display mode storage unit) provided in the storage unit of the control circuit 18.

  This process is ended by the end of S17 and S18, but this process is repeatedly performed at a predetermined cycle after the end, as long as the hand image is acquired and the display permission mode continues, the position indication image of S16 and S17 is displayed. The display permission state continues. That is, during this time, even if the finger state is not a predetermined finger state determined in advance, the position indication image is displayed based on the finger state at that time. However, in the display permission mode, when it is determined in S19 that the predetermined time has elapsed by the timer 18b started in S15, the display prohibition mode is forcibly set in S20, and this processing ends. Become. The process of S20 is the same as the process of S18.

  In this embodiment, the user indicates the intention of the operation by a hand shape (operation intention display finger state), and when this is indicated, the display permission mode is set for a certain period of time. Yes. In this case, the operation intention display finger state is not limited to the simple operation finger state such as the one-finger state (for example, see FIGS. 5 and 51), but is a state in which two fingers are extended (see FIG. 68). It can be variously defined as a state where three fingers are extended (see FIG. 69), a state where four fingers are extended (see FIG. 70), and the like. Further, even in the single-finger state, only the case where the finger is the index finger (see FIGS. 5 and 51) may be set as the operation intention display finger state, or only when the finger is the middle finger (see FIG. 71). An operation intention display finger state may be used. Further, even in the two-finger state, only when the fingers are the index finger and the middle finger (see FIG. 68), the operation intention display finger state may be set. Further, any one or more of these may be defined as an operation intention display finger state. From the viewpoint of preventing erroneous operations, it is desirable to determine a hand state (finger state) that is not performed by the user as an operation intention display finger state. From the viewpoint of processing, it is desirable to determine a hand state (finger state) that is easier to identify.

  Hereinafter, a fifth embodiment of the present invention will be described.

  Note that the identification of the predetermined finger states such as the operation intention display finger state and the simple operation finger state in S2 of FIG. 52 may not be performed based on the captured image as described above. For example, among the inputs by the touch operation performed on the touch operation surface 12a1 (remote operation surface) by the control circuit 18 during a predetermined period, the number of inputs by the touch operation performed at different positions on the touch operation surface 12a1. (Number of operations specifying means), and a case where the number of inputs specified within the period is a predetermined number (number for simple operation or number for operation intention display) is specified as a predetermined finger state You may do it. In this case, for example, when three touch operations are performed at different positions within one second (or simultaneously), it is specified that there are three fingers, and the predetermined number of fingers is three. Can be specified as the predetermined finger state. Specifically, as shown in FIG. 59, when a touch operation is performed on the touch operation surface 12a1 in a state where the display prohibition mode (and the input prohibition mode) is set, the control circuit 18 The timer 18b is started and the time counting is started by counting up the counter 18c, and the number of inputs by the touch operation including the touch operation at the start of the time counting is counted within the predetermined period t0. However, when a touch operation is performed on the touch operation surface 12a1, touch position information (for example, position coordinates in the touch operation surface 12a1) is acquired and a retouch operation to the same position in the touch operation surface 12a1 is performed. Do not count that input. If the number of inputs when the predetermined period t0 has elapsed is the same as the predetermined number of fingers that define the predetermined finger state, the finger state of the current operator's hand is the predetermined finger state. A display permission mode (and an input permission mode) is set only for a predetermined display permission period from the time of the identification. When the display permission period ends, the display prohibition mode (and the input prohibition mode) is set again.

  Hereinafter, a sixth embodiment of the present invention will be described.

  In the touch operation input process shown in FIG. 53, when the display permission mode is set, the control content corresponding to the touch operation input to the remote touch operation surface 12a is executed, that is, the execution of the corresponding control content is permitted. When the display prohibit mode is set, the control content corresponding to the touch operation input to the remote touch operation surface 12a is not executed, that is, the execution of the corresponding control content is prohibited. However, it corresponds to touch operation input to the remote touch operation surface 12a when a predetermined predetermined finger state (in this case, a finger state for operation intention display) is specified, not when the display permission mode is set. The control content corresponding to the touch operation input to the remote touch operation surface 12a when execution of the control content to be performed is permitted and the predetermined predetermined finger state is not specified rather than when the display prohibition mode is set Can be configured to be prohibited.

  Specifically, when a predetermined finger state (in this case, an operation intention display finger state) is specified in advance, the touch operation input to the remote touch operation surface 12a is permitted to be received and determined in advance. When the predetermined finger state is not specified, the operation input reception control for prohibiting the reception of the touch operation input to the remote touch operation surface 12a can be executed. Hereinafter, an example of the specific processing flow will be described with reference to FIG.

  In S111, the control circuit 18 determines whether or not there is a touch operation on the operation panel 12a. The process of S111 is the same as S101 of FIG. In S112, the control circuit 18 determines whether or not the current finger state is a predetermined finger state (in this case, an operation intention display finger state). In the present embodiment, as in the process of S2 of FIG. 52 or S12 of FIG. 55, etc., not only the process of specifying whether or not the predetermined finger state is predetermined, but also as the current finger state is specified. Then, a process of storing the specified finger state in a predetermined storage area (finger state storage unit) provided in the storage unit of the control circuit 18 is performed. In S112, based on the finger state information stored in the storage area (finger state storage unit), it is determined whether or not the current finger state is a predetermined finger state. When it is determined that the predetermined finger state is determined in advance, the process proceeds to S113, and when it is determined that the predetermined finger state is not determined, the process is terminated.

  In S113, the touch operation position on the operation panel 12a, that is, the touch operation position on the touch operation surface 12a1 is specified. This process is the same as S103 in FIG. In subsequent S114, control corresponding to the touch operation position on the operation panel 12a is performed. This process is the same as S104 in FIG. Although the present process ends upon completion of S114, the present process is repeatedly performed at a predetermined cycle even after the completion.

  Note that the position instruction image display process in the embodiment for executing the touch operation input process shown in FIG. 56 may be any one of FIGS. 52 and 55 and a position instruction image display process described later.

  The seventh embodiment of the present invention will be described below.

  In the above embodiment, the display permission mode is set to permit the display of the position indication image when a predetermined predetermined finger state is specified, and when the predetermined predetermined finger state is not specified. The display prohibition mode is set to prohibit the display of the position instruction image. However, such display mode switching control is performed when the vehicle is in a predetermined traveling state and is not in the predetermined traveling state. Can be configured to set the display permission mode as the display mode. In this case, the vehicle running state detecting means for detecting the running state of the vehicle is provided. FIG. 60 shows an example of such processing. Here, a vehicle speed sensor (vehicle speed detection means) 30 for detecting the vehicle speed of the vehicle is connected to the control circuit 18, and the control circuit 18 is predetermined by the vehicle. The display mode switching control is executed when the vehicle is in a vehicle running state that exceeds the vehicle speed level (vehicle speed threshold), and when the vehicle is not in the vehicle running state, the display permission mode is set as the display mode regardless of the finger state. It is supposed to be set.

  Specifically, in S21, the control circuit 18 determines whether an image of the operator's hand has been acquired. This is the same as the processing of S1 in FIG. When the image of the operator's hand is acquired, the process proceeds to S22, and when the image of the operator's hand is not acquired, this process ends.

  In S22, the current running state of the vehicle is specified. That is, it is determined whether or not the vehicle is in a traveling state exceeding a predetermined vehicle speed level. The control circuit 18 stores a vehicle speed threshold value in a predetermined storage unit such as a ROM in advance, determines whether the vehicle speed detected by the vehicle speed sensor 30 exceeds the vehicle speed threshold value, specifies the vehicle running state as the vehicle speed threshold, The case where it does not exceed is specified as a low-speed driving state (including a stop state). Here, 5 km / h or the like is fixedly determined as the vehicle speed threshold. Then, in S23, it is determined whether or not the current vehicle is in the vehicle running state described above. If it is determined that the vehicle is in the vehicle running state, the process proceeds to S24. On the other hand, if it is determined in S23 that the vehicle is not in a traveling state (when a low-speed traveling state including a stop state is specified), the process proceeds to S26.

  In S24, it is specified whether or not the operator's hand (operating hand) is in a predetermined finger state (in this case, an operation intention display finger state). The predetermined finger state is specified by the same process as the process of S2 in FIG. In subsequent S25, if it is determined that the current finger state is the predetermined predetermined finger state, the process proceeds to S26, and if it is determined not to be the predetermined predetermined finger state, the process proceeds to S29.

  In S26, the display permission mode in which the display of the position indication image is permitted is set as the display mode. The set display permission mode is stored in a predetermined storage area (display mode storage unit) of the control circuit 18. In subsequent S27, the control circuit 18 specifies the positional relationship between the display screen of the display device 15 and the photographed hand. This process is the same as the process of S5 in FIG. In subsequent S28, the position instruction image 200H is displayed in an overlapping manner on the main image 200B displayed on the display screen of the display device 15 (position instruction image display means). This process is the same as S6 in FIG.

  Specifically, when it is determined in S23 that the current vehicle is not in the predetermined vehicle running state (when the low-speed running state including the stop state is specified), the processing of S27 and S28 is performed. Regardless of the finger state of the operator, the position indication image 200H based on the finger state is displayed (see (c) of FIGS. 4, 5, and 51), but in S22, the vehicle is running. And when it is determined in S25 that the current operator's finger state is a predetermined finger state (in this case, an operation intention display finger state), by the processing of S27 and S28, A position indication image 200H based on a predetermined finger state determined in advance by the operator is displayed (see FIGS. 5 and 51 (c)).

  On the other hand, if it is determined in S25 that the current finger state is not a predetermined finger state determined in advance (in this case, an operation intention display finger state), the process proceeds to S29. In S29, the display mode is set to a display prohibition mode in which the display of the position indication image is prohibited, and this is stored in a predetermined storage area (display mode storage unit) of the control circuit 18. Thereby, the display of the position instruction image 200H performed in S27 and S28 is not executed, and the position instruction image 200H is not displayed on the display screen of the display device 15, for example, as shown in FIG.

  Although the present process is terminated by the end of S28 and S29, the present process is repeatedly performed at a predetermined cycle after the termination.

  In the present embodiment, the predetermined vehicle speed level (vehicle speed threshold) is set to 5 km / h, and the display permission mode is always set in a slow running state of the vehicle that does not exceed the vehicle speed level. It has come to be. However, as shown in FIG. 63, this vehicle speed level is set within a vehicle speed range that is higher than 0 km / h and lower than 5 km, or within a vehicle speed range that is greater than or equal to 5 km / h and less than 10 km / h (the vehicle is traveling slowly). Alternatively, it may be set to 0 km / h so that the display permission mode is always set when the vehicle is stopped. In addition, this vehicle speed level may be set within a vehicle speed range of 10 km / h or more and less than 30 km (low-speed driving state of the vehicle), or within a vehicle speed range of 30 km / h or more and less than 60 km / h (in the vehicle). (High speed running state), or may be set within a vehicle speed range of 60 km / h or higher (high speed running state of the vehicle).

  In the process of FIG. 60, the display mode switching control is performed when the vehicle is in a predetermined vehicle travel state (higher speed than the vehicle speed threshold), and when the vehicle is not in the predetermined vehicle travel state (lower speed than the vehicle speed threshold). ), The display permission mode is fixedly set. However, when the vehicle is in a predetermined vehicle traveling state (higher speed than the vehicle speed threshold), the display prohibition mode is fixedly set and the predetermined vehicle traveling is performed. An embodiment in which the display mode switching control is performed when the vehicle is not in a state (lower speed than the vehicle speed threshold) may be employed.

  Hereinafter, an eighth embodiment of the present invention will be described.

  In the above embodiment, the display permission mode is set to permit the display of the position indication image when a predetermined predetermined finger state is specified, and when the predetermined predetermined finger state is not specified. The display prohibit mode is set to prohibit the display of the position indication image. However, when the driver is specified as the operator of the touch operation surface 12a1, the display mode switching control is performed in the display mode. When the switching control is executed and the vehicle is not in the traveling state, the display permission mode can be set as the display mode regardless of the finger state. The specific processing flow will be described below with reference to FIG.

  In S31, the control circuit 18 determines whether an image of the operator's hand has been acquired. This is the same as the processing of S1 in FIG. When the image of the operator's hand is acquired, the process proceeds to S32, and when the image of the operator's hand is not acquired, this process is terminated.

  In S32, it is specified whether or not the operator who makes his / her hand face the operation panel 12a is a driver (operator specifying means). The operator can be specified based on, for example, a captured image (FIG. 28) of the camera 20 shown in FIG. Specifically, by identifying the direction in which the arm extends from the captured image of the camera 20 and determining whether or not the direction extends from the driver's seat 2D side, the operator Whether or not is a driver. In S33, it is determined whether or not the operator of the operation panel 12a is a driver. If it is determined that the operator is a driver, the process proceeds to S34. On the other hand, if it is determined in S33 that the operator of the operation panel 12a is a person other than the driver, the process proceeds to S36.

  In S34, it is specified whether or not the driver's hand (operating hand) as an operator is in a predetermined finger state (in this case, a finger state for operation intention display). The predetermined finger state is specified by the same process as the process of S2 in FIG. In subsequent S35, it is determined whether or not the current finger state is a predetermined predetermined finger state. If it is determined that the present finger state is a predetermined predetermined finger state, the process proceeds to S36, and a predetermined predetermined finger state is determined. If it is determined not to be in the state, the process proceeds to S39.

  In S36, the display permission mode in which the display of the position indication image is permitted is set as the display mode. The set display permission mode is stored in a predetermined storage area (display mode storage unit) of the control circuit 18. In subsequent S37, the control circuit 18 specifies the positional relationship between the display screen of the display device 15 and the photographed hand. This process is the same as the process of S5 in FIG. In subsequent S38, the position instruction image 200H is displayed in an overlapping manner on the main image 200B displayed on the display screen of the display device 15 (position instruction image display means). This process is the same as S6 in FIG.

  Specifically, when a person who is different from the driver is specified as an operator of the operation panel 12a in S33, the finger of the operator is determined regardless of the finger state of the operator by the processes of S37 and S38. While position indication image 200H based on the state is displayed (see (c) of FIGS. 4, 5, and 51), in S32, the driver is specified as the operator of operation panel 12a, and in S35. When it is determined that the finger state is a predetermined finger state (in this case, an operation intention display finger state), a position based on the predetermined finger state performed by the driver is obtained by the processing of S37 and S38. The instruction image 200H is displayed (see (c) of FIGS. 5 and 51).

  On the other hand, if it is determined in S35 that the current finger state is a predetermined predetermined finger state (in this case, an operation intention display finger state), the process proceeds to S39. In S39, the display mode is set to a display prohibition mode in which the display of the position indication image is prohibited, and this is stored in a predetermined storage area (display mode storage unit) of the control circuit 18. Thereby, the display of the position instruction image 200H performed in S37 and S38 is not executed, and the position instruction image 200H is not displayed on the display screen of the display device 15 as shown in FIG. 54C, for example.

  This process is ended by the end of S38 and S39. This process is repeatedly performed at a predetermined cycle even after the process is completed.

  The ninth embodiment of the present invention will be described below.

  In the present embodiment, the position indication image display process described above is performed as a process in which both of FIGS. 60 and 61 are combined. That is, the traveling state of the vehicle (for example, the vehicle speed of the vehicle) is detected, and whether or not the operator of the remote touch operation surface 12a is a driver is determined. A predetermined predetermined finger state is specified when the operator of the specified remote touch operation surface 12a is a driver. The display of the position instruction image 200H is permitted only when the position instruction image 200H is displayed, but the display of the position instruction image 200H is prohibited when the predetermined predetermined finger state is not specified. When the vehicle is not in the predetermined traveling state, or when the operator of the specified remote touch operation surface 12a is not a driver, or when both of them occur at the same time, the specified finger state is set. Regardless, it is possible to perform processing that permits display of the position indicating image 200H. The specific processing flow will be described below with reference to FIG.

  That is, when a hand image of the operator on the remote touch operation surface 12a is acquired in S41 (the same processing as S1 in FIG. 52), it is determined in S42 and S43 whether or not the vehicle is in a vehicle speed traveling state. If the vehicle is traveling at a high speed, the process proceeds to S44 and S45 to determine whether or not the operator of the remote touch operation surface 12a is a driver (FIG. 61). Processing similar to S32 and S33).

  If it is determined in S42 to S45 that the vehicle is in a vehicle speed traveling state and the operator of the remote touch operation surface 12a is a driver, the process proceeds to S46 and S47, and the operator's hand (operator) is It is specified whether or not the predetermined finger state is predetermined (the operation intention display finger state here) (the same processing as S2 and S3 in FIG. 52), and the current finger state is a predetermined finger state that is determined in advance. If it is determined that the predetermined finger state is not established, the process proceeds to S51.

  In S48, the display permission mode is set as the display mode, and this is stored in a predetermined storage area (display mode storage unit). In subsequent S49, the control circuit 18 specifies the positional relationship between the display screen of the display device 15 and the photographed hand, and in S50, on the main image 200B displayed on the display screen of the display device 15, The position instruction image 200H is displayed in an overlapping manner (position instruction image display means). These processes are the same as the processes of S4 to S6 in FIG.

  More specifically, if it is determined in S42 that the vehicle is not in a vehicle running state, if a person different from the driver is specified as an operator of the operation panel 12a in S44, S48 to S50 are performed. Through the processing, the position instruction image 200H based on the finger state is displayed regardless of the finger state of the operator (see FIGS. 4, 5 and 51 (c)). On the other hand, in S42, the vehicle is assumed to be in the vehicle running state, and in S44, the driver is specified as the operator of the operation panel 12a. When the intention display finger state) is specified, the position instruction image 200H based on the predetermined finger state performed by the driver is displayed by the processing of S48 to S50 ((c) of FIGS. 5 and 51). reference).

  On the other hand, if it is determined in S47 that the current finger state is not a predetermined finger state (in this case, an operation intention display finger state), the process proceeds to S51. In S51, the display prohibit mode is set as the display mode, and this is stored in a predetermined storage area (display mode storage unit). Thereby, the display of the position instruction image 200H implemented in S49 and S50 is not executed, and the position instruction image 200H is not displayed on the display screen of the display device 15, for example, as shown in FIG.

  This process is terminated by the end of S50 and S51. This process is repeatedly performed at a predetermined cycle even after the process is completed.

  In the processing of FIG. 62, the display mode switching control is performed when the vehicle is in a predetermined vehicle running state (higher speed than the vehicle speed threshold) and the operator is a driver. In the embodiment, the display permission mode is fixedly set when the vehicle is not in a predetermined vehicle running state (lower speed than the vehicle speed threshold) or when the operator is not a driver. The display prohibit mode is fixedly set when the operator is a driver and the display mode switching control is performed when the vehicle is not in a predetermined vehicle running state and the operator is a driver. In other cases (when the operator is not a driver), the display permission mode may be fixedly set.

  The tenth embodiment of the present invention will be described below.

  In the above embodiment, the camera 12b serving as the hand image photographing means is a camera that photographs a fixed area in the vehicle interior facing the touch operation surface (front surface) 12a1 of the touch panel 12a. It has the structure image | photographed from the back surface 12a2 side of 12a. However, the present invention is not limited to such a configuration. For example, as shown in FIG. 27, the touch operation surface 12a1 of the touch panel 12a is attached to the center console portion C of the vehicle so as to be photographed obliquely from above, and an approaching object approaching the touch operation surface 12a1 is displayed on the touch operation surface 12a1. You may comprise so that it may image | photograph from the side. In this case, unlike the above-described embodiment, when a specific image (for example, a hand image) in the photographed image is extracted and displayed on the display unit 15, the process of reversing the image horizontally can be omitted. Further, the camera 12b serving as the hand image photographing means may be used as the camera 20 for photographing the peripheral area including the touch operation surface 12a1 of the operation panel 12a.

  The eleventh embodiment of the present invention will be described below.

  Among the above-described embodiments, for all operation screens displayed on the display device 15, the display mode switching control is executed only on the predetermined operation screen for the configuration that executes the display mode switching control as described above. In other operation screens, the display permission mode or the display prohibition mode may be fixedly set.

  Based on the control command from the control circuit 18, the display device 15 can switch a plurality of operation screens 200 on which input by a pressing operation on the touch operation surface 12a1 is accepted on the display screen (operation screen display means). . For example, the menu selection screen 2003 shown in FIG. 65, the air volume setting screen 2004 of the car air conditioner shown in FIG. 66, the volume setting screen 2005 of the car audio shown in FIG. 67, and the car navigation system already described. The map display screen 2002 (FIG. 5C), the character input screen 2001 (FIG. 4C), and the like can be switched and displayed according to various inputs. Among these operation screens, display mode execution enable / disable information that makes it possible to specify the operation screen that is the execution target of the display mode switching control is stored in a predetermined storage area (display mode execution enable / disable information storage means) of the control circuit 18. The control circuit 18 refers to the display mode execution availability information when performing the position indication image display processing, and based on this, the operation screen displayed by the display device 15 is as described above. The display mode switching control is configured to be executed on the specified operation screen only when the display mode switching control is specified as an execution target. The specific processing flow will be described below with reference to FIG.

  In S61, the control circuit 18 determines whether an image of the operator's hand has been acquired. This is the same as the processing of S1 in FIG. When the image of the operator's hand is acquired, the process proceeds to S62, and when the image of the operator's hand is not acquired, this process is terminated.

  In S62, the control circuit 18 specifies the type of the operation screen 200 that is currently displayed, and specifies whether or not the display mode switching control is a predetermined execution target. Here, the display mode execution availability information stored in the predetermined storage area of the control circuit 18 includes a complicated operation in which the pressing operation on the touch operation surface 12a1 exceeds the predetermined operation burden level in the operation screen. The operation screen is determined as an execution target of the display mode switching control, and a simple operation screen including only an operation in which the pressing operation on the touch operation surface 12a1 does not exceed the operation load level is determined as a fixed setting target of the display permission mode. . The control circuit 18 refers to the display mode execution enable / disable information to specify whether or not the currently displayed operation screen 200 is an execution target of the display mode switching control.

  Note that the display mode execution availability information can be stored as information obtained by classifying screen types as follows. That is, as a complex operation screen, one or more operation input images displayed on the display device 15 (or all pressing operation areas defined on the touch operation surface 12a1 corresponding to the operation input image (operation icon) 200I). An operation screen in which the ratio SI / SA occupying the entire screen (or the entire area of the touch operation surface 12a1) exceeds a predetermined ratio S0 can be determined, and an operation screen not exceeding can be determined as a simple operation screen. (Example 1 in FIG. 72). Further, the number KI of one or more operation input images (or pressing operation areas determined on the touch operation surface 12a1 corresponding to the operation input images) 200I displayed on the display device 15 as the complex operation screen is determined in advance. An operation screen exceeding the given number K0 can be determined, and an operation screen not exceeding can be determined as a simple operation screen (example 2 in FIG. 72). Further, as a complex operation screen, an interval between one or more operation input images 200I displayed on the display device 15 (or between adjacent pressing operation areas defined on the touch operation surface 12a1 corresponding to the operation input image). An operation screen whose interval (LI) is narrower than a predetermined interval L0 can be determined, and an operation screen that does not exceed the predetermined interval L0 can be determined as a simple operation screen (example 3 in FIG. 72). Here, screens 2001, 2003, 3004, and 2005 as shown in FIGS. 65 to 67 and FIG. 5C are simplified operation screens, and a character input screen 2002 as shown in FIG. Name input screen) is defined as a complex operation screen.

  In S64, the control circuit 18 refers to the display mode execution availability information that can specify the complex operation screen and the simple operation screen, and the currently displayed operation screen 200 is the display mode switching control execution target (here, If it is determined whether or not it is a complicated operation screen, the process proceeds to S63. If it is determined in S63 that the currently displayed operation screen 200 is an execution target of display mode switching control, the process proceeds to S69. If it is determined that the operation screen 200 is not an execution target of display mode switching control, the process proceeds to S66. .

  In S64, it is specified whether or not the operator's hand (operating hand) is in a predetermined finger state (in this case, an operation intention display finger state) (a process similar to S2 in FIG. 52). In subsequent S65, if it is determined that the current finger state is the predetermined predetermined finger state, the process proceeds to S66, and if it is determined not to be the predetermined predetermined finger state, the process proceeds to S69.

  In S66, the display permission mode in which the display of the position indication image is permitted is set as the display mode. The set display permission mode is stored in a predetermined storage area (display mode storage unit) of the control circuit 18. In subsequent S67, the control circuit 18 specifies the positional relationship between the display screen 200 of the display device 15 and the photographed hand (similar to S5 in FIG. 52). In subsequent S68, the position instruction image 200H is displayed in an overlapping manner on the main image 200B displayed on the display screen 200 of the display device 15 (position instruction image display means: same as S6 in FIG. 52).

  More specifically, if it is determined in S62 that the currently displayed operation screen 200 is not an execution target of the display mode switching control (here, it is specified that it is a simple operation screen), S67 and While the position instruction image 200H based on the finger state is displayed by the process of S68 regardless of the finger state of the operator (see (c) of FIGS. 4, 5, and 51), in S62, When it is determined that the currently displayed operation screen 200 is an execution target of the display mode switching control (in this case, when the operation screen 200 is identified as a complex operation screen), the operation is performed by the processing of S67 and S68. The position instruction image 200H based on the operating intention display finger state is displayed (see FIG. 5 and FIG. 51 (c)).

  On the other hand, if it is determined in S65 that the current finger state is not a predetermined finger state, the process proceeds to S69. In S69, the display mode is set to a display prohibition mode in which the display of the position indication image 200H is prohibited, and this is stored in a predetermined storage area (display mode storage unit) of the control circuit 18. Thereby, the display of the position instruction image 200H implemented in S67 and S68 is not executed, and the position instruction image 200H is not displayed on the display screen 200 of the display device 15, for example, as shown in FIG. .

  Although the present process is terminated by the end of S68 and S69, the present process is repeatedly performed at a predetermined cycle after the termination.

  In the process of FIG. 64, the display mode switching control is performed on the complex operation screen, and the display permission mode is fixedly set on the simple operation screen. May be fixedly set, and the display mode switching control may be performed on the simple operation screen.

  The twelfth embodiment of the present invention will be described below.

  In the second to eleventh embodiments, when a predetermined finger state predetermined as the finger state of the operator is specified, the position indication image in the first embodiment is only for a predetermined display permission period. While the highlighted display of 200H is performed, in other cases, the position instruction image 200H is not displayed, and the finger state for displaying the operation intention is determined as the predetermined finger state. (Example 1 in FIG. 73), another characteristic finger state may be used, or a simple operation finger state (a finger state in which an operation can be easily performed (a state in which a specific number or type of fingers are extended): 73 example 2) may be sufficient.

  Hereinafter, embodiments of other embodiments will be described.

  In the above-described embodiment, the processed image (real finger image) 200H in which the contour shape of the photographed finger is reflected as it is is displayed as the position instruction image 200H, but the fingertip Q ( That is, an image completely different from the finger shape may be displayed, such as a fingertip position image (pointer or the like: see FIG. 50) 200H2 indicating a photographed finger pointing position). If highlighting is to be performed, only the fingertip region including the fingertip Q (for example, up to the first joint of the finger or an area within a predetermined radius centered on the fingertip position Q) may be displayed with emphasis. In this case, it is specified which position on the display screen the fingertip Q of the photographed finger faces, and a predetermined position indication image such as a pointer is displayed at the specified screen position. Further, as the position indication image 200H, a pseudo finger image (here, a bar-like image: see FIG. 49) 200H1 or the like that simply shows the photographed finger length direction F and the fingertip Q may be displayed.

  Moreover, you may comprise so that the operation target finger P in 1st embodiment mentioned above may correspond with the finger | toe specified in the already described simple operation finger state.

  Moreover, in the said embodiment, although the touch panel is provided and comprised as a remote operation part of this invention, an operation part provided with switches with pressing operation, such as a push switch and a seesaw switch, may be sufficient. In the above embodiment, a resistance film type touch panel is used as the operation panel 12a. However, another type of touch panel may be used.

  Further, as the operation panel 12a, an operation panel that detects the position of the touch operation by performing image processing on a photographed image photographed by photographing means such as a camera may be employed instead of the touch panel. Specifically, the operation panel 12a is a translucent panel whose surface 12a1 forms a touch operation surface, and a wavelength predetermined for an approaching object that approaches the operation panel 12a from the touch operation surface 12a1 side. A light source 12c for irradiating the band light through the operation panel 12a is provided. Then, the camera 12b serving as the hand image photographing means captures the approaching object from the back surface 12a2 side of the operation panel 12a1 in such a manner that the light emitted from the light source 12c captures the reflected light reflected by the approaching object. It is set as a photographing means. Further, the control circuit 18 functions as a light reflection area specifying means for specifying a light reflection area in which the intensity of the reflected light exceeds a predetermined threshold intensity on the image photographed by the camera 12b. Based on this light reflection area, it is configured to function also as an input receiving means for receiving a touch input to the light reflection area.

  According to this configuration, when a light reflection area exceeding a predetermined threshold intensity is specified in the captured image of the camera 12b, it is determined that a touch operation input has been made in the specified area, and the input is accepted. That is, the captured image of the camera 12b is configured to be used not only for displaying the position indication image as in the above-described embodiment, but also for specifying the light reflection area. Further, in this case, when the area of the specified light reflection area exceeds a predetermined threshold area, a touch operation input to the light reflection area can be received.

  Of course, each of the above-described embodiments can be implemented in combination with each element.

10 Vehicle Navigation Device 12 Operation Information Input Unit (Remote Operation Unit)
12a Operation panel 12b Camera (hand image photographing means)
12c Light source 15 Display device (display means)
15a touch panel 18 control circuit (input receiving means, position instruction image display means, simple operation finger state specifying means, operation input control means, display mode setting means, operator specifying means, running state specifying means, position instruction image emphasizing display means , Finger identification means, operation target finger designation means, fingertip identification means, designated finger information acquisition means)
18a Image processing unit 20 Camera 30 Vehicle speed sensor 40 Camera 41, 42 Camera 50 In-vehicle LAN
100 ECU
150 Captured image 200B Main image 200I Switch image (operation icon)
200H Position indication image (hand image)
H Approaching object (hand / finger)

Claims (39)

In a display device comprising a display means having a display screen and a remote operation unit having a remote operation surface for remotely performing operations on the display screen, fingers are opposed to the front side of the remote operation unit A display method for displaying a position indicating image in a form superimposed or superimposed on an image forming a background at a screen position on the display screen corresponding to the finger position.
A finger specifying step of specifying a finger in the opposite hand on the front side of the remote control unit;
A designated finger information obtaining step for obtaining, as designated finger information, approaching distance information reflecting the approaching distance relationship between the fingers identified by the finger identifying step to the remote operation surface;
An operation target finger that is used for the operation from the specified finger to the remote operation unit or is estimated to be used is selected from the fingers specified by the finger specifying step based on the obtained approach distance information , An operation target finger designating step that preferentially designates the finger of the fingertip closer to the remote operation surface;
A display step of displaying the position indicating image indicating the specified pointing position of the finger in a form synthesized or superimposed on a background image on the display screen, wherein the position indicating image is designated among the displayed position indicating images; A position indication image emphasis display step performed by highlighting the position indication image corresponding to the operation target finger relative to a position indication image corresponding to a finger different from the operation target finger;
A display method characterized by comprising: The display method according to claim 1 , wherein the finger specifying step is a fingertip specifying step of specifying a fingertip from the hand, and the finger is specified in a form of specifying the fingertip.   The display method according to claim 1, wherein in the operation target finger designation step, one finger is designated as the operation target finger. The display method according to claim 1, wherein the operation target finger designation step designates two or three fingers as the operation target fingers. A light source is irradiated with light from a light source to a hand approaching the remote operation surface, and has a photographing step of photographing the hand from the back side of the remote operation surface by photographing means,
The designated finger information acquisition step acquires the brightness of the image of the hand imaged by the imaging unit as the approach distance information,
In the operation target finger specifying step, among the fingers specified in the finger specifying step, the finger with the highest luminance of the fingertip specified in the fingertip specifying step is preferentially specified as the operation target finger. The display method according to any one of claims 1 to 4 , wherein: A light source is irradiated with light from a light source to a hand approaching the remote operation surface, and has a photographing step of photographing the hand from the back side of the remote operation surface by photographing means,
The display method according to any one of claims 1 to 5 , wherein the finger specifying step specifies the finger from the captured image of the hand. A photographing step of photographing a hand approaching the remote operation surface by photographing means from the back side of the remote operation surface, and the display step of displaying the position indication image is generated based on the image of the photographed hand display method according to to is for displaying a position indication image claims 1 any one of claims 6. In a display device comprising a display means having a display screen and a remote operation unit having a remote operation surface for remotely performing operations on the display screen, fingers are opposed to the front side of the remote operation unit A display method for displaying a position indicating image in a form superimposed or superimposed on an image forming a background at a screen position on the display screen corresponding to the finger position.
The remote operation unit is a touch operation unit that detects the presence or absence of a touch operation and a touch position on the touch operation surface while the remote operation surface forms a touch operation surface,
A finger specifying step of specifying a finger in the opposite hand on the front side of the remote control unit;
An operation target finger specifying step of using a finger that has performed a touch operation on the touch operation surface among the specified fingers to be used for an operation to the remote operation unit, or to be specified as an operation target finger that is estimated to be used,
A display step of displaying the position indicating image indicating the specified pointing position of the finger in a form synthesized or superimposed on a background image on the display screen, wherein the position indicating image is designated among the displayed position indicating images; A position indication image emphasis display step performed by highlighting the position indication image corresponding to the operation target finger relative to a position indication image corresponding to a finger different from the operation target finger;
A display method characterized by comprising: The display method according to any one of claims 1 to 8 , wherein the position indication image emphasis display step performs the emphasis display in a form in which an emphasis level is changed stepwise over a predetermined time. . The position instruction image highlighting step includes the position instruction image corresponding to the operation target finger and the position instruction corresponding to a finger different from the operation target finger at the start of the highlight display as the highlight display. The image is displayed with the same emphasis level, and the relative emphasis level of the position indication image corresponding to the operation target finger is changed stepwise as time elapses from the start of the emphasis display. The display method according to claim 9, wherein display is performed to execute display. The display method according to claim 9 or 10 , wherein the position indicating image highlighting step starts the highlighting as the operation target finger is specified by the operation target finger specifying step. The position indication image highlight display step, after a predetermined time has elapsed since the specified the operation target finger by the manipulation target finger specification step, according to claim 9 or claim 10 in which to start the highlight Display method. A hand image capturing step of capturing an image of a hand facing the remote operation surface;
The finger specifying step specifies a finger of a hand facing the remote operation surface based on a photographed hand image,
The position instruction image highlighting step generates a real finger image based on a photographed hand image that enables at least the contour of the finger of the identified hand to be identified, and the generated real finger image is the position instruction. The display method according to claim 1, wherein the display method is an image. The position indication image highlighting step displays either a pseudo finger image in which at least the finger extending direction can be specified or a fingertip position image in which at least the fingertip position of the finger can be specified as the position indication image. The display method according to claim 1, which is: The display method according to claim 1, wherein the highlighting is performed on an outline of the position indication image. The display method according to claim 1 , wherein the highlighting is performed by changing a display color of the position indication image. The display method according to any one of claims 1 to 14 , wherein the highlighting is performed in a form of changing a shade of a display color of the position indication image. The display method according to claim 1 , wherein the highlighting is performed by changing a shape of the position indication image. The display method according to claim 1 , wherein the highlighting is performed by changing the transparency of the position indication image. The display method according to any one of claims 1 to 14 , wherein the highlighting is performed by blinking the position indication image. After the operation target finger is specified in the operation target finger specifying step, the operation target finger tracking step of tracking the movement of the finger specified as the operation target finger,
The operation target finger designation step continuously designates the same finger as the finger designated as the operation target finger as the operation target finger based on a tracking result obtained by the operation target finger tracking step. The display method according to any one of claims 1 to 20 . The finger specifying step sequentially specifies the fingers of the hand that exist within a finger-identifiable range predetermined on the near side of the remote operation unit at predetermined time intervals,
By acquiring the position information for each finger sequentially specified by the finger specifying step and storing the position information as tracking information in a predetermined tracking information storage unit, the finger sequentially specified by the finger specifying step is stored. Finger tracking step to track the movement of each finger,
The operation target finger tracking step specifies tracking information of the operation target finger from the tracking information for each finger stored in the finger tracking step, and designates the operation target finger as the operation target finger based on the tracking information of the operation target finger The display method according to claim 21, wherein the displayed movement of the finger is tracked. In the finger tracking step, if there is a finger being tracked that does not satisfy a predetermined standard condition with the finger actually specified when the finger is specified in the finger specifying step, the tracking is performed. Switching the recognition of the finger as an untrackable finger, and storing the information related to the finger when it is recognized as untrackable as tracking return determination information in a predetermined tracking return determination information storage unit,
Further, the finger tracking step is stored when there is an actually specified finger that does not satisfy the reference condition with the finger being tracked when the finger is specified by the finger specifying step. On the basis of the tracking return determination information is determined whether or not the finger actually identified is the same as the finger that has been recognized as untraceable in the past, The finger corresponding to the tracking return determination information, which has been made untraceable, recognizes that it has reappeared at the position of the finger actually identified, and resumes the movement tracking of the finger. 23. The display method according to claim 22 , wherein if the determination is made, the finger actually identified is recognized as a newly appearing finger, and movement tracking of the finger is newly started. The tracking information includes at least the latest position information of the finger specified by the finger specifying step, and the tracking return determination information also includes the last position before the corresponding finger is recognized as untrackable. The display method according to claim 23, wherein information is included. In the finger tracking step, when the finger is specified by the finger specifying step, an actually specified finger that does not satisfy the reference condition with the finger being tracked is specified, and the tracking is performed. When tracking return determination information of a plurality of fingers is stored in the return determination information storage unit, the position information of the finger actually identified and the non-trackable finger included in the tracking return determination information Based on the last position information, the finger that is not traceable corresponding to the tracking return determination information including the position information indicating the position closest to the actually identified finger is actually identified. The display method according to claim 24, wherein the finger is determined to be the same finger as the finger. The finger tracking step includes the finger being tracked that does not satisfy the reference condition with the finger actually identified when the finger is identified in the finger identifying step, and the finger being tracked is tracked If it is recognized as unacceptable, the information on the finger when it is recognized as untraceable, and the unrecognizable recognition order in which the context of unrecognizable recognition time with other untraceable fingers can be specified, Store in the tracking return determination information storage unit as tracking return determination information,
Further, in the finger tracking step, when the finger is specified by the finger specifying step, an actually specified finger that does not satisfy the reference condition with the finger being tracked is specified, and When tracking return determination information of a plurality of fingers is stored in the tracking return determination information storage unit, the position information of the finger that is actually specified, and the untrackable information included in the tracking return determination information Based on the last position information of the finger, the tracking return determination information including the position information indicating the position closest to the actually specified finger is specified, and one piece of the tracking return determination information is specified In this case, when the finger that is not traceable corresponding to the tracking return determination information is determined to be the same finger as the actually specified finger, two or more tracking return determination information is specified In With reference to the untraceable recognition order included in the tracking return determination information, the finger that is untraceable corresponding to the latest tracking unrecognized time among the tracking return determination information is actually identified. The display method according to claim 24 or 25 , wherein the finger is determined to be the same finger as the finger. In a display device comprising a display means having a display screen and a remote operation unit having a remote operation surface for remotely performing operations on the display screen, fingers are opposed to the front side of the remote operation unit A display method for displaying a position indicating image in a form superimposed or superimposed on an image forming a background at a screen position on the display screen corresponding to the finger position.
A finger specifying step of specifying a finger in the opposite hand on the front side of the remote control unit;
An operation target finger designation step for designating an operation target finger to be used for the operation from the identified finger to the remote operation unit or estimated to be used;
A display step of displaying the position indicating image indicating the specified pointing position of the finger in a form synthesized or superimposed on a background image on the display screen, wherein the position indicating image is designated among the displayed position indicating images; A position indication image emphasis display step performed by highlighting the position indication image corresponding to the operation target finger relative to a position indication image corresponding to a finger different from the operation target finger;
And the finger identifying step sequentially identifies the fingers of the hand that are within a finger identifiable range predetermined on the near side of the remote control unit at predetermined time intervals,
By acquiring the position information for each finger sequentially specified by the finger specifying step and storing the position information as tracking information in a predetermined tracking information storage unit, the finger sequentially specified by the finger specifying step is stored. Finger tracking step to track the movement of each finger,
After the operation target finger is specified in the operation target finger specifying step, the finger stored in the tracking information storage unit by the finger tracking step is used to track the movement of the finger specified as the operation target finger. An operation target finger tracking step of identifying the tracking information of the operation target finger from each tracking information, and tracking the movement of the finger designated as the operation target finger based on the tracking information of the operation target finger;
And the operation target finger designation step continuously designates the same finger as the finger designated as the operation target finger as the operation target finger based on the tracking result of the operation target finger tracking step. And
The finger tracking step includes
Among each finger being tracked that stores the tracking information for each finger specifying step, the finger is determined in advance with the finger actually specified when the finger is specified by the finger specifying step. If there is a finger that does not meet the reference criteria, the recognition is switched to the finger being tracked as an untrackable finger, and other untrackable fingers along with information about the finger when it is recognized as untrackable. The tracking unrecognized order in which the context of the non-trackable recognition time can be specified is stored as tracking return determination information in a predetermined tracking return determination information storage unit,
The finger tracking step is stored when there is an actually specified finger that does not satisfy the reference condition with the finger being tracked when the finger is specified by the finger specifying step. On the basis of the tracking return determination information, it is determined whether the finger actually identified is the same as the finger that was previously recognized as untraceable. Recognize that the finger that was not trackable corresponding to the tracking return determination information has reappeared at the position of the finger that was actually identified, and resume the movement tracking of the finger. If the finger has been identified, the finger actually identified is recognized as a newly appearing finger, and movement tracking of the finger is newly started. The tracking information includes at least the finger identifying step. The latest position of the finger specified by Are included, the even track return determination information includes last position information before the corresponding finger is recognized as the tracking impossible,
Further, the finger tracking step includes an actually specified finger that does not satisfy the reference condition with the finger being tracked when the finger is specified in the finger specifying step, and the tracking return is performed. If the determination information storage unit stores the tracking return determination information of a plurality of fingers, the position information of the finger actually identified and the last of the untrackable fingers included in the tracking return determination information When the tracking return determination information including the position information indicating the position closest to the finger that is actually specified is specified based on the position information, and one piece of the tracking return determination information is specified When the finger that is not traceable corresponding to the tracking return determination information is determined to be the same finger as the actually specified finger, while two or more tracking return determination information is specified, Add them Referring to the untraceable recognition order included in the return determination information, the finger that is not trackable corresponding to the latest untrackable recognition time among the tracking return determination information is actually identified. A display method characterized in that the finger is determined to be the same finger as the finger. The finger tracking step is performed when an actually specified finger that does not satisfy the reference condition with the finger being tracked is specified when the finger is specified in the finger specifying step. If the distance between the respective positions indicated by the position information of the finger specified in (2) and the last position information of the non-trackable finger included in the tracking return determination information is less than a predetermined reference distance While it is allowed to recognize that the finger that is actually identified and the finger that is not trackable corresponding to the tracking return determination information are the same, if the distance is not less than the reference distance, the finger is actually identified. The display method according to any one of claims 24 to 27 , wherein it is prohibited to recognize that the finger that has been made and the finger that cannot be tracked corresponding to the tracking return determination information are the same. . Said tracking return decision information, when the corresponding finger is recognized as the tracking disabled, any one of the tracking information to claims 24 which is the tracking information of the finger which has been stored in the storage unit according to claim 28 Display method described in. In the finger tracking step, when the finger is specified by the finger specifying step and the specified finger is recognized as a newly appearing finger, new identification information is given to the newly appearing finger, The tracking information of each finger is stored in the tracking information storage unit in a form associated with the identification information,
In the finger tracking step, when the tracking finger to which the identification information is given is recognized as a non-trackable finger, the tracking return determination information is associated with the identification information of the finger. 30. The display method according to claim 29, wherein the display method is stored in a tracking return determination information storage unit. The finger tracking step includes the tracking information stored in the tracking information storage unit and further the tracking stored in the tracking return determination information storage unit when the finger is specified by the finger specifying step. Based on the return determination information, whether or not the finger actually specified by the finger specifying step is the finger being tracked or whether or not the finger is untrackable is attached. 31. The display method according to claim 30, wherein determination is made for each finger, and the determination result is stored in a predetermined tracking state storage unit in a form associated with the identification information as a tracking state. In the finger tracking step, when the finger is specified by the finger specifying step and the specified finger is recognized as a newly appearing finger, new identification information is given to the newly appearing finger, The display method according to any one of claims 22 to 29 , wherein the tracking information of each finger is stored in the tracking information storage unit in a form associated with the identification information. The finger tracking step erases the identification information and various information stored in association with the identification information when the finger is not identified when the finger is identified by the finger identifying step. The display method according to any one of claims 30 to 32 , wherein the display method can be initialized. In the finger tracking step, each tracking target storing the tracking information for each finger specifying step is changed to a corresponding next tracking information that is estimated to be positioned at the next finger specifying step. A next position estimation step that is calculated based on the next position of each finger being tracked calculated in the next position estimation step, and each finger that is actually specified in the next finger specification step. A pair of the next position and the actual position satisfying the reference condition defined with respect to a positional relationship with an actual position is specified, and is actually specified by the finger being tracked and the finger specifying step which is one of the pair The finger is determined to be the same finger, and the movement tracking of the finger is continued as if the finger being tracked was moved to the actually specified finger position determined to be the same. Claim 2 To display method according to any one of claims 33. The tracking information may include each position information of the corresponding finger being tracked identified by the finger identifying step a plurality of times,
The display method according to claim 34, wherein the next position estimation step calculates the next position of the corresponding finger being tracked based on the position information. In the case where the tracking information includes each position information of the corresponding finger being tracked specified in the previous and previous finger specifying steps, the next position estimation step is based on both of the position information. Calculating the next position of the finger being tracked, while the tracking information includes the corresponding position information of the finger being tracked identified by the previous finger identifying step, and When the position information of the corresponding finger being tracked specified by the step is not included, the position information specified by the previous finger specifying step is close to the position indicated by the position information. 36. The display method according to claim 35, wherein the next position of the finger being tracked is calculated based on a position of an edge of the finger identifiable range. 37. A display device comprising means for realizing each step in the display method according to any one of claims 1 to 36 . A display device according to claim 37 is provided,
The display screen is a screen for operating a device, and the remote operation surface is provided at a position different from the display screen, and is an operation surface for remotely performing operation input on the display screen. An input receiving unit that receives an input corresponding to a pressing operation position based on a pressing operation on the remote operation surface; and an image of a hand facing the remote operation surface. Hand image photographing means for photographing; and position indication image highlighting means for displaying a position indication image generated based on the image of the photographed hand at a position on the display screen pointed to by the hand. An operation input device comprising: The device is a vehicle-mounted device, the display screen is arranged to be visible by the driver on the vehicle front side of the driver, and the remote control surface is provided at a position closer to the driver than the display screen. The operation input device according to claim 38 .

Images