JP2017187922A - In-vehicle information processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an in-vehicle information processing system allowing for intuitive operation with reduced delay.SOLUTION: An in-vehicle information processing system 10 includes: a display unit 11 having at least one screen; a touch operation unit 12 which detects a touch of an operating hand of an operator; an imaging unit 13 which images at least a part of the operating hand and the touch operation unit 12; and a control unit 14 which performs image processing within a predetermined time on the basis of an image captured by the imaging unit 13, while associating position coordinates in an operation area R2 on the screen with position coordinates in a predetermined area R1 of the touch operation unit 12, to superimpose at least a part of the operating hand on the screen.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an in-vehicle information processing system mounted on a vehicle in order to control information related to the vehicle.

  Conventionally, an operator performs a pointer operation or a cursor operation displayed on a screen, a screen scroll operation, a selection operation, an input operation, and the like by performing a touch operation on a touch pad installed on a center console. Is possible.

  For example, the operation input device disclosed in Patent Literature 1 reduces the influence of disturbance factors including vibration of a running vehicle, and smoothes the cursor movement on the display unit based on the touch operation on the touch pad. It is possible.

  In addition, for example, the operation device disclosed in Patent Document 2 can operate a pointer or cursor displayed on the screen and scroll the screen by a touch operation by an operator on a touch pad installed in the center console. And

JP2015-174648 JP, 2006-012313, A

  However, the conventional devices disclosed in Patent Documents 1 and 2 respond to an operation of the operator by drawing a pointer, a cursor, a movement vector, or the like on the screen. Such an operation is not intuitive. In addition, since the correspondence between the position of the operator's operator's hand and the position of the pointer on the screen is not clear, it takes a long time for the operator to watch the pointer on the screen. Therefore, it is desired to realize an interface that enables more intuitive operation.

  Therefore, a system is effective in which the operator's operator's hand imaged by the camera is displayed on the screen so that the operation can be performed in a state closer to the actual feeling. In this system, in order to identify the touch operation position by the operator's operator, a method of performing image processing based on the captured image can be considered. However, this method causes an increase in calculation processing time and causes a time delay between the actual movement of the operator's operator and the movement on the screen. As a result, the operator is expected to feel uncomfortable with the operation.

  An object of the present invention made in view of such a viewpoint is to provide an in-vehicle information processing system capable of an intuitive operation with little delay.

An in-vehicle information processing system according to a first aspect that solves the above problem is as follows.
A display unit having at least one screen;
A touch operation unit that detects contact by an operator's operator;
An imaging unit that images at least a part of the operator and the touch operation unit;
Corresponding the position coordinates in the operation area on the screen and the position coordinates in the predetermined area of the touch operation unit, and performing image processing within a predetermined time based on the image captured by the imaging unit, A control unit that superimposes and displays at least a part of the operator on the screen.

In the in-vehicle information processing system according to the second aspect,
The imaging unit images a positional relationship between the touch operation unit and the operator,
The control unit superimposes and displays at least a part of the operating hand on the screen so as to correspond to the position of the operating hand in the touch operation unit.

In the in-vehicle information processing system according to the third aspect,
The control unit performs image processing for extracting at least a part of the operating hand based on the image captured by the imaging unit.

In the vehicle information processing system according to the fourth aspect,
The control unit performs image processing for changing a display magnification of the operator based on a size ratio between the predetermined region of the touch operation unit and the operation region on the screen.

In the in-vehicle information processing system according to the fifth aspect,
The control unit identifies position coordinates in the predetermined area of the touch operation unit where contact by the operator is detected based on detection information from the touch operation unit.

  According to the in-vehicle information processing system according to the first aspect, an intuitive operation with little delay is possible.

  Moreover, according to the vehicle-mounted information processing system according to the second aspect, it is possible to shorten the time for the operator to watch the screen.

  Further, according to the in-vehicle information processing system according to the third aspect, it is possible to faithfully superimpose and display the operator's operator's hand on the screen.

  Further, according to the in-vehicle information processing system according to the fourth aspect, it is possible to superimpose and display the operator's hand with an optimum size according to the screen.

  Further, according to the in-vehicle information processing system according to the fifth aspect, it is possible to reduce the time delay as compared with the case where the position coordinates are identified by image processing.

It is a mimetic diagram showing the whole in-vehicle information processing system concerning this embodiment. It is a functional block diagram which shows schematic structure of the vehicle-mounted information processing system of FIG. It is the schematic diagram which showed an example of the image displayed on a display part. It is a figure which shows typically a cross section when a touch operation part is seen from a side surface direction. It is a schematic diagram which shows an example of the correspondence of the predetermined area | region of a touch operation part, and the operation area on the screen which comprises a display part. It is the schematic diagram which showed the mode of the image processing which an in-vehicle information processing system performs. It is the schematic diagram which illustrated the correspondence of the function item displayed on the screen which comprises a display part, and a touchpad. It is the schematic diagram which illustrated the mode of operation which the operator's operator hand superimposed and displayed on the screen performed virtually. It is a flowchart which shows an example of operation | movement of the vehicle-mounted information processing system of FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a schematic diagram showing an entire in-vehicle information processing system 10 according to the present embodiment. FIG. 2 is a functional block diagram showing a schematic configuration of the in-vehicle information processing system 10 of FIG. The in-vehicle information processing system 10 includes a display unit 11, a touch operation unit 12, an imaging unit 13, a control unit 14, and a storage unit 15. FIG. 3 is a schematic diagram illustrating an example of an image displayed on the display unit 11. FIG. 3A shows an example of a menu screen, and FIG. 3B shows an example of a map screen. FIG. 4 is a diagram schematically illustrating a cross section when the touch operation unit 12 is viewed from the side surface direction.

  The in-vehicle information processing system 10 associates the position coordinates in the operation area on the screen configuring the display unit 11 with the position coordinates in the predetermined area of the touch operation unit 12 and images captured by the imaging unit 13. Based on the above, the operator's operating hand is superimposed on the screen. That is, based on the touch operation by the operator on the touch operation unit 12, the operator superimposed on the screen operates the screen virtually at the corresponding position. The in-vehicle information processing system 10 causes the movement of the operator superimposed on the screen to correspond to the movement of the actual operator's operator captured by the imaging unit 13. Here, the operator is, for example, a driver driving the vehicle or a passenger sitting in the passenger seat, and the operator is, for example, a driver on the center console side or the passenger's own hand.

  As shown in FIG. 1, the display unit 11 has at least one screen. The display unit 11 may be configured by an arbitrary display device such as a liquid crystal display. When the display unit 11 is configured by a liquid crystal display, for example, the display unit 11 is disposed on an instrument panel. The display device constituting the display unit 11 may be a touch panel display or a display incapable of touch operation. In the following description, it is assumed that the display unit 11 is a display that cannot be touched.

  The in-vehicle information processing system 10 may include a so-called head-up display type device in addition to or instead of the display unit 11. In this case, the head-up display type device has a light emitting unit that generates display information as display light, reflects the generated display light toward an observer such as a driver, and the virtual image through the front windshield. Is displayed. The observer is not limited to the driver and may be a passenger sitting in the passenger seat.

  The display unit 11 displays information on the vehicle, function items for controlling the information, or a combination thereof. Hereinafter, these are collectively referred to as “display contents”. The information about the vehicle includes, for example, information such as air conditioning, car navigation, audio, an image around the vehicle by an electronic mirror, a vehicle speed, a traveling position of the vehicle in a plurality of lanes, or an inter-vehicle distance. The function items for controlling the information include, for example, “return”, “forward”, “home”, “decision”, “various menus”, “temperature high / low”, “current location”, “volume high / low” ”,“ Enlargement / reduction ”,“ speed fast / slow ”,“ lane change ”, or“ distance long / short ”. The display unit 11 may display each item as a character or an icon.

  For example, as shown to Fig.3 (a), the display part 11 displays various menus as a function item for controlling the information regarding a vehicle. Specifically, the display unit 11 displays “APPS” as an item for displaying various applications, “TEL” as an item for using a telephone, and “A / C” as an item for controlling an air conditioner. , “NAVI” is displayed as a menu for using the car navigation, and “AUDIO” is displayed as a menu for using the audio. Similarly, the display unit 11 displays “HOME” as an item for returning to the home screen, and “RETURN” as an item for returning to the previous screen.

  For example, as shown in FIG. 3B, the display unit 11 displays map information that is a part of the car navigation system as information about the vehicle. Further, the display unit 11 has functions such as “Destination setting”, “HOME”, and “RETURN” so as to be superimposed on the map information as a combination of information on the vehicle and functional items for controlling the information. Display items.

  The display unit 11 displays the operator's hand so as to overlap the display content. As shown in FIG. 3, the display unit 11 makes the operator's operating hand translucent and displays the above display content behind it. The display unit 11 is not limited to this, and may display an opaque operation hand so that the display content behind the operator is temporarily hidden when the operator's operation hand is superimposed.

  In the following, the degree of semi-transmission, that is, the transmittance is described as being constant without depending on the overlapping position, but is not limited thereto, and may be changed for each overlapping position. For example, at the position where the function items are displayed, the display unit 11 raises the transmittance above a predetermined value so that the operator can sufficiently recognize which item the operator should select. It may be displayed. Conversely, at a position where only the background is displayed, the display unit 11 may display the transmittance lower than a predetermined value.

  Moreover, the display part 11 may display the operator's operator hand which attached the gradation. The gradations described herein may include any step change with respect to light and darkness, color, or transmittance, or a combination thereof. The display unit 11 is preferably displayed in gradation by an arbitrary method in which the operator can easily view the display content behind. For example, the display unit 11 displays the operator by gradually increasing the brightness of the operator, gradually changing to a lighter color, or gradually increasing the transmittance as the fingertip of the operator's operator superimposed on the display approaches. May be.

  As the transmittance is lower, the reality of the operator's operator to be superimposed is improved, but it becomes difficult for the operator to visually recognize the display content behind. Conversely, the higher the transmittance, the easier it is for the operator to visually recognize the display content behind, but the reality is reduced. Therefore, it is preferable that the display unit 11 displays each display content by an arbitrary display method that allows the operator to easily view the display contents while ensuring the reality of the operator's operator to be superimposed.

  Although the display unit 11 has been described as superimposing an operator's hand in the real world on a virtual space in which the above-described display content is displayed, the present invention is not limited to this. For example, the display unit 11 may superimpose display contents and the like from the front surface of the operator's operator displayed on the screen as in a so-called mixed reality.

  As shown in FIG. 1, the touch operation part 12 is arrange | positioned at a center console, for example. The touch operation unit 12 includes a touch pad 121 and a tact switch 122 as shown in FIG. As shown in FIG. 4, the operator places his / her arm and wrist on the armrest and palmrest, respectively, and makes a part of the operation hand, for example, a finger contact the touchpad 121.

  The touch pad 121 detects contact by a contact object such as an operator's operating hand or a stylus pen at a corresponding contact position. In particular, the touch pad 121 detects contact with a part of the operator's operating hand, for example, a finger at a corresponding contact position. The operator operates information displayed on each screen constituting the display unit 11 by performing a touch operation on the touch operation unit 12, particularly the touch pad 121. The touch pad 121 is formed of, for example, transparent glass, and a touch sensor configured by an arbitrary method such as a resistive film method, a capacitance method, a surface acoustic wave method, an infrared method, or an electromagnetic induction method can be used. . In the following description, it is assumed that the touch pad 121 is a capacitive touch pad.

  As shown in FIG. 4, the tact switch 122 is disposed immediately below the touch pad 121 and supported by the substrate. When the touch pad 121 is displaced downward by the operator's pressing, the tact switch 122 is turned on by the pressing. The tact switch 122 is turned off when the push of the operator is released and the touch pad 121 returns to the original position. When the operator depresses the touch pad 121, the tact switch 122 is turned on to obtain a click feeling.

  In FIG. 4, one tact switch 122 is arranged at the central portion immediately below the touch pad 121. However, the tact switch 122 is not limited to this, and any number can be used as long as the pressure from the touch pad 121 can be detected. It may be arranged at any place. For example, one tact switch 122 may be disposed on the outer periphery of the tact switch 122 immediately below the touch pad 121, or a plurality of tact switches 122 may be disposed at dispersed positions. For example, by arranging a plurality of tact switches 122 immediately below the touch pad 121, the touch operation unit 12 is configured to be able to detect a pressure from the touch pad 121 for each predetermined area of the touch pad 121. Also good. That is, the touch operation unit 12 is configured to be able to detect at which position on the touch pad 121 the operator has pressed when a plurality of fingers of the operator are in contact with the touch pad 121 at the same time. Also good.

  The configuration unit arranged immediately below the touch pad 121 is not limited to the tact switch 122, and may be any configuration as long as the pressing from the touch pad 121 can be detected. For example, instead of the tact switch 122, a pressure sensitive sensor such as a piezoelectric element may be disposed immediately below the touch pad 121.

  In addition to the touch pad 121 and the tact switch 122, the touch operation unit 12 may include a filter in order to remove an unnecessary detection signal of the touch pad 121 due to vibration during traveling of the vehicle.

  The imaging unit 13 includes at least one camera and is disposed on, for example, a roof panel. The camera which comprises the imaging part 13 is arrange | positioned so that the inside of a vehicle may be imaged from a roof panel. More specifically, the imaging unit 13 captures an image of at least a part of an operator's operator who performs a touch operation with the touch operation unit 12 and the touch operation unit 12 from above.

  The imaging unit 13 may image at least a part of the operator's operating hand, for example, only five fingers, but it is preferable to image the entire operator's operating hand including the back of the hand. Here, the entire operating hand is the entire part including from the vicinity of the wrist of the operator to the fingertip. In this case, the imaging unit 13 is preferably arranged above the operator, such as a roof panel, in order to easily image the entire operator's operator.

  The imaging unit 13 is not limited to such an arrangement, and may be arranged at an arbitrary place as long as at least a part of the operator's operating hand can be imaged. For example, the imaging unit 13 may be arranged directly below the transparent touch pad 121 and may capture a part of the operator's operator performing a touch operation with the touch pad 121 from below. In this case, for example, by changing the palm rest portion in FIG. 4 to an arbitrary transparent support member, the imaging unit 13 can also image the entire operator's operating hand including the back of the hand.

  The imaging unit 13 is preferably composed of a camera with a wide dynamic range so that the operator's hand can be clearly imaged both in a bright state during the day and in a dark state at night. The image captured by the camera may be a black and white image or a color image.

  The imaging unit 13 is not limited to a configuration with a camera having a wide dynamic range, and may be configured with a camera capable of imaging in a bright state in the daytime. In this case, the imaging unit 13 may irradiate the operator's operating hand on the touch pad 121 with a spotlight from above in order to clearly capture the operator's operating hand even at night.

  When the vehicle performs automatic driving, it is assumed that the operator leans on the reclining seat and leans on the seat in a relaxed state. At this time, if the position of the touch operation unit 12 is fixed, the operator needs to perform the touch operation by extending his arm while leaning on the seat, and feels inconvenience. Therefore, for example, by configuring the center console on which the touch operation unit 12 is disposed so as to be lowered rearward in conjunction with the movement of the reclining seat, the operator can easily perform the touch operation without extending the arm. Can do. In the case of such a configuration, the imaging unit 13 needs to image the operator on the touch pad 121 in accordance with each position of the touch operation unit 12 that is linked to the movement of the reclining seat.

  Therefore, in addition to the above, the imaging unit 13 is preferably configured with a camera having a wide angle of view. The imaging unit 13 is not limited to this, and may be configured such that the angle of the camera itself changes in conjunction with a change in the position of the touch operation unit 12 even if the camera has a narrow angle of view. Similarly, the imaging unit 13 may be configured such that the camera itself translates in conjunction with a change in the position of the touch operation unit 12. Further, the position of the touch operation unit 12 that changes in conjunction with the movement of the reclining seat is limited to two positions, for example, a position at the time of manual operation and a position at the time of automatic operation, so as to correspond to each position. Two cameras may be arranged.

  The control unit 14 is a processor that controls and manages the entire in-vehicle information processing system 10 including each functional block of the in-vehicle information processing system 10. The control unit 14 includes a processor such as a CPU (Central Processing Unit) that executes a program that defines a control procedure, and the program is stored in the storage unit 15, for example.

  The control unit 14 acquires the contact information detected on the touch pad 121 from the touch operation unit 12 as an input signal. Specifically, the control unit 14 acquires detection information related to a contact object, for example, contact with an operator's finger and a corresponding contact position. The control unit 14 identifies accurate position coordinates on the touch pad 121 where the touch operation is performed based on the detection information regarding the corresponding contact position.

  The control unit 14 acquires a signal related to the on state or the off state of the tact switch 122 from the touch operation unit 12. Specifically, when the operator presses the tact switch 122 via the touch pad 121, the control unit 14 acquires an on-state signal. When the operator stops pressing the touch pad 121 and releases the pressing of the tact switch 122, the control unit 14 acquires an off-state signal. The control unit 14 identifies the on state or the off state of the tact switch 122 based on the acquired signal.

  The control unit 14 selects a corresponding item on the screen constituting the display unit 11 when the touch pad 121 detects contact by a part of the operator's operating hand. At this time, the control unit 14 highlights the item. Highlighting is to display a predetermined item with emphasis. The control unit 14 provides feedback to the operator that the above items are in a selected state by highlighting. For example, as illustrated in FIG. 3A, when contact with the operator's finger is detected at a corresponding position on the touch pad 121, the control unit 14 highlights the function item “NAVI” on the screen. At this time, the control unit 14 superimposes and displays the operator's operating hand at a corresponding position based on the image captured by the imaging unit 13.

  Subsequently, when the tact switch 122 is turned on by pressing the touch pad 121 by a part of the operator's operating hand, the control unit 14 determines the selection of a predetermined item on the screen. The operation for confirming selection of a predetermined item on the screen is not limited to this, and may be an arbitrary operation such as a double tap on the touch pad 121, for example. In this case, the touch operation unit 12 may not have the tact switch 122.

  For example, in FIG. 3A, when the tact switch 122 is turned on by being pushed by a part of the operator's operator's hand, the control unit 14 confirms the selection of the item “NAVI” displayed on the screen. Let At this time, the control unit 14 causes the same movement to be displayed on the screen in accordance with a part of the operator's operator touching the touch pad 121, for example, pressing of the index finger or double tap.

  The control unit 14 causes the display unit 11 to display information on the vehicle, function items for controlling the information, or a combination thereof. Further, the control unit 14 superimposes and displays at least a part of the operator's operator's hand on the screen at a display magnification based on the size of the operation area on the screen constituting the display unit 11 by image processing to be described later. Let

  The control unit 14 refers to various information stored in the storage unit 15. Specifically, the control unit 14 refers to information related to the vehicle or information related to a function item for controlling the information. The control unit 14 refers to information regarding the on state or the off state of the tact switch 122. The control unit 14 refers to the image information captured by the imaging unit 13. The control unit 14 refers to information on the operator's operator who has been image-processed and is finally superimposed on the display unit 11.

  The storage unit 15 can be configured by a semiconductor memory, a magnetic memory, or the like, and stores various information described above, a program for operating the in-vehicle information processing system 10, and the like. The storage unit 15 also functions as a work memory. For example, the storage unit 15 stores information related to the operator's operator who has been subjected to image processing, which is finally superimposed on the display unit 11.

  Below, with reference to FIG.5 and FIG.6, the image processing which the vehicle-mounted information processing system 10 performs is demonstrated in detail.

  FIG. 5 is a schematic diagram illustrating an example of a correspondence relationship between a predetermined area of the touch operation unit 12 and an operation area on the screen configuring the display unit 11. FIG. 5A shows a predetermined region R1 of the touch operation unit 12. FIG. 5B shows an operation area R <b> 2 on the screen constituting the display unit 11.

  The control unit 14 sets a predetermined region R1 of the touch operation unit 12 and an operation region R2 on the screen constituting the display unit 11. The predetermined area R1 of the touch operation unit 12 is an area for the operator to perform a touch operation with a part of the operator. For example, the predetermined region R1 of the touch operation unit 12 is a part or the entire region of the touch pad 121. The operation area R2 on the screen constituting the display unit 11 is an area on the screen that can be virtually operated by the operator's operator hand superimposed on the screen. For example, the operation area R2 on the screen constituting the display unit 11 is a part or the entire area of the screen.

  As shown in FIG. 5A, the predetermined region R <b> 1 of the touch operation unit 12 is set on the back side of the touch operation unit 12 so that the operator's operating hand is superimposed on the entire touch operation unit 12. Is preferred. The back side of the touch operation unit 12 is, for example, the back side of the touch pad 121 constituting the touch operation unit 12. That is, as shown in FIGS. 4 and 5, the back side of the touch operation unit 12 is an area on the touch pad 121 that is farthest from the wrist when the operator's arm and wrist are arranged on the armrest and palmrest, respectively. .

  The predetermined region R1 of the touch operation unit 12 is not limited to this, and may be an arbitrary partial region on the touch pad 121 or the entire region as described above. When the predetermined area R1 of the touch operation unit 12 is an arbitrary partial area on the touch pad 121, the areas on the touch pad 121 other than the predetermined area R1 are configured not to react to the touch operation. May be.

  On the other hand, the operation area R2 on the screen constituting the display unit 11 is set at the upper part of the screen so as to correspond when the predetermined area R1 is set on the back side of the touch pad 121. That is, the back side and the near side of the touch pad 121 correspond to the upper part and the lower part of the screen, respectively. Although it is most intuitive to make each correspond in this way, the correspondence relationship between the touch pad 121 and the screen constituting the display unit 11 is not limited to this. For example, the above correspondence relationship may be reversed, and the near side and the far side of the touch pad 121 may correspond to the upper part and the lower part of the screen, respectively. In this case, in order to superimpose the operator's operation hand on the entire touch operation unit 12, the operation region R2 on the screen constituting the display unit 11 is associated with the predetermined region R1 on the back side of the touch operation unit 12. , May be set at the lower part thereof.

  The control unit 14 associates the set position coordinates in the predetermined region R1 of the touch operation unit 12 with the position coordinates in the operation region R2 on the screen constituting the display unit 11. For example, when the predetermined area R1 of the touch operation unit 12 is a square area of a part or the whole of the touch pad 121, and the operation area R2 on the screen is a part of or a whole square area of the screen. think of. In this case, the control unit 14 associates the four vertices of the predetermined area R1 with the four vertices of the operation area R2. The control unit 14 can determine the correspondence between the position coordinates of each point located in the square area connecting the vertices by identifying the correspondence between the position coordinates of the four vertices.

  Such processing may be executed as calibration at an initial stage when the in-vehicle information processing system 10 is mounted on the vehicle, or may be executed as needed.

  FIG. 6 is a schematic diagram illustrating a state of image processing performed by the in-vehicle information processing system 10. FIG. 6A shows the state of the operator's operator performing a touch operation on the touch operation unit 12. FIG. 6B shows the state of the operator's operator hand superimposed on the screen constituting the display unit 11.

  The control unit 14 acquires image information captured by the camera from the imaging unit 13. As illustrated by a region R3 in FIG. 6, the captured image includes at least a part of an operator's hand performing a touch operation with the touch operation unit 12 and the touch operation unit 12, particularly the touch pad 121. That is, the imaging unit 13 images the positional relationship between the touch operation unit 12 and the operator's operating hand. In addition, as described above, the control unit 14 associates the position coordinates in the predetermined region R1 with the position coordinates in the operation region R2, so that it corresponds to the position of the operator's operating hand in the touch operation unit 12. In addition, at least a part of the operator can be displayed in a superimposed manner on the screen.

  When the operator's operator's hand is superimposed and displayed on the display unit 11, the control unit 14 performs image processing for extracting a part or the whole of the operator's operator's hand based on the above image. That is, the control unit 14 removes image information such as an external background from the contour of the operator's operator's hand. Note that by surrounding the periphery of the touch pad 121 with a black edge, the control unit 14 can easily extract the operator's operator based on the captured image.

  Further, when the captured image is a black and white image, the control unit 14 may or may not color the operating part by image processing. In order to further improve the reality of the operator's operator displayed on the display unit 11, the control unit 14 is preferably colored by image processing.

  On the other hand, when the captured image is a color image, it is preferable that the control unit 14 directly superimposes and displays the image on the screen in accordance with the actual color and brightness of the operator's operator. The control unit 14 is not limited to this, and in order to make it easier to visually recognize the display content behind, the controller 14 eliminates the color and brightness of the operator part, and instead performs image processing that adds a predetermined color, for example. You may go. Further, the control unit 14 may perform image processing in which the color and brightness of the operation part are excluded, and the operation part is completely colorless and transparent. In this case, the control unit 14 displays only a part near the contour of the operator on the screen.

  In the following description, it is assumed that the image captured by the imaging unit 13 is a color image, and the control unit 14 superimposes and displays the image on the screen as it is in accordance with the actual color and brightness of the operator's operator. That is, the description will be made assuming that the control unit 14 does not need to perform image processing relating to color, brightness, and the like.

  Based on the ratio of the size between the predetermined area R1 of the touch operation unit 12 in the captured image and the operation area R2 on the screen constituting the display unit 11, the control unit 14 performs the operation of the operator. Determine the hand display magnification. For example, when the predetermined area R1 of the touch operation unit 12 is a square area of a part or the whole of the touch pad 121, and the operation area R2 on the screen is a part of or a whole square area of the screen. think of. In this case, the control unit 14 calculates the ratio between the length of each side of the predetermined region R1 of the touch operation unit 12 and the length of each corresponding side of the operation region R2 on the screen. Based on the ratio, the control unit 14 determines the display magnification of the operator's operator's hand displayed on the display unit 11 in a superimposed manner.

  Based on the determined display magnification, the control unit 14 superimposes and displays the captured operator's operator on the display unit 11 in an enlarged or reduced state or as it is.

  The control unit 14 may make the display magnification of the operator's operator the same as the above ratio, or may be a different value based on the above ratio. The display magnification determination process may be executed simultaneously with the above-described calibration at the initial stage when the in-vehicle information processing system 10 is mounted on the vehicle, or may be executed as needed. Once the display magnification is determined, the control unit 14 may fix the magnification or change it according to the situation. For example, the control unit 14 may change the display magnification of the operator's operator between daytime and nighttime, or may appropriately change the display magnification according to the operator's setting.

  For example, when the display unit 11 is configured by a plurality of screens, the control unit 14 may change the display magnification of the operator's operator to be superimposed and displayed based on the size of the operation area R2 of each screen. Good. Without being limited to this, the control unit 14 derives, for example, an average value based on the size of the operation area R2 of each screen, and the operator's operator's hand that performs superimposition display based on the average value. The display magnification may be constant.

  The control unit 14 may change the display magnification of the operator's operator according to not only the size of the operation area R2 on the screen but also the content displayed on the screen. For example, when the display unit 11 displays a map or a function item and the operator operates each, the control unit 14 reduces the display magnification of the operator to be lower than usual so that the operator can easily operate. Then, the display unit 11 may be displayed in a superimposed manner.

  In addition, the control unit 14 may change the display magnification for each operator in order to match the size of the operator displayed superimposed on the display unit 11 between operators having different sizes of the operator. Good. On the other hand, the control unit 14 makes the display magnification of the operator superimposed on the display unit 11 constant among operators having different sizes of the operator, and matches the actual operator's size of the operator. May be displayed in a superimposed manner.

  The control unit 14 preferably performs image processing within a predetermined time based on the image captured by the imaging unit 13. The predetermined time is a time delay between the timing of the operation by the operator's actual operator and the timing of the movement of the operator's operator superimposed on the screen, and the operator is not aware of it. Means a time delay. That is, it is preferable that the control unit 14 completes the image processing within a time sufficiently shorter than a time delay in which the operator feels uncomfortable with the operation due to the reaction speed and the recognition ability of the operator. For example, it is preferable that the control unit 14 limit the image processing only to the above-described extraction of the operator's operator's hand imaged and adjustment of the display magnification.

  That is, the position coordinates in the predetermined region R1 of the touch operation unit 12 where the touch operation by the operator's operator is detected is not identified by image processing of the captured image by the imaging unit 13, but as described above. It is preferable that identification is performed based on detection information from the touch operation unit 12, particularly the touch pad 121.

  Although the control unit 14 is described as performing two image processes as described above, the present invention is not limited to this, and three or more image processes may be performed within a predetermined time. In this case, for example, the position coordinates in the predetermined region R1 of the touch operation unit 12 where the touch operation by the operator's operator is detected may be identified by image processing of the captured image by the imaging unit 13.

  When performing the above-described image processing, the control unit 14 refers to information about the predetermined region R1 of the touch operation unit 12 and the operation region R2 on the screen constituting the display unit 11 from the storage unit 15. That is, the control unit 14 refers to the information regarding the position coordinates in the predetermined region R1 of the touch pad 121 corresponding to the detection information. The control unit 14 refers to information regarding the position coordinates in the operation region R2 of each screen constituting the display unit 11. The control unit 14 refers to information regarding the display magnification of the operator's operator's hand to be displayed in a superimposed manner determined by calibration or the like.

  Below, with reference to FIG.7 and FIG.8, an example of operation which an operator performs using the vehicle-mounted information processing system 10 is demonstrated in detail.

  FIG. 7 is a schematic view illustrating the correspondence between the function items displayed on the screen configuring the display unit 11 and the touch pad 121. The control unit 14 determines at least one area in the operation region R2 on the screen, and displays on the screen at least one function item for the operator to control information related to the vehicle in association with each area. . For example, the control unit 14 further divides the operation region R2 set at the top of the screen into seven areas R21, R22, R23, R24, R25, R26, and R27, and “APPS”, “TEL”, “ The function items of “A / C”, “NAVI”, “AUDIO”, “HOME”, and “RETURN” are displayed corresponding to the respective areas. The seven areas R21 to R27 correspond to the areas R11, R12, R13, R14, R15, R16, and R17 in the predetermined region R1 of the touch operation unit 12, respectively.

  The number of areas divided by the control unit 14 is not limited to seven, and it is sufficient that at least one area is determined. Moreover, the function item displayed on each area is not limited to the above, and may be any item for controlling information on the vehicle. The display method of each function item is not limited to the display using characters illustrated in FIG. 7, and may be any display method that can be visually recognized by the operator, such as a display using an icon such as an arrow.

  For example, when the operator wants to activate car navigation, the operator performs a touch operation in the area R14. When the touch pad 121 detects a contact by a part of the operator's operating hand in the area R14, the control unit 14 highlights the item “NAVI” displayed in the area R24. At this time, the control unit 14 superimposes and displays the operator's operating hand at a corresponding position based on the image captured by the imaging unit 13.

  Subsequently, when the tact switch 122 is turned on by pressing the touch pad 121 by a part of the operator's operator, the control unit 14 determines the selection of the item “NAVI” displayed in the area R24. At this time, the control unit 14 displays the same movement on the screen in accordance with the pressing of a part of the operator's operator touching the area R14, for example, the index finger.

  Similarly, when the operator performs a touch operation in the area R11, the control unit 14 performs processing related to “APPS” displayed in the area R21. When the operator performs a touch operation in the area R12, the control unit 14 performs a process related to “TEL” displayed in the area R22. When the operator performs a touch operation in the area R13, the control unit 14 performs processing related to “A / C” displayed in the area R23. When the operator performs a touch operation in the area R15, the control unit 14 performs a process related to “AUDIO” displayed in the area R25. When the operator performs a touch operation in the area R16, the control unit 14 performs a process related to “HOME” displayed in the area R26. When the operator performs a touch operation in the area R17, the control unit 14 performs a process related to “RETURN” displayed in the area R27.

  In FIG. 7, the control unit 14 has been described as displaying each function item in association with each area defined in the operation region R2, but the present invention is not limited to this. As shown in FIG. 8, the control unit 14 may assign a function to each part of the operator's operating hand and select and execute various functions on the screen based on the operation of each part in the touch operation unit 12. In this case, the control unit 14 does not need to display each function item for each area in order to select and execute various functions.

  FIG. 8 is a schematic view illustrating an operation that is virtually performed by the operator's operator superimposed and displayed on the screen. Instead of displaying each function item on the screen, the control unit 14 assigns various functions to each part of the operator's operating hand, for example, each of three fingers. More specifically, the control unit 14 assigns the functions “RETURN”, “NAVI”, and “HOME” to the thumb, index finger, and middle finger of the operator's operation hand, respectively. The kind and number of fingers to which various functions are assigned are not limited to the above, and any function may be assigned to any number of fingers.

  For example, when the operator causes the index finger to contact the touch pad 121, the control unit 14 acquires a detection signal related to contact on the touch pad 121 from the touch operation unit 12. At this time, the control unit 14 identifies the type of finger touching the touch pad 121 based on the captured image acquired from the imaging unit 13. That is, the control unit 14 identifies the position coordinates of the contact on the touch pad 121 based on the detection information acquired from the touch operation unit 12, and the finger touching based on the image information acquired from the imaging unit 13. Identify the type.

  The control unit 14 may display an arbitrary signal on the screen when the contact by the operator's operating hand is identified. When the control unit 14 does not identify contact by the operator's operating hand, the control unit 14 does not display the signal on the screen. For example, in the former case, the control unit 14 may display the item “NAVI” in the vicinity of the index finger of the operator superimposed on the screen, or may display it with a predetermined icon. The control unit 14 may further highlight the displayed item. Further, the control unit 14 may display the circumference of the index finger so as to surround the index finger with a circle pointer or the like in order to feed back the identification of the contact with the index finger of the operator as visual information to the operator. As described above, the operator can easily recognize which finger is in contact with the touch pad 121 as visual information.

  Subsequently, when the operator depresses the touch pad 121 and the tact switch 122 is turned on, the control unit 14 determines the selection of the item “NAVI” displayed near the operator's index finger. At this time, the control unit 14 displays the same movement on the screen in accordance with the pressing of the index finger of the operator who is in contact with the corresponding position on the touch pad 121.

  Similarly, when the operator performs a touch operation with the thumb, the control unit 14 performs processing related to “RETURN”. When the operator performs a touch operation with the middle finger, the control unit 14 performs processing related to “HOME”.

  For example, when the operator causes the thumb, index finger, and middle finger to touch the touch pad 121 at the same time, the control unit 14 may simultaneously display all the above items on the screen. That is, the control unit 14 causes the items “RETURN”, “NAVI”, and “HOME” to be displayed simultaneously with characters or icons in the vicinity of the thumb, index finger, and middle finger of the operator superimposed on the screen. May be. The control unit 14 may further highlight each displayed item. In addition, the control unit 14 may display each finger so as to surround each finger with a circle pointer or the like in order to feed back the identification of the contact by each finger of the operator as visual information to the operator. Subsequently, when the operator presses the touch pad 121 with a specific finger, the control unit 14 determines the selection of the function assigned to the finger.

  In the above case, the control unit 14 may identify the pressed finger based on the image information captured by the imaging unit 13, for example. In addition, in order to identify the pressed finger with higher accuracy, the control unit 14 identifies the pressed finger based on detection signals from a plurality of tact switches 122 arranged immediately below the touch pad 121, for example. May be. That is, the control unit 14 identifies each position coordinate and the type of finger when the operator simultaneously touches each finger with the touch pad 121. Thereafter, the control unit 14 identifies a position on the touch pad 121 by the tact switch 122. Thereby, the control unit 14 identifies which finger function is to be selected.

  FIG. 9 is a flowchart showing an example of the operation of the in-vehicle information processing system 10.

  First, the control unit 14 performs calibration. That is, the control unit 14 associates the set position coordinates in the predetermined region R1 of the touch operation unit 12 with the position coordinates in the operation region R2 on the screen constituting the display unit 11 (step S10).

  Subsequently, the control unit 14 determines the display magnification of the operator's operator's hand to be superimposed and displayed on the display unit 11 by calibration or the like (step S11).

  Next, the control unit 14 determines whether the operator's operating hand is superimposed on the touch operation unit 12 based on the image captured by the imaging unit 13 (step S12).

  When it is determined that the operator's operating hand is superimposed on the touch operation unit 12, the control unit 14 proceeds to step S13. When it is determined that the operator's operating hand is not superimposed on the touch operation unit 12, the control unit 14 returns to step S12 again and waits until the operator's operating hand is superimposed.

  When it is determined that the operator's operator's hand is superimposed on the touch operation unit 12, the control unit 14 performs image processing for extracting part or all of the operator's operator's hand (step S13).

  Subsequently, the control unit 14 superimposes and displays the captured operator's hand based on the display magnification determined in step S11 (step S14).

  Subsequently, the control unit 14 determines whether detection information related to the touch operation is acquired from the touch operation unit 12 (step S15).

  When acquiring the detection information, the control unit 14 proceeds to step S16. When the detection information is not acquired, the control unit 14 returns to step S15 again and waits until the detection information is acquired.

  When acquiring the detection information, the control unit 14 highlights the function item corresponding to the position coordinate in the predetermined region R1 of the identified touch operation unit 12 (step S16).

  Subsequently, the control unit 14 determines whether a signal indicating that the tact switch 122 is on is acquired from the touch operation unit 12 (step S17).

  When the control unit 14 obtains an on-state signal, the control unit 14 proceeds to step S18. When the control unit 14 does not acquire the on-state signal, the control unit 14 returns to step S15 and determines again whether the detection information has been acquired.

  When acquiring the ON state signal, the control unit 14 determines the selection of the function item highlighted in Step S16 (Step S18).

  Thereafter, the flow ends.

  As described above, in the in-vehicle information processing system 10 according to the present embodiment, the operator's operator displayed on the display unit 11 virtually operates the information on the screen. Is possible. That is, the operator can access the screen in a state closer to the actual feeling. In addition, the operator can intuitively recognize the actual position of the operator and the positional relationship on the screen. Therefore, the in-vehicle information processing system 10 can reduce the time for the operator to watch the screen as compared with a conventional device that displays a pointer or the like.

  Since the in-vehicle information processing system 10 limits the time for performing image processing, it is possible to superimpose and display the operator's hand with a minimum delay. That is, the in-vehicle information processing system 10 can reduce a temporal movement shift between the actual operator and the operator superimposed and displayed on the screen. Thereby, the operator can operate the information displayed on the screen without a more uncomfortable feeling.

  Since the in-vehicle information processing system 10 performs image processing for extracting the operator's operating hand imaged by the imaging unit 13, the operator's operating hand can be faithfully superimposed on the screen. As a result, the operator can intuitively recognize the operating hand superimposed on the screen as his / her own hand.

  The in-vehicle information processing system 10 performs image processing to change the display magnification of the operator's operator imaged by the imaging unit 13, so that the operator's operator's operator is superimposed and displayed with an optimum size according to the screen. It is possible. As a result, the operator can easily see the display content displayed behind the operator while feeling the reality of the operator superimposed on the screen.

  The in-vehicle information processing system 10 can display an operator's operator in a superimposed manner with a minimum delay compared to the case where the position coordinates in the predetermined region R1 where the touch operation is detected are identified by image processing. Is possible. The in-vehicle information processing system 10 does not indirectly identify the position coordinates based on the image captured by the imaging unit 13, but directly identifies the position coordinates by the touch operation unit 12, and thus identifies the position coordinates with high accuracy. be able to. That is, since the in-vehicle information processing system 10 detects the position where the operator is actually touching by the touch operation unit 12, when the operator selects a function item displayed on the screen, a malfunction occurs. Less likely to cause.

  In the in-vehicle information processing system 10, the imaging unit 13 captures an image of the operator's entire operating hand, so that the operator can easily recognize that the operating hand superimposed on the screen is his / her hand. Furthermore, the operator can easily recognize which part of the operator is moving based on the display on the screen. Further, since the operator accurately recognizes the relationship between the actual position of the operator and the position on the screen, the operator can easily grasp the movement amount of the operator and the movable area on the screen.

  The in-vehicle information processing system 10 can accurately identify the operator's operator on the touch pad 121 by imaging the entire operator's operator with the imaging unit 13. That is, the in-vehicle information processing system 10 can accurately recognize the operator's operating hand as a human hand. Further, the in-vehicle information processing system 10 can identify each part of the operator's operator with higher accuracy. That is, the in-vehicle information processing system 10 can accurately identify which finger corresponds to the finger in the captured image. Further, the in-vehicle information processing system 10 can accurately identify the size of the entire operating hand, the proportion of the size of each part in the entire operating hand, and the like by imaging the entire operating hand. As a result, the in-vehicle information processing system 10 accurately matches the movement amount of the operator's operator's hand on the screen and the movable area on the screen with the actual movement of the operator's hand on the touch pad 121. It is possible to identify.

  The in-vehicle information processing system 10 sets a predetermined area R1 on the back side of the touch pad 121 and sets an operation area R2 on the upper part of the screen, so that the operator inevitably moves the entire touch operation unit 12. The operator's own hand is superimposed. At this time, the in-vehicle information processing system 10 can inevitably image the entire operator's operation hand by the imaging unit 13. In addition, by making the area on the touch pad 121 other than the predetermined area R1 non-responsive to the touch operation, the operator's consciousness is concentrated on the predetermined area R1. At this time, the vehicle-mounted information processing system 10 can capture the entire operator's operating hand more reliably.

  The in-vehicle information processing system 10 displays each function item corresponding to each area defined in the operation region R2, so that the operator can easily visually recognize the function item to be selected. In other words, the operator can intuitively recognize in which area on the touch operation unit 12 the touch operation should be performed when selecting a predetermined function item.

  On the other hand, the in-vehicle information processing system 10 can selectively execute various functions at arbitrary locations on the screen when assigning functions to each part of the operator's operator. That is, the in-vehicle information processing system 10 can minimize the operation range of the operator in the touch operation, and the operator can easily perform the touch operation.

  The in-vehicle information processing system 10 can clearly recognize which finger is in contact with the touch pad 121 as visual information by highlighting an item selected by contact by the operator. In addition, the operator can easily visually recognize at which position on the screen the operator displayed in a superimposed manner or which item is selected.

  The in-vehicle information processing system 10 gives a click feeling to the operator when the tact switch 122 is turned on, so that the operator can obtain tactile feedback by his / her own operation, and is more intuitive. Operation is possible. In other words, the tact switch 122 is used to confirm the selection, so that the operator can confirm the item selected by a natural action. In addition, the operator feels more reality with respect to the operator superimposed and displayed on the screen due to the reaction force acting on the finger. That is, the operator can easily feel the illusion that his / her actual operator is touching the screen directly.

  The in-vehicle information processing system 10 makes the operator's operator's hand to be superimposed and displayed semi-transparent, so that the operator can easily visually recognize the information displayed on the screen while ensuring the reality. That is, the operator can operate information displayed on the screen more intuitively.

  The in-vehicle information processing system 10 can more easily image the entire operator's operating hand by installing the imaging unit 13 above the touch operation unit 12.

  It will be apparent to those skilled in the art that the present invention can be realized in other predetermined forms other than the above-described embodiments without departing from the spirit or essential characteristics thereof. Accordingly, the above description is illustrative and not restrictive. The scope of the invention is defined by the appended claims rather than by the foregoing description. Some of all changes that fall within the equivalent scope shall be included therein.

DESCRIPTION OF SYMBOLS 10 In-vehicle information processing system 11 Display unit 12 Touch operation unit 121 Touch pad 122 Tact switch 13 Imaging unit 14 Control unit 15 Storage unit R1 Predetermined regions R11, R12, R13, R14, R15, R16, R17 Area R2 Operation region R21 , R22, R23, R24, R25, R26, R27 Area R3 area

Claims (5)

A display unit having at least one screen;
A touch operation unit that detects contact by an operator's operator;
An imaging unit that images at least a part of the operator and the touch operation unit;
Corresponding the position coordinates in the operation area on the screen and the position coordinates in the predetermined area of the touch operation unit, and performing image processing within a predetermined time based on the image captured by the imaging unit, A control unit that superimposes and displays at least a part of the operating hand on the screen,
In-vehicle information processing system. The imaging unit images a positional relationship between the touch operation unit and the operator,
The control unit causes at least a part of the operation hand to be superimposed on the screen so as to correspond to the position of the operation hand in the touch operation unit.
The in-vehicle information processing system according to claim 1. The control unit performs image processing for extracting at least a part of the operating hand based on the image captured by the imaging unit.
The in-vehicle information processing system according to claim 1 or 2. The control unit performs image processing for changing a display magnification of the operator based on a ratio of a size between the predetermined region of the touch operation unit and the operation region on the screen.
The in-vehicle information processing system according to any one of claims 1 to 3. The control unit identifies position coordinates in the predetermined area of the touch operation unit where contact by the operator is detected based on detection information from the touch operation unit.
The in-vehicle information processing system according to any one of claims 1 to 4.

Images