JP2020126282A - Input system and input method - Google Patents

Abstract

To realize a comfortable gesture operation input in a vehicle.SOLUTION: A display 30 presents information to passengers in a vehicle. A gesture detection sensor 50 recognizes a passenger's gesture operation performed in a gesture recognition space set near the display 30. A display 41 is installed outside the gesture recognition space. An optical plate 42 is installed between the gesture recognition space and the display 41, and images the gesture guide image displayed on the display 41 in the gesture recognition space. The display 30 displays information that reflects gesture operation recognized by the gesture detection sensor 50.SELECTED DRAWING: Figure 5

Description

The present invention relates to an input system and an input method using gesture input in a vehicle.

In recent years, development of an aerial display for displaying an image in the air has been advanced. For example, a technique has been developed in which a real image is formed in the air at an equal distance on the opposite side of the real image by passing the image through a special optical plate (see, for example, Patent Document 1). The optical plate is formed by laminating two optical members formed by laminating strip-shaped mirror-like glasses so as to cross each other and joining them. Light emitted from a display or the like is reflected by two optical members that are orthogonal to each other and forms an image again in the air on the opposite side.

Further, in recent years, development of a gesture operation in which an operation input is performed by a motion of a hand without touching a touch panel or a button has been advanced. Since the gesture operation does not need to touch the operation target, the operation target can be operated from a distant position.

Patent 4865088

In recent years, the electrification of vehicles has been accelerated, and the opportunities for touch operation and button operation inside the vehicle are increasing. In the vehicle, the occupant basically needs to operate various devices while sitting on a seat, and depending on the position of the operation part, the occupant needs to extend his/her hand greatly. Therefore, if the gesture operation is introduced into the operation of the device in the vehicle, it can be expected that the burden on the operation of the passenger can be reduced.

However, since the gesture operation is performed in the air, the operation input is not recognized by the device if it is out of the recognition range of the gesture operation. The situation becomes a great stress to the passenger who is performing the gesture operation.

The present invention has been made in view of such a situation, and an object thereof is to provide a technique capable of comfortably performing a gesture operation in a vehicle.

In order to solve the above problems, an input system according to an aspect of the present invention is a display for presenting information to an occupant in a vehicle, and an occupant performed in a gesture recognition space set near the display. Sensor for recognizing the gesture operation, a display installed outside the gesture recognition space, and a gesture guide image displayed on the display installed between the gesture recognition space and the display. And an optical plate for forming an image in the recognition space. The display displays information that reflects the gesture operation recognized by the sensor.

It should be noted that any combination of the above constituents, the expression of the present invention converted between an apparatus, a system, a method, a program, a recording medium storing the program, an automatic driving vehicle equipped with them, and the like are also included in the present invention. Is effective as a mode.

According to the present invention, it is possible to comfortably perform a gesture operation in a vehicle.

1A and 1B are diagrams showing a configuration example of gesture input. 2A and 2B are diagrams showing a configuration example in which a gesture input and an aerial display are combined. FIGS. 3A to 3C are diagrams showing an installation example in a vehicle of the input system according to the embodiment of the present invention. It is a figure which shows an example of the installation method of an optical plate and a display. It is a block diagram which shows the structure of the input system which concerns on embodiment of this invention. 6A to 6C are diagrams showing a specific example of the gesture operation using the input system according to the embodiment of the present invention. It is a flow chart which shows operation of the input system concerning an embodiment of the invention. FIG. 8A and FIG. 8B are diagrams showing an installation example in the vehicle of the input system according to the modified example.

1A and 1B are diagrams showing a configuration example of gesture input. FIG. 1A is a configuration example in which a camera is used as the gesture detection sensor 50. In the configuration example, a gesture detection camera is installed below the display 30 for displaying information. The orientation of the camera is set such that a space away from the screen of the display 30 by a predetermined distance is within the angle of view. The space within the angle of view of the camera in front of the screen is the gesture recognition space S1.

FIG. 1B is a configuration example in which a highly sensitive capacitive touch panel sensor is used as the gesture detection sensor 50. In the configuration example, a capacitive touch panel sensor is installed on the surface of the display 30. The non-contact space that is close to the surface of the display 30 is the gesture recognition space S1. The user can perform a hover operation in this gesture recognition space S1. The gesture recognition space S1 becomes wider as the sensitivity of the touch panel sensor is higher. Note that an infrared sensor, an ultrasonic sensor, or the like can be used instead of the camera or the capacitive touch panel sensor.

Gesture input, particularly the gesture input of the configuration example shown in FIG. 1A, has the following problems. (1) It is difficult for the operator to perform a gesture operation within the range of the gesture recognition space S1. The gesture recognition space S1 is an empty space seen from the operator, and it is difficult for the operator to grasp the spatial range. (2) Displaying a guide for gesture operation on the display 30 is troublesome for anyone other than the operator. (3) In order to operate the display 30 displayed in front of the eyes and the device in front of the eyes, it is natural to directly touch and operate, and it is difficult to appeal the merit of the gesture operation.

Therefore, in the embodiment of the present invention, the gesture input and the aerial display are used in combination. Specifically, an aerial display is used to display a gesture operation guide image in the air in the gesture recognition space S1.

2A and 2B are diagrams showing a configuration example in which a gesture input and an aerial display are combined. FIG. 2A is a configuration example in which a camera is used as the gesture detection sensor 50, and FIG. 2B is a configuration example in which a capacitive touch panel sensor is used as the gesture detection sensor 50.

The aerial display is realized by the display 41 and the optical plate 42. The optical plate 42 that has been subjected to the special processing described above forms the image displayed on the display 41 installed on one side of the optical plate 42 at a spatially symmetrical position on the opposite side of the optical plate 42. Can be made. The installation positions of the optical plate 42 and the display 41 are determined based on the relationship between the position of the guide image I1 to be displayed in the gesture recognition space S1 set in front of the display 30 and the reflection.

Combining gesture input and aerial display has the following advantages. (1) By displaying the aerial gesture operation guide visible only to the operator, the gesture operation within the range of the gesture recognition space S1 becomes easy. (2) By displaying the aerial gesture operation guide visible only to the operator, no annoyance occurs to anyone other than the operator. (3) Since only the gesture operation guide is displayed, the operator is less visually annoyed. (4) It becomes possible to intuitively operate devices other than the display displayed in front of the eyes and the device in front of the eyes. (5) The amount of information to be presented can be secured by providing an aerial display for a gesture operation guide in addition to the display 30 for displaying normal information.

The aerial display technology as of 2017 has lower brightness and lower resolution than general liquid crystal displays and organic EL displays, and is not suitable for displaying small characters and pictures. In addition, since the image floats in the air in the semi-transparent state, the contrast is low, and the visibility is reduced in a bright environment. Therefore, the aerial display is preferably used together with the normal display 30. Since the aerial display technology as of 2017 has a narrower viewing angle than a general display, it is advantageous from the viewpoint of presenting information only to a specific operator.

Hereinafter, an example of using an input system combining a gesture input and an aerial display in a vehicle will be described. The above-described aerial display requires the display 41 and the optical plate 42, and a space below the display 30 to store the display 41 and the optical plate 42 is required.

FIGS. 3A to 3C are diagrams showing an installation example in a vehicle of the input system according to the embodiment of the present invention. FIG. 3A is a schematic view of the vicinity of the driver's seat in the vehicle. A display 30 for presenting information is installed on the dashboard 4. As the display 30 for presenting information, for example, a center display of a car navigation device can be used. It may be a display of a smartphone or a tablet fixed to a holder on the dashboard 4.

The gesture recognition space S1 is set to a space below and near the display surface of the display 30. Specifically, it is installed on the inclined surface of the center console 5 extending from the installation position of the display 30 of the dashboard 4 to the lower front side while inclining. Although the gesture recognition space S1 is set on the upper side of the inclined surface in FIG. 3A, it may be set on the central portion or the lower side of the inclined surface. The steering wheel 3a is installed on the right side of the inclined surface of the center console 5, and the driver can easily reach the gesture recognition space S1 with his/her left hand.

FIG. 3B is a schematic diagram showing the positional relationship between the display 30 and the optical plate 42. The optical plate 42 is installed parallel to the inclined surface of the center console 5. The guide image Ia of the left arrow is displayed in the air above the optical plate 42. FIG. 3C is a schematic view of the positional relationship among the display 30, the optical plate 42, and the display 41 as seen from the side surface direction. From the driver's viewpoint E1, the guide image Ia appears to stand out on the inclined surface of the center console 5.

FIG. 4 is a diagram showing an example of a method of installing the optical plate 42 and the display 41. The display 41 is installed in the storage box 45. The inside of the storage box 45 is subjected to low reflection processing. A normal liquid crystal display module (LCM) can be used for the display 41. A light control film (LCF) 43 is attached to the display surface of the display 41. The light control film 43 is a film that suppresses diffused light and improves the parallelism of light, and can improve the brightness and visibility when the display 41 is viewed from the front. The optical plate 42 is installed at the position of the upper lid of the storage box 45.

Thus, by covering the display 41 installed on one side of the optical plate 42 with the storage box 45, the periphery of the display 41 can be darkened, and an image is formed in the air on the opposite side of the optical plate 42. The visibility of the image I1 can be improved. Further, by attaching the light control film 43 to the display 41 and applying the low reflection processing to the inside of the storage box 45, it is possible to prevent the formation of a ghost image in the air. The storage box 45 shown in FIG. 4 is installed, for example, inside the inclined surface of the center console 5 shown in FIG.

FIG. 5 is a block diagram showing the configuration of the input system 2 according to the embodiment of the present invention. The input system 2 includes a control device 10, a display 30, an aerial image display device 40, and a gesture detection sensor 50. The aerial image display device 40 includes a display 41 and an optical plate 42 as main components. The light control film 43 shown in FIG. 4 is not essential and can be omitted.

The control device 10 includes a processing unit 11, an I/O unit 12, and a recording unit 13. The processing unit 11 includes a screen control unit 111, a guide control unit 112, a detection information acquisition unit 113, an operation content determination unit 114, and a device control unit 115. The function of the processing unit 11 can be realized by cooperation of hardware resources and software resources. CPU, GPU, DSP, FPGA, ROM, RAM and other LSIs can be used as hardware resources. Programs such as an operating system, applications, and firmware can be used as software resources. The recording unit 13 is a non-volatile memory and includes a recording medium such as a NAND flash memory chip, SSD, and HDD.

The control device 10 may be mounted in a dedicated housing or may be mounted in a head unit such as a car navigation device or a display audio. In the latter case, a board on which the function of the control device 10 is mounted may be added in the housing, or the existing hardware resources thereof may be utilized in a time-sharing manner. Further, it is also possible to use a hardware resource of an information device brought in from the outside such as a smartphone or a tablet.

The display 30 is a display installed in the vehicle compartment as described above, and a liquid crystal display or an organic EL display can be used. The gesture detection sensor 50 is a sensor that recognizes a gesture operation performed by a passenger in a gesture recognition space S1 set near the display 30 in the vehicle interior. As described above, a camera, a non-contact type touch panel or the like can be used.

The I/O unit 12 outputs the image signal supplied from the processing unit 11 to the display 30, the image signal supplied from the processing unit 11 to the display 41, and the detection signal supplied from the gesture detection sensor 50. Is output to the processing unit 11.

The screen control unit 111 generates image data to be displayed on the display 30 and outputs the image data to the display 30 for display. The guide control unit 112 generates image data to be displayed in the air as a gesture guide image in the gesture recognition space S1 and outputs the image data to the display 41 for display. For example, the guide control unit 112 displays, as a gesture guide image, a symbol image indicating the operation content in the air. For example, graphic symbol marks such as circles, triangles, squares, crosses, arrows, and crosses may be displayed in the air, or icons representing the operation content may be displayed in the air.

Further, the guide control unit 112 may display an image defining the range of the gesture recognition space S1 in the air as the gesture guide image. For example, the image of the frame of the gesture recognition space S1 may be an aerial image. In addition, an image of a point may be displayed in the air at the position of each vertex of the gesture recognition space S1.

The detection information acquisition unit 113 acquires the detection information detected by the gesture detection sensor 50. For example, the image data of the gesture recognition space S1 captured by the camera is acquired. The operation content determination unit 114 determines the operation content based on the detection information acquired by the detection information acquisition unit 113. For example, a hand is detected as an object from the acquired image, and the detected movement of the hand is followed. The operation content determination unit 114 identifies the gesture operation content from the detected hand movement. The search range of the hand in the image may be narrowed down to the area near the guide image.

The screen control unit 111 causes the display 30 to display an image that reflects the gesture operation determined by the operation content determination unit 114. The screen control unit 111 displays an image (mark, icon, pictogram, symbol, etc.) indicating completion of acceptance of the gesture operation. Further, the screen control unit 111 displays an image (for example, a processing-in-progress icon, a processing-completion icon) indicating the status of the device operation corresponding to the received gesture operation.

The device control unit 115 executes the operation content corresponding to the gesture operation determined by the operation content determination unit 114 with respect to the device in the vehicle. For example, the operation of the car navigation device, the operation of the display audio, the operation of the air conditioner, the operation of the power window, the operation of turning on/off the room lamp, and the like are executed. In addition, driving operation of the vehicle such as turning on/off of a turn signal, gear shift, horn ringing, passing, start/end of a wiper may be executed.

FIGS. 6A to 6C are diagrams showing a specific example of the gesture operation using the input system 2 according to the embodiment of the present invention. FIG. 6A is an example of a case where a device that is in front of the operator's eyes (around where the gesture operation is performed) executes a function that is not displayed.

In FIG. 6A, a display 30 is a display of a car navigation device, and the volume of voice guidance of the car navigation device is changed by a gesture operation. In the gesture recognition space S1 above the optical plate 42, a left arrow guide image Ia and a right arrow guide image Ib are imaged in the air. The operator lowers the volume with a gesture of leftwardly guiding the guide image Ia, and increases the volume with a gesture of rightwardly guiding the right guide image Ib. In FIG. 6A, by moving the guide image Ia indicated by the left arrow to the left, the volume is reduced and the scale of the volume bar 30a displayed on the display 30 is reduced.

FIG. 6B shows an example of operating a device that is not in front of the operator's eyes. In the gesture recognition space S1 above the optical plate 42, the guide image Ic of the up arrow is imaged in the air. The operator blinks the guide image Ic of the up arrow in the upward direction, and the right blinker blinks. When the right blinker blinks, an icon 30b indicating that the right blinker is blinking is displayed on the screen of the display 30. Although not shown, while the right blinker is blinking, the guide image of the down arrow is imaged in the air, and the right blinker is turned off by the gesture that the operator glances down the guide image of the down arrow.

FIG. 6C is an example in the case of operating a device that is hard to reach, but in front of the operator. In FIG. 6C, the display 30 is a display of the car navigation device, and flicks or swipes the map displayed on the display 30. A map 30c is displayed on the screen of the display 30, and a guide image Ia indicated by the left arrow and a guide image Ib indicated by the right arrow are imaged in the air in the gesture recognition space S1 above the optical plate 42. The operator flicks or swipes the map 30c leftward with the gesture of leftwardly guiding the guide image Ia, and flicks or swipes the map 30c rightward with the gesture of rightwardly guiding the guide image Ib. To be done. When the hand movement is less than the predetermined speed, the flick operation is performed, and when the hand movement is more than the predetermined speed, the swipe operation is performed. In FIG. 6C, the map 30c is flicked to the left by moving the guide image Ia indicated by the left arrow to the left.

FIG. 7 is a flowchart showing the operation of the input system 2 according to the embodiment of the present invention. The guide control unit 112 displays a predetermined guide image in the air in the gesture recognition space S1 (S10). The detection information acquisition unit 113 acquires detection information based on the gesture operation of the operator detected by the gesture detection sensor 50 (S11). The operation content determination unit 114 identifies the operation content based on the acquired detection information (S12). The screen control unit 111 causes the display 30 to display an image indicating completion of acceptance of the specified operation content (S13). The device control unit 115 controls the device according to the specified operation content (S14).

As described above, according to the present embodiment, by displaying the guide image in the gesture recognition space S1, the gesture operation within the range of the gesture recognition space S1 is facilitated, and the probability that the gesture operation will be missed is increased. It can be greatly reduced. Therefore, the passenger in the vehicle can comfortably perform the gesture operation.

Further, when the guide image is displayed in the air in the direction of the driver, a passenger sitting in the passenger seat cannot see the guide image due to the characteristics of the narrow viewing angle. Therefore, the visual annoyance of passengers other than the target person does not occur. Further, if the guide image is displayed in the air only during the gesture operation, the visual annoyance of the subject does not occur. In addition, the amount of information presented can be secured by using it together with an existing display for displaying information. The amount of information presented is limited only by the aerial display.

The present invention has been described above based on the embodiment. It should be understood by those skilled in the art that these embodiments are mere examples, and that various modifications can be made to the combinations of their respective constituent elements and respective processing processes, and that such modifications are also within the scope of the present invention. By the way.

FIGS. 8A and 8B are diagrams showing an installation example of the input system 2 according to the modified example in a vehicle. In the modification shown in FIG. 8A, the gesture recognition space S1 is installed on the central portion of the dashboard 4. The aerial image display device 40 is installed inside the central portion of the dashboard 4. The display 30 for presenting information is installed at a position close to the gesture recognition space S1 on the inclined surface of the center console 5. As the display 30 for presenting information, for example, a display of a head unit such as a display audio can be used.

In the modification shown in FIG. 8B, the aerial image display device 40 including the display 41 and the optical plate 42 is embedded in the upper portion of the joint portion of the steering column 3b with the steering wheel 3a. The guide image I1 is formed behind the steering wheel 3a (above the steering column 3b) when viewed from the driver. A display in the instrument panel 6 can be used as a display for presenting information.

Further, in FIG. 4, an example in which a liquid crystal display module is used for the display 41 has been described, but in applications where the display image is limited, a plurality of light emitting diodes may be installed on the substrate to form the display 41. .. For example, when only the bright spots are displayed at the vertices of the gesture recognition space S1, the display 41 can be created by installing eight light emitting diodes at predetermined positions on the substrate.

When using the display device 41 in which the attenuation due to the passage of the liquid crystal layer does not occur, the aerial image may look dazzling in the vehicle interior at night. The guide controller 112 may adjust the brightness of the light emitting diode according to the brightness inside the vehicle. The brightness inside the vehicle is determined based on the illuminance information detected by an illuminance sensor (not shown) installed in the vehicle. The illuminance at the current position may be acquired from a server of the Meteorological Agency or a private weather company via a wireless communication network. The guide controller 112 lowers the brightness of the light emitting diode as the illuminance inside the vehicle is lower. The brightness of the light emitting diode can be controlled by adjusting the drive current or the PWM ratio. When the liquid crystal display module is used for the display 41, the original display brightness is low, so that it is not necessary to reduce the brightness at night, but the brightness control when using the liquid crystal display module is not excluded.

The embodiment may be specified by the following items.

[Item 1]
A display (30) for presenting information to passengers in the vehicle,
A sensor (50) for recognizing a gesture operation of an occupant performed in a gesture recognition space (S1) set near the display (30);
A display device (41) installed outside the gesture recognition space (S1),
An optical plate (42) installed between the gesture recognition space (S1) and the display device (41) for forming an image of the gesture guide image displayed on the display device (41) in the gesture recognition space (S1). And
The display (30) displays information reflecting the gesture operation recognized by the sensor (50),
Input system (2).
This facilitates the gesture operation within the range of the gesture recognition space (S1), and improves the operability of the gesture operation.
[Item 2]
The display device (41) displays a symbol image showing the operation content as the gesture guide image,
The optical plate (42) forms an image of the symbol image in the gesture recognition space (S1),
The input system (2) according to item 1.
Thereby, the operability by the gesture operation can be further improved.
[Item 3]
The display (41) displays, as the gesture guide image, an image defining a range of the gesture recognition space (S1),
The optical plate (42) images an image defining the range of the gesture recognition space (S1) in the gesture recognition space (S1).
The input system (2) according to item 1.
This makes it easier to recognize the extension of the gesture recognition space (S1).
[Item 4]
The display (30) is a center display (30) installed on the dashboard (4),
The gesture recognition space (S1) is set in a space below and near the display surface of the center display (30).
The input system (2) according to any one of Items 1 to 3.
As a result, as compared with the case where the center display (30) is touch-operated, the distance to reach the hand is shortened, so that the operability is improved.
[Item 5]
The gesture recognition space (S1) is set on the central part of the dashboard (4),
The display (30) is installed at a position close to the gesture recognition space (S1) of the center console (5),
The input system (2) according to any one of Items 1 to 3.
Accordingly, the gesture guide image is displayed at the same height as the windshield, and thus the visibility of the gesture guide image during driving can be improved.
[Item 6]
Recognizing the gesture operation of the passenger performed in the gesture recognition space (S1) set near the display (30) for presenting information to the passenger in the vehicle (1),
An optical plate (42) installed between the gesture recognition space (S1) and the display (41) installed outside the gesture recognition space (S1) is used to display the display (41). Forming an image of the displayed gesture guide image in the gesture recognition space (S1);
Causing the display (30) to display information that reflects the recognized gesture operation.
input method.

2 input system, 3a steering wheel, 3b steering column, 4 dashboard, 5 center console, 6 instrument panel, 10 control device, 11 processing unit, 12 I/O unit, 13 recording unit, 14 communication unit, 30 display, 40 aerial image display device, 41 display device, 42 optical plate, 43 light control film, 45 storage box, 50 gesture detection sensor, 111 screen control unit, 112 guide control unit, 113 detection information acquisition unit, 114 operation content determination unit, 115 Device control unit.

Claims (6)

A display for presenting information to passengers in the vehicle,
A sensor that recognizes a gesture operation of an occupant performed in a gesture recognition space set near the display,
An indicator installed outside the gesture recognition space,
An optical plate that is installed between the gesture recognition space and the display, and forms an image of the gesture guide image displayed on the display in the gesture recognition space,
The display displays information that reflects the gesture operation recognized by the sensor,
Input system. As the gesture guide image, the display displays a symbol image showing the operation content,
The optical plate forms an image of the symbol image in the gesture recognition space,
The input system according to claim 1. The display, as the gesture guide image, displays an image defining a range of the gesture recognition space,
The optical plate forms an image in the gesture recognition space that defines a range of the gesture recognition space.
The input system according to claim 1. The display is a center display installed on the dashboard,
The gesture recognition space is set in a space below and near the display surface of the center display,
The input system according to any one of claims 1 to 3. The gesture recognition space is set on the central part of the dashboard,
The display is installed at a position close to the gesture recognition space of the center console,
The input system according to any one of claims 1 to 3. A step of recognizing a passenger's gesture operation performed in a gesture recognition space set in the vicinity of a display for presenting information to a passenger in a vehicle;
An optical plate installed between the gesture recognition space and a display installed outside the gesture recognition space is used to form a gesture guide image displayed on the display in the gesture recognition space. Steps,
Causing the display to display information that reflects the recognized gesture operation.
input method.

Images