WO2018230526A1

WO2018230526A1 - Input system and input method

Info

Publication number: WO2018230526A1
Application number: PCT/JP2018/022305
Authority: WO
Inventors: 中井　潤; 隆大河平
Original assignee: パナソニックＩｐマネジメント株式会社
Priority date: 2017-06-13
Filing date: 2018-06-12
Publication date: 2018-12-20
Also published as: JP2020126282A

Abstract

This input system has: a display; a sensor; an indicator; and an optical plate. The display presents information to an occupant in a vehicle. The sensor recognizes an occupant's gesture carried out in a gesture recognition space set near the display. The indicator is installed outside the gesture recognition space. The optical plate is installed between the gesture recognition space and the indicator, and forms, in the gesture recognition space, a gesture guide image that has been displayed on the indicator. The display displays information reflecting the gesture recognized by the sensor.

Description

Input system and input method

This disclosure relates to an input system and an input method using gesture input in a vehicle.

In recent years, development of aerial displays that display images in the air has been underway. For example, a technique for forming a real image in the air at an equal distance on the opposite side of the real image by passing the image through a special optical plate has been developed (see, for example, Patent Document 1). The optical plate forms one plate by crossing and joining two optical members formed by laminating strip-shaped mirror glass. Light emitted from a display or the like is reflected by two orthogonal optical members and forms an image again in the air on the opposite side.

In recent years, the development of gesture operations that allow operation input by hand movement without touching the touch panel or buttons has been promoted. Since the gesture operation does not need to touch the operation target, the operation target can be operated from a remote position.

Japanese Patent No. 4865088

This disclosure provides a technique that allows a gesture operation to be comfortably performed in a vehicle.

The input system according to one aspect of the present disclosure includes a display, a sensor, a display, and an optical plate. The display presents information to passengers in the vehicle. The sensor recognizes a passenger's gesture operation performed in a gesture recognition space set in the vicinity of the display. The indicator is installed outside the gesture recognition space. The optical plate is installed between the gesture recognition space and the display, and forms a gesture guide image displayed on the display in the gesture recognition space. The display displays information reflecting the gesture operation recognized by the sensor.

Note that any combination of the above components, the expression of the present disclosure converted between an apparatus, a system, a method, a program, a recording medium on which the program is recorded, an autonomous driving vehicle equipped with the same, and the like are also disclosed in the present disclosure. It is effective as an embodiment of

According to the present disclosure, it is possible to perform a gesture operation comfortably in the vehicle.

FIG. 1A is a diagram illustrating a configuration example of gesture input. FIG. 1B is a diagram illustrating a configuration example of gesture input. FIG. 2A is a diagram illustrating a configuration example in which gesture input and an aerial display are combined. FIG. 2B is a diagram illustrating a configuration example in which gesture input and an aerial display are combined. FIG. 3A is a diagram illustrating an installation example in the vehicle of the input system according to the embodiment of the present disclosure. FIG. 3B is a diagram illustrating an installation example in the vehicle of the input system according to the embodiment of the present disclosure. FIG. 3C is a diagram illustrating an installation example in the vehicle of the input system according to the embodiment of the present disclosure. FIG. 4 is a diagram illustrating an example of an installation method of the optical plate and the display. FIG. 5 is a block diagram illustrating a configuration of the input system according to the embodiment of the present disclosure. FIG. 6A is a diagram illustrating a specific example of the gesture operation using the input system according to the embodiment of the present disclosure. FIG. 6B is a diagram illustrating a specific example of the gesture operation using the input system according to the embodiment of the present disclosure. FIG. 6C is a diagram illustrating a specific example of the gesture operation using the input system according to the embodiment of the present disclosure. FIG. 7 is a flowchart showing an operation of the input system according to the embodiment of the present disclosure. FIG. 8A is a diagram illustrating an installation example of the input system according to the modification in the vehicle. FIG. 8B is a diagram illustrating an installation example of the input system according to the modification in the vehicle.

Prior to the description of the embodiment of the present disclosure, the problems in the prior art will be briefly described. In recent years, the electrification of vehicles is accelerating, and opportunities for touch operations and button operations in vehicles are increasing. In the vehicle, the passenger basically needs to operate various devices while sitting on the seat, and depending on the position of the operation unit, it is necessary to greatly extend his hand. Therefore, if a gesture operation is introduced into the operation of equipment in the vehicle, it can be expected that the burden on the operation of the passenger will be reduced. However, since the gesture operation is performed in the air, the operation input is not recognized by the device if it is out of the recognition range of the gesture operation. This situation is a great stress for passengers who perform gesture operations.

FIG. 1A and FIG. 1B are diagrams showing a configuration example of gesture input. FIG. 1A is a configuration example in which a camera is used as the gesture detection sensor 50. In this configuration example, a gesture detection camera is installed below the information display 30. The direction of the camera is set so that a space that is a predetermined distance away from the screen of the display 30 is within the angle of view. A space that falls within the angle of view of the camera in front of the screen is a gesture recognition space S1.

FIG. 1B is a configuration example in which a highly sensitive capacitive touch panel sensor is used as the gesture detection sensor 50. In the configuration example, a capacitive touch panel sensor is installed on the surface of the display 30. A non-contact space close to the surface of the display 30 is a gesture recognition space S1. The user can perform a hover operation in the gesture recognition space S1. The gesture recognition space S1 becomes wider as the sensitivity of the touch panel sensor is higher. Note that an infrared sensor, an ultrasonic sensor, or the like can be used instead of the camera or the capacitive touch panel sensor.

The gesture input, particularly the gesture input of the configuration example shown in FIG. 1A has the following problems. (1) It is difficult for the operator to perform a gesture operation in the range of the gesture recognition space S1. The gesture recognition space S1 is an empty space as seen from the operator, and it is difficult for the operator to grasp the space range. (2) Displaying a gesture operation guide on the display 30 is troublesome for non-operators. (3) In order to operate the display 30 displayed in front of the eyes or the device in front of the eyes, it is natural to directly touch and operate, and it is difficult to appeal the merit of the gesture operation.

Therefore, in the embodiment of the present disclosure, the gesture input and the aerial display are used in combination. Specifically, using the aerial display, the gesture operation guide video is displayed in the air in the gesture recognition space S1.

FIG. 2A and FIG. 2B are diagrams showing a configuration example in which gesture input and an aerial display are combined. FIG. 2A is a configuration example in which a camera is used as the gesture detection sensor 50, and FIG. 2B is a configuration example in which a capacitive touch panel sensor is used as the gesture detection sensor 50.

The aerial display is realized by the display 41 and the optical plate 42. The optical plate 42 subjected to the special processing described above forms an image displayed on the display 41 installed on one side of the optical plate 42 in a line-symmetric space position on the opposite side of the optical plate 42. Can be made. The installation positions of the optical plate 42 and the display device 41 are determined based on the relationship between the position of the guide image I1 to be displayed in the gesture recognition space S1 set in front of the display 30 and the reflection.

The combination of gesture input and aerial display has the following advantages. (1) By displaying an aerial gesture operation guide that is visible only to the operator, it is easy to perform a gesture operation within the range of the gesture recognition space S1. (2) By displaying an aerial gesture operation guide that is visible only to the operator, there is no trouble for anyone other than the operator. (3) Since only the gesture operation guide is displayed, there is little visual trouble for the operator. (4) It is possible to intuitively operate devices other than the display displayed in front of the eyes and the device in front of the eyes. (5) The amount of information to be presented can be ensured by installing an aerial display for gesture operation guide separately from the display 30 for displaying normal information.

The aerial display technology as of 2017 has lower luminance and lower resolution than general liquid crystal displays and organic EL (organic electro-luminescence: OEL) displays, and is suitable for displaying small characters and pictures. Not. In addition, since the image floats in the air in a semi-transparent state, the contrast is low, and the visibility is lowered in a bright environment. Therefore, the aerial display is preferably used in combination with the normal display 30. Since the aerial display technology as of 2017 has a narrower viewing angle than a general display, there is an advantage in terms of presenting information only to a specific operator.

Hereinafter, an example in which an input system combining a gesture input and an aerial display is used in a vehicle will be described. The above-described aerial display requires the display 41 and the optical plate 42, and a space for storing the display 41 and the optical plate 42 is required below the display 30.

3A to 3C are diagrams illustrating an installation example of the input system according to the embodiment of the present disclosure in a vehicle. FIG. 3A is a schematic view of the vicinity of the driver's seat in the vehicle. A display 30 for presenting information is installed on the dashboard 4. For example, a center display of a car navigation apparatus can be used as the information presentation display 30. In addition, the display of a smart phone or a tablet fixed to the holder on the dashboard 4 may be used.

The gesture recognition space S <b> 1 is set to a space below the display surface of the display 30. Specifically, it is installed on the inclined surface of the center console 5 that extends while tilting forward and downward from the installation position of the display 30 of the dashboard 4. In FIG. 3A, the gesture recognition space S1 is set on the upper side of the inclined surface, but may be set on the center or lower side of the inclined surface. The steering wheel 3a is installed on the right side of the inclined surface of the center console 5, and the driver can easily reach the gesture recognition space S1 with the left hand.

FIG. 3B is a schematic diagram showing the positional relationship between the display 30 and the optical plate 42. An optical plate 42 is installed in parallel with the inclined surface of the center console 5. A guide image Ia with a left arrow is displayed in the air above the optical plate 42. FIG. 3C is a schematic view of the positional relationship among the display 30, the optical plate 42, and the display device 41 viewed from the side surface direction. From the viewpoint E1 of the driver, the guide video Ia appears to appear on the inclined surface of the center console 5.

FIG. 4 is a diagram illustrating an example of an installation method of the optical plate 42 and the display device 41. The display device 41 is housed and installed in a storage box 45. The inside of the storage box 45 is subjected to low reflection processing. A normal liquid crystal display module (LCM) can be used for the display device 41. A light control film (LCF) 43 is attached to the surface of the display 41 on the display surface side. The light control film 43 is a film that suppresses diffused light and improves the parallelism of light, and can improve luminance and visibility when the display device 41 is viewed from the front. The optical plate 42 is installed at the position of the upper lid of the storage box 45.

Thus, by covering the display 41 installed on one side of the optical plate 42 with the storage box 45, the periphery of the display 41 can be darkened and an image is formed in the air on the opposite side of the optical plate 42. The visibility of the guide video can be improved. Further, by attaching a light control film 43 to the display 41 and applying a low reflection process to the inside of the storage box 45, it is possible to prevent a ghost image from being formed in the air. The storage box 45 shown in FIG. 4 is installed inside the inclined surface of the center console 5 shown in FIG. 3A, for example.

FIG. 5 is a block diagram illustrating a configuration of the input system 2 according to the embodiment of the present disclosure. The input system 2 includes a control device 10, a display 30, an aerial image display device 40, and a gesture detection sensor 50. The aerial image display device 40 includes a display 41 and an optical plate 42 as main members. The light control film 43 shown in FIG. 4 is not essential and can be omitted.

The control device 10 includes a processing unit 11, an input / output unit (I / O unit) 12, and a recording unit 13. The processing unit 11 includes a screen control unit 111, a guide control unit 112, a detection information acquisition unit 113, an operation content determination unit 114, and a device control unit 115. The function of the processing unit 11 can be realized by cooperation of hardware resources and software resources. Hardware resources include CPU (central processing unit), GPU (graphics processing unit), DSP (digital signal processor), FPGA (field-programmable gate array), ROM (read-only memory), RAM (random-access memory), Other LSIs (large-scale integration) can be used. Programs such as operating system, application, firmware, etc. can be used as software resources. The recording unit 13 is a nonvolatile memory, and includes a recording medium such as a NAND flash memory chip, an SSD (solid-state drive), and an HDD (hard disk drive).

The control device 10 may be mounted in a dedicated housing, or may be mounted in a head unit such as a car navigation device or display audio. In the case of the latter, the form which adds the board | substrate which mounted the function of the control apparatus 10 in those housing | casings may be sufficient, and the form which utilizes those existing hardware resources by a time division may be sufficient. Moreover, the form which utilizes the hardware resource of the information equipment brought in from the outside, such as a smart phone and a tablet, may be sufficient.

The display 30 is a display installed in the vehicle interior as described above, and a liquid crystal display or an organic EL display can be used. The gesture detection sensor 50 is a sensor for recognizing a passenger's gesture operation performed in the gesture recognition space S1 set in the vicinity of the display 30 in the vehicle interior. As described above, a camera, a non-contact type touch panel, or the like can be used.

The I / O unit 12 outputs an image signal supplied from the processing unit 11 to the display 30, outputs an image signal supplied from the processing unit 11 to the display 41, and a detection signal supplied from the gesture detection sensor 50. Is output to the processing unit 11.

The screen control unit 111 generates all image data to be displayed on the display 30, and outputs and displays the image data on the display 30. The guide control unit 112 generates image data to be displayed in the air as a gesture guide video in the gesture recognition space S1, and outputs and displays the image data on the display device 41. For example, the guide control unit 112 displays a symbol image indicating the operation content in the air as a gesture guide image. For example, graphic symbol marks such as circles, triangles, squares, crosses, arrows, and crosses may be displayed in the air, or icons representing the operation contents may be displayed in the air.

Further, the guide control unit 112 may display an image defining the range of the gesture recognition space S1 in the air as a gesture guide image. For example, the image of the frame of the gesture recognition space S1 may be an aerial image. In addition, a point image may be displayed in the air at the position of each vertex in the gesture recognition space S1.

The detection information acquisition unit 113 acquires detection information detected by the gesture detection sensor 50. For example, the image data of the gesture recognition space S1 photographed by the camera is acquired. The operation content determination unit 114 determines the operation content based on the detection information acquired by the detection information acquisition unit 113. For example, a hand is detected as an object from the acquired image, and the detected movement of the hand is followed. The operation content determination unit 114 specifies the gesture operation content from the detected hand movement. Note that the hand search range in the image may be narrowed down to a nearby region where the guide video is displayed.

The screen control unit 111 causes the display 30 to display an image reflecting the gesture operation determined by the operation content determination unit 114. The screen control unit 111 displays an image (a mark, an icon, a pictogram, a symbol, or the like) indicating that the gesture operation has been accepted. In addition, the screen control unit 111 displays an image (for example, an icon during processing and an icon indicating completion of processing) indicating the state of device operation corresponding to the accepted gesture operation.

The device control unit 115 executes the operation content corresponding to the gesture operation determined by the operation content determination unit 114 for the device in the vehicle. For example, an operation of a car navigation device, an operation of display audio, an operation of an air conditioner, an operation of a power window, an operation of turning on / off a room lamp, and the like are executed. In addition, you may perform driving operation of vehicles, such as turning on / off of a blinker, a gear shift, a sound of a horn, passing, and start / end of a wiper.

6A to 6C are diagrams illustrating specific examples of the gesture operation using the input system 2 according to the embodiment of the present disclosure. FIG. 6A is an example in the case of executing a function that is not displayed on a device in front of the operator's eyes (in the vicinity of performing the gesture operation).

6A, a display 30 is a display of the car navigation device, and the volume of the voice guidance of the car navigation device is changed by a gesture operation. In the gesture recognition space S1 above the optical plate 42, a left guide image Ia and a right guide image Ib are imaged in the air. The volume is decreased by the gesture of the operator receiving the guide image Ia indicated by the left arrow in the left direction, and the volume is increased by the gesture of receiving the guide image Ib indicated by the right arrow in the right direction. In FIG. 6A, the volume of the volume bar 30a displayed on the display 30 is reduced by lowering the volume by moving the guide image Ia indicated by the left arrow in the left direction.

FIG. 6B shows an example of operating a device that is not in front of the operator. In the gesture recognition space S1 above the optical plate 42, a guide image Ic with an upward arrow is imaged in the air. The right blinker blinks with a gesture in which the operator receives the upward arrow guide image Ic upward. When the right turn signal blinks, an icon 30b indicating that the right turn signal blinks is displayed on the screen of the display 30. Although not shown, while the right blinker is blinking, a guide image indicated by a down arrow is formed in the air, and the right blinker is turned off by a gesture in which the operator receives the guide image indicated by the down arrow downward.

FIG. 6C shows an example of operating a device that is in front of the operator but is difficult to reach. In FIG. 6C, the display 30 is a display of the car navigation apparatus, and flicks or swipes the map displayed on the display 30. A map 30c is displayed on the screen of the display 30, and a left guide image Ia and a right guide image Ib are formed in the air in the gesture recognition space S1 above the optical plate 42. The operator flicks or swipes the map 30c in the left direction with a gesture that causes the left arrow guide image Ia to move left, and flicks or swipes the map 30c in the right direction with a gesture that causes the right arrow guide image Ib to move in the right direction. Is done. A flick operation is performed when the hand movement is less than a predetermined speed, and a swipe operation is performed when the hand movement is greater than the predetermined speed. In FIG. 6C, the map 30c is flicked to the left by receiving the guide image Ia of the left arrow in the left direction.

FIG. 7 is a flowchart showing the operation of the input system 2 according to the embodiment of the present disclosure. The guide control unit 112 displays a predetermined guide image in the air in the gesture recognition space S1 (S10). The detection information acquisition unit 113 acquires detection information based on an operator's gesture operation detected by the gesture detection sensor 50 (S11). The operation content determination unit 114 identifies the operation content based on the acquired detection information (S12). The screen control unit 111 causes the display 30 to display an image indicating completion of reception of the specified operation content (S13). The device control unit 115 controls the device according to the specified operation content (S14).

As described above, according to the present embodiment, by displaying the guide video in the gesture recognition space S1, the gesture operation within the gesture recognition space S1 is facilitated, and the probability that the gesture operation will be missed is increased. It can be greatly reduced. Therefore, a passenger in the vehicle can perform a gesture operation comfortably.

Also, if the guide video is displayed in the air toward the driver, the guide video cannot be seen by the passenger sitting in the passenger seat due to the narrow viewing angle characteristics. Therefore, the visual annoyance of passengers other than the subject does not occur. Further, if the guide video is displayed in the air only during the gesture operation, the visual inconvenience of the subject does not occur. Moreover, the amount of information presentation can be ensured by using together with the existing display for information display. Only the aerial display limits the amount of information presented.

The present disclosure has been described based on the embodiments. Those skilled in the art will understand that these embodiments are exemplifications, and that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present disclosure. By the way.

8A and 8B are diagrams showing an installation example of the input system 2 according to the modification in the vehicle. In the modification shown in FIG. 8A, the gesture recognition space S <b> 1 is installed on the center portion of the dashboard 4. The aerial image display device 40 is installed inside the center of the dashboard 4. The information presentation display 30 is installed at a position close to the gesture recognition space S <b> 1 on the inclined surface of the center console 5. For example, a display of a head unit such as display audio can be used as the information presentation display 30.

8B, an aerial video display device 40 having a display 41 and an optical plate 42 is embedded in the upper part of the joint portion of the steering column 3b with the steering wheel 3a. The guide image I1 is imaged in the back of the steering wheel 3a (above the steering column 3b) as viewed from the driver. The display in the instrument panel 6 can be used as a display for presenting information.

Moreover, although the example which uses a liquid crystal display module for the display 41 was demonstrated in FIG. 4, in the use with which a display image is limited, you may create the display 41 by installing several light emitting diodes on a board | substrate. . For example, when only a bright spot is displayed at the vertex of the gesture recognition space S1, the display device 41 can be created simply by installing eight light emitting diodes at predetermined positions on the substrate.

When using the display 41 that does not cause attenuation due to passing through the liquid crystal layer, the aerial image may appear dazzling in the passenger compartment at night. The guide control unit 112 may adjust the luminance of the light emitting diode according to the brightness in the vehicle. The brightness in the vehicle is determined based on illuminance information detected by an illuminance sensor (not shown) installed in the vehicle. The illuminance at the current position may be acquired from a server of the Japan Meteorological Agency or a private weather company via a wireless communication network. The guide control unit 112 decreases the luminance of the light emitting diode as the illuminance in the vehicle is lower. The luminance of the light emitting diode can be controlled by adjusting the drive current or the PWM (pulse width modulation) ratio. When a liquid crystal display module is used for the display device 41, the original display luminance is low, so that it is not necessary to reduce the luminance at night. However, luminance control when using the liquid crystal display module is not excluded.

Note that the embodiment may be specified by the following items.

[Item 1]
The input system (2) includes a display (30), a sensor (50), a display (41), and an optical plate (42). The display (30) presents information to passengers in the vehicle. The sensor (50) recognizes the occupant's gesture operation performed in the gesture recognition space (S1) set in the vicinity of the display (30). The display (41) is installed outside the gesture recognition space (S1). The optical plate (42) is installed between the gesture recognition space (S1) and the display (41), and images the gesture guide image displayed on the display (41) in the gesture recognition space (S1). The display (30) displays information reflecting the gesture operation recognized by the sensor (50).

Thereby, the gesture operation within the range of the gesture recognition space (S1) is facilitated, and the operability of the gesture operation can be improved.

[Item 2]
In the input system (2) according to item 1, the display device (41) displays a symbol image indicating the operation content as a gesture guide image, and the optical plate (42) displays the symbol image as a gesture recognition space (S1). To form an image.

This makes it possible to further improve the operability by gesture operation.

[Item 3]
In the input system (2) according to item 1, the display (41) displays an image defining the range of the gesture recognition space (S1) as a gesture guide image, and the optical plate (42) is a gesture recognition space. An image defining the range of (S1) is imaged in the gesture recognition space (S1).

This makes it easier to recognize the outer edge of the gesture recognition space (S1).

[Item 4]
In the input system (2) according to any one of items 1 to 3, the display (30) is a center display (30) installed on the dashboard (4), and the gesture recognition space (S1) is a center display. It is set in the space below the front with respect to the display surface of (30).

Thereby, compared with the case where the center display (30) is touch-operated, the distance for reaching the hand is shortened, so that the operability is improved.

[Item 5]
In the input system (2) according to any one of items 1 to 3, the gesture recognition space (S1) is set on the center of the dashboard (4), and the display (30) is the center console (5). It is installed at a position close to the gesture recognition space (S1).

Thereby, since the gesture guide image is displayed at the same height as the windshield, the visibility of the gesture guide image during driving can be improved.

[Item 6]
The input method includes a step of recognizing a passenger's gesture operation performed in a gesture recognition space (S1) set in the vicinity of a display (30) for presenting information to a passenger in the vehicle (1). . Moreover, the input method uses an optical plate (42) installed between the gesture recognition space (S1) and the display (41) installed outside the gesture recognition space (S1). 41) imaging the gesture guide image displayed in 41) in the gesture recognition space (S1). Further, the input method includes a step of displaying information reflecting the recognized gesture operation on the display (30).

The present disclosure relates to a technique capable of performing a gesture operation comfortably in a vehicle, and is particularly useful as an input system and an input method.

2 Input System

3a Steering Wheel

3b Steering Column 4 Dashboard 5 Center Console 6 Instrument Panel 10 Control Unit 11 Processing Unit 12 I / O Unit 13 Recording Unit 30 Display 30a Volume Bar 30b Icon 30c Map 40 Aerial Video Display Device 41 Indicator 42 Optical plate 43 Light control film 45 Storage box 50 Gesture detection sensor 111 Screen control unit 112 Guide control unit 113 Detection information acquisition unit 114 Operation content determination unit 115 Device control unit E1 Viewpoint I1, Ia, Ib, Ic Guide video S1 Gesture recognition space

Claims

A display for presenting information to passengers in the vehicle;
A sensor for recognizing a passenger's gesture operation performed in a gesture recognition space set in the vicinity of the display;
An indicator installed outside the gesture recognition space;
An optical plate that is installed between the gesture recognition space and the display and forms a gesture guide image displayed on the display on the gesture recognition space;
The display displays information reflecting a gesture operation recognized by the sensor;
Input system.
The display device displays a symbol image indicating the operation content as the gesture guide image,
The optical plate images the symbol image in the gesture recognition space;
The input system according to claim 1.
The indicator displays an image defining a range of the gesture recognition space as the gesture guide image,
The optical plate forms an image defining a range of the gesture recognition space on the gesture recognition space;
The input system according to claim 1.
The display is a center display installed on the dashboard,
The gesture recognition space is set in a space in front of and below the display surface of the center display.
The input system according to claim 1.
The gesture recognition space is set on the center of the dashboard,
The display is installed at a position close to the gesture recognition space of a center console.
The input system according to claim 1.
Recognizing a passenger's gesture operation performed in a gesture recognition space set in the vicinity of a display for presenting information to a passenger in the vehicle;
Using the optical plate installed between the gesture recognition space and a display installed outside the gesture recognition space, the gesture guide image displayed on the display is imaged on the gesture recognition space. Steps,
Displaying information reflecting the recognized gesture operation on the display,
input method.