WO2020085521A1

WO2020085521A1 - Input/output device and vehicle comprising same

Info

Publication number: WO2020085521A1
Application number: PCT/KR2018/012526
Authority: WO
Inventors: 이동규; 박도연; 오주현
Original assignee: 엘지전자 주식회사
Priority date: 2018-10-23
Filing date: 2018-10-23
Publication date: 2020-04-30

Abstract

The present invention relates to an input/output device mounted on a vehicle. The input/output device comprises: an image sensor configured to sense a hand of a passenger riding in the vehicle; a haptic module configured to generate a tactile effect on the hand by using vibrations of the air; and a processor for controlling the haptic module to generate different types of tactile effects on the basis of the position of the hand.

Description

Input / output devices and vehicles containing them

The present invention relates to an input / output device that enables information to be exchanged between a vehicle and a passenger on a vehicle, and a vehicle including the same.

Vehicle means a means of transportation that can move people or luggage using kinetic energy. Examples of vehicles are automobiles and motorcycles.

For the safety and convenience of users of the vehicle, various sensors and devices are provided in the vehicle, and the functions of the vehicle are diversified.

The function of the vehicle may be divided into a convenience function for promoting the convenience of the driver and a safety function for promoting the safety of the driver and / or pedestrian.

First, the convenience function has a development motivation related to driver convenience, such as giving the vehicle an infotainment (information + entertainment) function, supporting a partial autonomous driving function, or helping secure a driver's view such as a night vision or a blind spot. . For example, adaptive cruise control (ACC), smart parking assist system (SPAS), night vision (NV), head up display (HUD), around view monitor (around view monitor, AVM), and adaptive headlight system (AHS).

The safety function is a technology that ensures driver safety and / or pedestrian safety, lane departure warning system (LDWS), lane keeping assist system (LKAS), and autonomous emergency braking, AEB) functions.

In order to further improve the above-mentioned convenience functions and safety functions, communication technologies specialized for vehicles have been developed. For example, V2I (Vehicle to Infrastructure) that enables communication between the vehicle and the infrastructure, V2V (Vehicle to Vehicle) that enables communication between the vehicle and the vehicle, and V2X (Vehicle to Everything) that enables communication between the vehicle and the vehicle. There is this.

Although the functions of the vehicle are diversified, the input / output device that receives input from the user or outputs various information to the user is limited to the touch screen. Since the function desired by the passenger cannot be quickly selected by the touch screen alone, various user input devices formed to execute the function immediately are provided in the vehicle. For example, the user input device may include a button type, a lever type, and a jog wheel type input device.

As the functions of the vehicle are diversified, a number of user input devices are inevitably provided in the vehicle. For example, as a heat seat is developed, a heat seat button associated with on / off of the heat seat is additionally provided in the vehicle. Although the heat seat is a function mainly used only in winter, there is a problem that the heat seat button must always occupy a predetermined space at one point of the vehicle.

User demand for a new type of user interface is increasing, which allows passengers to quickly select necessary functions while efficiently configuring space inside the vehicle.

The present invention aims to solve the above and other problems.

One object of the present invention is to provide a new type of input / output device capable of sensing a non-contact user input and providing non-contact feedback.

One object of the present invention is to provide an input / output device that allows an occupant to quickly and easily select a specific function while efficiently using a space.

One object of the present invention is to provide an input / output device capable of quickly executing a specific function based on a gesture of a passenger, while performing authentication for the passenger.

The present invention provides an input / output device and a vehicle equipped with the input / output device.

The input / output device may include an image sensor configured to sense a hand of a passenger on the vehicle; A haptic module configured to generate a tactile effect on the hand using vibration of air; And a processor that controls the haptic module to generate different tactile effects based on the position of the hand.

According to one embodiment, a three-dimensional space separated from the input / output device is defined, and the processor may determine whether to generate the tactile effect based on the position of the hand with respect to the three-dimensional space.

According to an embodiment, at least one of the size and position of the 3D space may be differently defined according to the passenger.

According to one embodiment, the processor may adjust the light output intensity of the image sensor based on the three-dimensional space.

According to one embodiment, the processor generates a tactile effect of the first method when the hand is located in the first area of the 3D space, and the first method when the hand is located in the second area of the 3D space. The haptic module may be controlled to generate a second tactile effect.

According to an embodiment of the present disclosure, the processor sets a first part and a second part in the hand based on the position of the hand, and a vibration of the first size occurs in the first part, and the second part The haptic module may be controlled to generate vibrations of a second size different from the first size.

According to one embodiment, the input / output device further includes a communication unit, and the processor outputs visual information for guiding at least one of the three-dimensional space and the position of the hand to the display provided in the vehicle from the display. The display can be controlled as much as possible.

According to an embodiment, the processor or the image sensor determines whether an object entering the 3D space is the hand, and the processor does not generate the tactile effect when the object is not the hand. The haptic module can be controlled so as not to.

According to an embodiment, the processor may control the haptic module to generate vibrations of different sizes according to a distance between the hand and the image sensor.

According to an embodiment, the input / output device further includes a gesture sensor that tracks a gesture defined by at least one movement of the passenger's hand and finger, and the processor uses the gesture sensor to position the hand. Can detect.

According to an embodiment, a three-dimensional space separated from the input / output device is defined, and the image sensor may be selectively activated according to whether the hand is located in the three-dimensional space.

According to one embodiment, the image sensor detects the vein information located on the hand, and the processor selects a function corresponding to the gesture in response to the gesture satisfying a reference condition, and It can be determined whether or not to execute based on the vein information detected by the image sensor.

According to an embodiment of the present disclosure, the input / output device further includes a communication unit, and when the authentication of the vein information fails, the processor includes a user registration screen configured to register the vein information on a display provided in the vehicle. The communication unit can be controlled to be output.

According to an embodiment, when the gesture satisfies a reference condition, the processor may execute different functions according to vein information sensed by the image sensor.

According to an embodiment, when the vein authentication of the passenger using the image sensor is performed once, the processor may deactivate the image sensor until the passenger leaves the vehicle.

According to an embodiment, a three-dimensional space separated from the input / output device is defined, and when the new passenger's hand, which has not been vein authenticated, enters the three-dimensional space, the processor detects the deactivated image sensor. Activation and vein authentication for the hand of the new passenger may be performed.

According to an embodiment, the input / output device further includes a communication unit, and the processor, when the vein authentication is performed, enables a user mode corresponding to the passenger to be executed in one or more electronic devices provided in the vehicle. , It is possible to control the communication unit.

According to an embodiment, the input / output device further includes a communication unit, and the processor receives a personal profile corresponding to the passenger from an external device, and when vein authentication is performed using the image sensor, the personal profile is added to the personal profile. Based on this, the communication unit is controlled so that a menu selectable by the passenger is output from a display provided in the vehicle, and functions included in the menu may be varied according to the passenger.

According to one embodiment, the input / output device further includes a touch panel configured to sense the touch of the passenger, the haptic module is spaced apart from the edge of the touch panel, and the image sensor is located below the touch panel. The overlapping arrangement, the processor may simultaneously receive the passenger's touch input through the touch panel and the passenger's vein information through the image sensor.

Meanwhile, the input / output device according to the present invention includes an image sensor configured to detect a vein located in the hand of a passenger on the vehicle; A haptic module configured to generate a tactile effect on the hand using vibration of air; And a processor that performs vein authentication for the passenger using the image sensor, and grants different privileges to the passenger according to the vein authentication, wherein a three-dimensional space is defined based on the image sensor, and The processor generates the tactile effect of the first method when the hand is located in the first area of the 3D space, and the tactile effect of the second method when the hand is located in the second area of the 3D space. The haptic module may be controlled, and when the vein authentication is completed, the haptic module may be controlled to end generation of a tactile effect that was occurring.

The effects of the input / output device and the vehicle including the same according to the present invention will be described as follows.

According to the present invention, the vein authentication can be used to provide a different user interface to the passenger, and the user can move the hand to the correct position because it generates a tactile effect that guides the vein authentication area to facilitate vein authentication. have.

1 is a block diagram illustrating an input / output device according to an embodiment of the present invention

2A and 2B are exemplary views for explaining the arrangement of some components described in FIG. 1.

3 is a flowchart illustrating a control method by the input / output device of FIG. 1.

4A, 4B, and 4C are exemplary views for describing a 3D space of an input / output device

4D and 4E are conceptual views for explaining haptic intensity in a 3D space

5 is a flow chart for explaining a control method for generating different tactile effects on the hands of passengers.

6 is an exemplary view for explaining an example according to the control method of FIG. 5

7 is a flowchart for explaining a control method of an input / output device related to a 3D space

8A and 8B are exemplary views for explaining different 3D spaces according to passengers.

9 is a flowchart for explaining a control method of an input / output device that performs various controls depending on whether an object entering a 3D space is a hand;

10 is a flowchart for explaining a control method of an input / output device that performs various controls based on gesture input.

11A and 11B are exemplary views for describing the embodiments of FIG. 10.

12 is a flowchart for explaining a control method of an input / output device related to vein authentication.

13 is a flowchart illustrating a control method of an input / output device providing a user interface using vein authentication.

Hereinafter, exemplary embodiments disclosed herein will be described in detail with reference to the accompanying drawings, but the same or similar elements are assigned the same reference numbers regardless of the reference numerals, and overlapping descriptions thereof will be omitted. The suffixes "modules" and "parts" for the components used in the following description are given or mixed only considering the ease of writing the specification, and do not have meanings or roles distinguished from each other in themselves. In addition, in describing the embodiments disclosed in the present specification, when it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed herein, detailed descriptions thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical spirit disclosed in the specification is not limited by the accompanying drawings, and all modifications included in the spirit and technical scope of the present invention , It should be understood to include equivalents or substitutes.

Terms including ordinal numbers such as first and second may be used to describe various components, but the components are not limited by the terms. The terms are used only for the purpose of distinguishing one component from other components.

When an element is said to be "connected" or "connected" to another component, it is understood that other components may be directly connected to or connected to the other component, but other components may exist in the middle. It should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that no other component exists in the middle.

Singular expressions include plural expressions unless the context clearly indicates otherwise.

In this application, the terms "comprises" or "have" are intended to indicate the presence of features, numbers, steps, actions, components, parts or combinations thereof described herein, one or more other features. It should be understood that the existence or addition possibilities of fields or numbers, steps, operations, components, parts or combinations thereof are not excluded in advance.

1 is a block diagram illustrating an input / output device according to an embodiment of the present invention.

The present invention provides a new type of input / output device capable of receiving input from a passenger on a vehicle and outputting various information related to the vehicle to the passenger, and a vehicle having the input / output device.

The input / output device 100 is provided in the vehicle 1 and may be formed of an independent device detachable from the vehicle 1 or integrally installed in the vehicle 1 to be a part of the vehicle 1.

Hereinafter, for convenience of description, the input / output device 100 will be described as having a separate configuration independent from the vehicle 1. However, this is only an embodiment of the present invention, and the operation and control method of all the input / output devices 100 described in the present specification includes at least one electronic control unit provided in the vehicle 1. Can be performed by.

Referring to FIG. 1, the input / output device 100 includes at least one of a communication unit 110, an image sensor 130, a haptic module 150, a gesture sensor 160, a touch panel 170, and a processor 190. Includes.

The communication unit 110 is configured to perform communication with various components provided in the vehicle 1. For example, the communication unit 110 may receive various information provided through a CAN (controller are network). As another example, the communication unit 110 may perform communication with all devices capable of communication such as a vehicle, a mobile terminal and a server, and other vehicles. This may be called V2X (Vehicle to everything) communication. V2X communication can be defined as a technology that exchanges or shares information such as traffic conditions while communicating with road infrastructure and other vehicles while driving.

The communication unit 110 is configured to perform communication with one or more displays 10 provided in the vehicle 1.

Furthermore, the communication unit 110 may receive information related to driving of the vehicle from most devices provided in the vehicle 1. Information transmitted from the vehicle 1 to the input / output device 100 is referred to as “vehicle driving information”.

The vehicle driving information includes vehicle information and surrounding information of the vehicle. Based on the frame of the vehicle 1, information related to the inside of the vehicle may be defined as vehicle information and information related to the outside of the vehicle as surrounding information.

The vehicle information means information about the vehicle itself. For example, the vehicle information includes the vehicle's driving speed, driving direction, acceleration, angular velocity, position (GPS), weight, vehicle occupancy, vehicle braking force, vehicle maximum braking force, air pressure of each wheel, and centrifugal force applied to the vehicle. , Driving mode of the vehicle (whether autonomous driving mode or manual driving), parking mode of the vehicle (autonomous parking mode, automatic parking mode, manual parking mode), whether the user is in the vehicle and information related to the user It can contain.

The surrounding information refers to information related to other objects located within a predetermined range around the vehicle and information related to the outside of the vehicle. For example, the condition of the road surface where the vehicle is driving (friction force), weather, distance from the vehicle in front (or rear), the relative speed of the vehicle in front (or rear), and the curvature of the curve when the driving lane is a curve, vehicle Ambient brightness, information related to an object existing in a reference area (constant area) based on a vehicle, whether an object enters / departs from the predetermined area, whether a user exists around the vehicle, and information related to the user (for example, For example, whether the user is an authenticated user).

In addition, the surrounding information includes: ambient brightness, temperature, sun location, object information (people, other vehicles, signs, etc.) located in the vicinity, type of road surface being driven, topographical features, lane information, and driving lane ) Information, information necessary for autonomous driving / autonomous parking / automatic parking / manual parking mode.

In addition, the surrounding information includes an object (object) existing around the vehicle and a distance to the vehicle 1, a possibility of collision, the type of the object, a parking space in which the vehicle can park, and an object for identifying the parking space (for example, , Parking lines, strings, other vehicles, walls, etc.).

The vehicle driving information is not limited to the example described above, and may include all information generated from components provided in the vehicle 1.

The processor 190 may control one or more displays 10 provided in the vehicle 1 using the communication unit 110.

Specifically, the processor 830 may determine whether at least one condition is satisfied among a plurality of preset conditions based on vehicle driving information received through the communication unit 110. Depending on the satisfied condition, the processor 830 may control the one or more displays in different ways.

In relation to a preset condition, the processor 830 may detect that an event has occurred in the electrical equipment and / or application provided in the vehicle 1 and determine whether the detected event satisfies the preset condition. At this time, the processor 830 may detect that an event has occurred from information received through the communication unit 110.

The application is a concept including a widget or a home launcher, and means all types of programs that can be driven in the vehicle 1. Therefore, the application may be a web browser, a video playback, message transmission and reception, schedule management, a program that performs the functions of the application update.

Furthermore, the application includes Forward Collision Warning (FCW), Blind Spot Detection (BSD), Lane Departure Warning (LDW), Pedestrian Detection (PD), Curve Speed Warning (Curve Speed Warning, CSW) and turn by turn navigation (TBT).

For example, when an event occurs, when there is a missed call, when there is an application to be updated, when a message arrives, start on, start off, autonomous driving on / off, display activation key is pressed (LCD awake key), an alarm, an incoming call, and a missed notification.

As another example, the event occurrence may be a case in which an alert set in ADAS (advanced driver assistance system) or a function set in ADAS is performed. For example, when a forward collision warning occurs, when a blind spot detection occurs, when a lane departure warning occurs, a lane keeping warning When assist warning occurs, an event may be considered to occur when an emergency braking function is performed.

As another example, when changing from a forward gear to a reverse gear, when an acceleration greater than a predetermined value occurs, when a deceleration greater than a predetermined value occurs, when a power unit is changed from an internal combustion engine to a motor, or at a motor Even if it is changed to an internal combustion engine, it can be considered that the event occurred.

In addition, it can be considered that an event has occurred even when various ECUs provided in the vehicle 1 perform a specific function.

When the generated event is satisfied with a preset condition, the processor 190 controls the communication unit 110 such that information corresponding to the satisfied condition is displayed on the one or more displays 1.

The image sensor 130 is configured to generate a hand image corresponding to the hand of a passenger who has boarded the vehicle 1. Specifically, it is possible to detect the hand of a passenger located in a predetermined area and acquire a hand image of the hand.

A biometric recognition may be performed by a hand image obtained by the image sensor 130, a gesture of a passenger may be extracted and used as a user input, and the position of the hand may be calculated as a coordinate value.

For example, a biometric technology may be applied to the image sensor 130.

Here, the biometric technology refers to a technology for identifying or identifying an individual's identity using biometric information. The image sensor 130 may acquire biometric information such as a fingerprint, a face, an iris, and a vein, and implement a biometric technology based on the acquired biometric information.

For example, the image sensor 130 may acquire a hand image, extract a vein pattern from the hand image, and identify an individual using the vein pattern.

The vein recognition technology is a technology for identifying an individual using a vein pattern image acquired through infrared rays. Human veins are located under a complex shape directly under the skin, and the vein pattern is known to be individual and does not change with age.

Because the veins are inside the body, there is little risk of theft or forgery, and the condition of the skin's surface has no effect on perception. Since the image sensor 130 to which the vein recognition technology is applied uses infrared rays, it is made of a non-contact device with high user convenience. The vein pattern is stable and can be found relatively clearly, so a low resolution camera can be used.

The image sensor 130 may extract vein information from the hand image to enable vein recognition. Passenger identification may be performed based on the vein information.

The image sensor 130 may track the position of the hand using the hand image. The image sensor 130 may calculate the position of the hand as a 2D and / or 3D coordinate value using the hand image.

The haptic module 150 generates various tactile effects that the user can feel. A typical example of the tactile effect generated by the haptic module 150 may be vibration. The intensity and pattern of vibration generated in the haptic module 150 may be controlled by a user's selection or setting of the processor 190. For example, the haptic module 150 may synthesize and output different vibrations or sequentially output them.

In addition to vibration, the haptic module 150 includes a pin arrangement vertically moving with respect to a contact skin surface, stimulation of the ejection force or suction force of air through a jet or intake, a grazing on the skin surface, contact with an electrode, and electrostatic force. Various tactile effects, such as the effect by and the effect by reproducing the feeling of cold and warm using an element capable of absorbing heat or generating heat can be generated.

The haptic module 150 may not only transmit a tactile effect through direct contact, but may also implement the above so that the user can feel the tactile effect through muscle sensations such as fingers or arms.

The haptic module 150 may generate a tactile effect by generating vibration of air using ultrasonic waves. Specifically, in order to generate a tactile effect on the skin, it is possible to output ultrasound having a frequency of 40 kHz or more, which is a threshold value that can be felt by receptors in the skin.

Two or more haptic modules 150 may be provided according to a configuration aspect of the input / output device 100.

The gesture sensor 160 may track a gesture defined by at least one movement of a passenger's hand and finger. Specifically, the gesture sensor 160 may detect the gesture and determine which one of the preset gestures corresponds to the gesture. Furthermore, the gesture sensor 160 may calculate the position of the hand as a coordinate value of 2D and / or 3D.

The function of the gesture sensor 160 may be implemented by the image sensor 130.

The touch panel 170 may be configured in a film form having a touch pattern. The image sensor 130 may be disposed on the rear surface of the touch panel 170.

The touch panel 170 may be integrally formed with a display (not shown). For example, the touch sensor may be disposed on the substrate of the display, or may be provided inside the display. As such, the touch panel 170 may form a touch screen together with a display, and in this case, the touch screen may function as a user input unit that receives a touch input.

Although not illustrated in FIG. 1, the input / output device 100 may further include a display.

The display may display graphic objects corresponding to various information.

Displays include liquid crystal displays (LCDs), thin film transistor-liquid crystal displays (TFT LCDs), organic light-emitting diodes (OLEDs), and flexible displays, 3 At least one of a dimensional display (3D display) and an electronic ink display (e-ink display) may be included.

The display may be implemented as a head up display (HUD). When the display is implemented as a HUD, the display may include a projection module to output information through a wind shield or an image projected on the window.

The display may include a transparent display. The transparent display can be attached to a wind shield or window.

The transparent display can display a predetermined screen while having a predetermined transparency. Transparent display, to have transparency, the transparent display is transparent Thin Film Elecroluminescent (TFEL), transparent OLED (Organic Light-Emitting Diode), transparent LCD (Liquid Crystal Display), transmissive transparent display, transparent LED (Light Emitting Diode) display It may include at least one of. The transparency of the transparent display can be adjusted.

The display forms a mutual layer structure with the touch input unit or is integrally formed, thereby realizing a touch screen.

The processor 190 controls one or more components provided in the input / output device 100, and controls overall operation of the input / output device 100. The processor 190 may provide or process appropriate information or functions to a user by processing signals, data, information, etc. input or output through the above-described components or by driving an application program stored in a memory (not shown). have.

The processor 190 includes application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors (processors), It may be implemented using at least one of controllers, micro-controllers, microprocessors, and electrical units for performing other functions.

At least some of the components illustrated in FIG. 1 may operate in cooperation with each other to implement an operation, control, or control method of the input / output device 100 according to various embodiments described below. Also, the operation, control, or control method of the input / output device 100 may be implemented on the input / output device 100 by driving at least one application program stored in a memory.

For example, the processor 190 may control the haptic module 150 so that the haptic module 150 operates in different ways based on the hand image generated by the image sensor 130.

Furthermore, the processor 190 may perform vein authentication based on the hand image and provide a user interface accordingly.

2A and 2B are exemplary views for explaining the arrangement of some of the components described in FIG. 1. FIG. 2A is a top view of the input / output device 100 viewed from top to bottom, and FIG. 2B is a side view viewed in the direction A shown in FIG. 2A.

As illustrated in FIG. 2A, a touch panel 170 is disposed on an upper portion of the input / output device 100, and one or more haptic modules may be disposed at an edge of the touch panel 170.

As illustrated in FIG. 2B, at least one of the image sensor 130 and the gesture sensor 160 may be disposed on the rear surface of the touch panel 170. The touch panel 170 may be made transparent so that the image sensor 130 and / or the gesture sensor 160 can operate. Alternatively, the touch panel 170 has a hollow, and at least one of the image sensor 130 and the gesture sensor 160 may be disposed at a position overlapping the hollow.

Although not shown in the drawing, the touch panel 170 has a hollow, and the haptic module 150 may be disposed at a position overlapping the hollow.

A plurality of haptic modules 150 may be spaced apart. In this case, the processor 190 may calculate a target point and a target intensity for generating a tactile effect, and determine the output intensity of each haptic module such that vibration having the target intensity occurs at the target point. For example, the first haptic module may output a first ultrasound having a first frequency, and the second haptic module may output a second ultrasound having a second frequency. The point at which the first ultrasonic wave intersects the second ultrasonic wave becomes the target point, and vibration at the target point has the target intensity by the first and second ultrasonic waves.

The processor 190 may simultaneously receive a passenger's touch input through the touch panel 150 and vein information of the passenger through the image sensor 130.

The touch panel 150 may be omitted from the input / output device 100, and in this case, the touch panel 150 may be replaced with a window or a rubber pad.

3 is a flowchart illustrating a control method by the input / output device of FIG. 1, and FIGS. 4A, 4B, and 4C are conceptual views for explaining a three-dimensional space of the input / output device.

Referring to FIG. 3, the processor 190 detects the hand of a passenger who has boarded the vehicle 1 (S310).

A three-dimensional space separated from the input / output device 100 is defined, and the processor 190 may determine whether a passenger's hand is located in the three-dimensional space. Furthermore, the processor 190 may convert a hand position into a 2D and / or 3D coordinate system based on the coordinate system of the 3D space.

Here, the three-dimensional space means a space where a passenger can input a gesture input using finger and hand movements, and corresponds to an invisible virtual area. The input / output device 100 may track the hand of a passenger located in the 3D space, and may generate a control command to execute a predetermined function according to the movement of a finger and / or hand located in the 3D space.

In the present invention, the three-dimensional space is described as having a hexahedral shape, as shown in 4a, for convenience of explanation in the present invention, but in various sizes and shapes according to embodiments Dimensional space can be defined.

As shown in FIG. 4A, the input / output device 100 is for receiving user input from a passenger and is disposed in a center fascia of the vehicle 1 or is provided in the vehicle 1 It can be placed between a red stone and a passenger seat. Accordingly, a driver and / or a front passenger may execute various functions related to the vehicle 1 through the input / output device 100.

The processor 190 may calculate the hand position using the hand image received from the image sensor 150. It is also possible for the processor 190 to directly receive hand position information for the hand position from the image sensor 130.

The input / output device 100 may further include a gesture sensor 160 that tracks a gesture defined by at least one of a passenger's hand and finger. In this case, the processor 190 may detect the position of the hand using the gesture sensor 160.

The processor 190 may receive an internal image photographing the inside of the vehicle 1 from a camera provided in the vehicle 1. In this case, the processor 190 may calculate the hand position using the internal image.

A proximity sensor may be further provided in the input / output device 100. The processor 190 may detect whether a predetermined object is located in the 3D space by the proximity sensor, and determine whether the predetermined object corresponds to a passenger's hand using the image sensor 130.

The proximity sensor refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object in the vicinity using mechanical force or infrared light, etc., without mechanical contact. The proximity sensor may be disposed in the inner region of the input / output device 100 wrapped by the touch panel 170 described above or near the touch panel 170.

The processor 190 may calculate the position of the hand by comprehensively considering sensing information received from at least one of a sensor provided in the vehicle 1 and a sensor provided in the input / output device 100.

Next, the processor 190 may perform vein authentication using a vein located in the hand (S330).

Vein authentication means identifying a passenger using vein information generated by the image sensor 130. The processor 190 may check the identity of the passenger by using the remnant after seeing infrared light into the blood vessel. In other words, the processor 190 may perform vein authentication using the hand image and / or vein information generated from the image sensor 130.

The processor 190 compares vein information generated by the image sensor 130 to internal identification information stored inside the input / output device 100 and / or external identification information stored outside the input / output device 100. By doing so, vein authentication can be performed.

Here, the external identification information may include information stored in a memory provided in the vehicle 1, information stored in a server or a cloud, and information stored in a mobile terminal carried by a passenger.

The input / output device 100 may provide different interfaces according to a result of vein authentication. For example, if the passenger is a person who is authorized to use the vehicle 1, the input / output device 100 may provide a user interface to access all functions that can be executed in the vehicle 1 . Alternatively, if the passenger is a person who does not have the authority to use the vehicle 1, the input / output device 100 may provide a user interface that can access only limited functions.

The processor 190 may execute a user mode corresponding to the result of the vein authentication so that different interfaces are provided according to the result of the vein authentication. Furthermore, when the vein authentication is performed, the processor 190 may control the communication unit 110 such that a user mode corresponding to a passenger is executed on one or more electronic devices provided in the vehicle 1.

The vein information generated by the image sensor 130 serves as an ID and password for accessing the input / output device 100 and / or the vehicle 1. In other words, a login may be made by vein information.

The processor 190 searches for any one piece of identification information that matches the vein information among the stored one or more pieces of identification information. If there is the searched identification information, a user interface corresponding to the searched identification information is provided, and if there is no searched identification information, a limited user interface is provided.

The processor 190 may output different tactile effects based on the position of the hand (S350).

Depending on the structure of the input / output device 100, an area where vein authentication can be performed is limited. For example, as illustrated in FIGS. 4A and 4B, the vein authentication area 410 may be formed in a portion of the 3D space 400 where the input / output device 100 can acquire a hand image. Intravenous authentication may be performed only when a hand is positioned in the vein-authenticable area 410.

The processor 190 may generate a tactile effect that guides the direction of movement of the hand such that the passenger's hand is located in the vein authenticable area 410. The processor 190 generates a tactile effect of the first method when the hand is located in the first area of the 3D space, and a tactile effect of the second method when the hand is located in the second area of the 3D space. To occur, the haptic module 150 can be controlled.

For example, as illustrated in FIG. 4C, when the hand needs to move to the left, a tactile effect 420 in the form of a triangle pointing to the left occurs, and when the hand needs to move to the right, a triangle shape pointing to the right The tactile effect 430 may occur. When the hand is located in the vein authenticable area 410, a tactile effect 440 in a circular shape may occur.

Here, the tactile effect of a specific form means that the user can intuitively and / or recognize the specific form through tactile sense.

Even if the tactile effect of the same type, the tactile effect of a greater intensity is generated as the hand is closer to the vein authenticable region 410, and the tactile effect of a smaller intensity is generated as it is moved away from the vein authenticable region 410. Can occur. In other words, the processor 190 may control the haptic module 150 to generate vibrations of different sizes according to the distance between the image sensor 130 and the hand.

The position of the hand can be represented by a coordinate system consisting of three axes orthogonal to each other. For example, the position of the hand may be represented by the x-axis, y-axis, and z-axis. In this case, the processor 190 may control the haptic module 150 to generate different tactile effects according to at least one of the hand x-, y-, and z-coordinates.

The passenger may move his or her hand to the vein-authenticable area 410 where vein authentication is possible according to the tactile effect.

When the vein authentication is completed, the processor 190 ends the output of the tactile effect being output (S370), and may grant different privileges to the passengers according to the vein authentication (S390).

If the vein authentication is not completed, the processor 190 controls the haptic module 150 to generate different tactile effects based on the hand position.

When the vein authentication is completed, the processor 190 ends the output of the tactile effect being output. Thereafter, the processor 190 may additionally generate a tactile effect that guides the success or failure of vein recognition.

When the vein authentication is successful, a login by a specific ID is performed and a user interface corresponding to the specific ID is provided. Alternatively, if vein authentication fails, a limited user interface is provided according to login failure.

The user interface includes a menu that guides a plurality of functions available to passengers (or icons capable of executing applications), and at least one of a menu configuration, a menu type, and a background image is used for vein authentication. It depends on the vein information.

According to the present invention, the vein authentication can be used to provide different user interfaces to the passengers, and the user can place their hands in the correct position because the tactile effect of guiding the vein authentication area 410 is made to facilitate vein authentication. Can be moved.

The vein authenticable area 410 is an invisible virtual space, but an appropriate tactile effect is transmitted to the passenger through vibration of the air according to the position of the hand. As a result, the driver may perform driving with one hand and perform vein authentication with the other hand while looking at the front.

4D and 4E are exemplary views for explaining haptic intensity in a 3D space.

The processor 190 may adjust the haptic strength of the tactile effect in various ways according to the hand position.

For example, as illustrated in FIG. 4D, a tactile effect occurs when the passenger's hand is located at the edge of the 3D space 400, and the stronger the passenger's hand approaches the vein authentication area 410 A tactile effect of intensity can occur.

For another example, as illustrated in FIG. 4E, in order to guide the edge of the 3D space 400, the tactile effect of the greatest intensity occurs when the passenger's hand is located at the edge of the 3D space 400 And, when it is located at the edge of the vein authenticable area 410, the tactile effect of the weakest intensity occurs, and when it is located at the center of the vein authenticable area 410, a tactile effect of the middle intensity may occur.

5 is a flowchart for explaining a control method for generating different tactile effects on the hand of a passenger, and FIG. 6 is an exemplary view for explaining an example according to the control method of FIG. 5.

Referring to FIG. 5, the processor 190 may set the first part and the second part in the hand based on the position of the hand (S510).

At least one of the position and size of each part may vary depending on the position of the hand. Furthermore, the hand may be divided into three or more parts, and different tactile effects may occur in each part.

Next, the processor 190 may control the haptic module 150 such that a first vibration occurs in a first portion and a second vibration occurs in a second portion (S530).

For example, the first vibration may be a vibration having a first magnitude, and the second vibration may be a vibration having a second magnitude. For another example, the first vibration may be a vibration having a first shape, and the second vibration may be a vibration having a second shape. In conclusion, the first vibration and the second vibration are vibrations in which at least one of size, shape, vibration pattern, and repetition period is different.

As shown in FIG. 6, when the hand is located on the right side with respect to the input / output device 100, a tactile effect that guides the hand to move to the left may occur. For example, a tactile effect 420 in the form of a triangle pointing to the left may occur.

At this time, the processor 190 may set the first part 610 and the second part 620 for the entire area of the hand. The first portion 610 outputs a first vibration to generate a first tactile effect of a circular shape with a constant radius, and the second portion 620 has a second tactile effect of a distorted circular shape with a non-uniform radius. The second vibration can be output to occur. The passenger may intuitively and / or recognize the tactile effect 420 in the form of a triangle pointing to the left by the first vibration and the second vibration.

7 is a flowchart illustrating a control method of an input / output device related to a 3D space, and FIGS. 8A and 8B are exemplary views for explaining different 3D spaces according to passengers.

The processor 190 may set a 3D space based on information received through the communication unit 110. Passenger information is generated by various sensors provided in the vehicle 1 and may be received by the input / output device 100 through the communication unit 110.

The passenger information may include various information related to the passenger, such as gender, age, hand size, arm length, presence of disability, etc., as an example of the vehicle driving information.

At least one of the size, shape, and location of the 3D space may be defined differently according to the passenger (or passenger information).

The 3D space is a space where a passenger can input a gesture input, but the shoulder position and arm length are different depending on the passenger. The input / output device 100 according to the present invention sets different three-dimensional spaces according to the passengers in order to provide a customized input space for the passengers.

For example, as shown in FIG. 8A, a first three-dimensional space 812 is set for the first passenger 820, and as shown in FIG. 8B, a second for the second passenger 830 The 3D space 814 can be set.

On the other hand, since the 3D space is a virtual area that is invisible, the occupant has a problem that it is impossible to determine where the 3D space customized for himself is.

The processor 830 may provide a passenger with a non-contact tactile effect guiding the edge of the 3D space using the haptic module 150. For example, a tactile effect such as a virtual wall at the edge of the 3D space may be generated by the haptic module 150.

Furthermore, the processor may output visual information guiding the 3D space to the display 10 (S730). The display 10 is disposed inside the vehicle 1, and the processor 190 displays a control command to control the visual information to be output from the display 10 through the communication unit 110. 10).

The visual information includes an image corresponding to the interior interior of the vehicle 1, and a graphic object corresponding to the 3D space may be displayed on the image. Furthermore, a graphic object that guides the position of the hand based on the 3D space may be displayed on the image.

The processor 190 may adjust the light output intensity of the image sensor 130 based on the 3D space (S750).

The processor 190 may perform vein authentication using the image sensor 130. Since at least one of the location, size, and shape of the 3D space varies depending on the passenger, the processor 180 may also set at least one of the location, size, and shape of the vein-authenticable area according to the passenger. . This is to provide an optimal interface space for passengers. The processor 190 adjusts the light output intensity of the image sensor 130 to enable vein authentication in the vein authentication possible area.

9 is a flowchart illustrating a control method of an input / output device that performs various controls depending on whether an object entering a 3D space is a hand.

The processor 190 may determine whether an object entering the 3D space is a hand (S910).

The processor 190 is based on at least one of internal information generated by various sensors provided inside the input / output device 100 and internal information generated by various sensors provided outside the input / output device 100. It is possible to determine whether the object entering the 3D space is a hand.

If the object is not a hand, the haptic module 150 is controlled so that a tactile effect does not occur (S930). Alternatively, when the object is a hand, the haptic module 150 is controlled to generate different tactile effects according to the hand position. In other words, the processor 190 may determine whether to generate a tactile effect based on the position of the hand in the 3D space.

The processor 190 or the image sensor 130 determines whether an object entering the 3D space is the hand, and the processor 190, if the object is not the hand, the tactile effect The haptic module 150 is controlled so that it does not occur.

The processor 190 may selectively activate the image sensor 130 according to whether the hand is located in the 3D space (S950). The image sensor 130 may be selectively activated according to whether a passenger's hand is located in the 3D space.

The processor 190 may activate the image sensor 130 for vein authentication when the passenger's hand is located in the vein authentication capable area 410 or is expected to enter the vein authentication capable area 410. have. When it is determined that the vein authentication cannot be performed, the processor 190 may deactivate the image sensor 130.

Meanwhile, the input / output device 100 according to the present invention may perform vein authentication using the image sensor 130 and receive gesture input using the gesture sensor 160 at the same time.

Although the gesture sensor 160 is described as an example of a component of the input / output device 100, the gesture sensor 160 may be an independent device provided in the vehicle 1. In this case, the input / output device 100 may control the input to the vehicle 1 and the output by the vehicle 1 in cooperation with the gesture sensor 160.

10 is a flowchart illustrating a control method of an input / output device that performs various controls based on gesture input, and FIGS. 11A and 11B are exemplary views for describing the embodiments of FIG. 10.

The processor 190 may select a function corresponding to the gesture in response to a gesture defined by at least one movement of a passenger's hand and finger satisfying a reference condition (S1010).

For example, as illustrated in FIG. 11A, when the first gesture for pointing the palm toward the input / output device 100 with all fingers open is detected, the first function may be selected.

Next, the processor 190 may determine whether to execute the selected function based on vein information detected by the image sensor (S1030).

The processor 190 senses a gesture input, and at the same time, acquires vein information on a vein of the hand inputting the gesture input, and may perform vein authentication using the vein information.

As a result of vein authentication, when the passenger has the authority to execute the first function, the processor 190 may execute the first function. Alternatively, if the passenger does not have the authority to execute the first function, the processor 190 may limit / block the execution of the first function.

Meanwhile, the processor 190 may execute different functions according to vein information even when the same gesture is input. When the gesture satisfies the reference condition, the processor 190 may execute different functions according to the vein information detected by the image sensor 130.

For example, as illustrated in FIG. 11B, when a second gesture indicating ok is detected, different functions may be selected and executed according to whether vein authentication has been performed. Specifically, the processor 190 may execute a second function in response to the second gesture when vein authentication is successful, and execute a third function in response to the second gesture when vein authentication fails.

If authentication of the vein information fails, the processor 190 may output a user registration screen to the display 100 provided in the vehicle 1 so that the vein information is registered (S1050).

Passengers who fail vein authentication can register themselves to the vehicle 1 and / or the input / output device 100 through the user registration screen, and can use various functions as registered users.

12 is a flowchart illustrating a control method of an input / output device related to vein authentication.

When the vein authentication of the passenger is performed once, the processor 180 may deactivate the image sensor 110 until the passenger gets off from the vehicle 1 (S1210).

This is to prevent the vein authentication from being repeated multiple times unnecessarily, taking into account the peculiarity of boarding the same seat until the passengers get off after boarding the vehicle 1.

The processor 180 may control the communication unit 110 to execute a user mode corresponding to the passenger (S1230). Login is made by vein authentication, and a user interface corresponding to vein authentication may be provided according to a method described later in FIG. 13.

When the new passenger's hand enters the 3D space, the processor 180 activates the deactivated image sensor (S1250), and may perform vein authentication for the new passenger's hand (S1270).

For example, the processor 180 may determine whether a new passenger's hand enters a 3D space based on vehicle driving information received through the communication unit 110. For example, the processor 180 may determine in real time that a new passenger's hand enters a three-dimensional space based on an internal image photographing the interior of the vehicle 1.

For another example, the processor 180 may use the proximity sensor to determine whether a new passenger's hand enters a three-dimensional space. Since the input / output device 100 may be disposed between the driver's seat and the front passenger's seat, the processor 190 may determine which seat the hand of a passenger is on board based on the direction in which the hand enters. The hand entering from the right edge of the 3D space to the left with respect to the front of the vehicle 1 corresponds to the hand of the passenger in the passenger seat, and the hand entering from the left edge of the 3D space to the right is in the driver's seat. It may correspond to the hand of the driver on board.

Here, the new passenger means a person who has not been authenticated with vein. For example, if only the vein authentication for the driver has been made, the passenger in the passenger seat may be a new passenger. Alternatively, if only vein authentication is performed for the passenger in the passenger seat, the driver may be a new passenger.

The processor 190 may provide different user interfaces according to the occupants of the hand entering the 3D space.

When the driver's hand enters the 3D space, the processor 190 may provide a user interface for the driver. For example, the user interface for a driver may be that a driver's home screen including an icon for a function for the driver is displayed on the display 10 of the vehicle 1.

Alternatively, the processor 190 provides a user interface for the passenger when the hand of the passenger in the passenger seat enters the three-dimensional space. Different icons may be included in the user interface for the driver and the user interface for the passenger.

For example, the user interface for the driver may include a start icon for turning off or turning on the ignition of the vehicle 1, and a driving mode icon for changing the driving mode of the vehicle 1. Alternatively, the user interface for the passenger does not include the start icon and the driving mode icon, but may include a passenger seat air conditioning control icon capable of controlling the air conditioning system corresponding to the passenger seat of the vehicle 1.

13 is a flowchart for explaining a control method of an input / output device that provides a user interface using vein authentication.

The processor 190 may receive at least one personal profile from an external device (S1310).

The personal profile is for setting a user interface provided by the vehicle 1 and / or the input / output device 100, and a menu for executing at least one of the functions and functions specialized for the passenger. It can contain.

The external device providing the personal profile may be a server or a terminal carried by a passenger.

When the vein authentication is performed, the processor 190 may select a menu selectable by the user based on the user's personal profile corresponding to vein authentication (S1330). Furthermore, the processor 190 may control the communication unit 110 so that the selected menu is output from the display 10 provided in the vehicle 1 (S1350).

The processor 190 controls the display 10 to output a menu selectable by the passenger based on the personal profile, and functions included in the menu are varied according to the passenger.

According to the present invention, since the passenger is provided with a customized user interface by simply acting to place his or her hand in the vein authentication capable area 410 set by the input / output device 100, the convenience of the passenger increases. .

The above-described present invention can be embodied as computer readable codes (or applications or software) on a medium on which a program is recorded. The above-described control method of the autonomous vehicle can be realized by codes stored in a memory or the like.

The computer-readable medium includes any kind of recording device in which data readable by a computer system is stored. Examples of computer-readable media include a hard disk drive (HDD), solid state disk (SSD), silicon disk drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. This includes, and is also implemented in the form of a carrier wave (eg, transmission over the Internet). Also, the computer may include a processor or a control unit. Accordingly, the above detailed description should not be construed as limiting in all respects, but should be considered illustrative. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims

An input / output device mounted on a vehicle,

An image sensor configured to sense a hand of a passenger on the vehicle;

A haptic module configured to generate a tactile effect on the hand using vibration of air; And

An input / output device including a processor that controls the haptic module such that different tactile effects are generated based on the position of the hand.
According to claim 1,

A three-dimensional space separated from the input / output device is defined,

The processor,

And determining whether to generate the tactile effect based on the position of the hand in the 3D space.
According to claim 2,

The input / output device, characterized in that at least one of the size and position of the three-dimensional space is differently defined according to the passenger.
According to claim 3,

The processor,

An input / output device characterized by adjusting the light output intensity of the image sensor based on the three-dimensional space.
According to claim 2,

The processor,

When the hand is located in the first area of the 3D space, a tactile effect of the first method occurs, and when located in the second area of the 3D space, a tactile effect of the second method different from the first method occurs. An input / output device characterized by controlling the haptic module.
According to claim 2,

The processor,

The first part and the second part are set in the hand based on the position of the hand, and a vibration of a first size occurs in the first part, and a second size different from the first size is generated in the second part. Input and output device, characterized in that to control the haptic module to generate vibration.
According to claim 2,

Further comprising a communication unit,

The processor,

An input / output device characterized in that the display is controlled such that visual information guiding at least one of the three-dimensional space and the position of the hand is displayed on the display provided on the vehicle.
According to claim 2,

The processor or the image sensor determines whether an object entering the 3D space is the hand,

The processor is configured to control the haptic module so that the tactile effect does not occur when the object is not the hand.
According to claim 1,

The processor,

An input / output device, characterized in that the haptic module is controlled to generate vibrations of different sizes according to the distance between the hand and the image sensor.
According to claim 1,

Further comprising a gesture sensor for tracking the gesture defined by the movement of at least one of the hand and fingers of the passenger,

The processor,

An input / output device characterized by sensing the position of the hand using the gesture sensor.
The method of claim 10,

A three-dimensional space separated from the input / output device is defined,

The image sensor is an input / output device characterized in that it is selectively activated according to whether the hand is located in the 3D space.
The method of claim 10,

The image sensor detects the vein information located in the hand,

The processor

An input / output device characterized in that in response to the gesture satisfying a reference condition, a function corresponding to the gesture is selected, and whether or not the selected function is executed is determined based on the vein information detected by the image sensor. .
The method of claim 12,

Further comprising a communication unit,

The processor,

If the authentication for the vein information fails, the input / output device characterized in that the communication unit is controlled such that a user registration screen configured to register the vein information is output on a display provided in the vehicle.
The method of claim 10,

The processor,

When the gesture satisfies a reference condition, an input / output device characterized by executing different functions according to vein information detected by the image sensor.
According to claim 1,

The processor,

When the vein authentication of the passenger using the image sensor is performed once, the input / output device deactivating the image sensor until the passenger leaves the vehicle.
The method of claim 15,

A three-dimensional space separated from the input / output device is defined,

The processor,

An input / output device characterized in that when the hand of a new passenger who has not been vein authenticated enters the 3D space, activates the deactivated image sensor and performs vein authentication on the hand of the new passenger.
The method of claim 15,

Further comprising a communication unit,

The processor,

When the vein authentication is performed, an input / output device characterized in that the communication unit is controlled such that a user mode corresponding to the passenger is executed in one or more electronic devices provided in the vehicle.
According to claim 1,

Further comprising a communication unit,

The processor,

The personal profile corresponding to the passenger is received from an external device,

When vein authentication using the image sensor is performed, the communication unit is controlled so that a menu selectable by the passenger is output from a display provided in the vehicle based on the personal profile,

The functions included in the menu are input / output devices characterized in that they vary according to the passenger.
According to claim 1,

Further comprising a touch panel configured to sense the touch of the passenger,

The haptic module is spaced apart from the edge of the touch panel,

The image sensor is disposed overlapping the lower portion of the touch panel,

The processor is an input / output device characterized in that it simultaneously receives a touch input of the passenger through the touch panel and vein information of the passenger through the image sensor.
An input / output device mounted on a vehicle,

An image sensor configured to detect a vein located in the hand of a passenger in the vehicle;

A haptic module configured to generate a tactile effect on the hand using vibration of air; And

And a processor for performing vein authentication for the passenger using the image sensor and granting different privileges to the passenger according to the vein authentication,

A 3D space is defined based on the image sensor,

The processor,

If the hand is located in the first area of the 3D space, a haptic module of the first method occurs, and when the hand is located in the second area of the 3D space, the haptic module of the second method is generated. Control,

When the vein authentication is completed, an input / output device characterized in that the haptic module is controlled to end the generation of the tactile effect that was occurring.