KR101588184B1 - Control apparatus for vechicle, vehicle, and controlling method for vehicle - Google Patents

Abstract

A control apparatus for a vehicle comprises: a voice receiving unit receiving a voice of a driver; a gesture detection unit detecting a gesture of a driver; and a control unit determining whether any one of objects is selected by the voice and the gesture and generating a control signal so that a vehicle display can provide augmented reality of the selected object based on a determination result. The control unit guides the driver to the selected object which is a destination.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a vehicle control apparatus, a vehicle control method,

A vehicle control apparatus, a vehicle, and a control method of the vehicle.

In addition to basic driving functions, the vehicle can perform additional functions for operator convenience such as audio function, video function, navigation function, air conditioning control, seat control, lighting control and so on.

As a technique for performing various functions in such a vehicle, a gesture detection technique using a hand and a voice recognition technology are highlighted. Hand gesture detection and speech recognition are the most intuitive techniques because hands and mouth are the most liberal part of a person's body and are the closest to the way they routinely manipulate their tools.

An object of the present invention is to provide a vehicle control device for controlling a vehicle display so as to determine a destination desired by a driver according to a gesture and voice of a driver and to provide an augmented reality and a guidance for a destination.

A vehicle and a control method of a vehicle that determine a destination according to a gesture and voice of a driver and provide an augmented reality and a guidance on a destination.

A vehicle control apparatus according to one aspect includes: a voice receiving unit for receiving a voice of a driver; A gesture sensing unit for sensing a driver's hand; A camera module mounted on a windshield or a dashboard of the vehicle; Detecting at least one of an end point of the hand and a direction line of the wrist based on the detection result of the gesture detection unit and detecting at least any one of an end point of the hand and a direction line of the wrist and one or more objects And a control unit for generating a control signal so that the vehicle display provides the augmented reality of the selected object based on the determination result, wherein the vehicle display further includes a virtual image corresponding to the selected object mounted on the windshield, The controller outputs the map information, and the map information includes a route guide indicator with the selected object as a destination. The controller determines the selected object based on the image received from the camera module, , And the object is a building outside the vehicle that can be the destination for the route guidance, or It can be a place.

The vehicle display is also mounted on an AVN (Audio Video Navigation) device.

In addition, the control unit judges whether any one of the objects is selected when the voice matches the preset command.

delete

delete

delete

delete

In addition, in a state where the control unit is performing a route guidance to a certain object, when the other object is selected by the voice and the gesture, the control unit performs the route guidance using the selected object as a destination.

Further, the control unit generates a three-dimensional virtual image.

A vehicle according to another aspect includes: a voice receiving unit for receiving a voice of a driver; A gesture sensing unit for sensing a driver's hand; A camera module mounted on a windshield or a dashboard of the vehicle; Detecting at least one of an end point of the hand and a direction line of the wrist based on the detection result of the gesture detection unit and detecting at least any one of an end point of the hand and a direction line of the wrist and one or more objects A control unit for determining whether or not the control unit is turned on; And a vehicle display providing an augmented reality of the selected object, wherein the vehicle display is mounted on the windshield of the vehicle and outputs a virtual image and map information corresponding to the selected object, The control unit includes a route guidance indicator, and the control unit determines the selected object based on the image received from the camera module, performs the route guidance with the selected object as a destination, Of buildings or places.

The vehicle display is also mounted on an AVN (Audio Video Navigation) device.

delete

delete

In addition, the control unit judges whether any one of the objects is selected when the voice matches the preset command.

In addition, when the control unit is performing route guidance with an object as a destination, when another object is selected by voice and gesture, the route guidance is performed with the selected object as a destination.

Further, the vehicle display displays a three-dimensional virtual image.

According to another aspect of the present invention, there is provided a method of controlling a vehicle, comprising: receiving a driver's voice and sensing a driver's hand; Detecting at least one of an end point of the hand and a direction line of the wrist based on the detection result of the sensing step and detecting at least any one of an end point of the hand and a direction line of the wrist and one or more objects based on the voice Judging whether or not it has been selected; Providing an augmented reality of the selected object; And performing a route guidance with the selected object as a destination, wherein performing the route guidance includes outputting a virtual image and map information corresponding to an object selected on the windshield of the vehicle, Wherein the determining comprises determining a selected object based on an image received from a camera module mounted on a windshield or dashboard of the vehicle, It may be a building or a place outside the vehicle that can be the destination when performing the guidance.

delete

In addition, the sensing step includes determining whether any one of the objects is selected when the voice matches the preset command.

Further, the step of providing the augmented reality includes displaying the three-dimensional virtual image.
The gesture sensing unit includes a light source and an optical sensor. The light source forms a three-dimensional space with respect to x-axis, y-axis, and z-axis. The optical sensor senses light absorbed or reflected by the driver's hand, The control unit can detect at least one of the end point of the hand and the direction line of the wrist based on the detection result of the optical sensor.

According to the vehicle control apparatus, the vehicle, and the control method of the vehicle, the driver can be provided with an intuitive augmented reality and guidance on the destination by instructing a desired destination in the vehicle by hand or voice.

In addition, according to the vehicle control apparatus, the vehicle, and the control method of the vehicle, the driver can be provided with the augmented reality and the route guidance by instructing the destination located around the vehicle at any time during the driving through hands or voice.

BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a schematic view of the appearance of a vehicle on which a vehicle control device is mounted. Fig.
2 is an internal view of a vehicle equipped with a vehicle control device.
3 is an internal view of a vehicle equipped with a vehicle control device.
4 is a control block diagram of a control apparatus for a vehicle.
5 is an example of a three-dimensional space formed by a signal sensed by the gesture sensing unit.
6 is an illustration of an object corresponding to a direction line of a wrist or an end point of a hand;
7 is a diagram showing waveforms of voice and command words.
8 is a diagram of an augmented reality and map information of an object provided by a vehicle display mounted on a windshield.
9 is an image provided by the vehicle display mounted on the AVN apparatus.
10 is an exemplary view showing a vehicle display mounted on a windshield to output a navigation screen.
11 is a control block diagram of the vehicle.
12 is a flowchart of a control method of a vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS The objectives, specific advantages and novel features of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: FIG. It should be noted that, in the present specification, the reference numerals are added to the constituent elements of the drawings, and the same constituent elements are assigned the same number as much as possible even if they are displayed on different drawings. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. In this specification, the terms first, second, etc. are used to distinguish one element from another, and the element is not limited by the terms.

Hereinafter, the vehicle control apparatus 100 and the vehicle 10 will be described with reference to Figs. 1 to 11. Fig. Fig. 1 is a schematic view for explaining an appearance of a vehicle 10 on which a vehicle control apparatus 100 is mounted, and Fig. 2 is an internal view of the vehicle 10 on which the vehicle control apparatus 100 is mounted.

1, a vehicle 10 according to an embodiment includes a main body 1 forming an outer appearance of a vehicle 10, wheels 51 and 52 for moving the vehicle 10, wheels 51 and 52, A door 71 and 72 for shielding the interior of the vehicle 10 from the outside, a driver 60 for driving the inside of the vehicle 10, Glass 31, and side mirrors 81 and 82 that provide the driver with a field of view behind the vehicle 10.

The wheels 51 and 52 include a front wheel 51 provided on the front of the vehicle and a rear wheel 52 provided on the rear side of the vehicle and the driving device 60 includes a front wheel 51 and a rear wheel 52, (51) or the rear wheel (52). Such a drive unit 60 may employ an engine for generating a rotational force by burning fossil fuel or a motor for generating a rotational force by receiving power from a capacitor (not shown).

The doors 71 and 72 are provided so as to be rotatable on the left and right sides of the main body 1 so that the driver can ride inside the vehicle 10 at the time of opening. Shield.

The front glass 31 is provided on the front upper side of the main body 1 so that a driver inside the vehicle 10 can obtain time information in front of the vehicle 10. The windshield glass is also called a windshield glass.

On the windshield 31, a vehicle display 200 can be mounted.

The vehicle display 200 mounted on the windshield 31 displays the augmented reality of the object selected by the driver, the map information, the current vehicle speed, the remaining amount of fuel, It is a projection device that minimizes eye movements for the operator to view the dashboard or AVN (Audio Video Navigation) screen.

The object is a building or a place where the AVN apparatus 300 can be a destination when the navigation function is performed, and becomes a unit of the augmented reality provided by the vehicle display 200.

Augmented Reality (AR) is a technology that superimposes three-dimensional virtual images on a realistic image or background that the driver sees and displays them as one image.

The vehicle display 200 mounted on the windshield 31 overlaps and displays a three-dimensional virtual image of the object on the real image of the object reflected on the windshield 31 in order to provide the driver with an augmented reality of an object .

The vehicle display 200 mounted on the windshield 31 may be, for example, a Head-Up Display (HUD) implemented with a transparent thin film transistor.

In this case, as shown in FIG. 2, the vehicle display 200 is mounted on the windshield 31 and is transparent, so that it does not interfere with the driver's view.

The vehicle display 200 includes a front glass 31 as well as a rear glass (not shown) that provides the driver of the vehicle 10 in the vehicle 10 with a view of the rear of the vehicle 10, (Not shown) that provides a field of view of the camera.

Vehicle display 200 may also be mounted on AVN device 300, which will be described in detail below with respect to FIG.

The side mirrors 81 and 82 include a left side mirror 81 provided on the left side of the main body 1 and a right side mirror 82 provided on the right side. Side information and rear-side time information.

The vehicle 10 according to one embodiment includes a vehicle control device 100 and an AVN (Audio Video Navigation) device 300.

The AVN apparatus 300 is an apparatus for integrally performing an audio function, a video function, and a navigation function. In addition to the above functions, it is also possible to perform additional functions such as a telephone call.

A vehicle display 200 may be mounted on the AVN apparatus 300 to provide audio functions, video functions and navigation functions to the driver and the vehicle display 200 mounted on the AVN apparatus 300 may be mounted on the vehicle 10, The center fascia 33, which is the central area of the dashboard 32 of the vehicle. A detailed description related thereto will be described later.

3 is an internal view of the vehicle 10 on which the vehicle control device 100 is mounted, and Fig. 4 is a control block diagram of the onboard control device 100. Fig.

The vehicle control apparatus 100 may be mounted on the center console 40 between the driver's seat 21 and the passenger's seat 22 as shown in FIG. 3A, or may be mounted on the top of the front glass 31 as shown in FIG. 3B .

4, the vehicle control apparatus 100 includes a voice receiving unit 110 for receiving a driver's voice, a gesture sensing unit 120 for sensing a driver's gesture, and components of the vehicle control apparatus 100 A control unit 130 for controlling the entire system, and a storage unit 140 for storing map information corresponding to one or more objects.

The voice receiving unit 110 receives voice from the driver and generates an electrical signal. Voice includes all kinds of sound including human voice. Such speech can be represented by a sound wave having a frequency and a magnitude.

The voice receiving unit 110 may be micro-implemented, for example. The microphone includes a carbon type, a crystal type, a movable coil type, a ribbon type, a condenser type, and a semiconductor type microphone. However, the microphones are not limited to the types of microphones presented, but may be implemented in various types of microphones.

The gesture sensing unit 120 senses the hand gesture by photographing the driver's hand.

Hereinafter, a hand gesture will be described as an example, but it is also possible that the gesture detection unit 120 detects a gesture such as an eye or other types of gestures.

3A, when the onboard controller 100 is mounted on the center console 31, the gesture sensing unit 120 may sense a gesture of a moving hand on the top of the center console 31. [

3B, when the on-vehicle control device 100 is mounted on the front glass 31, the gesture sensing unit 120 can sense a gesture of a moving hand at the inner bottom of the front glass 31 have.

The gesture detection unit 120 can be implemented, for example, by a camera, and can track the driver's hand while being rotated 180 degrees by the servo motor so that the driver's hand is positioned within the camera field of view. Also, the gesture sensing unit 120 may be implemented by a stereo camera for acquiring three-dimensional coordinates of the driver's hand or 3D modeling, or an infrared camera including an infrared light source and an infrared sensor.

Also, the gesture sensing unit 120 may be implemented by an ultrasonic sensor, for example. In this case, the gesture sensing unit 120 senses the movement of the driver's hand in real time within the sensing area.

Also, the gesture sensing unit 120 may be implemented as a Leap Motion device implemented by three infrared light sources and two optical sensors.

In addition, the gesture sensing unit 120 may be implemented as a variety of devices for sensing a hand gesture.

5 is an example of a three-dimensional space formed by a signal sensed by the gesture sensing unit 120. In FIG.

5, the gesture sensing unit 120 forms a three-dimensional space with respect to x, y, and z axes corresponding to each light source using three infrared light sources, Can be detected.

In this three-dimensional space, the gesture sensing unit 120 senses the driver's hand as a gesture.

For example, the gesture sensing unit 120 can sense the driver's hand by sensing infrared rays absorbed or reflected by the driver's hand although it is irradiated from an infrared light source, thereby generating an output signal.

The control unit 130 determines one or more objects to be displayed to the driver and detects the direction line l of the wrist or the end point p of the hand and detects the direction line l of the detected wrist or the end point p of the hand, And the voice of the driver received by the voice receiving unit 110, whether or not any one of the at least one object is selected.

For example, the control unit 130 may estimate one or more objects to be displayed to the driver at the current position, based on the current position information of the vehicle 10 received from the GPS (Global Positioning System).

The control unit 130 may estimate one or more objects to be displayed to the driver based on the outside image received from the camera module (not shown) mounted on the windshield 31 or the dashboard 32. [ The outside image means an image that is seen by the driver through the windshield 31 of the vehicle 10.

However, one or more objects shown to the driver are not necessarily limited to these, but may be estimated or determined in various ways.

The control unit 130 detects the direction line 1 of the wrist or the end point p of the hand based on the output signal of the gesture sensing unit 120.

For example, the controller 130 calculates the gradient of the boundary of the hand based on the output signal of the gesture sensing unit 120 to detect the direction I of the wrist and the end point p of the hand first can do.

The control unit 130 detects the end point p of the hand based on the output signal of the gesture sensing unit 120 and tracks the movement of the end point p of the hand, It can also be tracked.

Further, the control unit 130 determines an object corresponding to the direction line l of the detected wrist or the end point p of the hand among at least one object shown to the driver.

6 is an illustration of an object corresponding to a direction line l of a wrist or an end point p of a hand.

Referring to FIG. 6, the controller 130 determines an object ob2 corresponding to a direction line l of the detected wrist or an end point p of the hand among one or more objects ob1 and ob2 shown to the driver .

The object ob2 corresponding to the directional line 1 of the wrist means an object ob2 shown to the user at the point where the directional line 1 of the detected wrist meets the windshield 31. [

The object ob2 corresponding to the end point p of the hand is positioned at a position where the straight line and the front glass 31 meet when a straight line parallel to the direction line l of the wrist is drawn from the detected end point p of the hand, (Ob2) shown to the user.

The control unit 130 determines whether the voice received from the voice receiving unit 110 coincides with a command stored in the storage unit 140 to be described later.

7 is a diagram showing waveforms of voice and command words.

The control unit 130 determines that the object ob2 corresponding to the direction line l of the current wrist and the end point p of the hand is "selected "

For example, when "save" is stored as a command in the storage unit 140 and the voice receiving unit 110 receives "save" as a voice, the control unit 130 determines whether the current direction of the wrist l It is determined that the object ob2 corresponding to the end point p of the hand has been selected.

As shown in Fig. 7, the coincidence between the voice and the command can be judged according to whether or not the waveform of the voice and the waveform of the command coincide with each other.

The control unit 130 then generates a control signal for the vehicle display 200 to provide the augmented reality for the selected object to the driver.

The vehicle display 200 may be mounted on the windshield 31 or mounted on the AVN apparatus 300, as described above.

8 is a diagram of an augmented reality and map information of an object provided by the vehicle display 200 mounted on the windshield 31. Fig.

8, the controller 130 generates a virtual image for the selected object ob2, and generates a control signal for causing the vehicle display 200 to project the virtual image generated on the windshield 31 .

In this case, the controller 130 controls the vehicle display 200 mounted on the front glass 31 to project the virtual image on the real image displayed to the driver on the windshield 31, thereby providing the augmented reality to the driver can do.

The virtual image may be an image stored in the storage unit 140, which will be described later.

The control unit 130 also controls the vehicle display 200 mounted on the windshield 31 to display not only the virtual image but also the map information inf for the selected object ob on the windshield 310 You may.

Map information refers to general navigation information related to an object such as an object's business name, geographical position, and the like.

In addition, the controller 130 controls the vehicle display 200 mounted on the front glass 31 to display various information (not shown) such as the current speed of the vehicle, the remaining amount of fuel, It is possible to control the projection.

The vehicle display 200 mounted on the windshield 31 may be, for example, a Head-Up Display (HUD) implemented with a transparent thin film transistor.

Further, the vehicle display 200 mounted on the front glass 31 may display a three-dimensional image to the driver.

On the other hand, the vehicle display 200 may be mounted on the AVN apparatus 300, as well as mounted on the windshield 31.

9 is an image provided by the vehicle display 200 mounted on the AVN apparatus 300. Fig.

The vehicle display 200 mounted on the AVN apparatus 300 can be mounted on the center fascia 33 which is the central area of the dashboard 32 of the vehicle 10, It is possible to provide an augmented reality for the selected object ob2 as shown in FIG.

In this case, based on the map information corresponding to the current position of the vehicle 10, the control unit 130 displays the virtual image of the selected region of the selected object ob2, The augmented reality can be provided to the driver by controlling the vehicle display 200 mounted on the AVN apparatus 300 to display the augmented reality together.

The real image may be an image obtained from a camera module (not shown) mounted on the front glass 31 of the vehicle 10 or the dashboard 32.

The virtual image may be a three-dimensional image stored in the storage unit 140, which will be described later.

The control unit 130 may also control the vehicle display 200 mounted on the AVN apparatus 300 to display not only virtual images but also map information inf for the selected object ob to the driver.

The vehicle display 200 mounted on the AVN apparatus 300 may be formed using a plasma display panel (PDP), a light emitting diode (LED), or a liquid crystal display It may be implemented.

On the other hand, the controller 130 can perform a route guidance using the selected object ob2 as a destination.

In this case, if another object ob2 is selected by voice and gesture in a state in which the controller 130 has already performed the route guidance to an object, As a priority.

The route guidance can be performed by controlling the vehicle display 200 mounted on the windshield 31 or the vehicle display 200 mounted on the AVN apparatus 300 to output a navigation screen.

10 is an exemplary view showing the vehicle display 200 mounted on the windshield 31 to output a navigation screen.

10, the control unit 130 generates a route guidance image with the selected object ob2 as a destination, and displays the vehicle display 200 mounted on the front glass 31 to project the route guidance image to the driver. Can be controlled.

In this case, the control unit 130 may generate the route guidance indicator v and other map information (not shown) as a route guidance image to perform the route guidance to the driver.

Also, the route guidance image (e.g., the route guidance indicator v) may be generated as a three-dimensional virtual image.

In addition, the control unit 130 can provide the augmented reality to the driver by controlling the vehicle display 200 mounted on the windshield 31 to project various virtual images onto the windshield 31. [

9, the controller 130 generates a route guidance image with the selected object ob2 as a destination, and displays the route guidance image on a vehicle display (not shown) mounted on the AVN apparatus 300 200 can be controlled.

In this case, the vehicle display 200 mounted on the AVN apparatus 300 transmits the route guidance indicator v and the map information inf, which are the destination of the selected object ob2, To the driver.

Also, the route guidance image (e.g., the route guidance indicator v) may be generated as a three-dimensional virtual image.

In addition, the control unit 130 may provide the augmented reality to the driver by controlling the vehicle display 200 mounted on the AVN apparatus 300 to output various virtual images to the navigation screen.

3, the control unit 130 includes a ROM 133 storing a control program for controlling a processor 131, a control unit 100 for a vehicle, and a control unit (RAM) 132 used as a storage area corresponding to various operations performed in the on-vehicle control apparatus 100, or may include a RAM (Random Access Memory)

In addition, a separate circuit board electrically connected to the controller 130 may include a processing board (not shown) including a processor 131, a RAM 132, or a ROM 133.

The processor 131, the RAM 132, and the ROM 133 may be interconnected via an internal bus.

The control unit 130 may also be used to refer to components including the processor 131, the RAM 132, and the ROM 133. [

The control unit 130 may also be used to refer to components including a processor 131, a RAM 132, a ROM 133, and a processing board (not shown).

The storage unit 140 includes a program area 141 in which a program for the functional operation of the vehicular controller 100 is stored and a data area 142 in which data according to use is stored. The data area 142 stores virtual images, map information, commands necessary for voice recognition, and the like corresponding to one or more objects (ob).

Map information refers to general navigation information related to an object such as an object's business name, geographical position, and the like. The map information may include location information (e.g., underground or tunnel), path information (e.g., speed camera location, bifurcation, highway), traffic information For example, the location of the A restaurant and the B gas station).

The term " command " means a unit of data used when the control unit 130 performs control according to the driver's voice. For example, the command can be a variety of commands such as "voice recognition mode," "<ordinal number>," "yes / no", "route <number>," "call to <name> And includes a speech waveform for the phrase.

3, the control unit 130 stores signals or data input from the ROM 133 storing the control program and the outside of the onboard control unit 100, or various operations performed on the onboard control unit 100 (RAM) 132 used as a storage area corresponding to the storage unit 140. However, it is also possible that the storage unit 140 includes a RAM and a ROM.

Also, the storage unit 140 may be implemented as a separate unit from the controller 130, but may be implemented in the controller 130 to include RAM and ROM. In this case, the storage unit 140 may be implemented in combination with other components constituting the control unit 130 on any one of the substrates or in the apparatus.

The gesture detection unit 120, the control unit 130 and the storage unit 140 may be implemented as modules in the vehicle 10, And can be connected through a network.

In this case, the wired and wireless network includes a wired communication network, a wireless communication network, and a local area network, and may include a combination of a wired communication network, a wireless communication network, and a local area network.

The wired communication network includes wired Ethernet, and includes a wide area network (WAN), a value added network (VAN), and a CAN (Controller Area Network).

The wireless communication network may be connected to an access point (AP) by accessing a wireless network at a location where an access point (not shown) is installed. The wireless communication network supports the IEEE 802.11 standard of the Institute of Electrical and Electronics Engineers (IEEE).

The local area network includes a bluetooth communication network, a bluetooth low energy, an infrared data association (IrDA), a Wi-Fi communication network, an ultra wideband (UWB) (Zigbee) communication network.

Hereinafter, the voice receiving unit 110, the gesture detecting unit 120, the control unit 130, and the storage unit 140 implemented in the vehicle 10 will be referred to as a voice receiving unit 210, a gesture detecting unit 220, (230), and a storage unit (240).

11 is a control block diagram of the vehicle 10. Fig.

11, the vehicle 10 includes a voice receiving unit 210 for receiving voice, a gesture sensing unit 220 for sensing a gesture of a driver, a control unit 230 for controlling overall components of the vehicle 100 A storage unit 240 for storing map information corresponding to at least one object, and a vehicle display 200 for outputting a virtual image.

The gesture detection unit 220, the control unit 230 and the storage unit 240 are connected to the voice receiving unit 210, the gesture detection unit 120, the control unit 130, (140), and duplicate descriptions thereof are omitted.

The vehicle display 200 may be mounted on the windshield 31, as shown in Fig. 2, and mounted on the AVN apparatus 300 as shown in Fig.

The vehicle display 200 mounted on the windshield 31 projects a three-dimensional virtual image of the selected object on the windshield 31 in accordance with the control signal output from the controller 230.

In addition, the vehicle display 200 mounted on the front glass 31 can project not only a three-dimensional virtual image but also map information on the selected object on the windshield 31. [

Also, the vehicle display 200 mounted on the front glass 31 may be a head-up display (HUD) implemented with a transparent thin film transistor.

The vehicle display 200 mounted on the AVN apparatus 300 outputs a 3D virtual image of the selected object on the navigation screen according to the control signal generated by the controller 230.

In this case, the vehicle display 200 mounted on the AVN apparatus 300 displays the three-dimensional virtual image of the selected object ob2 together with the real image of the surrounding region of the selected object ob2 on the navigation screen .

The real image may be an image obtained from a camera module (not shown) mounted on the front glass 31 of the vehicle 10 or the dashboard 32.

The vehicle display 200 mounted on the AVN apparatus 300 may be formed using a plasma display panel (PDP), a light emitting diode (LED), or a liquid crystal display It may be implemented.

In addition, the vehicle 10 may include a proximity sensor for detecting rear or side obstacles or other vehicles, and a rain sensor for detecting rainfall and precipitation.

Hereinafter, a control method of the vehicle 10 will be described with reference to Fig. 12 is a flowchart of a control method of a vehicle.

First, the vehicle receives voice from the driver and senses the driver's gesture (S1110).

In this case, the vehicle generates a voice received from the driver and an electrical signal corresponding to the gesture of the driver.

The gesture of the driver may be, for example, a gesture of the driver's hand.

The vehicle can photograph the driver's hand using a camera, an infrared sensor, an ultrasonic sensor, a lip motion device, etc., and can sense the wrist direction line of the hand or the end point of the hand.

Then, the vehicle determines one or more objects to be displayed to the driver, and determines whether any one of the at least one object is selected based on the detected gesture and the received voice (S1120).

Specifically, the vehicle can estimate one or more objects to be displayed to the driver at the current position, based on the current position information of the vehicle received from the GPS.

In addition, the vehicle may estimate one or more objects to be displayed to the driver based on an outside image received from a camera module (not shown) mounted on the top or bottom of the windshield.

However, one or more objects shown to the driver are not necessarily limited to these, but may be estimated or determined in various ways.

Further, the vehicle detects the hand gesture based on the output signal.

For example, the vehicle detects the direction line of the driver's wrist or the end point of the hand. The vehicle can also track the movement of the end point of the hand by tracking the trajectory along which the end point of the hand moves.

Further, the vehicle determines an object corresponding to the detected hand gesture of one or more objects estimated or determined. For example, the vehicle determines an object corresponding to the detected wrist direction line or the end point of the hand.

When the vehicle receives a voice corresponding to a preset command, it determines that an object corresponding to the hand gesture is "selected ". For example, when the vehicle 10 receives a voice that is the same as the preset command "sleeping ", it is determined that an object existing at the wrist direction line or the end position of the hand is selected.

Then, the elevator provides the augmented reality and map information of the selected object to the driver (S1130).

In this case, the vehicle can output the virtual image and map information of the selected object through the vehicle display 200 mounted on the windshield or AVN device.

In addition, the vehicle may output a virtual image of a selected object on the real image when the virtual image is output.

The real image may actually be the outer image seen through the windshield to the driver, or it may be the outer image obtained from a camera module mounted on the windshield or dashboard of the vehicle.

The map information includes general navigation information related to the object such as an object's business name, geographical location, and the like.

The map information may include location information (e.g., underground or tunnel), path information (e.g., speed camera location, bifurcation, highway), traffic information For example, the location of the A restaurant and the B gas station).

Subsequently, the vehicle performs a route guidance with the selected object as a destination (S1140).

The route guidance can be performed by controlling the vehicle display mounted on the windshield or the display mounted on the AVN device to output a navigation screen that is the destination of the selected object.

In the above-described embodiment, some of the components constituting the vehicular controller 100 and the vehicle 10 may be implemented as a kind of 'module'. Here, 'module' means a hardware component such as software or a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), and the module performs certain roles. However, a module is not limited to software or hardware. A module may be configured to reside on an addressable storage medium and may be configured to execute one or more processors.

Thus, by way of example, a module may include components such as software components, object-oriented software components, class components and task components, and processes, functions, attributes, procedures, Microcode, circuitry, data, databases, data structures, tables, arrays, and variables, as will be appreciated by those skilled in the art. The functionality provided by the components and modules may be combined into a smaller number of components and modules or further separated into additional components and modules. In addition, the components and modules may execute one or more CPUs within the device.

Meanwhile, the vehicle control method described above can be implemented as a computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording media storing data that can be decoded by a computer system. For example, it may be a ROM (Read Only Memory), a RAM (Random Access Memory), a magnetic tape, a magnetic disk, a flash memory, an optical data storage device, or the like. In addition, the computer-readable recording medium may be distributed and executed in a computer system connected to a computer network, and may be stored and executed as a code readable in a distributed manner.

It will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the appended claims. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

1: Body
10: Vehicle
21: driver's seat 22:
31: front glass
32: Dashboard
33: Center Fesia
51: front wheel
52: rear wheel
60: Driving device
71, 72: Door
81, 82: side mirror
100: vehicle control device
110:
120: Gesture detection unit
130:
131: Processor
132: Ram 132: Rom
140:
141: program area 142: data area
200: Vehicle display
210:
220: Gesture detection unit
230:
231: Processor
232: Ram 233: Rom
240:
241: program area 242: data area
300: Abe Lee's device

Claims (22)

A voice receiving unit for receiving a voice of a driver;
A gesture sensing unit for sensing a driver's hand;
A camera module mounted on a windshield or a dashboard of the vehicle; And
Detecting at least one of an end point of the hand and a direction line of the wrist based on the detection result of the gesture detection unit and detecting at least one of an end point of the hand and a direction line of the wrist, And a control unit for generating a control signal so that the vehicle display provides the augmented reality of the selected object based on the determination result,
Wherein the vehicle display is mounted on the front glass and outputs a virtual image and map information corresponding to the selected object,
Wherein the map information includes a route guidance indicator with the selected object as a destination,
Wherein the control unit judges the selected object based on an image received from the camera module, performs a route guidance using the selected object as a destination,
Wherein the object is a building or a place outside the vehicle that can be a destination when performing the route guidance. The method according to claim 1,
Wherein the vehicle display is mounted on an AVN (Audio Video Navigation) apparatus. delete delete delete delete The method according to claim 1,
Wherein the control unit determines whether one of the objects is selected when the voice coincides with a predetermined command. The method according to claim 1,
Wherein the control unit performs a route guidance using the selected object as a destination when the voice and the gesture select another object in a state that the control unit is performing a route guidance to an object. The method according to claim 1,
Wherein the control unit generates a three-dimensional virtual image. A voice receiving unit for receiving a voice of a driver;
A gesture sensing unit for sensing a driver's hand;
A camera module mounted on a windshield or a dashboard of the vehicle;
Detecting at least one of an end point of the hand and a direction line of the wrist based on the detection result of the gesture detection unit and detecting at least one of an end point of the hand and a direction line of the wrist, A control unit for determining whether or not any one of them is selected; And
A vehicle display providing an augmented reality of a selected object,
Wherein the vehicle display is mounted on a windshield of a vehicle and outputs a virtual image and map information corresponding to the selected object,
Wherein the map information includes a route guidance indicator with the selected object as a destination,
Wherein the control unit judges the selected object based on an image received from the camera module, performs a route guidance using the selected object as a destination,
Wherein the object is a building or a place outside the vehicle that can be the destination in the course guidance. delete 11. The method of claim 10,
The vehicle display is mounted on an AVN (Audio Video Navigation) device. delete 11. The method of claim 10,
Wherein the control unit judges whether one of the objects is selected when the voice coincides with a preset command. 11. The method of claim 10,
Wherein, when the control unit is performing a route guidance to an object, the control unit performs the route guidance with the selected object as a destination, when another object is selected by the voice and the gesture. 11. The method of claim 10,
Wherein the vehicle display displays a three-dimensional virtual image. Receiving a driver's voice and sensing a driver's hand;
Detecting at least one of an end point of the hand and a direction line of the wrist based on the detection result of the sensing step and detecting at least one of an end point of the hand and a direction line of the wrist, Determining whether one of the objects is selected;
Providing an augmented reality of the selected object; And
Performing a route guidance with the selected object as a destination,
Wherein the step of performing the route guidance includes outputting virtual image and map information corresponding to the selected object on the windshield of the vehicle,
Wherein the map information includes a route guidance indicator with the selected object as a destination,
Wherein the determining comprises determining the selected object based on an image received from a camera module mounted on a windshield or dashboard of the vehicle,
Wherein the object is a building or a place outside the vehicle that can be a destination when performing the route guidance. delete 18. The method of claim 17,
Wherein the sensing includes determining whether one of the objects has been selected when the voice matches a predetermined command. 18. The method of claim 17,
Wherein the step of providing the augmented reality includes displaying a three-dimensional virtual image. The method according to claim 1,
The gesture sensing unit includes a light source and an optical sensor, and the light source forms a three-dimensional space with respect to x-axis, y-axis, and z-axis, and the optical sensor senses light absorbed or reflected by the hands of the driver and,
Wherein the controller detects at least one of an end point of the hand and a direction line of the wrist based on the detection result of the optical sensor. 11. The method of claim 10,
The gesture sensing unit includes a light source and an optical sensor, and the light source forms a three-dimensional space with respect to x-axis, y-axis, and z-axis, and the optical sensor senses light absorbed or reflected by the hands of the driver and,
Wherein the control unit detects at least one of an end point of the hand and a direction line of the wrist based on the detection result of the optical sensor.

Images