WO2015093905A1

WO2015093905A1 - Vehicle driving assistance device and vehicle having same

Info

Publication number: WO2015093905A1
Application number: PCT/KR2014/012611
Authority: WO
Inventors: 박영경; 홍윤표; 이제훈; 최원석
Original assignee: 엘지전자 주식회사
Priority date: 2013-12-20
Filing date: 2014-12-19
Publication date: 2015-06-25
Also published as: KR20150072942A

Abstract

The present invention relates to a vehicle driving assistance device and a vehicle having the same. The vehicle driving assistance device, according to one embodiment of the present invention, comprises: a camera capable of photographing side and rear parts of a vehicle; and a processor which calculates a degree of risk of the side and rear parts of the vehicle on the basis of a stereo image received from a stereo camera and side entrance information, and which generates side and rear part risk level information on the basis of the degree of risk. Thus, side and rear part risk level information corresponding to a driving situation can be displayed.

Description

Vehicle driving assistance device and vehicle having same

The present invention relates to a vehicle driving assistance apparatus and a vehicle having the same, and more particularly, to a vehicle driving assistance apparatus capable of displaying side and rear danger level information corresponding to a driving situation, and a vehicle having the same.

The vehicle is a device for moving in the direction desired by the user on board. An example is a car.

On the other hand, for the convenience of the user using the vehicle, various types of sensors, electronic devices, etc. are provided. In particular, various devices for driving convenience of the user have been developed.

An object of the present invention is to provide a vehicle driving assistance apparatus capable of displaying the side rear hazard level information corresponding to a driving situation and a vehicle having the same.

Vehicle driving assistance apparatus according to an embodiment of the present invention for achieving the above object, the rear-side rearview risk of the vehicle based on the camera capable of photographing the rear and rear, the stereo image received from the stereo camera, and the side entry information And a processor for generating lateral rear hazard level information based on the risk.

On the other hand, the vehicle driving assistance apparatus according to an embodiment of the present invention for achieving the above object, based on the radar for detecting the distance to the rear and rear of the vehicle, the distance information detected from the radar, and the side entry information, the vehicle And a processor for computing the lateral rear hazards of and generating lateral rear hazard level information based on the hazards.

On the other hand, the vehicle driving assistance apparatus according to an embodiment of the present invention for achieving the above object, based on the rider to perform scanning on the rear and rear of the vehicle, the scan image received from the rider, and the side entry information, the vehicle And a processor for computing the lateral rear hazards of and generating rear hazard level information based on the hazards.

On the other hand, a vehicle according to an embodiment of the present invention for achieving the above object, a display, a sensor unit for sensing a vehicle state, a steering drive unit for driving a steering device, a brake drive unit for driving a brake device, driving the engine A control unit for controlling an engine driving unit, a suspension driving unit for driving a suspension device, a steering driving unit, a brake driving unit, a power source driving unit, a suspension driving unit, a camera capable of photographing the vehicle in the rear and rear, and a stereo image received from a stereo camera, and a side surface. And a vehicle driving assistance device having a processor configured to calculate a side rear hazard level of the vehicle based on the entry information and to display, on the display, the generated side rear hazard level information based on the risk level.

The vehicle driving assistance apparatus according to an embodiment of the present invention calculates the rear and rear risks of the vehicle based on the stereo image received from the camera capable of photographing the rear and rear of the vehicle, and the side entry information, and based on the risks, By generating the rear danger level information, it is possible to display the side rear danger level information corresponding to the vehicle driving situation. Accordingly, user convenience may be increased.

Meanwhile, at least one of the display color, the size of the display level, and the brightness of the display level may be varied according to the calculated risk, so that the user may intuitively change the risk. Can be recognized.

On the other hand, when the calculated risk exceeds the allowable value, the vehicle and the driver of the vehicle can be protected by generating a side entry prevention control signal for controlling at least one of the steering driver and the brake driver.

Meanwhile, the vehicle driving assistance apparatus may further include a stereo camera for capturing a stereo image of the front, and based on the stereo image, when detecting an object, the vehicle driving assistance apparatus detects the distance information by using the stereo image and uses the distance information. Thus, the object can be separated and detected, thereby reducing the data processing speed and the like.

On the other hand, the vehicle driving assistance apparatus according to another embodiment of the present invention, based on the distance information detected from the radar for detecting the distance to the rear and rear of the vehicle, and the side entry information, calculates the rear and rear risk of the vehicle, By generating the side rear danger level information based on the degree of danger, it is possible to display the side rear danger level information corresponding to the vehicle driving situation. Accordingly, user convenience may be increased.

On the other hand, the vehicle driving assistance apparatus according to another embodiment of the present invention, based on the scanned image and the side entry information received from the rider performing the scanning of the rear and rear of the vehicle, calculates the rear and rear risk of the vehicle, By generating the side rear danger level information based on the degree of danger, it is possible to display the side rear danger level information corresponding to the vehicle driving situation. Accordingly, user convenience may be increased.

1 is a view showing the appearance of a vehicle having a stereo camera according to an embodiment of the present invention.

FIG. 2 is a view illustrating an appearance of a stereo camera attached to the vehicle of FIG. 1.

3A to 3B illustrate various examples of an internal block diagram of a vehicle driving assistance apparatus according to an embodiment of the present invention.

4A-4B illustrate various examples of internal block diagrams of the processor of FIGS. 3A-3B.

5A through 5B are views referred to for describing an operation of the processor of FIGS. 4A through 4B.

6A to 6D are views referred to for describing the operation of the vehicle driving assistance apparatus of FIGS. 3A to 3B.

FIG. 7 is an example of an internal block diagram of the electronic control apparatus in the vehicle of FIG. 1.

8 is a flowchart illustrating a method of operating a vehicle driving assistance apparatus according to an embodiment of the present invention.

9 to 16B are views referred to for describing the operating method of FIG. 8.

17 is a flowchart illustrating a method of operating a vehicle driving assistance apparatus according to another embodiment of the present invention.

18 is a diagram illustrating an exterior of a vehicle including a mono camera according to another embodiment of the present invention.

19A illustrates an example of an internal block diagram of a vehicle driving assistance apparatus according to another embodiment of the present invention.

19B illustrates an example of an internal block diagram of a vehicle driving assistance apparatus according to another embodiment of the present invention.

20A-20B illustrate various examples of internal block diagrams of the processor of FIG. 19A.

21A through 21B are views for explaining an operation of the processor of FIG. 20A.

Hereinafter, with reference to the drawings will be described the present invention in more detail.

The suffixes "module" and "unit" for components used in the following description are merely given in consideration of ease of preparation of the present specification, and do not impart any particular meaning or role by themselves. Therefore, the "module" and "unit" may be used interchangeably.

The vehicle described herein may be a concept including an automobile and a motorcycle. In the following, a vehicle is mainly described for a vehicle.

Meanwhile, the vehicle described herein may be a concept including a vehicle having an engine, a hybrid vehicle having an engine and an electric motor, an electric vehicle having an electric motor, and the like. Hereinafter, the description will be given mainly on a vehicle provided with an engine.

Meanwhile, the vehicle driving assistance apparatus described in the present specification may be referred to as an Advanced Driver Assistance Systems (ADAS) or an Advanced Driver Assistance Apparatus (ADAA). Hereinafter, a vehicle driving assistance apparatus for a vehicle and a vehicle having the same according to various embodiments of the present disclosure will be described.

Referring to the drawings, the vehicle 200 includes wheels 103FR, 103FL, 103RL,... Rotated by a power source, a steering wheel 150 for adjusting the traveling direction of the vehicle 200, and a front of the vehicle. First and second side cameras 197L, which are provided in the front stereo camera 195 and the left and

right side mirrors

107L and 107R for acquiring an image, respectively, for acquiring images for the left and right rear, respectively. 197R).

The stereo camera 195 may include a plurality of cameras, and the stereo image obtained by the plurality of cameras may be signal processed in the vehicle driving assistance apparatus (100 of FIG. 3).

Meanwhile, the drawing illustrates that the stereo camera 195 includes two cameras.

In the drawings, the first and

second side cameras

197L and 197R each have one camera, but alternatively, a plurality of cameras may be provided, and in particular, a stereo camera may be provided. It is possible. In the following description, the first and

second side cameras

197L and 197R each have a single camera.

Referring to the drawings, the stereo camera module 195 may include a first camera 195a having a first lens 193a and a second camera 195b having a second lens 193b.

Meanwhile, the stereo camera module 195 includes a first light shield 192a and a second light for shielding light incident on the first lens 193a and the second lens 193b, respectively. The shield 192b may be provided.

The stereo camera module 195 of the drawing may be a structure that can be attached to or detached from the ceiling or the windshield of the vehicle 200.

A vehicle driving assistance device (100 in FIG. 3) having such a stereo camera module 195 obtains a stereo image of the front of the vehicle from the stereo camera module 195 and based on the stereo image, a disparity. ) Detect, perform object detection on the at least one stereo image based on the disparity information, and continuously track the movement of the object after object detection.

On the other hand, similarly, the vehicle driving assistance apparatus (100 of FIG. 3) provided with the 1st and

2nd side cameras

197L and 197R has the vehicle left, Acquire an image for the right rear sequentially, perform disparity detection based on the sequentially acquired image, perform object detection on the image based on the disparity information, and then detect the object. , Continuously, you can track the movement of the object.

The vehicle driving assistance apparatus 100 of FIGS. 3A to 3B may process the stereo image received from the stereo camera 195 based on computer vision to generate vehicle related information. The vehicle related information may include vehicle control information for direct control of the vehicle, or vehicle driving assistance information for driving guide to the vehicle driver.

First, referring to FIG. 3A, the vehicle driving assistance apparatus 100 of FIG. 3A includes a communication unit 120, an interface unit 130, a memory 140, a processor 170, a power supply unit 190, and a stereo camera. 195, and first and

second side cameras

197L and 197R. In addition, an audio input unit (not shown) and an audio output unit (not shown) may be provided.

The communication unit 120 may exchange data with the mobile terminal 600 or the server 500 in a wireless manner. In particular, the communication unit 120 may exchange data wirelessly with a mobile terminal of a vehicle driver. As a wireless data communication method, various data communication methods such as Bluetooth, WiFi Direct, WiFi, and APiX are possible.

The communication unit 120 may receive weather information, road traffic information, for example, TPEG (Transport Protocol Expert Group) information from the mobile terminal 600 or the server 500. In the meantime, the vehicle driving assistance apparatus 100 may transmit the real-time traffic information grasped based on the stereo image to the mobile terminal 600 or the server 500.

On the other hand, when the user rides in the vehicle, the user's mobile terminal 600 and the vehicle driving assistance device 100 may perform pairing with each other automatically or by executing the user's application.

The interface unit 130 may receive vehicle-related data or transmit a signal processed or generated by the processor 170 to the outside. To this end, the interface unit 130 may perform data communication with the ECU 770, the AVN (Audio Video Navigation) device 400, the sensor unit 760, etc. in the vehicle by wired communication or wireless communication. have.

The interface unit 130 may receive map information related to vehicle driving through data communication with the AVN device 400.

The interface unit 130 may receive sensor information from the ECU 770 or the sensor unit 760.

Here, the sensor information includes vehicle direction information, vehicle position information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle tilt information, vehicle forward / reverse information, battery information, fuel information, tire information, vehicle It may include at least one of lamp information, vehicle interior temperature information, vehicle interior humidity information.

Such sensor information may include heading sensors, yaw sensors, gyro sensors, position modules, vehicle forward / reverse sensors, wheel sensors, vehicle speed sensors, It may be obtained from a vehicle body tilt sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor by steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, and the like. Meanwhile, the position module may include a GPS module for receiving GPS information.

Meanwhile, among the sensor information, the vehicle driving direction information, the vehicle position information, the vehicle angle information, the vehicle speed information, the vehicle tilt information, the side entry information, the backward information, and the like related to the vehicle driving may be called vehicle driving information.

The memory 140 may store various data for operations of the overall vehicle driving assistance apparatus 100, such as a program for processing or controlling the processor 170.

The audio output unit (not shown) converts an electrical signal from the processor 170 into an audio signal and outputs the audio signal. To this end, a speaker or the like may be provided. The audio output unit (not shown) may output sound corresponding to the operation of the input unit 110, that is, the button.

The audio input unit (not shown) may receive a user voice. To this end, a microphone may be provided. The received voice may be converted into an electrical signal and transmitted to the processor 170.

The processor 170 controls the overall operation of each unit in the vehicle driving assistance apparatus 100.

In particular, the processor 170 performs computer vision-based signal processing. Accordingly, the processor 170 obtains a stereo image of the front of the vehicle from the stereo camera 195, performs a disparity operation on the front of the vehicle based on the stereo image, and based on the calculated disparity information. , Object detection may be performed on at least one of the stereo images, and after the object detection, the movement of the object may be continuously tracked.

In particular, when detecting an object, the processor 170 may perform lane detection, vehicle detection, pedestrian detection, traffic sign detection, road surface detection, and the like. Can be.

The processor 170 may perform a distance calculation on the detected surrounding vehicle, a speed calculation of the detected surrounding vehicle, a speed difference calculation with the detected surrounding vehicle, and the like.

Meanwhile, the processor 170 sequentially acquires images of the rear left and right sides of the vehicle from the first and

second side cameras

197L and 197R, and based on the sequentially obtained images, the disparity The detection may be performed, and object detection may be performed on the image based on the disparity information, and after the object detection, the movement of the object may be continuously tracked.

In particular, the processor 170 may calculate surrounding vehicles located at the rear left and right of the vehicle, calculate a distance from the surrounding vehicle, a speed of the surrounding vehicle, or a relative speed with the surrounding vehicle.

Meanwhile, the processor 170 may receive weather information, road traffic information, for example, TPEG (Transport Protocol Expert Group) information through the communication unit 120.

On the other hand, the processor 170, the vehicle driving assistance apparatus 100 may grasp, in real time, the traffic situation information around the vehicle, which is determined based on the stereo image.

The processor 170 may receive map information or the like from the AVN device 400 through the interface unit 130.

Meanwhile, the processor 170 may receive sensor information from the ECU 770 or the sensor unit 760 through the interface unit 130. Here, the sensor information includes vehicle direction information, vehicle position information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle tilt information, vehicle forward / reverse information, battery information, fuel information, tire information, vehicle It may include at least one of lamp information, vehicle interior temperature information, vehicle interior humidity information.

On the other hand, the processor 170 detects the distance of the surrounding vehicle, the speed of the surrounding vehicle, or the speed difference with the surrounding vehicle based on the image, and determines the distance from the surrounding vehicle and the speed of the surrounding vehicle based on the image. Or, based on the difference in speed with the surrounding vehicle, it is possible to calculate the rear and rear risk of the vehicle, it is possible to generate the rear and rear hazard level information based on the calculated risk.

In this case, at least one of the display color, the size of the display level, and the brightness of the display level may vary according to the calculated risk.

On the other hand, when the calculated risk exceeds the allowable value, the processor 170 may generate a side entrance prevention control signal for controlling at least one of the steering driver and the brake driver provided in the vehicle.

On the other hand, the processor 170, based on at least one of the distance between the surrounding vehicle located in the rear and rear of the vehicle, the speed of the surrounding vehicle or the speed difference with the surrounding vehicle, the processor 170 with the surrounding vehicle located in the rear and rear of the vehicle The estimated collision time (TTC) can be calculated.

The power supply unit 190 may supply power required for the operation of each component under the control of the processor 170. In particular, the power supply unit 190 may receive power from a battery inside the vehicle.

The stereo camera 195 may include a plurality of cameras. Hereinafter, as described in FIG. 2 and the like, two cameras are provided.

The stereo camera 195 may be detachable from the ceiling or the windshield of the vehicle 200, and may include a first camera 195a having a first lens 193a and a second camera having a second lens 193b. 195b.

On the other hand, the stereo camera 195 has a first light shield 192a and a second light shield for shielding light incident on the first lens 193a and the second lens 193b, respectively. The part 192b may be provided.

The first and

second side cameras

197L and 197R are attachable to both side mirrors 107L and 107R of the vehicle 200, respectively, to capture images of the left rear and right rear of the vehicle, respectively. It can work.

On the other hand, the first and

second side cameras

197L and 197R may be disposed under the case covering the left and right side mirrors 107L and 107R to protect against foreign matter.

Next, referring to FIG. 3B, the vehicle driving assistance apparatus 100 of FIG. 3B may further include an input unit 110 and a display 180 as compared to the vehicle driving assistance apparatus 100 of FIG. 3A. Hereinafter, only the description of the input unit 110 and the display 180 will be described.

The input unit 110 may include a plurality of buttons or a touch screen attached to the vehicle driving assistance apparatus 100, particularly, the stereo camera 195. Through a plurality of buttons or a touch screen, the vehicle driving assistance apparatus 100 may be turned on and operated. In addition, various input operations may be performed.

The display 180 may display an image related to the operation of the vehicle driving assistance apparatus. For displaying such an image, the display 180 may include a cluster or a head up display (HUD), a display in a side mirror, or the like in front of a vehicle. On the other hand, when the display 180 is a HUD, it may include a projection module for projecting an image on the windshield of the vehicle 200.

4A-4B illustrate various examples of internal block diagrams of the processor of FIGS. 3A-3B, and FIGS. 5A-5B are diagrams referred to in describing the operation of the processor of FIGS. 4A-4B.

First, referring to FIG. 4A, FIG. 4A is an example of an internal block diagram of the processor 170. The processor 170 in the vehicle driving assistance apparatus 100 includes an image preprocessor 410 and a disparity calculator 420. ), An object detector 434, an object tracking unit 440, and an application unit 450 may be provided.

The image preprocessor 410 receives a stereo image from the stereo camera 195 and performs preprocessing.

In detail, the image preprocessing unit 410 may include noise reduction, rectification, calibration, color enhancement, and color space conversion for a stereo image. CSC), interpolation, camera gain control, and the like. Accordingly, a stereo image that is clearer than a stereo image captured by the stereo camera 195 may be obtained.

The disparity calculator 420 receives a stereo image signal-processed by the image preprocessor 410, performs stereo matching on the received stereo images, and performs stereo matching. Obtain a disparity map. That is, disparity information about the stereo image of the front of the vehicle may be obtained.

In this case, the stereo matching may be performed in units of pixels of stereo images or in units of predetermined blocks. The disparity map may refer to a map in which stereo parallax information of stereo images, that is, left and right images, is numerically represented.

The segmentation unit 432 may perform segmentation and clustering on at least one of the stereo images based on the disparity information from the disparity calculator 420.

In detail, the segmentation unit 432 may separate the background and the foreground from at least one of the stereo images based on the disparity information.

For example, an area in which the disparity information is less than or equal to a predetermined value in the disparity map may be calculated in the background, and the corresponding part may be excluded. Thereby, the foreground can be relatively separated.

As another example, an area in which the disparity information is greater than or equal to a predetermined value in the disparity map may be calculated in the foreground and a corresponding portion may be extracted. Thereby, the foreground can be separated.

As described above, by separating the foreground and the background based on the disparity information information extracted based on the stereo image, the signal processing speed, the signal processing amount, and the like can be shortened in the subsequent object detection.

Next, the object detector 434 may detect the object based on the image segment from the segmentation unit 432.

That is, the object detector 434 may detect an object with respect to at least one of the stereo images based on the disparity information information.

In detail, the object detector 434 may detect an object with respect to at least one of the stereo images. For example, an object can be detected from the foreground separated by image segments.

Next, an object verification unit 436 classifies and verifies the separated object.

To this end, the object verification unit 436 may include an identification method using a neural network, a support vector machine (SVM) method, a method of identifying by AdaBoost using a haar-like feature, or a histograms of oriented gradients (HOG). Techniques can be used.

The object checking unit 436 may check the objects by comparing the objects stored in the memory 140 with the detected objects.

For example, the object checking unit 436 may check surrounding vehicles, lanes, road surfaces, signs, dangerous areas, tunnels, and the like, which are positioned around the vehicle.

The object tracking unit 440 performs tracking on the identified object. For example, in order to sequentially identify the object in the obtained stereo images, calculate the motion or motion vector of the identified object, track the movement of the object, etc. based on the calculated motion or motion vector. Can be. Accordingly, it is possible to track surrounding vehicles, lanes, road surfaces, signs, dangerous areas, tunnels, and the like, which are located around the vehicle.

Next, the application unit 450 may calculate a risk of the vehicle 200 based on various objects located around the vehicle, for example, another vehicle, a lane, a road surface, a sign, and the like. In addition, it is possible to calculate the possibility of colliding with the vehicle ahead, whether the vehicle slips.

The application unit 450 may output, as vehicle driving assistance information, a message for informing the user of such information, based on the calculated risk, the possibility of colliding or the slip. Alternatively, a control signal for attitude control or driving control of the vehicle 200 may be generated as vehicle control information.

4B is another example of an internal block diagram of a processor.

Referring to the drawings, the processor 170 of FIG. 4B has the same internal configuration unit as the processor 170 of FIG. 4A, but the signal processing order is different. Only the differences are described below.

The object detector 434 may receive a stereo image and detect an object with respect to at least one of the stereo images. Unlike FIG. 4A, based on the disparity information, for the segmented image, the object may be detected directly from the stereo image, instead of detecting the object.

Next, the object verification unit 436 classifies the detected and separated objects based on the image segments from the segmentation unit 432 and the objects detected by the object detection unit 434. , Verify.

Meanwhile, a signal processing process similar to that of FIG. 4A or 4B may also be performed on the images acquired by the first and

second side cameras

197L and 197R.

That is, the image preprocessor 410 sequentially receives images acquired by the first and

second side cameras

197L and 197R, performs preprocessing, and a disparity calculator. ) 420 receives the ordered image signal-processed by the image preprocessor 410, performs matching on the received sequential images, and obtains a disparity map according to the matching. Can be. That is, disparity information about the left and right rear of the vehicle can be obtained.

Thereafter, the segmentation unit 432, the object detection unit 434, and the object confirmation unit 436 may check the surrounding vehicles and the like on the left and right rear sides of the vehicle.

In addition, the object tracking unit 440 may track the movement of the corresponding object, based on the identified motion or motion vector of the objects in the sequential images. Accordingly, it is possible to track the surrounding vehicles located in the rear left and right of the vehicle.

5A and 5B are views referred to for describing a method of operating the processor 170 of FIG. 4A based on stereo images obtained in the first and second frame sections, respectively.

First, referring to FIG. 5A, during the first frame period, the stereo camera 195 acquires a stereo image.

The disparity calculator 420 in the processor 170 receives the stereo images FR1a and FR1b signal-processed by the image preprocessor 410 and performs stereo matching on the received stereo images FR1a and FR1b. To obtain a disparity map 520.

The disparity map 520 is a leveling disparity between the stereo images FR1a and FR1b. The greater the disparity level is, the closer the vehicle is to the distance, and the smaller the disparity level is, the lower the disparity map is. We can calculate that distance of is far.

On the other hand, when displaying such a disparity map, the disparity map may be displayed such that the larger the disparity level, the higher the luminance, and the smaller the disparity level, the lower the luminance.

In the drawing, in the disparity map 520, the first to

fourth lanes

528a, 528b, 528c, 528d and the like have corresponding disparity levels, respectively, the construction area 522, the first front vehicle 524. For example, each of the second front vehicles 526 has a corresponding disparity level.

The segmentation unit 432, the object detection unit 434, and the object confirmation unit 436 may perform segment, object detection, and detection on at least one of the stereo images FR1a and FR1b based on the disparity map 520. Perform object verification.

In the figure, using the disparity map 520, it illustrates that object detection, and verification for the second stereo image FR1b is performed.

That is, in the image 530, the first to

fourth lanes

538a, 538b, 538c, 538d, the construction area 532, the first front vehicle 534, and the second front vehicle 536 detect an object. And confirmation can be performed.

Next, referring to FIG. 5B, during the second frame period, the stereo camera 195 acquires a stereo image.

The disparity calculator 420 in the processor 170 receives the stereo images FR2a and FR2b signal-processed by the image preprocessor 410 and performs stereo matching on the received stereo images FR2a and FR2b. To obtain a disparity map 540.

In the drawing, in the disparity map 540, the first to

fourth lanes

548a, 548b, 548c, 548d and the like have corresponding disparity levels, respectively, the construction area 542, the first front vehicle 544. For example, each of the second front vehicles 546 has a corresponding disparity level.

The segmentation unit 432, the object detection unit 434, and the object confirmation unit 436 may perform segment, object detection, and detection on at least one of the stereo images FR2a and FR2b based on the disparity map 520. Perform object verification.

In the figure, using the disparity map 540, it illustrates that object detection, and verification is performed on the second stereo image FR2b.

That is, in the image 550, the first to

fourth lanes

558a, 558b, 558c and 558d, the construction area 552, the first front vehicle 554, and the second front vehicle 556 detect the object. And confirmation can be performed.

Meanwhile, the object tracking unit 440 may perform tracking on the identified object by comparing FIGS. 5A and 5B.

In detail, the object tracking unit 440 may track the movement of the corresponding object based on the motion or the motion vector of each object identified in FIGS. 5A and 5B. Accordingly, tracking of a lane, a construction area, a first front vehicle, a second front vehicle, and the like located around the vehicle can be performed.

6A to 6B are views referred to for describing the operation of the vehicle driving assistance apparatus of FIG. 3.

First, FIG. 6A is a diagram illustrating a situation in front of a vehicle captured by the stereo camera 195 provided in a vehicle. In particular, the vehicle front situation is displayed in a bird eye view.

Referring to the drawings, from the left to the right, the first lane 642a, the second lane 644a, the third lane 646a, the fourth lane 648a is located, the first lane 642a and the second A construction area 610a is located between the lanes 644a, a first front vehicle 620a is located between the second lane 644a and the third lane 646a, and the third lane 646a and the fourth lane. It can be seen that the second front vehicle 630a is disposed between the lanes 648a.

Next, FIG. 6B illustrates displaying the vehicle front situation detected by the vehicle driving assistance apparatus together with various types of information. In particular, the image as shown in FIG. 6B may be displayed on the display 180 or the AVN device 400 provided in the vehicle driving assistance apparatus.

6B illustrates that information display is performed based on an image captured by the stereo camera 195, unlike FIG. 6A.

Referring to the drawings, from the left to the right, the first lane 642b, the second lane 644b, the third lane 646b, the fourth lane 648b, the first lane 642b and the second lane A construction area 610b is located between the lanes 644b, a first front vehicle 620b is located between the second lane 644b and the third lane 646b, and the third lane 646b and the fourth lane. It can be seen that the second front vehicle 630b is disposed between the lanes 648b.

The vehicle driving assistance apparatus 100 performs signal processing based on the stereo image captured by the stereo camera 195 to provide the construction area 610b, the first front vehicle 620b, and the second front vehicle 630b. You can check the object. In addition, the first lane 642b, the second lane 644b, the third lane 646b, and the fourth lane 648b may be identified.

On the other hand, in the drawing, to illustrate the object confirmation for the construction area 610b, the first front vehicle 620b, the second front vehicle 630b, it is illustrated that each is highlighted with a border.

On the other hand, the vehicle driving assistance apparatus 100 is based on the stereo image captured by the stereo camera 195, the distance to the construction area 610b, the first front vehicle 620b, the second front vehicle 630b. Information can be calculated.

In the drawing, the calculated first distance information 611b, second distance information 621b, and third distance corresponding to each of the construction area 610b, the first front vehicle 620b, and the second front vehicle 630b. It illustrates that information 631b is displayed.

Meanwhile, the vehicle driving assistance apparatus 100 may receive sensor information about the vehicle from the ECU 770 or the sensor unit 760. In particular, the vehicle speed information, the gear information, the yaw rate information (yaw rate) indicating the speed at which the rotation angle (concave angle) of the vehicle changes, and the angle information of the vehicle can be received, and such information can be displayed.

In the drawing, the vehicle speed information 672, the gear information 671, and the yaw rate information 673 are displayed on the vehicle front image upper portion 670, and the angle of the vehicle is displayed on the vehicle front image lower portion 680. While the information 682 is illustrated, various examples are possible. In addition, the width information 683 of the vehicle and the curvature information 681 of the road may be displayed together with the angle information 682 of the vehicle.

On the other hand, the vehicle driving assistance apparatus 100 may receive speed limit information, etc. for the road on which the vehicle is traveling, through the communication unit 120 or the interface unit 130. In the figure, it illustrates that the speed limit information 640b is displayed.

The vehicle driving assistance apparatus 100 may display various pieces of information illustrated in FIG. 6B through the display 180. Alternatively, the vehicle driving assistance device 100 may store various pieces of information without additional display. In addition, the information may be used for various applications.

FIG. 6C is a diagram illustrating a situation in which the left and right rear vehicles are photographed by the left and

right side cameras

197L and 197R provided in the vehicle. In particular, the vehicle left and right rear view is displayed in a bird eye view.

Referring to the drawings, from the left to the right, the first lane 642a, the second lane 644a, the third lane 646a, the fourth lane 648a are located, the second lane 644a and the third Between the lanes 646a, the vehicle 200 of the user himself is located, followed by the first rear vehicle 660a, between the third lane 646a and the fourth lane 648a, followed by For example, the second rear vehicle 670a is positioned between the first lane 642a and the second lane 644a.

Next, FIG. 6D illustrates a vehicle right rear image 670 captured by the vehicle driving assistance apparatus 100.

In the drawings, for convenience of description, the image photographed by the right side camera 197R is shown as being inverted left and right. Hereinafter, the image captured by the right side camera 197R will be described based on the inverted left and right as shown in the figure.

The vehicle right rear image 670 may include a second lane 644b, a third lane 646b, and a fourth lane 648b among four lanes, and in particular, the third lane 646b and the fourth lane. Between the lanes 648b, a first rear vehicle 660b may be included.

The processor 170 of the vehicle driving assistance apparatus 100 may determine an object for the first rear vehicle 660b by signal processing based on the right rear image 670 of the vehicle. In addition, the second lane 644b, the third lane 646b, and the fourth lane 648b may be identified.

On the other hand, unlike the drawing, in order to indicate the object recognition for the first rear vehicle 660b, similar to FIG. 6B, it is also possible to highlight the border.

Meanwhile, the processor 170 of the vehicle driving assistance apparatus 100 based on the image photographed by the right side camera 197R, distance information about the first rear vehicle 660b and the speed of the first rear vehicle 660b. Information, relative speed information between the first rear vehicle 660b and the vehicle 200 may be detected.

The processor 170 of the vehicle driving assistance apparatus 100 may perform various applications, such as determining a risk on the right rear side, using such information.

Referring to the drawings, the vehicle 200 may include an electronic control apparatus 700 for controlling the vehicle. The electronic control apparatus 700 can exchange data with the vehicle driving assistance apparatus 100 and the AVN apparatus 400 described above.

The electronic control apparatus 700 includes an input unit 710, a communication unit 720, a memory 740, a lamp driver 751, a steering driver 752, a brake driver 753, a power source driver 754, and a sunroof driver. 755, suspension driver 756, air conditioning driver 757, window driver 758, airbag driver 759, sensor unit 760, ECU 770, display unit 780, audio output unit 785. The power supply unit 790 may be provided.

The input unit 710 may include a plurality of buttons or a touch screen disposed in the vehicle 200. Through a plurality of buttons or touch screens, it is possible to perform various input operations.

The communication unit 720 may exchange data with the mobile terminal 600 or the server 500 in a wireless manner. In particular, the communication unit 720 may exchange data wirelessly with the mobile terminal of the vehicle driver. As a wireless data communication method, various data communication methods such as Bluetooth, WiFi Direct, WiFi, and APiX are possible.

The communication unit 720 may receive weather information, road traffic information, for example, TPEG (Transport Protocol Expert Group) information from the mobile terminal 600 or the server 500.

On the other hand, when the user is in the vehicle, the mobile terminal 600 and the electronic control apparatus 700 of the user can perform pairing with each other automatically or by executing the user's application.

The memory 740 may store various data for operating the entire electronic control apparatus 700, such as a program for processing or controlling the ECU 770.

The lamp driver 751 may control turn on / off of lamps disposed inside and outside the vehicle. In addition, it is possible to control the intensity, direction, etc. of the light of the lamp. For example, control of a direction indicator lamp, a brake lamp, and the like can be performed.

The steering driver 752 may perform electronic control of a steering apparatus in the vehicle 200. As a result, the traveling direction of the vehicle can be changed.

The brake driver 753 may perform electronic control of a brake apparatus in the vehicle 200. For example, the speed of the vehicle 200 may be reduced by controlling the operation of the brake disposed on the vehicle. As another example, by varying the operation of the brakes disposed on the left wheels and the right wheels, the traveling direction of the vehicle 200 may be adjusted to the left or the right.

The power source driver 754 may perform electronic control of the power source in the vehicle 200.

For example, when the fossil fuel-based engine is a power source, the power source driver 754 may perform electronic control of the engine. Thereby, the output torque of an engine, etc. can be controlled.

As another example, when the electric-based motor is a power source, the power source driver 754 may perform control on the motor. Thereby, the rotation speed, torque, etc. of a motor can be controlled.

The sunroof driver 755 may perform electronic control of a sunroof apparatus in the vehicle 200. For example, the opening or closing of the sunroof can be controlled.

The suspension driver 756 may perform electronic control of a suspension apparatus in the vehicle 200. For example, when the road surface is curved, the suspension device may be controlled to control the vibration of the vehicle 200 to be reduced.

The air conditioning driver 757 may perform electronic control of an air cinditioner in the vehicle 200. For example, when the temperature inside the vehicle is high, the air conditioner may operate to control the cool air to be supplied into the vehicle.

The window driver 758 may perform electronic control of a suspension apparatus in the vehicle 200. For example, the opening or closing of the left and right windows of the side of the vehicle can be controlled.

The airbag driver 759 may perform electronic control of an airbag apparatus in the vehicle 200. For example, in case of danger, the airbag can be controlled to burst.

The sensor unit 760 senses a signal related to traveling of the vehicle 100. To this end, the sensor unit 760 may include a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / reverse sensor, and a wheel sensor. , A vehicle speed sensor, a vehicle body tilt sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor by steering wheel rotation, a vehicle internal temperature sensor, a vehicle internal humidity sensor, and the like.

Accordingly, the sensor unit 760 includes vehicle direction information, vehicle position information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle tilt information, vehicle forward / reverse information, battery information, fuel information, A sensing signal may be acquired for tire information, vehicle lamp information, vehicle interior temperature information, vehicle interior humidity information, and the like.

On the other hand, the sensor unit 760 may include an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake temperature sensor (ATS), a water temperature sensor (WTS), and a throttle. Position sensor (TPS), TDC sensor, crank angle sensor (CAS), etc. may be further provided.

The ECU 770 may control the overall operation of each unit in the electronic control apparatus 700.

By the input by the input unit 710, a specific operation may be performed, or a signal sensed by the sensor unit 760 may be received and transmitted to the vehicle driving assistance apparatus 100, and the map information may be transmitted from the AVN device 400. It can receive, and can control the operation of the driving unit (751, 752, 753, 754, 756).

In addition, the ECU 770 may receive weather information, road traffic condition information, for example, TPEG (Transport Protocol Expert Group) information from the communication unit 720.

The display unit 780 may display an image related to the operation of the vehicle driving assistance apparatus. For displaying such an image, the display unit 780 may include a cluster or a head up display (HUD) on the front surface of the vehicle. On the other hand, when the display unit 780 is a HUD, it may include a projection module for projecting an image on the windshield of the vehicle 200. Meanwhile, the display unit 780 may include a touch screen that can be input.

The audio output unit 785 converts the electrical signal from the ECU 770 into an audio signal and outputs the audio signal. To this end, a speaker or the like may be provided. The audio output unit 785 may output a sound corresponding to the operation of the input unit 710, that is, the button.

The power supply unit 790 may supply power required for the operation of each component under the control of the ECU 770. In particular, the power supply unit 790 may receive power from a battery (not shown) in the vehicle.

FIG. 8 is a flowchart illustrating a method of operating a vehicle driving assistance apparatus according to an embodiment of the present invention, and FIGS. 9 to 16B are views for explaining the operation method of FIG. 8.

According to an embodiment of the present invention, the vehicle driving assistance apparatus 200 is based on an image photographed by the first and

second side cameras

197L and 197R of the plurality of

cameras

195, 197L and 197R. The rear and rear risk of the can be calculated, and based on the risk, can be controlled to display the rear and rear risk level information on the display.

In addition, the vehicle driving assistance apparatus 200 may control to display an estimated time of collision (TCT) with a neighboring vehicle located in the rear and rear of the vehicle.

Therefore, the first and

second side cameras

197L and 197R of the plurality of

cameras

195, 197L and 197R will be described. For convenience of explanation, the right side camera 197R will be described mainly.

First, referring to FIG. 8, the processor 170 of the vehicle driving assistance apparatus 100 receives an image from the right side camera 197R (S810).

The processor 170 of the vehicle driving assistance apparatus 100 receives the first image and the second image sequentially or frame by frame through the right side camera 197R.

Next, the processor 170 of the vehicle driving assistance apparatus 100 receives at least one of sensor information and map information of the vehicle through the interface unit 130 (S820).

The sensor information of the vehicle may be received from the ECU 770 or the sensor unit 760 of the vehicle 200. In addition, the map information may be received from the AVN device 400 in the vehicle 200.

Meanwhile, the vehicle driving direction information, the vehicle position information, the vehicle angle information, the vehicle speed information, the vehicle tilt information, the side entry information, the backward information, and the like, related to the vehicle driving, may be referred to as the vehicle driving information among the sensor information.

Meanwhile, in relation to an embodiment of the present invention, the driving information of the vehicle may include vehicle speed information, vehicle traveling direction information, side entry information, and the like.

In relation to an embodiment of the present invention, the side entry information may include a side entry signal by a user's manipulation of a specific input unit (eg, a side entry button). Alternatively, the side entry information may be automatically generated by the ECU 770 based on the steering information by steering wheel rotation.

Next, the processor 170 of the vehicle driving assistance apparatus 100 determines whether the side entry information is received from the vehicle driving information (S830), and when the side entry information is received, the image, the vehicle driving information, and the map. Based on at least one of the information, the risk of calculating the rear and rear of the vehicle (S840). In addition, the processor 170 of the vehicle driving assistance apparatus 100 generates side rear danger level information based on the calculated risk level (S850).

The processor 170 of the vehicle driving assistance apparatus 100 may determine whether side entry information is included in sensor information of the vehicle received through the interface unit 130. When an operation signal for entering the right side is received by the user's operation, the processor 170 of the vehicle driving assistance apparatus 100 may determine that the side entry information is received.

In addition, the processor 170 of the vehicle driving assistance apparatus 100 calculates a side rear danger level of the vehicle based on at least one of an image, sensor information of the vehicle, and map information.

To this end, the processor 170 of the vehicle driving assistance apparatus 100 first performs image-based object detection.

The first image and the second image are sequentially obtained from the right side camera 197R, and the processor 170 of the vehicle driving assistance apparatus 100 uses the difference between the first image and the second image, and thus, for the right rear side, is obtained. Disparity information can be obtained.

The processor 170 of the vehicle driving assistance apparatus 100 may perform segment, object detection, and object verification on at least one of the first image and the second image by using the obtained disparity information. have. In addition, tracking of the identified object, for example, a right rear vehicle, may be performed.

Accordingly, confirmation, distance information acquisition, and speed information acquisition (or relative speed information acquisition) may be performed on the neighboring vehicles located on the right rear side of the vehicle. In addition, during object detection and confirmation, lane detection and road surface detection can be performed.

In addition, the processor 170 of the vehicle driving assistance apparatus 100 may acquire image-based vehicle surrounding information, that is, distance information with a vehicle located at a right rear side, and speed information of a vehicle located at a right rear side, acquired on an image basis. Or, based on the relative speed information, when entering the right side, the risk for the right rear can be calculated.

Alternatively, the processor 170 of the vehicle driving assistance apparatus 100 may interface with the distance information with the vehicle located at the right rear, the speed information of the vehicle located at the right rear, or the relative speed information acquired based on the image. By using at least one of the map information and the sensor information of the vehicle, which are received through the unit 130, the risk of the right rear side may be calculated when the right side is entered.

For example, the processor 170 of the vehicle driving assistance apparatus 100 may calculate speed information or relative speed information of the vehicle located at the right rear side based on the vehicle position information (GPS information) among the sensor information. The risk calculation may be performed based on the speed information or the relative speed information of the vehicle located at the right rear side.

As another example, the processor 170 of the vehicle driving assistance apparatus 100 may calculate the speed information or the relative speed information of the vehicle located at the right rear side based on the vehicle position information (GPS information) among the sensor information. , Based on the map information, the curvature information of the currently running road can be calculated, and the risk calculation can be performed based on the curvature information of the road, the speed information of the vehicle located at the right rear side, or the relative speed information.

On the other hand, when the vehicle having the first distance and the first speed is at the right rear side, the processor 170 of the vehicle driving assistance device 100 calculates the degree of risk of the first level, and the second In time, when a vehicle having a second distance closer to the first distance or having a second speed faster than the first speed is in the right rear side, the vehicle may calculate the degree of risk of the second level higher than the risk of the first level. have.

The processor 170 of the vehicle driving assistance apparatus 100 may generate side rear danger level information according to the calculated risk level. In addition, the generated rear and rear danger level information may be controlled to be output to the display 180 or to an audio output unit (not shown).

For example, when the rear and rear danger level information is displayed on the display 180, at least one of the display color, the size of the display level, and the brightness of the display level may vary according to the calculated risk for the user's intuitive recognition. Can be

The higher the risk, the more reddish the display color, the larger the level of the display level, the larger the area of the displayed area, the higher the brightness of the display level, or a combination thereof. As a result, the user may intuitively recognize the risk.

9 illustrates an example of an internal block diagram of a processor for displaying risk level information.

Referring to the drawings, the processor 170 of the vehicle driving assistance apparatus 100 may include a lane detector 910, a distance detector 920, a speed detector 930, and a risk calculator 940.

The processor 170 of the vehicle driving assistance apparatus 100 receives map information from the AVN device 400 through the interface unit 130, receives a stereo image Sim from the stereo camera 195, and controls the ECU. The vehicle driving information Scar may be received from the 770 or the sensor unit 760.

Here, the vehicle driving information may include vehicle driving direction information, vehicle position information (GPS information), vehicle angle information, vehicle speed information, vehicle tilt information, side entry information, backward information, and the like. The vehicle driving information may be part of the sensor information.

The lane detection unit 910 performs lane detection based on the first to third images Sim1, Sim2, and Sim3 from the front stereo camera 195 and the first and

second side cameras

197L and 197R, respectively. can do.

The stereo image Sim1 may be received from the dual front stereo camera 195, and the lane detector 910 may detect the lane based on the disparity of the stereo image Sim1.

Meanwhile, the mono images Sim2 and Sim3 may be received from the first and

second side cameras

197L and 197R, and the lane detection unit 910 may be configured according to the difference between the mono images Sim2 and Sim3 that are sequentially obtained. Based on this, the lane can be detected.

Similarly, the distance detector 920 and the speed detector 930 may be configured from the front stereo camera 195 and the first and

second side cameras

197L and 197R, respectively. Based on Sim3), the distance of the front vehicle, the speed, the distance of the right rear vehicle, the speed, the distance of the left rear vehicle, and the speed can be detected.

The lane detecting unit 910 is based on the first to third images Sim1, Sim2, and Sim3 from the front stereo camera 195 and the first and

second side cameras

197L and 197R, respectively. The lane behind the vehicle right side, the lane behind the vehicle left side can be detected.

The risk calculator 940 may calculate the risk based on the detected lane information, the speed information of the surrounding vehicle, the distance information, the vehicle driving information Scar, and the map information Smp.

For example, the risk calculator 940 may be configured such that the closer the distance to the surrounding vehicle is, the higher the speed of the surrounding vehicle is, the higher the current speed of the vehicle 200 is, the greater the steering angle of the driving vehicle is, the map information. Alternatively, the smaller the curvature of the road based on the image, the narrower the distance between the lane and the lane by the detected lane, or the greater the curvature of the road surface, the higher the risk.

In addition, the risk calculator 940 may classify and output the front risk information Sfd and the side risk information Ssd according to the calculated risk.

In an embodiment of the present invention, the side risk information is displayed on the display 180 by using the side risk information Ssd. And, such side risk information, the size, color, etc. of the level is changed according to the risk. Accordingly, the user can intuitively recognize the risk level.

When the calculated risk exceeds the allowable value, the processor 170 may generate a vehicle control signal Scoc for controlling at least one of a steering driver and a brake driver provided in the vehicle.

10A illustrates another example of an internal block diagram of the processor of FIG. 9.

Referring to the drawings, the processor 170 of the vehicle driving assistance apparatus 100 is an image-based vehicle based on each image received from the front stereo camera 195, the first and

second side cameras

197L and 197R. Peripheral information can be detected. Here, the image-based vehicle surrounding information may include distance information, surrounding vehicle distance information, speed information, lane information, road surface information, display panel information, and the like.

As described above, the disparity information may be calculated based on the stereo image received from the front stereo camera 195, and the segment, the object detection, and the object confirmation may be performed based on the calculated disparity information.

Based on this, the processor 170 of the vehicle driving assistance apparatus 100 includes a vehicle detection 1010, a distance detection 1011, a lane detection 1012, a road surface detection 1014, and a visual driving for the front of the vehicle. Visual odometry 1016 may be performed.

Meanwhile, the disparity information is calculated based on the sequential mono images received from the first and

second side cameras

197L and 197R, and the segment, the object detection, and the object confirmation are performed based on the calculated disparity information. Can be.

Based on this, the processor 170 of the vehicle driving assistance apparatus 100 includes a vehicle detection 1010, a distance detection 1011, a lane detection 1012, a road surface detection 1014, Visual odometry 1016 may be performed.

Next, the processor 170 of the vehicle driving assistance apparatus 100 may receive sensor information from the ECU 770 or the sensor unit 760.

The processor 170 of the vehicle driving assistance apparatus 100 performs vehicle dead reckoning 1030 based on vehicle driving information from the ECU 770 or the sensor unit 760.

In addition, the processor 170 of the vehicle driving assistance apparatus 100 performs the vehicle ego motion tracking 1040 based on the vehicle dead reckoning. At this time, in addition to vehicle dead reckoning, it is also possible to perform vehicle egomotion tracking 1040 based on visual odometry.

The processor 170 of the vehicle driving assistance apparatus 100 may perform a curvature calculation 1050 of the driving road based on the vehicle egomotion tracking 1040.

Meanwhile, the processor 170 of the vehicle driving assistance apparatus 100 performs the tracking direction 1060 of the vehicle based on the curvature calculation 1050, the lane detection 1010, and the road surface detection 1014. do.

In addition, the processor 170 of the vehicle driving assistance apparatus 100 may collide with the vehicle based on the progress tracking of the vehicle 1060, the vehicle detection 1010, the distance detection 1011, and the road surface detection 1014. Risk calculation, such as collision risk, may be performed 1065.

Next, the processor 170 of the vehicle driving assistance apparatus 100 may generate hazard level information, in particular, side hazard level information Ssd, based on the calculated risk information. The generated side danger level information Ssd may be output to the outside, in particular, to the display 180. Accordingly, the display 180 can display the side danger level information so that the user can recognize it.

On the other hand, the processor 170 determines (1080) whether or not the calculated risk exceeds the allowable value, and, if applicable, to control at least one of the steering drive unit and the brake drive unit provided in the vehicle, side entry prevention control The signal Sse may be generated 1090. On the other hand, if it does not exceed, it may be controlled to perform the normal driving (1085).

The generated side entry prevention control signal Sse is output to the outside, in particular, to the ECU 770 of the vehicle, and the ECU 770 includes a steering driver 752, a brake driver 753, and a power source driver ( 754, the suspension driver 756 may be controlled.

For example, when the right entry prevention control signal is generated by the processor 170 of the vehicle driving assistance apparatus 100, based on this, the steering driver 752 in the vehicle 200 controls the steering apparatus to perform left steering. Control or the brake drive unit 753 may operate the right brake. Accordingly, the vehicle and the vehicle driver can be protected.

On the other hand, although not shown in FIG. 10A, after the risk calculation 1065, the processor 170 of the vehicle driving assistance apparatus 100 is based on the risk calculation 1065 and the vehicle egomotion tracking 1040. Vehicle path prediction may be performed. In addition, the processor 170 may calculate whether the vehicle is slipped based on the predicted vehicle path. When the vehicle is in a sleep state, the slip prevention control signal may be generated to control at least one of the steering driver 752, the brake driver 753, the power source driver 754, and the suspension driver 756.

FIG. 10B illustrates another example of an internal block diagram of the processor of FIG. 9.

Referring to FIG. 10B, it is almost similar to FIG. 10A, except that map information from the AVN device 400 is received, and map matching 1020 is further performed. Only the differences are described below.

The processor 170 of the vehicle driving assistance apparatus 100 performs map matching based on the vehicle driving information among the sensor information from the ECU 770 or the sensor unit 760 and the map information from the AVN device 400. Perform 1020. In addition, the position of the current vehicle on the map may be varied based on vehicle driving information such as vehicle speed information.

Next, the processor 170 of the vehicle driving assistance apparatus 100 may further, in addition to the vehicle egomotion tracking 1040, calculate the curvature of the driving road based on the map matching information at the time of calculating the curvature of the driving road 1050. Calculation 1050 may be performed.

The processor 170 of the vehicle driving assistance apparatus 100 maps the current vehicle onto a map on the basis of GPS information, which is vehicle position information, and map information from the AVN device 400. You can.

11A to 11C are views referred to for explaining a risk determination for the right rear side.

FIG. 11A shows a curved road, in which a vehicle 200 is driving between a first lane 1242a and a second lane 1244a and to the right rear, in particular, between a second lane 1244a and a third lane 1246a. It illustrates that there is a rear vehicle 1220a.

The vehicle driving assistance apparatus 100 of the vehicle 200 includes a front stereo camera 195 and first and

second side cameras

197L and 197R, and accordingly, the front area CA1 and the left rear area ( CA2) and the right rear region CA3 can be photographed respectively.

FIG. 11B illustrates an image 1200b captured by the right side camera 197R at a first time, and FIG. 11C illustrates an image 1200c captured by a right side camera 197R at a second time after the first time. To illustrate.

The processor 170 of the vehicle driving assistance apparatus 100 detects the first to

fourth lanes

1242b, 1244b, 1246b, and 1248b based on the captured

images

1200b and 1200c, respectively, and the rear vehicle 1220b. Detection can be performed. In addition, distance detection and speed detection may be performed on the rear vehicle 1220b.

FIG. 11B shows the detected vehicle 1220b with a thicker edge in the meaning of highlighting, and displays distance information 1223b and speed information 1244b, respectively, and FIG. 11C shows the detected vehicle 1220b. For example, the frame is thickly displayed in the meaning of highlighting, and the distance information 1223c and the speed information 1244c are respectively shown.

Referring to FIG. 11C, it can be seen that the distance to the rear vehicle is reduced and the speed of the rear vehicle is higher than that of FIG. 11B. Based on this information, the processor 170 of the vehicle driving assistance apparatus 100 may set the risk for the right rear side higher than the first time at the second time.

12A through 16B illustrate various examples of outputting risk level information corresponding to the risk of FIG. 11B and the risk of FIG. 11C.

First, FIG. 12A corresponds to the risk of FIG. 11B, and illustrates that the first level of danger level information 1310 is displayed on the display 180a attached to the inner surface of the right side mirror 107R.

Next, FIG. 12B corresponds to the risk of FIG. 11C, and illustrates that the second level of danger level information 1315 is displayed on the display 180a attached to the inner surface of the right side mirror 107R.

Accordingly, the user can easily check the right rear danger level information through the right side mirror 107R when attempting to enter the right side. In addition, by confirming the risk level information corresponding to the variable risk, it is possible to intuitively recognize the actual degree of risk.

In the drawing, FIGS. 12A to 12B illustrate dangerous level information by varying the display level size, but alternatively, the display color may vary, the brightness of the display level may vary, and the like.

Next, FIG. 13A illustrates that the risk level information 1320 of the first level is displayed on the cluster 300 inside the vehicle, and FIG. 13B corresponds to the risk of FIG. 11C. For example, the risk level information 1325 of the second level is displayed on the cluster 300 inside the vehicle.

Accordingly, when the user attempts to enter the right side, the user can easily check the right rear danger level information through the cluster 300.

On the other hand, it is also possible to display the estimated time to collision (TTC) with the surrounding vehicles located in the rear and rear of the vehicle, calculated by the vehicle driving assistance apparatus 200, on the cluster 300 inside the vehicle. Accordingly, the user can simply check the estimated collision time.

Next, FIG. 14A corresponds to the risk level of FIG. 11B, in which an image is projected onto the output area 1109 of the vehicle windshield by the HUD, and the risk level information 1330 of the first level is displayed on the output area 1330. 14B illustrates that the risk level information 1335 of the second level is displayed in the output area 1109 of the vehicle windshield, corresponding to the risk level of FIG. 11C.

Accordingly, the user can simply check the right rear danger level information through the output area 1109 of the windshield while watching the front.

On the other hand, in the output area 1109 of the vehicle windshield, it is also possible to display the estimated time of collision (TTC) with the surrounding vehicle located in the rear and rear of the vehicle calculated by the vehicle driving assistance device 200. Do. Accordingly, the user can simply check the estimated collision time.

15A to 15B illustrate information displayed on the output area 1109 of the windshield of the vehicle by the HUD method in a state where the internal camera 1500 for detecting the user's gaze is provided inside the vehicle. In accordance with the eye movement, the movement is illustrated.

The processor 170 of the vehicle driving assistance apparatus 100 determines a gaze direction of the user based on an image including the user acquired by the in-vehicle camera 1500, and according to the gaze direction, the rear and rear danger level information. The display position of can be varied.

FIG. 15A corresponds to FIG. 14A, but considering that the user's gaze is on the right, the risk level information 1131 of the first level is displayed on the right side more than in FIG. 14A.

Meanwhile, FIG. 15B corresponds to FIG. 14B, but considering that the user's gaze is on the right side, the second level of danger level information 1136 is displayed on the right side than in FIG. 14A.

Accordingly, in view of the user's eyes, by displaying the information, the ease of use can be increased.

Next, FIGS. 16A to 16B illustrate outputting dangerous level information through the audio output unit.

FIG. 16A corresponds to the risk of FIG. 11B, and illustrates that the risk level information of the first level is output to the sound 1340 through the audio output unit.

FIG. 16B corresponds to the risk of FIG. 11C, and illustrates that the second level of danger level information is output to the sound 1340 through the audio output unit.

Accordingly, the user can easily grasp the danger level information only by listening to the sound.

Meanwhile, in FIGS. 8 to 16B, signal processing based on a mono image is described based on the fact that the first and

second side cameras

197L and 197R of the vehicle are mono cameras. It is also possible that the first and

second side cameras

197L and 197R are stereo cameras. Accordingly, the processor 170 may process the signal based on the stereo image.

On the other hand, the operation method of the vehicle driving assistance apparatus according to an embodiment of the present invention described above with reference to Figs. 8 to 16B, when attempting to enter the side while driving, the risk is calculated, and the side risk level information is displayed.

Such contents can be modified and applied to calculating the rear danger level and displaying the rear danger level information when the vehicle is backed up as shown in FIG. 17.

Since the operation method of FIG. 17 is similar to the operation method of FIG. 8, the following description focuses on the difference.

Referring to the drawings, steps 1610 to 1620 to S1620 correspond to steps 810 to 820 of FIG. 8, and thus description thereof is omitted.

In operation 1630, the processor 170 of the vehicle driving assistance apparatus 100 determines whether a backward signal is received from vehicle driving information (S1630), and if applicable, a stereo image and driving of the vehicle. Based on at least one of the information and the map information, the rear risk of the vehicle is calculated (S1640). The processor 170 of the vehicle driving assistance apparatus 100 displays the rear danger level information on the display 180 based on the calculated risk level (S1650).

The processor 170 of the vehicle driving assistance apparatus 100 combines the left rear stereo image and the right rear stereo image received from the first and

second side cameras

197L and 197R to determine a rear risk of the vehicle. The degree of risk can be calculated based on the combination and at least one of the driving information and the map information of the vehicle.

Accordingly, even when the vehicle is reversed, the user can easily recognize the rear danger level information that varies according to the calculated risk.

Referring to the drawings, the vehicle 200 according to another embodiment of the present invention, the wheels (150FR, 135FL, 135RL, ...) rotating by the power source, the handle 150 for adjusting the traveling direction of the vehicle 200 And a mono camera 193 and a radar 194 for photographing the vehicle 200 in front of the vehicle 200, and first and

second side radars

199L and 199R for detecting a distance to the lateral rear of the vehicle.

The mono image through the mono camera 193 and the distance information through the radar 194 and the first and

second side radars

199L and 199R may be signal processed in the vehicle driving assistance apparatus 2100 of FIG. 19A. have.

Referring to the drawings, an internal block of the vehicle driving assistance device 2100 of FIG. 19A is similar to an internal block of the vehicle driving assistance device 100 of FIG. 3, but based on a stereo image from a stereo camera 195. The difference is that the signal processing is performed based on the mono image from the mono camera 193 and the distance information through the

radars

194, 199L and 199R. In the following, only the differences are described.

The mono camera 193 may be detachable from the ceiling or the windshield of the vehicle 200 and may include a single camera having a lens.

The radar 194 may be detachable from the ceiling or the windshield of the vehicle 200, transmit radio waves of a predetermined frequency in front of the vehicle, and receive radio waves reflected from an object in front of the vehicle.

The processor 2170 may calculate distance information based on the difference between the transmission wave and the reception wave in the radar 194. In addition, by matching the mono image and the distance information through the mono camera 193, it is possible to perform segment, object detection, confirmation and the like.

Meanwhile, the processor 2170 may use distance information received from the

radars

199L and 199R to calculate a risk on the vehicle rear and rear. That is, based on distance information received from the

radars

199L and 199R, distance detection and speed detection may be performed on the lateral rear object.

On the other hand, it is also possible to further use a mono camera for image acquisition for the rear and rear of the vehicle.

Next, FIG. 19B illustrates an example of an internal block diagram of a vehicle driving assistance apparatus according to another embodiment of the present invention.

Referring to the drawings, the inner block of the vehicle driving assistance device 2150 of FIG. 19B is similar to the inner block of the vehicle driving assistance device 2100 of FIG. 19A, but the first and second blocks for detecting distance to the rear and rear sides of the vehicle are shown. The difference is provided with the first and second side riders (Lidar) 2102L, 2102R, which perform scanning on an external object rather than the second side radars (199L, 199R).

To describe the difference, the first and second

side riders Lidar

2102L and 2102R may acquire scanned images for left and right rear and rear situations by laser scanning, and the processor 2270 may include: Based on the scanned images received from the first and second

side riders Lidar

2102L and 2102R, signal processing may be performed, that is, side rear hazard level information may be generated.

20A-20B illustrate various examples of internal block diagrams of the processor of FIG. 19A, and FIGS. 21A-21B are diagrams referred to for describing the operation of the processor of FIG. 20A.

The processor internal blocks of FIGS. 20A to 20B are similar to those of the processors of FIGS. 4A to 4B except that they do not include a disparity calculator.

Instead, there is a difference in receiving the distance information Sd corresponding to the disparity information from the external radar 194. The distance information Sd is input to the segmentation unit 432 and may be used when segmenting an image.

21A to 21B are views referred to for describing a method of operating the processor 170 of FIG. 20A, based on a mono image acquired in each of the first and second frame periods.

Referring to FIGS. 21A to 21B, the method is similar to FIGS. 5A to 5B except that since a mono image and radar based distance information are used, a disparity map does not need to be generated.

First, referring to FIG. 21A, during a first frame period, the mono camera 193 acquires a mono image FR1, and the radar obtains distance information Sd1.

Accordingly, the segmentation unit 432, the object detection unit 434, and the object confirmation unit 436 in the processor 2170 may be configured to include the first segment in the mono image FR1 based on the radar based distance information Sd1. The lanes to the

fourth lanes

538a, 538b, 538c, 538d, the construction area 532, the first front vehicle 534, and the second front vehicle 536 can be detected and confirmed.

Next, referring to FIG. 21B, during the second frame period, the mono camera 193 obtains a mono image FR2 and the radar obtains distance information Sd2.

Accordingly, the segmentation unit 432, the object detection unit 434, and the object confirmation unit 436 in the processor 2170 may be configured to include the first segment in the mono image FR2 based on the radar based distance information Sd2. The lanes to the

fourth lanes

558a, 558b, 558c, and 558d, the construction area 552, the first front vehicle 554, and the second front vehicle 556 can be detected and confirmed.

Meanwhile, the object tracking unit 440 may perform tracking on the identified object by comparing FIG. 21A with FIG. 21B.

In detail, the object tracking unit 440 may track the movement of the corresponding object based on the motion or the motion vector of each object identified in FIGS. 21A and 21B. Accordingly, tracking of a lane, a construction area, a first front vehicle, a second front vehicle, and the like located around the vehicle can be performed.

On the other hand, the vehicle driving assistance device 2100 including the mono camera 193 and the

radar

194, 199L, 199R described with reference to FIGS. 18 to 21, the side when entering the side, as described with reference to FIGS. The rearward risk may be determined, and the rearward rearward danger level information may be generated according to the determined risk. In addition, the generated rear and rear danger level information may be displayed on the display 180.

On the other hand, the vehicle driving assistance device 2150 including the mono camera 193, the radar 194, and the first and

second side riders

2102L and 2102R described in FIG. 19B is described with reference to FIGS. 8 to 17. As described above, when entering the side, the side rear risk may be determined, and the side rear risk level information may be generated according to the determined risk. In addition, the generated rear and rear danger level information may be displayed on the display 180.

The vehicle driving assistance apparatus and the vehicle having the same according to an embodiment of the present invention are not limited to the configuration and method of the embodiments described above, but the embodiments may be modified in various ways so that various modifications can be made. All or part of the examples may be optionally combined.

On the other hand, the vehicle driving assistance apparatus or the method of operating the vehicle of the present invention can be implemented as code that the processor can read on a recording medium that can be read by the processor provided in the vehicle driving assistance apparatus or the vehicle. The processor-readable recording medium includes all kinds of recording devices that store data that can be read by the processor. Examples of the processor-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like, and may also be implemented in the form of a carrier wave such as transmission over the Internet. . The processor-readable recording medium can also be distributed over network coupled computer systems so that the processor-readable code is stored and executed in a distributed fashion.

In addition, although the preferred embodiment of the present invention has been shown and described above, the present invention is not limited to the specific embodiments described above, but the technical field to which the invention belongs without departing from the spirit of the invention claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or the prospect of the present invention.

The present invention is applicable to a vehicle driving assistance apparatus and a vehicle having the same, and in particular, to a vehicle driving assistance apparatus capable of displaying side and rear danger level information corresponding to a driving situation and a vehicle having the same.

Claims

A camera capable of photographing the rear side of the vehicle; And

And a processor configured to calculate a lateral rear hazard level of the vehicle based on the image received from the camera and side entry information, and to generate lateral rear hazard level information based on the risk level. Device.
The method of claim 1,

The processor,

Based on the image, detecting a distance of a surrounding vehicle located at the rear and rear of the vehicle, a speed of the surrounding vehicle, or a speed difference with the surrounding vehicle,

And calculating the rear and rear risks of the vehicle based on a distance from the surrounding vehicle, a speed of the surrounding vehicle, or a speed difference from the surrounding vehicle.
The method of claim 2,

The processor,

Calculating an estimated collision time with a neighboring vehicle located at the rear and rear of the vehicle based on at least one of the distance of the surrounding vehicle, the speed of the surrounding vehicle, or the speed difference with the surrounding vehicle; Auxiliary devices.
The method of claim 1,

The processor,

A distance detector configured to detect a distance of a neighboring vehicle located in the side rear of the vehicle based on the image;

A speed detector configured to detect a speed of the surrounding vehicle based on the image; And

And a risk calculator configured to calculate a side rear hazard level of the vehicle based on the detected distance and speed, and generate the side rear hazard level information based on the calculated risk level. .
The method of claim 1,

The side rear hazard level information, at least one of the display color, the size of the display level, the brightness of the display level is variable according to the calculated risk.
The method of claim 1,

The processor,

And controlling the at least one of a steering driver and a brake driver included in the vehicle when the calculated risk level exceeds an allowable value.
The method of claim 1,

And a display for displaying the rear and rear danger level information.

The display,

And at least one of a projection module configured to output a projection image to an output area of the cluster of the vehicle or the windshield of the vehicle.
The method of claim 1,

It further comprises a vehicle interior camera for detecting the eyes of the user;

The processor,

Based on the image including the user obtained from the in-vehicle camera, the direction of the user's gaze is determined, and the display position of the lateral rear hazard level information is controlled to vary according to the gaze direction. Auxiliary devices.
The method of claim 1,

And a stereo camera for acquiring a stereo image of the front of the vehicle.

The processor,

A disparity calculator configured to perform a disparity operation of the stereo image;

An object detector configured to detect an object of at least one of the stereo images based on the disparity information of the stereo image;

And an object tracking unit that performs tracking on the detected object.
The method of claim 9,

The processor,

A segmentation unit segmenting an object in the stereo image based on disparity information of the stereo image; And

And an object checking unit to classify the detected object.

The object detector,

Based on the segmented object, for at least one of the stereo images

A vehicle driving assistance device, characterized in that for performing object detection.
The method of claim 1,

An interface unit for exchanging data with at least one device in the vehicle;

The interface unit receives sensor information of the vehicle,

The processor,

And calculating a rear and rear risk of the vehicle based on at least one of the image and sensor information of the vehicle.
The method of claim 11,

The processor,

Based on the image, image-based vehicle surrounding information including a distance from a surrounding vehicle and a speed difference from the surrounding vehicle is detected,

Based on the sensor information and the image-based vehicle information, perform a vehicle motion estimation,

Based on the image-based vehicle surrounding information, tracking the direction of travel of the vehicle,

And calculating the rear and rear risks of the vehicle based on the tracked direction of the vehicle and the image-based vehicle surrounding information.
The method of claim 12,

The interface unit further receives map information,

The processor,

Performing map matching on the vehicle based on the map information, the vehicle driving information and the position information among the sensor information,

And tracking the driving direction of the vehicle based on the map information and the image-based vehicle surrounding information.
A radar for detecting a distance to the rear and rear of the vehicle;

And a processor configured to calculate a lateral rear hazard level of the vehicle based on distance information detected from the radar and side entry information, and to generate rear hazard level information based on the hazard level. Device.
A rider performing scanning of the lateral rear of the vehicle;

And a processor configured to calculate a lateral rear hazard level of the vehicle based on the scanned image received from the rider and side entry information, and to generate rear hazard level information based on the hazard level. Device.
display;

A sensor unit for sensing a vehicle state;

A steering driver for driving a steering device;

A brake drive unit for driving a brake device,

An engine driver for driving an engine;

A suspension driver for driving the suspension device;

A controller for controlling the steering driver, the brake driver, the power source driver, and the suspension driver; And

Based on the camera capable of photographing the rear side of the vehicle, the stereo image received from the stereo camera, and the side entry information, the rear side rear hazard level of the vehicle is calculated and based on the risk side, the rear side rear side danger level information is generated. And a vehicle driving assistance device having a processor for controlling to display on the display.
The method of claim 16,

The side rear hazard level information displayed on the display is characterized in that at least one of the display color, the size of the display level, the brightness of the display level is variable according to the calculated risk.
The method of claim 16,

The processor,

When the calculated risk exceeds an allowable value, generates a lateral entry prevention control signal for controlling at least one of the steering driver and the brake driver;

The control unit,

And at least one of the steering driver and the brake driver based on the side restraint control signal.
The method of claim 16,

The display,

And at least one of a projection module configured to output a projection image to an output area of a cluster of the vehicle or the windshield of the vehicle.
The method of claim 16,

The vehicle driving assistance device,

It further comprises a vehicle interior camera for detecting the eyes of the user;

The processor,

And determine the gaze direction of the user based on an image including the user acquired by the in-vehicle camera, and control the display position of the side rear danger level information to vary according to the gaze direction.