KR101927162B1 - Sensor fusion system and method thereof - Google Patents

Abstract

A sensor fusion system and method thereof are disclosed.
The sensor fusion system considering the day / night camera recognition characteristic according to the embodiment of the present invention is characterized in that the radar data including at least one of the estimated lane information ahead of the vehicle measured through the radar, the distance between the preceding vehicle and the relative speed, A radar unit for measuring; A camera sensor unit for measuring camera data including forward object and lane information based on image image processing obtained through a camera installed in front of the vehicle; An automatic light-side sensor for measuring the brightness of light received from the outside; A day / night judging unit for judging whether the day / night judging information and the light brightness information are collected by the camera, And data for performing data association of the camera data by correcting the daytime gate to the nighttime gate that has been increased in the longitudinal direction in consideration of the camera recognition characteristics at night, And a correlation unit.

Description

TECHNICAL FIELD [0001] The present invention relates to a sensor fusion system,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a sensor fusion system and a method thereof, and more particularly, to a sensor fusion system and a method thereof in consideration of day and night camera recognition characteristics.

Generally, various systems are installed in the vehicle for protecting the driver and passengers, improving the driving assistance and ride comfort, and these systems are being improved by utilizing various sensors and information communication technologies.

In particular, a technology for recognizing lanes and performing automatic steering by using an image sensor through a camera has been put into practical use. In general, an image recognition processing apparatus provided in an automobile is provided with image information of a running lane and images Information on the left and right lanes, and displays the information on the display means through the display means, thereby providing convenient lane recognition for the driver, and information on the state of the road and the driving information of the surrounding vehicles.

In recent years, a sensor fusion system capable of extracting necessary information by fusing image information and radar information collected through the image sensor and radar has been developed.

The sensor fusion system is used to provide autonomous navigation system that recognizes lane information by using camera and automatic steering of vehicle or smart cruise control function of vehicle, but there is limit to information recognition through camera, Is required.

Figure 1 shows the daytime and nighttime fusion performance in a sensor fusion system mounted on a conventional passenger car.

Referring to FIG. 1, there is a problem that the fusion performance is rapidly deteriorated due to the difference in distance between the camera and the radar during nighttime driving, by applying one fusion logic even when the camera has different day / night recognition performance. In other words, the existing sensor fusion system is a system that compensates the shortcomings of radar / camera by simply using the information of camera and radar, and thus there is a desperate need to improve the performance of day / night recognition.

On the other hand, US Patent No. 6,590,521 discloses an object recognition system including a radar, an image sensor, and a controller, but has a problem in that it can not provide a solution against deterioration in day / night recognition performance.

Patent Document 1: United States Patent No. 6,590,521 (issued on July 2003)

The embodiment of the present invention is to provide a sensor fusion system and method in which night and night are distinguished and applied to fusion logic as the performance of recognition of the preceding vehicle at night is lowered.

According to one aspect of the present invention, a sensor fusion system that takes into consideration the day / night camera recognition characteristics is provided with a radar system including at least one of radar data including at least one of the estimated lane information ahead of the radar, A radar unit for measuring the radar unit; A camera sensor unit for measuring camera data including forward object and lane information based on image image processing obtained through a camera installed in front of the vehicle; An automatic light-side sensor for measuring the brightness of light received from the outside; A day / night judging unit for judging whether the day / night judging information and the light brightness information are collected by the camera, And correcting the daytime gate to a gate for nighttime that has been increased in the longitudinal direction in consideration of the camera recognition characteristics at night when it is determined to be the nighttime and to perform the data association of the camera data with the radar data And a correlation unit.

In addition, the day / night determining unit may include a logic circuit for determining whether the camera day / night judgment information and the light brightness information are all at night, at night.

The data correlation unit may further include a vehicle data fusion module that performs the data association by dividing the daytime gate into a daytime gate and the nighttime gate that is increased compared to the daytime gate; A lane data fusion module that fuses the estimated lane information of the radar unit and the lane information of the camera sensor unit to recognize a front lane of each lane; And a control module for measuring the preceding vehicle information by giving a correlation between the laser recognition result and the camera recognition result using the nighttime gate having a size increased in the vehicle data fusion module at night.

The radar unit may determine the preceding vehicle as at least one vehicle type of a passenger car, a truck, and a bus if the measured object has acceleration.

In addition, the vehicle data fusion module can further perform gate re-sizing by further dividing the nighttime gates according to the vehicle type according to nighttime camera recognition characteristics.

In addition, the vehicle data fusion module may perform gate re-sizing by further dividing the night gates by a predetermined unit distance according to camera recognition characteristics at night.

Meanwhile, a sensor fusion method considering a day / night camera recognition characteristic of a sensor fusion system of a vehicle according to an embodiment of the present invention includes: a) synchronizing a radar measurement result and a camera measurement result along a time axis; b) correcting the daytime gate to a nighttime gate that has been increased in the longitudinal direction by using the recognition characteristics of the camera in the nighttime according to the result of the day / night judgment; c) measuring a preceding vehicle information by performing a data association which gives a correlation between a laser recognition result and a camera recognition result using a night gateway; And d) tracking the position and movement information of the preceding vehicle by reflecting lane information on the preceding vehicle information.

In addition, the step b) may include determining whether the day / night determination information in the camera and the brightness of light received from the outside are daytime or nighttime.

Between the steps c) and d), the estimated lane information and the lane information obtained respectively from the radar unit and the camera sensor unit are fused so that the width of both lanes in front, the lateral position of the vehicle in the lane, The lane information including at least one of the distance of the lane, the shape of the lane, and the radius of curvature of the road.

The step b) may further include correcting the night gates by further distinguishing the night gates by the vehicle type or the distance from the preceding vehicle according to the camera recognition characteristics at night.

The embodiments of the present invention have an effect of improving the recognition performance of the vehicle ahead vehicle by performing the data association for the radar and the camera data by applying the gate size classified into the daytime and the nighttime to the fusion logic.

The effect of changing the gate size according to the vehicle type and thus preventing the camera data from being excluded in the data association is further reflected by further reflecting the distance from the preceding vehicle type or the preceding vehicle depending on the camera recognition characteristic at night have.

Figure 1 shows the daytime and nighttime fusion performance in a sensor fusion system mounted on a conventional passenger car.
FIG. 2 shows day and night recognition characteristics of a conventional camera according to the type of the vehicle.
FIG. 3 is a block diagram illustrating a sensor fusion system in consideration of day / night camera recognition characteristics according to an embodiment of the present invention.
4 shows a state in which the gate size is corrected by dividing the main / night according to the embodiment of the present invention.
5 shows a sensor fusion method according to the night and day division of the sensor fusion system in the embodiment of the present invention.
FIG. 6 illustrates a method of gate-by-object re-sizing according to an embodiment of the present invention.
FIG. 7 illustrates a gate re-sizing method according to an object distance according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when an element is referred to as "comprising ", it means that it can include other elements as well, without excluding other elements unless specifically stated otherwise. Also, the terms " part, "" module," and " module ", etc. in the specification mean a unit for processing at least one function or operation and may be implemented by hardware or software or a combination of hardware and software have.

Now, a sensor fusion system in consideration of day / night camera recognition characteristics according to an embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 2 shows day and night recognition characteristics of a conventional camera according to the type of the vehicle.

Referring to FIG. 2 of the accompanying drawings, graphs of measurement distances between camera and radar for daytime and nighttime for passenger cars, trucks, and buses (special vehicles) are shown.

When comparing A and A 'for a car, it can be seen that the distance measured by the camera becomes larger at nighttime (ie, the difference in the + direction becomes larger).

When comparing trucks B and B ', it can be seen that the distance measured by the camera at night becomes very inaccurate (ie, all + and - direction differences occur).

In the case of buses / special vehicles, when comparing C and C ', it can be seen that the distance measured by the camera becomes smaller at nighttime (ie, the difference in the - direction becomes larger).

As a result, in the present invention, the sensor fusion system of a vehicle applying the Fusion logic by dividing the day and night is proposed.

3 is a block diagram illustrating a sensor fusion system in consideration of day / night camera recognition characteristics according to an embodiment of the present invention.

3, the sensor fusion system 100 according to an exemplary embodiment of the present invention includes a radar unit 110, a camera sensor unit 120, an auto-light sensor 130, a day / night determination unit 140, and a data correlation unit 150.

The radar unit 110 measures radar data including at least one of the distance between the preceding vehicle and the preceding vehicle, the relative speed, and the vehicle type data measured in real time via radio waves or a laser, .

The camera sensor unit 120 transmits the camera data including the preceding vehicle information and the lane information to the data correlation unit 150 based on the image image processing obtained through the camera installed in front of the vehicle.

In addition, the camera sensor unit 120 transmits the main / night judgment information inside the camera to the day / night determiner 140. [

The auto-light sensor 130 measures the brightness of light received from the outside and transmits the light brightness information to the day / night determiner 140.

The day / night determiner 140 determines whether the day / night judgment information received from the camera sensor unit 120 and the light brightness information received from the automatic light sensor 130 are the current day or night.

For example, the day / night determiner 140 includes a logic circuit that determines (changes) from day to night only when both pieces of information received from the camera sensor unit 120 and the automatic light sensor 130 are all at night. Conversely, if both pieces of information are weekly, they can be changed to daytime.

This is because a judgment error may occur on the day when the weather is very cloudy, so that even if any information is inputted during the daytime, the daytime state can be maintained and the nighttime can be determined only when both information are confirmed at night.

The data correlation unit 150 includes a vehicle data fusion module 151 for fusing various sensor data collected from the radar unit 110, the camera sensor unit 120 and the day / night determination unit 140, a lane data fusion module 152 and a target selection module 153. [

When the vehicle data fusion module 151 receives the radar data and the camera data collected from the radar unit 110 and the camera sensor unit 120, the vehicle data fusion module 151 performs a data association for giving correlation to each data It is possible to estimate the distance to the preceding vehicle and track the preceding vehicle accordingly. At this time, the preceding vehicle can judge the presence or absence of acceleration of an object reflected through the lighter, and an object having no acceleration is determined as a fixed object.

Particularly, the vehicle data fusion module 151 according to the embodiment of the present invention uses a day and night determination result to determine a gate size for performing data association for the radar data and the camera data, And carries out the correction according to the information of the object (for example, the vehicle type).

4 shows a state in which the gate size is corrected by dividing the main / night according to the embodiment of the present invention.

4, the vehicle data fusion module 151 according to an embodiment of the present invention defines a gate that has been used in the past as a daytime gate, Is defined as a night gate.

Here, the gate size is a virtual area of a predetermined size generated at the position of an object based on radar data. In the past, one gate (daytime gate) There is a problem of recognition of the preceding vehicle due to the error between the data and the camera data.

That is, the data association is performed only when both the radar recognition result and the camera recognition result are present in the daytime gate as shown in (A) of FIG. On the other hand, when the camera recognition result is not present in the daytime gate as in (B) at night, there is a problem that the recognition rate of the preceding vehicle at night is lowered by excluding the camera recognition result and utilizing only the radar recognition result.

Accordingly, the vehicle data fusion module 151 increases the gate size in the longitudinal direction during data association using the recognition characteristics of the camera described in FIG. 2, defines the gate of the corrected size as the night gate (B) at night.

The lane data fusion module 152 fuses the estimated lane information and the lane information obtained respectively from the radar unit 110 and the camera sensor unit 120 to recognize the front two-lane status.

For example, the lane data fusion module 152 can recognize at least one of the width of the front lane, the lateral position of the vehicle in the lane, the distance to both lanes, the shape of the lane, and the radius of curvature of the road.

The control module 153 correlates the laser recognition result with the camera recognition result using the nighttime gate having the increased size at the vehicle data fusion module 151 at night according to the result of the day / night judgment, thereby measuring the preceding vehicle information. The position and movement information (direction, speed, etc.) of the preceding vehicle can be tracked by reflecting the lane information processed by the lane data fusion module 152.

The sensor fusion method according to the day and night division of the sensor fusion system according to the embodiment of the present invention described above will be briefly described below.

Meanwhile, FIG. 5 shows a sensor fusion method according to the night and day division of the sensor fusion system according to the embodiment of the present invention.

Referring to FIG. 5, the sensor fusion system 100 according to the embodiment of the present invention synchronizes the radar measurement result and the camera measurement result along a time axis (S11).

The sensor fusion system 100 corrects the daytime gate to the nighttime gate that is increased in the longitudinal direction by using the recognition characteristics of the camera in the nighttime according to the result of the day / night judgment (S12).

The sensor fusion system 100 carries out a data association that gives a correlation between the laser recognition result and the camera recognition result using the nightgate whose size is increased when it is judged as night according to the result of the day / (S13).

The sensor fusion system 100 may track the position and movement information of the preceding vehicle by reflecting lane information processed in the lane data fusion module 152 to the preceding vehicle information (S14).

As described above, according to the embodiment of the present invention, by applying the gate size classified into the daytime and the nighttime to the fusion logic, data association is performed with respect to the radar and the camera data, .

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary,

For example, in the embodiment of the present invention described above with reference to FIG. 4, data association is performed by dividing into a daytime gate and a nighttime gate according to the day / night division, but the present invention is not limited thereto, It is possible to perform gate resizing by dividing the vehicle type further.

FIG. 6 illustrates a method of gate-by-object re-sizing according to an embodiment of the present invention.

Referring to FIG. 6, according to the embodiment of the present invention, in addition to the application of the main / night division gate size described above, the recognition result of the truck / bus at night and the difference in distance between the recognition result of the passenger car and the preceding vehicle The data size can be increased by increasing the gate size according to the vehicle type after determining the gate size for the night.

In other words, as shown in FIG. 6, the night truck gate size is extended by a certain length in the lower direction based on the night passenger car gate size, and the gate size of the night special truck or bus is shifted in the downward direction from the truck gate size, Can be performed.

Therefore, it is possible to further change the gate size according to the vehicle type by reflecting the preceding vehicle type according to the camera recognition characteristic at night, and to prevent the camera data from being excluded in the data association.

FIG. 7 illustrates a gate re-sizing method according to an object distance according to an embodiment of the present invention.

Referring to FIG. 7, according to an embodiment of the present invention, it is also possible to divide the gate size of a passenger car at night into 50m units, and to add logic that varies the upper (+) and lower (-) directions in units of 10m.

Therefore, it is possible to further change the gate size according to the vehicle type and to prevent the camera data from being excluded in the data association, further reflecting the preceding vehicle type or the distance from the preceding vehicle depending on the camera recognition characteristic at night It is effective.

The embodiments of the present invention are not limited to the above-described apparatuses and / or methods, but may be implemented through a program for realizing functions corresponding to the configuration of the embodiment of the present invention, a recording medium on which the program is recorded And such an embodiment can be easily implemented by those skilled in the art from the description of the embodiments described above.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, It belongs to the scope of right.

100: sensor fusion system 110: radar unit
120: camera sensor unit 130: automatic light sensor
140: Day / Night Judgment Unit 150: Data Correlation Unit
151: vehicle data fusion module 152: lane data fusion module
153: Control module

Claims (10)

A radar unit for measuring radar data including at least one of a distance between a preceding vehicle, a relative speed, and vehicle type data, which is estimated through a radar;
A camera sensor unit for measuring camera data including forward object and lane information based on image image processing obtained through a camera installed in front of the vehicle;
An automatic light-side sensor for measuring the brightness of light received from the outside;
A day / night judging unit for judging whether the day / night judging information and the light brightness information are collected by the camera, And
And a data correlation unit for correcting the daytime gate to a gate for nighttime that has been increased in the longitudinal direction in consideration of the camera recognition characteristic at night when the determination is made at night and performing data association of the camera data with the radar data, Sensor fusion system. The method according to claim 1,
The day /
And a logic circuit for determining that the camera is in the nighttime when both the day / night determining information and the light brightness information in the camera are at night. The method according to claim 1,
Wherein the data correlator comprises:
A vehicle data fusion module for performing the data association by dividing the daytime gate into a daytime gate and the nighttime gate that is increased compared to the daytime gate;
A lane data fusion module that fuses the estimated lane information of the radar unit and the lane information of the camera sensor unit to recognize a front lane of each lane; And
And a control module that measures the preceding vehicle information by giving a correlation between the laser recognition result and the camera recognition result using the nighttime gate whose size is increased in the vehicle data fusion module at night. The method of claim 3,
The radar unit includes:
If the measured object has an acceleration, it is determined that the preceding vehicle is a preceding vehicle, and the preceding vehicle is determined to be at least one vehicle type of a passenger car, a truck, and a bus. 5. The method of claim 4,
The vehicle data fusion module comprising:
Wherein the nighttime gate is divided into the nighttime gates according to the nighttime camera recognition characteristics, and gate re-sizing is performed. 5. The method of claim 4,
The vehicle data fusion module comprising:
Wherein the nighttime gate is further classified by a predetermined unit distance according to camera recognition characteristics at night to perform gate re-sizing. A sensor fusion method in consideration of day / night camera recognition characteristics of a sensor fusion system of a vehicle,
a) synchronizing a radar measurement result and a camera measurement result along a time axis;
b) correcting the daytime gate to a nighttime gate that has been increased in the longitudinal direction by using the recognition characteristics of the camera in the nighttime according to the result of the day / night judgment;
c) measuring a preceding vehicle information by performing a data association which gives a correlation between a laser recognition result and a camera recognition result using a night gateway; And
and d) tracking the position and movement information of the preceding vehicle by reflecting lane information on the preceding vehicle information. 8. The method of claim 7,
The step b)
And determining whether the day / night determination information in the camera and the brightness of light received from the outside are collected to be a daytime or a nighttime. 8. The method of claim 7,
Between the step c) and the step d)
The estimated lane information and the lane information obtained from the radar unit and the camera sensor unit are fused to calculate the width of the front lane side lane, the lateral position of the vehicle in the lane, the distance from the lane to both lanes, the shape of the lane, And recognizing the lane information including the one lane information. 8. The method of claim 7,
The step b)
And further correcting the nighttime gate by further dividing the nighttime gate by the type of vehicle or the distance from the preceding vehicle according to the nighttime camera recognition characteristic.

Images