KR20220159575A - Vehicle AVM camera system for omnidirectional collision prevention and control method thereof - Google Patents

Abstract

The present invention relates to a vehicle AVM camera system and a control method thereof. The AVM camera system of the present invention comprises: a lidar data processing unit; a candidate detecting unit; and a first filtering unit.

Description

Vehicle AVM camera system for omnidirectional collision prevention and control method thereof {Vehicle AVM camera system for omnidirectional collision prevention and control method thereof}

This embodiment relates to a vehicle AVM camera system for omnidirectional collision avoidance and a control method thereof.

The contents described in this section simply provide background information on the embodiments of the present invention and do not constitute prior art.

Recently, with the development of the automobile industry and the continuous increase in the requirements for safety functions, research on advanced driver assistance systems (ADAS) is being actively conducted.

One of the most popular research areas in ADAS is LKAS (Lane Keeping Assistance system), which detects the currently driving lane, checks departure information, and controls it so that it does not deviate. LKAS is one of the essential ADAS functions because it is often necessary to release the steering wheel by operating the air conditioning button in the vehicle or by other external factors while driving, which can lead to a serious accident.

In order to guarantee high performance in LKAS, technology for accurately detecting lanes is essential. In general, when detecting a lane, a relatively inexpensive camera sensor is used, but the camera sensor is less reliable than the LIDAR sensor in adverse weather conditions. Although the price of lidar sensors is currently expensive, the development of lane recognition algorithms using lidar sensors is essential because the price of sensors is falling and their performance is improving as technology development and demand increase.

This embodiment relates to a lane recognition method using a lidar sensor, which detects a lane candidate group based on brightness information in lidar data, and from the detected lane candidate group, vehicle steering wheel information, density information, reliability information, and moving information. The purpose is to improve the accuracy of the lane-based LKAS using lidar by selecting the final lane candidate group by performing filtering based on average information.

In this embodiment, the lidar data processing unit for acquiring lidar data from the lidar sensor and detecting ground data in the lidar data; a candidate detection unit that detects a lane candidate group based on brightness information among the ground data; a first filtering unit to remove an erroneous candidate group from the lane candidate group based on at least one of characteristics of the lane candidate group and vehicle state information; a second filtering unit for measuring reliability of the next best candidate group filtered through the first filtering unit and removing unreliable candidate groups from the next best candidate group; and a selection unit for selecting a final lane candidate group by calculating a moving average for the lane candidate group filtered through the second filtering unit.

In addition, according to another aspect of the present embodiment, a ground data detection process of acquiring lidar data from a lidar sensor and detecting ground data in the lidar data; a lane candidate group detection process of detecting a lane candidate group based on brightness information among the ground data; a first filtering process of removing an erroneous candidate group from the lane candidate group based on at least one of characteristics of the lane candidate group and vehicle state information; a second filtering process of measuring reliability of the next best candidate group filtered through the first filtering process and removing unreliable candidate groups from the next best candidate group; and a selection process of selecting a final lane candidate group by calculating a moving average for the lane candidate group filtered through the second filtering process.

As described above, according to the present embodiment, a lane candidate group is detected based on brightness information in lidar data, and based on steering wheel information, density information, reliability information, and moving average information of the vehicle from the detected lane candidate group, By performing filtering to select the final lane candidate group, there is an effect of improving the accuracy of the lane-based LKAS using lidar.

1 is a block diagram schematically illustrating an AVM camera system according to an exemplary embodiment.
2 is a flowchart for explaining a lane recognition method according to the present embodiment.
3 is a conceptual diagram of omnidirectional collision avoidance based on a camera sensor according to the present embodiment.
4 is a camera sensor-based omnidirectional collision avoidance diagram using the target technology according to the present embodiment.
5 is a diagram for explaining a method of removing an unreliable candidate group according to the present embodiment.
6 is a diagram for explaining a final lane candidate group according to the present embodiment.

Hereinafter, this embodiment will be described in detail with reference to the accompanying drawings. In describing the present invention, '... Terms such as 'unit' and 'module' refer to a unit that processes at least one function or operation, and may be implemented as hardware, software, or a combination of hardware and software.

This embodiment describes a lane recognition method using at least one LiDAR (Light Detection and Ranging) sensor disposed in an autonomous vehicle, but this is according to an embodiment, and detects an object using lidar data. It can be applied to various fields of detection.

1 is a block diagram schematically illustrating an AVM camera system according to an exemplary embodiment. Hereinafter, the AVM camera system 100 according to the present embodiment and related components will be described with reference to FIG. 1 .

The AVM camera system 100 is a device that detects a lane in conjunction with a lidar sensor (not shown), and detects a lane ahead at a predetermined distance based on lidar data collected from the lidar sensor. Specifically, the AVM camera system 100 removes height information from the point cloud data obtained from the lidar sensor, and detects an area in which a lane is estimated to be located based on brightness information as a lane candidate group. Thereafter, the AVM camera system 100 selects a final lane candidate group by performing filtering based on vehicle steering wheel information, density information, reliability information, and moving average information from the detected lane candidate group.

As shown in FIG. 1, the AVM camera system 100 according to this embodiment includes a lidar data processing unit 110, a candidate detection unit 120, a first filtering unit 130, a second filtering unit 140, a setting It includes a part 150 and a guide part 160. Here, components included in the AVM camera system 100 are not necessarily limited thereto. For example, the AVM camera system 100 may be implemented in a form including a LiDAR sensor as a component.

The lidar data processing unit 110 processes the lidar data obtained from the lidar sensor to detect lidar data or ground data.

On the other hand, the lidar sensor is mounted on one side of the vehicle, and emits a laser toward the periphery (front) of the vehicle. The laser emitted by the lidar sensor may be scattered or reflected back to the vehicle.

The lidar sensor represents distance information measured using a laser in the form of a point cloud in 3D space, and transfers lidar data including this distance information to the AVM camera system 100. . For example, each point in the point cloud may include distance information of how many meters away from the target and the robot coordinate system in the X, Y, and Z directions from the position of the sensor, and the reflection coefficient value of the target.

The form of such LiDAR data can be confirmed through FIG. 3 . On the other hand, Fig. 3 (a) and (b) shows a comparison between the image obtained from the camera and corresponding lidar data obtained from the lidar sensor for each tunnel section.

The lidar data processing unit 110 according to the present embodiment may detect ground data by removing height information from lidar data, which is 3D stereoscopic data. That is, the lidar data processing unit 110 may detect ground data as shown in FIG. 4 by extracting only the floor surface by removing unnecessary Z-axis (height information perpendicular to the ground) in order to easily extract lanes.

The candidate detector 120 performs a function of detecting a next best candidate group based on brightness information among ground data.

The currently developed next-best algorithm using the lidar sensor mainly uses the Otsu algorithm when generating a next-best candidate group. The Otsu algorithm is a binarization method that finds T that minimizes the intra-class variance between two classes or maximizes the inter-class variance between two classes when data is classified into two classes based on a threshold value T. way.

On the other hand, when only the bottom surface is extracted based on 64-channel lidar, the total number of data is about 13,000, whereas the number of data corresponding to the next best is less than about 250, which is less than 2% numerically. That is, there is a limit to finding a threshold value T that divides the two classes with this data.

A characteristic of the lane is that the brightness value is larger than that of the surrounding data. Due to this fact, the candidate detector 120 detects a lane candidate group based on brightness information among ground data.

The candidate detector 120 according to the present embodiment calculates an x-axis gradient change waveform for the brightness change of the ground data, and detects a lane candidate group within the ground data based on a differential waveform obtained by differentiating the calculated gradient change waveform.

More specifically, the candidate detector 120 detects, as a next best candidate group, a region in which the amount of change in brightness increases by more than a preset threshold based on the differential waveform and then decreases by more than a preset threshold within a certain range.

For example, referring to FIG. 5 , looking at the primary derivative waveform of the brightness from the left to the right of the reference, it can be seen that the amount of change in brightness greatly increases and then greatly decreases. Due to this, the candidate detection unit 120 according to the present embodiment sets the next best candidate group when the brightness change amount greatly increases and then the brightness change amount greatly decreases in the N-th data. Here, N may be determined according to a width length between lanes in a predefined road.

Referring to FIG. 6 , a next best candidate group detected through the candidate detection unit 120 according to the present embodiment may be identified. Meanwhile, in FIG. 6 , the next best candidate group data obtained by the candidate detection unit 120 as the brightness differential value is displayed on the point cloud map.

In this embodiment, the candidate detection unit 120 has an effect of removing a stop line that may be misrecognized as a lane by detecting a lane candidate group based on brightness information among ground data.

Meanwhile, when a lane candidate group is detected through the candidate detector 120, many lanes are detected in the candidate group, but non-lane data (false positive) may sometimes be included. Therefore, in order to accurately detect lanes, it is necessary to remove erroneous data within a range that does not remove true lane data (true positive).

The first filtering unit 130 performs a function of removing an erroneous candidate group from the lane candidate group based on at least one of the characteristics of the lane candidate group detected by the candidate detection unit 120 and the state information of the vehicle.

In this embodiment, the first filtering unit 130 may collect steering wheel information as vehicle state information, and remove error candidate groups from the lane candidate group based on the collected steering wheel information. In this case, vehicle steering wheel information may be acquired through CAN communication with the vehicle.

For example, referring to FIG. 7 , the first filtering unit 130 may remove the error candidate group by calculating Euclidean distance between the vehicle and the lane candidate group based on steering wheel information of the vehicle. In more detail, the first filtering unit 130 may remove, as an error candidate group, the Euclidean distance information between the linear equation of the vehicle's heading and the lane candidate group based on the steering wheel information is less than a preset threshold.

In this case, the candidate detection unit 120 has an effect of excluding an error candidate group corresponding to a mark drawn on the road floor, a water puddle, or an uneven floor surface from the lane candidate group.

On the other hand, lanes are mainly composed of solid lines or dotted lines, and when lane candidate groups are generated, they exist densely on the actual lanes.

Due to this fact, the first filtering unit 130 determines whether other data exists (ex: density information) within a preset contiguous range based on the next best candidate group as a characteristic of the next best candidate group, and uses this to determine the error Candidates can be excluded from the next best candidate group.

For example, the first filtering unit 130 examines whether other data exists within a preset threshold range, based on the detected lane candidate group. Thereafter, the first filtering unit 130 removes the second best candidate group because it is very likely to be an erroneous candidate group if another next best candidate group does not exist within the preset threshold range.

The second filtering unit 140 measures reliability of the next best candidate group filtered through the first filtering unit 130 and removes unreliable candidate groups from the next best candidate group.

The second filtering unit 140 represents a lane of the N-order equation corresponding to the lane candidate group filtered through the first filtering unit 130, and calculates a normal distance between the lane candidate group and the lane candidate group based on the lane of the N-order equation. Thus, the above reliability can be measured.

For example, the second filtering unit 140 determines how reliable the lane candidate group is and whether it can be used for actual LKAS driving based on the calculated normal distance with the lane candidate group, and excludes the lane candidate group with low reliability. can do.

Referring to (a) and (b) of FIG. 8 as an example, a circle represents a next-best candidate group, and a line is an example of drawing an N-order equation made of this candidate group.

8(a) is an example of drawing lanes with high reliability, and it can be confirmed that lane candidate groups are densely present based on an N-order equation.

FIG. 8(b) is an example of drawing a lane with low reliability, and it can be confirmed that the density of lane candidate groups based on the N-order equation is lower than that of FIG. 8(a).

Meanwhile, in FIG. 1, it is illustrated that the first filtering unit 130 and the second filtering unit 140 in the AVM camera system 100 are implemented as separate devices, but it is not necessarily limited thereto, and the first filtering unit ( 130) and the second filtering unit 140 may be implemented as one filtering device.

The selection unit 150 selects a final lane candidate group by calculating a moving average for the next best candidate group filtered through the second filtering unit 140 . This operation of the selector 150 flattens the noise of the data as a kind of tracking process.

In this embodiment, the selector 150 calculates a moving average representing an average value of slopes of lane equations corresponding to a lane candidate group based on accumulated data collected during a certain point in time. Here, the moving average is a reactive index showing an average value for a predetermined period from the past to the present, and it is preferable that the selection unit 150 calculates an average value for calculating the moving average based on 10 to 30 accumulated data, but this It is not limited.

On the other hand, for a long period of time, the next-best data slowly changes. Accordingly, when the value is rapidly different at any one point in time, it cannot physically exist. Due to this point, the selection unit 150 selects a final lane candidate group by filtering lane candidate groups having a change value that exceeds a preset threshold based on the calculated moving average. That is, the selector 150 removes lane data when the value is rapidly different by using the moving average.

The guide unit 160 performs a function of generating and providing way point information for guiding the vehicle to drive without departing from the lane based on the final lane candidate group. At this time, the parameter was adjusted so that the waypoint information was located at the center of the detected lane considering that the lane width in Korea is about 3m to 3.5m. If this data is passed on to the control device, LKAS is possible, and information of the X and Y coordinates to be moved by the vehicle is stored in the way point information.

Meanwhile, FIG. 9(a) shows a lane equation from which the final lane is recognized, and FIG. 9(b) shows waypoints generated based on the lane equation. Based on this, when performing LKAS, the vehicle moves along the corresponding waypoint.

2 is a flowchart for explaining a lane recognition method according to the present embodiment.

The AVM camera system 100 acquires lidar data from lidar sensors and detects ground data within lidar data (S202). In step S202, the AVM camera system 100 may detect ground data by removing height information from lidar data, which is 3D stereoscopic data.

The AVM camera system 100 detects a lane candidate group based on the brightness information among the ground data detected in step S202 (S204). In step S204, the AVM camera system 100 calculates a gradient change waveform of the x-axis for the brightness change of the ground data, and based on the differential waveform obtained by differentiating the calculated tilt change waveform, the brightness change amount increases by more than a preset threshold value and then becomes constant. An area that decreases by more than a predetermined threshold value within the range is detected as a next best candidate group.

At this time, the AVM camera system 100 may set the detected lane candidate group as a region of interest.

The AVM camera system 100 removes the error candidate group based on at least one of the characteristics of the lane candidate group and vehicle state information (S206). In step S206, the AVM camera system 100 may remove, as the error candidate group, the Euclidean distance information between the linear equation of the vehicle's heading and the lane candidate group based on the steering wheel information, which is less than a preset threshold.

In addition, the AVM camera system 100 examines whether other data exists within a preset threshold range based on the detected lane candidate group, and if another lane candidate group does not exist within the preset threshold range according to the inspection result, Since it is very likely to be an error candidate, it can be eliminated.

The AVM camera system 100 measures the reliability of the next best candidate group and removes the unreliable candidate group (S208). In step S208, the AVM camera system 100 may indicate a lane of the N-order equation corresponding to the lane candidate group, and calculate the normal distance between the lane candidate group and the N-order equation based on the lane of the N-order equation to measure reliability.

The AVM camera system 100 selects a final lane candidate group by calculating a moving average for the lane candidate group (S210). In step S210, the AVM camera system 100 calculates a moving average representing the average value of the inclination of the lane equation corresponding to the lane candidate group based on the accumulated data collected during a certain point in time, and calculates a predetermined threshold based on the calculated moving average A final lane candidate group is selected by filtering a lane candidate group having a deviation change value.

The AVM camera system 100 generates and provides waypoint information for guiding the vehicle to drive without departing from the lane based on the final lane candidate group selected in step S210 (S212).

Here, since steps S202 to S212 correspond to the operation of each component of the AVM camera system 100 described above, further detailed descriptions are omitted.

In FIG. 2, it is described that each process is sequentially executed, but is not necessarily limited thereto. In other words, since it will be applicable by changing and executing the process described in FIG. 2 or executing one or more processes in parallel, FIG. 2 is not limited to a time-series sequence.

As described above, the lane recognition method described in FIG. 2 is implemented as a program and can be read using computer software (CD-ROM, RAM, ROM, memory card, hard disk, magneto-optical disk, storage device, etc.) can be recorded in

The above description is merely an example of the technical idea of the present embodiment, and various modifications and variations can be made to those skilled in the art without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of the present embodiment, but to explain, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of this embodiment should be construed according to the claims below, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of rights of this embodiment.

100: AVM camera system 110: lidar data processing unit
120: candidate detection unit 130: first filtering unit
140: second filtering unit 150: selection unit
160: guide unit

Claims (8)

LiDAR data processing unit for detecting ground data within lidar data;
a candidate detection unit that detects a lane candidate group based on brightness information among the ground data; and
A first filtering unit to remove an error candidate group from the lane candidate group based on at least one of characteristics of the lane candidate group and vehicle state information;
The first filtering unit,
Steering wheel information of the vehicle is collected, Euclidean distance information between a straight line equation of the heading of the vehicle calculated based on the steering wheel information and the lane candidate group is calculated, and the Euclidean distance The AVM camera system characterized in that the next best candidate group whose information is less than a predetermined threshold value is removed as the error candidate group. According to claim 1,
The lidar data processing unit,
AVM camera system, characterized in that for detecting the ground data from which height information perpendicular to the ground has been removed by removing Z-axis information from the lidar data. According to claim 1,
The candidate detection unit,
An AVM camera system, characterized in that for calculating an x-axis gradient change waveform for the brightness change of the ground data, and detecting the lane candidate group in the ground data based on a differential waveform obtained by differentiating the calculated gradient change waveform. According to claim 3,
The candidate detection unit,
The AVM camera system according to claim 1 , wherein an area in which the amount of change in brightness increases by more than a predetermined threshold based on the differential waveform and then decreases by more than a predetermined threshold within a predetermined range is detected as the lane candidate group. According to claim 1,
The first filtering unit,
The AVM camera system, characterized in that for removing the error candidate group by utilizing density information indicating whether other data exists within a predetermined adjacent range based on the lane candidate group. According to claim 1,
The selection department,
The moving average representing the average value of the slope of the lane equation corresponding to the lane candidate group is calculated based on the cumulative data collected during a certain point in time, and the lane candidate group having a change value beyond a preset threshold based on the moving average is calculated The AVM camera system, characterized in that for selecting the final lane candidate group by filtering. According to claim 1,
The AVM camera system further comprises a guide unit generating way point information for guiding the vehicle to drive without departing from the lane based on the final lane candidate group. A ground data detection process of obtaining lidar data from a lidar sensor and detecting ground data in the lidar data;
a lane candidate group detection process of detecting a lane candidate group based on brightness information among the ground data;
a first filtering process of removing an erroneous candidate group from the lane candidate group based on at least one of characteristics of the lane candidate group and vehicle state information;
a second filtering process of measuring reliability of the next best candidate group filtered through the first filtering process and removing unreliable candidate groups from the next best candidate group; and
A selection process of selecting a final lane candidate group by calculating a moving average for the next lane candidate group filtered through the second filtering process;
The first filtering process,
Steering wheel information of the vehicle is collected, Euclidean distance information between a straight line equation of the heading of the vehicle calculated based on the steering wheel information and the lane candidate group is calculated, and the Euclidean distance Eliminate a next best candidate group whose information is less than a preset threshold as the error candidate group;
The second filtering process,
Indicates lanes of first to N (N = natural number) degree equations corresponding to the lane candidate group filtered through the first filtering process, and between the lane candidate groups based on the lanes of the first to Nth order equations A lane recognition method characterized in that the reliability is measured by calculating a normal distance.

Images