KR102188164B1 - Method of Road Recognition using 3D Data - Google Patents

Abstract

The present invention relates to a road recognition method, comprising: generating a 3D disparity map for an image acquired through a stereo camera, generating a U disparity map from the 3D disparity map, the U disparity map Simplifying by processing, detecting a boundary candidate based on the 3D disparity map from the U disparity map, and determining a boundary of a road by filtering the detected boundary candidate.

Description

Road Recognition Using 3D Data {Method of Road Recognition using 3D Data}

The present invention relates to a road recognition method using 3D data, and more particularly, to a method of detecting a boundary of a road using a 3D disparity map.

Transportation means such as a vehicle may photograph the surroundings using a three-dimensional camera, and recognize a boundary surface of a road and an obstacle around the road based on the captured image.

In the conventional road recognition technology, the ground area is determined by generating u displacement and v displacement, which are accumulated by dividing the disparity map of a stereo camera in the vertical and horizontal directions. In such a conventional technique, complex methods such as clustering and regression analysis are used to generate u displacement and v displacement respectively, and it is difficult to provide real-time information by using a method that takes a long computation time such as RANSAC or Hough transform for modeling. .

An object of the present invention for solving these conventional problems is to provide accurate road information to a user by generating and processing a U disparity map from 3D image data without generating a separate V displacement.

In order to achieve these objects, the road recognition method according to the present invention includes the steps of generating a 3D disparity map for an image acquired through a stereo camera, generating a U disparity map from the 3D disparity map, the Simplifying by processing a U disparity map, detecting a boundary candidate from the U disparity map based on the 3D disparity map, and determining a boundary of a road by filtering the detected boundary candidate can do.

According to various embodiments, the processing and simplification of the U disparity map may include assigning a distance weight to the U disparity map and performing a binarization process based on a threshold value on the U disparity map. . The distance weight may increase as the distance to the object increases. The threshold value may be determined based on information about a standard deviation of a road area set in advance. The step of performing the binarization process based on the threshold value may include setting a value of a point having a U disparity value lower than the threshold value to 0.

According to various embodiments, the detecting of the boundary candidate includes finding a disparity value of a pixel corresponding to a boundary in a designated column and determining a position in the 3D disparity map having the disparity value. Can include.

According to various embodiments, the determining of the boundary of the road includes detecting a peak by connecting points detected as the boundary candidates, and determining a boundary of the road based on a width of the detected peak. can do. The detecting of the peak includes detecting a peak by connecting points detected as the boundary candidates in a first direction, and connecting points detected as the boundary candidates in a second direction opposite to the first direction to generate a peak. It may include the step of detecting. Determining the boundary of the road based on the width of the detected peak may include removing the detected peak when the width of the detected peak is less than a specified range.

The vehicle according to the present invention includes a stereo camera, a 3D map generator that generates a 3D disparity map for an image acquired through the stereo camera, and a U map that generates a U disparity map from the 3D disparity map. A generation unit, a U map processing unit that processes and simplifies the U disparity map, a boundary detection unit that detects a boundary candidate from the U disparity map based on the 3D disparity map, and filters the detected boundary candidate It may include a filtering unit that determines the boundary of.

As described above, since the road recognition method according to the present invention does not perform a separate process for generating V displacement, accurate road recognition information can be provided to the user in real time.

In addition, the road recognition method according to the present invention can determine an accurate road boundary by detecting unnecessary peaks through bidirectional filtering.

1 is a block diagram of a road recognition system according to the present invention.
2 is a block diagram illustrating a configuration of a U map processing unit according to various embodiments of the present disclosure.
3 is a flowchart illustrating a road recognition process according to various embodiments of the present disclosure.
4 is an exemplary diagram for applying a learning-based threshold according to various embodiments of the present disclosure.
5 is an exemplary diagram illustrating detection of a road boundary candidate according to various embodiments of the present disclosure.
6 is an exemplary diagram illustrating error removal filtering performed by a filtering unit according to various embodiments of the present disclosure.
7 is an exemplary diagram in which a filtering process is applied to an actual image according to various embodiments of the present disclosure.
8 is an exemplary diagram illustrating edge detection according to various embodiments of the present disclosure.

Hereinafter, exemplary embodiments of the present invention will be described in more detail with reference to the accompanying drawings. However, in describing the embodiments, descriptions of technical contents that are well known in the technical field to which the present invention pertains and are not directly related to the present invention will be omitted as much as possible. This is to more clearly convey the core of the present invention by omitting unnecessary description.

1 is a block diagram of a road recognition system according to the present invention. The road recognition system 100 may be installed in a vehicle such as a vehicle. The information processed through the road recognition system 100 may be delivered to a driver, may be used for a user driving assistance program, or may be used for an unmanned driving system.

Referring to FIG. 1, the road recognition system 100 may include a stereo camera 110, a 3D map generator 120, a U map generator 130, a U map processor 140, a boundary detector 150, a filtering unit 160, and an output unit 170. .

The stereo camera 110 may capture an image or video by arranging two photographing lenses at designated intervals. The stereo camera 110 can simultaneously photograph a subject through two photographing lenses and obtain two images at the same time.

The 3D map generator 120 may generate a 3D map by analyzing image information captured through the stereo camera 110. The 3D map generator 120 may analyze an object, a size of the object, a distance to the object, and the like through comparison of the image captured through each lens. For example, a stereo image captured through each lens included in the stereo camera 110 may be a two-dimensional image that can be represented by an x-axis and a y-axis. The 3D map generator 120 may compare images captured by each lens at the same time and generate a 3D image that can be expressed in the x-axis, y-axis, and z-axis by reflecting the distance between the lenses. .

According to various embodiments of the present disclosure, the 3D map generator 120 may generate a 3D disparity map according to a difference caused by positions of photographing lenses, a distance between lenses, and an installation angle.

The U map generator 130 may generate a U disparity map from a 3D disparity map. The U disparity map may be obtained by stacking histograms of distance values located on the same horizontal axis using values of the x-axis and z-axis of the 3D image.

In the conventional road recognition technology, the ground area is determined by generating u displacement and v displacement, which are accumulated by dividing the disparity map of a stereo camera in the vertical and horizontal directions. In this prior art, complex methods such as clustering and regression analysis are used to generate u displacement and v displacement, respectively, and it is difficult to provide a real-time system by using a method that takes a long computation time such as RANSAC or Hough transform for modeling. .

On the other hand, the road recognition technology according to the present invention does not use a separate V disparity map and does not use a complex algorithm such as RANSAC or Hough transform used for straight line detection, and thus can maintain a fast data processing speed. The processed data may be transmitted to the driving assistance system in real time, and road recognition information may be provided to the user.

The U map processor 140 may process the generated U disparity map to apply weights and perform binarization processing. The U map processor 140 may increase the reliability of data by multiplying the weight according to the U disparity value by using not only the number of simple pixels but also actual distance information. Also, the U map processor 140 may perform a binarization operation using a learning-based threshold. Information on the operation of the U-map processor 140 may be provided through FIGS. 2 and 4.

The boundary detection unit 150 may connect points that become boundary in the processed U disparity map. The boundary detection unit 150 may refer to a 3D dispatch map to indicate the positions of the boundary points in the image. Detailed information on the operation of the boundary detection unit 150 may be provided through FIG. 5.

The filtering unit 160 may remove some of the peaks generated by points detected as boundary candidates through filtering. The filtering unit 160 may detect a boundary of a road that may substantially give meaning to a user by removing peaks less than a specified width in a process of connecting points detected as boundary candidates. Detailed information on the operation of the filtering unit 160 may be provided through FIGS. 6 and 7.

The output unit 170 may output an image or a range value reflecting the filtered boundary value. The output unit 170 may display an image reflecting the filtered boundary value on a screen or provide it to a driving assistance system to be used (negative path recognition, forward vehicle tracking, collision avoidance, etc.).

2 is a block diagram illustrating a configuration of a U map processing unit according to various embodiments of the present disclosure.

Referring to FIG. 2, the U map processing unit 140 may include a weight calculation unit 210 and a threshold value application unit 220.

The weight calculator 210 may reflect the distance weight to the U disparity value provided from the U map generator 130. The weight calculator 210 may multiply the U disparity value by a weight according to a distance (a higher weight as the distance increases) to utilize not only the number of simple pixels but also actual distance information. The distance weight may be reflected by Equation 1 below.

[Equation 1]

: U 디스패리티 값

: U disparity value

: 거리 가중치

: Distance weight

The threshold value application unit 220 may simplify a part of the area corresponding to the road by binarizing it. The threshold value application unit 220 may store standard deviation information on a disparity distribution of an area corresponding to a road. The standard deviation information may be set based on height information for general road topography.

The threshold value application unit 220 may determine a threshold value for removing a U disparity value less than a specified range according to the standard deviation information. For example, the threshold value application unit 220 may determine a threshold value for removing points within a range of 60% or less according to the standard deviation information among all U disparity points. The threshold value applying unit 220 may make values less than a certain height to 0 and output only values greater than the height. Through FIG. 5, information on the operation of the threshold value application unit 220 may be provided.

3 is a flowchart illustrating a road recognition process according to various embodiments of the present disclosure.

Referring to FIG. 3, in operation 310, the 3D map generator 120 may generate a 3D disparity map according to a difference caused by a location of photographing lenses, a distance between lenses, and an installation angle.

In operation 320, the U map generator 130 may generate a U disparity map from a 3D disparity map. The U map generator 130 may be obtained by stacking histograms of distance values located on the same horizontal axis using values of the x-axis and the z-axis of the 3D image.

In operation 330, the U map processor 140 may process the generated U disparity map to apply weights and perform binarization processing. The U map processor 140 may reflect the distance weight to the U disparity value provided from the U map generator 130. In addition, the U-map processing unit 140 may simplify a part of the area corresponding to the road by binarizing it.

In operation 340, the boundary detection unit 150 may connect boundary points in the processed U disparity map. The boundary detection unit 150 may refer to a 3D dispatch map to indicate the positions of the boundary points in the image.

In operation 350, the filtering unit 160 may remove some of the points detected as boundary candidates through filtering. The filtering unit 160 may detect a boundary of a road that may substantially give meaning to a user by removing peaks less than a specified width in a process of connecting points detected as boundary candidates.

In operation 360, the output unit 170 may display an image reflecting the filtered boundary value on a screen or provide it to a driving support system to be used (negotiated path recognition, forward vehicle tracking, collision avoidance, etc.).

4 is an exemplary diagram for applying a learning-based threshold according to various embodiments of the present disclosure.

Referring to FIG. 4, the threshold value application unit 220 may simplify a part of an area corresponding to a road by binarizing it.

The first image 410 shows the U disparity distribution before applying the threshold. The first image 410 simply reflects the normal height difference of the road in the middle/bottom of the image in addition to the part that becomes the boundary of the actual road.

The threshold value application unit 220 may store standard deviation information on a disparity distribution of an area corresponding to a road. The standard deviation information may be set based on height information for general road topography.

The threshold value application unit 220 may determine a threshold value for removing a U disparity value less than a specified range according to the standard deviation information. For example, the threshold value application unit 220 may obtain the second image 420 by removing points within a range of 60% or less according to the standard deviation information among all U disparity points of the first image 410.

In the second image 420, U disparity distributions in the middle/bottom of the image are all processed as 0 according to the application of the threshold, so that the actual boundary of the road can be easily determined.

5 is an exemplary diagram illustrating detection of a road boundary candidate according to various embodiments of the present disclosure.

Referring to FIG. 5, the boundary detection unit 150 may connect boundary points in the binarized U disparity map 420. The boundary detector 150 may refer to a 3D disparity map 510 that is an input image to indicate the positions of the boundary points in the image.

The boundary detector 150 may find a disparity value (row number) of a pixel corresponding to a boundary in a specific column of the binarized U disparity map 420. The boundary detector 150 may search for a location having the disparity value in the 3D disparity map 510 by scanning from bottom to top. The boundary detector 150 may store location information of a matched point.

For example, the boundary detection unit 150 may find the location of the boundary point 411 in the binarized U disparity map 420 with reference to the 3D disparity map 520. The boundary detector 150 may check the disparity value (row number) of the boundary point 411 and scan the 3D disparity map 510 from the bottom to the top to find the matching point 511. The boundary detector 150 may match the position of the point 511 in an actual image to determine the point 521 and store the result.

The road recognition technology according to the present invention does not use a separate V disparity map, and does not use a complex algorithm such as RANSAC or Hough transform used for straight line detection, thereby maintaining a fast data processing speed. The processed data can be delivered to the driving assistance system in real time and can be provided to the user.

6 is an exemplary diagram illustrating error removal filtering performed by a filtering unit according to various embodiments of the present disclosure.

Referring to FIG. 6, the filtering unit 160 may remove a portion where an error occurs when points detected as boundary candidates are connected through filtering. The filtering unit 160 may detect peaks in both directions and calculate a width through Equation 2 below to remove peaks less than a predetermined width.

[Equation 2]

In FIG. 6, the filtering unit 160 may connect points detected in a first direction (eg, from left to right) at a boundary 610 before filtering. Through this, the filtering unit 160 may detect the first peak 601 and the third peak 603. The filtering unit 160 may detect the second peak 602 by connecting points detected in the second direction (eg, from right to left).

The filtering unit 160 may determine a width of each peak at the boundary 620 in which the peak is detected. The filtering unit 160 may remove a peak whose width of the detected peak is smaller than the width determined by the above equation.

For example, when the first peak 601, the second peak 602, and the third peak 603 are detected, the filtering unit 160 maintains the first peak 601 having a width greater than the specified width, and the second peak 601 is less than the specified width. Boundary 630 can be obtained by removing peak 602 and third peak 603. In various embodiments, the filtering unit 160 may additionally perform a box filtering operation so that the result of the boundary 630 can be smoothed and recognized as a continuous boundary.

7 is an exemplary diagram in which a filtering process is applied to an actual image according to various embodiments of the present disclosure.

Referring to FIG. 7, the filtering unit 160 may detect a peak through first derivative. The filtering unit 160 may detect a peak by connecting points detected in a first direction (eg, from left to right), and detect a peak by connecting points detected in a second direction (eg, from right to left). can do. Through this, the boundary 710 before filtering may be determined.

The filtering unit 160 may determine the width of each of the detected peaks. The filtering unit 160 may remove a peak 721 less than a designated width among the detected peaks. Peak 722 that is larger than the specified width is not removed and can be detected as a road boundary. The filtering unit 160 may output the boundary 730.

8 is an exemplary diagram illustrating edge detection according to various embodiments of the present disclosure.

Referring to FIG. 8, the road recognition system 100 may detect a boundary 801 between a road and surrounding objects from images acquired through a stereo camera 110.

The road recognition system 100 may generate and process a U disparity map, and may determine locations of boundary points by referring to the 3D disparity map.

The road recognition system 100 may detect only meaningful peaks through a filtering process and remove unnecessary peaks to determine a boundary 801 that can be substantially used by a user.

The road recognition system 100 does not use a separate V disparity map, and does not use complex algorithms such as RANSAC or Hough transform, which are used for straight line detection, so it can maintain fast data processing speed. The processed data can be delivered to the driving assistance system in real time and can be provided to the user.

In the present specification and drawings, a preferred embodiment of the present invention has been disclosed, and although specific terms are used, these are merely used in a general meaning to easily explain the technical content of the present invention and to aid understanding of the present invention. It is not intended to limit the scope of. In addition to the above-described embodiments, it is obvious to those of ordinary skill in the art that other modifications based on the technical idea of the present invention can be implemented.

100: road recognition system 110: stereo camera
120: 3D map generation unit 130: U map generation unit
140: U map processing unit 150: boundary detection unit
160: filtering unit 170: output unit

Claims (10)

Generating a 3D disparity map for an image acquired through a stereo camera; Generating a U disparity map from the 3D disparity map; Processing and simplifying the U disparity map; Detecting a boundary candidate from the U disparity map based on the 3D disparity map; And filtering the detected boundary candidate to determine a boundary of the road.
The processing and simplification of the U disparity map may include assigning a distance weight to the U disparity map; And performing a binarization process based on a threshold value on the U disparity map. delete The method of claim 1, wherein the distance weight is
Road recognition method, characterized in that the larger the distance to the object is increased. The method of claim 1, wherein the threshold is
A road recognition method, characterized in that it is determined based on standard deviation information of a preset road area. The method of claim 1, wherein the step of performing the binarization process based on the threshold value
And setting a value of a point having a U disparity value lower than the threshold value to 0. The method of claim 1, wherein detecting the boundary candidate
Finding a disparity value of a pixel corresponding to a boundary in a designated column; And
Determining a location in the 3D disparity map having the disparity value. The method of claim 1, wherein determining the boundary of the road
Detecting a peak by connecting points detected as the boundary candidates;
And determining a boundary of the road based on a width of the detected peak. The method of claim 7, wherein detecting the peak
Detecting a peak by connecting points detected as boundary candidates in a first direction; And
And detecting a peak by connecting points detected as the boundary candidates in a second direction opposite to the first direction. The method of claim 7, wherein determining the boundary of the road based on the width of the detected peak
If the width of the detected peak is less than the specified range, removing the detected peak; road recognition method comprising a. Stereo camera; A 3D map generator that generates a 3D disparity map for the image acquired through the stereo camera; A U map generator for generating a U disparity map from the 3D disparity map; A U map processing unit that processes and simplifies the U disparity map; A boundary detector for detecting a boundary candidate from the U disparity map based on the 3D disparity map; And a filtering unit configured to determine a boundary of a road by filtering the detected boundary candidates.
And the U map processing unit assigns a distance weight to the U disparity map and performs a binarization process based on a threshold value on the U disparity map.

Images