RU2802991C1 - Method for detecting extended linear objects in an image - Google Patents

Abstract

FIELD: image processing.

SUBSTANCE: invention relates to methods for selecting extended linear objects in an image and can be widely used in detecting road marking lines to determine its position on a given vector road map by an unmanned vehicle. A method for detecting extended linear objects in an image, which consists in the fact that during the movement of a vehicle (V) from video cameras, images of the roadway are obtained, individual linear objects of road markings are detected on the image using a fast windowed Hough transform. Image filtering for the development of linear objects of road markings is performed by applying a morphological filter "closing" to the image with a radius of the order of the characteristic width of the road marking line. The detected segments are grouped into sets taking into account the constraint on the angle of the break, a group of broken line segments is approximated, the resulting road marking lines in pixel coordinates are recalculated into metric coordinates on the roadway plane relative to the vehicle.

EFFECT: accuracy of recognition of extended linear objects of complex shape in the image.

1 cl, 2 dwg

Description

The invention relates to the field of image processing, namely to methods for isolating extended linear objects in an image and can be widely used in detecting road marking lines for an unmanned vehicle to determine its position on a given vector road map, as well as in constructing a road map from satellite images of the Earth , searching for dislocations in crystals on individual X-ray topo-tomography projections.

The following methods for detecting extended linear objects in an image are known from the technical field.

The method for detecting a road lane, known from patent RU 2746631, includes obtaining an image of a section of a road lane and processing it. The resulting image is simultaneously processed using a convolutional neural network and using a road marking detection algorithm, after which masks are generated in parallel for two intermediate images obtained as a result of processing, a weighted addition of the masks is performed pixel by pixel and conversion to a top view, forming the final mask, then performed transforming the pixel coordinates of the final mask from the image coordinate system into the vehicle coordinate system, after which each pixel of the final mask is assigned a weight coefficient, which is determined by the brightness of the corresponding pixel, then the resulting pixels of the final mask are approximated into a polynomial that defines the line of the middle of the road lane, and the deviation of the position of the vehicle is calculated means relative to the middle of the road lane.

To use neural networks to detect road markings, it is necessary to have a unique training sample of images, which requires a lot of time. In addition, when implementing this method, only the main traffic lane is detected, whereas in some traffic situations, for example, when turning right or left, it is necessary to have information about the number of lanes in the current direction and in which of them the vehicle is located.

There is a known method for extracting a road section from a road video image (CN107977608), which includes identifying all possible line segments in the image using the Hough transform; grouping detected line segments into different types; extracting the following normalized features of all line segments; selecting an optimal group of line segments in which one or more possible road edges exist; obtaining road edges from M image frames in a certain scene; calculating a histogram of the distribution of edges of the M image frames to obtain the main portion and the auxiliary portion of the road distribution; then extracting the histogram value as the first characteristic value of the two distribution sections; calculating the lengths of all lines in sections; using the longest line as the road boundary of a continuous image frame; and determination of the road surface area.

The description of this method does not disclose information about how, in the process of selecting a set of line segments, the coordinates of their beginning and end are determined.

There is a known method for detecting road lines (CN114037970), in which images of a chessboard are obtained from video cameras, an internal reference calibration of the camera is performed, and the internal reference matrix of the camera and the distortion coefficient are obtained; correct the resulting road image in real time, extract information about the edge features of the corrected image to obtain a binary image and determine the area of interest in accordance with the line of the roadway; transform the binary image to a virtual top view using virtual rotation, use morphological filtering to remove noise from the binary image and improve information about the lines of the roadway; extracting the pixel point coordinates of the left and right road lines using the sliding window method, and then applying the random sampling consistency method to fit the road lines; extracting road lines, back-projecting the road lines onto the corrected image using reverse perspective transformation for rendering.

The disadvantage of this method is that it is possible to detect only the main road lane, whereas when planning the movement of a vehicle, it is important to understand how many lanes there are on the road, for example, to make a left turn you need to change lanes to the far left.

The method disclosed in source CN114037970, in technical essence, is closest to the claimed invention and can act as a prototype.

The problem to be solved by the claimed invention is the creation of a method for detecting extended linear objects in an image in the form of broken lines with restrictions on the angle of break and the extent of the desired boundaries, providing the ability to accurately identify linear objects of various shapes - rectilinear, arc-shaped, circles, multiple parallel straight lines, a priori without specifying information about the expected location of the boundaries in the image and their number.

The technical result of the claimed invention is to increase the accuracy of recognition of extended linear objects of complex shape in the image.

The technical result is achieved by implementing a method for detecting extended linear objects in an image, in which, while a vehicle is moving, images of the roadway are obtained from video cameras, the resulting image is converted to a virtual top view using virtual rotation, and the image is filtered to reveal linear objects road markings using a morphological filter; individual linear road marking objects are detected in the image using the fast windowed Hough transform.

The peculiarity of the proposed method is that image filtering for the manifestation of linear road marking objects is performed by applying a “closure” morphological filter with a radius of the order of the characteristic width of the road marking line to the image, and applying a “opening” morphological filter with a radius greater than the characteristic width of the lines road markings, obtain an image of road markings by performing a pixel-by-pixel subtraction of images obtained using morphological filters “closure” and “opening”, reduce the channels of the road markings image by performing a pixel-by-pixel maximum operation between channels to obtain a single-channel image of road markings, for the purpose of detection of individual road marking lines, a windowed fast Hough transform is applied to the resulting single-channel image, the detected segments are grouped into sets taking into account the constraint on the bend angle, a group of segments of a polyline is approximated, the resulting road marking lines in pixel coordinates are converted into metric coordinates on the plane of the roadway relative to the vehicle using camera calibration parameters.

Figure 1 shows examples of images obtained by step-by-step implementation of the claimed method for detecting extended linear objects, where a is an image of road markings with vaguely defined boundaries as a result of applying the operation of pixel-by-pixel subtraction of images obtained using morphological filters “closure” and “opening”, b - image obtained as a result of applying the fast windowed Hough transform, c - image obtained as a result of grouping detected segments, d - image obtained as a result of approximating groups of segments with broken lines.

Figure 2 shows a diagram for determining the transition angle φ between the segments u , v , k .

A method for detecting extended linear objects, including capturing a video image and processing it by a computing unit, is carried out by installing at least one video camera in a vehicle connected to the specified computing unit equipped with a high-performance module, for example, an industrial computer of the Nuvo-5000 brand on based on amd64 architecture.

The claimed method for detecting extended linear objects, in the preferred embodiment, is implemented as follows (Fig. 1, a - d ).

The video camera is installed and fixed on the vehicle, a calibration procedure is performed, as a result of which internal parameters (focal length, pixel size, distortion coefficients) and external parameters (three-dimensional displacement and rotation of the video camera relative to the vehicle) are obtained, while dynamic calibration is supported, in which the external parameters (in particular, the three-dimensional rotation of the video camera relative to the vehicle) change over time and are estimated in some way known from the prior art (for example, based on inertial sensors, cameras, lidars, or any combination of these sensors).

While the vehicle is moving, images of the road surface are obtained from video cameras, the resulting image is converted to a virtual top view using virtual rotation, and the distortion of the RGB image is corrected.

In the control unit, the image is filtered to display linear objects of road markings using morphological filters “closure” and “opening”, which are determined through basic morphological filters.

Let the RGB image be defined as I, while - splitting image I into channels R, G and B, respectively.

Let us introduce the notation - image region size I , the upper left corner of which has coordinates (x, y).

Then, the basic morphological filters through which the “closure” and “opening” morphological filters are determined have the form:

- morphological filter “Building”

Let's denote - the result of applying the morphological filter “increasing” with an increasing region of size w by h.

Then for

- morphological filter “Narrowing”

Let's denote - the result of applying the morphological filter “narrowing” with a narrowing region of size w by h.

Then for

.

Then, the morphological filters “closure” and “opening” of the image are expressed through basic morphological filters as follows:

- morphological “closure” of the image;

- morphological “unlocking” of the image.

Then, an image of road markings is obtained by performing the operation of pixel-by-pixel subtraction of images obtained using the morphological filters “closure” and “opening” (Fig. 1, a ):

,

where r ₁ is the characteristic width of the road markings in pixels, r ₂ is a value greater than the characteristic width of the road markings in pixels.

Then, the reduction of the road marking image channels is carried out by performing a pixel-by-pixel maximum operation between the channels to obtain a single-channel road marking image.

Let - three-channel image. Let's denote - pixel values in the image . Let us denote the result of channel reduction by pixel-by-pixel maximum as . - single-channel image, such that .

In order to detect individual road marking lines, a windowed fast Hough transform is applied to the resulting single-channel image, in which the image is divided into square fragments with the same overlap area, a fast Hough transform (FHT) is applied to each selected image fragment, and segments in the image are searched.

Then, the resulting segments are grouped into unordered sets, taking into account the restriction of broken lines on the break angle; in each of the groups the order of traversal is determined, the group is approximated by a broken line and is checked for length.

At the stage of searching for segments in the image (Fig. 1, b ), the selected fragment of the original image will be considered the input image for the FPC. The result of the fast Hough transform is a single-channel image called the Hough image.

Each pixel of the resulting Hough image, parameterized as (s, t), contains the sum along the line of the input image defined by the intersection points with the opposite edges of the image (s,0) and (s+ t, n), where s is the integer shift of the pattern, and t – integer slope.

Thus, each row t of the resulting Hough image contains sums along all segments with a slope of t. To search for a contrast segment corresponding to a road marking line, it is necessary to select a line of the Hough image whose dispersion would be the greatest, that is, select the slope for which the dispersion of the sums of brightness along the segments is greatest, and at the same time the dispersion must exceed a certain specified threshold value. Then, the point from this line with the maximum integral brightness value will correspond to the most contrasting segment relative to the background in a given window.

At the stage of grouping segments (Fig. 1, c ), a pairwise search of segments is carried out, in which a pair of segments is considered to belong to the same boundary if the following conditions are met:

1. The distance between the segments is smaller ;

2. The angle between the segments lies in the range .

It should be noted that the distance between segments is defined as the distance between the nearest points on the segments, and the angle between the segments is defined as the angle between two rays emanating from the intersection point of the segments to their most distant ends. For non-intersecting segments, this angle is further determined by parallel translation of the segments until the two closest ends coincide at one vertex.

Thus, a set of segments are formed, found as a result of the fast windowed Hough transform, for which the above conditions 1 and 2 are met (in terms of the distance between the segments and the angle between them).

The grouping of segments, which are essentially the vertices of the graph G= (V, E), where V is the set of segments according to the results of the fast windowed Hough transform, is carried out using a well-known prior art algorithm for searching for connected components in the graph.

In the process of approximating groups of segments with broken lines (Fig. 1, d ), all boundaries in the image are divided into open rectilinear and curvilinear boundaries and circles. The latter is typical for images of road markings on a roundabout. The approximation of a group of segments is divided, respectively, into approximation by a single straight line segment (a broken line with one link) and approximation by a broken line with more than one link (circles are also approximated by a broken line, the beginning and end of which coincide). For each approximation result, the length of the polyline or segment is calculated as the sum of the distances between the vertices. If the calculated length exceeds the specified one, then the result is sent to the output of the algorithm. For the case of approximating a group of segments with a single straight line segment, a pair of the most distant points is sought among all ends of the segments in the group. This pair of points specifies the coordinates of the ends of the segment. Next, for all other ends of the segments in the group, their proximity to the formed segment is checked through the distance from a specific point to the constructed segment. If the proportion of close points is large, then the marking line is considered rectilinear, and the resulting segment is its approximation.

If it was not possible to approximate with a single straight segment, then a group of segments of a broken line is approximated.

Recall that the group of segments is also a connected component of the graph G. Let us denote its graph .

Let's call the transition angle between three vertices-segments And – the angle formed by rays coming out from the middle of one of the segments and passing through the middles of the other two (Fig. 2).

Perform a search in the connectivity component such a path that all transition angles between three consecutive vertices in the path satisfy the above condition 2 on the angle between the segments, and this path would also be of maximum length.

To do this, check that each trio of vertices , there is such a traversal of the triple along existing edges , that the transition angle when going around satisfies the above condition 2.

If such a bypass does not satisfy condition 2 and , then add an edge V , If and at the same time a new detour or satisfies the transition angle constraint, that is, condition 2.

To find the order of traversal of all segments, we will use Floyd's algorithm. To do this, we assign a weight to each edge ( u , v ) equal to the distance between the most distant ends of the segments u, v. The output of the algorithm will be a matrix of shortest paths D between all pairs of vertices and a matrix P storing information about the numbers of phases at which the shortest distance between pairs of vertices was obtained. Next, from all the shortest paths, the path whose value is maximum is selected. If the choice of the maximum path is not unambiguous (the maximum values are close within a given error), then the maximum path with the minimum number of traversed vertices is selected. Next, using information about the phases P, the traversal of the vertices-segments is restored.

Due to the peculiarities of calculations, the algorithm described above searches only for open straight and curved boundaries. The result of approximating a group of segments covering a circle will be a broken line corresponding to only half of the circle. This is due to the fact that a complete traversal of the circle will have a minimum value in the resulting matrix D (since the shortest distance between the beginning and the end will be zero), and a half traversal will have a maximum value. Therefore, an alternative path is searched from the ends of the selected path using the pre-calculated Floyd matrix. If such a path exists and it does not have common vertices with the previously found path (except for the beginning and end), then these paths are combined into one. Next, when the order of traversing the segments is found, a broken line is constructed, the vertices of which are the points of intersection or the nearest ends of segments adjacent in the path, if the segments do not intersect.

Based on the results of approximating groups of segments with broken lines, the resulting road marking lines in pixel coordinates at each moment in time are recalculated into metric two-dimensional coordinates (x, y) in the plane of the road surface relative to the vehicle using the calibration parameters of the video camera.

Claims (1)

A method for detecting extended linear objects in an image, in which, while a vehicle is moving, images of the road surface are obtained from video cameras, the resulting image is converted to a virtual top view, the image is filtered in the control unit to reveal linear objects of road markings, detection is performed on the image individual linear road marking objects using the fast windowed Hough transform, characterized in that the image filtering for the manifestation of linear road marking objects is performed by applying a morphological filter “closure” with a radius of the order of the characteristic width of the road marking line to the image; opening" with a radius greater than the characteristic width of the road marking lines, obtain a road marking image by performing the operation of pixel-by-pixel subtraction of images obtained using the morphological filters "closure" and "opening", reduce the channels of the road marking image by performing a pixel-by-pixel maximum operation between the channels to obtain single-channel image of road markings, in order to detect individual road marking lines, a windowed fast Hough transform is applied to the resulting single-channel image, the detected segments are grouped into sets taking into account the constraint on the break angle, the group of segments is approximated by a polyline, the resulting road marking lines in pixel coordinates are converted into metric coordinates on the plane of the roadway relative to the vehicle.