CN114723946B - Preferential direction deviation early warning system and method based on semantic segmentation - Google Patents
Preferential direction deviation early warning system and method based on semantic segmentation Download PDFInfo
- Publication number
- CN114723946B CN114723946B CN202210374860.6A CN202210374860A CN114723946B CN 114723946 B CN114723946 B CN 114723946B CN 202210374860 A CN202210374860 A CN 202210374860A CN 114723946 B CN114723946 B CN 114723946B
- Authority
- CN
- China
- Prior art keywords
- straight line
- road
- edge
- point
- representing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 230000011218 segmentation Effects 0.000 title claims abstract description 37
- 238000012545 processing Methods 0.000 claims abstract description 28
- 238000001514 detection method Methods 0.000 claims abstract description 13
- 230000008569 process Effects 0.000 claims abstract description 5
- 239000011159 matrix material Substances 0.000 claims description 83
- 230000006870 function Effects 0.000 claims description 29
- 238000001914 filtration Methods 0.000 claims description 12
- 238000003708 edge detection Methods 0.000 claims description 9
- 238000000638 solvent extraction Methods 0.000 claims description 7
- 230000009699 differential effect Effects 0.000 claims description 6
- 238000012946 outsourcing Methods 0.000 claims description 5
- 230000001174 ascending effect Effects 0.000 claims description 3
- 238000004891 communication Methods 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 230000036299 sexual function Effects 0.000 claims 1
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000007797 corrosion Effects 0.000 description 1
- 238000005260 corrosion Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000011478 gradient descent method Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a preferential direction deviation early warning system and method based on semantic segmentation, wherein the system comprises the following steps: the device comprises a semantic segmentation module, an offset detection module, a preference module and an offset early warning module; the method comprises the following steps: 1. extracting distribution information of road pixels by using a semantic segmentation network; 2. the two processes of identifying the road edge are performed in parallel to obtain two groups of results; 3. the preferred module evaluates the two results output by the offset detection module to select a root preferred result, so that the robustness of the algorithm is enhanced; 4. and obtaining an early warning result through processing the road edge information. The invention can realize the processing of the road edge information and give the offset early warning according to the processing, thereby ensuring the road safety.
Description
Technical Field
The invention belongs to the field of computer vision, relates to the technologies of semantic segmentation, hough straight line detection, minimum outsourcing triangle and the like, and provides a method for giving the condition of shifting a road in the advancing direction.
Background
At present, a lot of algorithms for detecting offset focus on segmentation and offset early warning of a highway drivable area generally only use clustering, edge detection, a filter-based and feature-based machine learning method in an image processing technology, and have poor effects on a general road, and compared with a special highway, the road in a general scene is usually narrower, the visual field is also smaller, and a plurality of saw-tooth or wavy edges can exist. These methods on roads are not applicable because they are limited to special scenes (e.g. require lane lines) and do not take into account the shape of the general road edges.
Disclosure of Invention
The invention aims to solve the defects in the prior art, and provides a preferential direction deviation early warning system and method based on semantic segmentation, so that the edge of a road can be accurately identified, the deviation early warning under a general road is realized, and the road safety is ensured.
The invention adopts the following method proposal for solving the technical problems:
the invention relates to a preferential direction deviation early warning system based on semantic segmentation, which is characterized by comprising the following components: the device comprises a semantic segmentation module, an offset detection module, a preference module and an offset early warning module;
the semantic segmentation module is provided with a semantic segmentation network F θ And is a nonlinear function with respect to the input picture, where θ is the semantic segmentation network F θ Parameters of (2); partitioning a network F using semantics θ Extracting features from the input current RGB picture X and outputting a road probability matrix P=F containing each pixel point θ (X), wherein X is a tensor of 3 XHXW, H, W respectively represent the height and width of the current RGB picture, P is a tensor of H XW, let P i Is the road probability value corresponding to the ith pixel point in P, P i ∈[0，1]，i＝1，2，...，H×W；
Using a binarization processing function F t Binarizing the road probability matrix P to obtain a classification result matrix S=F of all pixel points t (P) and S is H W tensor, let S i Is the classification value corresponding to the ith pixel point in S, S i E {0,1}, i=1, 2,..h×w, if S i =1, indicating that the ith pixel is a pixel on the road, if S i =0, indicating that the i-th pixel is not a pixel on the road;
the offset detection module adopts two methods to process the classification result matrix S in parallel and correspondingly obtains two kinds of the classification result matricesStraight line group L of each road edge 1 ，L 2 Each road edge straight line group comprises two straight lines which respectively represent two side edges of a road;
the preferential module is based on two road edge straight line groups L 1 ，L 2 Constructing a loss function loss:
wherein s is 1 ，s 2 The slope, k of two straight lines in any road edge straight line group corresponding to the current RGB picture X 1 ，k 2 Is the slope of two lines in the line group of the edge of the preferred road corresponding to the RGB picture input last time, d 1 ，d 2 Respectively represent the slope s 1 ，s 2 Corresponding intersection point coordinates, p, of two straight lines in the road edge straight line group 1 ，p 2 The coordinates of the central pixel point of the current RGB picture X; lambda (lambda) 1 ，λ 2 ，λ 3 ，λ 4 Is a set of hyper-parameters;
the preferential module utilizes loss function to respectively make two groups of road edge straight line groups L 1 ，L 2 Evaluating, so as to select a line group with a better road edge as L;
and the offset early warning module obtains a current offset result according to the distance between the center coordinate of the lower boundary of the current RGB picture X and the edge straight line group L, and further carries out early warning according to the offset result calculated by the continuous RGB pictures in a period of time.
The invention relates to a semantic segmentation-based preferential direction deviation early warning method which is characterized by comprising the following steps of:
step 1, constructing a semantic segmentation network F θ Is a nonlinear function with respect to the input picture, where θ is the semantic segmentation network F θ Parameters of (2); partitioning a network F using semantics θ Extracting features from the input current RGB picture X and outputting a road probability matrix P=F containing each pixel point θ (X), wherein X is a tensor of 3 XH X WH, W respectively represent the height and width of the current RGB picture, P is the tensor of H W, let P i Is the road probability value corresponding to the ith pixel point in P, P i ∈[0，1]，i＝1，2，...，H×W；
Using a binarization processing function F t Binarizing the road probability matrix P to obtain a classification result matrix S=F of all pixel points t (P) and S is H W tensor, let S i Is the classification value corresponding to the ith pixel point in S, S i E {0,1}, i=1, 2,..h×w, if S i =1, indicating that the ith pixel is a pixel on the road, if S i =0, indicating that the i-th pixel is not a pixel on the road;
step 2, processing the classification result matrix S according to the first and second branches respectively, and obtaining two linear groups L 1 ，L 2 ：
The first branch comprises:
step 2.1a, using edge detection filter bank with differential action [ K ] 1 ，K 2 ]Performing valid convolution operation on the classification result matrix S to obtain two groups of edge information of the current RGB picture XAnd->K 1 Representing the first filter, K 2 Representing a second filter->Representing a first filter result matrix,/for the first filter result matrix>Representing a second filter result matrix;
calculating a first filtering result matrixAnd S is B ∈R H×W For the first filtering result matrix S B Performing expansion treatment to obtain an expansion result matrix S D And S is D ∈R H×W ；
Step 2.2a, using a Gaussian filter on the expansion result matrix S D Processing to obtain Gaussian filter result matrix S G ；
Step 2.3a, calculating a Gaussian filter result matrix S G The ith pixel point S of (1) G (i) Is a binary value of (2) Thereby obtaining a binary Gaussian filter result matrix S 'after binarization' G ；
Step 2.4a, the binarized binary Gaussian filter result matrix S' G The coordinates of n pixel points with 1 value are stored in a list Point, and the point= [ (x) 1 ，y 1 )，(x 2 ，y 2 )，...，(x n ，y n )]；(x n ，y n ) Representing the coordinates of the nth pixel point with the value of 1;
processing the list Point by using a Hough straight Line detection algorithm to obtain a straight Line set line= [ (k) 1 ，b 1 )，(k 2 ，b 2 )，...，(k m ，b m )]Wherein k is m ，b m Respectively representing the slope and intercept of the mth straight line;
selecting r straight lines before intersection times ordering from the straight Line set Line and storing the same into the straight Line set L out Wherein is denoted as L out ＝[(k′ 1 ，b′ 1 )，(k′ 2 ，b′ 2 )，...，(k′ r ，b′ r )]Wherein k' r ，b′ r Respectively representing the slope and intercept of the r-th straight line, where r=min (α 1 ，m)，α 1 Is a preset super parameter;
step 2.5a, using a peak finding method to find the straight line set L out Processing to obtain two slope intervals, and collecting straight line set L out The straight lines in the two slope intervals are respectively stored in the list L p1 ，L p2 ；
Step 2.6a, for the list L p1 ，L p2 Each straight line in the two-point sampling device randomly samples two points respectively to correspondingly obtain two point sets P 1 ，P 2 Respectively calculating point sets P by using least square method 1 ，P 2 Corresponding best fit straight line (s 1 ，b 1 )，(s 2 ，b 2 ) Is stored in a straight line set L 1 ；s 1 Representing a set of points P 1 Slope of best fit straight line, b 1 Representing a set of points P 1 Intercept of the corresponding best fit line, s 2 Representing a set of points P 2 Slope of best fit straight line, b 2 Representing a set of points P 2 Intercept of the corresponding best fit line;
the second branch comprises:
step 2.1b, performing multiple open operations on the classification result matrix S to obtain an open operation result matrix S m And S is m ∈R H×W ；
Step 2.2b using edge detection filter bank with differential action [ K ]' 1 ，K′ 2 ]For the open operation result matrix S m Performing valid convolution operation to obtain two groups of edge information of the current RGB picture XAnd->K′ 1 Representing a third filter, K' 2 Representing a fourth filter->Representing a third filter result matrix,/for the filter result matrix>Representation ofFourth filter result matrix
Calculating a second filtering result matrixAnd S' B ∈R H×W Finding a second filtering result matrix S 'by utilizing findcontour method' B Set of all edge point sets in (c) = { E 1 ，E 2 ，...，E i′ ，...，E n′ E, where E i′ A set of points representing the i' th edge, and E i′ ＝[(x i′，1 ，y i′，1 )，(x i′，2 ，y i′，2 )，...(x i′，m′ ，y i′，m′ )]，i′＝1，2，...，n′；(x i′，m′ ，y i′，m′ ) Point set E representing the ith edge i′ The abscissa of the m 'th point in (b), m' being E i′ The number of edge points of the corresponding communicating blocks, n is S B The number of the medium communication blocks;
the area of the convex polygon corresponding to the Edge point of each communicating block is respectively calculated, and the Edge point of the communicating block with the largest area is reserved in a point set Edge;
step 2.3b, finding out the point with the smallest ordinate in the point set Edge, and marking the ordinate as y min Calculate the point set P tri ＝{(x，y)|(x，y)∈Edge，y≤α 2 ×(H-y min ) -a }; (x, y) represents the coordinates of any one Edge point in the point set Edge, α 2 Is a preset super parameter;
step 2.4b, processing the point set Ptri by using a minimum triangle method to obtain three points of the minimum triangle, finding out the point with the minimum longitudinal coordinate value in the three points, taking the connection line between the point with the minimum longitudinal coordinate value and the rest two points as straight lines on two sides of the road, and storing the straight lines in a straight line set L 2 In (a) and (b);
step 3, based on two road edge straight line groups L 1 ，L 2 Constructing a loss function loss:
wherein s is 1 ，s 2 The slope, k of two straight lines in any road edge straight line group corresponding to the current RGB picture X 1 ，k 2 Is the slope of two lines in the line group of the edge of the preferred road corresponding to the RGB picture input last time, d 1 ，d 2 Respectively represent the slope s 1 ，s 2 Corresponding intersection point coordinates, p, of two straight lines in the road edge straight line group 1 ，p 2 The coordinates of the central pixel point of the current RGB picture X; lambda (lambda) 1 ，λ 2 ，λ 3 ，λ 4 Is a set of hyper-parameters;
for two groups of road edge straight line groups L respectively by using loss function 1 ，L 2 Evaluating, so as to select a line group with a better road edge as L;
and 4, obtaining a current offset result according to the distance between the center coordinate of the lower boundary of the current RGB picture X and the edge straight line group L, and then carrying out early warning according to the offset result calculated by the continuous RGB pictures in a period of time.
The preferred direction deviation early warning method based on semantic segmentation is also characterized in that the peak searching method in the step 2.5a in the first branch is to search two intervals with the length of length and the most aggregated numerical values in a preset interval [ -s, s), wherein s and length are super parameters, and the peak searching method is carried out according to the following steps:
step 2.5.1, collecting the straight line set L out The straight lines in (a) are arranged in ascending order according to the slope to obtain a ordered straight line set L' out ；
Step 2.5.2 defining and initializing interval I as [ -s, -s+length), score set Score as empty list []The distance d=α of the center points of the current interval I and the possible interval W is defined and initialized 3 s；α 3 Representing a preset multiple;
step 2.5.3, calculating a Score score=g (βd) +λf (I) and storing in a Score set Score, wherein f representsStatistics L' out A function of the number of lines with a medium slope in interval I, λ, β being a hyper-parameter, g representing a function of sigmoid;
step 2.5.4, if score > t, assigning I to W and then executing step 2.5.5; otherwise, directly executing the step 2.5.5;
step 2.5.5, after adding length to the two end values of the interval I, if the left end value is greater than or equal to s, performing step 2.5.6, otherwise returning to step 2.5.3 for sequential execution;
step 2.5.6, selecting the two regions corresponding to the largest scores from the Score set Score as two optimal regions, and setting L' out Straight lines with medium slope in two optimal intervals are correspondingly stored in a list L p1 ，L p2 Is a kind of medium.
Compared with the prior art, the invention has the beneficial effects that:
1. according to the invention, the second branch utilizes the principle that parallel lines of the three-dimensional space are projected to the two-dimensional space and then intersect at shadow vanishing points, and proper points are selected for triangle outsourcing to obtain the road edge, so that the influence of shielding of objects in the road can be reduced, and more accurate results are brought to the road deviation.
2. The first branch in the invention is to obtain the equation of the straight line of the road edge by utilizing the edge profile information of the road according to the regression method, and the peak value searching method is more beneficial to removing interference information and finding the straight line of the road edge, thereby avoiding the situation that one road edge is identified as two.
3. The preferential module in the system constructs a set of evaluation algorithm according to the continuity of road position change and the centralization of data in a normal scene, and can reasonably give the reliability of a method result and select the reliability so as to integrate the respective advantages of the two methods, thereby enabling the deviation early warning result to be more accurate and stable.
Drawings
FIG. 1 is a flow chart of the connection relationship between the inside of each module and the module in the embodiment of the invention;
FIG. 2 is a schematic diagram of module association in an embodiment of the present invention;
FIG. 3 is a simplified diagram of a network architecture of an expression partitioning network used in an embodiment of the present invention;
FIG. 4 shows an expansion structure element used in branch one of the direction shift early warning method according to the embodiment of the present invention.
Detailed description of the preferred embodiments
In this embodiment, a preferential direction deviation early warning system based on semantic segmentation inputs a picture into a semantic segmentation module to obtain segmentation results, the results are transmitted to two methods in a deviation detection module to obtain two sets of road edge information, the two sets of edge information are input into the preferential module to select a better set of results, and finally a center line and a midpoint of a lower boundary of the picture are calculated in the deviation early warning module to obtain a result. As shown in fig. 2, specifically, the method includes: the device comprises a semantic segmentation module, an offset detection module, a preference module and an offset early warning module;
the semantic segmentation module is provided with a semantic segmentation network F θ And is a nonlinear function with respect to the input picture, where θ is the semantic segmentation network F θ Parameters of (2); partitioning a network F using semantics θ Extracting features from the input current RGB picture X and outputting a road probability matrix P=F containing each pixel point θ (X), wherein X is a tensor of 3 XHXW, H, W respectively represent the height and width of the current RGB picture, P is a tensor of H XW, let P i Is the road probability value corresponding to the ith pixel point in P, P i ∈[0，1]，i＝1，2，...，H×W；
Using a binarization processing function F t Binarization processing is carried out on the road probability matrix P to obtain a classification result matrix S=F of all pixel points t (P) and S is H W tensor, let S i Is the classification value corresponding to the ith pixel point in S, S i E {0,1}, i=1, 2,..h×w, if S i =1, indicating that the ith pixel is a pixel on the road, if S i =0, indicating that the i-th pixel is not a pixel on the road;
in the embodiment, the input image size is 1920x1080, the bit depth is 24, the network main body part adopts BiSeNetv2 semantic segmentation network, and the compression is doubled to 1x3x960x540 for reducing the calculation amountThe small tensor X is input into BiSeNetv2, the network structure is shown in figure 3, in the training stage, the pictures of the road are collected for classification marking, road pixel points are marked separately from other pixel points, then the output channel number of the BiSeNetv2 is changed to 1, as the probability that the predicted pixel is the road, the result and the mark obtained through the network are used for constructing BCELoss and Diceloss, the final Loss is loss=BCELoss+lambda Diceloss (lambda=1 in the embodiment), the error is propagated to the parameter theta through a gradient descent method, and the parameters are loaded in the prediction mode and the tensor P with the size of 1X1X960X540 is output. F (F) t ：R H×W →{0，1} H×W The input is subjected to element-wise operation, a threshold t (0.7 in this embodiment) is preset, and for each element of the input, if t is greater than t, the output is 1, otherwise, the output is 0.
The offset detection module adopts two methods to process the classification result matrix S in parallel and correspondingly obtains two road edge straight line groups L 1 ，L 2 Each road edge straight line group comprises two straight lines which respectively represent two side edges of a road;
the preferential module is based on two road edge straight line groups L 1 ，L 2 Constructing a loss function loss:
wherein s is 1 ，s 2 The slope, k of two straight lines in any road edge straight line group corresponding to the current RGB picture X 1 ，k 2 Is the slope of two lines in the line group of the edge of the preferred road corresponding to the RGB picture input last time, d 1 ，d 2 Respectively represent the slope s 1 ，s 2 Corresponding intersection point coordinates, p, of two straight lines in the road edge straight line group 1 ，p 2 The coordinates of the central pixel point of the current RGB picture X; lambda (lambda) 1 ，λ 2 ，λ 3 ，λ 4 Is a set of hyper-parameters; in the present embodiment, lambda 1 ，λ 2 ，λ 3 ，λ 4 Respectively 1，1，0.001，1。
The preferential module utilizes loss function to respectively make two groups of road edge straight line groups L 1 ，L 2 Evaluating, so as to select a line group with a better road edge as L;
the offset early warning module obtains a current offset result according to the distance between the center coordinate of the lower boundary of the current RGB picture X and the edge straight line group L, and then carries out early warning according to the offset result calculated by the continuous RGB pictures in a period of time.
In this embodiment, as shown in fig. 1, a preferential direction deviation early warning method based on semantic segmentation is performed according to the following steps:
step 1, constructing a semantic segmentation network F θ Is a nonlinear function with respect to the input picture, where θ is the semantic segmentation network F θ Parameters of (2); partitioning a network F using semantics θ Extracting features from the input current RGB picture X and outputting a road probability matrix P=F containing each pixel point θ (X), wherein X is a tensor of 3 XHXW, H, W respectively represent the height and width of the current RGB picture, P is a tensor of H XW, let P i Is the road probability value corresponding to the ith pixel point in P, P i ∈[0，1]，i＝1，2，...，H×W；
Using a binarization processing function F t Binarization processing is carried out on the road probability matrix P to obtain a classification result matrix S=F of all pixel points t (P) and S is H W tensor, let S i Is the classification value corresponding to the ith pixel point in S, S i E {0,1}, i=1, 2,..h×w, if S i =1, indicating that the ith pixel is a pixel on the road, if S i =0, indicating that the i-th pixel is not a pixel on the road;
step 2, processing the classification result matrix S according to the first and second branches respectively, and obtaining two linear groups L 1 ，L 2 ：
Branch one includes:
step 2.1a, using edge detection filter bank with differential action [ K ] 1 ，K 2 ]Performing valid convolution on the classification result matrix SOperation is carried out to obtain two groups of edge information of the current RGB picture XEye->K 1 Representing the first filter, K 2 Representing a second filter->Representing a first filter result matrix,/for the first filter result matrix>Representing a second filter result matrix;
calculating to obtain a first filtering result matrixAnd S is B ∈R H×W For the first filtering result matrix S B Performing expansion treatment to obtain an expansion result matrix S D And S is D ∈R H×W ；
Edge detection filter bank [ K ] in the present embodiment 1 ，K 2 ]Is thatThe anchor point is the center of the convolution kernel, the padding mode adopts 0 complementation, and the action process of one convolution kernel is as follows (K is adopted 1 Examples):
in this embodiment, as shown in fig. 4, the structural element adopted for expansion is an anchor point as a structural element center point, the coordinates represented by the squares in the gray area are set as H, and the two-dimensional tensor after the canny edge detection processing is set as F, so that the value of the output tensor G at the k/l position can be calculated to G by the following equation k，l ＝max i，j∈H {F k+i-1，l+j-1 The value of the point beyond F is treated as 0.
Step 2.2a, using Gaussian filter to expand the result matrix S D Processing to obtain Gaussian filter result matrix S G ；
Step 2.3a, calculating a Gaussian filter result matrix S G Binarized value of i-th pixel point SG (i)
Thereby obtaining a binary Gaussian filter result matrix S 'after binarization' G ；
Step 2.4a, binarizing the binary Gaussian filter result matrix S' G The coordinates of n pixel points with 1 value are stored in a list Point, and the point= [ (x) 1 ，y 1 )，(x 2 ，y 2 )，...，(x n ，y n )]；(x n ，y n ) Representing the coordinates of the nth pixel point with the value of 1;
processing the list Point by using a Hough straight Line detection algorithm to obtain a straight Line set line= [ (k) 1 ，b 1 )，(k 2 ，b 2 )，...，(k m ，b m )]Wherein k is m ，b m Respectively representing the slope and intercept of the mth straight line;
selecting r straight lines before intersection times ordering from the straight Line set Line and storing the same into the straight Line set L out Wherein is denoted as L out ＝[(k′ 1 ，b′ 1 )，(k′ 2 ，b′ 2 )，...，(k′ r ，b′ r )]Wherein k' r ，b′ r Respectively representing the slope and intercept of the r-th straight line, where r=min (α 1 ，m)，α 1 Is a preset super parameter; in the present embodiment alpha 1 ＝100；
Step 2.5a, utilizing a wave crest searching method to search the straight line set L out Processing to obtain two slope intervalsAnd collect straight lines L out The straight lines in the two slope intervals are respectively stored in the list L p1 ，L p2 ；
In this embodiment, the peak searching method searches for two sections with the length of length, where s is a super parameter, and the peak searching method is performed according to the following steps:
step 2.5.1, straight line set L out The straight lines in (a) are arranged in ascending order according to the slope to obtain a ordered straight line set L' out ；
Step 2.5.2 defining and initializing interval I as [ -s, -s+length), score set Score as empty list []The distance d=α of the center points of the current interval I and the possible interval W is defined and initialized 3 s；α 3 Representing a preset multiple;
in the present embodiment alpha 3 ＝2
Step 2.5.3, calculating a Score score=g (βd) +λf (I) and storing in a Score set Score, wherein f represents the statistic L' out A function of the number of lines with a medium slope in interval I, λ, β being a hyper-parameter, g representing a function of sigmoid;
in this embodiment s=3, β=1, length=0.5, λ=0.05, and t=0.5.
Step 2.5.4, if score > t, assigning I to W and then executing step 2.5.5; otherwise, directly executing the step 2.5.5;
step 2.5.5, after adding length to the two end values of the interval I, if the left end value is greater than or equal to s, performing step 2.5.6, otherwise returning to step 2.5.3 for sequential execution;
step 2.5.6, selecting the two regions corresponding to the largest scores from the Score set Score as two optimal regions, and setting L' out Straight lines with medium slope in two optimal intervals are correspondingly stored in a list L p1 ，L p2 Is a kind of medium.
Step 2.6a, pair list L p1 ，L p2 Each straight line in the two-point sampling device randomly samples two points respectively to correspondingly obtain two point sets P 1 ，P 2 Using least squares methodComputing a set of points P 1 ，P 2 Corresponding best fit straight line (s 1 ，b 1 )，(s 2 ，b 2 ) Is stored in a straight line set L 1 ；s 1 Representing a set of points P 1 Slope of best fit straight line, b 1 Representing a set of points P 1 Intercept of the corresponding best fit line, s 2 Representing a set of points P 2 Slope of best fit straight line, b 2 Representing a set of points P 2 Intercept of the corresponding best fit line;
the least squares method used in this embodiment uses the square of the distance of the estimated y from the actual y as the Loss, i.e., the result is thatThe smallest group (s, b), i.e. +.>Substituting the above formula into two pile of points to obtain (s 1 ，b 1 )，(s 2 ，b 2 ) Two straight lines added to the empty straight line set L 1 Is a kind of medium.
The second branch comprises:
step 2.1b, performing multiple open operations on the classification result matrix S to obtain an open operation result matrix S m And S is m ∈R H ×W ；
In this embodiment, the corrosion and expansion use 3x3 structural elements, the anchor point is located in the center, and the minimum (large) value of the whole area is solved and output to the anchor point.
Step 2.2b using edge detection filter bank with differential action [ K ]' 1 ，K′ 2 ]Split operation result matrix S m Performing valid convolution operation to obtain two groups of edge information of the current RGB picture XAnd->K′ 1 Representing a third filter, K' 2 Representing a fourth filter->Representing a third filter result matrix,/for the filter result matrix>Representing a fourth filter result matrix
Calculating to obtain a second filtering result matrixAnd S' B ∈R H×W Finding a second filtering result matrix S 'by utilizing findcontour method' B Set of all edge point sets in (c) = { E 1 ，E 2 ，...，E i′ ，...，E n′ E, where E i′ A set of points representing the i' th edge, and E i′ ＝[(x i′，1 ，y i′，1 )，(x i′，2 ，y i′，2 )，...(x i′，m′ ，y i′，m′ )]，i′＝1，2，...，n′；(x i′，m′ ，y i′，m′ ) Point set E representing the ith edge i′ The abscissa of the m 'th point in (b), m' being E i′ The number of edge points of the corresponding communicating blocks, n is S B The number of the medium communication blocks;
in this example, K' 1 ，K′ 2 And K is equal to 1 ，K 2 Consistent; the method for obtaining the area of the convex polygon is to divide the convex polygon into a plurality of triangles, then divide the triangle by 2 and then take the modulus to calculate the area of each triangle, for example, for the point set E i Let the point inside it be { A } 1 ，A 2 ，A 3 ，...，A m Then its area is:
step 2.3b, finding out the point with the smallest ordinate in the point set Edge, and marking the ordinate as y min Calculate the point set P tri ＝{(x，y)|(x，y)∈Edge，y≤α 2 ×(H-y min ) -a }; (x, y) represents the coordinates of any one Edge point in the point set Edge, α 2 Is a pre-set super parameter.
In the present embodiment
Step 2.4b, processing the point set P by using a minimum package triangle method tri Obtaining three points of the minimum outsourcing triangle, firstly finding out the point with the minimum longitudinal coordinate value in the three points, then taking the connection line between the point with the minimum longitudinal coordinate value and the rest two points as the straight line of two sides of the road respectively and storing the straight line into a straight line set L 2 In (a) and (b);
step 3, based on two road edge straight line groups L 1 ，L 2 The loss function is constructed as follows:
wherein s is 1 ，s 2 The slope, k of two straight lines in any road edge straight line group corresponding to the current RGB picture X 1 ，k 2 Is the slope of two lines in the line group of the edge of the preferred road corresponding to the RGB picture input last time, d 1 ，d 2 Respectively represent the slope s 1 ，s 2 Corresponding intersection point coordinates, p, of two straight lines in the road edge straight line group 1 ，p 2 The coordinates of the central pixel point of the current RGB picture X; lambda (lambda) 1 ，λ 2 ，λ 3 ，λ 4 Is a set of hyper-parameters;
for two groups of road edge straight line groups L respectively by using loss function 1 ，L 2 Evaluating, so as to select a line group with a better road edge as L;
and 4, obtaining a current offset result according to the distance between the center coordinate of the lower boundary of the current RGB picture X and the edge straight line group L, and then carrying out early warning according to the offset result calculated by the continuous RGB pictures in a period of time.
Claims (3)
1. A preferential direction deviation early warning method based on semantic segmentation is characterized by comprising the following steps:
step 1, constructing a semantic segmentation network F θ Is a nonlinear function with respect to the input picture, where θ is the semantic segmentation network F θ Parameters of (2); partitioning a network F using semantics θ Extracting features from the input current RGB picture X and outputting a road probability matrix P=F containing each pixel point θ (X), wherein X is a tensor of 3 XHXW, H, W respectively represent the height and width of the current RGB picture, P is a tensor of H XW, let P i Is the road probability value corresponding to the ith pixel point in P, P i ∈[0，1]，i＝1，2，...，H×W；
Using a binarization processing function F t Binarizing the road probability matrix P to obtain a classification result matrix S=F of all pixel points t (P) and S is H W tensor, let S i Is the classification value corresponding to the ith pixel point in S, S i E {0,1}, i=1, 2,..h×w, if S i =1, indicating that the ith pixel is a pixel on the road, if S i =0, indicating that the i-th pixel is not a pixel on the road;
step 2, processing the classification result matrix S according to the first and second branches respectively, and obtaining two linear groups L 1 ，L 2 ：
The first branch comprises:
step 2.1a, using edge detection filter bank with differential action [ K ] 1 ，K 2 ]Performing valid convolution operation on the classification result matrix S to obtain two groups of edge information of the current RGB picture XAnd->K 1 Representing the first filter, K 2 Representing a second filter->Representing a first filter result matrix,/for the first filter result matrix>Representing a second filter result matrix;
calculating a first filtering result matrixAnd S is B ∈R H×W For the first filtering result matrix S B Performing expansion treatment to obtain an expansion result matrix S D And S is D ∈R H×W ；
Step 2.2a, using a Gaussian filter on the expansion result matrix S D Processing to obtain Gaussian filter result matrix S G ；
Step 2.3a, calculating a Gaussian filter result matrix S G The ith pixel point S of (1) G (i) Is a binary value of (2) Thereby obtaining a binary Gaussian filter result matrix S 'after binarization' G ；
Step 2.4a, the binarized binary Gaussian filter result matrix S' G The coordinates of n pixel points with 1 value are stored in a list Point, and the point= [ (x) 1 ，y 1 )，(x 2 ，y 2 )，...，(x n ，y n )]；(x n ，y n ) Representing the coordinates of the nth pixel point with the value of 1;
processing the list Point by using a Hough straight line detection algorithm to obtainTo the Line set line= [ (k) 1 ，b 1 )，(k 2 ，b 2 )，...，(k m ，b m )]Wherein k is m ，b m Respectively representing the slope and intercept of the mth straight line;
selecting r straight lines before intersection times ordering from the straight Line set Line and storing the same into the straight Line set L out Wherein is denoted as L out ＝[(k′ i ，b′ i )，(k′ 2 ，b 2 )，...，(k′ r ，b′ r )]Wherein k' r ，b′ r Respectively representing the slope and intercept of the r-th straight line, where r=min (α 1 ，m)，α 1 Is a preset super parameter;
step 2.5a, using a peak finding method to find the straight line set L out Processing to obtain two slope intervals, and collecting straight line set L out The straight lines in the two slope intervals are respectively stored in the list L p1 ，L p2 ；
Step 2.6a, for the list L p1 ，L p2 Each straight line in the two-point sampling device randomly samples two points respectively to correspondingly obtain two point sets P 1 ，P 2 Respectively calculating point sets P by using least square method 1 ，P 2 Corresponding best fit straight line (s 1 ，b 1 )，(s 2 ，b 2 ) Is stored in a straight line set L 1 ；s 1 Representing a set of points P 1 Slope of best fit straight line, b 1 Representing a set of points P 1 Intercept of the corresponding best fit line, s 2 Representing a set of points P 2 Slope of best fit straight line, b 2 Representing a set of points P 2 Intercept of the corresponding best fit line;
the second branch comprises:
step 2.1b, performing multiple open operations on the classification result matrix S to obtain an open operation result matrix S m And S is m ∈R H×W ；
Step 2.2b using edge detection filter bank with differential action [ K ]' 1 ，K′ 2 ]For the open operation result matrixS m Performing valid convolution operation to obtain two groups of edge information of the current RGB picture XAnd->K′ 1 Representing a third filter, K' 2 Representing a fourth filter->Representing a third filter result matrix,/for the filter result matrix>Representing a fourth filter result matrix;
calculating a second filtering result matrixAnd S' B ∈R H×W Finding a second filtering result matrix S 'by utilizing findcontour method' B Set of all edge point sets in (c) = { E 1 ，E 2 ，...，E i′ ，...，E n′ E, where E i′ A set of points representing the i' th edge, and E i′ ＝[(x i′，1 ，y i′，1 )，(X i′，2 ，y i′，2 )，...(x i′，m′ ，y i′，m′ )]，i′＝1，2，...，n′；(x i′，m′ ，y i′，m′ ) Point set E representing the ith edge i′ The abscissa of the m 'th point in (b), m' being E i′ The number of edge points of the corresponding communicating blocks, n is S B The number of the medium communication blocks;
the area of the convex polygon corresponding to the Edge point of each communicating block is respectively calculated, and the Edge point of the communicating block with the largest area is reserved in a point set Edge;
step 2.3b, finding out the point with the smallest ordinate in the point set Edge,its squat is marked as y min Calculate the point set P tri ＝{(x，y)|(x，y)∈Edge，y≤α 2 ×(H-y min ) -a }; (x, y) represents the coordinates of any one Edge point in the point set Edge, α 2 Is a preset super parameter;
step 2.4b, processing the point set P by using a minimum outsourcing triangle method tri Obtaining three points of the minimum outsourcing triangle, firstly finding out the point with the minimum longitudinal coordinate value in the three points, then taking the connection line between the point with the minimum longitudinal coordinate value and the rest two points as the straight line of two sides of the road respectively and storing the straight line into a straight line set L 2 In (a) and (b);
step 3, based on two road edge straight line groups L 1 ，L 2 Constructing a loss function loss:
wherein s is 1 ，s 2 The slope, k of two straight lines in any road edge straight line group corresponding to the current RGB picture X 1 ，k 2 Is the slope of two lines in the line group of the edge of the preferred road corresponding to the RGB picture input last time, d 1 ，d 2 Respectively represent the slope s 1 ，s 2 Corresponding intersection point coordinates, p, of two straight lines in the road edge straight line group 1 ，p 2 The coordinates of the central pixel point of the current RGB picture X; lambda (lambda) 1 ，λ 2 ，λ 3 ，λ 4 Is a set of hyper-parameters;
for two groups of road edge straight line groups L respectively by using loss function 1 ，L 2 Evaluating, so as to select a line group with a better road edge as L;
and 4, obtaining a current offset result according to the distance between the center coordinate of the lower boundary of the current RGB picture X and the edge straight line group L, and then carrying out early warning according to the offset result calculated by the continuous RGB pictures in a period of time.
2. The semantic segmentation-based preferential direction shift early warning method according to claim 1, wherein the peak searching method in step 2.5a in the first branch is to search two intervals with the length of length, wherein the length is the super parameter, and the peak searching method is performed by the following steps:
step 2.5.1, collecting the straight line set L out The straight lines in (a) are arranged in ascending order according to the slope to obtain a ordered straight line set L' out ；
Step 2.5.2 defining and initializing interval I as [ -s, -s+length), score set Score as empty list []The distance d=α of the center points of the current interval I and the possible interval W is defined and initialized 3 s；α 3 Representing a preset multiple;
step 2.5.3, calculating a Score score=g (βd) +λf (I) and storing in a Score set Score, wherein f represents the statistic L' out A function of the number of lines with a medium slope in interval I, λ, β being a hyper-parameter, g representing a function of sigmoid;
step 2.5.4, if score > t, assigning I to W and then executing step 2.5.5; otherwise, directly executing the step 2.5.5;
step 2.5.5, after adding length to the two end values of the interval I, if the left end value is greater than or equal to s, performing step 2.5.6, otherwise returning to step 2.5.3 for sequential execution;
step 2.5.6, selecting the two regions corresponding to the largest scores from the Score set Score as two optimal regions, and setting L' out Straight lines with medium slope in two optimal intervals are correspondingly stored in a list L p1 ，L p2 Is a kind of medium.
3. A preferential direction shift early warning system realized by the preferential direction shift early warning method based on semantic segmentation as claimed in claim 1, characterized by comprising: the device comprises a semantic segmentation module, an offset detection module, a preference module and an offset early warning module;
the semantic segmentation module is provided with a semantic segmentation network F θ And is a non-line for inputting picturesA sexual function, where θ is the semantic segmentation network F θ Parameters of (2); partitioning a network F using semantics θ Extracting features from the input current RGB picture X and outputting a road probability matrix P=F containing each pixel point θ (X), wherein X is a tensor of 3 XHXW, H, W respectively represent the height and width of the current RGB picture, P is a tensor of H XW, let P i Is the road probability value corresponding to the ith pixel point in P, P i ∈[0，1]，i＝1，2，....，H×W；
Using a binarization processing function F t Binarizing the road probability matrix P to obtain a classification result matrix S=F of all pixel points t (P) and S is H W tensor, let S i Is the classification value corresponding to the ith pixel point in S, S i E {0,1}, i=1, 2,..h×w, if S i =1, indicating that the ith pixel is a pixel on the road, if S i =0, indicating that the i-th pixel is not a pixel on the road;
the offset detection module adopts two methods to process the classification result matrix S in parallel and correspondingly obtains two road edge straight line groups L 1 ，L 2 Each road edge straight line group comprises two straight lines which respectively represent two side edges of a road;
the preferential module is based on two road edge straight line groups L 1 ，L 2 Constructing a loss function loss:
wherein s is 1 ，s 2 The slope, k of two straight lines in any road edge straight line group corresponding to the current RGB picture X 1 ，k 2 Is the slope of two lines in the line group of the edge of the preferred road corresponding to the RGB picture input last time, d 1 ，d 2 Respectively represent the slope s 1 ，s 2 Corresponding intersection point coordinates, p, of two straight lines in the road edge straight line group 1 ，p 2 The coordinates of the central pixel point of the current RGB picture X; lambda (lambda) 1 ，λ 2 ，λ 3 ，λ 4 Is a set of hyper-parameters;
the preferential module utilizes loss function to respectively make two groups of road edge straight line groups L 1 ，L 2 Evaluating, so as to select a line group with a better road edge as L;
and the offset early warning module obtains a current offset result according to the distance between the center coordinate of the lower boundary of the current RGB picture X and the edge straight line group L, and further carries out early warning according to the offset result calculated by the continuous RGB pictures in a period of time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210374860.6A CN114723946B (en) | 2022-04-11 | 2022-04-11 | Preferential direction deviation early warning system and method based on semantic segmentation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210374860.6A CN114723946B (en) | 2022-04-11 | 2022-04-11 | Preferential direction deviation early warning system and method based on semantic segmentation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114723946A CN114723946A (en) | 2022-07-08 |
| CN114723946B true CN114723946B (en) | 2024-02-27 |
Family
ID=82244320
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210374860.6A Active CN114723946B (en) | 2022-04-11 | 2022-04-11 | Preferential direction deviation early warning system and method based on semantic segmentation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114723946B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019192397A1 (en) * | 2018-04-04 | 2019-10-10 | 华中科技大学 | End-to-end recognition method for scene text in any shape |
| CN111439259A (en) * | 2020-03-23 | 2020-07-24 | 成都睿芯行科技有限公司 | Agricultural garden scene lane deviation early warning control method and system based on end-to-end convolutional neural network |
-
2022
- 2022-04-11 CN CN202210374860.6A patent/CN114723946B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019192397A1 (en) * | 2018-04-04 | 2019-10-10 | 华中科技大学 | End-to-end recognition method for scene text in any shape |
| CN111439259A (en) * | 2020-03-23 | 2020-07-24 | 成都睿芯行科技有限公司 | Agricultural garden scene lane deviation early warning control method and system based on end-to-end convolutional neural network |
Non-Patent Citations (1)
| Title |
|---|
| 呼长春 ; 詹永照 ; 刘志强 ; .基于边缘特征的车道偏移检测与预警.计算机应用与软件.2009,(第01期),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114723946A (en) | 2022-07-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110738207B (en) | Character detection method for fusing character area edge information in character image | |
| CN111666921B (en) | Vehicle control method, apparatus, computer device, and computer-readable storage medium | |
| Torresen et al. | Efficient recognition of speed limit signs | |
| Chen et al. | An effective hybrid atrous convolutional network for pixel-level crack detection | |
| Zhou et al. | Deep learning-based crack segmentation for civil infrastructure: Data types, architectures, and benchmarked performance | |
| CN103065314B (en) | Image communicated domain rapid marking method based on linear description | |
| Lai et al. | Traffic Signs Recognition and Classification based on Deep Feature Learning. | |
| CN112308826B (en) | Bridge structure surface defect detection method based on convolutional neural network | |
| CN108009554A (en) | A kind of image processing method and device | |
| CN103886308A (en) | Pedestrian detection method through soft cascade classifiers according to polymerization channel characteristics | |
| CN105069451B (en) | A kind of Car license recognition and localization method based on binocular camera | |
| CN109034136A (en) | Image processing method, device, picture pick-up device and storage medium | |
| CN104915642B (en) | Front vehicles distance measuring method and device | |
| CN108038486A (en) | A kind of character detecting method | |
| CN111382658B (en) | Road traffic sign detection method in natural environment based on image gray gradient consistency | |
| Pham et al. | Automatic detection and measurement of ground crack propagation using deep learning networks and an image processing technique | |
| CN111242046A (en) | Ground traffic sign identification method based on image retrieval | |
| CN116883650A (en) | Image-level weak supervision semantic segmentation method based on attention and local stitching | |
| Barodi et al. | An enhanced artificial intelligence-based approach applied to vehicular traffic signs detection and road safety enhancement | |
| Kulambayev et al. | A deep learning-based approach for road surface damage detection | |
| Yildiz et al. | Hybrid image improving and CNN (HIICNN) stacking ensemble method for traffic sign recognition | |
| CN107832732B (en) | Lane line detection method based on treble traversal | |
| Eslami et al. | Comparison of deep convolutional neural network classifiers and the effect of scale encoding for automated pavement assessment | |
| CN116596921B (en) | Method and system for sorting incinerator slag | |
| CN114723946B (en) | Preferential direction deviation early warning system and method based on semantic segmentation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |