CN114596551A - Vehicle-mounted forward-looking image crack detection method - Google Patents

Abstract

The invention provides a method for detecting cracks in a vehicle-mounted front-view image. The method includes pavement semantic segmentation of front-view images, classification of pavement image blocks, crack detection, crack type discrimination and statistics. The processing process of the method of the invention is not approximate, and can effectively avoid the influence of weeds and debris on road surface detection. After introducing the histogram grayscale threshold segmentation algorithm based on saliency detection, the cracks are clear and easy to extract, and the crack detection of complex images can be adapted. And it has universal applicability and can be widely used in the field of pavement disease detection.

Description

A method for detecting cracks in vehicle-mounted front-view images

technical field

The invention relates to the technical field of road disease detection, in particular to a method for detecting cracks in a vehicle-mounted front-view image.

Background technique

At present, forward-looking road images have the characteristics of diverse background textures, multi-targets, complex lighting environments, and complex collection scenes, which make cracks one of the most difficult targets to identify in damage types. The front-view shooting of the vehicle camera, the scene is complex, in addition to the highway, it also includes the auxiliary facilities of the road construction and the background of the off-road environment; the noise is complex, including noises such as shadows, lane lines, oil spots and water spots;

There are three main types of pavement crack detection technologies today: artificial vision detection, digital image processing and three-dimensional laser detection methods. Among them, artificial visual detection has problems such as slow speed, low efficiency, high risk, and poor comprehensiveness. The detection steps of pavement crack identification technology based on image processing generally include the following contents: first, the crack image is collected by the vehicle-mounted camera, and then the collected image is preprocessed, crack detection and feature extraction are performed to obtain the crack type, crack geometric characteristic parameters and damage severity. and other information (Reference: Jiang Wenbo, Luo Qiurong, Zhang Xiaohua. A review of concrete road crack detection methods based on digital images [J]. Journal of Xihua University (Natural Science Edition), 2018(1):13.). Pavement crack detection technology based on image processing has become a common method in the field of pavement crack detection due to its accuracy, safety, robustness and real-time performance. The three-dimensional laser-based pavement crack detection method is not disturbed by light and shadow, and has high recognition accuracy and fast speed; however, the detection effect of small cracks is poor, and the hardware cost is high, so it cannot be widely used.

Therefore, this paper has a good theoretical significance and practical application value for the research on the road crack detection technology based on the vehicle front-view image, which can provide a more general, more efficient, more stable and more intelligent road crack identification technology for the real-time monitoring of road surface cracks. It can also provide a scientific evaluation system and theoretical basis for pavement crack detection.

SUMMARY OF THE INVENTION

In view of the above technical problems, the purpose of the present invention is to overcome the deficiencies of the prior art, and to propose a method for detecting cracks in a vehicle-mounted front-view image. The method includes pavement semantic segmentation of front-view images, classification of pavement image blocks, crack detection, crack type discrimination and statistics.

To achieve the above object, the present invention proposes a method for detecting cracks in a vehicle-mounted front-view image, comprising the steps of:

S1: For the vehicle front-view image, use the CNN road semantic segmentation method to separate the road surface and the background area, and obtain the road surface image;

S2: Divide the pavement image obtained in step S1 into n×n regular-sized image blocks, and use the ResNet network to perform binary classification on the image blocks to obtain cracked and non-cracked pavement image blocks;

S3: Perform image enhancement and other preprocessing on the cracked pavement image block obtained in step S2, and use the improved HC saliency detection and gray histogram threshold segmentation method for crack detection to obtain a crack segmentation image;

S4: Perform mathematical morphology crack contour extraction on the crack detection image obtained in step S3, determine the type of crack by histogram projection method, and establish a two-dimensional coordinate system to calculate information such as the length and width of the crack.

Preferably, the image preprocessing method in step S3 includes image grayscale correction processing, histogram equalization processing and median filter denoising processing. The improved HC saliency detection method consists of HC saliency detection and fine-scale enhancement. Part of the composition, after detection, the crack segmentation is performed by the grayscale histogram threshold segmentation method, including the following:

(1) HC-based saliency detection: The saliency value of a pixel is defined by the color difference from all other pixels in the image, and the expression is as follows:

In the formula, D(I _K , I _I ) is the color distance metric of two pixels in the space L·a·b. After the above formula is extended by the pixel level, the salient value of each color can be obtained. The formula is as follows:

where c ₁ is the color value in pixel I _k , n is the number of different pixel colors, f _j is the frequency of pixel color C _j in image I, and the true color space contains 256*256*256 possible colors, More than the total pixels of the image, computationally expensive. Quantize each color channel to 12, reduce the color to 12*12*12, discard the color that appears in low frequency, and keep the color that appears in high frequency. The quantization method of the RGB color space will produce flaws. In order to reduce the error, a smoothing operation is used to replace the salient value of each color with the weighted average of the salient values of similar colors, but similar colors need to measure the distance in the Lab color space. Let m = n/4 nearest neighbor colors to improve the saliency value of color c ₁ , the formula is as follows:

是颜色c和它的m个最近邻之间的距离；In the formula,

is the distance between color c and its m nearest neighbors;

(2) Fine-scale saliency enhancement: The fine-scale saliency enhancement algorithm can reduce the saliency value of image noise. Both the crack and the background texture have a certain scale. Converting the pavement crack image to a scale suitable for the crack will highlight the local characteristics of the crack. The scale transformation formula is as follows:

I _t+1 (x,y)=I(U _N,t (x,y))×h( )

t＝1；In the formula, t represents the image scale, t=0 is the original image, U _N,t (x, y) is the M×N area around the pixel (x, y) at the t scale, h( ) is the scale change kernel function. The size of the fracture varies with the image resolution. The shape of the fracture is narrow, long and narrow, and the scale should not be too thin. Take N=2,

t=1;

The fine-scale saliency enhancement algorithm simulates the linear expansion process of fractures. The fracture fine-scale saliency enhancement formula is as follows:

θ∈{0°,45°,90°,135°}

In the formula, I _u is the gray mean value of the crack-scale image. Let S _zu be the saliency mean, w _z (x, y) be the corresponding saliency weight at point (x, y), the formula is as follows:

In the formula, G _{N, θ} (x, y) is the sum of the gray values of the linear neighborhood in the θ direction with the point (x, y) as the center, and N=3. Let the number of original image pixel points be N. According to statistical rules, the proportion of candidate points in the original image is at most 10%. When the candidate point is a crack, the linear neighborhood points with a surrounding length of L do not need to be screened again;

Grayscale histogram threshold segmentation method: For pavement crack images, when the grayscale values of the target area and the background area are different, the grayscale histogram is displayed as two peaks with a valley between the peaks. When the two peaks correspond to the central gray value of the target area and the background area, respectively, the gray value corresponding to the valley value can be used as the threshold for image segmentation. Assuming that the gray value corresponding to the trough is T, then T is used as the segmentation threshold, the area composed of pixels with gray value less than T is taken as the crack area, and the pixels with gray value greater than T are taken as the background area. However, the pixel distribution of the crack area and the pavement area obeys a normal distribution, namely:

where f ₁ (i) is the distribution function of the fracture region;

It can be seen from the normal distribution characteristics that in the above formula, a ₁ is the central gray value of all pixels in the crack area, u ₁ is the average gray value of all pixels in the crack area, and δ ₁ ² is the mean square error of the gray value. . Likewise, f ₂ (i) represents the distribution function of the background region;

The qualifications are as follows:

When T lets f(T) take the minimum value, T is the desired threshold. Due to the limited range of gray value variation in the crack image, it does not cover all gray value ranges. Therefore, we can take advantage of this point, when designing the algorithm, only the number of pixels corresponding to the gray value in the crack is read to improve the operation efficiency of the algorithm;

Preferably, in the step S4, the crack contour extraction uses the projection method to identify the crack image and distinguish its crack type, and establish a two-dimensional coordinate system to calculate the information of the length and width of the crack, which mainly includes the following methods and features:

(1) Crack feature extraction and type identification based on projection method: extract features from images containing cracks, and identify different types of pavement cracks according to different eigenvalues. The binary image after crack segmentation is projected in different directions, and the geometric characteristics of the crack are determined by counting the number of pixel points. The projection result has the following characteristics:

①The horizontal projection amplitudes of transverse cracks are quite different, and the data changes obviously. When projecting it to the vertical direction, there is still an obvious difference in the projection amplitude, but it is relatively smooth. The number of pixels in each row is roughly the same;

②When the longitudinal crack is projected to the horizontal direction, there is still an obvious difference in its projection amplitude, while when it is projected to the vertical direction, its projection amplitude has an obvious maximum peak;

③ The horizontal direction of the network crack is similar to the vertical direction projection curve, and the amplitude fluctuation is roughly the same;

(2) Quantification of network cracks: The network crack adopts an envelope rectangle to represent the periphery of the crack. The determination of the outer envelope rectangle is determined by searching for the edge points of the crack target in the crack segmentation algorithm, and the four envelope rectangles are obtained. edge point, and then take its extreme value as the geometric parameter of the envelope rectangle. Therefore, for the quantification of network cracks, the envelope rectangle method is used to calculate the damage area:

A=Hμ·Wμ

Considering the conversion parameter t between the front-view image coordinate system and the road plane coordinate system, the formula is converted to:

A=Hμ·Wμ·t

(3) Calculation of linear crack length: Since the skeleton of a linear crack is composed of several small segments of skeleton, the length of each segment is calculated and summed, and the result is the length of the crack;

First, find the starting and ending points of the fracture skeleton, assuming that the corresponding coordinates are (x ₁ , y ₁ ) and (x _m , y _n ) respectively, and the coordinates of any two adjacent points are (x _t , y _t ) and (x _t+1 , y _t+1 ), the total length L of the crack can be calculated according to the following formula:

In the formula, D is the total length of the image, and Lt is the length of two adjacent points;

Considering the conversion parameter t between the front-view image coordinate system and the road plane coordinate system, the formula conversion is as follows:

(4) Calculation of linear crack width: The crack width can be briefly expressed by the ratio of the sum of the number of pixels in the skeleton to the crack length. The formula is as follows:

W=sum/L

Considering the conversion parameter t between the front-view image coordinate system and the road plane coordinate system, the formula conversion is as follows:

W=sum/L·t

where W is the width of the crack;

From the above, the present invention provides a method for detecting cracks in a vehicle-mounted front-view image, which introduces a deep learning classification method, a HC saliency algorithm, and an image threshold segmentation algorithm based on a grayscale histogram. The process has no approximation and can be used in application fields such as road disease detection, which can effectively improve the accuracy and stability of road crack detection.

Description of drawings

The description of this summary will become apparent and easily understood in conjunction with the following drawings, in which:

1 is a flow chart of a method for detecting cracks in a vehicle-mounted front-view image of the present invention

Figure 2 Three types of network verification results

Figure 3 Image preprocessing diagram

Figure 4 Comparison of saliency detection algorithms

Figure 5. Comparison of threshold segmentation methods

Figure 6 Binarization of significance detection results

Figure 7 Comparison of threshold segmentation after HC saliency detection algorithm and single histogram-based threshold segmentation

Figure 8 Comparison of skeleton extraction results

Detailed ways

A method for detecting cracks in a vehicle-mounted front-view image of the present invention will be described in detail below with reference to FIGS. 1 to 8 and Tables 1 to 5. FIG.

As shown in FIG. 1 , a method for detecting cracks in a vehicle-mounted front-view image of the present invention includes the following steps:

S1: For the vehicle front-view image, use the CNN road semantic segmentation method to separate the road surface and the background area, and obtain the road surface image;

S2: Divide the pavement image obtained in step S1 into image blocks with a regular size of 10cm×10cm, and use the ResNet network to classify the image blocks to obtain cracked and non-cracked pavement image blocks;

S3: Perform image enhancement and other preprocessing on the cracked pavement image block obtained in step S2, and use the improved HC saliency detection and gray histogram threshold segmentation method for crack detection to obtain a crack segmentation image;

S4: Perform mathematical morphology crack contour extraction on the crack detection image obtained in step S3, determine the type of crack by histogram projection method, and establish a two-dimensional coordinate system to calculate information such as the length and width of the crack.

The method of the present invention is further described below by means of implementation examples.

Data set: According to the collected image data and grid number, manually classify the images after the checkerboard grid block, and classify the block images into two categories: cracked images and non-cracked images, all with cracks are one. Classes, all without cracks are classified as one class, and divided into training set and validation set, the number of images contained in training set and validation set is 5000 and 400 respectively.

Evaluation indicators: Calculate the precision rate (Pr), recall rate (Re), F-measure Implementation steps:

a) For the collected images, use the CNN road semantic segmentation method to separate the road surface and the background area, and obtain the road surface image, according to. According to the calibration and measurement method of the camera, the segmented road area image is divided into 10cm×10cm grid. For the grid-blocked images, ResNet50 residual network, VGG network, and MobilenetV2 lightweight mobile network are used to classify the training set image blocks under the same parameters. The parameters are shown in Table 1. The VGG convolutional neural network, MobilenetV2 lightweight mobile network and ResNet50 residual network are verified for the same image block, and the results are shown in Figure 2. Select 400 pictures, and test them for Top-1 picture accuracy rate, precision rate, recall rate, and F-measure. The results are shown in Table 2, which proves that the ResNet50 residual network can effectively classify the cracks. The image block is used for the subsequent crack image preprocessing and detection and identification, which meets the requirements of the experiment.

Table 1

Table 2

b) Preprocessing the classified images with cracks, firstly, through grayscale transformation, the contrast of the cracked area is improved, so that the image reaches the ideal grayscale range. Secondly, through image grayscale correction processing, histogram equalization processing and median filter denoising processing, the image is preprocessed. The processing result is shown in Figure 3. While retaining and highlighting the image details, the noise of the image is also removed. It solves the contradiction between denoising and preserving details.

c) After image preprocessing, image threshold segmentation of gray histogram is performed on the image. The improved HC saliency detection algorithm is compared with the FT saliency detection algorithm, the AC saliency detection algorithm, and the LC saliency detection algorithm to obtain their respective saliency maps and binarize them, as shown in Figure 4 and Figure 6, the results prove the present invention Compared with the other three traditional saliency detection algorithms, the improved HC saliency detection algorithm is more suitable for crack detection.

The iterative global thresholding method, Otsu method global thresholding method and threshold segmentation method based on gray histogram are respectively applied to the same image to conduct comparative experiments, and the optimal segmentation method is obtained through analysis. The valley value of the graph can achieve good results as a threshold to segment the image.

Combined with the gray histogram threshold segmentation algorithm based on the improved HC saliency detection algorithm and a single threshold segmentation based on gray histogram to segment the image and calculate the precision (Precision), recall (Recall), F-measure (F-measure). -Measure) three indicators, the processing result is shown in Figure 7, and the corresponding accuracy rate, recall rate and F-measure of the crack detection algorithm are shown in Table 3. It can be clearly seen from the chart that the algorithm of the present invention has a great effect on the detection and segmentation of pavement cracks. A huge improvement, which can effectively segment the target area from the background area.

table 3

d) Perform the crack contour extraction based on mathematical morphology on the image after threshold segmentation, and the result is shown in Figure 8. The connected domain acquisition and skeleton extraction of the image through the constructed structural elements can clearly obtain the outline of the edge of the grayscale change of the binarized image, can splicing the fractured area, and can effectively remove the isolated points. Then, the projection method is used to extract and identify the crack features, and distinguish whether it is a transverse crack, a longitudinal crack, or a network crack. And the geometric characteristic parameters of cracks are obtained, which is convenient to accurately evaluate the damage condition of highway pavement and reflect the actual pavement condition.

e) 40 test images were selected for the experiment. After a series of the above operations, the results of extracting cracks and the results of manual crack identification were compared to illustrate the accuracy of the algorithm in this paper for crack identification. After manual inspection, it is concluded that there are 74 transverse cracks, 53 longitudinal cracks, and 29 network cracks on the pavement in the test section. Based on this, the algorithm used in this paper is used to detect cracks, and some experimental results are obtained. See Table 4 and Table 5.

Table 4

table 5

In the method for detecting cracks in vehicle front-view images of the present invention, after introducing the histogram grayscale threshold segmentation algorithm based on saliency detection, the cracks are clear and easy to extract, which can be adapted to crack detection of complex images, and has universal applicability and can be widely used in the field of road damage detection.

The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within the range.

Claims (3)

1. a method for vehicle-mounted front-view image crack detection, is characterized in that, comprises the steps: S1: For the vehicle front-view image, use the CNN road semantic segmentation method to separate the road surface and the background area, and obtain the road surface image; S2: Divide the pavement image obtained in step S1 into n×n regular-sized image blocks, and use the ResNet network to perform binary classification on the image blocks to obtain cracked and non-cracked pavement image blocks; S3: Perform image enhancement and other preprocessing on the cracked pavement image block obtained in step S2, and use the improved HC saliency detection and gray histogram threshold segmentation method for crack detection to obtain a crack segmentation image; S4: Perform mathematical morphology crack contour extraction on the crack detection image obtained in step S3, determine the type of crack by histogram projection method, and establish a two-dimensional coordinate system to calculate information such as the length and width of the crack. 2. The method for detecting cracks in a vehicle-mounted front-view image according to claim 1, wherein the image preprocessing method in the step S3 comprises image grayscale correction processing, histogram equalization processing and median value Filtering and denoising processing, the improved HC saliency detection method consists of two parts: HC saliency detection and fine-scale enhancement. After detection, the gray histogram threshold segmentation method is used for crack segmentation, which mainly includes the following methods and features: (1) HC-based saliency detection: The saliency value of a pixel is defined by the color difference from all other pixels in the image, and the expression is as follows:

In the formula, D(I _K , I _I ) is the color distance metric of two pixels in the space L·a·b. After the above formula is extended by the pixel level, the salient value of each color can be obtained. The formula is as follows:

where c ₁ is the color value in pixel I _k , n is the number of different pixel colors, f _j is the frequency of pixel color C _j in image I, and the true color space contains 256*256*256 possible colors, More than the total pixels of the image, computationally expensive. Quantize each color channel to 12, reduce the color to 12*12*12, discard the color that appears in low frequency, and keep the color that appears in high frequency. The quantization method of the RGB color space will produce flaws. In order to reduce the error, a smoothing operation is used to replace the salient value of each color with the weighted average of the salient values of similar colors, but similar colors need to measure the distance in the Lab color space. Let m = n/4 nearest neighbor colors to improve the saliency value of color c ₁ , the formula is as follows:

In the formula,

is the distance between color c and its m nearest neighbors; (2) Fine-scale saliency enhancement: The fine-scale saliency enhancement algorithm can reduce the saliency value of image noise. Both the crack and the background texture have a certain scale. Converting the pavement crack image to a scale suitable for the crack will highlight the local characteristics of the crack. The scale transformation formula is as follows: I _t+1 (x,y)=I(U _N,t (x,y))×h( ) In the formula, t represents the image scale, t=0 is the original image, U _N,t (x, y) is the M×N area around the pixel (x, y) at the t scale, h( ) is the scale change kernel function. The size of the fracture varies with the image resolution. The shape of the fracture is narrow, long and narrow, and the scale should not be too thin. Take N=2,

t=1; The fine-scale saliency enhancement algorithm simulates the linear expansion process of fractures. The fracture fine-scale saliency enhancement formula is as follows:

θ∈{0°,45°,90°,135°} In the formula, I _u is the gray mean value of the crack-scale image. Let S _zu be the saliency mean, w _z (x, y) be the corresponding saliency weight at point (x, y), the formula is as follows:

In the formula, G _{N, θ} (x, y) is the sum of the gray values of the linear neighborhood in the θ direction with the point (x, y) as the center, and N=3. Let the number of original image pixel points be N. According to statistical rules, the proportion of candidate points in the original image is at most 10%. When the candidate point is a crack, the linear neighborhood points with a surrounding length of L do not need to be screened again; Grayscale histogram threshold segmentation method: For pavement crack images, when the grayscale values of the target area and the background area are different, the grayscale histogram is displayed as two peaks with a valley between the peaks. When the two peaks correspond to the central gray value of the target area and the background area, respectively, the gray value corresponding to the valley value can be used as the threshold for image segmentation. Assuming that the gray value corresponding to the trough is T, then T is used as the segmentation threshold, the area composed of pixels with gray value less than T is taken as the crack area, and the pixels with gray value greater than T are taken as the background area. However, the pixel distribution of the crack area and the pavement area obeys a normal distribution, namely:

where f ₁ (i) is the distribution function of the fracture region; It can be seen from the normal distribution characteristics that in the above formula, a ₁ is the central gray value of all pixels in the crack area, u ₁ is the average gray value of all pixels in the crack area, and δ ₁ ² is the mean square error of the gray value. . Likewise, f ₂ (i) represents the distribution function of the background region; The qualifications are as follows:

When T lets f(T) take the minimum value, T is the desired threshold. Due to the limited range of gray value variation in the crack image, it does not cover all gray value ranges. Therefore, this point can be used, when designing the algorithm, only the number of pixels corresponding to the gray value in the crack is read to improve the operation efficiency of the algorithm. 3. a kind of vehicle-mounted front-view image crack detection method according to claim 1, is characterized in that, in described step S4, crack contour extraction uses the method of projection to identify crack image and distinguish its crack type, establish two. The dimensional coordinate system calculates the information of the length and width of the crack, mainly including the following methods and features: (1) Crack feature extraction and type identification based on projection method: extract features from images containing cracks, and identify different types of pavement cracks according to different eigenvalues. The binary image after crack segmentation is projected in different directions, and the geometric characteristics of the crack are determined by counting the number of pixel points. The projection result has the following characteristics: ①The horizontal projection amplitudes of transverse cracks are quite different, and the data changes obviously. When projecting it to the vertical direction, there is still an obvious difference in the projection amplitude, but it is relatively smooth. The number of pixels in each row is roughly the same; ②When the longitudinal crack is projected to the horizontal direction, there is still an obvious difference in the projection amplitude, while when it is projected to the vertical direction, the projection amplitude has an obvious maximum peak; ③ The horizontal direction of the network crack is similar to the vertical direction projection curve, and the amplitude fluctuation is roughly the same; (2) Quantification of network cracks: The network crack adopts an envelope rectangle to represent the periphery of the crack. The determination of the outer envelope rectangle is determined by searching for the edge points of the crack target in the crack segmentation algorithm, and the four envelope rectangles are obtained. edge point, and then take its extreme value as the geometric parameter of the envelope rectangle. Therefore, for the quantification of network cracks, the envelope rectangle method is used to calculate the damage area: A=Hμ·Wμ Considering the conversion parameter t between the front-view image coordinate system and the road plane coordinate system, the formula is converted to: A=Hμ·Wμ·t (3) Calculation of linear crack length: Since the skeleton of a linear crack is composed of several small segments of skeleton, the length of each segment is calculated and summed, and the result is the length of the crack; First, find the starting and ending points of the fracture skeleton, assuming that the corresponding coordinates are (x ₁ , y ₁ ) and (x _m , y _n ) respectively, and the coordinates of any two adjacent points are (x _t , y _t ) and (x _t+1 , y _t+1 ), the total length L of the crack can be calculated according to the following formula:

In the formula, D is the total length of the image, and Lt is the length of two adjacent points; Considering the conversion parameter t between the front-view image coordinate system and the road plane coordinate system, the formula conversion is as follows:

(4) Calculation of linear crack width: The crack width can be briefly expressed by the ratio of the sum of the number of pixels in the skeleton to the crack length. The formula is as follows: W=sum/L Considering the conversion parameter t between the front-view image coordinate system and the road plane coordinate system, the formula conversion is as follows: W=sum/L·t where W is the width of the crack.

Images