CN114331986A - Dam crack identification and measurement method based on unmanned aerial vehicle vision - Google Patents
Dam crack identification and measurement method based on unmanned aerial vehicle vision Download PDFInfo
- Publication number
- CN114331986A CN114331986A CN202111575229.4A CN202111575229A CN114331986A CN 114331986 A CN114331986 A CN 114331986A CN 202111575229 A CN202111575229 A CN 202111575229A CN 114331986 A CN114331986 A CN 114331986A
- Authority
- CN
- China
- Prior art keywords
- crack
- image
- calculating
- pixel
- cracks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
A dam crack identification and measurement method based on unmanned aerial vehicle vision comprises the following steps: step 1: collecting images, namely collecting RGB images of the dam body to be detected, wherein the collected images can completely cover a detection area; step 2: carrying out rough detection on the cracks; and step 3: carrying out fine crack identification based on visual saliency; and 4, step 4: performing edge extraction; and 5: calculating crack characteristic information; step 6: performing image splicing, and reducing a dam body panoramic view through image splicing so as to determine the position information of the local cracks relative to the dam body; and 7: and (5) positioning and evaluating the cracks. The invention aims to solve the technical problems that the dam crack detection error is large in complex environment and the accuracy in crack geometric information measurement is low in the prior art, and provides a dam crack identification and measurement method based on unmanned aerial vehicle vision.
Description
Technical Field
The invention relates to the technical field of machine vision detection and measurement, in particular to a dam crack identification and measurement method based on unmanned aerial vehicle vision.
Background
Cracks are important diseases which endanger the structural safety of a dam body of a hydropower station, and the position detection and the geometric form, particularly the width measurement of the cracks are important contents of the dam body detection. Traditional detection method mainly needs to build a scaffold, hang a basket or use a telescope based on artificial vision detection, and consumed manpower and material resources are large, detection potential safety hazards are large, detection efficiency is low, and detection effect often depends on the experience abundance of detection personnel.
With the development of artificial intelligence technology and unmanned aerial vehicle industry, more and more operation and maintenance personnel introduce unmanned aerial vehicles into operation and maintenance work of industries such as dams or roads and bridges by combining AI routing inspection, collect images of areas to be detected by remotely controlling the unmanned aerial vehicles, identify cracks by using a computer vision technology,
for example, patent document No. 2017103826144 proposes a surface crack recognition method based on saliency detection, which first transforms an image from RGB to Lab color space, then pre-segments the image by using a superpixel segmentation algorithm, clusters the image by combining a mean shift algorithm, then extracts a color saliency map and a texture saliency map of the image, and finally adaptively segments a crack region.
Patent document with application number 201811332035X proposes a bridge bottom crack detection method based on unmanned aerial vehicle vision, and the method carries a visible light camera through an unmanned aerial vehicle to orderly shoot a series of RGB images of a bridge block according to a preset track, processes the images in a numbering mode, then carries out steps of enhancing and denoising the images to extract crack features, and finally binaryzation segmentation of crack regions is carried out. A three-layer BP neural network is also established in the method for classifying the types of the cracks.
When the prior art faces a complex background environment, the technical defects that the algorithm identification precision is low, the speed is low, and the real-time and efficient crack detection requirement is difficult to meet exist, and meanwhile, the crack risk assessment is inaccurate due to the fact that the measurement of information such as the length or the width of a crack is lacked in the detection process.
Disclosure of Invention
The invention aims to solve the technical problems that the dam crack detection error is large in complex environment and the accuracy in crack geometric information measurement is low in the prior art, and provides a dam crack identification and measurement method based on unmanned aerial vehicle vision.
A dam crack identification and measurement method based on unmanned aerial vehicle vision comprises the following steps:
step 1: collecting images, namely collecting RGB images of the dam body to be detected, wherein the collected images can completely cover a detection area;
step 2: carrying out rough detection on the cracks;
and step 3: carrying out fine crack identification based on visual saliency;
and 4, step 4: performing edge extraction;
and 5: calculating crack characteristic information;
step 6: carrying out image splicing; restoring a dam body panoramic view through image splicing so as to determine the position information of the local cracks relative to the dam body;
and 7: and (5) positioning and evaluating the cracks.
In the step 2, a pre-trained convolutional neural network model is used for detecting whether cracks exist in the image, if the cracks are not detected, the next fine detection is not needed, and therefore the algorithm efficiency is improved; if the crack is detected, outputting a crack boundary frame in the image through the model, thereby filtering complex background interference information and improving the accuracy of subsequent crack edge extraction and the precision of crack measurement.
In step 3, according to the difference of the image cracks relative to the image background in color and brightness, the image saliency map obtained by the FT algorithm is used, the image threshold is calculated in a self-adaptive mode by combining the mean shift algorithm, and then the cracks are segmented in a binary mode.
In step 4, extracting crack edge pixel points by using a Canny edge detection algorithm, smoothing an image by using a Gaussian filter, calculating the amplitude and the direction of a gradient by using finite difference of first-order partial derivatives, performing non-maximum suppression on the gradient amplitude, and detecting and connecting edges by using a dual-threshold algorithm.
In step 5, firstly calibrating a camera, calculating and acquiring internal and external parameters of the camera, calculating conversion between an image coordinate system and a world coordinate system, and then calculating the width by adopting a shortest distance method according to crack edge pixels; calculating the Euclidean distance between the starting point and the ending point of the crack as the length of the crack; and taking the included angle of the fitting straight line of the crack as the included angle of the crack.
Further comprising step 7: positioning and evaluating the cracks; the position of the crack relative to the dam body is positioned, so that operation and maintenance staff can conveniently and accurately determine the position information of the crack; determining the damage grade of the crack according to the length, width, angle and area information of the crack in the step 5; and the development trend of cracks is predicted, the dam body damage work is done in time, and the accidents are prevented.
In step 2, when performing rough crack detection, the following steps are specifically adopted:
2.1: collecting a data set; collecting dam body crack data, wherein the collection environment comprises cloudy days, foggy days, light facing, backlight and the like, and ensuring that the collected data set contains various conditions in a general detection environment;
2.2, expanding the data set; expanding a data set by image rotation, brightness conversion, shearing and Gaussian noise increasing;
2.3, labeling the data set; in semi-supervised learning, an image labeling tool is used for labeling a data set, the speed and the precision of feature extraction are improved, data are made into a specified format, and a specified file corresponding to an image is generated;
2.4, optimizing the network; (1) aiming at the problem of inconsistent crack sizes, redesigning the prior frame size by using a K-means + + algorithm and matching the prior frame size to a corresponding characteristic layer; (2) a multispectral channel attention module is introduced into a feature extraction network, so that the network can independently learn the weight of each channel and enhance the information propagation among features, thereby enhancing the distinguishing capability of the network on the foreground and the background, and randomly inputting images with different sizes in the training iteration process so as to enhance the generalization capability of the model; (3) the CIoU _ loss is adopted as a regression loss function of a target detection task, the overlapping area and the central point distance between a prediction frame and a target frame are considered in the CIoU _ loss, when the target frame wraps the prediction frame, the distance of 2 frames is directly measured, so that the information of the central point distance of the boundary frame and the scale information of the width-to-height ratio of the boundary frame are considered, meanwhile, the length-to-width ratio of the prediction frame and the target frame is also considered, the boundary regression result is better, and a loss function calculation formula is as follows
Wherein IoU is the intersection sum of the prediction box and the target boxThe ratio of union, the center of the prediction frame is represented by b, and the center of the target frame is represented by bgtIndicating that ρ (·) represents the euclidean distance, c represents the diagonal distance between the intersected prediction box and the target box, which circumscribes the smallest rectangle, α is a weight coefficient,
v represents a parameter for the uniformity of the aspect ratio,
wgtand hgtRepresent the width and height of the target box, w and h represent the width and height of the prediction box, respectively;
2.5, training a model; the manufactured data set is brought into an optimized network for training, and the model is continuously iterated through parameter adjustment, so that the loss function of the model is reduced to the minimum; finally generating a crack detection model;
2.6, detecting a crack area; calling a crack detection model, inputting an image to be detected into the model, and outputting the category and the boundary frame of the crack end to end through the model;
in step 3, when performing fine crack identification based on visual saliency, the following steps are specifically adopted:
3.1, calculating an image saliency map; intercepting a crack area image through a crack boundary identified by a crack detection model, carrying out Gaussian filtering denoising on the image, and then converting the color space of the image from RGB to Lab; calculating the average value of L, a and b of the whole picture; calculating Euclidean distances between the L, a and b values of each pixel and the mean values of the three L, a and b values of the image according to a formula in an algorithm to obtain a saliency map; normalizing, namely dividing the significant value of each pixel in the image by the maximum significant value to obtain a final significant map; the saliency of a pixel can be calculated using the following formula
S(p)＝||Iμ-Iωhc(p)||
Wherein, IμIs a feature vector of the average image, Iωhc(p) is Lab color feature of pixel p after Gaussian smoothing, and | | is L2 paradigm, that is, the former term and the latter term are calculated in Lab color spaceThe Euclidean distance of;
3.2, dividing the original picture into K large blocks by using a Meanshift method, and calculating the average value Sk of a saliency map corresponding to each divided block by using a 3.1 saliency calculation method;
3.3, calculating the average value S mu of the whole saliency map;
3.4, calculating an adaptive threshold value Ta-2S mu;
3.5, when Sk > Ta, the region is considered as a significance map;
and 3.6, carrying out binarization segmentation on the salient region to obtain a segmented image with refined cracks.
In the step 4, a Canny algorithm is adopted to detect the crack edge, and the method specifically comprises the following steps:
4.1, eliminating noise; convolution noise reduction is carried out by using a Gaussian smoothing filter, and noise points around the crack are eliminated; the gaussian kernel used for gaussian filtering is the product of two one-dimensional gaussians, x and y, and the standard deviation σ in both dimensions is usually the same, and is of the form:
convolution noise reduction using a gaussian smoothing filter;
4.2, calculating the amplitude and the direction of the gradient; operating according to the step of Sobel filtering; first using a pair of convolution arrays acting in the x and y directions, respectively, i.e.
Recalculating gradient magnitude and direction
4.3, non-maximum suppression;
4.4, hysteresis thresholds, i.e. hysteresis high and low thresholds; specifically, if the amplitude of a pixel location exceeds a high threshold, the pixel is retained as an edge pixel; if the magnitude of the pixel location is less than the low threshold, the pixel is excluded; if the magnitude of a pixel location is between 2 thresholds, then the pixel is retained only when connected to a pixel above the high threshold.
In step 5, when calculating the crack characteristic information, the method includes measuring the length and width of the crack, and specifically includes the following steps:
5.1, calibrating a camera to obtain internal and external parameters; after the crack image identification is completed, calculating the edge coordinates of the crack at each point in the image for the image containing the crack, and calculating the actual size of the crack according to the coordinate transformation relation; the internal reference and the external reference of the camera can be directly obtained by using a Zhangyingyou calibration method; as shown in the following formula
Wherein, a is an internal parameter matrix of the camera, R is a rotation matrix of the external parameter, T is a translation vector, α ═ f/dx, β ═ f/dy, f is a focal length, and dx and dy are width and height of the pixel respectively; gamma represents the deviation of pixel point in x, y direction and (u)0,v0) Is a reference point;
5.2, converting pixel coordinates into world coordinates; the relationship between the pixel coordinate system and the world coordinate system is shown in the following matrix:
wherein [ X Y Z]TRepresenting the world coordinate system, [ u v ]]TRepresenting a pixel coordinate system with coordinates in the upper left corner of the image, ZCIs the corresponding Z-axis coordinate value under the camera coordinate system;
5.3, calculating the width of the crack; adopting a shortest distance method, taking a vertical crack as an example, marginalizing a crack image, and taking a left edge point of each line as a center; calculating the minimum value of the distances of the right edge points in a plurality of rows adjacent to the row as the width of the crack of the row; the Euclidean distance calculation method of the crack width is as follows
Wherein w is the crack width, Z under monocular cameral＝Zr0; maximum value w of crackmax＝max(wi) Average value of cracks is
wiThe width of the ith row of the crack is shown, and N is the total row number;
5.4, calculating the length; the skeleton information obtained after thinning operation is a thin line graph with single pixel arrangement, and the principle of calculating the dam crack length information can be that the Euclidean distance between the starting point and the ending point of the thinned crack is calculated as the length information of the dam crack;
5.5, calculating an included angle; because the dam body crack graph has an irregular shape, the crack image needs to be fitted in the processing process to form a straight line capable of being calculated;
5.6, calculating the area; the minimum circumscribed rectangular area of the crack was taken as the area of the crack.
Compared with the prior art, the invention has the following technical effects:
1. the invention provides a crack identification and measurement method based on unmanned aerial vehicle vision, which is characterized in that an unmanned aerial vehicle carries a high-definition camera to automatically inspect a dam body, and the crack image identification algorithm provided by the invention is used for automatically processing the collected image, so that the crack of the dam body is quickly and accurately identified, the length and the width of the crack are measured, and the dam body maintenance and damage prediction efficiency is improved.
2. The invention provides a crack detection method based on improved YOLOv5, which improves the speed and the precision of crack detection by improving a YOLOv5 network model, and improves the accuracy of subsequent crack segmentation and measurement by detecting crack regions and filtering background interference information.
3. The invention provides a Meanshift crack segmentation method based on saliency image detection. By introducing a significance detection step into the traditional Meanshift segmentation algorithm, the accuracy and robustness of crack segmentation are improved.
Drawings
The invention is further illustrated by the following examples in conjunction with the accompanying drawings:
FIG. 1 is a flow chart of a method for identifying and measuring cracks based on unmanned aerial vehicle vision in the invention;
FIG. 2 is a flow chart of a crack detection method based on improved YOLOv5 in the invention;
FIG. 3 is a flow chart of fine crack identification based on visual saliency in the present invention;
fig. 4 is a schematic view of a crack region segmented by the method of the present invention.
Detailed Description
As shown in figure 1, the dam crack identification and measurement method based on unmanned aerial vehicle vision comprises the following steps:
1. collecting an image; the dam body RGB image acquisition method based on the Zen-Si H20T video camera is characterized in that a Xinsi M300RTK unmanned aerial vehicle in Dajiang is used, the Zen-Si H20T video camera is carried, the unmanned aerial vehicle flies according to a preset track, and RGB images of the dam body are acquired in sequence by adjusting information such as hovering postures of the unmanned aerial vehicle, angles and focal lengths of the video camera.
2. Crack gross detection based on YOLOv 5; detecting whether the image has cracks by using a pre-trained YOLOv5 convolutional neural network model, and if the image has no cracks, performing the next fine detection; if the image is detected to contain the crack, outputting a boundary frame of the crack through the model, thereby filtering complex background interference information and improving the accuracy of subsequent crack edge extraction and the precision of crack measurement. The crack growth detection procedure based on YOLOv5 is shown in fig. 2.
3. Fine crack identification based on visual saliency; according to the difference of the image cracks relative to the image background in color and brightness, a saliency map of the image obtained by an FT algorithm is used, then an image threshold value is calculated in a self-adaptive mode by combining a mean shift algorithm (MeanShift), and finally the cracks are segmented in a binary mode. The flow chart of the fine crack identification based on the visual saliency is shown in fig. 3, and the image of the segmented crack binary region is shown in fig. 4.
4. Extracting edges; extracting crack edge pixel points by using a Canny edge detection algorithm, and smoothing the image by using a Gaussian filter; calculating the magnitude and direction of the gradient by using the finite difference of the first-order partial derivatives; carrying out non-maximum suppression on the gradient amplitude; edges are detected and connected using a dual threshold algorithm.
5. Calculating crack characteristic information; and calibrating the camera to obtain the internal and external parameters of the camera. Then calculating the width by adopting a shortest distance method according to the crack edge pixels; calculating the Euclidean distance between the starting point and the ending point of the crack as the length of the crack; and taking the included angle of the fitting straight line of the crack as the included angle of the crack.
6. Image splicing; and restoring a dam body panoramic view through image splicing so as to determine the position information of the local cracks relative to the dam body. The optional image stitching algorithm comprises SIFT, SURF and ORB matching methods.
7. Positioning and evaluating cracks; the position of the crack relative to the dam body is positioned, and operation and maintenance staff can conveniently and accurately determine the position information of the crack. And 5, determining the damage grade of the crack according to the length, width, angle and area information of the crack. And the development trend of cracks is predicted, the dam body damage work is done in time, and the accidents are prevented.
As shown in fig. 2, the present invention also provides a crack detection method, which comprises the following steps:
1) collecting a data set; dam crack data are collected by using a Zen H20T camera, the collection environment comprises cloudy days, foggy days, head-on light, backlight and the like, and the collected data set is guaranteed to contain various conditions in a general detection environment.
2) Performing expansion of the data set; the data set is extended by image rotation, brightness conversion, shearing, Gaussian noise adding and the like.
3) Labeling the data set; the data set is labeled using the LabelImg tool, the data is made into VOC format, and an xml file corresponding to the image is generated.
4) Optimizing the network; (1) aiming at the problem of inconsistent crack sizes, redesigning the prior frame size by using a K-means + + algorithm and matching the prior frame size to a corresponding characteristic layer; (2) a multispectral channel attention module is introduced into a feature extraction network, so that the network can independently learn the weight of each channel and enhance information propagation among features, thereby enhancing the distinguishing capability of the network on the foreground and the background, and randomly inputting images with different sizes in the training iteration process so as to enhance the generalization capability of the model. (3) The CIoU _ loss is adopted as a regression loss function of a target detection task, the overlapping area and the central point distance between a prediction frame and a target frame are considered in the CIoU _ loss, when the target frame wraps the prediction frame, the distance of 2 frames is directly measured, so that the information of the central point distance of the boundary frame and the scale information of the width-to-height ratio of the boundary frame are considered, meanwhile, the length-to-width ratio of the prediction frame and the target frame is also considered, the boundary regression result is better, and a loss function calculation formula is as follows
IoU is the ratio of the intersection and union of the predicted frame and the target frame, the center point of the predicted frame is denoted by b, and the center point of the target frame is denoted by bgtIndicating that ρ (·) represents the euclidean distance, c represents the diagonal distance between the intersected prediction box and the target box, which circumscribes the smallest rectangle, α is a weight coefficient,
v represents a parameter for the uniformity of the aspect ratio,
wgtand hgtRepresenting the width and height of the target box, w and h representing the width and height of the prediction box, respectively.
5) Training a model; and replacing the anchors value in the network with the anchors value obtained by clustering the crack data set, setting epochs to 10000, setting batch _ size to 64, setting the initial value of the learning rate to 0.01, training the model, minimizing the loss function, and iteratively generating a YOLOv5 crack detection model.
6) Detecting a crack region; and calling a YOLOv5 crack detection model, inputting the image to be detected into the model, and outputting the crack category and the boundary box end to end through the model.
In step 4), the network is a YOLOv5 network; in step 5), the model is a YOLOv5 model;
as shown in fig. 3, the present invention further includes a method for Meanshift crack segmentation based on saliency image detection, which includes the following steps:
1) calculating an image saliency map; and intercepting a crack region image through a crack boundary identified by a YOLOv5 model, carrying out Gaussian filtering denoising on the image, and then converting the color space of the image from RGB to Lab. The average of L, a, b is calculated for the entire picture. And calculating Euclidean distances between the L, a and b values of each pixel and the mean value of the three L, a and b values of the image according to a formula in the algorithm to obtain a saliency map. And (4) normalizing, and dividing the significant value of each pixel in the image by the maximum significant value to obtain a final significant map. The saliency of a pixel can be calculated using the following formula
S(p)＝||Iμ-Iωhc(p)||
Wherein, IμIs a feature vector of the average image, Iωhc(p) is the Lab color feature of the pixel p after Gaussian smoothing, and | | is an L2 paradigm, namely calculating the Euclidean distance between the former item and the latter item in the Lab color space;
2) dividing an original picture into K large blocks by using a Meanshift method, and calculating the average value Sk of a saliency map corresponding to each block;
3) calculating the average value S mu of the whole saliency map;
4) calculating an adaptive threshold value Ta-2S mu;
5) when Sk > Ta, the region is considered as a significance map;
6) carrying out binarization segmentation on the significant region to obtain a segmented image with refined cracks;
in step 4, the crack edge detection using the Canny algorithm specifically comprises the following steps:
4.1, eliminating noise; and convolution noise reduction is carried out by using a Gaussian smoothing filter, and noise points around the crack are eliminated. The gaussian kernel used for gaussian filtering is the product of two one-dimensional gaussians, x and y, and the standard deviation σ in both dimensions is usually the same, and is of the form:
where the noise is reduced by convolution using a gaussian smoothing filter, the selected gaussian having a size of 5The kernel is exemplified by
4.2, calculating the amplitude and the direction of the gradient; here operating according to the steps of Sobel filtering. First using a pair of convolution arrays acting in the x and y directions, respectively, i.e.
Recalculating gradient magnitude and direction
4.3, non-maximum suppression; used for excluding non-edge pixels, only some thin lines are kept;
4.4, hysteresis thresholds, i.e., hysteresis high and low thresholds. If the magnitude of the pixel location exceeds a high threshold, the pixel is retained as an edge pixel; if the magnitude of the pixel location is less than the low threshold, the pixel is excluded; if the magnitude of a pixel location is between 2 thresholds, then the pixel is retained only when connected to a pixel above the high threshold;
the invention also comprises a method for measuring the length and width of the dam crack, which comprises the following steps:
5.1, calibrating a camera to obtain internal and external parameters; after the crack image identification is completed, for the image containing the crack, in order to measure the actual size of the subsequent crack, the edge coordinates of the crack at each point in the image need to be calculated, and the actual size of the crack is calculated according to the coordinate transformation relation; the internal reference and the external reference of the camera can be directly obtained by using a Zhangyingyou calibration method. As shown in the following formula:
wherein A is an internal reference matrix of the camera, R is a rotation matrix of the external reference, and T is a translation directionAmount, α ═ f/dx, β ═ f/dy, f is the focal length, dx, dy are the width and height of the pixel, respectively; gamma represents the deviation of pixel point in x, y direction and (u)0,v0) Is a reference point;
5.2, converting pixel coordinates into world coordinates; the relationship between the pixel coordinate system and the world coordinate system is shown in the following matrix:
wherein [ X Y Z]TRepresenting the world coordinate system, [ u v ]]TRepresenting a pixel coordinate system with coordinates in the upper left corner of the image, ZCIs the corresponding Z-axis coordinate value under the camera coordinate system;
5.3, calculating the width of the crack; the method for calculating the Euclidean distance of the crack width comprises the following steps of adopting a shortest distance method, taking a vertical crack as an example, marginalizing a crack image, then taking a left edge point of each line as a center, and calculating the minimum value of the distance of a right edge point in 5 lines adjacent to the line as the crack width of the line, wherein the Euclidean distance of the crack width is calculated as follows:
wherein w is the crack width, Z under monocular cameral＝Zr0. Maximum value w of crackmax＝max(wi) Average value of cracks is
wiThe width of the ith row of the crack is shown, and N is the total row number;
5.4, calculating the length; the skeleton information obtained after thinning operation is a thin line graph with single pixel arrangement, and the principle of calculating the dam crack length information can be that the Euclidean distance between the starting point and the ending point of the thinned crack is calculated as the length information of the dam crack;
5.5, calculating an included angle; because the dam body crack graph has an irregular shape, the crack image needs to be fitted in the processing process to form a straight line capable of being calculated;
5.6, calculating the area; the minimum circumscribed rectangular area of the crack was taken as the area of the crack.
The dam crack identification and measurement method based on unmanned aerial vehicle vision comprises the steps of collecting crack images of a dam through a high-definition camera carried by an unmanned aerial vehicle, roughly identifying cracks through improved YOLOv5, segmenting crack regions through an improved significance detection algorithm, extracting edge information of the cracks through edge detection, calculating information such as the length and the width of the cracks, and finally positioning position information of the cracks relative to the dam through splicing images. The method can quickly and accurately identify the cracks of the dam body, measure the length and the width of the cracks, and improve the efficiency of dam body overhaul and damage prediction.
Claims (10)
1. A dam crack identification and measurement method based on unmanned aerial vehicle vision is characterized by comprising the following steps:
step 1: collecting images, namely collecting RGB images of the dam body to be detected, wherein the collected images can completely cover a detection area;
step 2: carrying out rough detection on the cracks;
and step 3: carrying out fine crack identification based on visual saliency;
and 4, step 4: performing edge extraction;
and 5: calculating crack characteristic information;
step 6: performing image splicing, and reducing a dam body panoramic view through image splicing so as to determine the position information of the local cracks relative to the dam body;
and 7: and (5) positioning and evaluating the cracks.
2. The method according to claim 1, wherein in step 2, a pre-trained convolutional neural network model is used to detect whether there is a crack in the image, and if no crack is detected, the next fine detection is not needed, so as to improve the algorithm efficiency; if the crack is detected, outputting a crack boundary frame in the image through the model, thereby filtering complex background interference information and improving the accuracy of subsequent crack edge extraction and the precision of crack measurement.
3. The method according to claim 1, wherein in step 3, according to the difference of the image cracks relative to the image background in color and brightness, a saliency map of the image obtained first by using FT (Frequency-tuned) saliency detection algorithm is used, then an image threshold value is calculated in an adaptive mode by combining with mean shift algorithm (Meanshift), and then the cracks are segmented by binarization.
4. The method of claim 1, wherein in step 4, Canny edge detection algorithm is used to extract crack edge pixels, gaussian filter is used to smooth the image, finite difference of first order partial derivatives is used to calculate magnitude and direction of gradient, non-maximum suppression is performed on gradient magnitude, and dual threshold algorithm is used to detect and connect edges.
5. The method according to claim 1, wherein in step 5, camera calibration is performed first, internal and external parameters of the acquisition camera are calculated for calculating the conversion from the image coordinate system to the world coordinate system, and then the width is calculated by using the shortest distance method according to the crack edge pixels; calculating the Euclidean distance between the starting point and the ending point of the crack as the length of the crack; and taking the included angle of the fitting straight line of the crack as the included angle of the crack.
6. The method of claim 1, further comprising step 7: positioning and evaluating the cracks; the position of the crack relative to the dam body is positioned, so that operation and maintenance staff can conveniently and accurately determine the position information of the crack; determining the damage grade of the crack according to the length, width, angle and area information of the crack in the step 5; and the development trend of cracks is predicted, the dam body damage work is done in time, and the accidents are prevented.
7. The method according to claim 1 or 2, characterized in that in step 2, in the course of the crack detection, the following steps are used:
2.1, collecting a data set; collecting dam body crack data, wherein the collection environment comprises cloudy days, foggy days, light facing, backlight and the like, and ensuring that the collected data set contains various conditions in a general detection environment;
2.2, expanding the data set; expanding a data set by image rotation, brightness conversion, shearing and Gaussian noise increasing;
2.3, labeling the data set; in semi-supervised learning, an image labeling tool is used for labeling a data set, the speed and the precision of feature extraction are improved, data are made into a specified format, and a specified file corresponding to an image is generated;
2.4, optimizing the network; (1) aiming at the problem of inconsistent crack sizes, redesigning the prior frame size by using a K-means + + algorithm and matching the prior frame size to a corresponding characteristic layer; (2) a multispectral channel attention module is introduced into a feature extraction network, so that the network can independently learn the weight of each channel and enhance the information propagation among features, thereby enhancing the distinguishing capability of the network on the foreground and the background, and randomly inputting images with different sizes in the training iteration process so as to enhance the generalization capability of the model; (3) the CIoU _ loss is adopted as a regression loss function of a target detection task, the overlapping area and the central point distance between a prediction frame and a target frame are considered in the CIoU _ loss, when the target frame wraps the prediction frame, the distance of 2 frames is directly measured, so that the information of the central point distance of the boundary frame and the scale information of the width-to-height ratio of the boundary frame are considered, meanwhile, the length-to-width ratio of the prediction frame and the target frame is also considered, the boundary regression result is better, and a loss function calculation formula is as follows
IoU is the ratio of the intersection and union of the predicted frame and the target frame, the center point of the predicted frame is denoted by b, and the center point of the target frame is denoted by bgtIndicating that rho (·) represents the Euclidean distance, and c represents the distance between the intersected prediction box and the target boxThe diagonal distance, which constitutes the circumscribed minimum rectangle, alpha is a weight coefficient,
v represents a parameter for the uniformity of the aspect ratio,
wgtand hgtRepresent the width and height of the target box, w and h represent the width and height of the prediction box, respectively;
2.5, training a model; the manufactured data set is brought into an optimized network for training, and the model is continuously iterated through parameter adjustment, so that the loss function of the model is reduced to the minimum; finally generating a crack detection model;
2.6, detecting a crack area; and calling a crack detection model, inputting the image to be detected into the model, and outputting the crack category and the boundary box end to end through the model.
8. The method according to claim 1 or 3, characterized in that in step 3, when carrying out fine crack identification based on visual significance, the following steps are adopted:
3.1, calculating an image saliency map; intercepting a crack area image through a crack boundary identified by a crack detection model, carrying out Gaussian filtering denoising on the image, and then converting the color space of the image from RGB to Lab; calculating the average value of L, a and b of the whole picture; calculating Euclidean distances between the L, a and b values of each pixel and the mean values of the three L, a and b values of the image according to a formula in an algorithm to obtain a saliency map; normalizing, namely dividing the significant value of each pixel in the image by the maximum significant value to obtain a final significant map; the saliency of a pixel can be calculated using the following formula
S(p)＝||Iμ-Iωhc(p)||
Wherein, IμIs a feature vector of the average image, Iωhc(p) is the Lab color feature of the pixel p after Gaussian smoothing, and | | is an L2 paradigm, namely calculating the Euclidean distance between the former item and the latter item in the Lab color space;
3.2, dividing the original picture into K large blocks by using a Meanshift method, and calculating the average value Sk of a saliency map corresponding to each divided block by using a 3.1 saliency calculation method;
3.3, calculating the average value S mu of the whole saliency map;
3.4, calculating an adaptive threshold value Ta-2S mu;
3.5, when Sk > Ta, the region is considered as a significance map;
and 3.6, carrying out binarization segmentation on the salient region to obtain a segmented image with refined cracks.
9. The method according to claim 1 or 4, wherein in step 4, the crack edge is detected by using a Canny algorithm, and the method comprises the following steps:
4.1, eliminating noise; convolution noise reduction is carried out by using a Gaussian smoothing filter, and noise points around the crack are eliminated; the gaussian kernel used for gaussian filtering is the product of two one-dimensional gaussians, x and y, and the standard deviation σ in both dimensions is usually the same, and is of the form:
convolution noise reduction using a gaussian smoothing filter;
4.2, calculating the amplitude and the direction of the gradient; operating according to the step of Sobel filtering; first using a pair of convolution arrays acting in the x and y directions, respectively, i.e.
Recalculating gradient magnitude and direction
4.3, non-maximum suppression;
4.4, hysteresis thresholds, i.e. hysteresis high and low thresholds; specifically, if the amplitude of a pixel location exceeds a high threshold, the pixel is retained as an edge pixel; if the magnitude of the pixel location is less than the low threshold, the pixel is excluded; if the magnitude of a pixel location is between 2 thresholds, then the pixel is retained only when connected to a pixel above the high threshold.
10. The method according to claim 1 or 5, wherein in step 5, when calculating the crack characteristic information, the method comprises measuring the length and width of the crack, and specifically comprises the following steps:
5.1, calibrating a camera to obtain internal and external parameters; after the crack image identification is completed, calculating the edge coordinates of the crack at each point in the image for the image containing the crack, and calculating the actual size of the crack according to the coordinate transformation relation; the internal reference and the external reference of the camera can be directly obtained by using a Zhangyingyou calibration method; as shown in the following formula
Wherein, a is an internal parameter matrix of the camera, R is a rotation matrix of the external parameter, T is a translation vector, α ═ f/dx, β ═ f/dy, f is a focal length, and dx and dy are width and height of the pixel respectively; gamma represents the deviation of pixel point in x, y direction and (u)0,v0) Is a reference point;
5.2, converting pixel coordinates into world coordinates; the relationship between the pixel coordinate system and the world coordinate system is shown in the following matrix:
wherein [ X Y Z]TRepresenting the world coordinate system, [ u v ]]TRepresenting a pixel coordinate system with coordinates in the upper left corner of the image, ZCIs the corresponding Z-axis coordinate value under the camera coordinate system;
5.3, calculating the width of the crack; adopting a shortest distance method, taking a vertical crack as an example, marginalizing a crack image, and taking a left edge point of each line as a center; calculating the minimum value of the distances of the right edge points in a plurality of rows adjacent to the row as the width of the crack of the row; the Euclidean distance calculation method of the crack width is as follows
Wherein w is the crack width, Z under monocular cameral＝Zr0; maximum value w of crackmax＝max(wi) Average value of cracks is
wiThe width of the ith row of the crack is shown, and N is the total row number;
5.4, calculating the length; the skeleton information obtained after thinning operation is a thin line graph with single pixel arrangement, and the principle of calculating the dam crack length information can be that the Euclidean distance between the starting point and the ending point of the thinned crack is calculated as the length information of the dam crack;
5.5, calculating an included angle; because the dam body crack graph has an irregular shape, the crack image needs to be fitted in the processing process to form a straight line capable of being calculated;
5.6, calculating the area; the minimum circumscribed rectangular area of the crack was taken as the area of the crack.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111575229.4A CN114331986A (en) | 2021-12-21 | 2021-12-21 | Dam crack identification and measurement method based on unmanned aerial vehicle vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111575229.4A CN114331986A (en) | 2021-12-21 | 2021-12-21 | Dam crack identification and measurement method based on unmanned aerial vehicle vision |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114331986A true CN114331986A (en) | 2022-04-12 |
Family
ID=81054464
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111575229.4A Pending CN114331986A (en) | 2021-12-21 | 2021-12-21 | Dam crack identification and measurement method based on unmanned aerial vehicle vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114331986A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114972329A (en) * | 2022-07-13 | 2022-08-30 | 江苏裕荣光电科技有限公司 | Image enhancement method and system of surface defect detector based on image processing |
| CN115272131A (en) * | 2022-08-22 | 2022-11-01 | 苏州大学 | Image Moire pattern removing system and method based on self-adaptive multi-spectral coding |
| CN116229191A (en) * | 2023-03-13 | 2023-06-06 | 东莞理工学院 | Target detection method based on normalized corner distance and target foreground information |
| CN117191011A (en) * | 2023-08-17 | 2023-12-08 | 北京自动化控制设备研究所 | Target scale recovery method based on inertial/visual information fusion |
| CN117876359A (en) * | 2024-03-04 | 2024-04-12 | 金品计算机科技(天津)有限公司 | Rubber product defect detection method, device, equipment and medium |
-
2021
- 2021-12-21 CN CN202111575229.4A patent/CN114331986A/en active Pending
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114972329A (en) * | 2022-07-13 | 2022-08-30 | 江苏裕荣光电科技有限公司 | Image enhancement method and system of surface defect detector based on image processing |
| CN114972329B (en) * | 2022-07-13 | 2022-10-21 | 江苏裕荣光电科技有限公司 | Image enhancement method and system of surface defect detector based on image processing |
| CN115272131A (en) * | 2022-08-22 | 2022-11-01 | 苏州大学 | Image Moire pattern removing system and method based on self-adaptive multi-spectral coding |
| CN116229191A (en) * | 2023-03-13 | 2023-06-06 | 东莞理工学院 | Target detection method based on normalized corner distance and target foreground information |
| CN116229191B (en) * | 2023-03-13 | 2023-08-29 | 东莞理工学院 | Target detection method based on normalized corner distance and target foreground information |
| CN117191011A (en) * | 2023-08-17 | 2023-12-08 | 北京自动化控制设备研究所 | Target scale recovery method based on inertial/visual information fusion |
| CN117876359A (en) * | 2024-03-04 | 2024-04-12 | 金品计算机科技(天津)有限公司 | Rubber product defect detection method, device, equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109948661B (en) | 3D vehicle detection method based on multi-sensor fusion | |
| CN106651872B (en) | Pavement crack identification method and system based on Prewitt operator | |
| CN114331986A (en) | Dam crack identification and measurement method based on unmanned aerial vehicle vision | |
| CN107045629B (en) | Multi-lane line detection method | |
| CN107424142B (en) | Weld joint identification method based on image significance detection | |
| CN110400322B (en) | Fruit point cloud segmentation method based on color and three-dimensional geometric information | |
| CN110120042B (en) | Crop image pest and disease damage area extraction method based on SLIC super-pixel and automatic threshold segmentation | |
| CN110516550B (en) | FPGA-based lane line real-time detection method | |
| CN110473221B (en) | Automatic target object scanning system and method | |
| CN111915704A (en) | Apple hierarchical identification method based on deep learning | |
| CN108052904B (en) | Method and device for acquiring lane line | |
| CN109993099A (en) | A kind of lane line drawing recognition methods based on machine vision | |
| CN110033431B (en) | Non-contact detection device and detection method for detecting corrosion area on surface of steel bridge | |
| CN111415363A (en) | Image edge identification method | |
| CN112330593A (en) | Building surface crack detection method based on deep learning network | |
| CN111354047B (en) | Computer vision-based camera module positioning method and system | |
| CN111461133B (en) | Express delivery surface single item name identification method, device, equipment and storage medium | |
| CN113343976B (en) | Anti-highlight interference engineering measurement mark extraction method based on color-edge fusion feature growth | |
| CN112883881B (en) | Unordered sorting method and unordered sorting device for strip-shaped agricultural products | |
| CN115797813B (en) | Water environment pollution detection method based on aerial image | |
| CN113538503A (en) | Solar panel defect detection method based on infrared image | |
| CN112861654A (en) | Famous tea picking point position information acquisition method based on machine vision | |
| CN113989604A (en) | Tire DOT information identification method based on end-to-end deep learning | |
| CN110348307B (en) | Path edge identification method and system for crane metal structure climbing robot | |
| CN115841633A (en) | Power tower and power line associated correction power tower and power line detection method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |