CN113379712A - Steel bridge bolt disease detection method and system based on computer vision - Google Patents
Steel bridge bolt disease detection method and system based on computer vision Download PDFInfo
- Publication number
- CN113379712A CN113379712A CN202110697289.7A CN202110697289A CN113379712A CN 113379712 A CN113379712 A CN 113379712A CN 202110697289 A CN202110697289 A CN 202110697289A CN 113379712 A CN113379712 A CN 113379712A
- Authority
- CN
- China
- Prior art keywords
- bolt
- image
- steel bridge
- mask
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
- G06F18/24133—Distances to prototypes
- G06F18/24137—Distances to cluster centroïds
- G06F18/2414—Smoothing the distance, e.g. radial basis function networks [RBFN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30164—Workpiece; Machine component
Abstract
The invention discloses a computer vision-based steel bridge bolt disease detection method and system, which comprises the following steps: acquiring an original steel bridge bolt image, marking the original steel bridge bolt image, and enhancing data to generate a data set; building a Mask-RCNN model and pre-training the Mask-RCNN model by using a COCO data set; training the Mask-RCNN model after pre-training by using the generated data set; correcting an original steel bridge bolt image into a front view of the bolt through perspective transformation; and (3) identifying bolt diseases: and detecting whether the bolt is fastened and falls off, and identifying and calculating a loosening angle if the bolt is loosened. The invention can realize long-distance nondestructive detection, is easy to deploy, has low cost, high speed and high precision, has low requirement on the skills of detection workers, and provides an intelligent and automatic solution for the nondestructive detection of the steel structure bridge bolt diseases.
Description
Technical Field
The invention relates to the technical field of bolt detection, in particular to a steel bridge bolt disease detection method and system based on computer vision.
Background
The steel structure bridge has the outstanding advantages of light weight, high strength, assembly construction, wide application range and the like, and is widely applied to the construction of main traffic hub projects such as large-span bridges and urban overpass bridges. The high-strength bolt is one of the connection modes commonly adopted in the current steel structure bridge, but under the action of dynamic loads such as vehicle load, wind load and the like, the bolt connection structure is easy to generate the phenomena of bolt loosening, fracture or shedding, so that the connection rigidity of the steel structure cannot be ensured, the bearing capacity of the structure is reduced, and the service safety performance of the structure is influenced.
In engineering practice, the common steel structure bridge bolt connection looseness detection technology can be divided into two types: in-situ sensing techniques and sensor-based sensing techniques. The former includes a torque tightening method, a hammering method, a strain gauge method and the like, and the latter measures related physical quantities reflecting bolt loosening based on acoustic elastic effect, guided wave, impedance and the like. However, due to the fact that the number of high-strength bolts in the large steel structure bridge is large, the operation procedure for checking bolt loosening in the mass of bolts is complex and the cost is high.
With the development and progress of computers and information technologies, monitoring and detection of steel structure bridge diseases by comprehensively using various machine vision technologies become research hotspots in the field of bridge nondestructive detection. Aiming at the problem of looseness of high-strength bolts of steel bridges, bolt looseness identification methods based on image processing technologies such as feature matching and threshold segmentation exist at present, but the methods are low in intelligent degree, often need manual intervention, and are difficult to position and identify bolt targets in images under complex background environments. In recent years, the target detection method based on deep learning has been more and more widely applied to apparent damage recognition in practical engineering due to the continuous improvement of recognition accuracy and speed. However, in the existing methods, on one hand, the requirements on image acquisition quality (angle, light and the like) are high, and the bolt loosening angle can be further judged only by comparing the front state and the rear state of the bolt, and on the other hand, other diseases such as bolt breakage or falling cannot be identified by the methods, so the methods are not suitable for detecting the bolt group diseases of large steel structure bridges.
Therefore, in order to improve the detection efficiency of bolt loosening diseases, a high-strength bolt disease detection system and method suitable for large-scale steel structure bridges are urgently needed to be provided by combining with a deep learning intelligent recognition technology.
Disclosure of Invention
The invention aims to solve the technical problems of low automation degree, complex deployment process, low detection efficiency and the like of the conventional steel structure bridge high-strength bolt looseness detection technology, and aims to provide a steel bridge bolt disease detection method and system based on computer vision to solve the problems.
The invention is realized by the following technical scheme:
a steel bridge bolt disease detection method based on computer vision comprises the following steps:
s1, image acquisition and preprocessing: collecting an original steel bridge bolt image, and carrying out target marking and classification on the original steel bridge bolt image by using a Labelme tool to obtain a classified steel bridge bolt image; generating a Mask and a JSON file according to the classified steel bridge bolt image, integrating the Mask and the JSON file with the original steel bridge bolt image into a steel bridge bolt image data set, and then performing data enhancement on the steel bridge bolt image data set to obtain an enhanced steel bridge bolt image data set;
s2, establishing a Mask-RCNN model: building a Mask-RCNN model in TensorFlow, and pre-training the Mask-RCNN model by using a COCO data set until the Mask-RCNN model converges to obtain a converged Mask-RCNN model;
s3, training a bolt disease recognition model: dividing the enhanced steel bridge bolt image data set in the S1 into a training set, a verification set and a test set; inputting the training set into the converged Mask-RCNN model for training to obtain a Mask-RCNN bolt disease identification model; and calculating a loss function of the Mask-RCNN bolt disease identification model by using the test set.
S4, correcting the original image: correcting the original steel bridge bolt image to obtain a corrected steel bridge bolt image;
s5, identifying bolt diseases: identifying the corrected steel bridge bolt image by using a Mask-RCNN bolt disease identification model to obtain a bolt Mask identification result of the corrected image; the bolt mask recognition result of the corrected image comprises a bolt fastening target, a bolt loosening target and a bolt falling target; screening out a bolt loosening target and a bolt falling target;
s6, bolt disease treatment: acquiring the position information of the bolt falling target according to the bolt falling target; obtaining position information of the bolt loosening target according to the bolt loosening target and calculating a bolt loosening angle according to the bolt loosening target; and sending the position information of the bolt falling target and the position information of the bolt loosening target to a maintainer, and sending early warning information to inform the maintainer to overhaul.
Further, in S1, the classified steel bridge bolt image includes three types: fastening a steel bridge bolt image, loosening the steel bridge bolt image and dropping the steel bridge bolt image; the data enhancement of the steel bridge bolt image dataset comprises: respectively rotating original steel bridge bolt images in the steel bridge bolt image data set by a plurality of angles of 30 degrees, 60 degrees, 90 degrees and 180 degrees in the data enhancement process, and calculating the coordinate parameters of each bolt target Mask after rotation; and after the operations are finished, integrating the Mask, the JSON file and the original steel bridge bolt image into an enhanced steel bridge bolt image data set.
Further, in S2, the establishing of the Mask-RCNN model in the tensrflow specifically includes: a Mask-RCNN model was constructed in Tensorflow based on ResNet-101.
Further, S3 specifically includes: dividing the enhanced steel bridge bolt image data set in the S1 into a training set, a verification set and a test set according to the ratio of 6:1: 3; the testing set is used for verifying the generalization capability of the converged Mask-RCNN model; inputting the training set into the converged Mask-RCNN model, and training the converged Mask-RCNN model in a combined mode to obtain a Mask-RCNN bolt disease identification model; respectively calculating loss functions of a training set and a testing set in the training process, and updating Mask-RCNN bolt disease identification model parameters by adopting a random gradient descent algorithm; the loss functions of the Mask-RCNN bolt disease recognition model in the training process comprise an RPN classification loss function, an RPN regression loss function, an MRCNN classification loss function, an MRCNN regression loss function and an MRCNN Mask loss function.
Further, in S4, the correcting the original steel bridge bolt image specifically includes: recognizing a bolt target in an original steel bridge bolt image by using a Mask-RCNN bolt disease recognition model to obtain a bolt target recognition frame, positioning coordinates of a bolt target center point, detecting an external quadrangle of the bolt center point by using an Alpha-shape algorithm, correcting the external quadrangle into a rectangle according to an actual proportion by perspective transformation, and performing bolt example segmentation on the corrected bolt image by using the Mask-RCNN bolt disease recognition model.
Further, in S4, detecting an external quadrangle of the point cloud of the center of the bolt cluster by using an Alpha-shape algorithm, where the specific calculation process is as follows: setting a judgment radius R to be 0.01, drawing a circle with the radius R through any two points of the point cloud, if no other data points exist in the circle, using the two points as boundary points of the point cloud, and calculating four points to form a quadrangle, wherein the quadrangle formed by connecting the four points is the circumscribed quadrangle of the bolt group; the external quadrangle perspective transformation in the image is a rectangle according to the arrangement quantity and the spacing of the bolts on the long edge and the wide edge of the actual external rectangle of the bolt, and the transformation process is as follows:
wherein [ x 'y' ] is the pixel coordinate before correction; [ x y ] is the pixel coordinate after rectification; m is a perspective transformation matrix; all unknowns in the M matrix can be solved by utilizing the four groups of angular point coordinates, the original image can be re-projected by utilizing the M matrix, and the projected image is the front view of the bolt image; and re-identifying the corrected bolt image by using a Mask-RCNN model to obtain a bolt Mask identification result of the corrected image.
Further, S5 includes screening out a bolt loosening target and a bolt falling target, using the position information of the bolt falling target in a disease early warning system, and performing example segmentation on the bolt loosening target to obtain a loosening bolt target front view with geometric significance; detecting the characteristic points of the scale marks in the loosened bolt target by using a self-adaptive Canny algorithm, fitting the center line of the characteristic points of the scale marks in the loosened bolt target by using Hough transformation, checking the RGB value at the position corresponding to the center line in the sub-block, and taking the straight line closest to the RGB value of the color of the scale marks as the real scale marks; if the two scale marks are on the same straight line, the bolt is not loosened, and if the two scale marks are staggered, the staggered angle is the loosening angle of the bolt.
Further, in S6, the calculating a bolt loosening angle according to the bolt loosening target specifically includes: the method comprises the steps of carrying out example segmentation on a bolt loosening target to obtain a plurality of bolt loosening sub-image blocks, detecting the bolt loosening sub-image blocks by using a self-adaptive Canny algorithm to obtain scale mark boundary characteristics of the bolt loosening sub-image blocks, detecting and identifying the scale mark boundary characteristics of the bolt loosening sub-image blocks by using Hough transformation to obtain scale mark dislocation angles, and finally calculating the bolt loosening angles according to the scale mark dislocation angles.
A steel bridge bolt disease detection system based on computer vision comprises image acquisition equipment, an image processing and identifying module, a display module and an automatic early warning module;
the image acquisition equipment is used for acquiring an original steel bridge bolt image and uploading the original steel bridge bolt image to the image processing and identifying module;
the image processing and identifying module is used for processing and identifying the original steel bridge bolt image through an algorithm to obtain a processed bolt image and a bolt mask identifying result;
the display module is used for displaying the processed bolt image and the bolt mask identification result; the display module is an interactive interface;
the automatic early warning module is used for pushing early warning information according to the bolt mask recognition result.
Further, the image acquisition equipment is unmanned aerial vehicle image acquisition equipment; acquiring an original steel bridge bolt image by using unmanned aerial vehicle image acquisition equipment, remotely connecting the unmanned aerial vehicle image acquisition equipment with a steel bridge bolt disease detection system through a wireless transmission network, uploading the original steel bridge bolt image to the system in real time, and processing the original steel bridge bolt image in real time; and displaying the processed bolt image and the bolt mask identification result on an interactive interface.
Compared with the prior art, the invention has the following advantages and beneficial effects:
the method and the system for detecting the bolt defect of the steel bridge based on the computer vision are different from the existing method for detecting the bolt looseness by machine vision, do not need to compare the front state and the rear state of the bolt, can automatically realize the image correction and finally identify the bolt defect, and have excellent practical performance on the bolt looseness of the large steel structure bridge.
The steel bridge bolt disease detection method and system based on computer vision can realize long-distance nondestructive detection, are easy to deploy, low in cost, high in speed, high in precision and low in requirement on the skills of detection workers, and provide an intelligent and automatic solution for the nondestructive detection of the steel bridge bolt disease.
According to the steel bridge bolt disease detection method and system based on computer vision, image acquisition equipment such as an unmanned aerial vehicle is remotely connected with a bolt loosening recognition system, bolt image acquisition, image correction processing, bolt target recognition and disease detection and analysis result front-end display are automatically realized, and the steel bridge bolt disease condition can be conveniently detected.
Drawings
The accompanying drawings, which are included to provide a further understanding of the embodiments of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the principles of the invention. In the drawings:
FIG. 1 is a schematic flow diagram of the present invention;
FIG. 2 is a schematic diagram of the Mask-RCNN model used in the present invention;
FIG. 3 is an original image in this embodiment;
FIG. 4 is an image after manual annotation in this embodiment;
FIG. 5 is an image after image rectification in this embodiment;
fig. 6 is a result of bolt defect recognition in this embodiment;
fig. 7 is a visualization result of the bolt loosening according to the method of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to examples and accompanying drawings, and the exemplary embodiments and descriptions thereof are only used for explaining the present invention and are not meant to limit the present invention.
Examples
As shown in fig. 1, a steel bridge bolt disease detection method based on computer vision includes the following steps:
s1, image acquisition and preprocessing: acquiring an original steel bridge bolt image, wherein the original steel bridge bolt image is shown in figure 3; performing target marking and classification on the original steel bridge bolt image by using a Labelme tool to obtain a classified steel bridge bolt image; the annotated image is shown in FIG. 4; generating a Mask and a JSON file according to the classified steel bridge bolt image, integrating the Mask and the JSON file with the original steel bridge bolt image into a steel bridge bolt image data set, and then performing data enhancement on the steel bridge bolt image data set to obtain an enhanced steel bridge bolt image data set;
the acquired original steel bridge bolt image is not limited in shooting angle, distance, light rays and the like, but the bolt and scale marks in the image are ensured to be clear and visible, the number of the bolt images is large enough, and an image library contains various bolt diseases so as to ensure the robustness of an identification model; marking the image by using a Labelme tool to generate a Mask and a JSON file, and completely framing a bolt target and a scale mark in the marking process; respectively rotating the original image by a plurality of angles of 30 degrees, 60 degrees, 90 degrees, 180 degrees and the like in the data enhancement process, and calculating the coordinate parameters of each bolt target Mask in the original image after rotating; and integrating the Mask, the JSON file and the original image into a data set after the operations are finished.
S2, establishing a Mask-RCNN model: building a Mask-RCNN model in TensorFlow, and pre-training the Mask-RCNN model by using a COCO data set until the Mask-RCNN model converges to obtain a converged Mask-RCNN model;
s3, training a bolt disease recognition model: dividing the enhanced steel bridge bolt image data set in the S1 into a training set, a verification set and a test set; inputting the training set into the converged Mask-RCNN model for training to obtain a Mask-RCNN bolt disease identification model; and calculating a loss function of the Mask-RCNN bolt disease identification model by using the test set.
S4, correcting the original image: correcting the original steel bridge bolt image to obtain a corrected steel bridge bolt image; the image after image rectification is shown in fig. 5;
s5, identifying bolt diseases: identifying the corrected steel bridge bolt image by using a Mask-RCNN bolt disease identification model to obtain a bolt Mask identification result of the corrected image; the bolt disease recognition result is shown in fig. 6; the bolt mask recognition result of the corrected image comprises a bolt fastening target, a bolt loosening target and a bolt falling target; screening out a bolt loosening target and a bolt falling target; the visualization of bolt loosening is shown in fig. 7;
s6, bolt disease treatment: acquiring the position information of the bolt falling target according to the bolt falling target; obtaining position information of the bolt loosening target according to the bolt loosening target and calculating a bolt loosening angle according to the bolt loosening target; and sending the position information of the bolt falling target and the position information of the bolt loosening target to a maintainer, and sending early warning information to inform the maintainer to overhaul.
In S1, the classified steel bridge bolt image includes three types: fastening a steel bridge bolt image, loosening the steel bridge bolt image and dropping the steel bridge bolt image; the data enhancement of the steel bridge bolt image dataset comprises: respectively rotating original steel bridge bolt images in the steel bridge bolt image data set by a plurality of angles of 30 degrees, 60 degrees, 90 degrees and 180 degrees in the data enhancement process, and calculating the coordinate parameters of each bolt target Mask after rotation; and after the operations are finished, integrating the Mask, the JSON file and the original steel bridge bolt image into an enhanced steel bridge bolt image data set.
In S2, the establishing of the Mask-RCNN model in TensorFlow specifically comprises: a Mask-RCNN model was constructed in Tensorflow based on ResNet-101. Constructing a Mask-RCNN model in Tensorflow on the basis of ResNet-101, wherein the structure of the ResNet-101 model is shown in Table 1, and the Mask-RCNN model is shown in FIG. 2; carrying out convolution pooling on an original steel bridge bolt image by using a Backbone, carrying out feature extraction by using FPN to generate 5 pyramid Featuremas, respectively adopting the pixel scale of (32/64/128/256/512) and the rectangular frame length-width ratio of (0.5/1/2) to generate a plurality of Anchors for each Featurema in the pyramid, combining the Anchors with Box delta data corresponding to the Anchors to generate a final RPNbox, and only retaining 1000-2000 RPNboxes after carrying out non-maximum suppression processing; distinguishing positive and negative samples according to the IOU value, calculating the difference between the positive sample and a real frame and a Mask value to be predicted, and using the values to calculate a final damage function; after a Mask-RCNN model is built, a COCO data set is used for pre-training the Mask-RCNN model until the Mask-RCNN model converges, and pre-trained network parameters are stored, so that transfer learning is conveniently carried out on bolt disease identification in the follow-up process.
TABLE 1 ResNet-101 architecture
S3 specifically includes: dividing the enhanced steel bridge bolt image data set in the S1 into a training set, a verification set and a test set according to the ratio of 6:1: 3; the testing set is used for verifying the generalization capability of the converged Mask-RCNN model; inputting the training set into the converged Mask-RCNN model, and training the converged Mask-RCNN model in a combined mode to obtain a Mask-RCNN bolt disease identification model; respectively calculating loss functions of a training set and a testing set in the training process, and updating Mask-RCNN bolt disease identification model parameters by adopting a random gradient descent algorithm; the loss functions of the Mask-RCNN bolt disease recognition model in the training process comprise an RPN classification loss function, an RPN regression loss function, an MRCNN classification loss function, an MRCNN regression loss function and an MRCNN Mask loss function.
In S4, the correcting the original steel bridge bolt image specifically includes: recognizing a bolt target in an original steel bridge bolt image by using a Mask-RCNN bolt disease recognition model to obtain a bolt target recognition frame, positioning coordinates of a bolt target center point, detecting an external quadrangle of the bolt center point by using an Alpha-shape algorithm, correcting the external quadrangle into a rectangle according to an actual proportion by perspective transformation, and performing bolt example segmentation on the corrected bolt image by using the Mask-RCNN bolt disease recognition model.
S4, detecting the circumscribed quadrangle of the point cloud of the center of the bolt cluster by using an Alpha-shape algorithm, wherein the specific calculation process is as follows: setting a judgment radius R to be 0.01, drawing a circle with the radius R through any two points of the point cloud, if no other data points exist in the circle, using the two points as boundary points of the point cloud, and calculating four points to form a quadrangle, wherein the quadrangle formed by connecting the four points is the circumscribed quadrangle of the bolt group; the external quadrangle perspective transformation in the image is a rectangle according to the arrangement quantity and the spacing of the bolts on the long edge and the wide edge of the actual external rectangle of the bolt, and the transformation process is as follows:
wherein [ x 'y' ] is the pixel coordinate before correction; [ x y ] is the pixel coordinate after rectification; m is a perspective transformation matrix; all unknowns in the M matrix can be solved by utilizing the four groups of angular point coordinates, the original image can be re-projected by utilizing the M matrix, and the projected image is the front view of the bolt image; and re-identifying the corrected bolt image by using a Mask-RCNN model to obtain a bolt Mask identification result of the corrected image.
S5, screening out a bolt loosening target and a bolt falling target, using the position information of the bolt falling target in a disease early warning system, and carrying out example segmentation on the bolt loosening target to obtain a loosening bolt target front view with geometric significance; detecting the characteristic points of the scale marks in the loosened bolt target by using a self-adaptive Canny algorithm, fitting the center line of the characteristic points of the scale marks in the loosened bolt target by using Hough transformation, checking the RGB value at the position corresponding to the center line in the sub-block, and taking the straight line closest to the RGB value of the color of the scale marks as the real scale marks; if the two scale marks are on the same straight line, the bolt is not loosened, and if the two scale marks are staggered, the staggered angle is the loosening angle of the bolt.
In S6, the calculating the bolt loosening angle according to the bolt loosening target specifically includes: the method comprises the steps of carrying out example segmentation on a bolt loosening target to obtain a plurality of bolt loosening sub-image blocks, detecting the bolt loosening sub-image blocks by using a self-adaptive Canny algorithm to obtain scale mark boundary characteristics of the bolt loosening sub-image blocks, detecting and identifying the scale mark boundary characteristics of the bolt loosening sub-image blocks by using Hough transformation to obtain scale mark dislocation angles, and finally calculating the bolt loosening angles according to the scale mark dislocation angles.
A steel bridge bolt disease detection system based on computer vision comprises image acquisition equipment, an image processing and identifying module, a display module and an automatic early warning module;
the image acquisition equipment is used for acquiring an original steel bridge bolt image and uploading the original steel bridge bolt image to the image processing and identifying module;
the image processing and identifying module is used for processing and identifying the original steel bridge bolt image through an algorithm to obtain a processed bolt image and a bolt mask identifying result;
the display module is used for displaying the processed bolt image and the bolt mask identification result; the display module is an interactive interface;
the automatic early warning module is used for pushing early warning information according to the bolt mask recognition result.
The image acquisition equipment is unmanned aerial vehicle image acquisition equipment; acquiring an original steel bridge bolt image by using unmanned aerial vehicle image acquisition equipment, remotely connecting the unmanned aerial vehicle image acquisition equipment with a steel bridge bolt disease detection system through a wireless transmission network, uploading the original steel bridge bolt image to the system in real time, and processing the original steel bridge bolt image in real time; and displaying the processed bolt image and the bolt mask identification result on an interactive interface.
The method can automatically perform bolt target detection and disease classification on the original steel bridge bolt image; remotely connecting the unmanned aerial vehicle image acquisition equipment with a bolt disease identification system through a wireless transmission network, uploading a bolt image to the system in real time, and processing the image in real time; and displaying the processed bolt image and result on an interactive interface, wherein the interface comprises a menu bar, a status bar and a tool bar, so that the interface can realize the functions of image importing, correction and storage of bolt disease identification results, automatic early warning and the like.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are merely exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A steel bridge bolt disease detection method based on computer vision is characterized by comprising the following steps:
s1, image acquisition and preprocessing: collecting an original steel bridge bolt image, and carrying out target marking and classification on the original steel bridge bolt image by using a Labelme tool to obtain a classified steel bridge bolt image; generating a Mask and a JSON file according to the classified steel bridge bolt image, integrating the Mask and the JSON file with the original steel bridge bolt image into a steel bridge bolt image data set, and then performing data enhancement on the steel bridge bolt image data set to obtain an enhanced steel bridge bolt image data set;
s2, establishing a Mask-RCNN model: building a Mask-RCNN model in TensorFlow, and pre-training the Mask-RCNN model by using a COCO data set until the Mask-RCNN model converges to obtain a converged Mask-RCNN model;
s3, training a bolt disease recognition model: dividing the enhanced steel bridge bolt image data set in the S1 into a training set, a verification set and a test set; inputting the training set into the converged Mask-RCNN model for training to obtain a Mask-RCNN bolt disease identification model; calculating a loss function of the Mask-RCNN bolt disease identification model by using the test set;
s4, correcting the original image: correcting the original steel bridge bolt image to obtain a corrected steel bridge bolt image;
s5, identifying bolt diseases: identifying the corrected steel bridge bolt image by using a Mask-RCNN bolt disease identification model to obtain a bolt Mask identification result of the corrected image; the bolt mask recognition result of the corrected image comprises a bolt fastening target, a bolt loosening target and a bolt falling target; screening out a bolt loosening target and a bolt falling target;
s6, bolt disease treatment: acquiring the position information of the bolt falling target according to the bolt falling target; and obtaining the position information of the bolt loosening target according to the bolt loosening target and calculating the bolt loosening angle according to the bolt loosening target.
2. The computer vision-based steel bridge bolt disease detection method of claim 1, wherein in S1, the classified steel bridge bolt image includes three types: fastening a steel bridge bolt image, loosening the steel bridge bolt image and dropping the steel bridge bolt image; the data enhancement of the steel bridge bolt image dataset comprises: respectively rotating original steel bridge bolt images in the steel bridge bolt image data set by a plurality of angles of 30 degrees, 60 degrees, 90 degrees and 180 degrees in the data enhancement process, and calculating the coordinate parameters of each bolt target Mask after rotation; and after the operations are finished, integrating the Mask, the JSON file and the original steel bridge bolt image into an enhanced steel bridge bolt image data set.
3. The method for detecting the steel bridge bolt disease based on the computer vision of claim 1, wherein in S2, the establishing of the Mask-RCNN model in the tensrflow specifically comprises: a Mask-RCNN model was constructed in Tensorflow based on ResNet-101.
4. The steel bridge bolt disease detection method based on computer vision of claim 1, wherein S3 specifically comprises: dividing the enhanced steel bridge bolt image data set in the S1 into a training set, a verification set and a test set according to the ratio of 6:1: 3; the testing set is used for verifying the generalization capability of the converged Mask-RCNN model; inputting the training set into the converged Mask-RCNN model, and training the converged Mask-RCNN model in a combined mode to obtain a Mask-RCNN bolt disease identification model; respectively calculating loss functions of a training set and a testing set in the training process, and updating Mask-RCNN bolt disease identification model parameters by adopting a random gradient descent algorithm; the loss functions of the Mask-RCNN bolt disease recognition model in the training process comprise an RPN classification loss function, an RPN regression loss function, an MRCNN classification loss function, an MRCNN regression loss function and an MRCNN Mask loss function.
5. The method for detecting the disease of the steel bridge bolt based on the computer vision of claim 1, wherein in the step S4, the correcting the original steel bridge bolt image is specifically as follows: recognizing a bolt target in an original steel bridge bolt image by using a Mask-RCNN bolt disease recognition model to obtain a bolt target recognition frame, positioning coordinates of a bolt target center point, detecting an external quadrangle of the bolt center point by using an Alpha-shape algorithm, correcting the external quadrangle into a rectangle according to an actual proportion by perspective transformation, and performing bolt example segmentation on the corrected bolt image by using the Mask-RCNN bolt disease recognition model.
6. The steel bridge bolt disease detection method based on computer vision of claim 5, characterized in that in S4, further comprising detecting an external quadrangle of a point cloud of a bolt cluster center point by using an Alpha-shape algorithm, the specific calculation process is as follows: setting a judgment radius R to be 0.01, drawing a circle with the radius R through any two points of the point cloud, if no other data points exist in the circle, using the two points as boundary points of the point cloud, and calculating four points to form a quadrangle, wherein the quadrangle formed by connecting the four points is the circumscribed quadrangle of the bolt group; the external quadrangle perspective transformation in the image is a rectangle according to the arrangement quantity and the spacing of the bolts on the long edge and the wide edge of the actual external rectangle of the bolt, and the transformation process is as follows:
wherein [ x 'y' ] is the pixel coordinate before correction; [ x y ] is the pixel coordinate after rectification; m is a perspective transformation matrix; all unknowns in the M matrix can be solved by utilizing the four groups of angular point coordinates, the original image can be re-projected by utilizing the M matrix, and the projected image is the front view of the bolt image; and re-identifying the corrected bolt image by using a Mask-RCNN model to obtain a bolt Mask identification result of the corrected image.
7. The steel bridge bolt disease detection method based on computer vision of claim 1, wherein S5 further comprises screening a bolt loosening target and a bolt falling target, using position information of the bolt falling target in a disease early warning system, and performing instance segmentation on the bolt loosening target to obtain a front view of the bolt loosening target with geometric significance; detecting the characteristic points of the scale marks in the loosened bolt target by using a self-adaptive Canny algorithm, fitting the center line of the characteristic points of the scale marks in the loosened bolt target by using Hough transformation, checking the RGB value at the position corresponding to the center line in the sub-block, and taking the straight line closest to the RGB value of the color of the scale marks as the real scale marks; if the two scale marks are on the same straight line, the bolt is not loosened, and if the two scale marks are staggered, the staggered angle is the loosening angle of the bolt.
8. The method for detecting the steel bridge bolt damage based on the computer vision of claim 1, wherein in S6, the calculating the bolt loosening angle according to the bolt loosening target specifically comprises: the method comprises the steps of carrying out example segmentation on a bolt loosening target to obtain a plurality of bolt loosening sub-image blocks, detecting the bolt loosening sub-image blocks by using a self-adaptive Canny algorithm to obtain scale mark boundary characteristics of the bolt loosening sub-image blocks, detecting and identifying the scale mark boundary characteristics of the bolt loosening sub-image blocks by using Hough transformation to obtain scale mark dislocation angles, and finally calculating the bolt loosening angles according to the scale mark dislocation angles.
9. A steel bridge bolt disease detection system based on computer vision is based on the steel bridge bolt disease detection method based on computer vision of any one of claims 1-8, and is characterized by comprising image acquisition equipment, an image processing and recognition module, a display module and an automatic early warning module;
the image acquisition equipment is used for acquiring an original steel bridge bolt image and uploading the original steel bridge bolt image to the image processing and identifying module;
the image processing and identifying module is used for processing and identifying the original steel bridge bolt image through an algorithm to obtain a processed bolt image and a bolt mask identifying result;
the display module is used for displaying the processed bolt image and the bolt mask identification result; the display module is an interactive interface;
the automatic early warning module is used for pushing early warning information according to the bolt mask recognition result.
10. The computer vision-based steel bridge bolt disease detection system as claimed in claim 9, wherein the image acquisition device is an unmanned aerial vehicle image acquisition device; acquiring an original steel bridge bolt image by using unmanned aerial vehicle image acquisition equipment, remotely connecting the unmanned aerial vehicle image acquisition equipment with a steel bridge bolt disease detection system through a wireless transmission network, uploading the original steel bridge bolt image to the system in real time, and processing the original steel bridge bolt image in real time; and displaying the processed bolt image and the bolt mask identification result on an interactive interface.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110697289.7A CN113379712B (en) | 2021-06-23 | 2021-06-23 | Steel bridge bolt disease detection method and system based on computer vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110697289.7A CN113379712B (en) | 2021-06-23 | 2021-06-23 | Steel bridge bolt disease detection method and system based on computer vision |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113379712A true CN113379712A (en) | 2021-09-10 |
| CN113379712B CN113379712B (en) | 2022-07-29 |
Family
ID=77578563
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110697289.7A Active CN113379712B (en) | 2021-06-23 | 2021-06-23 | Steel bridge bolt disease detection method and system based on computer vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113379712B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114407079A (en) * | 2022-01-24 | 2022-04-29 | 湖南中南智能装备有限公司 | Method for controlling installation of bolt of mechanical arm |
| CN114492146A (en) * | 2022-04-02 | 2022-05-13 | 武汉地震工程研究院有限公司 | Bolt group loosening positioning and quantitative analysis method and system based on transfer learning |
| CN114742820A (en) * | 2022-05-11 | 2022-07-12 | 西南交通大学 | Bolt looseness detection method and system based on deep learning and storage medium |
| CN114820620A (en) * | 2022-06-29 | 2022-07-29 | 中冶建筑研究总院（深圳）有限公司 | Bolt loosening defect detection method, system and device |
| CN115937165A (en) * | 2022-12-19 | 2023-04-07 | 中交公路长大桥建设国家工程研究中心有限公司 | Mobile bolt loosening intelligent detection and identification method and system |
| CN116894775A (en) * | 2023-09-11 | 2023-10-17 | 中铁大桥局集团第二工程有限公司 | Bolt image preprocessing method based on camera motion model recovery and super-resolution |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109670503A (en) * | 2018-12-19 | 2019-04-23 | 北京旷视科技有限公司 | Label detection method, apparatus and electronic system |
| CN110031476A (en) * | 2019-03-27 | 2019-07-19 | 仲恺农业工程学院 | A kind of multi-view stereo vision detection device and method that bridge pedestal nut-screw is loosened or ruptured |
| CN110097536A (en) * | 2019-04-10 | 2019-08-06 | 东南大学 | Hexagon bolt looseness detection method based on deep learning and Hough transformation |
| US20190257706A1 (en) * | 2018-02-22 | 2019-08-22 | Toyota Jidosha Kabushiki Kaisha | Axial-force measurement method, tightening determination method, axial-force measurement apparatus, and tightening determination apparatus for bolt |
| CN110570472A (en) * | 2018-06-05 | 2019-12-13 | 成都精工华耀科技有限公司 | Fastener loosening detection method based on artificial marking image |
| CN110852213A (en) * | 2019-10-30 | 2020-02-28 | 天津大学 | Template matching-based pointer instrument multi-condition automatic reading method |
| CN111002259A (en) * | 2019-10-23 | 2020-04-14 | 武汉理工大学 | Rail nut loosening detection and maintenance device and method based on CCD camera |
| CN111122140A (en) * | 2020-01-08 | 2020-05-08 | 国网湖北省电力有限公司电力科学研究院 | Device and method for diagnosing looseness of bolt of connecting node of pole material of power transmission tower |
| CN111380676A (en) * | 2020-04-23 | 2020-07-07 | 中国铁建重工集团股份有限公司 | Bolt looseness warning system and detection system for track fastener bolt |
| CN111461114A (en) * | 2020-03-03 | 2020-07-28 | 华南理工大学 | Multi-scale feature pyramid text detection method based on segmentation |
| CN111915561A (en) * | 2020-06-30 | 2020-11-10 | 西安理工大学 | Bolt state monitoring method based on image recognition and machine learning |
| CN112419297A (en) * | 2020-12-04 | 2021-02-26 | 中冶建筑研究总院（深圳）有限公司 | Bolt looseness detection method, device, equipment and storage medium |
-
2021
- 2021-06-23 CN CN202110697289.7A patent/CN113379712B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190257706A1 (en) * | 2018-02-22 | 2019-08-22 | Toyota Jidosha Kabushiki Kaisha | Axial-force measurement method, tightening determination method, axial-force measurement apparatus, and tightening determination apparatus for bolt |
| CN110570472A (en) * | 2018-06-05 | 2019-12-13 | 成都精工华耀科技有限公司 | Fastener loosening detection method based on artificial marking image |
| CN109670503A (en) * | 2018-12-19 | 2019-04-23 | 北京旷视科技有限公司 | Label detection method, apparatus and electronic system |
| CN110031476A (en) * | 2019-03-27 | 2019-07-19 | 仲恺农业工程学院 | A kind of multi-view stereo vision detection device and method that bridge pedestal nut-screw is loosened or ruptured |
| CN110097536A (en) * | 2019-04-10 | 2019-08-06 | 东南大学 | Hexagon bolt looseness detection method based on deep learning and Hough transformation |
| CN111002259A (en) * | 2019-10-23 | 2020-04-14 | 武汉理工大学 | Rail nut loosening detection and maintenance device and method based on CCD camera |
| CN110852213A (en) * | 2019-10-30 | 2020-02-28 | 天津大学 | Template matching-based pointer instrument multi-condition automatic reading method |
| CN111122140A (en) * | 2020-01-08 | 2020-05-08 | 国网湖北省电力有限公司电力科学研究院 | Device and method for diagnosing looseness of bolt of connecting node of pole material of power transmission tower |
| CN111461114A (en) * | 2020-03-03 | 2020-07-28 | 华南理工大学 | Multi-scale feature pyramid text detection method based on segmentation |
| CN111380676A (en) * | 2020-04-23 | 2020-07-07 | 中国铁建重工集团股份有限公司 | Bolt looseness warning system and detection system for track fastener bolt |
| CN111915561A (en) * | 2020-06-30 | 2020-11-10 | 西安理工大学 | Bolt state monitoring method based on image recognition and machine learning |
| CN112419297A (en) * | 2020-12-04 | 2021-02-26 | 中冶建筑研究总院（深圳）有限公司 | Bolt looseness detection method, device, equipment and storage medium |
Non-Patent Citations (7)
| Title |
|---|
| CHENG YUAN ET AL: "Near real-ti me bolt-l oosening detect ion using mask and region-based conv olutional neur al network", 《HTTPS://ONLINELIBRARY.WILEY.COM/DOI/10.1002/STC.2741》 * |
| LING XIAO ET AL: "Missing Small Fastener Detection", 《IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT》 * |
| QIAN ZHANG ET AL: "Research on segmentation of key parts for train safety based on deep learning", 《ELEVENTH INTERNATIONAL CONFERENCE ON INFORMATION OPTICS AND》 * |
| 吴杰: "桥梁结构损伤识别若干问题研究", 《中国博士学位论文全文数据库 (工程科技Ⅱ辑)》 * |
| 张阳: "建筑结构施工安全智能化监测关键技术研究", 《万方数据库》 * |
| 沈浩等: "基于深度学习的钢桁架桥螺栓病害智能识别方法", 《南京工业大学学报(自然科学版)》 * |
| 陈健雄: "基于实例分割的中低速磁浮接触轨紧固件松动识别", 《中国优秀硕士学位论文全文数据库 工程科技Ⅱ辑》 * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114407079A (en) * | 2022-01-24 | 2022-04-29 | 湖南中南智能装备有限公司 | Method for controlling installation of bolt of mechanical arm |
| CN114407079B (en) * | 2022-01-24 | 2024-01-30 | 湖南中南智能装备有限公司 | Method for controlling mechanical arm mounting bolt |
| CN114492146A (en) * | 2022-04-02 | 2022-05-13 | 武汉地震工程研究院有限公司 | Bolt group loosening positioning and quantitative analysis method and system based on transfer learning |
| CN114492146B (en) * | 2022-04-02 | 2022-07-08 | 武汉地震工程研究院有限公司 | Bolt group loosening positioning and quantitative analysis method and system based on transfer learning |
| CN114742820A (en) * | 2022-05-11 | 2022-07-12 | 西南交通大学 | Bolt looseness detection method and system based on deep learning and storage medium |
| CN114820620A (en) * | 2022-06-29 | 2022-07-29 | 中冶建筑研究总院（深圳）有限公司 | Bolt loosening defect detection method, system and device |
| CN115937165A (en) * | 2022-12-19 | 2023-04-07 | 中交公路长大桥建设国家工程研究中心有限公司 | Mobile bolt loosening intelligent detection and identification method and system |
| CN115937165B (en) * | 2022-12-19 | 2023-08-25 | 中交公路长大桥建设国家工程研究中心有限公司 | Intelligent detection and identification method and system for loosening of movable bolt |
| CN116894775A (en) * | 2023-09-11 | 2023-10-17 | 中铁大桥局集团第二工程有限公司 | Bolt image preprocessing method based on camera motion model recovery and super-resolution |
| CN116894775B (en) * | 2023-09-11 | 2024-01-23 | 中铁大桥局集团第二工程有限公司 | Bolt image preprocessing method based on camera motion model recovery and super-resolution |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113379712B (en) | 2022-07-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113379712B (en) | Steel bridge bolt disease detection method and system based on computer vision | |
| CN112818988B (en) | Automatic identification reading method and system for pointer instrument | |
| CN110910350B (en) | Nut loosening detection method for wind power tower cylinder | |
| CN104123542A (en) | Device and method for positioning wheel hub work piece | |
| CN105335973A (en) | Visual processing method for strip steel processing production line | |
| CN105373135A (en) | Method and system for guiding airplane docking and identifying airplane type based on machine vision | |
| CN102788572B (en) | Method, device and system for measuring attitude of lifting hook of engineering machinery | |
| CN112419299A (en) | Bolt loss detection method, device, equipment and storage medium | |
| CN115993096A (en) | High-rise building deformation measuring method | |
| CN115578315A (en) | Bridge strain close-range photogrammetry method based on unmanned aerial vehicle image | |
| CN113688817A (en) | Instrument identification method and system for automatic inspection | |
| CN115171045A (en) | YOLO-based power grid operation field violation identification method and terminal | |
| CN115331086A (en) | Brake shoe breaking and rivet losing fault detection method | |
| CN113706496B (en) | Aircraft structure crack detection method based on deep learning model | |
| CN113870326B (en) | Structural damage mapping, quantifying and visualizing method based on image and three-dimensional point cloud registration | |
| CN107767366B (en) | A kind of transmission line of electricity approximating method and device | |
| CN111597904B (en) | Identification method for inclination of tunnel cable bracket | |
| CN116524017B (en) | Underground detection, identification and positioning system for mine | |
| CN112184903A (en) | Method, device, equipment and medium for detecting high-voltage line tree obstacle risk points | |
| JP2011257244A (en) | Body shape detection device and method | |
| CN115597494B (en) | Precision detection method and system for prefabricated part preformed hole based on point cloud | |
| CN116448764A (en) | Automatic crack detection method for fatigue test of aircraft structure | |
| CN116051808A (en) | YOLOv 5-based lightweight part identification and positioning method | |
| CN115389516A (en) | Pavement disease de-weighting method based on high-precision positioning and deep neural network | |
| CN111815560A (en) | Photovoltaic power station fault detection method and device, portable detection equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |