CN115482207A - Bolt looseness detection method and system - Google Patents
Bolt looseness detection method and system Download PDFInfo
- Publication number
- CN115482207A CN115482207A CN202211067640.5A CN202211067640A CN115482207A CN 115482207 A CN115482207 A CN 115482207A CN 202211067640 A CN202211067640 A CN 202211067640A CN 115482207 A CN115482207 A CN 115482207A
- Authority
- CN
- China
- Prior art keywords
- bolt
- calibration line
- image
- partial contour
- bolts
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 90
- 238000013135 deep learning Methods 0.000 claims abstract description 13
- 238000000034 method Methods 0.000 claims description 24
- 238000012937 correction Methods 0.000 claims description 11
- 238000000605 extraction Methods 0.000 claims description 10
- 230000011218 segmentation Effects 0.000 claims description 9
- 239000000284 extract Substances 0.000 claims description 4
- 238000012549 training Methods 0.000 description 16
- 238000001914 filtration Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000002093 peripheral effect Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000001228 spectrum Methods 0.000 description 4
- 238000003709 image segmentation Methods 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000007850 degeneration Effects 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 230000001629 suppression Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 101100127891 Caenorhabditis elegans let-4 gene Proteins 0.000 description 1
- 208000003464 asthenopia Diseases 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Quality & Reliability (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a bolt looseness detection method, which comprises the following steps: 100: setting a detection area, and marking out a calibration line on each bolt of the detection area and a connecting piece connected with the bolt, wherein a first part of the calibration line is positioned on the bolt, and a second part of the calibration line is positioned on the connecting piece; 200: acquiring images of all bolts and connecting pieces connected with the bolts in the detection area range; 300: identifying and segmenting the image by adopting a deep learning Yolo model to obtain a plurality of segmented images, wherein a single bolt, a connecting piece corresponding to the single bolt and a corresponding calibration line are displayed in the segmented images; 400: extracting a first partial contour of the calibration line and a second partial contour of the calibration line from each segmented image; 500: and judging whether the bolt is loosened or not based on whether the angular deviation between the first partial contour of the calibration line and the second partial contour of the calibration line exceeds a set threshold value or not. Correspondingly, the invention also discloses a bolt looseness detection system.
Description
Technical Field
The present disclosure relates to inspection methods and systems, and particularly to a method and system for inspecting a fastener installation status.
Background
Bolts have found widespread use in manufacturing as a connector. The tightening condition of the bolts directly affects the performance of the equipment and can have catastrophic consequences if a problem occurs.
The traditional bolt inspection method mainly comprises the following steps: after the bolt is screwed for the first time, the state of screwing the bolt and the periphery by using the line mark is checked manually, whether the marked line of the bolt and the periphery is aligned or not is checked, and if the marked line of the bolt and the periphery is not aligned, the bolt is not loosened. The method has the disadvantages of large workload and low efficiency, and visual fatigue can be generated after workers check a large number of bolts, so that false detection and missed detection are caused.
Based on this, it is desirable to obtain a novel bolt looseness detection method that can improve the detection speed and the detection accuracy as compared with the conventional manual detection method.
Disclosure of Invention
One of the objectives of the present invention is to provide a bolt loosening detection method, which can get rid of the dependence on manual detection, and quickly and accurately identify the bolt state according to the processing and analysis of the visual image, so as to meet the requirements of industrial automation.
In order to achieve the above object, the present invention provides a bolt looseness detecting method, which includes the steps of:
100: setting a detection area, and marking a calibration line on each bolt of the detection area and a connecting piece connected with the bolt, wherein a first part of the calibration line is positioned on the bolt, and a second part of the calibration line is positioned on the connecting piece;
200: acquiring images of all bolts and connecting pieces connected with the bolts in the detection area range;
300: identifying and segmenting the image by adopting a deep learning Yolo model to obtain a plurality of segmented images, wherein a single bolt, a connecting piece corresponding to the single bolt and a corresponding calibration line are displayed in the segmented images;
400: extracting a first partial contour of the calibration line and a second partial contour of the calibration line from each segmented image;
500: and judging whether the bolt is loosened or not based on whether the angular deviation between the first partial contour of the calibration line and the second partial contour of the calibration line exceeds a set threshold value or not.
It can be seen that the concept of the present invention is: firstly, acquiring a detection image or video through an image acquisition device; then training a bolt recognition model through a deep learning Yolo algorithm, and then performing bolt segmentation according to the model; then identifying and extracting the outline of the divided calibration line of the bolt so as to respectively extract a first part positioned on the bolt and a second part positioned on the connecting piece; and judging whether the bolt is loosened or not by the included angle between the first part and the second part.
In some embodiments of the invention, the images of the bolts and the connecting pieces connected thereto within the detection area may be acquired by a movable camera, which may be realized by mounting the camera on a movable support. For example, by controlling the support to travel at a constant speed, images of all bolts and peripheral connecting pieces thereof in a certain area range can be acquired.
As described above, the image acquired in the detection area range of the present invention covers a plurality of bolts, and in order to improve the bolt looseness detection accuracy, the plurality of bolts in the image need to be divided into individual bolts.
In order to obtain a better segmentation effect, the scheme adopts a deep learning method. The deep learning Yolo algorithm is a one-stage algorithm, all the bounding boxes can be predicted only by sending the algorithm into a network once, and the algorithm is high in detection speed.
In some embodiments of the present invention, the bolt segmentation is performed based on the Yolov5 algorithm (or called Yolov5 model, yolov5 target detection network).
The Yolov5 target detection network is divided into four parts, namely an input end, a backhaul, a Neck and a Prediction. The input end carries out preprocessing operation on the input image, for example, the input image is sent to a Yolov5 target detection network after self-adaptive image scaling. The Backbone uses CSP (Cross Stage Partial), focus and SPP (Spatial Pyramid) network structures, integrates image gradient change into a characteristic diagram, and can reduce the parameter quantity and the operation quantity of the model, thereby realizing that the detection precision and speed can be still ensured under the condition of reducing the scale of the model. The Neck uses a network structure combining PAN and FPN to transfer image features to a Prediction layer, and can accurately reserve the spatial information of an image. The Prediction adopts GIOU _ Loss as a Loss function of the bounding box, and the NMS performs non-maximum suppression processing on the detection frame to obtain an optimal target frame. The CSP1_ X and CSP2_ X structures in the CSP are respectively applied to the backhaul and the hack. The SPP respectively adopts the maximal pooling of 5, 9 and 13, so that the problems of distortion and the like caused by operations such as image scaling, clipping and the like can be solved, and the problem of repeated extraction of image features by a network structure can also be solved.
Furthermore, in some embodiments of the present disclosure, the Yolov5s with the fastest recognition speed and higher accuracy is selected as the Yolo model.
In the identification process, a large number of acquired bolt images are divided into a training set, a verification set and a test set. The training set for model training can adopt a PASCAL VOC format, and the data set can be generated by labeling pictures through a LabelImg tool, only contains 1 category of 'bolt' and represents a bolt. In the training process, the upper limit of model recognition is approached by setting a training hyper-parameter of a bolt recognition model. Thus, the model can be trained according to the network structure, the hyper-parameters and the data set. In some embodiments, in order to increase the detection speed, the graphics card GPU is used for model training. Through model training, a trained Yolo model can be finally obtained to serve as a bolt recognition model.
After the bolt recognition model is obtained, the model can be deployed. And after the model is deployed, real-time bolt recognition can be carried out on the image. Meanwhile, a coordinate point (x) corresponding to the upper left corner of the square frame can be obtained left-top ,y left-top ) Coordinate point (x) of lower right corner right-bottom ,y right-bottom ) Identification type ClassID and probability Score. The result can be used for image segmentation of the bolt.
The segmented image obtained by the invention comprises a single bolt and a peripheral connecting piece thereof and also comprises a corresponding calibration line, so that the image segmented area is larger than the area identified by Yolov 5. When the image is segmented, the range of the frame of the Yolov5 model is expanded, and the segmented image contains the calibration line of the bolt, so that the later-stage bolt looseness detection is facilitated.
Further, the method for detecting bolt loosening according to the present invention further includes, between step 200 and step 300, the steps of: and carrying out shake correction on the image.
Further, a wiener filter algorithm may be employed to shake correct the image.
The basic principle of wiener filtering for image de-blurring is shown in formula (1):
in which H (u, v) is a degeneration function, S η (u,v)＝|N(u,v)| 2 Power spectrum, S, representing noise f (u,v)＝|F(u,v)| 2 Representing the power spectrum of the non-degraded image, G (u, v) representing the fourier transform of the degraded image,
representing the de-blurred picture.
Further, in order to facilitate the identification of the calibration line, in step 100 of the present invention, the calibration line is red.
Further, in step 400 of the bolt looseness detecting method according to the present invention: converting the segmentation image from the RGB image into an HSV image to extract a red region; and converting the red area into a binary image to extract a first partial contour of the calibration line and a second partial contour of the calibration line.
In this embodiment, the extraction of the red region by the present invention is performed under HSV space. In this way, the RGB image is first converted into the HSV image, and the red color has a certain range in three channels of H, S, and V, and thus can be easily recognized. Then, the present invention converts the identified red region into a binary image for subsequent contour detection.
Further, in step 500 of the bolt looseness detecting method of the present invention:
determining the minimum bounding rectangle of the first part outline of the calibration line and the second part outline of the calibration line;
calculating the coordinates of the central points of the minimum external rectangles through the vertex coordinates of the minimum external rectangles so as to determine the central points of the two minimum external rectangles;
calculating to obtain the included angle theta between the connecting line of the two central points and the X axis 1 ；
Calculating to obtain an included angle theta between the long side of the minimum circumscribed rectangle of the first outline part and the X axis 2 ；
Will theta 2 And theta 1 The difference in the values is compared with a set threshold value, and if the difference exceeds the threshold value, the bolt is considered to be loosened.
In the technical scheme, the detection of the contour can be carried out based on a binary image. The principle of contour detection is to find only the outermost boundary.
In some embodiments, the present invention uses an encoding method to assign different integer values to different boundaries, so as to determine whether it is an empty boundary or an outer boundary and their hierarchical relationship. For example, the pixel values of the input binary image are represented by f (i, j), and each time line scanning is performed, the method is terminated when the following two cases are met:
(1) f (i, j) ≥ 1, f (i, j + 1) =0; then f (i, j) is taken as the starting point of the hole boundary;
(2) f (i, j-1) =0, f (i, j) =1; then f (i, j) is taken as the starting point of the outer boundary;
the pixels on the boundary are then marked starting from the starting point and a unique integer value is assigned to the new boundary, called NBD. In the initial state, NBD =1, and when a new boundary is found, NBD is increased by 1. In this process, when f (i, j) =1, f (i, j + 1) =0, then f (i, j) is assigned to-NBD. After the binary image is subjected to the contour detection algorithm, the contour of the calibration line can be obtained.
In this way, the present invention can be implemented by screening the number of points or the area of the contours, for example selecting the two contours with the largest number of points, wherein the longest calibration line is the first part Contour of the calibration line drawn on the bolt 1 The other Contour is a second partial Contour of the calibration line on the connecting piece 2 。
Wherein for Contour 1 Let 4 vertices of the minimum bounding rectangle be (x) left-top-1 ,y left-top-1 )(x right-top-1 ,y right-top-1 )(x left-bottom-1 ,y left-bottom-1 )
、、、
(x right-bottom-1 ,y right-bottom-1 ) (ii) a For Contour2, let the 4 vertices of the minimum bounding rectangle be (x) left-top-2 ,y left-top-2 )、(x rig h t-top-2 ,y rig h t-top-2 )、
(x left-bottom-2 ,y left-bottom-2 )、(x rig h t-bottom-2 ,y rig h t-bottom-2 ). Then, according to the coordinates of the 4 vertices of the minimum bounding rectangle, the center point can be obtained, as shown in formula (2):
it should be noted that the formula (2) is the first partial Contour 1 Second partial Contour 2 The two smallest circumscribed rectangles have a common formula for the center point, and therefore the subscripts "1" and "2" are not distinguished in the formula.
After the central points of the two minimum circumscribed rectangles are obtained, the included angle theta between the connecting line between the two central points and the x axis can be obtained 1 As shown in formula (3):
then, contour is determined 1 The included angle between the long side of the minimum circumscribed rectangle and the x axis is as follows:
the long side of the minimum circumscribed rectangle is found firstly, and the long side can be obtained by comparing the squares of the lengths of the two sides, wherein the square of the lengths of the two sides is specifically shown in the formula (4). Then, the included angle theta between the long side of the minimum circumscribed rectangle and the X axis can be obtained 2 Specifically, the formula is shown in formula (5).
To obtain theta 1 And theta 2 After two angles, the angular deviation delt _ θ of the first part and the second part of the calibration line can be obtained, as shown in equation (6):
delt_θ＝|θ 2 -θ 1 |(6)
after the angle deviation is obtained, whether the bolt is loosened can be judged by setting a threshold value. For example, if the threshold value is set to 10 °, that is, if the angular deviation delt _ θ is greater than 10 °, it is determined that the bolt is loose.
Another objective of the present invention is to provide a bolt loosening detection system, which can get rid of the dependence on manual detection, so as to quickly and accurately identify the bolt state according to the processing and analysis of the visual image, thereby meeting the industrial automation requirements.
In order to achieve the above object, the present invention provides a bolt loosening detection system, including:
the image acquisition device acquires images of all bolts and connecting pieces connected with the bolts in the set detection area, wherein the bolts and the connecting pieces connected with the bolts are provided with calibration lines, a first part of each calibration line is positioned on the bolt, and a second part of each calibration line is positioned on the connecting pieces;
the deep learning Yolo module is used for identifying and segmenting the input image to obtain a plurality of segmented images, and a single bolt, a connecting piece corresponding to the single bolt and a calibration line are displayed in the segmented images;
a calibration line extraction module which extracts a first partial contour of the calibration line and a second partial contour of the calibration line from each of the divided images;
and the judging module is used for judging whether the bolt is loosened or not based on whether the angular deviation between the first partial contour of the calibration line and the second partial contour of the calibration line exceeds a set threshold value or not.
Further, the bolt looseness detection system further comprises a shake correction module, wherein the shake correction module is used for inputting the image into the deep learning Yolo module after carrying out shake correction on the image by adopting a wiener filtering algorithm.
Further, in the bolt looseness detection system, the calibration line is red, and the calibration line extraction module converts the segmentation image from an RGB image into an HSV image to extract a red region; the red region is then converted into a binary image to extract a first partial contour of the calibration line and a second partial contour of the calibration line.
Further, in the bolt loosening detection system according to the present invention, the determination module performs the following steps:
determining the minimum bounding rectangle of the first part outline of the calibration line and the second part outline of the calibration line;
calculating the coordinates of the central points of the minimum external rectangles through the vertex coordinates of the minimum external rectangles so as to determine the central points of the two minimum external rectangles;
calculating to obtain the included angle theta between the connecting line of the two central points and the X axis 1 ；
Calculating to obtain an included angle theta between the long side of the minimum circumscribed rectangle of the first outline part and the X axis 2 ；
Will theta 2 And theta 1 The difference is compared with a set threshold value, and if the difference exceeds the threshold value, the bolt is considered to be loosened.
Therefore, the bolt looseness detection method and the bolt looseness detection system can solve the problems that the traditional bolt looseness is time-consuming and labor-consuming and low in detection precision through identification based on the visual image, so that the detection precision and the detection speed are greatly improved, and the requirements that the detection precision is more than 95% and the detection speed is more than 1/24s, which are required by industrial automatic detection, can be met.
Drawings
Fig. 1 shows a flowchart of the steps of a method for detecting loosening of a bolt according to an embodiment of the present invention.
Fig. 2 shows a flowchart of the steps of the bolt looseness detection method according to an embodiment of the present invention, which identifies and segments a bolt.
Fig. 3 shows a structural diagram of a Yolov5 model adopted in an embodiment of the bolt looseness detection method.
Fig. 4 shows a flow of outline extraction of a calibration line in one embodiment of the bolt loosening detection method according to the present invention.
Fig. 5 shows a process of determining whether a bolt is loosened according to the bolt loosening detection method of the present invention.
Fig. 6 shows an example of the bolt looseness detection method according to the present invention, in one embodiment, the bolt is not loosened.
Fig. 7 shows an example of the bolt loosening detection method according to the present invention in which the bolt is loosened in one embodiment.
Detailed Description
The bolt loosening detection method and system according to the present invention will be further explained and illustrated with reference to the drawings and specific embodiments of the specification, however, the explanation and illustration should not be construed as unduly limiting the technical solution of the present invention.
Fig. 1 shows a flowchart of the steps of a method for detecting loosening of a bolt according to an embodiment of the present invention.
As shown in fig. 1, in an embodiment, a bolt loosening detection method includes the steps of:
100: setting a detection area, and marking out a calibration line on each bolt of the detection area and a connecting piece connected with the bolt, wherein a first part of the calibration line is positioned on the bolt, and a second part of the calibration line is positioned on the connecting piece; acquiring images of all bolts and connecting pieces connected with the bolts in the detection area range;
200: image shake correction
300: identifying and segmenting the image by adopting a deep learning Yolo model to obtain a plurality of segmented images, wherein a single bolt, a connecting piece corresponding to the single bolt and a corresponding calibration line are displayed in the segmented images;
400: extracting a first partial contour of the calibration line and a second partial contour of the calibration line from each segmented image;
500: and judging whether the bolt is loosened or not based on whether the angular deviation between the first partial contour of the calibration line and the second partial contour of the calibration line exceeds a set threshold value or not.
Accordingly, in some embodiments, the above-described bolt detection method is implemented based on a bolt detection system comprising:
the image acquisition device acquires images of all bolts and connecting pieces connected with the bolts in the set detection area, wherein the bolts and the connecting pieces connected with the bolts are provided with calibration lines, a first part of each calibration line is positioned on each bolt, and a second part of each calibration line is positioned on each connecting piece;
the image is subjected to shake correction by adopting a wiener filtering algorithm, and then the image is input into the deep learning Yolo module;
the deep learning Yolo module is used for identifying and segmenting the input image to obtain a plurality of segmented images, and a single bolt, a connecting piece corresponding to the single bolt and a calibration line are displayed in the segmented images;
a calibration line extraction module which extracts a first partial contour of the calibration line and a second partial contour of the calibration line from each of the divided images;
and the judging module is used for judging whether the bolt is loosened or not based on whether the angular deviation between the first partial contour of the calibration line and the second partial contour of the calibration line exceeds a set threshold value or not.
In some embodiments of the invention, the images of the bolts and the connecting members connected thereto within the detection area may be acquired by a movable camera, which may be implemented by mounting the camera on a movable support. For example, by controlling the support to travel at a constant speed, images of all bolts and peripheral connecting pieces thereof in a certain area range can be acquired.
In some embodiments of the present invention, in step 200, the image may be subjected to a shake correction using a wiener filtering algorithm. The basic principle of wiener filtering for image de-blurring is shown in formula (1):
in which H (u, v) is a degeneration function, S η (u,v)＝|N(u,v)| 2 Power spectrum, S, representing noise f (u,v)＝|F(u,v)| 2 Representing the power spectrum of the non-degraded image, G (u, v) representing the fourier transform of the degraded image,
representing the de-blurred picture.
As described above, since the image within the detection area acquired by the present invention covers a plurality of bolts, it is necessary to divide the plurality of bolts in the image into individual pieces.
Fig. 2 shows a flowchart of the steps of the bolt looseness detection method according to an embodiment of the present invention, which identifies and segments a bolt.
As shown in fig. 2, in this embodiment, the Yolov5 model is first selected for bolt identification and segmentation.
Fig. 3 shows a structural diagram of a Yolov5 model adopted in an embodiment of the bolt looseness detection method according to the present invention.
As shown in FIG. 3, the Yolov5 model is divided into four parts, namely an input end, a backhaul, a tack and a Prediction. The input end carries out preprocessing operation on the input image, for example, the self-adaptive image is zoomed and then is sent to a Yolov5 target detection network. The Backbone uses CSP (Cross Stage Partial), focus and SPP (Spatial Pyramid) network structures, integrates image gradient change into a characteristic diagram, and can reduce the parameter quantity and the operation quantity of the model, thereby realizing that the detection precision and speed can be still ensured under the condition of reducing the scale of the model. The Neck uses a network structure combining PAN and FPN to transfer image features to the Prediction, and can accurately reserve the spatial information of the image. The Prediction adopts GIOU _ Loss as a Loss function of the bounding box, and the NMS performs non-maximum suppression processing on the detection frame to obtain an optimal target frame. The CSP1_ X and CSP2_ X structures in the CSP are respectively applied to the Backbone and the Neck. The SPP respectively adopts the maximal pooling of 5, 9 and 13, so that the problems of distortion and the like caused by operations such as image scaling, clipping and the like can be solved, and the problem of repeated extraction of image features by a network structure can also be solved.
With continued reference to fig. 2, after selecting the Yolov5 model, the embodiment divides the training set, the verification set, and the test set according to the ratio of 7. The training set for model training adopts the PASCAL VOC format, and the data set can be generated by labeling pictures through a LabelImg tool, only contains 1 category of 'bolt' and represents a bolt.
In the training process, the upper limit of model recognition is approached by setting a training hyper-parameter of a bolt recognition model. In this embodiment, the model training parameters are shown in table 1.
Table 1.
| Parameter(s) | Parameter value |
| Epochs (which represents how many times all sample data will be "turned" during the training process) | 300 |
| batch-size (which represents the number of samples fed into the network at a time) | 32 |
| Mosaic (which represents a data enhancement algorithm) | 1.0 |
| Mixup (which represents the two graphs interpolated to scale to mix the samples) | 0.0 |
| Lr (which represents learning rate) | 0.01 |
| Momentum (which represents Momentum) | 0.8 |
| GPU (display card representation) | true |
And according to the network structure, the parameters and the data set, performing model training by adopting a graphics card GPU to obtain a trained Yolo model as a bolt recognition model.
After the bolt identification model is obtained, the model can be deployed. And after the model is deployed, real-time bolt recognition can be carried out on the image. Meanwhile, a coordinate point (x) corresponding to the upper left corner of the square frame can be obtained left-top ,y left-top ) Coordinate point (x) of lower right corner right-bottom ,y right-bottom ) The type ClassID is identified and the probability Score. The result can be used for image segmentation of the bolt.
The obtained segmentation image comprises a single bolt and a peripheral connecting piece thereof and also comprises a corresponding calibration line, so that the image segmentation area is larger than the area identified by Yolov 5. When the image is segmented, the range of the frame of the Yolov5 model is expanded by 1.8 times, and the segmented image comprises the calibration line of the bolt, so that the bolt looseness detection at the later stage is facilitated.
Fig. 4 shows a flow of outline extraction of a calibration line in one embodiment of the bolt looseness detection method according to the present invention.
In some embodiments, to facilitate the identification of the calibration line, in step 100 of the present invention, the calibration line is scribed in red. Based on this, as shown in fig. 4, when extracting the contour of the calibration line: firstly, the segmentation image is converted into an HSV image from an RGB image so as to extract a red area. The red region is then converted into a binary image for contour detection, the principle of which is to find only the outermost boundary.
In some more specific embodiments, the different boundaries are assigned different integer values using an encoding method so that it can be determined whether it is an empty boundary or an outer boundary and their hierarchical relationship. For example, the pixel values of the input binary image are represented by f (i, j), and each time line scanning is performed, the method is terminated when the following two cases are met:
(1) f (i, j) ≥ 1, f (i, j + 1) =0; then f (i, j) is taken as the starting point of the hole boundary;
(2) f (i, j-1) =0, f (i, j) =1; then f (i, j) is taken as the starting point of the outer boundary;
the pixels on the boundary are then marked starting from the starting point and a unique integer value is assigned to the new boundary, called NBD. In the initial state, NBD =1, and when a new boundary is found, NBD is increased by 1. In this process, when f (i, j) =1, f (i, j + 1) =0, then f (i, j) is assigned to-NBD. After the binary image is subjected to the contour detection algorithm, the contour of the calibration line can be obtained.
Then, two contours with the maximum points are selected by screening the points of the contours. The first partial Contour of the longest of which is the calibration line drawn on the bolt 1 The second length being the second partial Contour of the calibration line drawn on the connecting element 2 。
Fig. 5 shows a process of determining whether a bolt is loosened according to the bolt loosening detection method of the present invention.
As shown in fig. 5, in some embodiments, the following steps may be employed to determine whether the bolt is loosened:
determining a first partial Contour of a calibration line 1 And a second partial Contour of the calibration line 2 Respective minimum circumscribed rectangles;
contour through the first part 1 Vertex coordinates (x) of the minimum bounding rectangle of (1) left-top-1 ,y left-top-1 )(x right-top-1 ,y right-top-1 )(x left-bottom-1 ,y left-bottom-1 )
、、、(x right-bottom-1 ,y right-bottom-1 ) And a second partial Contour 2 Vertex coordinates (x) of the minimum bounding rectangle of (1) left-top-2 ,y left-top-2 )、(x rig h t-top-2 ,y rig h t-top-2 )、(x left-bottom-2 ,y left-bottom-2 )、(x rig h t-bottom-2 ,y rig h t-bottom-2 ) And calculating the coordinates of the central points of the minimum circumscribed rectangles to determine the central points of the two minimum circumscribed rectangles:
it should be noted that the formula (2) is the first partContour 1 Second partial Contour 2 The two minimum circumscribed rectangles have a common formula of the central point, so the difference of subscripts '1' and '2' is not made in the formula;
after the central points of the two minimum circumscribed rectangles are obtained, the included angle theta between the connecting line between the two central points and the x axis can be obtained 1 As shown in formula (3):
then, contour is determined 1 The included angle between the long edge of the minimum external rectangle and the x axis is as follows:
firstly, finding the long side of the minimum circumscribed rectangle, and obtaining the length of the two sides by comparing the squares of the lengths of the two sides, wherein the square of the lengths of the two sides is specifically shown in the formula (4):
then, the included angle theta between the long side of the minimum circumscribed rectangle and the x axis can be obtained 2 Specifically, the formula is shown as (5).
To obtain theta 1 And theta 2 After two angles, the angular deviation delt _ θ between the first partial contour and the second partial contour of the calibration line can be obtained, as shown in equation (6):
delt_θ＝|θ 2 -θ 1 |(6)
after the angle deviation is obtained, whether the bolt is loosened or not can be judged by setting a threshold value.
In some specific embodiments, the threshold may be set at 10 °. And if the angle deviation delt _ theta is less than 10 degrees, judging that the bolt is not loosened. Fig. 6 shows an example of the bolt looseness detection method according to the present invention, in one embodiment, the bolt is not loosened. If the angle deviation delt _ theta is larger than or equal to 10 degrees, the bolt is judged to be loosened, and fig. 7 shows an example of the bolt loosening detection method according to the invention in one embodiment.
In addition, the combination of the features in the present application is not limited to the combination described in the claims of the present application or the combination described in the embodiments, and all the features described in the present application may be freely combined or combined in any manner unless contradictory to each other.
It should also be noted that the above-listed embodiments are only specific embodiments of the present invention. It is apparent that the present invention is not limited to the above embodiments and similar changes or modifications thereto which can be directly or easily inferred from the disclosure of the present invention by those skilled in the art are intended to be within the scope of the present invention.
Claims (10)
1. A bolt looseness detection method is characterized by comprising the following steps:
100: setting a detection area, and marking out a calibration line on each bolt of the detection area and a connecting piece connected with the bolt, wherein a first part of the calibration line is positioned on the bolt, and a second part of the calibration line is positioned on the connecting piece;
200: acquiring images of all bolts and connecting pieces connected with the bolts in the detection area range;
300: identifying and segmenting the image by adopting a deep learning Yolo model to obtain a plurality of segmented images, wherein a single bolt, a connecting piece corresponding to the single bolt and a calibration line are displayed in the segmented images;
400: extracting a first partial contour of the calibration line and a second partial contour of the calibration line from each segmented image;
500: and judging whether the bolt is loosened or not based on whether the angular deviation between the first partial contour of the calibration line and the second partial contour of the calibration line exceeds a set threshold value or not.
2. The bolt looseness detection method of claim 1, further comprising, between step 200 and step 300, the steps of: and carrying out shake correction on the image.
3. The bolt looseness detection method of claim 2, wherein a wiener filter algorithm is used to perform shake correction on the image.
4. The bolt looseness detecting method of claim 1, wherein in step 100, the calibration line is red.
5. The bolt looseness detection method of claim 4, wherein in step 400: converting the segmentation image from the RGB image into an HSV image to extract a red region; the red area is converted into a binary image to extract a first partial contour of the calibration line and a second partial contour of the calibration line.
6. The bolt looseness detection method of claim 1, wherein in step 500:
determining the minimum bounding rectangle of the first part outline of the calibration line and the second part outline of the calibration line;
calculating the coordinates of the central points of the minimum external rectangles through the vertex coordinates of the minimum external rectangles so as to determine the central points of the two minimum external rectangles;
calculating to obtain the included angle theta between the connecting line of the two central points and the X axis 1 ；
Calculating to obtain an included angle theta between the long side of the minimum circumscribed rectangle of the first outline part and the X axis 2 ；
Will theta 2 And theta 1 The difference in the values is compared with a set threshold value, and if the difference exceeds the threshold value, the bolt is considered to be loosened.
7. A bolt loosening detection system, comprising:
the image acquisition device acquires images of all bolts and connecting pieces connected with the bolts in the set detection area, wherein the bolts and the connecting pieces connected with the bolts are provided with calibration lines, a first part of each calibration line is positioned on the bolt, and a second part of each calibration line is positioned on the connecting pieces;
the deep learning Yolo module is used for identifying and segmenting the input image to obtain a plurality of segmented images, and a single bolt, a connecting piece corresponding to the single bolt and a calibration line are displayed in the segmented images;
the calibration line extraction module extracts a first partial contour of the calibration line and a second partial contour of the calibration line from each segmented image;
and the judging module is used for judging whether the bolt is loosened or not based on whether the angular deviation between the first partial contour of the calibration line and the second partial contour of the calibration line exceeds a set threshold value or not.
8. The bolt looseness detection system of claim 7, further comprising a shake correction module that inputs the image into a deep learning Yolo module after shake correction of the image by using a wiener filter algorithm.
9. The bolt looseness detection system according to claim 7, wherein the calibration line is red, and the calibration line extraction module converts the divided image from an RGB image into an HSV image to extract a red region; the red region is then converted into a binary image to extract a first partial contour of the calibration line and a second partial contour of the calibration line.
10. The bolt loosening detection system of claim 7, wherein the determination module performs the steps of:
determining the minimum bounding rectangle of the first part outline of the calibration line and the second part outline of the calibration line;
calculating the coordinates of the central points of the minimum external rectangles through the vertex coordinates of the minimum external rectangles to determine the central points of the two minimum external rectangles;
calculating to obtain the included angle theta between the connecting line of the two central points and the X axis 1 ；
Calculating to obtain a first contourThe included angle theta between the long side of the part of the minimum circumscribed rectangle and the X axis 2 ；
Will theta 2 And theta 1 The difference is compared with a set threshold value, and if the difference exceeds the threshold value, the bolt is considered to be loosened.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211067640.5A CN115482207A (en) | 2022-09-01 | 2022-09-01 | Bolt looseness detection method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211067640.5A CN115482207A (en) | 2022-09-01 | 2022-09-01 | Bolt looseness detection method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115482207A true CN115482207A (en) | 2022-12-16 |
Family
ID=84421267
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211067640.5A Pending CN115482207A (en) | 2022-09-01 | 2022-09-01 | Bolt looseness detection method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115482207A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117593515A (en) * | 2024-01-17 | 2024-02-23 | 中数智科(杭州)科技有限公司 | Bolt loosening detection system and method for railway vehicle and storage medium |
-
2022
- 2022-09-01 CN CN202211067640.5A patent/CN115482207A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117593515A (en) * | 2024-01-17 | 2024-02-23 | 中数智科(杭州)科技有限公司 | Bolt loosening detection system and method for railway vehicle and storage medium |
| CN117593515B (en) * | 2024-01-17 | 2024-03-29 | 中数智科(杭州)科技有限公司 | Bolt loosening detection system and method for railway vehicle and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113658132B (en) | Computer vision-based structural part weld joint detection method | |
| CN111260616A (en) | Insulator crack detection method based on Canny operator two-dimensional threshold segmentation optimization | |
| CN111008961B (en) | Transmission line equipment defect detection method and system, equipment and medium thereof | |
| CN108416809B (en) | Steel drum threaded cap pose recognition method based on machine vision | |
| CN106682665B (en) | Seven-segment type digital display instrument number identification method based on computer vision | |
| CN112149543B (en) | Building dust recognition system and method based on computer vision | |
| CN109472788B (en) | Method for detecting flaw on surface of airplane rivet | |
| CN110596120A (en) | Glass boundary defect detection method, device, terminal and storage medium | |
| CN111667475B (en) | Machine vision-based Chinese date grading detection method | |
| CN110335233B (en) | Highway guardrail plate defect detection system and method based on image processing technology | |
| CN110910339B (en) | Logo defect detection method and device | |
| CN114331986A (en) | Dam crack identification and measurement method based on unmanned aerial vehicle vision | |
| CN113658131A (en) | Tour type ring spinning broken yarn detection method based on machine vision | |
| CN112232222B (en) | Bullet train axle box end cover bolt loss fault detection method based on image processing | |
| CN112419298A (en) | Bolt node plate corrosion detection method, device, equipment and storage medium | |
| CN113902641A (en) | Data center hot area distinguishing method and system based on infrared image | |
| CN111062331A (en) | Mosaic detection method and device for image, electronic equipment and storage medium | |
| CN115482207A (en) | Bolt looseness detection method and system | |
| CN115272664A (en) | Instrument panel display method and device, electronic equipment and storage medium | |
| CN115272350A (en) | Method for detecting production quality of computer PCB mainboard | |
| CN116228757B (en) | Deep sea cage and netting detection method based on image processing algorithm | |
| US10115195B2 (en) | Method and apparatus for processing block to be processed of urine sediment image | |
| CN114359156A (en) | Insulator string-dropping fault detection method based on image recognition | |
| CN115346138A (en) | Target detection method, device and equipment of aerial image based on unmanned aerial vehicle | |
| CN111583341B (en) | Cloud deck camera shift detection method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |