CN111627018B - Steel plate surface defect classification method based on double-flow neural network model - Google Patents

Steel plate surface defect classification method based on double-flow neural network model Download PDF

Info

Publication number
CN111627018B
CN111627018B CN202010484073.8A CN202010484073A CN111627018B CN 111627018 B CN111627018 B CN 111627018B CN 202010484073 A CN202010484073 A CN 202010484073A CN 111627018 B CN111627018 B CN 111627018B
Authority
CN
China
Prior art keywords
defect
neural network
detected
defects
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010484073.8A
Other languages
Chinese (zh)
Other versions
CN111627018A (en
Inventor
倪红军
张加俏
任福继
张振亚
贯大兴
吕帅帅
汪兴兴
张福豹
朱昱
张守阳
王凯旋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nantong University
Original Assignee
Nantong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nantong University filed Critical Nantong University
Priority to CN202010484073.8A priority Critical patent/CN111627018B/en
Publication of CN111627018A publication Critical patent/CN111627018A/en
Application granted granted Critical
Publication of CN111627018B publication Critical patent/CN111627018B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G01N2021/8854—Grading and classifying of flaws
    • G01N2021/8861—Determining coordinates of flaws
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G01N2021/8854—Grading and classifying of flaws
    • G01N2021/8874—Taking dimensions of defect into account
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G01N2021/8854—Grading and classifying of flaws
    • G01N2021/888—Marking defects
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G01N2021/8887—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges based on image processing techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20048—Transform domain processing
    • G06T2207/20061—Hough transform
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30136—Metal
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computing Systems (AREA)
  • Evolutionary Biology (AREA)
  • Signal Processing (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Biochemistry (AREA)
  • Immunology (AREA)
  • Pathology (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)
  • Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)

Abstract

The invention provides a steel plate surface defect classification method based on a double-flow neural network model, which comprises the following steps: s10, establishing a standard drawing library of the surface defects of the steel plate; s20, classifying defects of pictures to be detected, importing the pictures to be detected into a double-flow neural network to obtain a global priority value and a local characteristic value, and determining the defect type of each picture to be detected by integrating the global priority value and the local characteristic value; and S30, judging the defect position of the picture to be detected, and determining the position of the defect category in the classification result by using a YOLO network, wherein two residual error network modules in the YOLO network are replaced by two characteristic multiplexing network modules. According to the method for classifying the defects on the surface of the steel plate of the double-flow neural network model, the type of the defects is firstly judged by adopting the double-flow neural network model, and then a picture to be detected is imported into a YOLO network to predict the positions of the defects; the dual-flow neural network and the YOLO network are matched, so that defect positioning accuracy is improved, and detection efficiency is improved.

Description

Steel plate surface defect classification method based on double-flow neural network model
Technical Field
The invention relates to the technical field of steel plate defect detection, in particular to a steel plate surface defect classification method based on a double-flow neural network model.
Background
Sheet metal is an indispensable raw material in the mechanical industry, and the product quality of sheet metal is a key index for determining the price thereof. Due to the problems of equipment and process condition limitations, the surface of the metal sheet inevitably has defects of different forms and different types, and the size, the number and the distribution of the defects are greatly different. And due to the diversity and complexity of surface defects, iron and steel production enterprises in various countries pay attention to the detection of surface quality, and the detection technology and the detection level are improved without great expense.
The common defects on the surface of the steel plate can be divided into two major types according to the shape of the defects, namely planar defects and linear defects. Common deep learning models such as retnet models and space pyramid models are direct reading images, image features are extracted through convolution operation, and then defect types are predicted. This algorithm only considers the local characteristics of the defect in the convolution process.
The YOLO network model was proposed in 2016 at the earliest, and the later version YOLOv3 is not only faster in detection speed, but also more suitable for detection of small targets. The YOLO network contains 24 volume layers, 4 max pooling layers and two full connection layers. The roll base layer is used for acquiring image characteristics, the maximum pooling layer is used for reducing image pixels, and the full-connection layer is used for predicting image types and positions. YOLO uses the features of the full map to predict bounding boxes and classify objects within the boxes, which means that the YOLO network can use the full map information to achieve object classification and object location detection in the same image.
In the image detection process, the YOLO can realize the classification and detection of the targets through multi-layer convolution. For example, when a dog and a cat are present in a single picture, the YOLO network can classify the dog and the cat, distinguish which is the dog and which is the cat, and locate the position, and mark the target position by using a square frame. The target detection result is evaluated using a confidence value, and the calculation formula is as follows. The confidence value can be seen as the product of the classification probability Pr and the IOU value, both belonging to [0,1]. The IOU value is the ratio of the intersection of the predicted frame and the real frame area.
The invention patent of publication number CN110490842A discloses a strip steel surface defect detection method based on deep learning, which comprises the steps of extracting local information of the strip steel surface through a defect judging and defect classifying double-flow network model, comprehensively analyzing by combining a scale pyramid to obtain a heat-like diagram, and finally obtaining the type and position of a defect at the same time, wherein the defect judging and defect classifying double-flow network model comprises a defect judging branch and a defect classifying branch.
Although this method can detect defects, it can detect only one type of defect, and can feed back only one type of defect for the surface of a steel sheet where two or more types of defects exist at the same time, and the data provided is insufficient to guide production. In actual production, the problem in actual production cannot be solved easily because of inaccurate judgment, so that the defect exists for a long time, and feedback is needed through manual detection. And waste of manpower and material resources is caused.
Disclosure of Invention
In order to solve the problems, the invention provides a steel plate surface defect classification method based on a double-flow neural network model, which comprises the steps of firstly judging the types of defects by adopting the double-flow neural network model, and then guiding a picture to be detected into a YOLO network to predict the positions of the defects; the dual-flow neural network and the YOLO network are matched, so that defect positioning accuracy is improved, and detection efficiency is improved.
In order to achieve the above purpose, the invention adopts a technical scheme that:
a steel plate surface defect classification method based on a double-flow neural network model comprises the following steps: s10, establishing a standard drawing library of the surface defects of the steel plate; s20, classifying defects of pictures to be detected, importing the pictures to be detected into a double-flow neural network to obtain a global priority value and a local characteristic value, and determining the defect type of each picture to be detected by integrating the global priority value and the local characteristic value; and S30, judging the defect position of the picture to be detected, and determining the position of the defect category in the classification result by using a YOLO network, wherein two residual error network modules in the YOLO network are replaced by two characteristic multiplexing network modules.
Further, the step S10 includes the steps of: s11, each picture in the defect standard picture library contains a typical defect, and the picture is corrected and cut by using Hough transformation so that the size of the picture is 200 x 200dpi; and S12, marking the image by using labelImg software, marking the position of the defect in the image by using a rectangular real frame, recording the coordinate information of the upper left corner (xL, yL) and the lower right corner (xR, yR) of the rectangular frame, and marking each single defect by adopting a dense marking method in the marking process.
Further, the step S20 includes the steps of: s21, importing a picture to be detected into a double-flow neural network, extracting image features by the double-flow neural network, and distributing the extracted image features according to a certain weight to obtain two flow directions; s22, the weight of one image feature enters a global priority network, global feature priori is captured from the whole image, the global defect category is predicted in a large direction, and finally a global priority value y1 is obtained through the global priority network; s23, the weight of the other image feature enters a spatial pyramid convolution layer, firstly, the spatial pyramid convolution layer extracts multi-scale example image features, then, each generated example feature is mapped through a full connection layer, and finally, the local defect category prediction of the corresponding global defect category is carried out through a relevant area in the spatial pooling layer selection mapping, so that a local feature value y2 is obtained; and S24, using an aggregation layer to aggregate the global priority value and the local characteristic value of the picture to be detected, and combining the global priority value and the local characteristic value to obtain a defect type prediction result.
Further, the global priority network is based on a VGG-16network architecture and comprises a 2×2 pooling layer, 3 full connection layers FCa, FCb and FCg, and bypass connection is set between the FCa and the FCg, so that FCa bypasses the FCb and is directly connected with the FCg.
Further, the global defect class includes planar defects and linear defects; the local defect categories of the planar defect include: at least one of plaque (Pa), surface Pits (PS), and scale indentation (RS); the local defect categories of the linear defects include: at least one of reticulation (Cr), inclusion (In) and scratch (Sc).
Further, two residual error network modules in the YOLO network are replaced by two feature multiplexing network modules, each feature multiplexing network module comprises 3 convolution layers, each convolution layer can obtain the output of all previous convolution layers as input, and adjacent convolution layers are connected through the convolution layers and the pooling layers.
Further, the step S30 includes the steps of: s31, importing a steel plate image to be detected into a YOLO network based on classification priority, and unifying the size and the size of the steel plate image to be detected to 448×448dpi by adopting a bilinear interpolation method; s32, carrying out normalization processing on the steel plate image to be detected after the size adjustment, and converting the pixel value range of the steel plate image to be detected from [0,255] to [0,1] to obtain a first defect classification chart; the normalization formula is:
wherein x is i Representing image pixel point values, wherein min (x) and max (x) represent maximum and minimum values of image pixels; and S33, dividing the first defect classification map into S multiplied by S grids, and if the center of the target defect falls into a grid unit, the grid unit is responsible for detecting the object, and obtaining a position detection result of the target defect.
Further, a YOLO network parameter is set, the number K=6 of cluster clusters is taken, the convolution kernel size is 1*1, the convolution step size is 1, the initial learning rate of a model is 0.01, the number of samples selected by one training is 4, the weight attenuation regular term is set to be 0.0005, and asynchronous random gradient descent with the motion term of 0.9 is adopted.
Further, each of the grids predicts B prediction frames; the prediction frame packageContaining 5 data values (x, y, w, h, confidence); (x, y) is the offset of the center of the prediction block relative to the current grid and (w, h) is the length and width of the prediction block; the confidence value reflects whether the bounding box contains a target probability that coincides with the current bounding box and the actual bounding boxThe final detection result accords with the following formula:
wherein Pr (Object) is a target Object judgment parameter, pr (Object) =1 when the classification result has a target defect, and Pr (Object) =0 when the classification result has no target defect; pr (Class) i /Object) is a conditional probability of a class.
Compared with the prior art, the technical scheme of the invention has the following advantages:
according to the steel plate surface defect classification method based on the double-flow neural network model, one flow is used for judging the global defect type, and the other flow is used for judging the local defect under a certain global defect type; firstly judging the types of defects by adopting a double-flow neural network model, and then guiding the picture to be detected into a YOLO network to predict the defect position; the dual-flow neural network and the YOLO network are matched, so that defect positioning accuracy is improved, and detection efficiency is improved.
Drawings
The technical solution of the present invention and its advantageous effects will be made apparent by the following detailed description of the specific embodiments of the present invention with reference to the accompanying drawings.
FIG. 1 is a flow chart of a method for classifying surface defects of a steel plate based on a dual-flow neural network model according to an embodiment of the invention;
FIG. 2 is a flow chart of a method for classifying surface defects of a steel plate based on a dual-flow neural network model according to an embodiment of the invention;
FIG. 3 is a diagram of a global priority network architecture according to one embodiment of the present invention;
FIG. 4 shows the surface defect classification of a steel sheet according to an embodiment of the present invention;
FIG. 5 is a flowchart showing a method for determining a defect position on a steel sheet surface based on a YOLO network according to an embodiment of the present invention;
FIG. 6 is a graph showing the results of a test for the prior classification of defects in steel sheets by a dual-flow neural network according to an embodiment of the present invention;
fig. 7 shows the results of classification tests of the steel plate surface defect dataset by different algorithms.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to fall within the scope of the invention.
The embodiment provides a steel plate surface defect classification method based on a double-flow neural network model, which comprises the following steps as shown in fig. 1-2: s10, establishing a standard chart library of the surface defects of the steel plate. S20, classifying defects of the pictures to be detected, importing the pictures to be detected into a double-flow neural network to obtain global priority values and local characteristic values, and determining the defect type of each picture to be detected by integrating the global priority values and the local characteristic values. And S30, judging the defect position of the picture to be detected, and determining the position of the defect category in the classification result by using a YOLO network, wherein two residual error network modules in the YOLO network are replaced by two characteristic multiplexing network modules.
The step S10 includes the steps of: and S11, each picture in the defect standard picture library contains a typical defect, and the picture is corrected and cut by using Hough transformation so that the size of the picture is 200 x 200dpi. S12, marking the image by using labelImg software, marking the position of the defect in the image by using a rectangular real frame, and recording the left upper corner (x L ,y L ) And right sideLower corner (x) R ,y R ) And (3) adopting a dense labeling method in the labeling process, namely labeling each single defect. The defect standard chart library comprises defect types such as reticulate patterns (Cr), inclusion (In), plaque patches (Pa), surface pits Pitted Surface (PS), scratch scratches (Sc) and the like. Further, defects can be classified into two major categories, i.e., planar defects and linear defects, and local defect categories of planar defects include plaque (Pa), surface Pits (PS), inclusions (In), and the like; the partial defect type of the linear defect includes moire (Cr), scratch (Sc), and the like.
The step S20 includes the steps of: s21, importing the picture to be detected into a double-flow neural network, extracting image features by the double-flow neural network, and distributing the extracted image features according to a certain weight to obtain two flow directions.
S22, the weight of the image feature enters a global priority network, the global feature priori is captured from the whole image, the global defect category is firstly predicted in a large direction, and finally the global priority value y is obtained through the global priority network 1 . As shown in fig. 3, the global priority network is based on a VGG-16network architecture, and includes a 2×2 pooling layer and 3 full connection layers FCa, FCb, and FCg, and a bypass connection is set between the FCa and the FCg, so that FCa bypasses the FCb and is directly connected to the FCg. The features of FCa and FCb are fully utilized to reduce possible information loss and generate more accurate global priority values. Finally, obtaining the global priority value y through the global priority network 1 . As shown in fig. 4, the global defect class includes planar defects and linear defects. Since the linear defects and the planar defects are very different macroscopically, accurate discrimination can be performed by the global priority network.
S23, the weight of the other image feature enters a spatial pyramid convolution layer, firstly, the spatial pyramid convolution layer extracts multi-scale example image features, then, each generated example feature is mapped through a full connection layer, and finally, the local defect category prediction of the corresponding global defect category is carried out through a relevant area in the spatial pooling layer selection mapping, so that a local feature value y is obtained 2 . The local defect categories of the planar defect include: at least one of plaque (Pa), surface Pits (PS) and inclusions (In); the local defect categories of the linear defects include: at least one of texture (Cr) and scratch (Sc).
And S24, using an aggregation layer to aggregate the global priority value and the local characteristic value of the picture to be detected, and combining the global priority value and the local characteristic value to obtain a defect type prediction result. The formula is as follows:
where y is the aggregate score, W is a c 2c weight matrix, b is the bias, and c is the classification number. When the global priority network determines that the defect type is planar defect, the local prediction does not predict dot defect conclusion such as reticulate pattern, scratch and the like, and classification accuracy is improved. Taking the pits on the surface of the defect as an example, extracting image characteristic information and carrying out weight distribution. If the global priority network predicts that the defect is a planar defect with the probability of 0.8 and the probability of a linear defect is 0.2; the local feature network predicts that the defect is 0.5 of surface pits and the probability of reticulation is 0.5. Combining two networks in flow direction, the final result is: the probability of the defect being a surface pit is 0.4, and the probability of the defect being a reticulate pattern is 0.1. The defect may be determined to be a surface pitting.
As shown in fig. 5, the step S30 includes the steps of: s31, importing the steel plate image to be detected into a YOLO network based on classification priority, and unifying the size and the size of the steel plate image to be detected to 448×448dpi by adopting a bilinear interpolation method. The specific method of bilinear interpolation is as follows: if we want the value of the unknown function f at point p= (x, y), we assume that we know the values of the function f at four points q11= (x 1, y 1), q12= (x 1, y 2), q21= (x 2, y 1) and q22= (x 2, y 2). The final result of bilinear interpolation is:
s32, carrying out normalization processing on the steel plate image to be detected after the size adjustment, and converting the pixel value range of the steel plate image to be detected from [0,255] to [0,1] to obtain a first defect classification chart; the normalization formula is:
wherein x is i The values of the image pixels are represented, and min (x) and max (x) represent the maximum and minimum values of the image pixels. The information storage of the normalized image itself is not changed, but the pixel value of the image is ranged from [0,255]]Converted into [0,1]]And the subsequent neural network processing is convenient.
And S33, dividing the first defect classification map into S multiplied by S grids, and if the center of the target defect falls into a grid unit, the grid unit is responsible for detecting the object, and obtaining a position detection result of the target defect.
The two residual error network modules in the YOLO network are replaced by two characteristic multiplexing network modules for use, so that the model can receive the multi-layer convolution characteristics output by the intensive connection blocks before prediction is performed. Each characteristic multiplexing network module comprises 3 convolution layers, each convolution layer can obtain the output of all previous convolution layers as input, and adjacent convolution layers are connected through the convolution layers and the pooling layers. The method comprises the steps of setting a YOLO network parameter, taking the number K=6 of cluster clusters, the convolution kernel size is 1*1, the convolution step length is 1, the initial learning rate of a model is 0.01, the number of samples selected by one training is 4, the weight attenuation regular term is set to be 0.0005, and asynchronous random gradient descent with the motion term of 0.9 is adopted.
Predicting B prediction frames by each grid; the prediction block contains 5 data values (x, y, w, h, confidence). (x, y) is the offset of the center of the prediction block relative to the current grid, and (w, h) is the length and width of the prediction block. The confidence value reflects whether the bounding box contains a target probability that coincides with the current bounding box and the actual bounding boxI.e. comprising two parts: first, whether the grid contains the target Object Pr (Object) or not, and second, the accuracy of the grid prediction B. Wherein Pr (Object) is a target Object determination parameter, pr (Object) =1 when the classification result has a target defect, and Pr (Object) =0 when the classification result has no target defect. Pr (Class) i /Object) is a conditional probability of a class.
The IOU is the intersection ratio of the predicted frame and the real frame, and the calculation formula is as follows:
when there are C defects in the image, the conditional probability of C categories is Pr (Class i Object) representing the probability that the mesh contains the target Object and belongs to the i-th Object. The final output probability is therefore
Due to Pr (Class) i ) The value of (1) is 0,1]Therefore, the defect detection precision of the improved model is higher.
The final detection result accords with the following formula:
the steel plate defects were classified by the double-flow neural network with priority, and the experimental results are shown in fig. 6. After the network initialization is finished, the classification capability is not available, the accuracy of the training set is 0.65 at the initial stage, and then the accuracy is gradually increased along with the increase of the iteration times; the accuracy rate in the test set gradually rises along with the increase of the iteration times, and finally converges to 1; the loss values for the training set and the test set decrease as training progresses, eventually converging to 0.
As can be seen from fig. 7, the accuracy of classifying the defects of the steel plate by the method provided by the application is highest, and the average test accuracy can reach 99.7%.
The foregoing description is only exemplary embodiments of the present invention and is not intended to limit the scope of the present invention, and all equivalent structures or equivalent processes using the descriptions and the drawings of the present invention or directly or indirectly applied to other related technical fields are included in the scope of the present invention.

Claims (8)

1. The steel plate surface defect classification method based on the double-flow neural network model is characterized by comprising the following steps of:
s10, establishing a standard drawing library of the surface defects of the steel plate;
s20, classifying defects of pictures to be detected, importing the pictures to be detected into a double-flow neural network to obtain a global priority value and a local characteristic value, and determining the defect type of each picture to be detected by integrating the global priority value and the local characteristic value; and
s30, judging the defect position of the picture to be detected, and determining the position of the defect category in the classification result by using a YOLO network, wherein two residual error network modules in the YOLO network are replaced by two characteristic multiplexing network modules;
the step S20 includes the steps of:
s21, importing a picture to be detected into a double-flow neural network, extracting image features by the double-flow neural network, and distributing the extracted image features according to a certain weight to obtain two flow directions;
s22, the weight of the image feature enters a global priority network, the global feature priori is captured from the whole image, the global defect category is firstly predicted in a large direction, and finally the global priority value y is obtained through the global priority network 1 ；
S23, the weight of the other image feature enters a spatial pyramid convolution layer, firstly, the spatial pyramid convolution layer extracts multi-scale example image features, then, each generated example feature is mapped through a full connection layer, and finally, the local defect category prediction of the corresponding global defect category is carried out through a relevant area in the spatial pooling layer selection mapping, so that the image feature is obtainedTo the local feature value y 2 The method comprises the steps of carrying out a first treatment on the surface of the And
and S24, aggregating the global priority value and the local characteristic value of the picture to be detected by using an aggregation layer, and obtaining a defect type prediction result by combining the global priority value and the local characteristic value.
2. The method for classifying surface defects of steel sheet based on a dual-flow neural network model according to claim 1, wherein the step S10 comprises the steps of:
s11, each picture in the defect standard picture library contains a typical defect, and the picture is corrected and cut by using Hough transformation so that the size of the picture is 200 x 200dpi; and
s12, marking the image by using labelImg software, marking the position of the defect in the image by using a rectangular real frame, and recording the left upper corner (x L ,y L ) And lower right corner (x) R ,y R ) And (3) adopting a dense labeling method in the labeling process, namely labeling each single defect.
3. The method for classifying surface defects of steel plates based on a dual-flow neural network model according to claim 1, wherein the global priority network is based on a VGG-16network architecture and comprises a 2 x2 pooling layer and 3 full connection layers FCa, FCb and FCg, and bypass connection is arranged between the FCa and FCg so that FCa bypasses FCb and is directly connected with FCg.
4. The method for classifying surface defects of steel sheets based on a dual-flow neural network model according to claim 1, wherein the global defect class includes planar defects and linear defects; the local defect categories of the planar defect include: at least one of plaque (Pa), surface Pits (PS) and inclusions (In); the local defect categories of the linear defects include: at least one of texture (Cr) and scratch (Sc).
5. The method for classifying surface defects of steel plates based on a dual-flow neural network model according to claim 1, wherein two residual network modules in a YOLO network are replaced by two characteristic multiplexing network modules, each characteristic multiplexing network module comprises 3 convolution layers, each convolution layer can obtain the output of all previous convolution layers as input, and adjacent convolution layers are connected through the convolution layers and a pooling layer.
6. The method for classifying surface defects of steel sheet based on a dual-flow neural network model according to claim 5, wherein said step S30 comprises the steps of:
s31, importing a steel plate image to be detected into a YOLO network based on classification priority, and unifying the size and the size of the steel plate image to be detected to 448×448dpi by adopting a bilinear interpolation method;
s32, carrying out normalization processing on the steel plate image to be detected after the size adjustment, and converting the pixel value range of the steel plate image to be detected from [0,255] to [0,1] to obtain a first defect classification chart; the normalization formula is:
wherein x is i Representing image pixel point values, wherein min (x) and max (x) represent maximum and minimum values of image pixels; and
s33, dividing the first defect classification map into S multiplied by S grids, and if the center of the target defect falls into a grid unit, the grid unit is responsible for detecting an object, and obtaining a position detection result of the target defect.
7. The method for classifying the surface defects of the steel plate based on the double-flow neural network model according to claim 6, wherein the parameters of the YOLO network are set, the number of clustering clusters K=6, the convolution kernel size is 1*1, the convolution step length is 1, the initial learning rate of the model is 0.01, the number of samples selected by one training is 4, the weight attenuation regular term is set to 0.0005, and the asynchronous random gradient descent with the motion term of 0.9 is adopted.
8. The method for classifying surface defects of steel plates based on a dual-flow neural network model according to claim 7, wherein each grid predicts B prediction frames; the prediction box contains 5 data values (x, y, w, h, confidence); (x, y) is the offset of the center of the prediction block relative to the current grid and (w, h) is the length and width of the prediction block; the confidence value reflects the probability of whether the boundary of the predicted frame contains the target or not and the situation that the boundary of the current predicted frame coincides with the boundary of the real frameThe final detection result accords with the following formula:
wherein Pr (Object) is a target Object judgment parameter, pr (Object) =1 when the classification result has a target defect, and Pr (Object) =0 when the classification result has no target defect; pr (Class) i /Object) is a conditional probability of a class.
CN202010484073.8A 2020-06-01 2020-06-01 Steel plate surface defect classification method based on double-flow neural network model Active CN111627018B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010484073.8A CN111627018B (en) 2020-06-01 2020-06-01 Steel plate surface defect classification method based on double-flow neural network model

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010484073.8A CN111627018B (en) 2020-06-01 2020-06-01 Steel plate surface defect classification method based on double-flow neural network model

Publications (2)

Publication Number Publication Date
CN111627018A CN111627018A (en) 2020-09-04
CN111627018B true CN111627018B (en) 2023-08-04

Family

ID=72272547

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010484073.8A Active CN111627018B (en) 2020-06-01 2020-06-01 Steel plate surface defect classification method based on double-flow neural network model

Country Status (1)

Country Link
CN (1) CN111627018B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114022818B (en) * 2021-11-04 2024-07-16 广州市云景信息科技有限公司 Road black smoke identification method based on Hough transformation and double-current convolutional neural network
CN114841915A (en) * 2022-03-14 2022-08-02 阿里巴巴（中国）有限公司 Tile flaw detection method and system based on artificial intelligence and storage medium

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB201704373D0 (en) * 2017-03-20 2017-05-03 Rolls-Royce Ltd Surface defect detection
CN108596026B (en) * 2018-03-16 2020-06-30 中国科学院自动化研究所 Cross-view gait recognition device and training method based on double-flow generation countermeasure network
CN108345911B (en) * 2018-04-16 2021-06-29 东北大学 Steel plate surface defect detection method based on convolutional neural network multi-stage characteristics
CN109064461A (en) * 2018-08-06 2018-12-21 长沙理工大学 A kind of detection method of surface flaw of steel rail based on deep learning network
CN110490842B (en) * 2019-07-22 2023-07-04 同济大学 Strip steel surface defect detection method based on deep learning
CN111062915B (en) * 2019-12-03 2023-10-24 浙江工业大学 Real-time steel pipe defect detection method based on improved YOLOv3 model

Also Published As

Publication number Publication date
CN111627018A (en) 2020-09-04

Similar Documents

Publication Publication Date Title
CN111612784B (en) Steel plate surface defect detection method based on classification priority YOLO network
CN110660052B (en) Hot-rolled strip steel surface defect detection method based on deep learning
CN111223088B (en) Casting surface defect identification method based on deep convolutional neural network
CN110070008B (en) Bridge disease identification method adopting unmanned aerial vehicle image
CN113298757A (en) Metal surface defect detection method based on U-NET convolutional neural network
CN111627018B (en) Steel plate surface defect classification method based on double-flow neural network model
CN112906816B (en) Target detection method and device based on optical differential and two-channel neural network
CN115439458A (en) Industrial image defect target detection algorithm based on depth map attention
CN115082444B (en) Copper pipe weld defect detection method and system based on image processing
CN109584206B (en) Method for synthesizing training sample of neural network in part surface flaw detection
CN115457044B (en) Pavement crack segmentation method based on class activation mapping
CN117392097A (en) Additive manufacturing process defect detection method and system based on improved YOLOv8 algorithm
CN116245882A (en) Circuit board electronic element detection method and device and computer equipment
Wen et al. PCDNet: Seed operation–based deep learning model for pavement crack detection on 3D asphalt surface
CN113378642B (en) Method for detecting illegal occupation buildings in rural areas
CN116681647A (en) Color-coated sheet surface defect detection method and device based on unsupervised generation
CN116645351A (en) Online defect detection method and system for complex scene
CN116777865A (en) Underwater crack identification method, system, device and storage medium
CN111047614A (en) Feature extraction-based method for extracting target corner of complex scene image
CN115830302A (en) Multi-scale feature extraction and fusion power distribution network equipment positioning identification method
CN115205193A (en) Steel plate surface defect detection method based on microdefect YOLO network
CN112001388B (en) Method for detecting circular target in PCB based on YOLOv3 improved model
CN114821165A (en) Track detection image acquisition and analysis method
CN111369508A (en) Defect detection method and system for metal three-dimensional lattice structure
CN117408967B (en) Board defect detection method and system based on 3D visual recognition

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant