CN114638809A - Multi-scale micro-defect detection method based on PA-MLFPN workpiece surface - Google Patents

Multi-scale micro-defect detection method based on PA-MLFPN workpiece surface Download PDF

Info

Publication number
CN114638809A
CN114638809A CN202210285543.7A CN202210285543A CN114638809A CN 114638809 A CN114638809 A CN 114638809A CN 202210285543 A CN202210285543 A CN 202210285543A CN 114638809 A CN114638809 A CN 114638809A
Authority
CN
China
Prior art keywords
feature
defect
model
mlfpn
workpiece
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210285543.7A
Other languages
Chinese (zh)
Inventor
彭宏京
许名扬
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing Tech University
Original Assignee
Nanjing Tech University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing Tech University filed Critical Nanjing Tech University
Priority to CN202210285543.7A priority Critical patent/CN114638809A/en
Publication of CN114638809A publication Critical patent/CN114638809A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30164—Workpiece; Machine component

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Quality & Reliability (AREA)
  • Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)
  • Image Processing (AREA)

Abstract

The application discloses a multi-scale micro-defect detection method based on a PA-MLFPN workpiece surface, which comprises the following steps: and (3) constructing a PA-MLFPN feature extraction model. A VGG16 main feature extraction network model is used as a first model, PA-MLFPN is used as a middle model, and a prediction layer is used as a later model to form a detection model. And training a detection model by taking the surface image of the multi-scale micro-defect target workpiece as input and the defect positioning frame and classification as output. PA-MLFPN uses the hole convolution of different scaled rates to replace the traditional convolution in the TUM coder part at first, and uses the average pooling operation to carry out down-sampling; adding a bottom-up characteristic enhancement path on the basis of the original TUM to interpolate shallow characteristics into a deep layer; introducing an ECA module to carry out weight distribution on a channel at the second stage of the SFAM; the invention can detect the workpiece surface defect target with variable and smaller dimensions.

Description

Multi-scale micro-defect detection method based on PA-MLFPN workpiece surface
Technical Field
The invention relates to a multi-scale small defect detection method based on a PA-MLFPN workpiece surface, and belongs to the technical field of computer vision.
Background
The task of detecting the surface defects of the workpiece is an indispensable link in workpiece production, but due to the influence of factors such as workpiece production equipment and production process, various defects such as scratches, convex powder, scratches and the like often exist on the surface of the produced workpiece. These defects not only affect the appearance, but also may create various hidden troubles for the subsequent use of the workpiece, such as accelerated aging of the workpiece, which may affect the service life, and even affect the normal use of the workpiece. It is important to use suitable defect detection techniques to improve the production quality of the workpiece. The traditional surface defect detection task is usually completed manually, but the manual detection is often influenced by factors such as personal subjective experience, working environment and mental state, and the like, so that the detection is lack of standardization. Meanwhile, false detection and missing detection are easy to occur when the manual mode detects the defects which move at high speed or are small.
With the development of deep learning, more and more target detection algorithms based on the convolutional neural network are proposed. However, due to the characteristics of the convolutional network structure and the workpiece surface defects, the problems still existing in the task of completing the workpiece surface defect detection based on the convolutional neural network are considered as follows: the randomness of the surface defects of the workpiece leads to variable defect sizes. For example, two different types of defect targets may have similar dimensions, but may be represented in a more complex manner, or may vary in size within the same defect type. However, in the conventional convolutional network model, the feature layer for detecting the object in a specific size range mainly consists of a single-stage feature layer or two adjacent layers for final detection, which results in poor detection performance. In the actual workpiece surface defect detection task, defects are frequently large in small defect targets on the basis of variable shapes. The shallow feature layer in the convolutional neural network has higher resolution, and contains more position and detail information beneficial to small target detection. Deep features have stronger semantic information, but the resolution is very low, and the perception capability on small targets is poor.
Disclosure of Invention
The invention aims to provide a PA-MLFPN-based workpiece surface multi-scale micro-defect detection method to solve the problems in the prior art. A Multi-Level Feature Pyramid Network (MLFPN) is selected to solve the Multi-scale defect target problem by extracting Multi-Level Multi-scale features. And an improvement method is provided for a workpiece surface defect detection task on the basis of MLFPN, a Path-enhancement Multi-Level Feature Pyramid Network (PA-MLFPN) is provided to further utilize shallow features, and the shallow features are embedded into an SSD to detect workpiece surface defects, so that the detection precision of Multi-scale micro-defect targets is improved.
In order to achieve the purpose, the invention provides the following technical scheme: a multi-scale tiny defect detection method based on a PA-MLFPN workpiece surface is characterized in that the following steps A to D are executed to obtain a workpiece surface defect detection model, then a tiny defect sample image of a detected target workpiece is extracted based on VGG16 main feature extraction network, and the workpiece surface defect detection model is applied to obtain preset defect classifications corresponding to the multi-scale tiny defects in the target workpiece image respectively;
step A, constructing a PA-MLFPN characteristic extraction model by taking the micro-defect sample image as input and the multi-layer characteristic pyramid as output based on the MLFPN model
And B: based on a Prediction layer, taking a multi-layer characteristic pyramid image of a micro defect image as input, and taking preset defect classifications respectively corresponding to multi-scale micro defects in the image as output to construct a classification model;
step C: the PA-MLFPN feature extraction model is connected in sequence, the classification model is connected in sequence, and the input end of the classification model is connected with the output end of the PA-MLFPN feature extraction model, so that a detection model which takes the image of the multi-scale micro defects on the surface of the workpiece at a single defect position as input and takes the preset defects corresponding to the multi-scale micro defects in the image as output is formed;
step D: based on preset defect classifications corresponding to preset number of workpiece surface multi-scale micro-defect sample images containing single defect positions and the workpiece surface multi-scale micro-defect sample images, training a detection model by taking the micro-defect sample images as input and the preset defect classifications corresponding to the multi-scale micro-defects in the images as output, and obtaining a workpiece surface defect detection model.
Further, the multi-scale micro defects on the surface of the workpiece comprise preset defects including scratch defects, convex powder defects and scratch defects.
Further, the method for detecting the multi-scale micro defects on the surface of the workpiece based on the PA-MLFPN further includes obtaining multi-scale micro defect images on the surface of the workpiece including a single defect position, dividing data of the multi-scale micro defect image data set on the surface of the workpiece into a training set, a test set and a verification set according to a preset proportion, and specifically includes: on a workpiece production line, photographing and sampling each workpiece at a fixed position by using a sampling device; marking the surface defects of the collected workpiece surface images through Labelimg, and making the images subjected to surface marking through Labelimg into a data set in a VOC format; performing data enhancement on the images in the data set through imgauge data enhancement; the enhanced data set is randomly divided into a (training set + validation set) test set ratio of 9:1 and a training set: validation set ratio of 9: 1.
Further, in the aforementioned step A, the PA-MLFPN model comprises an FFMv1 feature fusion module, a basic feature layer, a multi-stage FFMv2 feature fusion module, a multi-stage improved TUM refinement U-shaped module, and an improved SFAM scale-based feature aggregation module, the FFMv1 feature fusion module is used as an input end of the PA-MLFPN model, the output end of the FFMv1 feature fusion module is connected with the input end of the basic feature layer, the output end of the basic feature layer is respectively connected with the input end of each FFMv2 feature fusion module and the input end of the first-stage improved TUM refinement U-shaped module, the multi-stage FFMv2 feature fusion module is alternately stacked and connected with the multi-stage improved TUM refining U-shaped module, the output end of the improved TUM refining U-shaped module at the last stage is connected with the input end of the improved SFAM dimension-based feature aggregation module, the output end of the improved SFAM scale-based feature aggregation module is used as the output end of the PA-MLFPN model; the step of training the PA-MLFPN model comprises the following steps 101 to 104:
101, based on an FFMv1 feature fusion module, taking conv4_3 and conv5_3 as input, taking a multi-workpiece surface multi-scale micro defect basic feature layer as output, and then entering step 102;
step 102, inputting a multi-scale micro defect basic characteristic layer on the surface of a workpiece as input into a first-stage TUM model for further characteristic extraction, outputting six-scale micro defect basic characteristic graphs output at different levels as output, and then entering step 103;
103, inputting the maximum feature graph in the six-scale basic feature graphs obtained by the basic feature layer and the first-stage TUM model into a first-stage FFMv2 feature fusion module to obtain a first-stage fusion feature graph; then, taking the feature map and the basic feature layer map of the first-level fusion as input, inputting the feature map and the basic feature layer map into a second-level FFMv2 model to obtain a second-level fusion feature map, taking the first-level fusion feature map and the basic feature layer map as input and taking the later-level fusion feature map as output in a mode of alternately stacking a multistage improved TUM refined U-shaped model and an FFMv2 feature fusion module to obtain a shallow feature pyramid feature map, and then entering step 104;
and 104, performing feature aggregation based on an SFAM module, taking pyramid feature layer graphs with the same size in the shallow feature pyramid feature graph, the middle feature pyramid feature graph and the deep feature pyramid feature graph as input, taking a multilayer feature pyramid graph as output, and then performing weight distribution on channels on the obtained multilayer feature pyramid by using an ECA module.
Further, the foregoing TUM refinement U-shaped model is set to (2, 2, 3) respectively in the encoder section using a scaled rate1), performing feature learning by convolution of 3 × 3 holes with step length of 1, performing downsampling by average pooling with step length of 2, and adding an information transmission path from shallow layer to deep layer on the basis of the original U-shaped structure to interpolate detail position information of the pyramid feature of the shallow feature to the deep layer, that is, the feature size of the feature map is reduced to half of the original size, that is, Conv (T) after convolution with convolution kernel size of 3 × 3 and step length of 2 by convolution of Tii) Then and Ci+1Adding element by element to obtain a result, and performing convolution with convolution kernel size of 3 × 3 and step size of 1 to obtain Ti+1The calculation formula is as follows.
Figure BDA0003558061210000041
Further, the foregoing SFAM-based assignment of weights on channels to the obtained multi-layer characteristic pyramid graph using the ECA module specifically includes the following steps:
step V-1: performing global average pooling on the feature layers of the multi-layer feature pyramid graph preliminarily aggregated by the SFAM module to obtain the weights of C channels, and then entering the step V-2;
step V-2: the method comprises the following steps of obtaining the weight coefficients of C channels by realizing local cross-channel interaction information through one-dimensional convolution with the kernel size k, wherein the kernel size k is determined through the function self-adaption of the channel number C, and the calculation formula is as follows:
Figure BDA0003558061210000042
wherein | t |oddAnd b and gamma respectively take 1 and 2, finally, the weight coefficients of the C channels are subjected to Sigmoid function to obtain C values between 0 and 1, the C values respectively correspond to the weight C of the original channel, and the C values are multiplied by the multilayer characteristic pyramid diagram to be weighted and then output, so that the multilevel characteristic pyramid characteristic diagram is obtained.
Further, in the foregoing step a, the fine tuning training of the obtained PA-MLFPN model is further performed, specifically:
during training, the classification loss is calculated by using the Focal loss, the regression loss is calculated by using Smooth L1, and the finally adopted loss function is the combination of the Focal loss and Smooth L1:
L＝Lfl+LSL1
the calculation formula of the classification loss Focal loss is as follows:
Figure BDA0003558061210000043
wherein y' is the prediction output, y is the label of the real sample, alpha is the weight of the positive and negative samples, and gamma is the weight of the samples which are easy to classify and the samples which are difficult to classify; the regression loss Smooth L1 calculation formula is as follows:
Figure BDA0003558061210000044
where x is the difference between the prediction box and the real box.
Further, the obtaining of the defect classification result meeting the preset condition for the actual sample verification specifically includes: setting an evaluation index accuracy rate P and a recall rate R for model evaluation, wherein the calculation formula is as follows:
Figure BDA0003558061210000045
Figure BDA0003558061210000051
wherein TP represents the number of correctly judged defect regions, TN represents the number of correctly judged background regions, FP represents the number of incorrectly judged defect regions, and FN represents the number of incorrectly judged defects as backgrounds;
setting average precision AP for evaluating the detection performance of the model on each type of defect on the test set, wherein the area enclosed by the PR curve and the horizontal and vertical coordinate axes is the AP of the type of defect, and the calculation formula is as follows:
Figure BDA0003558061210000052
setting the average precision average mAP of the detection results of the multi-class defects, wherein the mAP calculation formula of the detection results is as follows:
Figure BDA0003558061210000053
further, in the method for detecting the multi-scale micro defects on the surface of the workpiece based on the PA-MLFPN, the preset defect classification corresponding to the micro defect image is verified, the number of iteration steps of the loop is set to be 100, firstly, the batch size is set to be 32, the learning rate is initialized to be 5e-4, and when the number of iteration steps reaches 50, the batch size is reset to be 16, and the learning rate is 1 e-4.
Further, the defect classification result meeting the preset condition is obtained by verifying the actual sample, an early-stop method is adopted during training, the verification loss is calculated in each iteration, when the verification loss value reaches the local optimum, the iteration is continued for 6 times, and the training is stopped if the model is not converged any more.
Compared with the prior art, the multi-scale tiny defect detection method based on the PA-MLFPN workpiece surface has the following technical effects by adopting the technical scheme: aiming at the characteristics of multiple scales and more tiny defects on the surface of a workpiece, the traditional target detection algorithm cannot well detect the workpiece. A workpiece surface defect detection algorithm based on a path-enhanced multilevel feature pyramid network (PA-MLFPN) is provided. The method further enhances the representation capability of the small defect target on the basis of extracting multi-level and multi-scale defect characteristics, improves the detection precision of the targets with different scales and effectively solves the problem of missed detection of the small defect target.
Drawings
FIG. 1 is a flow chart of the present invention;
FIG. 2 is a diagram of a network model after PA-MLFPN is embedded in the detection model;
FIG. 3 is a schematic diagram of a downsampling portion convolution using holes;
FIG. 4 is a schematic view of feature integration of a feature enhancement path;
FIG. 5 is a modified TUM block diagram;
FIG. 6 is a modified SFAM structural diagram;
fig. 7 is a diagram showing the detection result of the surface defect of the workpiece.
Detailed Description
In order to better understand the technical content of the present invention, specific embodiments are described below with reference to the accompanying drawings.
As shown in FIG. 1, the process steps of the present invention are as follows: firstly, acquiring a surface defect image of a workpiece.
Secondly, marking defects and enhancing data to construct a workpiece surface defect data set, specifically comprising the following steps: on a workpiece production line, a sampling device is used for photographing and sampling each workpiece at a fixed position to construct a workpiece surface defect data set. And marking the surface defects of the collected workpiece surface images through labelimg. Dividing data of a workpiece surface defect data set into a training set, a testing set and a verification set according to a preset proportion, and making an image subjected to surface marking by Labelimg into a data set in a VOC format; performing data enhancement on the images in the data set through imgauge data enhancement; the enhanced data set is randomly divided into a (training set + validation set) test set ratio of 9:1 and a training set: validation set ratio of 9: 1.
Thirdly, building a PA-MLFPN; constructing a PA-MLFPN feature extraction model by taking the micro-defect image as input and taking the multi-layer feature pyramid image of the micro-defect image as output based on the MLFPN model;
fourthly, based on a Prediction layer, taking a multi-layer characteristic pyramid image of the micro defect image as input, and taking preset defect classifications respectively corresponding to the multi-scale micro defects in the image as output to construct a classification model;
fifthly, obtaining a micro defect sample image of the detected target workpiece based on a VGG16 main feature extraction network, wherein a PA-MLFPN feature extraction model is used for extracting first, a classification model is used for extracting second and subsequent connection, and the input end of the classification model is connected with the output end of the PA-MLFPN feature extraction model, so that an image of multi-scale micro defects on the surface of the workpiece at a single defect position is used as input, and preset defects corresponding to the multi-scale micro defects in the image are classified into output detection models;
and sixthly, training the detection model by taking the micro defect sample images as input and the preset defect classifications corresponding to the multi-scale micro defects in the images as output based on preset defect classifications corresponding to preset quantity of the multi-scale micro defect sample images on the surfaces of the workpieces containing the single defect positions and the multi-scale micro defect sample images on the surfaces of the workpieces. And obtaining a workpiece surface defect detection model.
As shown in fig. 2, based on the VGG16 network model, a main feature extraction network is constructed by taking a defect image as an input and taking conv4_3 and conv5_3 layer feature layers in VGG16 as outputs. A network model is formed by connecting PA-MLFPN and Prediction layer in series in the sequence of PA-MLFPN and Prediction layer; the PA-MLFPN model comprises an FFMv1 feature fusion module, a basic feature layer, a multi-level FFMv2 feature fusion module, a multi-level improved TUM refinement U-shaped module and an improved SFAM scale-based feature aggregation module, wherein a basic feature layer (Base feature) is obtained by FFMv1 and conv4_3 and conv5_3 are subjected to preliminary feature fusion for a series of next further feature fusion.
The FFMv1 feature fusion module is used as an input end of the PA-MLFPN model, the output end of the FFMv1 feature fusion module is connected with the input end of a basic feature layer, the output end of the basic feature layer is respectively connected with the input end of each FFMv2 feature fusion module and the input end of the first-stage improved TUM refinement U-shaped module, the multi-stage FFMv2 feature fusion modules are alternately stacked and connected with the multi-stage improved TUM refinement U-shaped modules, the output end of the last-stage improved TUM refinement U-shaped module is connected with the input end of the improved SFAM scale-based feature aggregation module, and the output end of the improved SFAM scale-based feature aggregation module is used as the output end of the PA-MLFPN model; the step of training the PA-MLFPN model comprises the following steps 101 to 104:
101, based on an FFMv1 feature fusion module, taking conv4_3 and conv5_3 as input, taking a multi-workpiece surface multi-scale micro defect basic feature layer as output, and then entering step 102;
step 102, inputting a multi-scale micro defect basic characteristic layer on the surface of a workpiece as input into a first-stage TUM model for further characteristic extraction, outputting six-scale micro defect basic characteristic graphs output at different levels as output, and then entering step 103;
step 103, specifically: inputting the maximum feature map in six scales of basic feature maps obtained by the basic feature layer and the first-stage TUM model into a first-stage FFMv2 feature fusion module to obtain a first-stage fusion feature map; then, taking the feature map and the basic feature layer map of the first-level fusion as input, inputting the feature map and the basic feature layer map into a second-level FFMv2 model to obtain a second-level fusion feature map, taking the first-level fusion feature map and the basic feature layer map as input and taking the later-level fusion feature map as output in a mode of alternately stacking a multistage improved TUM refined U-shaped model and an FFMv2 feature fusion module to obtain a shallow feature pyramid feature map, and then entering step 104;
and 104, performing feature aggregation based on an SFAM module, taking pyramid feature layer graphs with the same size in the shallow feature pyramid feature graph, the middle feature pyramid feature graph and the deep feature pyramid feature graph as input, taking a multilayer feature pyramid graph as output, and then performing weight distribution on channels on the obtained multilayer feature pyramid by using an ECA module.
And then inputting the multi-layer characteristic pyramid image of the micro defect image as input, and inputting the defect classification corresponding to the multi-scale micro defects in the image as output to a Prediction layer of a Prediction layer to obtain the defect classification corresponding to the micro defects on the surface of the workpiece.
As shown in fig. 3, the downsampling section uses a schematic of hole convolution, the TUM refinement U-mode model performs feature learning at the encoder section using 3 × 3 hole convolutions with a step size of 1 with scaled rates set to (2, 2, 3, 3, 1), respectively, and downsampling using an average pooling operation with a step size of 2.
As shown in FIG. 4, the feature combination manner of the feature enhancement pathSpecifically, on the basis of an original U-shaped structure, an information transmission path from a shallow layer to a deep layer is added to interpolate detail position information of a shallow layer feature pyramid feature to the deep layer, that is, Ti is subjected to convolution with a convolution kernel size of 3 × 3 and a step size of 2, the feature map size is reduced to half of the original size, that is, conv (Ti), and then element-by-element addition is performed on the feature map and Ci +1, and the obtained result is subjected to convolution with a convolution kernel size of 3 × 3 and a step size of 1 to obtain Ti +1, wherein the calculation formula is as follows:
Figure BDA0003558061210000081
as shown in fig. 5, in the improved TUM structure, the TUM refinement U-mode performs feature learning using a convolution of 3 × 3 holes with a step size of 1, with scaled rates set to (2, 2, 3, 3, 1) respectively, in the encoder section, and downsampling using an average pooling operation with a step size of 2. And acquiring multi-scale information by setting different scaled rates to obtain different sizes of receptive fields, avoiding information loss in the process of continuous convolution of down-sampling to reserve more shallow defect target information, and adjusting and outputting the number of channels by further convolution with the step length of 1 and the number of channels of 128 by 1 multiplied by 1 on the basis.
As shown in fig. 6, the structure of the improved SFAM; the specific process of adopting the ECA module in the second stage of the improved SFAM is as follows: instead of two fully connected layers, ECA modules are introduced to perform a one-dimensional convolution of size k, where k is adaptively determined by a function of the number of channels C. Therefore, the dimension reduction operation is avoided, and the interaction of local cross-channel information is carried out through the one-dimensional convolution with the self-adaptive size. Specifically, the feature layer initially aggregated in the first stage of the SFAM is first subjected to global average pooling to obtain the weight of each channel, and if there are C channels, the weights of the C channels are obtained in total.
And then, realizing local cross-channel interaction information to acquire weight through one-dimensional convolution with the size of k, wherein the kernel size k needs to be determined in a self-adaptive manner:
Figure BDA0003558061210000082
wherein | t |oddThe representation takes the nearest odd number and b and γ take 1 and 2, respectively, in the experiment. And finally, obtaining C values between 0 and 1 from the output of the previous step through a Sigmoid function, respectively corresponding to the weights of the original channels, multiplying the values by the input characteristics to perform weighting and then outputting.
The fine tuning training of the constructed model on the workpiece surface defect data set specifically comprises the following steps: during training, the classification loss is calculated by using Focal loss, and the regression loss is calculated by using Smooth L1. The loss function ultimately employed is the combination of Focal loss and Smooth L1: l ═ Lfl+LSL1The formula for calculating the classification loss is as follows:
Figure BDA0003558061210000083
wherein y is the prediction output, y is the label of the true sample, α is the positive and negative sample weight, γ is the sample weight of the easy classification and the sample weight of the difficult classification. The regression loss Smooth L1 calculation formula is as follows:
Figure BDA0003558061210000091
where x is the difference between the prediction box and the real box.
To evaluate the performance of the algorithms presented herein, the associated evaluation index Precision (Precision, P) and Recall (Recall, R) were used to perform model evaluation.
Wherein TP, TN and FP respectively represent the number of correctly determined defects, the number of correctly determined background regions and the number of erroneously determined defects for the background:
Figure BDA0003558061210000092
the Average Precision (AP) is used to evaluate the detection performance of the model on each type of defect on the test set, and the area enclosed by the PR curve and the horizontal and vertical axes is the AP of the defect as shown in the following formula:
Figure BDA0003558061210000093
the detection result of the multi-class defects is evaluated by adopting an Average Precision mean (mAP), and the mAP of the detection result is shown as the formula:
Figure BDA0003558061210000094
the network is trained by adopting a transfer learning method, a weight file is obtained by pre-training a VOC data set, and then fine adjustment is carried out on a workpiece surface defect data set.
The loop iteration step number is set to 100, the batch size is first set to 32, the learning rate is initialized to 5e-4, and when the iteration step number reaches 50, the batch size is reset to 16, and the learning rate is 1 e-4. And during training, an early stopping method (early stopping) is adopted to avoid overfitting caused by continuous training, the verification loss is calculated in each iteration, when the verification loss value reaches local optimum, the iteration is continued for 6 times, and the training is stopped if the model is not converged any more.
Fig. 7 is a result of detecting a surface defect of a workpiece, specifically, a scratch defect, a convex defect, and a scratch defect, by inputting a minute defect image in an implementation.
Aspects of the invention are described herein with reference to the accompanying drawings, in which a number of illustrative embodiments are shown. Embodiments of the invention are not limited to those shown in the drawings. It should be understood that the present invention can be realized by any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail, since the disclosed concepts and embodiments are not limited to any embodiment. In addition, some aspects of the present disclosure may be used alone, or in any suitable combination with other aspects of the present disclosure.
Although the present invention has been described with reference to the preferred embodiments, it is not intended to be limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, the protection scope of the present invention should be determined by the appended claims.
Compared with the prior art, the multi-scale tiny defect detection method based on the PA-MLFPN workpiece surface has the following technical effects by adopting the technical scheme: aiming at the characteristics of multiple scales and more tiny defects on the surface of a workpiece, the traditional target detection algorithm cannot well detect the workpiece. A workpiece surface defect detection algorithm based on a path-enhanced multilevel feature pyramid network (PA-MLFPN) is provided. The method further enhances the representation capability of the small defect target on the basis of extracting multi-level and multi-scale defect characteristics, improves the detection precision of the targets with different scales and effectively solves the problem of missed detection of the small defect target.

Claims (10)

1. A multi-scale tiny defect detection method based on a PA-MLFPN workpiece surface is characterized in that the following steps A to D are executed to obtain a workpiece surface defect detection model, then a tiny defect sample image of a detected target workpiece is extracted based on VGG16 main feature extraction network, and the workpiece surface defect detection model is applied to obtain preset defect classifications corresponding to the multi-scale tiny defects in the target workpiece image respectively;
step A, constructing a PA-MLFPN characteristic extraction model by taking the micro-defect sample image as input and the multi-layer characteristic pyramid as output based on the MLFPN model
And B: based on a Prediction layer, taking a multi-layer characteristic pyramid image of a micro defect image as input, and taking preset defect classifications respectively corresponding to multi-scale micro defects in the image as output to construct a classification model;
and C: the PA-MLFPN feature extraction model is connected in sequence, the classification model is connected in sequence, and the input end of the classification model is connected with the output end of the PA-MLFPN feature extraction model, so that a detection model which takes the image of the multi-scale micro defects on the surface of the workpiece at a single defect position as input and takes the preset defects corresponding to the multi-scale micro defects in the image as output is formed;
step D: based on preset defect classifications corresponding to preset number of workpiece surface multi-scale micro-defect sample images containing single defect positions and the workpiece surface multi-scale micro-defect sample images, training a detection model by taking the micro-defect sample images as input and the preset defect classifications corresponding to the multi-scale micro-defects in the images as output, and obtaining a workpiece surface defect detection model.
2. The method for detecting the multi-scale micro defects on the surface of the PA-MLFPN-based workpiece as claimed in claim 1, wherein the multi-scale micro defects on the surface of the workpiece comprise preset defects including scratch defects, convex powder defects and scratch defects.
3. The method for detecting the multi-scale micro defects on the surface of the PA-MLFPN-based workpiece according to claim 1, further comprising the steps of obtaining multi-scale micro defect images of the surface of each workpiece including a single defect position, dividing data of the multi-scale micro defect image data set of the surface of the workpiece into a training set, a testing set and a verification set according to a preset proportion, and specifically comprising the following steps: on a workpiece production line, photographing and sampling each workpiece at a fixed position by using a sampling device; marking the surface defects of the collected workpiece surface images through Labelimg, and making the images subjected to surface marking through Labelimg into a data set in a VOC format; performing data enhancement on the images in the data set through imgauge data enhancement; the enhanced data set is randomly divided into a (training set + validation set) test set ratio of 9:1 and a training set: validation set ratio of 9: 1.
4. The method for detecting the multi-scale micro defects on the surface of the PA-MLFPN-based workpiece according to claim 1, wherein in the step A, the PA-MLFPN model comprises an FFMv1 feature fusion module, a basic feature layer, a multi-level FFMv2 feature fusion module, a multi-level improved TUM refinement U-shaped module and an improved SFAM scale-based feature aggregation module, the FFMv1 feature fusion module is used as the input end of the PA-MLFPN model, the output end of the FFMv1 feature fusion module is connected with the input end of the basic feature layer, the output end of the basic feature layer is connected with the input end of each FFMv2 feature fusion module and the input end of the first-level improved TUM refinement U-shaped module, the multi-level FFMv2 feature fusion module is alternately connected with the multi-level improved TUM refinement U-shaped module in a stacking manner, the output end of the last-level improved TUM refinement U-shaped module is connected with the input end of the improved SFAM scale-based feature aggregation module, the output end of the improved SFAM scale-based feature aggregation module is used as the output end of the PA-MLFPN model; the step of training the PA-MLFPN model comprises the following steps 101 to 104:
101, based on an FFMv1 feature fusion module, taking conv4_3 and conv5_3 as input, taking a multi-workpiece surface multi-scale micro defect basic feature layer as output, and then entering step 102;
step 102, inputting a multi-scale micro defect basic characteristic layer on the surface of a workpiece as input into a first-stage TUM model for further characteristic extraction, outputting six-scale micro defect basic characteristic graphs output at different levels as output, and then entering step 103;
103, inputting a maximum feature map in six scales of basic feature maps obtained by a basic feature layer and a first-stage TUM model into a first-stage FFMv2 feature fusion module to obtain a first-stage fusion feature map; then, taking the feature map and the basic feature layer map of the first-level fusion as input, inputting the feature map and the basic feature layer map into a second-level FFMv2 model to obtain a second-level fusion feature map, taking the first-level fusion feature map and the basic feature layer map as input and taking the later-level fusion feature map as output in a mode of alternately stacking a multistage improved TUM refined U-shaped model and an FFMv2 feature fusion module to obtain a shallow feature pyramid feature map, and then entering step 104;
and 104, performing feature aggregation based on an SFAM module, taking pyramid feature layer graphs with the same size in the shallow feature pyramid feature graph, the middle feature pyramid feature graph and the deep feature pyramid feature graph as input, taking a multilayer feature pyramid graph as output, and then performing weight distribution on channels on the obtained multilayer feature pyramid by using an ECA module.
5. The PA-MLFPN-based workpiece surface of claim 3The method for detecting the scale tiny defects is characterized in that the TUM refinement U-shaped model performs feature learning by convolution of 3 x 3 holes with step length of 1 and scaled rates of (2, 2, 3, 3, 1) respectively in an encoder part, performs downsampling by average pooling operation with step length of 2, adds an information transmission path from a shallow layer to a deep layer on the basis of an original U-shaped structure to interpolate detail position information of a shallow layer feature pyramid feature to the deep layer, namely Ti is convolved with convolution kernel size of 3 x 3 and step length of 2, and the feature map size is reduced to half of the original Conv (T)i) Then and Ci+1Adding element by element to obtain a result, and performing convolution with convolution kernel size of 3 × 3 and step size of 1 to obtain Ti+1The calculation formula is as follows.
Figure FDA0003558061200000021
。
6. The PA-MLFPN-based workpiece surface multi-scale micro-defect detection method as claimed in claim 3, wherein the SFAM-based assignment of weights on channels to the obtained multilayer feature pyramid maps by using ECA module comprises the following specific steps:
step V-1: performing global average pooling on the feature layers of the multi-layer feature pyramid graph preliminarily aggregated by the SFAM module to obtain the weights of C channels, and then entering the step V-2;
step V-2: the method comprises the following steps of obtaining weight coefficients of C channels by realizing local cross-channel interaction information through one-dimensional convolution with the kernel size of k, wherein the kernel size k is determined through the function self-adaption of the channel number C, and the calculation formula is as follows:
Figure FDA0003558061200000031
wherein | t |oddRepresenting that the nearest odd number is taken, b and gamma respectively take 1 and 2, and finally, the weight coefficients of C channels obtain C values between 0 and 1 through Sigmoid function, and respectivelyAnd weighting and outputting the weight C corresponding to the original channel by multiplying the weight C by the multi-layer characteristic pyramid characteristic diagram to obtain the multi-stage characteristic pyramid characteristic diagram.
7. The method for detecting the multi-scale micro-defects on the surface of the PA-MLFPN-based workpiece as claimed in claim 1, wherein the step A further comprises performing fine tuning training on the obtained PA-MLFPN model, specifically:
during training, the classification loss is calculated by adopting the Focal loss, the regression loss is calculated by adopting Smooth L1, and the finally adopted loss function is the combination of the Focal loss and Smooth L1:
L＝Lfl+LSL1
the calculation formula of the classification loss Focal loss is as follows:
Figure FDA0003558061200000032
wherein y' is the prediction output, y is the label of the real sample, alpha is the weight of the positive and negative samples, and gamma is the weight of the samples which are easy to classify and the samples which are difficult to classify; the regression loss Smooth L1 calculation formula is as follows:
Figure FDA0003558061200000033
where x is the difference between the prediction box and the real box.
8. The PA-MLFPN workpiece surface-based multi-scale micro-defect detection method as claimed in claim 1, wherein a defect classification result meeting a preset condition is obtained for actual sample verification, specifically:
setting an evaluation index accuracy rate P and a recall rate R for model evaluation, wherein the calculation formula is as follows:
Figure FDA0003558061200000034
Figure FDA0003558061200000035
wherein TP represents the number of correctly judged defect regions, TN represents the number of correctly judged background regions, FP represents the number of incorrectly judged defect regions, and FN represents the number of incorrectly judged defects as backgrounds;
setting average precision AP for evaluating the detection performance of the model on each type of defect on the test set, wherein the area enclosed by the PR curve and the horizontal and vertical coordinate axes is the AP of the type of defect, and the calculation formula is as follows:
Figure FDA0003558061200000041
setting the average precision average mAP of the detection results of the multi-class defects, wherein the mAP calculation formula of the detection results is as follows:
Figure FDA0003558061200000042
。
9. the PA-MLFPN workpiece surface multi-scale micro-defect detection method based on the claim 1 is characterized in that in the verification of the preset defect classification corresponding to the micro-defect image, the iteration step number of the loop is set to be 100, firstly, the batch size is set to be 32, the learning rate is initialized to be 5e-4, and when the iteration step number reaches 50, the batch size is reset to be 16, and the learning rate is 1 e-4.
10. The PA-MLFPN workpiece surface multi-scale micro-defect detection method as claimed in claim 1, wherein a defect classification result meeting preset conditions is obtained for actual sample verification, an early stop method is adopted during training, verification loss is calculated every iteration, when the verification loss value reaches local optimum, iteration is continued for 6 times, and training is stopped if the model does not converge any more.
CN202210285543.7A 2022-03-22 2022-03-22 Multi-scale micro-defect detection method based on PA-MLFPN workpiece surface Pending CN114638809A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210285543.7A CN114638809A (en) 2022-03-22 2022-03-22 Multi-scale micro-defect detection method based on PA-MLFPN workpiece surface

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210285543.7A CN114638809A (en) 2022-03-22 2022-03-22 Multi-scale micro-defect detection method based on PA-MLFPN workpiece surface

Publications (1)

Publication Number Publication Date
CN114638809A true CN114638809A (en) 2022-06-17

Family

ID=81950699

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210285543.7A Pending CN114638809A (en) 2022-03-22 2022-03-22 Multi-scale micro-defect detection method based on PA-MLFPN workpiece surface

Country Status (1)

Country Link
CN (1) CN114638809A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115760990A (en) * 2023-01-10 2023-03-07 华南理工大学 Identification and positioning method of pineapple pistil, electronic equipment and storage medium

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115760990A (en) * 2023-01-10 2023-03-07 华南理工大学 Identification and positioning method of pineapple pistil, electronic equipment and storage medium
CN115760990B (en) * 2023-01-10 2023-04-21 华南理工大学 Pineapple pistil identification and positioning method, electronic equipment and storage medium

Similar Documents

Publication Publication Date Title
CN112101426B (en) Unsupervised learning image anomaly detection method based on self-encoder
CN110163302B (en) Indicator diagram identification method based on regularization attention convolution neural network
CN106875373B (en) Mobile phone screen MURA defect detection method based on convolutional neural network pruning algorithm
CN110717481B (en) Method for realizing face detection by using cascaded convolutional neural network
CN108549841A (en) A kind of recognition methods of the Falls Among Old People behavior based on deep learning
CN110188774B (en) Eddy current scanning image classification and identification method based on deep learning
CN107194418B (en) Rice aphid detection method based on antagonistic characteristic learning
CN111160249A (en) Multi-class target detection method of optical remote sensing image based on cross-scale feature fusion
CN110765865B (en) Underwater target detection method based on improved YOLO algorithm
CN114372955A (en) Casting defect X-ray diagram automatic identification method based on improved neural network
CN112818969A (en) Knowledge distillation-based face pose estimation method and system
CN113297929A (en) Convolutional neural network microseismic monitoring waveform identification method based on whole-process visualization
CN111161224A (en) Casting internal defect grading evaluation system and method based on deep learning
CN112818871B (en) Target detection method of full fusion neural network based on half-packet convolution
CN112819063B (en) Image identification method based on improved Focal loss function
CN112258470B (en) Intelligent industrial image critical compression rate analysis system and method based on defect detection
CN115409797A (en) PCB defect image detection method based on improved deep learning algorithm
CN110751195A (en) Fine-grained image classification method based on improved YOLOv3
CN114565048A (en) Three-stage pest image identification method based on adaptive feature fusion pyramid network
CN111127454A (en) Method and system for generating industrial defect sample based on deep learning
CN114596316A (en) Road image detail capturing method based on semantic segmentation
CN114638809A (en) Multi-scale micro-defect detection method based on PA-MLFPN workpiece surface
CN117237279A (en) Blind quality evaluation method and system for non-uniform distortion panoramic image
CN115565020A (en) Tooth surface damage identification method and device based on improved neural network
CN109740672B (en) Multi-stream feature distance fusion system and fusion method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination