WO2024255919A1 - 瑕疵检测方法及相关设备 - Google Patents

瑕疵检测方法及相关设备 Download PDF

Info

Publication number
WO2024255919A1
WO2024255919A1 PCT/CN2024/107018 CN2024107018W WO2024255919A1 WO 2024255919 A1 WO2024255919 A1 WO 2024255919A1 CN 2024107018 W CN2024107018 W CN 2024107018W WO 2024255919 A1 WO2024255919 A1 WO 2024255919A1
Authority
WO
WIPO (PCT)
Prior art keywords
defect
image
layer
category
convolution
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2024/107018
Other languages
English (en)
French (fr)
Inventor
黄伟强
陈哲玮
张容琛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Laboratory For Artificial Intelligence In Design Ltd
Royal College of Art
Hong Kong Polytechnic University HKPU
Original Assignee
Laboratory For Artificial Intelligence In Design Ltd
Royal College of Art
Hong Kong Polytechnic University HKPU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Laboratory For Artificial Intelligence In Design Ltd, Royal College of Art, Hong Kong Polytechnic University HKPU filed Critical Laboratory For Artificial Intelligence In Design Ltd
Publication of WO2024255919A1 publication Critical patent/WO2024255919A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks

Definitions

  • the present disclosure relates to the field of computers and defect detection technology, and in particular, to a defect detection method, device and system, electronic equipment, computer-readable storage medium and computer program product.
  • products or items may have various defects. Before these products or items leave the factory, they need to be tested for defects to prevent defective products or items from being circulated in the market.
  • the present disclosure provides a defect detection method, comprising: obtaining an image to be processed; processing the image to be processed by a feature extraction module in a deep learning defect detection model, the feature extraction module comprising an intermediate feature layer, the intermediate feature layer comprising a discard layer, the discard layer being used to randomly discard features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains a plurality of features after random discarding after being processed by the feature extraction module; processing the plurality of features after random discarding by a detection head in the deep learning defect detection model, to obtain a plurality of category prediction results and a plurality of position prediction results of defects in the image to be processed; obtaining a category detection result, a position detection result, a category uncertainty and a position uncertainty of the defect in the image to be processed according to the plurality of category prediction results and the plurality of position prediction results of the defect.
  • the present disclosure provides a defect detection device, comprising: an image acquisition module, for acquiring an image to be processed; an image processing module, for processing the image to be processed by a feature extraction module in a deep learning defect detection model, wherein the feature extraction module comprises an intermediate feature layer, wherein the intermediate feature layer comprises a discarding layer, and the discarding layer is used to randomly discard features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains a plurality of features after random discarding processing after being processed by the feature extraction module; the image processing module is also used to process the plurality of features after random discarding processing by a detection head in the deep learning defect detection model.
  • the image processing module is used to obtain multiple category prediction results and multiple position prediction results of the defects in the image to be processed; the image processing module is also used to obtain category detection results, position detection results, category uncertainty and position uncertainty of the defects in the image to be processed according to the multiple category prediction results and multiple position prediction results of the defects.
  • the present disclosure provides a defect detection system, comprising: an image acquisition component, used to obtain the image to be processed of the object to be detected in the production process; a defect detection device, connected to the image acquisition component, used to obtain the image to be processed from the image acquisition component; wherein the defect detection device comprises: an image processing module, used to process the image to be processed by a feature extraction module in a deep learning defect detection model, the feature extraction module comprising an intermediate feature layer, the intermediate feature layer comprising a discarding layer, the discarding layer being used to randomly discard the features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains a plurality of features after random discarding processing after being processed by the feature extraction module; the plurality of features after random discarding processing are processed by the detection head in the deep learning defect detection model to obtain a plurality of category prediction results and a plurality of position prediction results of the defects in the image to be processed; the category detection result, the position detection result, the category uncertainty and the position uncertainty of the defects in the image to be processed
  • An embodiment of the present disclosure provides a defect detection system, comprising: an image capture device for collecting the image to be processed of the object to be detected in the production process; a processor and a memory for storing computer programs; when the processor executes the computer program, the steps in the method described in any embodiment of the present disclosure are implemented.
  • An embodiment of the present disclosure provides an electronic device, including: one or more processors; and a memory configured to store one or more programs, so that when the one or more programs are executed by the one or more processors, the electronic device implements the method described in any embodiment of the present disclosure.
  • An embodiment of the present disclosure provides a computer-readable storage medium on which a computer program is stored.
  • the computer program When the computer program is run on a computer, the computer implements the method described in any embodiment of the present disclosure.
  • the embodiments of the present disclosure provide a computer program product, including a computer program.
  • the computer program When the computer program is executed by a computer, the method described in the embodiments of the present disclosure is implemented.
  • FIG. 1 is a flow chart of a defect detection method according to an exemplary embodiment of the present disclosure.
  • FIG. 2 is a schematic diagram of a deep learning defect detection model according to an exemplary embodiment of the present disclosure.
  • FIG. 3 is a schematic diagram of a deep learning defect detection model according to another exemplary embodiment of the present disclosure.
  • FIG4 is a schematic diagram of the first cross-stage convolution block and the second cross-stage convolution block shown in FIG3 .
  • FIG. 5 is a schematic diagram of the first intermediate layer and the second intermediate layer shown in FIG. 4 .
  • FIG. 6 is a schematic diagram of the first depth-wise separable convolutional layer and the second depth-wise separable convolutional layer shown in FIG. 5 .
  • FIG. 7 is a schematic diagram of the third cross-stage convolution block shown in FIG. 3 .
  • FIG8 is a schematic diagram of a deep learning defect detection model according to yet another exemplary embodiment of the present disclosure.
  • FIG. 9 is a schematic diagram of a deep learning defect detection model according to yet another exemplary embodiment of the present disclosure.
  • FIG. 10 is a schematic diagram of a spatial pyramid pooling block according to an exemplary embodiment of the present disclosure.
  • FIG. 11 is a schematic diagram of an alternative embodiment of a spatial pyramid pooling block in an embodiment of the present disclosure.
  • FIG12 is a schematic diagram of the attention block shown in FIG9 .
  • FIG. 13 is a schematic diagram of a defect detection system according to an exemplary embodiment of the present disclosure.
  • FIG. 14 is a schematic diagram of a defect detection system according to another exemplary embodiment of the present disclosure.
  • FIG. 15 is a schematic diagram of a defect detection system according to yet another exemplary embodiment of the present disclosure.
  • FIG. 16 is a schematic structural diagram of a defect detection device according to an exemplary embodiment of the present disclosure.
  • FIG. 17 schematically shows a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.
  • module refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof.
  • a processor or multiple processors or memories
  • each module or unit can be part of an overall module or unit that includes the function of the module or unit.
  • the terms “first”, “second”, and “third” are used for descriptive purposes only and are not to be understood as indicating or implying relative importance; the term “plurality” refers to two or more, unless otherwise expressly defined.
  • the terms “installed”, “connected”, “connected”, “fixed”, etc. should be understood in a broad sense. For example, “connected” can be a fixed connection, a detachable connection, or an integral connection; “connected” can be a direct connection or an indirect connection through an intermediate medium.
  • the specific meanings of the above terms in the present disclosure can be understood according to the specific circumstances.
  • the defect detection method provided in the embodiment of FIG1 can be executed by any electronic device, such as a terminal and/or a server, and the present disclosure does not limit this. As shown in FIG1 , the method provided in the embodiment of the present disclosure includes the following steps.
  • the image to be processed in the disclosed embodiment can be any image of an object to be detected that needs to be detected to see if it contains defects or flaws.
  • the object to be detected can be, for example, a sanitary and personal care product, such as medical and sanitary products, such as surgical gowns, protective clothing, disinfection wraps, masks, diapers, civilian rags, wipes, wet face towels, magic towels, soft towel rolls, beauty cosmetics, etc.; it can also be a home decoration product, such as wall coverings, tablecloths, bed sheets, bedspreads, etc.; it can also be a clothing product, such as lining, adhesive interlining, flakes, shaping cotton, various synthetic leather base fabrics, etc.; it can also be an industrial product, such as filter materials, insulating materials, cement packaging bags, geotextiles, covering fabrics, etc.; it can also be an agricultural product, such as crop protection fabrics, seedling cloths, irrigation fabrics, thermal insulation curtains, etc.; it can also be other products, such as space cotton, thermal insulation
  • the object to be detected is a mask (including a mask that has been sewn and a mask in the cutting process), and the corresponding image to be processed is a mask image.
  • the object to be detected is a mask cloth in high-speed motion in a production line (hereinafter referred to as a production line), and the corresponding image to be processed is a mask cloth image.
  • the mask cloth can be a non-woven fabric, but the present disclosure is not limited to this, and other suitable materials can also be used as the mask cloth.
  • the present disclosure relates to the field of non-woven material detection technology, mainly for the mask cloth defect detection subdivision field, and the method can also be referred to as a non-woven mask cloth defect detection method or a mask cloth defect detection method.
  • the process of making masks using non-woven fabrics mainly includes material preparation, cutting, sewing and finishing. After the raw materials are prepared, cutting is carried out. The non-woven fabric is laid flat and cut with a cutting machine to cut out the main part of the mask. Next is the sewing process, sewing the non-woven fabric, nose bridge strips, ear straps and other parts together to form a complete mask. Finally, finishing and quality inspection are required to ensure that each mask meets the quality standards.
  • the relevant technology mainly involves the finished mask.
  • the defect detection of masks does not involve the defect detection of mask cloth on the mask production line, and it is impossible to achieve quality management and control in the production stage.
  • the image to be processed is processed by a feature extraction module in a deep learning defect detection model
  • the feature extraction module includes an intermediate feature layer
  • the intermediate feature layer includes a discard layer
  • the discard layer is used to randomly discard the features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains multiple features after random discarding processing after being processed by the feature extraction module.
  • the deep learning defect detection model in the embodiment of the present disclosure is a model based on deep learning, which can process images, detect whether there are defects in the image, and predict the defect information of the defect when the defect is detected.
  • the defect information includes, for example, any one or more of the size, position, category, etc. of the defect.
  • the deep learning defect detection model can be based on a target detection model, such as any one or more of the YOLO (You Only Look Once) series (such as YOLOv3, YOLOv4, Yolov5, etc.), Faster R-CNN, SSD, CNN (convolutional neural network), recurrent neural network (RNN) and transformer (Transformer).
  • the YOLO model is used as an example, but the present disclosure is not limited to this.
  • the deep learning defect detection model in the disclosed embodiment is a real-time target detection model, which divides the input image to be processed into grids, each grid is responsible for detecting a target (i.e., defect), and predicting the bounding box (also referred to as anchor box) and category of the target.
  • the deep learning defect detection model includes a feature extraction module, which is the backbone network (Backbone) in the deep learning defect detection model, and is mainly responsible for extracting the features of the input image to be processed.
  • the backbone network may include convolution layers, pooling layers, and other possible network layers, etc., which can gradually extract deep features in the image to be processed, and are used to convert the image to be processed into multi-layer feature maps.
  • the backbone network may include a Conv module (convolution module), a C3 module (composed of three standard convolution layers, each convolution layer having a specific convolution kernel size and step size) and an SPPF module (spatial pyramid pooling layer, used to aggregate features at multiple scales), etc.
  • Conv module convolution module
  • C3 module Composed of three standard convolution layers, each convolution layer having a specific convolution kernel size and step size
  • SPPF module spatial pyramid pooling layer, used to aggregate features at multiple scales
  • Backbone's intermediate feature layer refers to the hidden layer (excluding feature pyramid and detection head) located after the input layer (if there is an output layer, it is located between the input layer and the output layer) in the Backbone network, which is used to generate feature maps of the image to be processed.
  • These intermediate feature layers capture the information of the image to be processed at different levels and scales.
  • the information in the intermediate feature layer can not only be used to generate the final detection results, but sometimes it can also be used for other tasks, such as feature fusion or further analysis.
  • the intermediate feature layer of Backbone can be fused with other network parts (such as FPN, i.e., feature pyramid network/module) to improve detection performance.
  • a dropout layer is added to the intermediate feature layer of the feature extraction module of the deep learning defect detection model.
  • Dropout simulates the integrated learning between different sub-networks by randomly “closing” or “erasing” a part of the neurons (or features) in the hidden layer so that the deleted neurons no longer transmit signals. This can effectively prevent the model from overfitting and improve its generalization ability on unseen data (i.e., data not included in the training data set).
  • unseen data i.e., data not included in the training data set.
  • Dropout can be regarded as a Bayesian approximation method to a certain extent.
  • the sparsity of the model parameters is simulated, which has a certain similarity with the posterior distribution of the parameters in Bayesian inference.
  • Dropout does not directly simulate the posterior distribution of Bayes, but provides a regularization effect through its randomness, which helps to reduce overfitting.
  • dropout is added to the intermediate feature layer of Backbone, and its ability to prevent overfitting can be used to improve the generalization performance of the model.
  • the complexity and number of parameters of the neural network can also be reduced. If dropout is only enabled during the training phase and disabled during the inference phase, it is used to prevent the model from overfitting. In the disclosed embodiment, dropout is enabled during both the training phase and the inference phase to simulate Bayesian uncertainty.
  • the detection head in the deep learning defect detection model processes the multiple features after random discarding to obtain multiple category prediction results and multiple position prediction results of the defects in the image to be processed.
  • the disclosed embodiment can randomly characterize the generation process of the network structure of the hidden layer by randomly closing the neurons in the hidden layer (making the node values of a part of the hidden layer 0), such as the number of layers of the neural network, the connection mode of the neurons in each layer, etc., and can obtain multiple different network structures of the intermediate feature layer, so as to make multiple predictions for the same image to be processed (the same target sample image in the training process) through multiple different network structures of the intermediate feature layer, thereby obtaining multiple category prediction results and multiple position prediction results for each defect in the image to be processed.
  • Uncertainty estimation is to calculate the distribution index of the results of multiple predictions.
  • the category prediction result refers to the category result of the defect predicted by the detection head of the deep learning defect detection model (such as the depthwise separable convolution block therein).
  • the position prediction result refers to the position of the defect in the image (including the image) predicted by the detection head of the deep learning defect detection model (such as the depthwise separable convolution block therein).
  • the position of the defect in the image to be processed and/or the target sample image may be an absolute position or a relative position.
  • the category prediction result may include the predicted probability that the defect belongs to each category, and the position prediction result may include the predicted center coordinates, width, height and other information of the bounding box of the defect in the image.
  • a category detection result, a position detection result, a category uncertainty, and a position uncertainty of the defect in the image to be processed are obtained according to the multiple category prediction results and the multiple position prediction results of the defect.
  • the detection head of the deep learning defect detection model in the disclosed embodiment is mainly responsible for predicting the category of defects in the image to be processed, the location of the bounding box, and the confidence, and can also predict the category uncertainty of the predicted category of the defect and the location uncertainty of the location.
  • the detection head can use multiple convolutional layers to further extract and integrate features, and then output multiple category prediction results and multiple location prediction results of the defect, and based on the multiple category prediction results and multiple location prediction results of the defect, obtain the category detection result, location detection result, category uncertainty, and location uncertainty of the defect in the image to be processed.
  • the category detection result and the position detection result refer to the category result of the defect output by the detection head of the deep learning defect detection model and the position of the defect in the image, respectively.
  • category uncertainty refers to an indicator used to evaluate the degree of certainty of whether the predicted category result matches the true category of the defect when the deep learning defect detection model is used to predict the category prediction result of the defect in the image (including the image to be processed and/or the target sample image). The higher the category uncertainty, the more uncertain the category prediction result of the deep learning defect detection model for the defect is, and the lower the category uncertainty, the more certain the category prediction result of the deep learning defect detection model for the defect is.
  • position uncertainty refers to an indicator of the degree of certainty of whether the predicted position result matches the actual position of the defect when the deep learning defect detection model is used to predict the position prediction result of the defect in the image (including the image to be processed and/or the target sample image).
  • the higher the position uncertainty the more uncertain the position prediction result of the defect by the deep learning defect detection model is, and the lower the position uncertainty, the more certain the position prediction result of the defect by the deep learning defect detection model is.
  • the confidence of the output bounding box in addition to predicting the position and category of the bounding box, can also be predicted.
  • the confidence of the predicted output is a value between 0 and 1, which is used to indicate how sure the model is about the detected defect.
  • the confidence can indicate the probability (also called target confidence) of whether there is a defect (also called a defect) in the bounding box, that is, whether the bounding box is just the background or contains defects.
  • it can also indicate the degree of match between the bounding box predicted by the model and the real box of the defect when the bounding box has a defect; it can also indicate the degree of match between the category result predicted by the model and the real category of the defect when the bounding box has a defect.
  • the confidence can be calculated by calculating the IoU (Intersection over Union, If IoU is greater than a certain threshold, the defect is considered to be detected, and the confidence is the IoU score of the bounding box; if IoU is less than the set threshold, the defect is considered not to be detected, and the confidence is 0.
  • IoU Intersection over Union
  • Uncertainty estimation is an effective method to measure the credibility of detection results.
  • the uncertainty in target detection tasks can be divided into data uncertainty and model uncertainty.
  • Data uncertainty mainly comes from the inherent noise in real data (such as the input image to be processed or the target sample image), and model uncertainty comes from the uncertainty of model parameters.
  • the category uncertainty in the disclosed embodiment may include both data uncertainty and model uncertainty, and the position uncertainty may also include both data uncertainty and model uncertainty. Among them, the proportion of data uncertainty is relatively small, which is related to the noise and the annotation quality of the sample.
  • the model may be overconfident and give wrong category detection results and position detection results. For example, in quality inspection on a production line, the model may mistakenly classify a defective/defective product as non-defective, or vice versa. This misclassification may lead to inefficient production lines and even affect the overall quality of the product. For production processes that rely on model prediction results for quality control, if the model cannot provide reliable and accurate predictions, then these predictions may not be trusted by end users.
  • the mean of multiple category prediction results of a defect may be calculated as the category detection result of the defect.
  • the category prediction result indicates the probability of the detected defect being a certain type of defect.
  • the category prediction result may also be referred to as the category confidence, and thus the category detection result may also be referred to as the category confidence mean.
  • the mean of multiple position prediction results of the defect may be used as the position detection result of the defect.
  • the position detection result may also be referred to as the coordinate mean.
  • category uncertainty is the uncertainty of category confidence, which is a probability distribution index of category confidence, that is, the probability distribution of category confidence represents category uncertainty, so it can also be called category confidence uncertainty.
  • category confidence variance is a probability distribution index of a bounding box, and the probability distribution of a bounding box represents position uncertainty.
  • the variance of the position prediction result can be calculated to obtain position uncertainty.
  • position uncertainty can also be called coordinate variance.
  • the model output includes [coordinate mean, coordinate variance (position uncertainty), category confidence mean, category confidence variance (category uncertainty), target confidence].
  • the target confidence indicates the probability of whether the detection is a defect.
  • the category confidence mean indicates the probability of which type of defect the detection is.
  • a confidence threshold can be set. If the target confidence of the model output is less than the confidence threshold, the defect with lower target confidence is detected, for example, prompting manual judgment.
  • the confidence threshold is a real number greater than 0 and less than 1, for example, it can be set to 0.3, but the present disclosure is not limited to this.
  • the coordinate mean is expressed as:
  • the coordinate variance is expressed as:
  • the mean and variance of category confidence are:
  • Object) can be expressed as (c represents the current category, k represents traversing all categories):
  • the present disclosure is not limited to this.
  • the category prediction result with the highest frequency among the multiple category prediction results of the defect may be selected as the category detection result of the defect
  • the position prediction result with the highest frequency among the multiple position prediction results of the defect may be selected as the position detection result of the defect. Since the category detection result and position detection result of the defect in the embodiment of the present disclosure comprehensively consider the multiple category prediction results and multiple position prediction results of the defect, the detection result (including the category detection result and the position detection result) outputted by the defect has higher accuracy.
  • the category prediction result output by the deep learning defect model is the category confidence mean.
  • the category confidence mean output by the model is subjected to the system post-processing process using the above formula (1).
  • the post-processing process actually adds a softmax process after the model output to map the category confidence to a real number between [0,1].
  • the post-processing process can also protect the bounding box NMS (non-maximum suppression) and the like.
  • the variance of multiple category prediction results of the defect may be calculated as the category uncertainty of the defect, and the variance of multiple position prediction results of the defect may be calculated as the position uncertainty of the defect.
  • ⁇ represents the adjustment term, which is a real number greater than 0 and less than 1.
  • Variance represents the variance of multiple category prediction results or the variance of multiple location prediction results.
  • Uncertainty represents category uncertainty or location uncertainty.
  • an uncertainty threshold may be set to filter prediction results with higher uncertainty.
  • Sensitivity to environmental factors may affect the detection performance of the model, such as environmental factors such as light, shadows, and cluttered backgrounds.
  • the disclosed embodiment optimizes the model to improve its stability and accuracy in the actual environment. For example, Bayesian estimation/Bayesian prediction is introduced to improve the generalization ability and robustness of the model to the natural environment. Bayesian estimation simulates randomness by adding a dropout layer to randomly erase the output of a part of the features. At the same time, the output of the detection head of the model is changed, and the output is interpreted as a probability distribution, not just a regression problem. Interpreting the output as a probability distribution can better capture the uncertainty of the location and category of the defect.
  • the model can not only give a deterministic estimate of the prediction, but also provide uncertainty measures about these estimates. This is particularly important for dealing with complex natural environments, because in such environments, the location, size, and appearance of the target may change due to a variety of factors.
  • Bayesian estimation and dropout layers By combining Bayesian estimation and dropout layers and interpreting the output as a probability distribution, a more robust and more generalized deep learning defect detection model can be constructed. Such a model can better adapt to changes in the natural environment and improve the accuracy and reliability of detection.
  • the defect detection method provided by the embodiment of the present disclosure is to extract the defect information in the feature extraction module of the deep learning defect detection model.
  • a new discard layer is added to the intermediate feature layer in the image processing, which simplifies the approximation of the posterior probability distribution, so that not only the category and location of the defects in the processed image can be predicted quickly and effectively, but also the category uncertainty and location uncertainty of the prediction results can be evaluated.
  • the disclosed embodiment has developed a new deep learning defect detection model that is more accurate, efficient and has good generalization ability. It can improve the accuracy of defect recognition without sacrificing the detection speed, and can adapt to the needs of various production environments and types of objects to be detected. It has broad application prospects. For example, it can be applied to mask fabric defect detection systems to identify defects in different types of masks.
  • the deep learning defect detection module 1 includes a feature extraction module 11 and a detection head 12.
  • the feature extraction module 11 includes an input block 111 and an intermediate feature layer 112.
  • the input block 111 is used to receive an image to be processed and send the image to be processed to the intermediate feature layer 112.
  • the intermediate feature layer 112 includes a discard layer 1121.
  • the input block 111 can also be used to pre-process the received input image to be processed, such as resizing and normalizing, so as to adapt to the input requirements of the model.
  • the image to be processed is processed by the feature extraction module 11 in the deep learning defect detection model 1, and the discarding layer 1121 in the intermediate feature layer 112 of the feature extraction module 11 is used to randomly discard the features extracted after the intermediate feature layer 112 processes the image to be processed, so that the image to be processed obtains multiple features (the specific number of samples can be set according to actual needs) after being processed by the feature extraction module 11.
  • the multiple features after the random discarding process are processed separately by the detection head 12 in the deep learning defect detection model 1 to obtain multiple category prediction results and multiple position prediction results of the defects in the image to be processed.
  • the detection head 12 also obtains the category detection results, position detection results, category uncertainty and position uncertainty of the defects in the image to be processed based on the multiple category prediction results and multiple position prediction results of the defects.
  • the intermediate feature layer includes a first cross-stage convolution block
  • the discard layer includes a first discard layer.
  • the first cross-stage convolution block includes a first cross-stage input layer, a first convolution layer, a first intermediate layer, a second convolution layer, a first fusion layer, a third convolution layer, and the first discard layer.
  • the first convolution layer is used to perform convolution processing on the first cross-stage input feature map input by the first cross-stage input layer, and input the first convolution feature map to the first intermediate layer.
  • the first intermediate layer is used to process the first convolution feature map, and input the first feature map to be fused to the first fusion layer.
  • the second convolution layer is used to perform convolution processing on the first cross-stage input feature map, and input the second feature map to be fused to the first fusion layer.
  • the first fusion layer is used to perform fusion processing on the first feature map to be fused and the second feature map to be fused, and input the first fused feature map to the third convolution layer.
  • the third convolution layer is used to perform convolution processing on the first fused feature map, and input the second convolution feature map to the first discard layer.
  • the first discard layer is used to randomly discard features in the second convolution feature map to obtain multiple first discard feature maps.
  • the image to be processed is processed by the feature extraction module to obtain multiple first discard feature maps after random discard processing.
  • the features include a plurality of said first discarded feature maps.
  • the cross-stage input layer (including the first to third cross-stage input layers), the intermediate input layer (including the first to third intermediate input layers), and the depth separation input layer (including the first to third depth separation input layers) in the embodiments of the present disclosure are all input layers of the corresponding modules in the model, and their names are only used to distinguish which module the input layer belongs to.
  • the cross-stage input layer indicates that it is the input layer of the cross-stage convolution block.
  • the input layer is used to preprocess the input feature map or image data, such as adjusting the size, so as to adapt to the input requirements of the corresponding module in the model.
  • the convolution feature map in the embodiment of the present disclosure refers to the feature map obtained after the convolution process;
  • the feature map to be fused (for example, including the first to sixth feature maps to be fused) refers to the feature map to be fused, and the fusion process here can be, for example, splicing or cascading, for example, the first feature map to be fused and the second feature map to be fused are cascaded to obtain the first fused feature map.
  • the discarded feature map (for example, including the first to third discarded feature maps) refers to the feature map generated by randomly discarding features through the corresponding dropout layer.
  • the spatial pooling feature map (for example, including the first to third spatial pooling feature maps) refers to the feature map generated after being processed by the spatial pyramid pooling block (for example, including the first to third spatial pyramid pooling blocks).
  • the intermediate feature layer further includes a second cross-stage convolution block and a third cross-stage convolution block
  • the dropout layer further includes a second dropout layer and a third dropout layer.
  • the input end of the second cross-stage convolution block is connected to the output end of the first cross-stage convolution block to receive a plurality of the first discarded feature maps respectively, process the plurality of the first discarded feature maps, and input a plurality of second discarded feature maps to the third cross-stage convolution block through the second discard layer.
  • the input end of the third cross-stage convolution block is connected to the output end of the second cross-stage convolution block to receive a plurality of the second discarded feature maps respectively, process the plurality of the second discarded feature maps, and output a plurality of third discarded feature maps through the third discard layer.
  • the second cross-stage convolution block includes a second cross-stage input layer, a fourth convolution layer, a second intermediate layer, a fifth convolution layer, a second fusion layer, a sixth convolution layer and the second drop layer.
  • the second cross-stage input layer is connected to the first drop layer, and is used to respectively use multiple first dropped feature maps as second cross-stage input feature maps.
  • the fourth convolution layer is used to perform convolution processing on the second cross-stage input feature map input by the second cross-stage input layer, and input the third convolution feature map to the second intermediate layer.
  • the second intermediate layer is used to process the third convolution feature map, and input the third feature map to be fused to the second fusion layer.
  • the fifth convolution layer is used to perform convolution processing on the second cross-stage input feature map, and input the fourth feature map to be fused to the second fusion layer.
  • the second fusion layer is used to fuse the third feature map to be fused and the fourth feature map to be fused, and input the second fused feature map to the sixth convolution layer.
  • the sixth convolution layer is used to perform convolution processing on the second fused feature map, and input the fourth convolution feature map to the second drop layer.
  • the second drop layer is used to perform random drop processing on the features in the fourth convolution feature map
  • a plurality of second discarded feature maps are obtained after the image to be processed is processed by the feature extraction module, and the plurality of features obtained after random discarding processing also include a plurality of the second discarded feature maps.
  • the cross-stage convolution blocks in the embodiments of the present disclosure adopt the idea of cross-stage partial connection, and enhance the diversity and robustness of the features by fusing the feature maps of different stages.
  • This design helps to improve the performance of the model, especially when dealing with complex target detection tasks.
  • the cross-stage convolution block adopts a staged processing approach, dividing the input feature map into multiple stages, each of which contains a series of convolution operations. At the end of each stage, the cross-stage convolution block fuses the feature map of the current stage with the feature map of the previous stage to obtain a richer feature representation.
  • This cross-stage connection method helps the model capture feature information of different scales and levels, and improves the model's ability to perceive the target.
  • the cross-stage convolutional block can also adopt a partial connection strategy, that is, instead of fusing all the feature maps of all stages, it selectively fuses some feature maps. This design can reduce the amount of calculation of the model while maintaining high performance.
  • the deep learning defect detection model further includes a feature pyramid module
  • the feature pyramid module includes a first spatial pyramid pooling block, a second spatial pyramid pooling block, and a third spatial pyramid pooling block.
  • the first spatial pyramid pooling block is connected to the first cross-stage convolution block, and is used to receive a plurality of the first discarded feature maps.
  • the second spatial pyramid pooling block is connected to the second cross-stage convolution block, and is used to receive a plurality of the second discarded feature maps, and processes the plurality of the second discarded feature maps respectively to obtain a plurality of second spatial pooling feature maps.
  • the third spatial pyramid pooling block is connected to the third cross-stage convolution block, and is used to receive a plurality of the third discarded feature maps, and processes the plurality of the third discarded feature maps respectively to obtain a plurality of third spatial pooling feature maps.
  • the first spatial pyramid pooling block is also connected to the second spatial pyramid pooling block, and is used to receive a plurality of the second spatial pooling feature maps, and obtain a plurality of first spatial pooling feature maps according to the plurality of the first discarded feature maps and the plurality of the second spatial pooling feature maps.
  • the second spatial pyramid pooling block is also connected to the third spatial pyramid pooling block, and is used to receive a plurality of the third spatial pooling feature maps, and obtain a plurality of second spatial pooling feature maps according to a plurality of the second discarded feature maps and a plurality of the third spatial pooling feature maps.
  • a feature pyramid module can be added between the backbone network (e.g., feature extraction module 11) and the detection head 12 to further process and enhance the features extracted by the backbone network.
  • the feature pyramid module can include one or more of FPN (Feature Pyramid Networks) and PANet (Path Aggregation Network), which combine feature information at different levels through multi-scale feature fusion to improve the detection performance of the model.
  • the feature pyramid module includes a spatial pyramid pooling block for illustration, but the present disclosure is not limited to this.
  • the Spatial Pyramid Pooling block uses the spatial pyramid pooling technique.
  • the idea of spatial pyramid pooling comes from the application of pyramid-shaped grid cells that cover receptive fields of different sizes. Each grid cell will perform the same pooling operation, and eventually all feature maps of different sizes will be connected together to form a feature vector of fixed length.
  • the spatial pyramid pooling block scales the input feature map and performs pooling operations at each scale.
  • This multi-scale pooling method enables the network to process inputs of arbitrary sizes and generate outputs of fixed sizes. It improves the flexibility and adaptability of the model by splicing feature maps of different scales together to obtain a multi-scale feature representation. Since the spatial pyramid pooling block uses this multi-scale pooling technique, it can flexibly process inputs of different sizes and generate feature representations of fixed length, so it is called the spatial pyramid pooling block.
  • the detection head includes a first depthwise separable convolution block, a second depthwise separable convolution block and a third depthwise separable convolution block.
  • the defect includes a first defect, a second defect and a third defect; the first depthwise separable convolution block is connected to the first spatial pyramid pooling block, and is used to receive a plurality of the first spatial pooling feature maps, and process the plurality of the first spatial pooling feature maps respectively, to obtain a plurality of first category prediction results and a plurality of first position prediction results of the first defect.
  • the second depthwise separable convolution block is connected to the second spatial pyramid pooling block, and is used to receive a plurality of the second spatial pooling feature maps, and process the plurality of the second spatial pooling feature maps respectively, to obtain a plurality of second category prediction results and a plurality of second position prediction results of the second defect.
  • the third depthwise separable convolution block is connected to the third spatial pyramid pooling block, and is used to receive a plurality of the third spatial pooling feature maps, and process the plurality of the third spatial pooling feature maps respectively, to obtain a plurality of third category prediction results and a plurality of third position prediction results of the third defect.
  • the detection head in the embodiment of the present disclosure may include two or more depth-separable convolution blocks (for example, including first to third depth-separable convolution blocks), and the number of depth-separable convolution blocks in the detection head corresponds to the number of cross-stage convolution blocks in the intermediate feature layer.
  • the model also includes a feature pyramid module
  • the number of depth-separable convolution blocks also corresponds to the number of spatial pyramid pooling blocks in the feature pyramid module.
  • the detection head is respectively illustrated as including 2 or 3 depth-separable convolution blocks, but the present disclosure is not limited to this. For example, more depth-separable convolution blocks may be included to extract defects of more scales.
  • each depthwise separable convolution block decomposes a complete convolution operation into two steps, namely Depthwise Convolution and Pointwise Convolution.
  • a feature map of 64 ⁇ 64 pixels and three channels of color for example, it can come from the output of the corresponding cross-stage convolution block or the output of the spatial pyramid pooling block) first undergoes the first convolution operation.
  • This convolution is performed entirely in a two-dimensional plane, and the number of filters is the same as the Depth (here refers to the number of channels) of the previous layer.
  • the convolution operation is the same as the network layer (not the network layer). Therefore, a three-channel feature map generates three feature maps after operation.
  • One of the filters contains a kernel of size 3 ⁇ 3.
  • the number of feature maps after Depthwise Convolution is the same as the depth of the input layer.
  • Pointwise Convolution the information of different feature maps at the same spatial position is used to combine these feature maps to generate a new feature map.
  • the size of the convolution kernel of Pointwise Convolution is 1 ⁇ 1 ⁇ M, where M is the depth of the previous layer.
  • the convolution operation here will perform a weighted combination of the feature mapp of the previous step in the depth direction to generate a new feature map.
  • 4 feature maps are also output, which is the same as the output dimension of conventional convolution. With the same input, 4 feature maps are also obtained.
  • the number of parameters of depthwise separable convolution is about 1/3 of that of conventional convolution, and the amount of calculation can be reduced by 8-9 times. Therefore, under the premise of the same number of parameters, the number of neural network layers using depthwise separable convolution can be made deeper.
  • the structure of the depth-separable convolution block in the detection head can refer to the depth-separable convolution layer in the intermediate feature layer (for example, the first to third depth-separable convolution layers), and the convolution kernel size of the depth-separable convolution block can be different from that of the depth-separable convolution layer, and the convolution method can be the same.
  • the convolution kernel size in the detection head can be 1 ⁇ 1 and 3 ⁇ 3 respectively.
  • the convolution kernel size of the depth-separable convolution layer in the intermediate feature layer can be 1 ⁇ 1, 3 ⁇ 3, 5 ⁇ 5, and 7 ⁇ 7 respectively.
  • the role of different convolution kernels is to have different feature receptive fields.
  • Depth-separable convolution can effectively reduce the amount of convolution parameters and thus enhance the processing speed.
  • 128x3x3x256 (c1 w h c2) 128 represents the number of input channels
  • 3x3 represents the convolution kernel size
  • 256 represents the number of output channels
  • the parameter amount is 294912.
  • the number of parameters is reduced by 88%.
  • Depthwise separable convolution can completely replace ordinary convolution, with the same activation method and connection method.
  • the depthwise separable convolution block may not be used in the detection head, that is, the detection head may include using normal convolution operations to achieve prediction.
  • the category detection result, the position detection result, the category uncertainty and the position uncertainty of the defect in the image to be processed are obtained, including: according to the multiple first category prediction results and the multiple first position prediction results of the first defect, the first category detection result, the first position detection result, the first category uncertainty and the first position uncertainty of the first defect are obtained; according to the multiple second category prediction results and the multiple second position prediction results of the second defect, the second category detection result, the second position detection result, the second category uncertainty and the second position uncertainty of the second defect are obtained; according to the multiple second category prediction results and the multiple second position prediction results of the second defect, the second category detection result, the second position detection result, the second category uncertainty and the second position uncertainty of the second defect are obtained; according to the multiple second category prediction results and the multiple second position prediction results of the second defect, the second category detection result, the second position detection result, the second category uncertainty and the second position uncertainty of the second defect are obtained; according to the multiple second category prediction results and the multiple second position prediction results of the second defect, the second category detection result, the second position detection result
  • the first defect, the second defect, the third defect, etc. refer to defects of different scales in the image to be processed, rather than being used to limit the number of defects.
  • the first intermediate layer includes a first intermediate input layer, a first depth-separable convolution layer and a first accumulation layer.
  • the first intermediate input layer is used to receive the first convolution feature map output by the first convolution layer, and input the first convolution feature map to the first depth-separable convolution layer and the first accumulation layer respectively.
  • the first depth-separable convolution layer is used to perform depth-separable convolution processing on the first convolution feature map, and input the first depth-separated convolution feature map to the first accumulation layer.
  • the first accumulation layer accumulates the first convolution feature map and the first depth-separated convolution feature map to obtain the first feature map to be fused.
  • the first depth-separable convolution layer includes a first depth-separation input layer, a first depth-convolution layer, a first point-convolution layer, and a first activation function layer.
  • the first depth-separation input layer is used to receive the first convolution feature map output by the first intermediate input layer, and input the first convolution feature map to the first depth-convolution layer and the first activation function layer, respectively.
  • the first depth-convolution layer is used to perform depth-convolution processing on the first convolution feature map, and input the first depth-convolution feature map to the first point-convolution layer.
  • the first point-convolution layer is used to perform point-convolution processing on the first depth-convolution feature map, and input the first point-convolution feature map to the first activation function layer.
  • the first activation function layer is used to process the first point-convolution feature map and the first convolution feature map to obtain the first depth-separated convolution feature map.
  • the feature extraction module further includes an attention block connected to the intermediate feature layer.
  • the attention block is used to process the image to be processed, obtain a downsampled feature map of the image to be processed, and input the downsampled feature map to the intermediate feature layer.
  • the deep learning defect detection model further includes a background removal module.
  • the background removal module is used to remove background information in the image to be processed, and input the image to be processed after the background information is removed into the feature extraction module.
  • background removal can be used to remove or separate parts irrelevant to the object to be detected from the image to be processed, so as to more accurately focus on the object to be detected itself and its possible defects.
  • These backgrounds may include factors such as factory production lines, conveyor belts, supporting structures, ambient lighting, etc., which may introduce noise, interfere with edges, or affect the extraction of features such as color and texture during image processing and analysis. By removing the background, you can focus more on the object to be detected itself, thereby more accurately identifying and locating defects.
  • Background removal may involve background modeling, threshold segmentation, morphological operations, etc., to separate the foreground (i.e., the object to be detected) and the background from the original image to be processed.
  • background removal does not mean completely eliminating all traces of the background in the image. Instead, it aims to reduce the interference of the background on the defect detection process while retaining enough contextual information to accurately judge the state and quality of the object to be detected.
  • the intermediate feature layer may include 3 or more cross-stage convolution blocks, and the number of depth-separable convolution blocks in the detection head is the same as the number of cross-stage convolution blocks in the intermediate feature layer.
  • the model includes a feature pyramid module, the number of spatial pyramid pooling blocks included in the feature pyramid module is the same as the number of depth-separable convolution blocks in the detection head.
  • the deep learning defect detection model provided by the embodiments of the present disclosure is exemplified below with reference to FIGS. 3 to 11 , but the present disclosure is not limited thereto.
  • FIG3 is a schematic diagram of a deep learning defect detection model of another exemplary embodiment of the present disclosure.
  • the intermediate feature layer 112 in FIG2 includes a first cross-stage convolution block, a second cross-stage convolution block, and a third cross-stage convolution block.
  • the detection head 12 includes a first depth-separable convolution block, a second depth-separable convolution block, and a third depth-separable convolution block.
  • the input block 111 inputs the received image to be processed into the first cross-stage convolution block.
  • the first depth-separable convolution block processes the plurality of first discarded feature maps respectively, and predicts a plurality of first category prediction results and a plurality of first position prediction results of the first defect in the image to be processed based on each first discarded feature map.
  • backbone network structure in the embodiments of the present disclosure is not limited to the cross-stage convolutional blocks listed above, and can also be replaced by at least one of RepVGG, EfficientNet, ConvNeXt, etc.
  • the first cross-stage convolution block also inputs a plurality of first discarded feature maps to the second cross-stage convolution block.
  • the second cross-stage convolution block processes each first discarded feature map and outputs a plurality of second discarded feature maps.
  • the second cross-stage convolution block inputs each second discarded feature map to the second depth-separable convolution block.
  • the second depth-separable convolution block receives multiple second discarded feature maps, processes each second discarded feature map, and predicts multiple second category prediction results and multiple second position prediction results of the second defect in the image to be processed.
  • the second cross-stage convolution block also inputs the plurality of second discarded feature maps to the third cross-stage convolution block, respectively.
  • the third cross-stage convolution block processes each second discarded feature map and outputs a plurality of third discarded feature maps.
  • the third cross-stage convolution block inputs each third discarded feature map to the third depth-separable convolution block.
  • the third depth-separable convolution block receives multiple third discarded feature maps, processes each third discarded feature map, and predicts multiple third category prediction results and multiple third position prediction results of the third defect in the image to be processed.
  • the detection head 12 is further used to perform Bayesian prediction/analysis based on multiple category prediction results and multiple position prediction results predicted by each depth-separable convolution block to obtain category detection results, position detection results, category uncertainty and position uncertainty of each defect.
  • Bayesian prediction processing (such as averaging and variance calculation) is performed on multiple first category prediction results and multiple first position prediction results of the first defect predicted output by the first depthwise separable convolution block to obtain the first category detection result, first position detection result, first category uncertainty and first position uncertainty of the first defect.
  • Bayesian prediction processing is performed on multiple second category prediction results and multiple second position prediction results of the second defect predicted output by the second depth-separable convolution block to obtain the second category detection result, second position detection result, second category uncertainty and second position uncertainty of the second defect.
  • Bayesian prediction processing is performed on multiple third category prediction results and multiple third position prediction results of the third defect predicted output by the third depth-separable convolution block to obtain the third category detection result, third position detection result, third category uncertainty and third position uncertainty of the third defect.
  • the obtained multiple category prediction results of the defects in the image to be processed include multiple first category prediction results of the first defect, multiple second category prediction results of the second defect, and multiple third category prediction results of the third defect.
  • the obtained multiple position prediction results of the defects in the image to be processed include multiple first position prediction results of the first defect, multiple second position prediction results of the second defect, and multiple third position prediction results of the third defect.
  • the obtained category detection result of the defects in the image to be processed includes the first category detection result of the first defect, the second category detection result of the second defect, and the third category detection result of the third defect.
  • the obtained position detection result of the defects in the image to be processed includes the first position detection result of the first defect, the second position detection result of the second defect, and the third position detection result of the third defect.
  • the obtained category uncertainty of the defects in the image to be processed includes the first category uncertainty of the first defect, the second category uncertainty of the second defect, and the third category uncertainty of the third defect.
  • the obtained position uncertainty of the defects in the image to be processed includes the first position uncertainty of the first defect, the second position uncertainty of the second defect, and the third position uncertainty of the third defect.
  • the category detection result i.e., the classification result
  • the position detection result is the location/position information of the defect in the image to be processed.
  • the dropout layer 1121 includes a first dropout layer and a second dropout layer.
  • the first dropout layer is located in the first cross-stage convolution block
  • the second dropout layer is located in the second cross-stage convolution block.
  • the input end of the first cross-stage input layer in the first cross-stage convolution block is connected to the input block 111 in the embodiment of FIG. 3 Output end.
  • the output end of the first cross-stage input layer is connected to the input end of the first convolutional layer and the second convolutional layer respectively.
  • the output end of the first convolutional layer is connected to the input end of the first intermediate layer.
  • the output end of the first intermediate layer is connected to the first input end of the first fusion layer.
  • the output end of the second convolutional layer is connected to the second input end of the first fusion layer.
  • the output end of the first fusion layer is connected to the input end of the third convolutional layer.
  • the input end of the third convolutional layer is connected to the input end of the first dropout layer.
  • the output end of the first dropout layer is connected to the input end of the second cross-stage input layer in the second cross-stage convolutional block and the input end of the first depth-separable convolutional block respectively.
  • the output end of the second cross-stage input layer in the second cross-stage convolution block is connected to the input ends of the fourth convolution layer and the fifth convolution layer respectively.
  • the output end of the fourth convolution layer is connected to the input end of the second intermediate layer.
  • the output end of the second intermediate layer is connected to the first input end of the second fusion layer.
  • the output end of the fifth convolution layer is connected to the second input end of the second fusion layer.
  • the output end of the second fusion layer is connected to the input end of the sixth convolution layer.
  • the input end of the sixth convolution layer is connected to the input end of the second dropout layer.
  • the output end of the second dropout layer is connected to the input end of the second depthwise separable convolution block.
  • the convolution kernel sizes of the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the fifth convolution layer and the sixth convolution layer may all be 1 ⁇ 1.
  • the first intermediate layer includes a first depthwise separable convolutional layer
  • the second intermediate layer includes a second depthwise separable convolutional layer
  • the input end of the first intermediate input layer of the first intermediate layer is connected to the output end of the first convolutional layer in the embodiment of FIG4.
  • the output end of the first intermediate input layer is respectively connected to the input end of the first depthwise separable convolutional layer and the input end of the first accumulation layer.
  • the output end of the first accumulation layer is connected to the input end of the first fusion layer in the embodiment of FIG4.
  • the input end of the second intermediate input layer of the second intermediate layer is connected to the output end of the fourth convolutional layer in the embodiment of FIG4.
  • the output end of the second intermediate input layer is respectively connected to the input end of the second depthwise separable convolutional layer and the input end of the second accumulation layer.
  • the output end of the second accumulation layer is connected to the input end of the second fusion layer in the embodiment of FIG4.
  • the first depth-separable convolution layer includes a first depth-separated input layer, a first depth-convolution layer, a first point-convolution layer, and a first activation function layer.
  • the second depth-separable convolution layer includes a second depth-separated input layer, a second depth-convolution layer, a second point-convolution layer, and a second activation function layer.
  • the input end of the first depth separation input layer is connected to the output end of the first intermediate input layer in the embodiment of Figure 5.
  • the output end of the first depth separation input layer is respectively connected to the input end of the first depth convolution layer and the input end of the first activation function layer.
  • the output end of the first depth convolution layer is connected to the input end of the first point convolution layer.
  • the output end of the first point convolution layer is connected to the first activation function layer.
  • the input end of the second depth separation input layer is connected to the output end of the second intermediate input layer in the embodiment of FIG5.
  • the output end of the second depth separation input layer is respectively connected to the input end of the second depth convolution layer and the input end of the second activation function layer.
  • the output of the second depth convolution layer is connected to the input of the second point convolution layer.
  • the output of the second point convolution layer is connected to the second activation function layer.
  • the drop layer further includes a third drop layer.
  • the third cross-stage convolution block includes a third cross-stage input layer, a seventh convolution layer, a third intermediate layer, an eighth convolution layer, a third fusion layer, a ninth convolution layer and the third drop layer.
  • the input end of the third cross-stage input layer is connected to the output end of the second cross-stage convolution block, for example, connected to the output end of the second drop layer, and the plurality of second drop feature maps output by the second drop layer are respectively used as the third cross-stage input feature maps.
  • the seventh convolutional layer is used to perform convolution processing on the third cross-stage input feature map input by the third cross-stage input layer, and input the fifth convolutional feature map to the third intermediate layer.
  • the third intermediate layer is used to process the fifth convolutional feature map, and input the fifth feature map to be fused to the third fusion layer.
  • the eighth convolutional layer is used to perform convolution processing on the third cross-stage input feature map, and input the sixth feature map to be fused to the third fusion layer.
  • the third fusion layer is used to perform fusion processing on the fifth feature map to be fused and the sixth feature map to be fused, and input the third fused feature map to the ninth convolutional layer.
  • the ninth convolutional layer is used to perform convolution processing on the third fused feature map, and input the sixth convolutional feature map to the third discard layer.
  • the third discard layer is used to randomly discard the features in the sixth convolutional feature map to obtain multiple third discarded feature maps. After the image to be processed is processed by the feature extraction module, the multiple features obtained after random discarding include multiple third discarded feature maps.
  • the third intermediate layer includes a third intermediate input layer, a third depth-separable convolution layer and a third accumulation layer.
  • the third intermediate input layer is used to receive the fifth convolution feature map output by the seventh convolution layer, and input the fifth convolution feature map to the third depth-separable convolution layer and the third accumulation layer respectively.
  • the third depth-separable convolution layer is used to perform depth-separable convolution processing on the fifth convolution feature map, and input the third depth-separated convolution feature map to the third accumulation layer.
  • the third accumulation layer accumulates the fifth convolution feature map and the third depth-separated convolution feature map to obtain the fifth feature map to be fused.
  • the third depth-separable convolution layer includes a third depth-separated input layer, a third depth-convolution layer, a third point-convolution layer and a third activation function layer.
  • the third depth-separated input layer is used to receive the fifth convolution feature map output by the third intermediate input layer, and input the fifth convolution feature map to the third depth-convolution layer and the third activation function layer, respectively.
  • the third depth-convolution layer is used to perform depth-convolution processing on the fifth convolution feature map, and input the third depth-convolution feature map to the third point-convolution layer.
  • the third point-convolution layer is used to perform point-convolution processing on the third depth-convolution feature map, and input the third point-convolution feature map to the third activation function layer.
  • the third activation function layer is used to process the third point-convolution feature map and the fifth convolution feature map to obtain the third depth-separated convolution feature map.
  • the activation function used by the activation function layer can be any appropriate activation function.
  • SiLU Silicon activation function
  • the SiLU function is differentiable in the entire domain, so the gradient calculation during the back propagation process is more stable.
  • the SiLU function has the characteristics of no upper bound, lower bound, smoothness, non-monotonicity, and a larger and more stable gradient in the deep model.
  • the convolution kernel sizes of the seventh convolution layer, the eighth convolution layer, and the ninth convolution layer may all be 1 ⁇ 1.
  • a depthwise separable convolution layer is introduced into the cross-stage convolution block of the disclosed embodiment.
  • the depthwise separable convolution layer reduces the number of parameters required for convolution calculation and improves the efficiency of using convolution kernel parameters by splitting the correlation between spatial dimension and channel (depth) dimension.
  • the convolution calculation process of the depthwise separable convolution layer is divided into two parts: depthwise convolution, which performs spatial convolution on each channel separately, that is, a single convolution kernel is used to perform convolution operation on each input channel, and then the outputs are spliced; pointwise convolution, which uses a unit convolution kernel (that is, a 1x1 convolution kernel) to perform channel convolution to integrate the information of each channel and obtain the final feature map.
  • a unit convolution kernel that is, a 1x1 convolution kernel
  • Depthwise separable convolution is that it reduces the computational complexity and the number of model parameters while maintaining the expressiveness of the model.
  • Depthwise separable convolution is a lightweight convolution method that improves the efficiency and performance of the model by reducing parameters and computation.
  • the feature extraction module 11 may further include a fourth cross-stage convolution block
  • the detection head 12 may further include a fourth depth-separable convolution block.
  • the structure of the fourth cross-stage convolution block may refer to the above-mentioned FIGS. 4 to 7.
  • the fourth cross-stage convolution block is connected to the third cross-stage convolution block and the fourth depth-separable convolution block in the detection head 12, and is used to receive a plurality of third discarded feature maps respectively, and process the plurality of third discarded feature maps to generate a plurality of fourth discarded feature maps.
  • the fourth cross-stage convolution block may include a fourth discard layer to output a plurality of fourth discarded feature maps to the fourth depth-separable convolution block.
  • the fourth depth-separable convolution block may predict and output a plurality of fourth category prediction results and a plurality of fourth position prediction results of a fourth defect.
  • the detection head 12 may obtain a fourth category detection result and a fourth category uncertainty of a fourth defect based on a plurality of fourth category prediction results, and obtain a fourth position detection result and a fourth position uncertainty of a fourth defect based on a plurality of fourth position prediction results.
  • the deep learning defect detection model 1 further includes a feature pyramid module 13 located between the feature extraction module 11 and the detection head 12 .
  • the embodiment of FIG9 introduces a feature pyramid module 13 between the feature extraction module 11 and the detection head 12 to solve the multi-scale problem in defect detection. Defects of different sizes present different features in the image, and the feature pyramid can extract and fuse these features at different scales.
  • the feature pyramid greatly improves the performance of small defect detection without substantially increasing the amount of computation of the original model.
  • the feature pyramid fuses feature layers of different sizes through upsampling and downsampling operations. In this way, the model can simultaneously obtain high-level language features.
  • the fused features not only help improve the detection performance of the model, but also make the model better adaptable to defect detection tasks of different sizes.
  • the intermediate feature layer in the feature extraction module 11 generates feature maps of different scales.
  • These feature maps contain more and more abstract information from the bottom to the top in the pyramid structure, corresponding to defects of different sizes.
  • the bottom feature map usually has a higher resolution and more detailed information, which is suitable for detecting small defects; while the high-level feature map has a lower resolution and more abstract semantic information, which is suitable for detecting large defects.
  • the feature pyramid fuses these feature maps of different scales through a series of upsampling and downsampling operations. Specifically, the feature map of the lower layer will be upsampled to match its resolution with the feature map of the higher layer, and then fused with the feature map of the higher layer by splicing or adding.
  • the fused feature map contains both the low-level detail information and the high-level semantic information, which helps to improve the model's detection performance for defects of different sizes.
  • multiple prediction results of different scales can be generated. These prediction results correspond to defects of different sizes, thus achieving multi-scale defect detection.
  • This mechanism enables the model to focus on both global information and local details in the image, thereby improving the accuracy and robustness of defect detection.
  • the feature pyramid module 13 includes a first spatial pyramid pooling block, a second spatial pyramid pooling block, and a third spatial pyramid pooling block.
  • the cross-stage convolution block can be configured to implement downsampling of the feature map.
  • feature maps of three different scales are input to the feature pyramid module 13 as input to the YOLO multi-scale detection head 12. Because the cross-stage convolution block can enable the downsampling function by defining parameters, more than three cross-stage convolutions can be used to implement the input of three feature maps of different scales to the feature pyramid module 13.
  • the first input end of the first spatial pyramid pooling block is connected to the output end of the first cross-stage convolution block, for example, connected to the output end of the first discarding layer, to receive multiple first discarding feature maps output by the first discarding layer.
  • the second input end of the first spatial pyramid pooling block is connected to the output end of the second spatial pyramid pooling block to receive multiple second spatial pooling feature maps output by the second spatial pyramid pooling block.
  • the first spatial pyramid pooling block processes each of the multiple first discarding feature maps and a corresponding second spatial pooling feature map in the multiple second spatial pooling feature maps, and outputs multiple first spatial pooling feature maps to the first depthwise separable convolution block in the detection head 12 through its output end.
  • the first depthwise separable convolution block in the detection head 12 processes each of the first spatial pooling feature maps, and predicts multiple first category prediction results and multiple first position prediction results of the first defect.
  • the detection head 12 performs Bayesian prediction on the multiple first category prediction results and multiple first position prediction results of the first defect to obtain the first category detection result, the first position detection result, the first category uncertainty and the first position uncertainty of the first defect.
  • the first input end of the second spatial pyramid pooling block is connected to the output end of the second cross-stage convolution block, for example,
  • the output end of the discarding layer is connected to receive multiple second discarding feature maps output by the second discarding layer.
  • the second input end of the second spatial pyramid pooling block is connected to the output end of the third spatial pyramid pooling block to receive multiple third spatial pooling feature maps output by the third spatial pyramid pooling block.
  • the second spatial pyramid pooling block processes each second discarding feature map in the multiple second discarding feature maps and a corresponding third spatial pooling feature map in the multiple third spatial pooling feature maps, and outputs multiple second spatial pooling feature maps to the second depth separable convolution block in the detection head 12 through its output end.
  • the second depth separable convolution block in the detection head 12 processes each second spatial pooling feature map respectively, and predicts multiple second category prediction results and multiple second position prediction results of the second defect.
  • the detection head 12 performs Bayesian prediction on the multiple second category prediction results and multiple second position prediction results of the second defect to obtain the second category detection result, second position detection result, second category uncertainty and second position uncertainty of the second defect.
  • the input end of the third spatial pyramid pooling block is connected to the output end of the third cross-stage convolution block, for example, connected to the output end of the third discard layer, to receive multiple third discard feature maps output by the third discard layer.
  • the third spatial pyramid pooling block processes each of the multiple third discard feature maps respectively, and outputs multiple third spatial pooling feature maps to the third depth separable convolution block in the detection head 12 through its output end.
  • the third depth separable convolution block in the detection head 12 processes each third spatial pooling feature map respectively, and predicts multiple third category prediction results and multiple third position prediction results of the third defect.
  • the detection head 12 performs Bayesian prediction on the multiple third category prediction results and multiple third position prediction results of the third defect, and obtains the third category detection result, third position detection result, third category uncertainty and third position uncertainty of the third defect.
  • the defect in the image to be processed also includes the third defect
  • the multiple category prediction results of the defect in the image to be processed obtained also include the third category prediction result of the third defect
  • the multiple position prediction results also include the third position prediction result of the third defect.
  • the obtained category detection result, position detection result, category uncertainty and position uncertainty of the defect in the image to be processed also include a third category detection result, a third position detection result, a third category uncertainty and a third position uncertainty of a third defect, respectively.
  • the feature pyramid module 13 can fuse convolutional features of receptive fields of different sizes under the original feature maps of three scales, realize multi-scale feature fusion, and enhance the robustness of the features.
  • the deep learning defect detection model 1 may further include a background removal module located before the feature extraction module 11.
  • the background removal module is used to remove background information in the image to be processed, and input the image to be processed after the background information is removed to the feature extraction module 11.
  • the image to be processed without removing the background information is taken as a mask cloth image for example, and the mask cloth image with the background information removed is referred to as the image to be processed.
  • FIG10 is a schematic diagram of a spatial pyramid pooling block of an exemplary embodiment of the present disclosure.
  • each spatial pyramid pooling block in the feature pyramid module 13 may have the same structure.
  • the spatial pyramid pooling block may include an input layer, a first convolution layer connected to the input layer, Three parallel maximum pooling layers connected to the first convolutional layer, a fusion layer connected to the three maximum pooling layers respectively, and a second convolutional layer connected to the fusion layer.
  • Each maximum pooling layer in the spatial pyramid pooling block can be used to reduce the number of parameters in the feature map, improve the calculation speed, and increase the receptive field. It is a downsampling operation. Through pooling, the model can pay more attention to global features rather than local locations. This dimensionality reduction process can retain some important feature information, improve fault tolerance, and also prevent overfitting to a certain extent.
  • the convolution kernel sizes of the first convolution layer and the second convolution layer may both be 1 ⁇ 1.
  • the convolution kernel sizes of the three parallel maximum pooling layers can be 3 ⁇ 3, 5 ⁇ 5, and 7 ⁇ 7, respectively.
  • the feature pyramid module in the embodiment of the present disclosure is not limited to the spatial pyramid pooling blocks exemplified in Figures 9 and 10, and other structures may be used instead.
  • convolution kernels of different sizes are used for parallel processing, and then the feature map after direct fusion processing is obtained.
  • Other structures, such as those shown in Figure 11, can be changed to serially process the feature map of the input value feature pyramid module with convolution kernels of different sizes, for example, using three convolution layers with convolution kernel sizes of 3 ⁇ 3, 5 ⁇ 5, and 7 ⁇ 7 in series, and then using shortcuts (direct connection or shortcuts) to jump to the feature maps output by different convolution kernels to achieve fusion.
  • the third difference between the embodiment of FIG. 9 and the embodiment of FIG. 3 is that the input block 111 may be an attention block 111'.
  • the attention block 111' is a slice operation performed on the image to be processed (b1, c1, w1, h1 respectively represent the batch size, number of channels, width, and height of the image to be processed) before entering the feature extraction module 11.
  • the specific operation is to obtain a value for every other pixel in an image to be processed, similar to neighboring downsampling, so that four images are obtained, and the four images are complementary, and there is no information loss.
  • the w1/h1 information is concentrated in the channel space, and the input channel is expanded by 4 times (the input of (b1, 4*c1, w1, h1) is obtained), that is, the spliced image becomes 4*c1 channels relative to the original c channels.
  • the obtained new image is subjected to a convolution operation (for example, the convolution kernel size is 1x1), and finally a double downsampled feature map without information loss is obtained, which is input to the first cross-stage convolution block.
  • the attention block 111' obtains a double-downsampled feature map without information loss, which can reduce the number of parameters, the number of network layers, the gradient and the number of calculations, while improving the precision and recall rate of the model. This enables the model to more accurately detect and identify defects in the image while maintaining high efficiency, especially those with small size and unclear features.
  • a spatial attention mechanism is used to design the attention block.
  • the attention block in the embodiment of the present disclosure is not limited to the structure shown in FIG12, and the channel attention block can also be implemented through the Squeeze-Excitation (compression-excitation, explicit modeling of the interdependence between feature channels, obtaining the importance of each channel by learning, and then weighting the features on each channel according to this importance, highlighting important features and suppressing unimportant features) operation.
  • the spatial attention block can also be implemented through dynamic convolution.
  • the model provided by the embodiment of the present disclosure is not limited to the above-mentioned structure.
  • the embodiment of the present disclosure provides adding a discard layer to the intermediate feature layer and adding a Bayesian loss function (such as the second loss function described below) during the training process of the model.
  • the method provided by the embodiment of the present disclosure can adapt to various network structures while maintaining compatibility with various network architectures. While being compatible with other algorithms, it also enables the integration of uncertainty estimates, providing flexibility in applications and further improving performance.
  • the model can be adapted to different image resolutions and blemish/defect detection scenarios.
  • the convolution operation allows the network to focus on local features in the image rather than the entire image. Due to this property of the convolutional layer, the model is able to process input images of different sizes without having to make major modifications to the model structure. This makes the model highly flexible and can adapt to a variety of image resolutions. In the field of defect detection, this means that regardless of whether the input image is high-definition or low-resolution, or how the size and position of the defects in the image vary, the model using convolutional layers can effectively perform defect detection. Therefore, this model can easily adapt to different defect detection scenarios, improving the versatility and practicality of the model.
  • the method provided by the embodiment of the present disclosure also includes: obtaining a target training data set, which includes a target sample image and annotated categories and annotated positions of target sample defects in the target sample image; inputting the target sample image into the deep learning defect detection model to obtain multiple category prediction results and multiple position prediction results of the target sample defects; obtaining category detection results, position detection results, category uncertainty and position uncertainty of the target sample defects based on the multiple category prediction results and multiple position prediction results of the target sample defects; constructing a loss function based on the annotated category and category detection result, annotated position and position detection result, and category uncertainty and position uncertainty of the target sample defects to train the deep learning defect detection model.
  • a data set for defect detection can be obtained, and divided into a target training data set and a target test data set according to a predetermined ratio, for example, a ratio of 9:1.
  • a dropout operation is performed during the forward propagation process.
  • multiple different network structures can be obtained by randomly closing neurons, thereby performing multiple predictions on the same target sample image to obtain multiple category prediction results and multiple position prediction results.
  • the mean of multiple category prediction results and the mean of multiple position prediction results are calculated to represent the category detection result and position detection result of the target sample defect, respectively.
  • the mean of multiple category prediction results and the variance of multiple position prediction results are calculated to represent the category uncertainty and position uncertainty of the target sample defect, respectively.
  • the training parameters of the model can be set as follows: batch size is set to 64, number of iterations (epoch) is set to 100, initial learning rate is set to 0.001, decay coefficient is set to 0.005, and dropout probability is set to 0.3. After dropout, the number of neurons in each layer becomes about half of the original number.
  • the loss function includes a first loss function and a second loss function; when the difference between the labeled category and the category detection result of the target sample defect is greater, and the difference between the labeled position and the position detection result is greater, the value of the first loss function is greater; when the category uncertainty and the position uncertainty of the target sample defect are greater, the value of the second loss function is greater.
  • the disclosed embodiment introduces the Bayesian method, and adds dropout in the model to create random uncertainty to realize the Bayesian posterior distribution.
  • the detection head of the model is optimized, and the mean and variance of the anchor box (i.e., the position prediction result) and the category (i.e., the category prediction result) are added to represent their distribution.
  • the second loss function can be a negative log-likelihood loss function to optimize the uncertainty estimation, which not only promotes the evolution of the model towards accurate prediction, but also encourages the model to accurately estimate the uncertainty of its prediction, thereby improving the detection accuracy and robustness of the model.
  • the first loss function L1 is as follows:
  • MSE represents the bounding box regression calculation
  • yi is the i-th bounding box coordinate in the real data (i.e., the marked position of the i-th target sample defect)
  • N is the number of bounding boxes (i.e., the number of candidate defects in the target sample)
  • N is a positive integer greater than or equal to 1
  • i is a positive integer greater than or equal to 1 and less than or equal to N.
  • CE represents the multivariate cross entropy calculation, Represents the category confidence of category c in the predicted category detection result (i.e., the probability of belonging to category c).
  • s c is the confidence of category c in the real data (i.e., the category confidence of category c in the labeled category).
  • C represents the number of categories, C is a positive integer greater than or equal to 1, and c is a positive integer greater than or equal to 1 and less than or equal to C.
  • BCE represents the binary cross entropy calculation, o pre represents the predicted target confidence, and o represents the confidence of the target in the real data, that is, the confidence of the labeled target in the target sample defect.
  • the second loss function L2 is as follows:
  • ⁇ i is the predicted position detection result of the defect of the i-th target sample, for example, the mean of the i-th bounding box coordinates predicted by the model.
  • yi is the annotated position of the defect of the i-th target sample, for example, the i-th bounding box coordinates in the real data.
  • N is the number of target sample defects, N is a positive integer greater than or equal to 1, and i is a positive integer greater than or equal to 1 and less than or equal to N.
  • ⁇ c represents the mean category confidence of the predicted category c, that is, the category detection result.
  • s c is the category confidence of category c in the real data. is the category confidence variance of the predicted category c, that is, the category uncertainty.
  • C represents the number of categories, C is a positive integer greater than or equal to 1, and c is a positive integer greater than or equal to 1 and less than or equal to C.
  • obtaining a target training data set includes: obtaining an initial sample image and the annotated category and annotated position of an initial sample defect in the initial sample image; performing any one or more of random cropping, rotation, scaling, illumination change simulation, and defect synthesis on the initial sample image to generate an augmented sample image and the annotated category and annotated position of an extended sample defect in the extended gain image.
  • the target sample image includes the initial sample image and the augmented sample image.
  • the initial sample image is subjected to defect synthesis processing to generate an augmented sample image, including: extracting initial sample defects from the initial sample image; synthesizing the initial sample defects with an augmented background image to generate the augmented sample image; and/or, learning the initial sample image through a deep learning generation model to generate the augmented sample image.
  • the quality of the dataset affects the accuracy and robustness of the model. For example, let’s take the detection of defects in the mask cloth image as an example. For example, in order to train an efficient and accurate model, a large number of high-quality images of mask cloth with defects are required to form a dataset. If the image quality in the dataset is low or the sample size is insufficient, the accuracy and robustness of the model may be affected.
  • the number of defective mask cloth images is less than the number of defective mask cloth images, that is, the number of positive and negative samples in the training data set is unbalanced.
  • the present disclosure solves the problem of data set quality by implementing a multi-stage data enhancement strategy. First, a large number of original mask cloth images are collected, and initial training data (including initial sample images) are generated through expert annotation. Then, one or more of image processing techniques such as random cropping, rotation, scaling and illumination change simulation, as well as synthetic defect generation methods, are used to expand the data set to obtain an amplified sample image.
  • the defective mask cloth image can be increased by using any one or more of the data enhancement methods of random cropping, rotation, scaling, illumination change simulation (such as brightness adjustment), defect synthesis, adding Gaussian noise, random occlusion, etc.
  • the illumination change simulation is to randomly adjust one or more of the saturation, brightness, contrast, etc. of the original defective mask cloth image, which can simulate the image being easily affected by changes in light and weather; random occlusion is to block random areas of the defective mask cloth image with black, which can simulate the situation that it is easy to be blocked by other objects during shooting;
  • Gaussian noise is to add noise whose probability density function obeys Gaussian distribution to the original defective mask cloth image, which can better simulate unknown real noise.
  • the disclosed embodiment uses data enhancement to randomly scale and crop multiple initial sample images for splicing, which greatly increases the number of small and micro defects in the training data set and improves the effect of small and micro defect detection.
  • the disclosed embodiment can also generate new defect samples based on real defect samples by introducing AIGC (stable diffusion) to increase data diversity.
  • AIGC stable diffusion
  • Any one or two of the following two methods of synthesizing defects can be used: the first is to extract the defect foreground and replace the background of other fabrics. The second is to learn real defect images through stable diffusion and then generate fake defect images.
  • new defect samples can be generated based on real defect samples, which can enrich the data set and help train more accurate models.
  • the generated defect samples can be used to simulate various possible defect situations, so as to more comprehensively test the performance of the defect detection system in practical applications.
  • obtaining a target training data set includes: obtaining candidate sample images and the annotated categories and annotated positions of candidate sample defects in the candidate sample images; obtaining output features of the candidate sample images through the feature extraction module; obtaining feature distances between the candidate sample images based on the output features of each candidate sample image; and selecting candidate sample images whose feature distances are greater than a first distance threshold among candidate sample images with the same annotated category, and/or candidate sample images whose feature distances are less than a second distance threshold among candidate sample images with different annotated categories as target sample images in the target training data set.
  • Euclidean distance may be used as the feature distance, but the present disclosure is not limited thereto.
  • the disclosed embodiment may also apply an adaptive sample selection mechanism to preferentially select those samples that are most effective in improving model performance for training. For example, during the training process, the features of the backbone network output of multiple samples are extracted to capture the similarities and differences between different samples. The feature distance of each sample is then calculated.
  • the feature distance d ij between the i-th sample ( xi ) and the j-th sample ( xj ) can be calculated by the following formula:
  • F k ( xi ) is the eigenvalue of sample xi in the kth dimension
  • F k ( xj ) is the eigenvalue of sample xj in the kth dimension
  • D is the dimension of the sample feature
  • D is a positive integer greater than or equal to 1
  • k is a positive integer greater than or equal to 1 and less than or equal to D.
  • i and j are both positive integers greater than or equal to 1, and i is not equal to j.
  • Calculating the characteristic distance of samples helps to understand the distribution of samples in the feature space. For example, you can select the farthest samples among the same type of samples (images corresponding to the same type of defects, that is, different samples with the same label, such as samples all labeled as "tape") and the closest samples among the different types of samples (images corresponding to different types of defects, such as one sample labeled as "stain” and another sample labeled as "crease”) to participate in back propagation, which is equivalent to using the most error-prone (most difficult to judge) samples to optimize the model weights to improve the accuracy and robustness of the model.
  • the sample selection criterion of the adaptive sample selection mechanism is to first count the feature distance, and then divide the first distance threshold and the second distance threshold according to the statistical value of the feature distance, for example, the first distance threshold (also called intra-class distance) is selected to be greater than 0.3, and the second distance threshold (also called inter-class distance) is selected to be less than 0.2.
  • the adaptive sample selection mechanism is a training strategy that is added after conventional training and is used for secondary training to solve the hard sample problem. After secondary training, the feature distribution of intra-class samples is more concentrated, and the feature distribution of inter-class samples is more discrete, thereby enhancing classification accuracy.
  • selecting the farthest samples among the samples of the same type means that these samples have the greatest difference in feature space, and even if they belong to the same category, it is difficult for the model to distinguish them correctly.
  • the model learn these difficult-to-distinguish samples the model can be prompted to pay more attention to the detailed differences within the category, thereby improving the accuracy of the model.
  • selecting the closest samples among samples of different types means that these samples have a high similarity in feature space, but in fact they belong to different categories. These samples are the most error-prone samples for the model because they have similar features but different labels. By letting the model learn these samples, the model's ability to distinguish between different categories can be strengthened, thereby improving the robustness of the model.
  • the deep learning defect detection model extracts the weight tensor of each convolutional layer in the deep learning defect detection model; obtain the Euclidean norm of each channel in the weight tensor; sort each convolutional layer according to the Euclidean norm of its channel, and prune the corresponding channel according to a pruning threshold or a pruning ratio.
  • Channel pruning is an effective model compression and acceleration technology, especially suitable for the needs of lightweight model design.
  • Channel pruning in the model can significantly reduce the number of model parameters and computational complexity while maintaining the performance of the model.
  • the contribution of the convolution channel to the network output is measured based on L2-norm (Euclidean norm), and channel pruning is performed according to the degree of contribution.
  • L2-norm Euclidean norm
  • the contribution of the convolution channel is measured based on L2-norm: L2-norm (Euclidean norm) can be used to evaluate the importance or contribution of the convolution channel.
  • L2-norm Euclidean norm
  • the L2-norm value of each convolution channel weight can be calculated. This value reflects the influence and contribution of the channel in the network.
  • its weight tensor is extracted.
  • the L2-norm value of each channel in the weight tensor is calculated.
  • operations such as squaring, summing, and square rooting the weight tensor are performed.
  • the calculated L2-norm value is saved for subsequent channel pruning decisions.
  • channel pruning is performed according to the degree of contribution: According to the L2-norm value calculated in the previous step, the importance of each convolution channel can be evaluated, and channel pruning can be performed according to a certain strategy.
  • the goal of channel pruning is to remove channels that contribute less to the network output, thereby reducing the number of parameters and computational complexity of the model.
  • a pruning threshold or pruning ratio is set. The pruning threshold or pruning ratio can be determined according to specific needs and experiments. For each convolutional layer, sort them according to the L2-norm value of its channels.
  • select channels with smaller contributions for pruning For example, set the weights of the corresponding channels to zero or directly remove these channels from the model. After pruning, retrain and fine-tune the model to restore or maintain its performance.
  • the Euclidean norm reflects the response weight of the channel to the output, and channels with lower response weights are pruned first.
  • the deep learning defect detection model in order to achieve a balance between detection speed and accuracy, is lightweight designed, the number of feature channels is reduced through channel pruning, and the original convolution layer is replaced with a depthwise separable convolution to reduce the parameters and computational complexity of the model, thereby achieving faster detection while maintaining high accuracy.
  • the target sample images include mask images (or mask cloth images) having different materials, textures, colors, and prints.
  • the embodiment of the present disclosure designs a mixed data set.
  • the mixed data set contains multiple types of mask samples, such as mask images with different mask materials, colors, textures, and print patterns.
  • transfer learning and domain adaptation techniques can be used, including one or more of feature space alignment, fine-tuning, metric learning, etc., to enable the model to adapt to new materials, colors, textures, printed patterns, etc.
  • the model is periodically retrained on newly collected data to maintain its generalization performance.
  • obtaining an image to be processed includes: obtaining the image to be processed from an image capturing device in an image acquisition component, wherein the image acquisition component is used to obtain the image to be processed of the object to be detected in the production process.
  • the method further includes: obtaining the current production speed of the object to be detected and the a current processing speed of the deep learning defect detection model on the image to be processed; and adjusting the number of parallel threads or parallel processes used by the deep learning defect detection model to process the image to be processed according to the current production speed and the current processing speed.
  • a real-time detection feedback mechanism is introduced in the disclosed embodiment to dynamically adjust the model inference time (i.e., the current processing speed of the model) by changing the number of parallel threads of model inference according to the current production line speed (or current production speed), thereby ensuring that the required detection speed is achieved while maintaining high accuracy.
  • the model is deployed using TensorRT after pruning, and the encapsulated TensorRT architecture supports multi-process and multi-threaded concurrency.
  • a dynamic adjustment mechanism is designed.
  • a thread pool can be used to manage the concurrent execution of reasoning tasks.
  • the thread pool can ensure the effective use of thread resources and avoid the overhead caused by frequent creation and destruction of threads.
  • a load balancing strategy can be used to ensure that the loads of each thread are relatively balanced to avoid situations where some threads are overloaded while other threads are idle.
  • TensorRT natively supports multi-threaded reasoning, which can be achieved through execute_async_v2 of IExecutionContext.
  • the disclosed embodiment also takes advantage of the multi-core CPU (Central Processing Unit) to serialize the TensorRT engine to disk, and then loads this serialized engine in multiple processes, so that each process can independently perform reasoning tasks, and maximize the resources of the CPU and GPU (Graphics Processing Unit).
  • CPU Central Processing Unit
  • the disclosed embodiments can use sensors or other monitoring devices to obtain the current production speed in real time; the current reasoning speed of the monitoring model, that is, the time required to process each image is used as its current processing speed. According to the comparison between the current production speed and the current processing speed of the model, the number of parallel reasoning threads of the model is adjusted. If the current production speed is accelerated, the number of parallel threads needs to be increased to speed up the reasoning speed; conversely, if the current production speed is slowed down, the number of parallel threads can be reduced to save resources.
  • the real-time detection feedback mechanism dynamically adjusting the number of parallel threads of model reasoning, and the efficient deployment of TensorRT, it can be ensured that the defect detection tasks on the production line are both accurate and efficient.
  • obtaining an image to be processed includes: obtaining the image to be processed from an image capturing device in an image acquisition component; if the grayscale value and grayscale distribution of the image to be processed do not meet preset conditions, sending a control signal to adjust the exposure parameters of the image capturing device; obtaining a new image to be processed from the image capturing device after adjusting the exposure parameters, until the image to be processed that meets the preset conditions is obtained.
  • an automatic exposure module is designed in the embodiment of the present disclosure.
  • an image in the current environment is captured by an image capture device such as a camera, and its grayscale distribution is analyzed. If the grayscale value and grayscale distribution of the image do not meet the preset conditions, the exposure parameters are adjusted, and the image is captured again for analysis. This cycle continues until the grayscale value and grayscale distribution of the captured image meet the requirements, and then it is used as the image to be processed for the input model.
  • the automatic exposure module can monitor the lighting conditions in real time, calculate the latest exposure parameters of the camera in real time, and suppress the influence of external light.
  • the desired preset condition is histogram equalization
  • the grayscale mean is between 130 and 140
  • the grayscale variance is smaller. The smaller the better.
  • the grayscale histogram, grayscale mean, and grayscale variance of the image collected by the camera are used to determine whether the preset conditions are met. If not, the target difference is used to determine whether to increase or decrease the exposure.
  • the method provided by the embodiment of the present disclosure further includes: if the category uncertainty and/or location uncertainty of the defect is greater than an uncertainty threshold, sending prompt information for the defect.
  • the entire system is end-to-end. It takes the acquired mask cloth image as input, and the model first deletes irrelevant background to obtain the image to be processed. Then, it performs feature extraction, and finally outputs the category detection result, position detection result, category uncertainty and position uncertainty of the defect.
  • the system can determine that the defect with category uncertainty and/or position uncertainty greater than the uncertainty threshold (whose value can be set according to actual needs) has a high uncertainty defect, and a prompt message can be triggered at this time to prompt the user that the prediction of the defect has high uncertainty. Further, the user can determine how to deal with the defect, such as whether to manually mark the location and category of the defect. That is, the disclosed embodiment can also use the category uncertainty and/or position uncertainty predicted by the model to assist the user in identifying the most informative data for marking, thereby improving the efficiency of the annotation process.
  • the method provided by the embodiment of the present disclosure also includes: counting pixel quantity information of each defect detected in the image to be processed; determining the defect level of each defect based on the pixel quantity information of each defect; and performing corresponding processing based on the defect level of each defect.
  • the defect detection system includes an algorithm layer and a data service layer/data service component.
  • the algorithm layer is deployed with the above-mentioned deep learning defect detection model to process the image to be processed and obtain defect data, such as whether the image to be processed has defects. If there are defects, it contains defect information such as the category detection result, position detection result, category uncertainty and position uncertainty of the defect.
  • the data service layer is responsible for analyzing and processing the defect data provided by the algorithm layer to determine the severity of the defect.
  • the defects are divided into four levels: the first level is a slight defect, such as a defect with less than 20 pixels on the long side; the second level is a general defect, such as a defect with more than or equal to 20 pixels on the long side and less than 60 pixels; the third level is a more serious defect, such as a defect with more than or equal to 60 pixels on the long side and less than 100 pixels; and the fourth level is a serious defect, such as a defect with more than 100 pixels on the long side.
  • the data service layer can formulate and execute corresponding processing strategies for defects of different levels.
  • corresponding processing is performed according to the defect level of each defect, including: in the image to be processed, defects at different defect levels are marked with frames with different thicknesses.
  • the data service layer uses the thickness of the defect area border to assist in determining the classification level of the defect.
  • the data service layer first counts the number of pixels of each defect, and then divides the defects into four levels according to the number of defect pixels. Then, the first level of defects are marked with the thinnest border in the image to be processed, and the fourth level of defects are marked with the thinnest border in the image to be processed.
  • the thickest border is used in the processed image, and then the marked defect image is provided to the user and displayed on the interactive interface, so that the user can immediately identify the severity of the defect and quickly decide whether to stop the machine for processing. This visual identification enables the operator to quickly and intuitively understand the severity of the defect during the subsequent manual inspection process.
  • the method provided by the embodiment of the present disclosure also includes: saving the image to be processed and the category detection results, position detection results, category uncertainty, position uncertainty, defect level, defect area image, image features of the defect area, detection time and detection serial number of the defects detected in the image to be processed; receiving and retrieving corresponding defect information based on one or more of the input image features, detection time and detection serial number of the defect area.
  • the defect detection system may also include a defect recording module.
  • the defect recording module will automatically record the level of the defect (i.e., defect level), the exact location (i.e., location detection result), and the detailed regional image of the defect point (i.e., defect or defect point).
  • the data service layer may also provide a defect point information retrieval function.
  • the image features of the defect/defect area (for example, the features extracted by the feature extraction module), the detection time, and the detection sequence number will be stored in the database and can be retrieved through the above three data. Through this function, historical defect records can be quickly retrieved and reviewed to support continuous improvement of quality according to the production process.
  • defect detection method and system When the defect detection method and system are applied to the defect detection of non-woven mask cloth, it can also be called a non-woven mask cloth defect detection method and system.
  • defect detection method and system When the defect detection method and system are applied to the defect detection of hygienic and personal care product materials, it can also be called a hygienic and personal care product material intelligent defect detection method and system.
  • the embodiment of the present disclosure also provides a defect detection system, including: an image acquisition component, used to obtain the image to be processed of the object to be detected in the production process; a defect detection device, connected to the image acquisition component, used to obtain the image to be processed from the image acquisition component.
  • the defect detection device includes: an image processing module, which is used to process the image to be processed through a feature extraction module in a deep learning defect detection model, the feature extraction module includes an intermediate feature layer, and the intermediate feature layer includes a discard layer, and the discard layer is used to randomly discard the features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains multiple features after random discarding after being processed by the feature extraction module; the multiple features after random discarding are processed by the detection head in the deep learning defect detection model to obtain multiple category prediction results and multiple position prediction results of the defects in the image to be processed; according to the multiple category prediction results and multiple position prediction results of the defects, the category detection results, position detection results, category uncertainty and position uncertainty of the defects in the image to be processed are obtained.
  • the image acquisition component includes: an image capture device, wherein the image capture device includes an area array camera.
  • the area array camera is used to capture an original image to be processed and perform pixel merging processing on the original image to be processed to generate the image to be processed.
  • An area array camera is a camera used to obtain images of the surface of an object to be detected. It can use different types of sensors. For example, it can be a CCD (Charge-Coupled Device) or array camera or CMOS area array camera, which consists of a fixed array of pixels, each pixel unit can capture an image at the same time, so that the entire image can be collected at one time. Area array cameras can provide higher resolution and sensitivity, especially in low light conditions.
  • binning can be used to optimize the basic camera algorithm to fuse multiple real pixels of an array camera such as a CCD camera, thereby achieving a clear image with a very short exposure time, which is an effective high-speed image capture method.
  • the pixels in the image are merged into "super pixels” or “virtual pixels", for example, 2 ⁇ 2 or 3 ⁇ 3 pixels are merged into one virtual pixel to enhance the image, so that the camera can shoot at a very low exposure time without underexposure, thereby avoiding blur and smearing caused by long exposure time.
  • the pixel binning process is performed in hardware or "on chip".
  • the merged super-pixels have higher signal strength, which helps to obtain clearer images with very short exposure times. Due to the shortened exposure time, the camera can capture images faster and continuously, thus achieving high-speed image capture.
  • the area array camera achieves pixel fusion through binning technology, thereby improving the initial brightness of the image at the hardware layer, ensuring that the camera can capture high-speed moving mask fabric within a shorter exposure time and avoiding motion blurred images.
  • the image acquisition component further includes: a bar light source, located at one side of the image capture device, and irradiating the object to be inspected moving on the production line at a predetermined angle.
  • a bar light source is a special light source with a narrow and long shape, usually using LED or CCFL as the light source.
  • the brightness of a bar light source is more uniform than that of an ordinary light source, which makes it very suitable for lighting or display, and can provide more uniform lighting within a certain space.
  • Bar light sources can be customized according to specific usage scenarios, including adjusting their length, brightness, color and other parameters to meet different lighting needs and achieve the best lighting effect.
  • Bar light sources have high durability and long life, and can maintain stable light output for a long time.
  • Bar light sources can not only adjust brightness and color, but also meet the application needs of various scenarios by changing the shape and direction of the light beam.
  • Bar light sources use LED technology, which is highly efficient and energy-saving, and is more energy-efficient than traditional fluorescent or incandescent lamps. At the same time, it also has better environmental performance, which helps to reduce energy consumption and environmental pollution.
  • the camera may also have its own light source.
  • other light sources may also be used, such as a surface light source, a ring light source, etc.
  • the predetermined angle is 45 degrees.
  • a strip light source is located on one side of the camera and illuminates the mask fabric moving on the production line at a 45-degree angle.
  • This specific lighting angle can maximize the visual contrast of defects, making them more obvious in the image captured by the camera.
  • the oblique lighting method can produce shadows and reflection effects, which help to highlight the unevenness or defects on the surface of the mask fabric. When light shines on the fabric at a certain angle, the defective part may produce different reflection or absorption effects due to different shapes, textures or depths. In this way, the camera can more easily distinguish the difference between defects and normal parts when capturing the image.
  • the 45-degree angle can ensure that the light can be fully It covers the surface of the fabric, while preventing the light from being too direct or too oblique, resulting in heavy or light shadows. Appropriate shadows and reflections help enhance the contrast of defects and make them more prominent in the image.
  • the narrow and long shape of the strip light source allows the light to be concentrated in a narrow area, thereby improving the uniformity and consistency of lighting. This is especially important for inspecting mask fabrics moving on an assembly line, as the movement of the fabric may cause changes in lighting conditions.
  • the stability and consistency of the strip light source helps ensure that the quality of the images captured by the camera is stable and reliable. By reasonably setting the position and angle of the strip light source, the visual contrast of the defects can be maximized, and the quality and accuracy of the images captured by the camera can be improved, thereby achieving effective detection of mask fabric defects.
  • the above 45 degrees does not require strict accuracy, and may have a certain error range, for example, greater than ⁇ 3 degrees.
  • the system includes an industrial computer component, the industrial computer component includes an industrial computer, and the defect detection device is deployed on the industrial computer.
  • a data service component is also deployed on the industrial computer, and the data service component is used to record and count the defect information detected in the production process.
  • the data service component manages all defect data (defect information, such as whether there is a defect in the image, the defect category, location, category uncertainty and location uncertainty, defect level, and any other data related to the defect) through MySQL.
  • the data service component can upload the detected defect information to the cloud server in real time, so as to centrally manage the data and facilitate remote monitoring, analysis and optimization of the production process.
  • the system provided by the embodiment of the present disclosure further includes: an alarm component connected to the defect detection device, configured to receive a trigger signal sent by the defect detection device when the defect detection device detects a defect, and send an alarm message according to the trigger signal.
  • the alarm component may include an alarm, but the present disclosure is not limited thereto.
  • the system provided by the embodiment of the present disclosure further includes: a display connected to the defect detection device, and used to display defect information detected by the defect detection device.
  • the defect detection system 120 may include an image acquisition component 121 and a defect detection device 122.
  • the defect detection device 122 may be arranged in a high-performance industrial computer component.
  • the defect detection device 122 may include an image processing module 1221.
  • the image acquisition component 121 includes an image capture device 1211 (e.g., a surface array camera) and a light source 1212 (e.g., a strip light source).
  • the image capture device 1211 is connected to the industrial computer component.
  • the industrial computer component includes an industrial computer.
  • the defect detection system 120 may further include an alarm component 123, and the alarm component 123 may be connected to the industrial computer component.
  • a high-speed area array camera is connected to a high-performance industrial computer to transmit the collected images to the industrial computer in real time.
  • the alarm component 123 is connected to a high-performance industrial computer.
  • the industrial computer detects that a cloth image/mask cloth image exists
  • a trigger signal is sent to the alarm, which emits an audible and visual signal as an alarm message to prompt workers to handle the corresponding defective masks.
  • the algorithm recognition component/algorithm layer and data service component are all run on high-performance industrial computers.
  • FIG14 shows an exemplary scenario of a defect detection system for mask production.
  • the hardware of the defect detection system 120 may include an image acquisition component/system (including a high-speed camera 1211 'and a strip light source 1212 ', responsible for acquiring image information, such as an image to be processed), a high-performance industrial control computer (Industrial Control Computer, IPC, abbreviation of industrial control computer, responsible for processing the acquired image information and generating detection results, such as category detection results and position detection results and category uncertainty and position uncertainty) 122 ', and optionally, an alarm.
  • IPC Industrial Control Computer
  • a camera link (Camera Link interface) 124 is provided between the high-performance industrial control computer 122 'and the high-speed camera 1211 '.
  • the high-performance industrial control computer 122 ' can be connected to the strip light source 1212 'through RS232.
  • the mask material or mask cloth moves along the moving direction of the mask nonwoven fabric on the cloth guide roller 126 toward the cutting machine 125.
  • the angle between the light emitted by the strip light source 1212' toward the mask cloth on the cloth guide roller 126 and the mask cloth can be, for example, 45 degrees.
  • the mask cloth image/material image captured by camera 1211 '(e.g., camera) is processed by the IPC for defect/flaw identification. If a defect is identified in the image, the IPC sends an audible and visual signal through an alarm to notify the workers on the assembly line to reject the defective product.
  • a high-speed camera 1211 ′ and a light source are installed in a position before the mask material cutting machine 125 .
  • a light source e.g., a bar light source 1212 ′
  • the camera needs to capture images in a short time (e.g., at least 25 frames per second).
  • the disclosed embodiment uses binning to fuse 2*2 pixels into a virtual pixel to enhance the image, enabling the camera to shoot at a very low exposure time without underexposure, thereby avoiding blur and smearing caused by long exposure.
  • the detection system proposed in the embodiment of the present disclosure overcomes the limitations of the related art that it relies too much on mechanical structures and does not adequately describe visual recognition methods.
  • the system uses advanced image capture equipment, intelligent recognition technology combined with deep learning algorithms, and efficient data service support to achieve real-time and accurate detection of defects in high-speed sports mask fabrics.
  • the defect detection system (also referred to as an intelligent defect detection system) provided by the embodiment of the present disclosure includes hardware, an algorithm layer/component (also referred to as a detection algorithm layer) and a data service layer/component.
  • the hardware components of the system include an industrial computer, a high-speed camera (high-speed camera), a light source and an alarm, which work together to capture images and issue warnings when defects are detected.
  • the algorithm layer includes technologies such as preprocessing, channel pruning, deep separable convolution and Bayesian optimization, which effectively improve the detection speed and accuracy.
  • the data service component includes image and data storage, defect data analysis, defect data reporting and automated defect labeling.
  • the detection algorithm is the core part of the detection system, while the hardware and data services make the defect detection system faster, more accurate and easier to use.
  • the camera may be, for example, a CIS camera (CMOS Image Sensor, complementary metal oxide semiconductor image sensor), a CCD camera, or the like.
  • CIS camera complementary metal oxide semiconductor image sensor
  • CCD camera complementary metal oxide semiconductor camera
  • the camera and light source are responsible for collecting image information on the surface of the mask cloth to obtain the image of the mask cloth;
  • the industrial computer is responsible for processing the collected image and identifying defects on the surface of the mask cloth;
  • the alarm is used to remind workers to deal with defective masks that do not meet quality standards.
  • the mask cloth image collected by the camera is processed by the industrial computer for defect identification, if a defect is identified in the image, the industrial computer will send an audible and visual signal through the alarm to notify the workers on the production line to remove the defective products.
  • the algorithm layer is the core of the system intelligence and is responsible for defect location and classification.
  • the algorithm layer includes image preprocessing, channel pruning, deep separable convolution processing, and Bayesian optimization modules.
  • the role of preprocessing is to preprocess the image before defect detection, and improve the difference between defects and background through image enhancement and filtering, so as to better identify defects;
  • channel pruning is based on the needs of lightweight model design, and the channel pruning of the model is carried out.
  • the contribution of the convolution channel to the network output is measured by L2-norm, and the channel is pruned according to the contribution degree to reduce its model parameters and improve the processing speed; deep separable convolution processing is also a model lightweight means.
  • Deep separable convolution can reduce 85% of the operation parameters compared with ordinary convolution operations, while ensuring that the accuracy is not lost;
  • Bayesian optimization is to optimize the model using the Bayesian method.
  • a dropout layer is added to the model to create randomness to simulate the Bayesian posterior distribution to enhance the accuracy and reliability of defect detection.
  • the Bayesian method can quantify the uncertainty of the prediction.
  • the model In a detection system, the model not only predicts outcomes, but also provides information about how confident those predictions are. This quantification of uncertainty allows the system to process information more intelligently, such as taking a more cautious approach to predictions that the model is not confident about.
  • the detection algorithm can predict four main types of mask material defects: stickers (stickers refer to the connection positions between each roll of mask cloth, which are pasted together on the assembly line by adhesive tape, so they can also be called adhesive tape), stains (stains), printing problems (some masks are patterned and printed, and there will be printing errors) and mask cloth creases.
  • stickers refer to the connection positions between each roll of mask cloth, which are pasted together on the assembly line by adhesive tape, so they can also be called adhesive tape
  • stains stains
  • printing problems some masks are patterned and printed, and there will be printing errors
  • mask cloth creases During online operation, the entire system is end-to-end. It takes the acquired image as input, and the model first removes irrelevant background, which refers to the area other than the mask cloth captured in the image. Efficient image feature extraction is achieved through channel pruning and depth-separable convolution. Then, Bayesian methods for random and epistemic uncertainty are incorporated into the detection architecture. Random uncertainty stems from the randomness of the training data itself
  • the Bayesian method includes: adding a dropout layer to the intermediate feature layer of the backbone to simulate the effect of the Bayesian posterior distribution to obtain different outputs about the features.
  • the distribution of these outputs provides an approximation of the prediction uncertainty, where the variability of the prediction results reflects the uncertainty of the model; modifying the detection head, adding the mean and variance tables of each parameter based on the model output including the coordinates, category and confidence of the bounding box
  • the loss function is adjusted to penalize excessive uncertainty and inaccurate predictions.
  • a negative log-likelihood loss function is added to the detection head to calculate the Bayesian probability loss. It has two functions: first, it penalizes excessive uncertainty (because if the model is very uncertain about its predictions, the loss will increase). Second, it penalizes inaccurate predictions (large prediction errors increase the loss).
  • the disclosed embodiment optimizes the defect category, location and uncertainty prediction process of the mask fabric defect detection system.
  • the output of the improved detection head is a probability distribution, and a new Bayesian loss, such as a negative log-likelihood loss, is added to increase the penalty for uncertain results, thereby optimizing back propagation.
  • Dropout is added to the backbone intermediate feature layer to simulate the uncertainty of the model parameters and realize the Bayesian posterior distribution.
  • Data uncertainty is determined by data noise, annotation variance, and sample quality (such as lighting, resolution, and contrast).
  • the model estimates the uncertainty of its predictions, so that the model improves its robustness and reliability.
  • the implementation of these innovations is important for improving the accuracy and efficiency of the mask fabric defect detection system, especially in a high-speed production environment. Through the combination of these technologies, effective detection of high-speed motion mask fabrics can be achieved, while improving the robustness and reliability of the system.
  • the data service layer is the information processing and decision support center in the mask fabric defect detection system. It is responsible for integrating the output of the algorithm layer and performing subsequent data management and decision making.
  • the data service layer includes defect and image data storage, defect data aggregation, defect data transmission, and automatic decision-making modules.
  • Image and defect data storage is responsible for collecting and saving all the original image data obtained through the image acquisition system and the result data processed by the algorithm layer (including one or more of the location of the defect, the defect type of the defect, the uncertainty of each defect, and the image characteristics of the defect area).
  • the defect data analysis will unify and analyze the various types of defect data detected, and summarize the scattered defect information into an intuitive report to facilitate monitoring and evaluation of production quality.
  • Defect data transmission ensures that the detected defect information can be quickly and accurately transmitted to the downstream links or quality control departments of the production line, providing support for real-time response and handling of defects.
  • Automatic decision making is based on aggregated analysis of defect data and the actual situation of the production line.
  • This module uses an intelligent decision-making algorithm to analyze the distribution of defect detection and uncertainty probability, and generates reports to help producers understand which types of defects are most common in the current mask production, the frequency of serious defects, etc., so that producers can understand the defects of the production process from a more macro perspective to optimize the production process.
  • the data service layer is responsible for storing, analyzing and transmitting inspection data, and implementing corresponding processing strategies according to the severity of defects.
  • the data service layer also enhances the efficiency of manual inspection, intuitively displays the defect level/defect level through the thickness of the defect border, and provides the function of retrieving defect information/defect information, supporting quality tracking and production process optimization.
  • the disclosed embodiments relate to a mask cloth defect detection system in a high-speed production environment.
  • the system integrates high-performance image capture hardware (camera and light source), advanced deep learning algorithms (including the deep learning defect detection model mentioned in the above embodiments), and a comprehensive data service layer to achieve real-time defect detection and classification of high-speed moving mask fabrics.
  • high-performance image capture hardware camera and light source
  • advanced deep learning algorithms including the deep learning defect detection model mentioned in the above embodiments
  • a comprehensive data service layer to achieve real-time defect detection and classification of high-speed moving mask fabrics.
  • the mask cloth defect detection system provided by the embodiment of the present disclosure, by setting an image acquisition component, an algorithm layer and a data service component, can efficiently and accurately detect defects on the mask cloth passing through without affecting the speed of the mask cloth production line, and automatically mark the defects when defects are detected, greatly improving production efficiency and product quality.
  • the system also ensures a high level of detection reliability. For example, in combination with the Bayesian method, an uncertainty score will be given for each predicted defect point (for example, refer to the above formula (2)). If the uncertainty score of a defect point is high, the defect point will be sent to manual marking.
  • High speed in the embodiments of the present disclosure means satisfying the global shutter, and the maximum frame rate is, for example, more than 60fps.
  • High speed means, for example, that when shooting mask cloth at a mask production speed of 500 pieces/min on an assembly line, the image does not appear to be dynamic blurred, and the camera frame sampling interval does not lose the mask cloth area information.
  • a mask cloth defect detection system which includes: an image acquisition component, including a high-speed array camera, a strip light source, and a camera and light source bracket.
  • the high-speed array camera is used to collect the surface image of the mask cloth, and the light source is used to highlight the visual difference between the defect and the background;
  • a high-performance industrial computer component including an industrial computer equipped with a high-performance GPU and a display, the industrial computer is used to detect the defects in the image collected by the camera in real time, and the display is installed at the end of the mask production line to facilitate workers to confirm the image information of the mask defect; an alarm component is installed at the end of the mask production line.
  • the alarm triggers to prompt the worker to deal with it in time; a defect recognition algorithm component improves the model by analyzing the defect characteristics and image characteristics, and optimizes its calculation speed and accuracy.
  • the image characteristics may include one or more of the changes in the background in the image, the changes in the surface texture of the detected object, the interference of the surrounding ambient light on the image, etc.
  • the processing involving image characteristics is to adjust the camera exposure and remove the image background.
  • the defect point characteristics are the anchor box distribution characteristics of the defect points and the Euclidean distance between the defect point feature classes.
  • the anchor box distribution characteristics of the defect points and the Euclidean distance between the defect point feature classes are analyzed. Clustering is performed according to the anchor box distribution characteristics to generate the most prioritized anchor box, and sorting and comparison are performed according to the Euclidean distance between the defect point feature classes to optimize the defect point classification accuracy.
  • the other is hard sample mining and optimization: First, according to the distribution of anchor boxes, find defects of different proportions. For example, for defects with an aspect ratio of 1:1, for defect A, calculate the Euclidean distance between A and other types of defects with an aspect ratio of 1:1. Then find the closest defect of a different type and mark it as N (Negative), and find the farthest defect of the same type and mark it as P (Positive). Then these three defect samples constitute a hard sample triplet. By constructing multiple hard sample triplets of defects with different aspect ratios, a hard sample dataset is finally constructed. Then, the model is deeply trained for the hard sample dataset to make the model more suitable for this hard sample, thereby achieving better detection results. That is, the most difficult defects to distinguish are selected for targeted optimization.
  • strip light sources are high power and high light intensity. Because the camera needs to shoot highly moving objects, the exposure The time cannot be too long, so a strong enough light source is needed to provide brightness. The width and length of the selected strip light source should be sufficient to cover the camera shooting area.
  • the data service component records and counts all defects detected on the assembly line, updates them to the database in real time, performs data analysis, and helps workers analyze the causes of defects, thereby optimizing the process.
  • the purpose of the present disclosure is to provide a mask cloth defect detection system, which aims to at least to some extent solve the problems encountered in the prior art in mask cloth defect detection, such as insufficient detection accuracy.
  • the system significantly improves the accuracy and efficiency of defect identification by introducing advanced image acquisition and processing technology, thereby improving the process flow in the relevant technical field.
  • the disclosed embodiments provide a complete online high-speed defect detection system, which is a systematic improvement and innovation for a specific business scenario (mask cloth moving at high speed on a production line).
  • the system lightweights the model as a whole, introduces Bayesian estimation into the model, enhances training samples through AIGC technology, solves the hard sample problem through quantitative learning to enable the model to converge better, improves image acquisition quality through binning, designs a camera automatic exposure algorithm, and designs human-computer interaction logic to optimize the defect decision-making process.
  • the present disclosure also provides a defect detection system, comprising: an image capture device for collecting the image to be processed of the object to be inspected in the production process; a processor and a memory for storing computer programs; when the processor executes the computer program, the steps of the method described in any of the above embodiments are implemented.
  • the memory includes a memory for storing a defect detection program (implementing the functions of the above algorithm layer) and a memory for storing a service data program (implementing the functions of the above data service layer).
  • Fig. 16 is a schematic diagram of the structure of a defect detection device according to an exemplary embodiment of the present disclosure.
  • the defect detection device 1500 provided by the embodiment of the present disclosure includes an image acquisition module 1510 and an image processing module 1520 .
  • the image acquisition module 1510 can be used to obtain an image to be processed.
  • the image processing module 1520 can be used to process the image to be processed through the feature extraction module in the deep learning defect detection model, the feature extraction module includes an intermediate feature layer, the intermediate feature layer includes a discard layer, the discard layer is used to randomly discard the features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains multiple features after random discarding processing after being processed by the feature extraction module.
  • the image processing module can also be used to process multiple features after random discarding through the detection head in the deep learning defect detection model to obtain multiple category prediction results and multiple position prediction results of defects in the image to be processed.
  • the image processing module may also be used to predict the defect according to the multiple categories and multiple positions.
  • the result is used to obtain a category detection result, a position detection result, a category uncertainty and a position uncertainty of the defect in the image to be processed.
  • the image processing module 1520 is also used to: obtain the first category detection result, first position detection result, first category uncertainty and first position uncertainty of the first defect according to multiple first category prediction results and multiple first position prediction results of the first defect; obtain the second category detection result, second position detection result, second category uncertainty and second position uncertainty of the second defect according to multiple second category prediction results and multiple second position prediction results of the second defect.
  • the image acquisition module 1510 is further used to: obtain the image to be processed from an image capturing device in an image acquisition component, wherein the image acquisition component is used to obtain the image to be processed of the object to be inspected in a production process.
  • the defect detection device 1500 may further include: a speed acquisition module, which may be used to obtain the current production speed of the object to be detected and the current processing speed of the deep learning defect detection model on the image to be processed.
  • the image processing module 1520 may also be used to adjust the number of parallel threads or parallel processes used by the deep learning defect detection model to process the image to be processed according to the current production speed and the current processing speed.
  • the image acquisition module 1510 is also used to: obtain the image to be processed from the image capturing device in the image acquisition component; if the grayscale value and grayscale distribution of the image to be processed do not meet the preset conditions, send a control signal to adjust the exposure parameters of the image capturing device; obtain a new image to be processed from the image capturing device after adjusting the exposure parameters, until the image to be processed that meets the preset conditions is obtained.
  • the defect detection device 1500 may further include: a sending module, configured to send prompt information for the defect if the category uncertainty and/or position uncertainty of the defect is greater than an uncertainty threshold.
  • the defect detection device 1500 may also include: a data processing module, which is used to count the pixel number information of each defect detected in the image to be processed; determine the defect level of each defect based on the pixel number information of each defect; and perform corresponding processing based on the defect level of each defect.
  • a data processing module which is used to count the pixel number information of each defect detected in the image to be processed.
  • the data processing module may also be used to: in the image to be processed, mark defects at different defect levels with frames of different thicknesses.
  • the defect detection device 1500 may further include: a data processing module, which can be used to save the image to be processed and the category detection results, position detection results, category uncertainty, position uncertainty, defect level, defect area image, image features of the defect area, detection time and detection serial number of the defects detected in the image to be processed; receive and retrieve corresponding defect information based on one or more of the input image features, detection time and detection serial number of the defect area.
  • a data processing module which can be used to save the image to be processed and the category detection results, position detection results, category uncertainty, position uncertainty, defect level, defect area image, image features of the defect area, detection time and detection serial number of the defects detected in the image to be processed.
  • the defect detection device 1500 may further include: a model training module, which may be used to obtain a target training data set, wherein the target training data set includes a target sample image and annotated categories and annotated locations of target sample defects in the target sample image; inputting the target sample image into the deep learning defect detection model to obtain multiple category prediction results and multiple location prediction results of the target sample defects; and According to the multiple category prediction results and multiple position prediction results of the target sample defects, the category detection results, position detection results, category uncertainty and position uncertainty of the target sample defects are obtained; according to the labeled category and category detection results, labeled position and position detection results, as well as category uncertainty and position uncertainty of the target sample defects, a loss function is constructed to train the deep learning defect detection model.
  • a model training module which may be used to obtain a target training data set, wherein the target training data set includes a target sample image and annotated categories and annotated locations of target sample defects in the target sample image
  • the target sample image includes a target sample image and annotated categories and annot
  • the model training module can also be used to: obtain the initial sample image and the annotated category and annotated position of the initial sample defect in the initial sample image; perform any one or more of random cropping, rotation, scaling, illumination change simulation, and defect synthesis on the initial sample image to generate an expanded sample image and the annotated category and annotated position of the extended sample defect in the expanded gain image.
  • the target sample image includes the initial sample image and the expanded sample image.
  • the model training module can also be used to: extract initial sample defects in the initial sample image; synthesize the initial sample defects with the augmented background image to generate the augmented sample image; and/or, learn the initial sample image through a deep learning generation model to generate the augmented sample image.
  • the model training module can also be used to: obtain candidate sample images and the labeled categories and labeled positions of candidate sample defects in the candidate sample images; obtain the output features of the candidate sample images through the feature extraction module; obtain the feature distances between the candidate sample images based on the output features of each candidate sample image; select candidate sample images whose feature distances are greater than a first distance threshold among candidate sample images with the same labeled categories, and/or candidate sample images whose feature distances are less than a second distance threshold among candidate sample images with different labeled categories, as target sample images in the target training data set.
  • the model training module can also be used in the training process of the deep learning defect detection model to: extract the weight tensor of each convolutional layer in the deep learning defect detection model; obtain the Euclidean norm of each channel in the weight tensor; sort each convolutional layer according to the Euclidean norm of its channel, and prune the corresponding channel according to a pruning threshold or a pruning ratio.
  • Fig. 17 schematically shows a schematic structural diagram of an electronic device 1600 according to an embodiment of the present disclosure.
  • the electronic device 1600 shown in Fig. 17 includes a processor 1610, and the processor 1610 can call and run a computer program from a memory to implement the method in the embodiment of the present disclosure.
  • the electronic device 1600 may further include a memory 1620.
  • the processor 1610 may call and run a computer program from the memory 1620 to implement the method in the embodiment of the present disclosure.
  • the memory 1620 may be a separate device independent of the processor 1610 , or may be integrated into the processor 1610 .
  • the electronic device 1600 may further include a transceiver 1630 , and the processor 1610 may control the transceiver 1630 to communicate with other devices, specifically, may send information or data to other devices, or receive information or data sent by other devices.
  • the processor 1610 , the memory 1620 , and the transceiver 1630 may implement bidirectional communication with each other via the communication bus 1640 .
  • processor of the embodiment of the present disclosure may be an integrated circuit chip with signal processing capabilities.
  • each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in software form.
  • the above-mentioned processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
  • DSP digital signal processor
  • ASIC application-specific integrated circuit
  • FPGA field programmable gate array
  • the methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure may be implemented or executed.
  • the general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
  • the steps of the method disclosed in the embodiments of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor.
  • the software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc.
  • the storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above-mentioned method in combination with its hardware.
  • the memory in the embodiments of the present disclosure may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories.
  • the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.
  • the volatile memory may be a random access memory (RAM), which is used as an external cache.
  • RAM synchronous link DRAM
  • SRAM static RAM
  • DRAM dynamic RAM
  • SDRAM synchronous DRAM
  • DDR SDRAM double data rate synchronous DRAM
  • ESDRAM enhanced SDRAM
  • SLDRAM synchronous link DRAM
  • DR RAM direct RAM
  • the embodiments of the present disclosure also provide a computer-readable storage medium for storing a computer program.
  • the computer program enables a computer to execute the corresponding processes in the various methods of the embodiments of the present disclosure.
  • the embodiments of the present disclosure also provide a computer program product, including computer program instructions.
  • the computer program instructions enable a computer to execute the corresponding processes in the various methods of the embodiments of the present disclosure.
  • the embodiments of the present disclosure also provide a computer program. When the computer program is run on a computer, it enables the computer to execute the corresponding processes in the various methods of the embodiments of the present disclosure. For the sake of brevity, it is not repeated here.
  • the present disclosure is applicable to the field of artificial intelligence technology, and is used to solve the problem in related technologies that defect prediction is inaccurate and category uncertainty and position uncertainty cannot be output, so as to achieve the effect of quickly and effectively predicting the category and location of defects and evaluating the category uncertainty and position uncertainty of the prediction results.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • Molecular Biology (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Probability & Statistics with Applications (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)

Abstract

本公开提供一种瑕疵检测方法及相关设备，属于计算机和缺陷检测技术领域。该方法包括：获得待处理图像；通过深度学习瑕疵检测模型中的特征提取模块处理待处理图像，特征提取模块包括中间特征层，中间特征层中包括丢弃层，丢弃层用于对中间特征层处理待处理图像后提取的特征进行随机丢弃，以使得待处理图像经过特征提取模块处理后获得经过随机丢弃处理后的多个特征；通过深度学习瑕疵检测模型中的检测头对经过随机丢弃处理后的多个特征进行处理，获得待处理图像中的瑕疵的多个类别预测结果、多个位置预测结果；根据瑕疵的多个类别预测结果和多个位置预测结果，获得待处理图像中的瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性。 (图1)

Description

瑕疵检测方法及相关设备
相关申请的交叉引用
本公开要求于2024年06月03日提交的申请号为202410709277.5、名称为“瑕疵检测方法及相关设备”的中国专利申请的优先权，并要求于2023年6月15日提交的申请号为63/508,275的美国专利申请的优先权，该中国专利申请和该美国专利申请的全部内容通过引用全部并入本文。
技术领域
本公开涉及计算机和缺陷检测技术领域，具体而言，涉及一种瑕疵检测方法、装置及系统、电子设备、计算机可读存储介质及计算机程序产品。
背景技术
在现实的生产生活场景中，产品或物品可能会存在各种缺陷。这些产品或物品在出厂之前，需要对其进行缺陷检测，以避免有缺陷的产品或物品流通到市场上。
发明内容
本公开实施例提供了一种瑕疵检测方法，包括：获得待处理图像；通过深度学习瑕疵检测模型中的特征提取模块处理所述待处理图像，所述特征提取模块包括中间特征层，所述中间特征层中包括丢弃层，所述丢弃层用于对所述中间特征层处理所述待处理图像后提取的特征进行随机丢弃，以使得所述待处理图像经过所述特征提取模块处理后获得经过随机丢弃处理后的多个特征；通过所述深度学习瑕疵检测模型中的检测头对经过随机丢弃处理后的多个特征进行处理，获得所述待处理图像中的瑕疵的多个类别预测结果、多个位置预测结果；根据所述瑕疵的多个类别预测结果和多个位置预测结果，获得所述待处理图像中的所述瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性。
本公开实施例提供了一种瑕疵检测装置，包括：图像获得模块，用于获得待处理图像；图像处理模块，用于通过深度学习瑕疵检测模型中的特征提取模块处理所述待处理图像，所述特征提取模块包括中间特征层，所述中间特征层中包括丢弃层，所述丢弃层用于对所述中间特征层处理所述待处理图像后提取的特征进行随机丢弃，以使得所述待处理图像经过所述特征提取模块处理后获得经过随机丢弃处理后的多个特征；所述图像处理模块还用于，通过所述深度学习瑕疵检测模型中的检测头对经过随机丢弃处理后的多个特征进行处 理，获得所述待处理图像中的瑕疵的多个类别预测结果、多个位置预测结果；所述图像处理模块还用于，根据所述瑕疵的多个类别预测结果和多个位置预测结果，获得所述待处理图像中的所述瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性。
本公开实施例提供了一种瑕疵检测系统，包括：图像采集组件，用于获取处于生产过程中的待检测对象的所述待处理图像；瑕疵检测装置，与所述图像采集组件连接，用于从所述图像采集组件获得所述待处理图像；其中，所述瑕疵检测装置包括：图像处理模块，用于通过深度学习瑕疵检测模型中的特征提取模块处理所述待处理图像，所述特征提取模块包括中间特征层，所述中间特征层中包括丢弃层，所述丢弃层用于对所述中间特征层处理所述待处理图像后提取的特征进行随机丢弃，以使得所述待处理图像经过所述特征提取模块处理后获得经过随机丢弃处理后的多个特征；通过所述深度学习瑕疵检测模型中的检测头对经过随机丢弃处理后的多个特征进行处理，获得所述待处理图像中的瑕疵的多个类别预测结果、多个位置预测结果；根据所述瑕疵的多个类别预测结果和多个位置预测结果，获得所述待处理图像中的所述瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性。
本公开实施例提供了一种瑕疵检测系统，包括：图像拍摄设备，用于采集处于生产过程中的待检测对象的所述待处理图像；处理器和用于存储计算机程序的存储器；当所述处理器执行所述计算机程序时，实现如本公开任一实施例所述方法中的步骤。
本公开实施例提供了一种电子设备，包括：一个或多个处理器；存储器，配置为存储一个或多个程序，当所述一个或多个程序被所述一个或多个处理器执行时，使得所述电子设备实现本公开任一实施例中所述的方法。
本公开实施例提供了一种计算机可读存储介质，其上存储有计算机程序，当所述计算机程序在计算机上运行时，使得计算机执行时实现本公开任一实施例中所述的方法。
本公开实施例提供了计算机程序产品，包括计算机程序，该计算机程序被计算机执行时实现本公开实施例中所述的方法。
附图说明
图1是本公开一示例性实施例的瑕疵检测方法的流程图。
图2是本公开一示例性实施例的深度学习瑕疵检测模型的示意图。
图3是本公开另一示例性实施例的深度学习瑕疵检测模型的示意图。
图4是图3所示的第一跨阶段卷积块和第二跨阶段卷积块的示意图。
图5是图4所示的第一中间层和第二中间层的示意图。
图6是图5所示的第一深度可分离卷积层和第二深度可分离卷积层的示意图。
图7是图3所示的第三跨阶段卷积块的示意图。
图8是本公开又一示例性实施例的深度学习瑕疵检测模型的示意图。
图9是本公开再一示例性实施例的深度学习瑕疵检测模型的示意图。
图10是本公开一示例性实施例的空间金字塔池化块的示意图。
图11是本公开实施例中的空间金字塔池化块的替代实施例的示意图。
图12是图9所示的注意力块的示意图。
图13是本公开一示例性实施例的瑕疵检测系统的示意图。
图14是本公开另一示例性实施例的瑕疵检测系统的示意图。
图15是本公开又一示例性实施例的瑕疵检测系统的示意图。
图16是本公开一示例性实施例的瑕疵检测装置的结构示意图。
图17示意性示出了根据本公开的一实施例的电子设备的示意性结构图。
具体实施方式
为了使得本公开的目的、技术方案和优点更为明显，下面将参照附图详细描述根据本公开的示例实施例。在附图中，相同的参考标号自始至终表示相同的元件。应当理解：这里描述的实施例仅仅是说明性的，而不应被解释为限制本公开的范围。
本公开实施例中，术语“模块”或“单元”是指有预定功能的计算机程序或计算机程序的一部分，并与其他相关部分一起工作以实现预定目标，并且可以通过使用软件、硬件(如处理电路或存储器)或其组合来全部或部分实现。同样的，一个处理器(或多个处理器或存储器)可以用来实现一个或多个模块或单元。此外，每个模块或单元都可以是包含该模块或单元功能的整体模块或单元的一部分。
在本公开中，术语“第一”、“第二”、“第三”仅用于描述的目的，而不能理解为指示或暗示相对重要性；术语“多个”则指两个或两个以上，除非另有明确的限定。术语“安装”、“相连”、“连接”、“固定”等术语均应做广义理解，例如，“连接”可以是固定连接，也可以是可拆卸连接，或一体地连接；“相连”可以是直接相连，也可以通过中间媒介间接相连。对于本领域的普通技术人员而言，可以根据具体情况理解上述术语在本公开中的具体含义。
本公开的描述中，需要理解的是，术语“上”、“下”、“左”、“右”、“前”、“后”等指示的方位或位置关系为基于附图所示的方位或位置关系，仅是为了便于描述本公开和简化描述，而不是指示或暗示所指的装置或单元必须具有特定的方向、以特定的方 位构造和操作，因此，不能理解为对本公开的限制。
在本说明书的描述中，术语“一个实施例”、“一些实施例”、“具体实施例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或特点包含于本公开的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不一定指的是相同的实施例或实例。而且，描述的具体特征、结构、材料或特点可以在任何的一个或多个实施例或示例中以合适的方式结合。
以上所述仅为本公开的优选实施例而已，并不用于限制本公开，对于本领域的技术人员来说，本公开可以有各种更改和变化。凡在本公开的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本公开的保护范围之内。
图1实施例提供的瑕疵检测方法可以有任意的电子设备执行，例如终端和/或服务器，本公开对此不做限定。如图1所示，本公开实施例提供的方法包括如下步骤。
在S110中，获得待处理图像。
本公开实施例中的待处理图像可以是任意需要检测其是否包含缺陷或瑕疵的待检测对象的图像。该待检测对象例如可以是卫生及个人护理产品，例如医疗、卫生用产品，如手术衣、防护服、消毒包布、口罩、尿片、民用抹布、擦拭布、湿面巾、魔术毛巾、柔巾卷、美容化妆品等；也可以是家庭装饰用产品，如贴墙布、台布、床单、床罩等；还可以是服装用产品，如衬里、粘合衬、絮片、定型棉、各种合成革底布等；也可以是工业用产品，如过滤材料、绝缘材料、水泥包装袋、土工布、包覆布等；也可以是农业用产品，如作物保护布、育秧布、灌溉布、保温幕帘等；还可以是其它产品，如太空棉、保温隔音材料、吸油毡、烟过滤嘴、袋包茶叶袋等。
在下面的实施例中，均以待检测对象为口罩(包括已经缝制成型的口罩和处于裁剪过程中的口罩)，对应的该待处理图像为口罩图像为例进行举例说明。在另一些实施例中，待检测对象为处于生产流水线(以下简称为生产线)高速运动的口罩布，对应的待处理图像为口罩布图像。例如，该口罩布可以是无纺布，但本公开并不限定于此，也可以采用其它合适的材料作为口罩布。此时，本公开涉及无纺布材料检测技术领域，主要面向口罩布瑕疵检测细分领域，该方法也可以称之为无纺口罩布瑕疵检测方法或口罩布瑕疵检测方法。
例如，采用无纺布制作口罩的工序主要包括材料准备、裁剪、缝制和整理等环节。在准备好原材料后，进行裁剪。将无纺布铺平，用切割机进行剪裁，剪出口罩的主体部分。接着是缝制环节，将无纺布、鼻梁条、耳带等部件缝制在一起，形成完整的口罩。最后，还需要进行整理和质量检查，确保每个口罩都符合质量标准，相关技术中主要涉及成品口 罩的瑕疵检测，未涉及处于口罩生产线上的口罩布的瑕疵检测，无法在生产阶段实现质量管理和控制。
由于近年来爆发的全球性流行病(例如covid-19)对公共安全造成了巨大的威胁，使得人们对口罩的需求激增。而口罩质量对于流行病的防控具有不可忽视的作用。因此，口罩生产企业需要在保证质量的前提下，提高口罩的生产效率。而相关技术中，主要采用人工检测方式，基于人工检测实现口罩质量控制的方式缺乏效率，而且存在标准不统一和精度低的问题。本公开实施例提供的方法通过机器视觉技术实现对处于生产流水线上的口罩布的自动化的瑕疵检测，能够在保证口罩质量的前提下，较大的提高生产效率，同时可以有效减少人工成本。
在S120中，通过深度学习瑕疵检测模型中的特征提取模块处理所述待处理图像，所述特征提取模块包括中间特征层，所述中间特征层中包括丢弃层，所述丢弃层用于对所述中间特征层处理所述待处理图像后提取的特征进行随机丢弃，以使得所述待处理图像经过所述特征提取模块处理后获得经过随机丢弃处理后的多个特征。
本公开实施例中的深度学习瑕疵检测模型是一种基于深度学习的、能够对图像进行处理，检测出图像中是否存在瑕疵，以及在检测出瑕疵时，预测出该瑕疵的瑕疵信息的模型，该瑕疵信息例如包括瑕疵的尺寸、位置、类别等中的任意一种或者多种。例如，该深度学习瑕疵检测模型可以基于目标检测模型，例如YOLO(You Only Look Once)系列(例如YOLOv3、YOLOv4、Yolov5等)、Faster R-CNN、SSD、CNN(卷积神经网络)、循环神经网络(RNN)和变换器(Transformer)等中的任意一种或者多种的组合。在下面的实施例中，以YOLO模型进行举例说明，但本公开并不限定于此。
本公开实施例中的深度学习瑕疵检测模型是一种实时目标检测模型，将输入的待处理图像分割成网格，每个网格负责检测一个目标(即瑕疵)，并预测目标的边界框(也称之为锚框)和类别。深度学习瑕疵检测模型包括特征提取模块，特征提取模块为深度学习瑕疵检测模型中的骨干网络(Backbone)，主要负责提取输入的待处理图像的特征。骨干网络可以包括卷积层、池化层和其它可能的网络层等，能够逐步提取待处理图像中的深层特征，用于将待处理图像转化为多层的特征图(feature maps)。这些特征图随后被用于目标检测任务，如边界框预测和类别分类。例如，骨干网络可以包含Conv模块(卷积模块)、C3模块(由三个标准卷积层组成，每个卷积层具有特定的卷积核大小和步长)和SPPF模块(空间金字塔池化层，用于在多个尺度上聚合特征)等，这些模块通过卷积、池化等操作，提取并融合不同层次的特征信息。
Backbone的中间特征层是指在Backbone网络中，位于输入层之后(如果存在输出层，则位于输入层和输出层之间)的隐藏层(不包括特征金字塔和检测头)，以用于产生待处理图像的特征图。这些中间特征层捕捉了待处理图像在不同层次和不同尺度上的信息。中间特征层的信息不仅可以被用于生成最终的检测结果，有时还可以被用于其他任务，如特征融合或进一步的分析。例如，可以利用Backbone的中间特征层与其他网络部分(如FPN，即特征金字塔网络/模块)进行融合，以提高检测性能。
本公开实施例中，在深度学习瑕疵检测模型的特征提取模块的中间特征层新增了丢弃(dropout)层，dropout通过随机的地“关闭”或“擦除”隐藏层中的一部分神经元(或特征)，使得被删除的神经元不再进行信号的传递，来模拟不同子网络之间的集成学习。这样可以有效地防止模型过拟合，并提高其在未见数据(即训练数据集中未包含的数据)上的泛化能力。通过dropout，模型被迫学习更加鲁棒的特征表示，因为它不能依赖于任何特定的神经元组合。dropout在一定程度上可以被看作是一种贝叶斯近似方法，通过随机地“关闭”一部分神经元，模拟了模型参数的稀疏性，这与贝叶斯推断中的参数后验分布有一定的相似性。dropout并不是直接模拟贝叶斯的后验分布，而是通过其随机性提供了一种正则化的效果，有助于减少过拟合。本公开实施例将dropout添加在Backbone的中间特征层，可以利用其防止过拟合的能力，提高模型的泛化性能。同时还可以降低神经网络的复杂度和参数数量。如果dropout只在训练阶段开启，在推理阶段关闭，则是用于防止模型过拟合。本公开实施例中，不论训练阶段还是推理阶段均开启dropout，以用于模拟贝叶斯不确定。
在S130中，通过所述深度学习瑕疵检测模型中的检测头对经过随机丢弃处理后的多个特征进行处理，获得所述待处理图像中的瑕疵的多个类别预测结果、多个位置预测结果。
本公开实施例在dropout的作用下，通过随机关闭隐藏层中的神经元(让一部分的隐藏层的节点值为0)，可以随机刻画隐藏层的网络结构的生成过程，例如神经网络的层数，每一层的神经元的连接方式等，能够得到该中间特征层的多个不同网络结构，从而对同一个待处理图像(在训练过程中是对同一个目标样本图像)通过该中间特征层的多个不同的网络结构进行多次预测，从而获得该待处理图像中每个瑕疵的多个类别预测结果和多个位置预测结果。不确定性估计就是计算多次预测的结果的分布指标。
本公开实施例中，类别预测结果是指通过深度学习瑕疵检测模型的检测头(例如其中的深度可分离卷积块)预测输出的该瑕疵所属的类别结果。位置预测结果是指通过深度学习瑕疵检测模型的检测头(例如其中的深度可分离卷积块)预测输出的该瑕疵在该图像(包 括待处理图像和/或目标样本图像)中所处的位置，可以是绝对位置，也可以是相对位置。类别预测结果可以包括预测出来的该瑕疵属于各个类别的概率，位置预测结果可以包括预测出来的该瑕疵在该图像中的边界框的中心坐标、宽度和高度等信息。
在S140中，根据所述瑕疵的多个类别预测结果和多个位置预测结果，获得所述待处理图像中的所述瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性。
本公开实施例中深度学习瑕疵检测模型的检测头在目标检测任务中，主要负责预测待处理图像中瑕疵的类别、边界框的位置以及置信度，还可以预测所预测出来的瑕疵的类别的类别不确定性和位置的位置不确定性。在一些实施例中，检测头可以利用多个卷积层来进一步提取和整合特征，然后输出瑕疵的多个类别预测结果、多个位置预测结果，并基于瑕疵的多个类别预测结果和多个位置预测结果，获得所述待处理图像中的所述瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性。
本公开实施例中，类别检测结果和位置检测结果分别是指深度学习瑕疵检测模型的检测头对外输出的该瑕疵的类别结果和该瑕疵在该图像中所处的位置。
本公开实施例中，类别不确定性是指采用深度学习瑕疵检测模型预测图像(包括待处理图像和/或目标样本图像)中的瑕疵的类别预测结果时，用于评估其预测的类别结果与该瑕疵的真实类别是否匹配的确定程度的指标，类别不确定性越高，表示该深度学习瑕疵检测模型对该瑕疵的类别预测结果越不确定，类别不确定性越低，表示该深度学习瑕疵检测模型对该瑕疵的类别预测结果越确定。
本公开实施例中，位置不确定性是指采用深度学习瑕疵检测模型预测图像(包括待处理图像和/或目标样本图像)中的瑕疵的位置预测结果时，用于评估其预测的位置结果与该瑕疵的真实位置是否匹配的确定程度的指标，位置不确定性越高，表示该深度学习瑕疵检测模型对该瑕疵的位置预测结果越不确定，位置不确定性越低，表示该深度学习瑕疵检测模型对该瑕疵的位置预测结果越确定。
本公开实施例中，在深度学习瑕疵检测模型中，除了预测边界框的位置和类别，还可以预测输出边界框的置信度，预测输出的置信度是一个介于0和1之间的数值，用于表示模型对检测到的瑕疵的确信程度。置信度可以表示边界框内是否有瑕疵(也可称之为疵点)的概率(也可以称之为目标置信度)，即边界框内是否只是背景还是包含瑕疵。可选的，还可以表示当边界框有瑕疵时，模型预测的边界框与瑕疵的真实框之间的匹配程度；也可以表示当边界框有瑕疵时，模型预测的类别结果与该瑕疵的真实类别之间的匹配程度。例如，置信度可以通过计算预测的边界框与实际瑕疵之间的IoU(Intersection over Union， 交并比)来得出的。如果IoU大于某个设定的阈值，则认为检测到了瑕疵，此时置信度即为该边界框的IoU得分；如果IoU小于设定的阈值，则认为未检测到瑕疵，置信度则为0。
不确定性估计是衡量检测结果可信程度的有效方法。目标检测任务中的不确定性可以分为数据不确定性和模型不确定性。数据不确定性主要来源于真实数据(例如输入的待处理图像或目标样本图像)中的固有噪声，模型不确定性则来源于模型参数的不确定性。本公开实施例中的类别不确定性中可以同时包含数据不确定性和模型不确定性，位置不确定性中也可以同时包含数据不确定性和模型不确定性。其中，数据不确定性占比相对较小，跟噪声、样本的标注质量相关。
当预测不确定性不足时，模型可能会过于自信地给出错误的类别检测结果和位置检测结果。例如，在生产线上的质量检测中，模型可能会错误地将一个有缺陷/瑕疵的产品分类为无缺陷，或者反之。这种错误的分类可能导致生产线的效率低下，甚至影响产品的整体质量。对于需要依赖模型预测结果进行质量控制的生产过程来说，如果模型不能提供可靠和准确的预测，那么这些预测结果就可能不会被最终用户所信任。
在示例性实施例中，可以计算瑕疵的多个类别预测结果的均值作为该瑕疵的类别检测结果。类别预测结果表示检出的瑕疵为哪种类型瑕疵的概率，类别预测结果也可以称之类别置信度，因此类别检测结果也可称之为类别置信度均值。可以将该瑕疵的多个位置预测结果的均值作为该瑕疵的位置检测结果。当采用坐标表示位置，则位置检测结果也可称之为坐标均值。
本公开实施例中，类别不确定性是类别置信度的不确定性，是类别置信度的概率分布指标，即类别置信度概率分布情况表示了类别不确定性，因此也可以称之为类别置信度不确定性，例如可以用类别置信度的方差来计算，此时可称之为类别置信度方差。位置不确定性是边界框的概率分布指标，边界框概率分布情况表示了位置不确定性，例如可以计算位置预测结果的方差来获得位置不确定性。当采用坐标表示位置、且计算坐标的方差作为位置不确定性时，位置不确定性也可称之为坐标方差。
例如，模型输出包括[坐标均值,坐标方差(位置不确定性),类别置信度均值，类别置信度方差(类别不确定性)，目标置信度]。其中，目标置信度表示该检出是否为疵点的概率。类别置信度均值表示该检出为哪种类型疵点的概率。
本公开实施例中，可以设置置信度阈值，若模型输出的目标置信度小于该置信度阈值，则将该目标置信度较低的瑕疵检出，例如提示人工进行判断。该置信度阈值为大于0且小于1的实数，例如可以设置为0.3，但本公开并不限定于此。
假设每个边界框的位置或坐标用(x，y，w，h)表示，其中x，y，w，h分别表示 边界框的中心横坐标和纵坐标、宽度和高度，则坐标均值表示为：坐标方差表示为： 再例如类别置信度均值和类别置信度方差分别是：则类别置信度p(c∣Object)可以表示为(c表示当前类别，k表示遍历所有类别)：
但本公开并不限定于此，例如也可以选择该瑕疵的多个类别预测结果中出现频次最高的类别预测结果作为该瑕疵的类别检测结果，选择该瑕疵的多个位置预测结果中出现频次最高的位置预测结果作为该瑕疵的位置检测结果。由于本公开实施例中的瑕疵的类别检测结果和位置检测结果综合考虑了该瑕疵的多个类别预测结果和多个位置预测结果，因此其输出的检测结果(包括类别检测结果和位置检测结果)的准确性更高。
需要说明的是，本公开实施例中，深度学习瑕疵模型预测输出的类别预测结果是类别置信度均值。模型输出的类别置信度均值采用上述公式(1)进行系统的后处理过程，后处理实际上是在模型输出后新增一个softmax过程，将类别置信度映射到[0,1]之间的实数。可选的，后处理过程还可以保护边界框NMS(非极大值抑制)等。
在示例性实施例中，可以计算瑕疵的多个类别预测结果的方差作为该瑕疵的类别不确定性，将该瑕疵的多个位置预测结果的方差作为该瑕疵的位置不确定性。
本公开不限于直接将方差作为不确定性，例如，还可以通过如下公式计算不确定性：
Uncertainty＝exp(-λ·variance)              (2)
其中λ表示调节项，为大于0且小于1的实数。variance表示多个类别预测结果的方差或多个位置预测结果的方差。Uncertainty表示类别不确定性或位置不确定性。
在一些实施例中，可以设置不确定阈值，以用于过滤不确定性较高的预测结果。
对环境因素的敏感性可能会影响模型的检测性能，诸如光线、阴影和杂乱背景等环境因素。本公开实施例通过优化模型，以提高其在实际环境中的稳定性和准确性。例如，引入贝叶斯估计/贝叶斯预测，来提高模型对自然环境的泛化能力和鲁棒性。贝叶斯估计通过新增了dropout层，随机擦除一部分特征的输出来模拟随机性。同时改了模型的检测头的输出，将输出解释为概率分布，而不仅仅是回归问题，将输出解释为概率分布可以更好地捕捉瑕疵的位置和类别的不确定性。这样，模型不仅能够给出预测的确定性估计，还能够提供关于这些估计的不确定性度量。这对于处理复杂的自然环境尤为重要，因为在这种环境中，目标的位置、大小和外观可能会受到多种因素的影响而发生变化。通过结合贝叶斯估计和dropout层以及将输出解释为概率分布，可以构建出更加健壮和泛化能力更强的深度学习瑕疵检测模型。这样的模型能够更好地适应自然环境中的变化，提高检测的准确性和可靠性。
本公开实施方式提供的瑕疵检测方法，通过在深度学习瑕疵检测模型的特征提取模块 中的中间特征层中新增丢弃层，简化了对后验概率分布的近似，从而不仅能够快速有效的对待处理图像中的瑕疵的类别和位置做出预测，还能够对预测结果的类别不确定性和位置不确定性做出评估。本公开实施例开发了一个更加精确、高效且具有良好泛化能力的新型的深度学习瑕疵检测模型。能够在不牺牲检测速度的情况下，提高瑕疵识别准确率，并能够适应各种生产环境和待检测对象类型的需求，具有广泛的应用前景，例如可以应用于口罩布料瑕疵检测系统中，以对不同口罩类型中的瑕疵进行识别。
如图2所示，本公开实施例提供的深度学习瑕疵检测模块1包括特征提取模块11和检测头12。特征提取模块11包括输入块111和中间特征层112。输入块111用于接收待处理图像，并将该待处理图像发送至中间特征层112。中间特征层112包括丢弃层1121。输入块111还可以用于对接收到的输入的待处理图像进行预处理，如调整尺寸和归一化，以便适应模型的输入要求。
通过深度学习瑕疵检测模型1中的特征提取模块11处理待处理图像，特征提取模块11的中间特征层112中的丢弃层1121，用于对中间特征层112处理待处理图像后提取的特征进行随机丢弃，以使得待处理图像经过特征提取模块11处理后获得经过随机丢弃处理后的多个(具体采样的个数，可以根据实际需求进行设定)特征。然后，通过深度学习瑕疵检测模型1中的检测头12对经过随机丢弃处理后的多个特征分别进行处理，获得待处理图像中的瑕疵的多个类别预测结果和多个位置预测结果。检测头12还根据瑕疵的多个类别预测结果和多个位置预测结果，获得待处理图像中的瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性。
在示例性实施例中，所述中间特征层包括第一跨阶段卷积块，所述丢弃层包括第一丢弃层。其中，所述第一跨阶段卷积块包括第一跨阶段输入层、第一卷积层、第一中间层、第二卷积层、第一融合层、第三卷积层和所述第一丢弃层。所述第一卷积层用于将所述第一跨阶段输入层输入的第一跨阶段输入特征图进行卷积处理，向所述第一中间层输入第一卷积特征图。所述第一中间层用于对所述第一卷积特征图进行处理，向所述第一融合层输入第一待融合特征图。所述第二卷积层用于对所述第一跨阶段输入特征图进行卷积处理，向所述第一融合层输入第二待融合特征图。所述第一融合层用于对所述第一待融合特征图和所述第二待融合特征图进行融合处理，向所述第三卷积层输入第一融合特征图。所述第三卷积层用于对所述第一融合特征图进行卷积处理，向所述第一丢弃层输入第二卷积特征图。所述第一丢弃层用于对所述第二卷积特征图中的特征进行随机丢弃，获得多个第一丢弃特征图，所述待处理图像经过所述特征提取模块处理后获得经过随机丢弃处理后的多个 特征包括多个所述第一丢弃特征图。
本公开实施例中的跨阶段输入层(包括第一至第三跨阶段输入层)、中间输入层(包括第一至第三中间输入层)、深度分离输入层(包括第一至第三深度分离输入层)均为模型中相应模块的输入层，其命名仅是为了区分该输入层属于哪个模块，例如跨阶段输入层表示其为跨阶段卷积块的输入层。输入层用于对输入的特征图或图像数据进行预处理，如调整尺寸，以便适应模型中相应模块的输入要求。
本公开实施例中的卷积特征图(例如包括第一至六卷积特征图)是指进行卷积处理后获得的特征图；待融合特征图(例如包括第一至第六待融合特征图)是指待进行融合处理的特征图，这里的融合处理例如可以是拼接或者级联，例如将第一待融合特征图和第二待融合特征图进行级联获得第一融合特征图。丢弃特征图(例如包括第一至第三丢弃特征图)是指通过相应的dropout层进行特征的随机丢弃处理生成的特征图。空间池化特征图(例如包括第一至第三空间池化特征图)是指经过空间金字塔池化块(例如包括第一至第三空间金字塔池化块)处理后生成的特征图。
在示例性实施例中，所述中间特征层还包括第二跨阶段卷积块和第三跨阶段卷积块，所述丢弃层还包括第二丢弃层和第三丢弃层。
其中，所述第二跨阶段卷积块的输入端连接所述第一跨阶段卷积块的输出端，以分别接收多个所述第一丢弃特征图，并对多个所述第一丢弃特征图进行处理，通过所述第二丢弃层向所述第三跨阶段卷积块输入多个第二丢弃特征图。所述第三跨阶段卷积块的输入端连接所述第二跨阶段卷积块的输出端，以分别接收多个所述第二丢弃特征图，并对多个所述第二丢弃特征图进行处理，通过所述第三丢弃层输出多个第三丢弃特征图。
其中，所述第二跨阶段卷积块包括第二跨阶段输入层、第四卷积层、第二中间层、第五卷积层、第二融合层、第六卷积层和所述第二丢弃层。所述第二跨阶段输入层连接所述第一丢弃层，用于分别将多个所述第一丢弃特征图作为第二跨阶段输入特征图。所述第四卷积层用于将所述第二跨阶段输入层输入的第二跨阶段输入特征图进行卷积处理，向所述第二中间层输入第三卷积特征图。所述第二中间层用于对所述第三卷积特征图进行处理，向所述第二融合层输入第三待融合特征图。所述第五卷积层用于对所述第二跨阶段输入特征图进行卷积处理，向所述第二融合层输入第四待融合特征图。所述第二融合层用于对所述第三待融合特征图和所述第四待融合特征图进行融合处理，向所述第六卷积层输入第二融合特征图。所述第六卷积层用于对所述第二融合特征图进行卷积处理，向所述第二丢弃层输入第四卷积特征图。所述第二丢弃层用于对所述第四卷积特征图中的特征进行随机丢 弃，获得多个第二丢弃特征图，所述待处理图像经过所述特征提取模块处理后获得经过随机丢弃处理后的多个特征还包括多个所述第二丢弃特征图。
本公开实施例中的跨阶段卷积块(例如包括第一至第三跨阶段卷积块)采用了跨阶段部分连接的思想，通过将不同阶段的特征图进行融合，增强了特征的多样性和鲁棒性。这种设计有助于提升模型的性能，特别是在处理复杂的目标检测任务时。具体来说，跨阶段卷积块采用了分阶段处理的方式，将输入的特征图划分为多个阶段，每个阶段都包含一系列卷积操作。在每个阶段的末尾，跨阶段卷积块会将当前阶段的特征图与前一阶段的特征图进行融合，从而得到更丰富的特征表示。这种跨阶段的连接方式有助于模型捕获到不同尺度和层次的特征信息，提高了模型对目标的感知能力。
在另一些实施例中，跨阶段卷积块还可以采用部分连接的策略，即不是将所有阶段的特征图都进行融合，而是选择性地融合部分特征图。这种设计可以减少模型的计算量，同时保持较高的性能。
在示例性实施例中，所述深度学习瑕疵检测模型还包括特征金字塔模块，所述特征金字塔模块包括第一空间金字塔池化块、第二空间金字塔池化块和第三空间金字塔池化块。所述第一空间金字塔池化块连接所述第一跨阶段卷积块，用于接收多个所述第一丢弃特征图。所述第二空间金字塔池化块连接所述第二跨阶段卷积块，用于接收多个所述第二丢弃特征图，并分别对多个所述第二丢弃特征图进行处理，获得多个第二空间池化特征图。所述第三空间金字塔池化块连接所述第三跨阶段卷积块，用于接收多个所述第三丢弃特征图，并分别对多个所述第三丢弃特征图进行处理，获得多个第三空间池化特征图。所述第一空间金字塔池化块还连接所述第二空间金字塔池化块，用于接收多个所述第二空间池化特征图，根据多个所述第一丢弃特征图和多个所述第二空间池化特征图获得多个第一空间池化特征图。所述第二空间金字塔池化块还连接所述第三空间金字塔池化块，用于接收多个所述第三空间池化特征图，根据多个所述第二丢弃特征图和多个所述第三空间池化特征图获得多个第二空间池化特征图。
本公开实施例中，可以在骨干网络(例如特征提取模块11)和检测头12之间增加特征金字塔模块，用于对骨干网络提取的特征进行进一步的处理和增强。在示例性实施例中，该特征金字塔模块可以包括FPN(Feature Pyramid Networks，特征金字塔网络)和PANet(Path Aggregation Network，路径聚合网络)等中的一种或多种，它们通过多尺度特征融合的方式，将不同层次的特征信息结合起来，提高模型的检测性能。在下面的实施例中，以特征金字塔模块包括空间金字塔池化块进行举例说明，但本公开并不限定与此。
空间金字塔池化块(Spatial Pyramid Pooling block，简写为SSP block)采用了空间金字塔池化的技术。空间金字塔池化的思想源自金字塔形状的网格单元的应用，这些网格单元覆盖了不同大小的感受野。每个网格单元都会进行相同的池化操作，最终将所有不同尺寸的特征图连接在一起，形成一个固定长度的特征向量。具体来说，空间金字塔池化块对输入的特征图进行尺度变化，并在每个尺度上进行池化操作。这种多尺度的池化方式使得网络能够处理任意大小的输入，并生成固定大小的输出。它通过将不同尺度的特征图拼接在一起，得到一个多尺度的特征表示，从而提高了模型的灵活性和适应性。由于空间金字塔池化块采用了这种多尺度池化的技术，使得它能够灵活处理不同大小的输入，并生成固定长度的特征表示，故称之为空间金字塔池化块。
在示例性实施例中，所述检测头包括第一深度可分离卷积块、第二深度可分离卷积块和和第三深度可分离卷积块。所述瑕疵包括第一瑕疵、第二瑕疵和第三瑕疵；所述第一深度可分离卷积块与所述第一空间金字塔池化块连接，用于接收多个所述第一空间池化特征图，并分别对多个所述第一空间池化特征图进行处理，获得所述第一瑕疵的多个第一类别预测结果和多个第一位置预测结果。所述第二深度可分离卷积块与所述第二空间金字塔池化块连接，用于接收多个所述第二空间池化特征图，并分别对多个所述第二空间池化特征图进行处理，获得所述第二瑕疵的多个第二类别预测结果和多个第二位置预测结果。所述第三深度可分离卷积块与所述第三空间金字塔池化块连接，用于接收多个所述第三空间池化特征图，并分别对多个所述第三空间池化特征图进行处理，获得所述第三瑕疵的多个第三类别预测结果和多个第三位置预测结果。
本公开实施例中的检测头可以包括两个或两个以上的深度可分离卷积块(例如包括第一至第三深度可分离卷积块)，检测头中的深度可分离卷积块的数量与中间特征层中的跨阶段卷积块的数量对应，当模型还包括特征金字塔模块时，深度可分离卷积块的数量还与特征金字塔模块中的空间金字塔池化块的数量对应。在下面的实施例中，分别以检测头中包括2个或3个深度可分离卷积块举例说明，本公开并不限定于此，例如还可以包括更多的深度可分离卷积块，以提取更多尺度的瑕疵。
本公开实施例中，每个深度可分离卷积块将一个完整的卷积运算分解为两步进行，分别为Depthwise Convolution(深度卷积)与Pointwise Convolution(点卷积)。深度卷积步骤中，例如一个大小为64×64像素、三通道彩色的特征图(例如可以来自对应的跨阶段卷积块的输出或空间金字塔池化块的输出)首先经过第一次卷积运算，此次的卷积完全是在二维平面内进行，且过滤器(Filter)的数量与上一层的Depth(深度，这里是指通道数 而非网络层级)相同。所以一个三通道的特征图经过运算后生成了3个特征图。其中一个Filter包含一个大小为3×3的Kernel(卷积核)。Depthwise Convolution完成后的Feature map数量与输入层的Depth相同。然后再通过Pointwise Convolution利用不同Feature map在相同空间位置上的信息，将这些Feature map进行组合生成新的Feature map。Pointwise Convolution的卷积核的尺寸为1×1×M，M为上一层的Depth。这里的卷积运算会将上一步的Feature mapp在深度方向上进行加权组合，生成新的Feature map。经过Pointwise Convolution之后，同样输出了4张Feature map，与常规卷积的输出维度相同。相同的输入，同样是得到4张Feature map，深度可分离卷积的参数个数是常规卷积的约1/3，运算量可缩小8-9倍。因此，在参数量相同的前提下，采用深度可分离卷积的神经网络层数可以做的更深。
本公开实施例中，检测头中的深度可分离卷积块的结构可参考中间特征层中的深度可分离卷积层(例如第一至第三深度可分离卷积层)，检深度可分离卷积块的卷积核大小可以与深度可分离卷积层不一样，卷积方式可以一样。例如，检测头中的卷积核大小可以分别为1×1和3×3。中间特征层中的深度可分离卷积层的卷积核大小可以分别为1×1，3×3，5×5，7×7。不同卷积核的作用是具有不同的特征感受野，深度可分离卷积可以有效减少卷积参数量从而加强处理速度。例如，对于正常卷积，例如128x3x3x256(c1 w h c2)128表示输入通道数，3x3表示卷积核大小，256表示输出通道数，参数量为294912。深度可分离卷积是先通过一个3x3的深度卷积对输入的每个通道进行卷积，参数量为128x3x3＝1152，再通过1x1点卷积对每个子区域进行卷积扩展通道，参数量为128x1x1x256＝32768。总参数量为1152+32768＝33930。对比普通卷积参数量减少了88％。但是精度只有轻微损失。深度可分离卷积是可以完全替换普通卷积的，激活方式都是一样，连接方式也一样。
在另一些实施例中，检测头中也可以不采用深度可分离卷积块，即检测头可以包括采用正常卷积操作实现预测。
其中，根据所述瑕疵的多个类别预测结果和多个位置预测结果，获得所述待处理图像中的所述瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性，包括：根据所述第一瑕疵的多个第一类别预测结果和多个第一位置预测结果，获得所述第一瑕疵的第一类别检测结果、第一位置检测结果、第一类别不确定性和第一位置不确定性；根据所述第二瑕疵的多个第二类别预测结果和多个第二位置预测结果，获得所述第二瑕疵的第二类别检测结果、第二位置检测结果、第二类别不确定性和第二位置不确定性；根据所述第 三瑕疵的多个第三类别预测结果和多个第三位置预测结果，获得所述第三瑕疵的第三类别检测结果、第三位置检测结果、第三类别不确定性和第三位置不确定性。
本公开实施例中，第一瑕疵、第二瑕疵以及第三瑕疵等是指待处理图像中不同尺度的瑕疵，而不是用于限定瑕疵的数量。
在示例性实施例中，所述第一中间层包括第一中间输入层、第一深度可分离卷积层和第一累加层。所述第一中间输入层用于接收所述第一卷积层输出的第一卷积特征图，并将所述第一卷积特征图分别输入至所述第一深度可分离卷积层和所述第一累加层。所述第一深度可分离卷积层用于对所述第一卷积特征图进行深度可分离卷积处理，向所述第一累加层输入第一深度分离卷积特征图。所述第一累加层对所述第一卷积特征图和所述第一深度分离卷积特征图进行累加处理，获得所述第一待融合特征图。
在示例性实施例中，所述第一深度可分离卷积层包括第一深度分离输入层、第一深度卷积层、第一点卷积层和第一激活函数层。所述第一深度分离输入层用于接收所述第一中间输入层输出的第一卷积特征图，并将所述第一卷积特征图分别输入至所述第一深度卷积层和所述第一激活函数层。所述第一深度卷积层用于对所述第一卷积特征图进行深度卷积处理，并向所述第一点卷积层输入第一深度卷积特征图。所述第一点卷积层用于对所述第一深度卷积特征图进行点卷积处理，并向所述第一激活函数层输入第一点卷积特征图。所述第一激活函数层用于对所述第一点卷积特征图和所述第一卷积特征图进行处理，获得所述第一深度分离卷积特征图。
在示例性实施例中，所述特征提取模块还包括与所述中间特征层连接的注意力块。所述注意力块用于对所述待处理图像进行处理，获得所述待处理图像的下采样特征图，并将所述下采样特征图输入至所述中间特征层。
在示例性实施例中，所述深度学习瑕疵检测模型还包括背景移除模块。所述背景移除模块用于移除所述待处理图像中的背景信息，并将移除所述背景信息之后的所述待处理图像输入至所述特征提取模块。
本公开实施例中，可以通过背景移除从待处理图像中移除或分离出与待检测对象无关的部分，以便更准确地聚焦于待检测对象本身及其可能存在的瑕疵。这些背景可能包括工厂生产线、传送带、支撑结构、环境光照等因素，在图像处理和分析过程中，它们可能引入噪声、干扰边缘或影响颜色、纹理等特征的提取。通过去除背景，可以更加专注于待检测对象本身，从而更准确地识别和定位瑕疵。背景移除可以涉及到如背景建模、阈值分割、形态学操作等，以从原始的待处理图像中分离出前景(即待检测对象)和背景。需要注意 的是，去除背景并不意味着完全消除背景在图像中的所有痕迹。相反，它的目的是减少背景对瑕疵检测过程的干扰，同时保留足够的上下文信息，以便能够准确判断待检测对象的状态和质量。
本公开实施例中，中间特征层可以包括3个或3个以上的跨阶段卷积块，检测头中的深度可分离卷积块的数量与中间特征层中的跨阶段卷积块的数量相同。当模型中包括特征金字塔模块时，特征金字塔模块中包括的空间金字塔池化块的数量与检测头中的深度可分离卷积块的数量相同。
下面结合图3至图11对本公开实施例提供的深度学习瑕疵检测模型进行举例说明，但本公开并不限定于此。
图3是本公开另一示例性实施例的深度学习瑕疵检测模型的示意图。如图3所示，图2中的中间特征层112包括第一跨阶段卷积块、第二跨阶段卷积块和第三跨阶段卷积块。可选的，检测头12包括第一深度可分离卷积块、第二深度可分离卷积块和第三深度可分离卷积块。输入块111将接收到的待处理图像输入第一跨阶段卷积块，第一跨阶段卷积块对该第一待处理图像进行处理后，获得该待处理图像的多个第一丢弃特征图，并将多个第一丢弃特征图分别输入至第一深度可分离卷积块进行处理。第一深度可分离卷积块分别对多个第一丢弃特征图进行处理，基于每个第一丢弃特征图预测出该待处理图像中的第一瑕疵的多个第一类别预测结果和多个第一位置预测结果。
需要说明的是，本公开实施例中backbone的网络结构并不限于所例举的跨阶段卷积块，例如还可以替换为RepVGG、EfficientNet、ConvNeXt等中的至少一种。
图3实施例中，第一跨阶段卷积块还将多个第一丢弃特征图分别输入至第二跨阶段卷积块。该第二跨阶段卷积块对每个第一丢弃特征图分别进行处理，输出多个第二丢弃特征图。
第二跨阶段卷积块将每个第二丢弃特征图分别输入至第二深度可分离卷积块。第二深度可分离卷积块接收多个第二丢弃特征图，分别对每个第二丢弃特征图进行处理，预测出该待处理图像中的第二瑕疵的多个第二类别预测结果和多个第二位置预测结果。
第二跨阶段卷积块还将多个第二丢弃特征图分别输入至第三跨阶段卷积块。该第三跨阶段卷积块对每个第二丢弃特征图分别进行处理，输出多个第三丢弃特征图。
第三跨阶段卷积块将每个第三丢弃特征图分别输入至第三深度可分离卷积块。第三深度可分离卷积块接收多个第三丢弃特征图，分别对每个第三丢弃特征图进行处理，预测出该待处理图像中的第三瑕疵的多个第三类别预测结果和多个第三位置预测结果。
继续参考图3，检测头12还用于根据每个深度可分离卷积块预测的多个类别预测结果和多个位置预测结果进行贝叶斯预测/分析，以获得每个瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性。
例如，针对第一深度可分离卷积块预测输出的第一瑕疵的多个第一类别预测结果和多个第一位置预测结果进行贝叶斯预测处理(例如求均值和方差)，获得第一瑕疵的第一类别检测结果、第一位置检测结果、第一类别不确定性和第一位置不确定性。
再例如，针对第二深度可分离卷积块预测输出的第二瑕疵的多个第二类别预测结果和多个第二位置预测结果进行贝叶斯预测处理，获得第二瑕疵的第二类别检测结果、第二位置检测结果、第二类别不确定性和第二位置不确定性。
再例如，针对第三深度可分离卷积块预测输出的第三瑕疵的多个第三类别预测结果和多个第三位置预测结果进行贝叶斯预测处理，获得第三瑕疵的第三类别检测结果、第三位置检测结果、第三类别不确定性和第三位置不确定性。
本公开实施例中，获得的待处理图像中的瑕疵的多个类别预测结果包括第一瑕疵的多个第一类别预测结果、第二瑕疵的多个第二类别预测结果和第三瑕疵的多个第三类别预测结果。获得的待处理图像中的瑕疵的多个位置预测结果包括第一瑕疵的多个第一位置预测结果、第二瑕疵的多个第二位置预测结果和第三瑕疵的多个第三位置预测结果。获得的待处理图像中的瑕疵的类别检测结果包括第一瑕疵的第一类别检测结果、第二瑕疵的第二类别检测结果和第三次的第三类别检测结果。获得的待处理图像中的瑕疵的位置检测结果包括第一瑕疵的第一位置检测结果、第二瑕疵的第二位置检测结果和第三瑕疵的第三位置检测结果。获得的待处理图像中的瑕疵的类别不确定性包括第一瑕疵的第一类别不确定性、第二瑕疵的第二类别不确定性和第三瑕疵的第三类别不确定性。获得的待处理图像中的瑕疵的位置不确定性包括第一瑕疵的第一位置不确定性、第二瑕疵的第二位置不确定性和第三瑕疵的第三位置不确定性。
图3中，以待处理图像为通过拍摄处于生产流水线上的口罩布图像为例，该类别检测结果即分类结果可以是胶纸/贴纸、污渍、印刷错误、折痕(口罩织物褶皱)等中的任一种。可以理解的是，这四种分类结果(例如C＝4)仅用于举例说明，并不限定一次，可以根据实际场景进行设置。位置检测结果即该瑕疵在该待处理图像中的定位/位置信息。
如图4所示，丢弃层1121包括第一丢弃层和第二丢弃层。第一丢弃层位于第一跨阶段卷积块中，第二丢弃层位于第二跨阶段卷积块中。
第一跨阶段卷积块中的第一跨阶段输入层的输入端连接图3实施例中的输入块111的 输出端。第一跨阶段输入层的输出端分别连接第一卷积层和第二卷积层的输入端。第一卷积层的输出端连接第一中间层的输入端。第一中间层的输出端连接第一融合层的第一输入端。第二卷积层的输出端连接第一融合层的第二输入端。第一融合层的输出端连接第三卷积层的输入端。第三卷积层的输入端连接第一丢弃层的输入端。第一丢弃层的输出端分别连接第二跨阶段卷积块中的第二跨阶段输入层的输入端和第一深度可分离卷积块的输入端。
第二跨阶段卷积块中的第二跨阶段输入层的输出端分别连接第四卷积层和第五卷积层的输入端。第四卷积层的输出端连接第二中间层的输入端。第二中间层的输出端连接第二融合层的第一输入端。第五卷积层的输出端连接第二融合层的第二输入端。第二融合层的输出端连接第六卷积层的输入端。第六卷积层的输入端连接第二丢弃层的输入端。第二丢弃层的输出端连接第二深度可分离卷积块的输入端。
图4实施例中，第一卷积层、第二卷积层、第三卷积层、第四卷积层、第五卷积层和第六卷积层的卷积核大小可以均为1×1。
如图5所示，第一中间层中包括第一深度可分离卷积层，第二中间层中包括第二深度可分离卷积层。
第一中间层的第一中间输入层的输入端连接图4实施例中的第一卷积层的输出端。第一中间输入层的输出端分别连接第一深度可分离卷积层的输入端和第一累加层的输入端。第一累加层的输出端连接图4实施例中的第一融合层的输入端。
第二中间层的第二中间输入层的输入端连接图4实施例中的第四卷积层的输出端。第二中间输入层的输出端分别连接第二深度可分离卷积层的输入端和第二累加层的输入端。第二累加层的输出端连接图4实施例中的第二融合层的输入端。
如图6所示，第一深度可分离卷积层包括第一深度分离输入层、第一深度卷积层、第一点卷积层和第一激活函数层。第二深度可分离卷积层包括第二深度分离输入层、第二深度卷积层、第二点卷积层和第二激活函数层。
第一深度分离输入层的输入端连接图5实施例中的第一中间输入层的输出端。第一深度分离输入层的输出端分别连接第一深度卷积层的输入端和第一激活函数层的输入端。第一深度卷积层的输出端连接第一点卷积层的输入端。第一点卷积层的输出端连接第一激活函数层。
第二深度分离输入层的输入端连接图5实施例中的第二中间输入层的输出端。第二深度分离输入层的输出端分别连接第二深度卷积层的输入端和第二激活函数层的输入端。第 二深度卷积层的输出端连接第二点卷积层的输入端。第二点卷积层的输出端连接第二激活函数层。
图7实施例中，丢弃层还包括第三丢弃层。如图7所示，第三跨阶段卷积块包括第三跨阶段输入层、第七卷积层、第三中间层、第八卷积层、第三融合层、第九卷积层和所述第三丢弃层。第三跨阶段输入层的输入端连接第二跨阶段卷积块的输出端，例如连接第二丢弃层的输出端，将第二丢弃层输出的多个第二丢弃特征图分别作为第三跨阶段输入特征图。
所述第七卷积层用于将所述第三跨阶段输入层输入的第三跨阶段输入特征图进行卷积处理，向所述第三中间层输入第五卷积特征图。所述第三中间层用于对所述第五卷积特征图进行处理，向所述第三融合层输入第五待融合特征图。所述第八卷积层用于对所述第三跨阶段输入特征图进行卷积处理，向所述第三融合层输入第六待融合特征图。所述第三融合层用于对所述第五待融合特征图和所述第六待融合特征图进行融合处理，向所述第九卷积层输入第三融合特征图。所述第九卷积层用于对所述第三融合特征图进行卷积处理，向所述第三丢弃层输入第六卷积特征图。所述第三丢弃层用于对所述第六卷积特征图中的特征进行随机丢弃，获得多个第三丢弃特征图，所述待处理图像经过所述特征提取模块处理后获得经过随机丢弃处理后的多个特征包括多个所述第三丢弃特征图。
继续参考图7，第三中间层包括第三中间输入层、第三深度可分离卷积层和第三累加层。所述第三中间输入层用于接收所述第七卷积层输出的第五卷积特征图，并将所述第五卷积特征图分别输入至所述第三深度可分离卷积层和所述第三累加层。所述第三深度可分离卷积层用于对所述第五卷积特征图进行深度可分离卷积处理，向所述第三累加层输入第三深度分离卷积特征图。所述第三累加层对所述第五卷积特征图和所述第三深度分离卷积特征图进行累加处理，获得所述第五待融合特征图。
继续参考图7，所述第三深度可分离卷积层包括第三深度分离输入层、第三深度卷积层、第三点卷积层和第三激活函数层。所述第三深度分离输入层用于接收所述第三中间输入层输出的第五卷积特征图，并将所述第五卷积特征图分别输入至所述第三深度卷积层和所述第三激活函数层。所述第三深度卷积层用于对所述第五卷积特征图进行深度卷积处理，并向所述第三点卷积层输入第三深度卷积特征图。所述第三点卷积层用于对所述第三深度卷积特征图进行点卷积处理，并向所述第三激活函数层输入第三点卷积特征图。所述第三激活函数层用于对所述第三点卷积特征图和所述第五卷积特征图进行处理，获得所述第三深度分离卷积特征图。
本公开实施例中，激活函数层(例如包括第一至第三激活函数层)使用的激活函数可以是任意适当的激活函数。在下面的实施例中，以采用SiLU(Swish激活函数)进行举例说明，但本公开并不限定于此。SiLU函数在整个定义域内都是可微的，因此在反向传播过程中的梯度计算更加稳定。SiLU函数在深层模型上，具有无上界有下界、平滑、非单调的特性，且梯度更大、更稳定。
图7实施例中，第七卷积层、第八卷积层和第九卷积层的卷积核大小可以均为1×1。
本公开实施例的跨阶段卷积块中引入了深度可分离卷积层，该深度可分离卷积层通过将空间维度和通道(深度)维度的相关性进行拆分，减少了卷积计算所需的参数数量，并提升了卷积核参数的使用效率。深度可分离卷积层的卷积计算过程分为两部分：深度卷积，对每个通道分别进行空间卷积，即对每个输入通道使用单一的卷积核进行卷积操作，然后将这些输出进行拼接；点卷积(Pointwise Convolution)，使用单位卷积核(即1x1的卷积核)进行通道卷积，以整合各个通道的信息，并得到最终的特征图。深度可分离卷积的优势在于它减少了计算复杂度和模型参数数量，同时保持了模型的表达能力。深度可分离卷积是一种轻量级的卷积方法，通过减少参数和计算量，提高了模型的效率和性能。
图8实施例与图3实施例的区别在于，特征提取模块11还可以包括第四跨阶段卷积块，检测头12还可以包括第四深度可分离卷积块。第四跨阶段卷积块的结构可参考上述图4至图7。第四跨阶段卷积块与第三跨阶段卷积块和检测头12中的第四深度可分离卷积块连接，用于分别接收多个第三丢弃特征图，并对多个第三丢弃特征图进行处理以生成多个第四丢弃特征图。第四跨阶段卷积块可以包括第四丢弃层，以输出多个第四丢弃特征图至第四深度可分离卷积块。第四深度可分离卷积块可以预测输出第四瑕疵的多个第四类别预测结果和多个第四位置预测结果。检测头12可以根据多个第四类别预测结果获得第四瑕疵的第四类别检测结果和第四类别不确定性，根据多个第四位置预测结果获得第四瑕疵的第四位置检测结果和第四位置不确定性。
图9实施例与图3实施例的区别之一在于，深度学习瑕疵检测模型1还包括位于特征提取模块11和检测头12之间的特征金字塔模块13。
图9实施例通过在特征提取模块11和检测头12之间引入特征金字塔模块13，以解决瑕疵检测中的多尺度问题。不同大小的瑕疵在图像中呈现出不同的特征，而特征金字塔可以在不同尺度下提取和融合这些特征。通过网络连接改变，特征金字塔在基本不增加原有模型计算量的情况下，大幅度提升了小瑕疵检测的性能。具体的，特征金字塔通过上采样和下采样操作，将不同尺寸的特征层进行融合。这样，模型可以同时获取到高层次的语 义信息和低层次的细节信息，从而提取出更好的特征。这种融合后的特征不仅有助于提升模型的检测性能，还可以使模型更好地适应不同大小的瑕疵检测任务。首先，特征提取模块11中的中间特征层会生成不同尺度的特征图。这些特征图在金字塔结构中，从底层到高层，分别包含了越来越抽象的信息，对应着不同大小的瑕疵。底层特征图通常具有更高的分辨率和更多的细节信息，适合用于检测小瑕疵；而高层特征图则具有更低的分辨率和更抽象的语义信息，适合用于检测大瑕疵。接下来，特征金字塔通过一系列的上采样和下采样操作，将这些不同尺度的特征图进行融合。具体来说，较低层的特征图会经过上采样操作，使其分辨率与较高层的特征图相匹配，然后通过拼接或相加的方式与较高层的特征图进行融合。这样，融合后的特征图既包含了低层的细节信息，又包含了高层的语义信息，有助于提升模型对不同大小瑕疵的检测性能。通过连接不同尺度的特征图，可以生成多个不同尺度的预测结果。这些预测结果分别对应着不同大小的瑕疵，从而实现了多尺度瑕疵检测。这种机制使得模型能够同时关注到图像中的全局信息和局部细节，从而提高了瑕疵检测的准确性和鲁棒性。
例如，如图9所示，特征金字塔模块13包括第一空间金字塔池化块、第二空间金字塔池化块和第三空间金字塔池化块。本公开实施例中，跨阶段卷积块可以设置实现特征图的降采样。图9实施例中，实现3个不同尺度的特征图输入到特征金字塔模块13，作为YOLO多尺度检测头12的输入。因为跨阶段卷积块可以通过定义参数来使能降采样的功能，所以可以使用3个以上跨阶段卷积，实现3个不同尺度的特征图输入给特征金字塔模块13。
第一空间金字塔池化块的第一输入端与第一跨阶段卷积块的输出端连接，例如与第一丢弃层的输出端连接，以接收第一丢弃层输出的多个第一丢弃特征图。第一空间金字塔池化块的第二输入端与第二空间金字塔池化块的输出端连接，以接收第二空间金字塔池化块输出的多个第二空间池化特征图。第一空间金字塔池化块分别对多个第一丢弃特征图中的每一个第一丢弃特征图与多个第二空间池化特征图中对应的一个第二空间池化特征图进行处理，通过其输出端向检测头12中的第一深度可分离卷积块输出多个第一空间池化特征图。检测头12中的第一深度可分离卷积块分别对每个第一空间池化特征图进行处理，预测出第一瑕疵的多个第一类别预测结果和多个第一位置预测结果。检测头12对第一瑕疵的多个第一类别预测结果和多个第一位置预测结果进行贝叶斯预测，获得第一瑕疵的第一类别检测结果、第一位置检测结果、第一类别不确定性和第一位置不确定性。
第二空间金字塔池化块的第一输入端与第二跨阶段卷积块的输出端连接，例如与第二 丢弃层的输出端连接，以接收第二丢弃层输出的多个第二丢弃特征图。第二空间金字塔池化块的第二输入端与第三空间金字塔池化块的输出端连接，以接收第三空间金字塔池化块输出的多个第三空间池化特征图。第二空间金字塔池化块分别对多个第二丢弃特征图中的每一个第二丢弃特征图与多个第三空间池化特征图中对应的一个第三空间池化特征图进行处理，通过其输出端向检测头12中的第二深度可分离卷积块输出多个第二空间池化特征图。检测头12中的第二深度可分离卷积块分别对每个第二空间池化特征图进行处理，预测出第二瑕疵的多个第二类别预测结果和多个第二位置预测结果。检测头12对第二瑕疵的多个第二类别预测结果和多个第二位置预测结果进行贝叶斯预测，获得第二瑕疵的第二类别检测结果、第二位置检测结果、第二类别不确定性和第二位置不确定性。
第三空间金字塔池化块的输入端与第三跨阶段卷积块的输出端连接，例如与第三丢弃层的输出端连接，以接收第三丢弃层输出的多个第三丢弃特征图。第三空间金字塔池化块分别对多个第三丢弃特征图中的每一个第三丢弃特征图进行处理，通过其输出端向检测头12中的第三深度可分离卷积块输出多个第三空间池化特征图。检测头12中的第三深度可分离卷积块分别对每个第三空间池化特征图进行处理，预测出第三瑕疵的多个第三类别预测结果和多个第三位置预测结果。检测头12对第三瑕疵的多个第三类别预测结果和多个第三位置预测结果进行贝叶斯预测，获得第三瑕疵的第三类别检测结果、第三位置检测结果、第三类别不确定性和第三位置不确定性。图9实施例中，待处理图像中的瑕疵还包括第三瑕疵，获得的待处理图像中的瑕疵的多个类别预测结果还包括第三瑕疵的第三类别预测结果，多个位置预测结果还包括第三瑕疵的第三位置预测结果。获得的待处理图像中的瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性分别还包括第三瑕疵的第三类别检测结果、第三位置检测结果、第三类别不确定性和第三位置不确定性。
本公开实施例中，特征金字塔模块13可以在原本3个尺度的特征图下，融合不同大小感受野的卷积特征，实现多尺度特征融合，增强特征的鲁棒性。
图9实施例与图3实施例的区别之二在于，深度学习瑕疵检测模型1还可以包括位于特征提取模块11之前的背景移除模块。背景移除模块用于移除待处理图像中的背景信息，并将移除背景信息之后的待处理图像输入至特征提取模块11。这里以未移除背景信息的待处理图像为口罩布图像进行举例说明，将移除背景信息的口罩布图像称之为待处理图像。
图10是本公开一示例性实施例的空间金字塔池化块的示意图。如图10所示，特征金字塔模块13中的每一个空间金字塔池化块(例如图9中的第一至第三空间金字塔池化块)可以具有相同的结构。空间金字塔池化块可以包括输入层、与输入层连接的第一个卷积层、 与第一个卷积层连接的三个并列的最大池化层、分别与三个最大池化层连接的融合层、以及与融合层连接的第二个卷积层。
空间金字塔池化块中的各个最大池化层可以用于降低特征图的参数量，提升计算速度，增加感受野，是一种降采样操作。通过池化，模型可以更关注全局特征而非局部出现的位置，这种降维的过程可以保留一些重要的特征信息，提升容错能力，并且还能一定程度上起到防止过拟合的作用。
图10实施例中，第一个卷积层和第二个卷积层的卷积核大小可以均为1×1。
继续参考图10，三个并联的最大池化层的卷积核大小可以分别为3×3，5×5，7×7。
需要说明的是，本公开实施例中的特征金字塔模块不限于采用图9和图10中例举的空间金字塔池化块，可以有其他结构可以替代。图10实施例中，对于同一个特征图，使用了不同大小的卷积核并行处理，然后获得直接融合处理后的特征图。其他的结构例如如图11所示，可以改为不同大小卷积核串行处理输入值特征金字塔模块的特征图，例如采用三个卷积核大小分别为3×3，5×5，7×7的卷积层串行，然后用shortcut(直连或捷径)方式，跳接不同卷积核输出的特征图实现融合。
图9实施例与图3实施例的区别之三在于，输入块111可以是注意力块111’。注意力块111’是待处理图像(b1,c1,w1,h1分别表示待处理图像的批次大小、通道数、宽度、高度)进入特征提取模块11前，对待处理图像进行切片操作，具体操作是在一张待处理图像中每隔一个像素拿到一个值，类似邻近下采样，这样就拿到了四张图像，四张图像互补，没有信息丢失，这样将w1/h1信息就集中到了通道空间，输入通道扩充了4倍(获得了(b1,4*c1,w1,h1)的输入)，即拼接起来的图像相对于原先的c个通道变成了4*c1个通道，最后将得到的新图像再经过卷积操作(例如卷积核大小为1x1)，最终得到了没有信息丢失情况下的二倍下采样特征图，输入至第一跨阶段卷积块。经过这样的处理，注意力块111’得到了没有信息丢失情况下的二倍下采样特征图，可以减少参数量、网络层数、梯度和计算次数，同时提高模型的精度和召回率，这使得模型在保持高效的同时，能够更准确地检测和识别图像中的瑕疵，尤其是那些尺寸较小、特征不明显的瑕疵。
图12实施例中使用了空间注意力机制来设计注意力块。可以理解的是，本公开实施例中的注意力块不限于采用图12所示的结构，还可以通过Squeeze-Excitation(压缩-激励，显式的建模特征通道之间的相互依赖关系，通过学习的方式获取每个通道的重要程度，然后依照这个重要程度来对各个通道上的特征进行加权，突出重要特征，抑制不重要的特征)操作来实现通道注意力块。此外还可以通过动态卷积实现空间注意力块。
可以理解的是，本公开实施例提供的模型并不限于上述例举的结构。本公开实施例提供的在中间特征层中加入丢弃层，在模型的训练过程中增加贝叶斯损失函数(例如下述第二损失函数)，本公开实施例提供的方法可以适应各种网络结构，在保持与各种网络架构 兼容的同时实现不确定性估计的集成，在应用中提供灵活性，并进一步提高性能。
本公开实施例中，通过卷积层的使用，可以使模型能够适应不同的图像分辨率和瑕疵/缺陷检测场景。卷积操作允许网络关注图像中的局部特征，而不是整个图像。由于卷积层的这种性质，模型能够处理不同尺寸的输入图像，而不必对模型结构进行大的修改。这使得模型具有很高的灵活性，可以适应各种图像分辨率。在缺陷检测领域，这意味着无论输入的图像是高清还是低分辨率，或者图像中缺陷的大小和位置如何变化，使用卷积层的模型都能够有效地进行缺陷检测。因此，这种模型可以轻松地适应不同的缺陷检测场景，提高了模型的通用性和实用性。
在示例性实施例中，本公开实施例提供的方法还包括：获得目标训练数据集，所述目标训练数据集中包括目标样本图像及所述目标样本图像中的目标样本瑕疵的标注类别和标注位置；将所述目标样本图像输入至所述深度学习瑕疵检测模型，获得所述目标样本瑕疵的多个类别预测结果和多个位置预测结果；根据所述目标样本瑕疵的多个类别预测结果和多个位置预测结果获得所述目标样本瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性；根据所述目标样本瑕疵的标注类别及类别检测结果、标注位置及位置检测结果、以及类别不确定性和位置不确定性构建损失函数以训练所述深度学习瑕疵检测模型。
本公开实施例中，可以获得用于瑕疵检测的数据集，按照预定比例例如按9：1的比例划分为目标训练数据集和目标测试数据集。在模型训练时，前向传播的过程中进行dropout操作，在dropout的作用下通过随机关闭神经元能够得到多个不同网络结构，从而对同一个目标样本图像进行多次预测，获得多个类别预测结果和多个位置预测结果。计算多个类别预测结果的均值和多个位置预测结果的均值分别表示目标样本瑕疵的类别检测结果和位置检测结果，计算多个类别预测结果的均值和多个位置预测结果的方差分别表示目标样本瑕疵的类别不确定性和位置不确定性。
例如，可以设置模型的训练参数：批量大小(batch size)设置为64，迭代次数(epoch)设置为100，初始学习速率设置为0.001，衰减系数设置为0.005，dropout概率为0.3。经过dropout后每层的神经元数量变为原来的大约一半。
在示例性实施例中，所述损失函数包括第一损失函数和第二损失函数；当所述目标样本瑕疵的标注类别和类别检测结果的差异越大，标注位置和位置检测结果的差异越大时，所述第一损失函数的取值越大；当所述目标样本瑕疵的类别不确定性和所述位置不确定性越大时，所述第二损失函数的取值越大。
本公开实施例引入贝叶斯方法，在模型中新增dropout创造随机不确定性来实现贝叶斯后验分布。同时优化模型的检测头，新增锚框(即位置预测结果)和类别(即类别预测结果)的均值和方差来表示其分布情况，例如，该第二损失函数可以为负对数似然损失函数，以优化不确定性估计，不仅推动模型向准确预测进化，也鼓励模型对其预测的不确定性进行准确估计，从而提高了模型的检测精度和鲁棒性。
例如，第一损失函数L1如下：
MSE是表示边界框回归计算，yi是真实数据中第i个边界框坐标(即第i个目标样本瑕疵的标注位置)，是模型预测的第i个边界框坐标(即第i个目标样本瑕疵的位置检测结果)，N是边界框的数量(即目标样本候选瑕疵的数量)，N为大于或等于1的正整数，i为大于或等于1且小于或等于N的正整数。CE是表示多元交叉熵计算，表示预测的类别检测结果中类别c的类别置信度(即属于类别c的概率)。sc真实数据中类别c的置信度(即标注类别中类别c的类别置信度)。C表示类别数量，C为大于或等于1的正整数，c为大于或等于1且小于或等于C的正整数。BCE是表示二元交叉熵计算，opre表示预测的目标置信度，o表示真实数据中目标的置信度，即目标样本瑕疵中标注的目标的置信度。
再例如，第二损失函数L2如下：
μi是预测的第i个目标样本瑕疵的位置检测结果，例如是模型预测的第i个边界框坐标的均值。yi是第i个目标样本瑕疵的标注位置，例如是真实数据中第i个边界框坐标。是预测的第i个目标样本瑕疵的位置不确定性，例如是模型预测的第i个边界框坐标的方差。N是目标样本瑕疵的数量，N为大于或等于1的正整数，i为大于或等于1且小于或等于N的正整数。μc表示预测的类别c的类别置信度均值，即类别检测结果。sc是真实数据中类别c的类别置信度。是预测的类别c的类别置信度方差，即类别不确定性。C表示类别数量，C为大于或等于1的正整数，c为大于或等于1且小于或等于C的正整数。
在示例性实施例中，获得目标训练数据集，包括：获取初始样本图像及所述初始样本图像中的初始样本瑕疵的标注类别和标注位置；对所述初始样本图像进行随机裁剪、旋转、缩放、光照变化模拟、瑕疵合成中的任意一项或多项处理，生成扩增样本图像及所述扩展增益图像中的扩展样本瑕疵的标注类别和标注位置。其中，所述目标样本图像包括所述初始样本图像和所述扩增样本图像。
在示例性实施例中，对所述初始样本图像进行瑕疵合成处理，生成扩增样本图像，包括：提取所述初始样本图像中的初始样本瑕疵；将所述初始样本瑕疵与扩增背景图像合成，生成所述扩增样本图像；和/或，通过深度学习生成模型学习所述初始样本图像，生成所述扩增样本图像。
数据集质量会影响模型的准确性和鲁棒性。例如还是以检测口罩布图像中的瑕疵 为例，为了训练一个高效准确的模型，需要大量高质量的具有瑕疵的口罩布图像以形成数据集。如果数据集中的图像质量低或样本量不足，模型的准确性和鲁棒性可能会受到影响。
实际情况中，存在瑕疵的口罩布图像数量比不存在瑕疵的口罩布图像的数量要少，即训练数据集中的正负样本数量不平衡。本公开通过实施一种多阶段数据增强策略解决数据集质量问题。首先，采集大量原始的口罩布图像，并通过专家标注生成初始训练数据(包括初始样本图像)。接着，利用图像处理技术如随机裁剪、旋转、缩放和光照变化模拟，以及合成瑕疵生成方法等中的一种或多种扩充数据集，以获得扩增样本图像。
本公开实施例中，可以通过采用随机裁剪、旋转、缩放、光照变化模拟(例如亮度调整)、瑕疵合成、添加高斯噪声、随机遮挡等中的任意一项或多项的数据增强方式来增加存在瑕疵的口罩布图像。光照变化模拟是随机调整原始的存在瑕疵的口罩布图像的饱和度、亮度、对比度等中的一种或者多种，可以模拟拍摄图像容易受到光照和天气变化的影响；随机遮挡是将存在瑕疵的口罩布图像的随机区域用黑色遮挡，可以模拟在拍摄时容易被其它物品遮挡的情况；高斯噪声是在原始的存在瑕疵的口罩布图像中添加概率密度函数服从高斯分布的噪声，可以更好地模拟未知的真实噪声。
相关技术中的模型难以在检测速度和准确性之间达到较好的平衡。这可能导致在实际应用中对小微瑕疵点的检测性能不令人满意，因为小微瑕疵点需要处理高分辨率图像，从而加剧算力负担，降低检测速度。本公开实施例通过数据增强将多张初始样本图像随机缩放裁剪进行拼接，大量增加了训练数据集中的小微瑕疵点，提高了小微瑕疵点检测的效果。
本公开实施例还可以通过引入了AIGC(stable diffusion)根据真实的瑕疵样本生成新的瑕疵样本，增加数据多样性。可以采用如下两种合成瑕疵方法中的任意一种或者两种：第一种是将瑕疵前景提取出来，替换其他布料的背景。第二种是通过stable diffusion学习真瑕疵图像，然后生成假的瑕疵图像。将AIGC引入瑕疵样本生成领域，可以根据真实的瑕疵样本生成新的瑕疵样本，可以丰富数据集，帮助训练更准确的模型。生成的瑕疵样本可以用于模拟各种可能的瑕疵情况，从而在实际应用中更全面地测试瑕疵检测系统的性能。
在示例性实施例中，获得目标训练数据集，包括：获取候选样本图像及所述候选样本图像中的候选样本瑕疵的标注类别和标注位置；通过所述特征提取模块获得所述候选样本图像的输出特征；基于各个候选样本图像的输出特征，获得各个候选样本图像之间的特征距离；选择标注类别相同的候选样本图像中特征距离大于第一距离阈值的候选样本图像，和/或，标注类别不同的候选样本图像中特征距离小于第二距离阈值的候选样本图像，作为所述目标训练数据集中的目标样本图像。
本公开实施例中可以使用欧几里得距离来作为特征距离，但本公开并不限定于此。
本公开实施例还可以应用一种自适应的样本选择机制，优先选择那些对模型性能提升最有效的样本进行训练。例如，在训练过程中，提取多个样本的骨干网络输出的特征，以捕捉到不同样本之间的相似性和差异性。然后计算各个样本的特征距离。
例如，可以通过如下公式计算第i个样本(xi)和第j个样本(xj)之间的特征距离dij：
Fk(xi)是样本xi在第k维的特征值，Fk(xj)是样本xj在第k维的特征值，D是样本的特征的维度，D为大于或等于1的正整数，k为大于或等于1且小于或等于D的正整数。i和j均为大于或等于1的正整数，且i不等于j。
计算样本的特征距离有助于理解样本在特征空间中的分布。例如，可以选择同类型样本(同种类型瑕疵对应的图像，就是label(标签)相同的不同样本，例如均被标注为“胶纸”的样本)中距离最远的，不同类型样本(不同种类型瑕疵对应的图像，例如一个样本被标注为“污渍”，另一个样本被标注为“折痕”)中距离最近的样本参与反向传播，相当于使用最容易出错(最难判断)的样本来优化模型的权重，以提高模型的准确性和鲁棒性。
本公开实施例中，自适应的样本选择机制的样本选择标准是先统计特征距离，然后根据特征距离的统计值划分第一距离阈值和第二距离阈值，例如选择第一距离阈值(也称类内距离)大于0.3，第二距离阈值(也称类间距离)小于0.2。自适应的样本选择机制这个训练策略是在常规训练后新增的，用于解决hard sample(困难样本)问题进行的二次训练。经过二次训练后，令类内样本特征分布更加集中，类间样本特征分布更加离散，从而增强分类准确性。
基于特征距离的计算结果，选择同类型样本中距离最远的样本，意味着这些样本在特征空间中的差异最大，即使它们属于同一类别，模型也很难将它们正确区分。通过让模型学习这些难以区分的样本，可以促使模型更加关注类别内部的细节差异，从而提高模型的准确性。同时，选择不同类型样本中距离最近的样本，意味着这些样本在特征空间中的相似度较高，但实际上它们属于不同的类别。这些样本是模型最容易出错的样本，因为它们具有相似的特征但标签不同。通过让模型学习这些样本，可以强化模型对不同类别之间的区分能力，从而提高模型的鲁棒性。使用这种策略进行反向传播，相当于使用最容易出错(最难判断)的样本来优化模型的权重。这样，模型在训练过程中会更加关注这些具有挑战性的样本，从而提高其在面对复杂和多变的环境时的泛化能力和鲁棒性。
在示例性实施例中，在所述深度学习瑕疵检测模型的训练过程中：提取所述深度学习瑕疵检测模型中的每个卷积层的权重张量；获得所述权重张量中每个通道的欧几里得范数；对每个卷积层根据其通道的欧几里得范数进行排序，根据剪枝阈值或剪枝比例，对相应的通道进行剪枝。
通道剪枝是一种有效的模型压缩和加速技术，特别适用于轻量化模型设计的需求。在模型中进行通道剪枝，可以显著减少模型的参数数量和计算复杂度，同时保持模型的性能。
本公开实施例，基于L2-norm(欧几里得范数)度量卷积通道对网络输出的贡献，并根据贡献程度进行通道剪枝。首先，基于L2-norm度量卷积通道的贡献：L2-norm(欧几里得范数)可以用来评估卷积通道的重要性或贡献程度。对于模型中的每个卷积层，可以计算每个卷积通道权重的L2-norm值。这个值反映了该通道在网络中的影响力和贡献。具体的，对于模型中的每个卷积层，提取其权重张量(weight tensor)。计算权重张量中每个通道的L2-norm值。例如对权重张量进行平方、求和和开方等操作。将计算得到的L2-norm值保存下来，用于后续的通道剪枝决策。然后，根据贡献程度进行通道剪枝：根据上一步计算得到的L2-norm值，可以评估每个卷积通道的重要性，并根据一定的策略进行通道剪枝。通道剪枝的目标是移除那些对网络输出贡献较小的通道，从而减少模型的参数数量和计算复杂度。具体的，设定一个剪枝阈值或剪枝比例。剪枝阈值或剪枝比例可以根据具体需求和实验来确定。对于每个卷积层，根据其通道的L2-norm值进行排序。根据设定的剪枝阈值或剪枝比例，选择贡献较小的通道进行剪枝。例如，将对应通道的权重设置为零或直接从模型中移除这些通道。在剪枝后，对模型进行重新训练和微调，以恢复或保持其性能。欧几里得范数反映了该通道对输出的响应权重，优先裁剪响应权重较低的通道。
在一些实施例中，针对检测速度与准确性的平衡，对深度学习瑕疵检测模型进行轻量化设计，通过通道剪枝降低特征通道数量，使用深度可分离卷积替换原始卷积层，减少模型的参数和计算量，实现更快的检测，同时保持高准确性。
在示例性实施例中，所述目标样本图像包括具有不同材料、纹理、颜色、印花图案的口罩图像(或口罩布图像)。
为了增加模型的泛化能力，本公开实施例设计了一个混合数据集，以口罩布图像为例，该混合数据集中包含了多种类型的口罩样本，例如具有采用不同的口罩材料、颜色、纹理和印花图案等的口罩图像。
对于某些特定类型的口罩材料，例如带有印花图案的口罩布，由于其图案的图案类型可能是千变万化的，训练样本不可能包含所有图案的口罩样本图像，导致模型可能无法实现高检测准确性。本公开实施例中可以利用迁移学习和领域自适应技术，包括：特征空间对齐，fine-tuning(微调)，度量学习等中的一种或多种，使模型能够适应新的材料、颜色、纹理、印花图案等类型。
在一些实施例中，模型定期在新收集的数据上进行再训练，以维持其泛化性能。
在示例性实施例中，获得待处理图像，包括：从图像采集组件中的图像拍摄设备获得所述待处理图像，所述图像采集组件用于获取处于生产过程中的待检测对象的所述待处理图像。其中，所述方法还包括：获得所述待检测对象的当前生产速度和所述 深度学习瑕疵检测模型对所述待处理图像的当前处理速度；根据所述当前生产速度和所述当前处理速度调整用于所述深度学习瑕疵检测模型处理所述待处理图像的并行线程或并行进程数量。
本公开实施例中引入实时检测反馈机制，根据当前生产线速度(或当前生产速度)改变模型推理并行线程数量来动态调整模型推理时间(即模型的当前处理速度)，确保在保持高准确性的同时达到所需的检测速度。
例如，模型经过剪枝后使用TensorRT进行部署，经过封装的TensorRT架构支持多进程，多线程并发。同时考虑到硬件长时间高负载会影响寿命，所以设计了动态调整的机制。本公开实施例中，可以使用线程池来管理推理任务的并发执行。线程池可以确保线程资源的有效利用，避免频繁创建和销毁线程带来的开销。设计一个任务调度器，负责将推理任务分配给空闲的线程。可以使用负载均衡策略，确保各个线程的负载相对均衡，避免某些线程过载而其他线程空闲的情况。
TensorRT原生支持多线程推理，通过IExecutionContext的execute_async_v2即可实现。除此之外，本公开实施例还利用多核CPU(Central Processing Unit，中央处理器)的优势，将TensorRT引擎序列化到磁盘，然后在多个进程中加载这个序列化的引擎，令每个进程都可以独立地执行推理任务，将CPU和GPU(Graphics Processing Unit，图形处理器)的资源发挥到极致。
本公开实施例可以使用传感器或其他监控设备实时获取当前生产速度；监控模型当前的推理速度，即处理每张图像所需的时间作为其当前处理速度。根据当前生产速度和模型的当前处理速度的对比，调整模型的推理并行线程数量。如果当前生产速度加快，需要增加并行线程数量以加快推理速度；反之，如果当前生产速度减慢，可以减少并行线程数量以节省资源。通过结合实时检测反馈机制、动态调整模型推理并行线程数量以及TensorRT的高效部署，可以确保生产线上的瑕疵检测任务既准确又高效。
在示例性实施例中，获得待处理图像，包括：从图像采集组件中的图像拍摄设备获得所述待处理图像；若所述待处理图像的灰度值和灰度分布不满足预设条件，则发送控制信号调整所述图像拍摄设备的曝光参数；从调整曝光参数后的所述图像拍摄设备获得新的待处理图像，直到获得满足所述预设条件的所述待处理图像。
为了提高模型对环境因素的鲁棒性，本公开实施例设计了自动曝光模块。在每次系统开启时，通过图像拍摄设备例如相机采集当前环境下的图像，分析其灰度分布情况，如果该图像的灰度值和灰度分布不满足预设条件，则调整曝光参数，再次采集图像进行分析，不断循环，直到采集到的图像的灰度值和灰度分布满足要求，才作为输入模型的待处理图像。该自动曝光模块可以实时监测光照条件，实时计算相机最新的曝光参数，抑制外界光线的影响。
例如，期望的预设条件是直方图均衡，灰度均值在130到140之间，灰度方差越 小越好。根据相机采集的图像的灰度直方图，灰度均值，灰度方差进行判断，是否达到预设条件，不满足则根据目标差值决定是增加曝光还是减少曝光。
在示例性实施例中，本公开实施例提供的方法还包括：若所述瑕疵的类别不确定性和/或位置不确定性大于不确定阈值，则发送针对所述瑕疵的提示信息。
例如，在线操作期间，整个系统是端到端的。它以获取的口罩布图像为输入，模型首先删除不相关的背景，获得待处理图像。然后，它执行特征提取，最后输出瑕疵的类别检测结果、位置检测结果和类别不确定性和位置不确定性。系统可以确定类别不确定性和/或位置不确定性大于不确定性阈值(其值可以根据实际需求进行设置)的瑕疵具有高不确定性的瑕疵，此时可以触发提示信息，以提示用户该瑕疵的预测具有高不确定性，进一步的，可由用户来确定如何对该瑕疵进行处理，例如是否由人工来进行瑕疵的位置和类别的标注。即本公开实施例还可以利用模型预测的类别不确定性和/或位置不确定性来协助用户识别最具信息量的数据以进行标记，从而提高注释过程的效率。
在示例性实施例中，本公开实施例提供的方法还包括：统计所述待处理图像中检测出的各个瑕疵的像素数量信息；根据各个瑕疵的像素数量信息确定各个瑕疵的瑕疵等级；根据各个瑕疵的瑕疵等级，执行相应的处理。
本公开实施例提供的瑕疵检测系统中，包括算法层和数据服务层/数据服务组件。算法层中部署有上述深度学习瑕疵检测模型，以用于对待处理图像进行处理，获得瑕疵数据，例如该待处理图像是否存在瑕疵，如果存在瑕疵，则包含该瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性等瑕疵信息。数据服务层负责分析和处理算法层提供的瑕疵数据，以确定瑕疵的严重程度。例如，通过统计瑕疵的像素数量信息(例如瑕疵的长边包含的像素个数)，将瑕疵分为四个等级：第一等级为轻微瑕疵，如长边像素小于20的瑕疵；第二等级为一般瑕疵，如长边像素大于或等于20且小于60的瑕疵；第三等级为较严重瑕疵，如长边像素大于或等于60且小于100的瑕疵；而第四等级则为严重瑕疵，如长边像素大于100的瑕疵。基于此分级系统，数据服务层可以为不同等级的瑕疵制定并执行相应的处理策略，例如检测到严重瑕疵时，直接触发报警器，提示用户(例如工人)马上处理。轻微瑕疵可以不触发报警器，只在交互界面进行提示，由用户自行决定是否处理。以确保瑕疵检测和处理的高效性和可靠性。
在示例性实施例中，根据各个瑕疵的瑕疵等级，执行相应的处理，包括：在所述待处理图像中，将处于不同瑕疵等级的瑕疵分别用具有不同粗细程度的边框标识。
本公开实施例中，数据服务层利用瑕疵区域边框的粗细来辅助判定瑕疵的分类等级，边框越粗表示瑕疵等级越高，边框越细表示瑕疵等级越低。例如数据服务层先统计每个瑕疵的像素数量信息，然后根据瑕疵像素数量信息将瑕疵分为四个等级，然后将第一等级的瑕疵在该待处理图像中采用最细的边框标识，将第四等级的瑕疵在该待 处理图像中采用最粗的边框标识，然后将标识好的瑕疵图像提供给用户，在交互界面显示，方便用户马上能够辨别出瑕疵严重程度，从而快速决策是否需要停机处理。这种视觉标识使得在后续的人工检查过程中，操作员能够迅速且直观地了解瑕疵的严重程度。
在示例性实施例中，本公开实施例提供的方法还包括：保存所述待处理图像以及所述待处理图像中检测出的瑕疵的类别检测结果、位置检测结果、类别不确定性、位置不确定性、瑕疵等级、瑕疵区域图像、瑕疵区域的图像特征、检出时间和检出序号；接收并根据输入的瑕疵区域的图像特征、检出时间和检出序号中的一项或多项，检索相应的瑕疵信息。
本公开实施例提供的瑕疵检测系统还可以包括瑕疵记录模块，当检测到瑕疵时，该瑕疵记录模块会自动记录瑕疵的等级(即瑕疵等级)、确切位置(即位置检测结果)以及瑕点(即瑕疵或瑕疵点)的详细区域图像。为了进一步提升效率，数据服务层还可以提供了瑕点信息检索功能，瑕疵/瑕疵区域的图像特征(例如可以是特征提取模块提取的特征)，检出时间，检出序号都会保存在数据库中，可以通过上述三个数据进行检索。通过这一功能，可以快速检索和回顾历史瑕疵记录，支持质量根据生产过程的持续改进。
当该瑕疵检测方法及系统应用于检测无纺口罩布的瑕疵检测时，也可称之为无纺口罩布瑕疵检测方法及系统。当该瑕疵检测方法及系统应用于卫生及个人护理产品材料的瑕疵检测时，也可以称之为卫生及个人护理产品材料智能缺陷检测方法及系统。
本公开实施例还提供了一种瑕疵检测系统，包括：图像采集组件，用于获取处于生产过程中的待检测对象的所述待处理图像；瑕疵检测装置，与所述图像采集组件连接，用于从所述图像采集组件获得所述待处理图像。
其中，所述瑕疵检测装置包括：图像处理模块，用于通过深度学习瑕疵检测模型中的特征提取模块处理所述待处理图像，所述特征提取模块包括中间特征层，所述中间特征层中包括丢弃层，所述丢弃层用于对所述中间特征层处理所述待处理图像后提取的特征进行随机丢弃，以使得所述待处理图像经过所述特征提取模块处理后获得经过随机丢弃处理后的多个特征；通过所述深度学习瑕疵检测模型中的检测头对经过随机丢弃处理后的多个特征进行处理，获得所述待处理图像中的瑕疵的多个类别预测结果、多个位置预测结果；根据所述瑕疵的多个类别预测结果和多个位置预测结果，获得所述待处理图像中的所述瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性。
在示例性实施例中，所述图像采集组件包括：图像拍摄设备，所述图像拍摄设备包括面阵相机。其中，所述面阵相机用于采集原始待处理图像，并对所述原始待处理图像进行像素合并处理，以生成所述待处理图像。
面阵相机是指用于获取待检测对象表面图像的相机，可以采用不同类型的传感器， 例如可以为CCD(Charge-Coupled Device，电荷耦合器件)或阵列相机(array camera)或CMOS面阵相机，由固定排布的像素阵列组成，每个像素单元都能够同时捕捉图像，从而可以一次性地采集整个图像。面阵相机可以提供更高的分辨率和灵敏度，特别是在较低的光照条件下。
在一些实施例中，可以通过binning(像素合并)来优化基础相机算法，以融合面阵相机例如CCD相机的多个真实像素，从而实现具有非常短曝光时间的清晰图像，是一种有效的高速图像捕获方法。像素合并处理时将图像中的像素合并为“超级像素”或“虚拟像素”，例如，将2×2或3×3等个像素融合为一个虚拟像素来增强图像，使相机能够在非常低的曝光时间下拍摄而不会曝光不足，从而避免因曝光时间过长而产生模糊和拖尾。
在CCD相机中，像素合并过程在硬件或“芯片上”进行。合并后的超级像素具有更高的信号强度，这有助于在非常短的曝光时间内获得更清晰的图像。由于曝光时间缩短，相机可以更快地连续捕捉图像，从而实现高速图像捕获。
例如，面阵相机通过binning技术实现像素融合，从而在硬件层提高图像初始亮度，保证相机能够在较小的曝光时间内拍摄高速运动的口罩布料，避免出现运动模糊的图像。
在示例性实施例中，所述图像采集组件还包括：条形光源，位于所述图像拍摄设备的一侧，以预定角度斜向照射在生产线上移动的所述待检测对象上。
条形光源是一种特殊的光源，其形状窄而长，通常采用LED或CCFL作为光源。条形光源的亮度比普通光源更为均匀，这使得它非常适合用于照明或显示，能够在一定的空间范围内提供更为均匀的光照。条形光源可以根据具体的使用场景进行定制，包括调整其长度、亮度以及颜色等参数，以满足不同的照明需求，实现最佳的照明效果。条形光源具有较高的耐用性和长寿命，可以在较长时间内保持稳定的光输出。条形光源不仅可以调节亮度和颜色，还可以通过改变光束的形状及方向，满足多种场景应用需求。条形光源采用LED技术，具有高效节能的特点，比传统的荧光灯或白炽灯更为节能。同时，它也具有更好的环保性能，有助于减少能源消耗和环境污染。
需要说明的是，本公开实施例的瑕疵检测系统中，也可以在相机中自带光源。在另一些实施例中，也可以采用其它光源，例如面光源、环形光源等。
在示例性实施例中，所述预定角度为45度。
例如，条形光源位于相机的一侧，以45度角斜向照射在生产流水线上移动的口罩布料。这种特定的照明角度可以最大化瑕点的可视对比度，使其在相机捕捉的图像中更为明显。首先，斜向照射的方式能够产生阴影和反射效果，这些效果有助于凸显口罩布料表面的不平整或瑕疵部分。当光线以一定角度照射在布料上时，瑕疵部分可能会因为形状、质地或深度的不同，而产生不同的反射或吸收效果。这样，相机在捕捉图像时就能更容易地区分出瑕疵与正常部分的差异。45度角既能保证光线能够充分 覆盖到布料的表面，又能避免光线过于直接或过于斜射而导致阴影过重或过浅。适当的阴影和反射效果有助于增强瑕点的对比度，使其在图像中更加突出。条形光源的窄长形状使得光线能够集中在一个较窄的区域内，从而提高了光照的均匀性和一致性。这对于检测流水线上移动的口罩布料尤为重要，因为布料的移动可能会导致光照条件的变化。条形光源的稳定性和一致性有助于确保相机捕捉到的图像质量稳定可靠。通过合理设置条形光源的位置和角度，可以最大化瑕点的可视对比度，提高相机捕捉图像的质量和准确性，从而实现对口罩布料瑕疵的有效检测。
需要说明的是，上述45度不要求严格精确，可以具有一定的误差范围，例如大于±3度均可以。
在示例性实施例中，所述系统包括工控机组件，所述工控机组件包括工控机，所述瑕疵检测装置部署在所述工控机上。
在示例性实施例中，所述工控机上还部署有数据服务组件。所述数据服务组件用于对所述生产过程中检测到的瑕疵信息进行记录和统计。
在一些实施例中，数据服务组件通过MySQL管理所有瑕点数据(瑕疵信息，例如图像中是否存在瑕疵，瑕疵的类别、位置、类别不确定性和位置不确定性、瑕疵等级等与瑕疵相关的任意数据)。
在另一些实施例中，数据服务组件可以将检测到的瑕疵信息实时上传到云端服务器。这样可以进行数据的集中管理，同时也方便进行远程监控、分析和优化生产流程。
在示例性实施例中，本公开实施例提供的系统还包括：报警组件，与所述瑕疵检测装置连接，用于在所述瑕疵检测装置检测到瑕疵时，接收所述瑕疵检测装置发送的触发信号，并根据所述触发信号发送报警信息。例如，该报警组件可以包括报警器，但本公开并不限定于此。
在示例性实施例中，本公开实施例提供的系统还包括：显示器，与所述瑕疵检测装置连接，用于显示所述瑕疵检测装置检测出的瑕疵信息。
下面结合图13至图15对本公开实施例提供的系统进行举例说明，但本公开并不限定于此。
如图13所示，瑕疵检测系统120可以包括图像采集组件121和瑕疵检测装置122。例如该瑕疵检测装置122可以设置在高性能工控机组件中。该瑕疵检测装置122中可以包括图像处理模块1221。图像采集组件121包括图像拍摄设备1211(例如面阵相机)和光源1212(例如，条形光源)。图像拍摄设备1211与工控机组件连接。工控机组件包括工控机。
可选的，瑕疵检测系统120还可以包括报警组件123，报警组件123可以与工控机组件连接。
例如，高速面阵相机与高性能工控机相连接，将采集到的图像实时回传到工控机上。报警组件123与高性能工控机相连接，当工控机检测到布面图像/口罩布图像存在 瑕点/瑕疵时，发送触发信号给报警器，报警器发出声光信号作为报警信息，提示工人处理对应瑕疵口罩。其中，算法识别组件/算法层和数据服务组件均在高性能工控机上运行。
图14显示了口罩生产的缺陷检测系统的示例性场景。如图14所示，瑕疵检测系统120的硬件可以包括图像采集组件/系统(包括高速相机1211’和条形光源1212’，负责采集图像信息，例如待处理图像)、高性能工控机(Industrial Control Computer，IPC，工业控制计算机的简写，负责处理采集到的图像信息，并生成检测结果，例如包括类别检测结果和位置检测结果以及类别不确定性和位置不确定性)122’，可选的，还可以包括报警器。高性能工控机122’与高速相机1211’之间具有相机链路(Camera Link接口)124。高性能工控机122’可以与条形光源1212’通过RS232进行连接。高速相机1211’和条形光源1212’位于口罩材料的切割机125之前的位置。这里假设口罩材料为无纺布。口罩材料或口罩布沿着口罩无纺布运动方向在导布辊126上朝向切割机125运动。条形光源1212’朝向导布辊126上的口罩布发射的光线与该口罩布之间的角度例如可以为45度。
参考图14，相机1211’(例如摄像头)采集到的口罩布图像/材料图像由IPC进行处理，以进行缺陷/瑕疵识别。如果在图像中识别出缺陷，则IPC通过报警器发送声光信号，通知装配线上的工人拒收缺陷产品。
从图14可以看出，高速相机1211’和光源(例如条形光源1212’)安装在口罩材料切割机125之前的位置。由于检测环境中的空间非常有限，并且在生产线上移动口罩材料的速度非常快，因此相机需要在短时间内(例如，至少每秒25帧)捕获图像。然而，当相机以如此高的速度拍摄图像时，由于曝光时间的限制，图像会变得模糊和失真。因此，本公开实施例使用binning将2*2个像素融合为一个虚拟像素(virtual pixel)来增强图像，使相机能够在非常低的曝光时间下拍摄，而不会出现曝光不足，从而避免了因长时间曝光导致的模糊和拖影。
本公开实施例提出的检测系统克服了相关技术中对机械结构过分依赖，对视觉识别方法描述不足的局限。该系统利用先进的图像捕捉设备、结合深度学习算法的智能识别技术，以及高效的数据服务支持，实现对高速运动口罩布料中瑕疵的实时、准确检测。
如图15所示，本公开实施例提供的瑕疵检测系统(也可称之为智能缺陷检测系统)包括硬件、算法层/组件(也可以称之为检测算法层)和数据服务层/组件。系统的硬件组成包括工控机、高速摄像头(高速相机)、光源以及报警器，它们共同工作以捕捉图像，并在检测到瑕疵时发出警告。算法层包括预处理、通道剪枝、深度可分离卷积和贝叶斯优化等技术，有效提高了检测速度和准确性。数据服务组件包括图像和数据存储、瑕点数据分析、瑕点数据报告和自动化瑕点标注。检测算法是检测系统的核心部分，而硬件和数据服务使缺陷检测系统更快、更准确、更便于使用。
相机例如可以采用CIS相机(CMOS Image Sensor，互补金属氧化物半导体图像传感器)、CCD相机等。
例如，相机和光源负责采集口罩布表面图像信息以获得口罩布图像；工控机负责处理采集的图像，识别口罩布表面的瑕点；报警器用于提示工人处理不符合质量标准的次品口罩。当相机采集的口罩布图像，由工控机处理进行瑕疵识别，如果在图像中识别出瑕疵，工控机会通过报警器发出声光信号，通知生产线上的工人剔除有瑕疵的产品。
算法层是系统智能的核心，负责瑕点定位和分类。算法层包括图像预处理、通道剪枝、深度可分离卷积处理、贝叶斯优化模块。预处理的作用是在瑕疵检测之前，对图像进行预处理，通过图像增强和滤波提高瑕点和背景的差异性，从而更好地识别瑕点；通道剪枝是基于轻量化模型设计的需求，对模型进行通道剪枝，通过L2-norm度量卷积通道对网络输出的贡献，根据贡献程度进行通道剪枝，以减少其模型参数，提高处理速度；深度可分离卷积处理也是一种模型轻量化手段，通过Deepwise Conv和Pointwise Conv简化卷积运算过程，深度可分离卷积对比普通卷积运算，可降低85％的运算参数，同时保证精度不受损失；贝叶斯优化是运用贝叶斯方法优化模型，首先在模型添加dropout层创造随机性模拟贝叶斯后验分布，增强瑕疵检测的准确性和可靠性。贝叶斯方法能够对预测的不确定性进行量化。在检测系统中，模型不仅能预测出结果，还能提供关于这些预测有多确信的信息。这种不确定性的量化使得系统能更智能地处理信息，如对那些模型不够确信的预测采取更谨慎的处理方式。
在该实施例中，检测算法可以预测口罩材料瑕疵的四种主要类型：贴纸(贴纸是指口罩布每卷之间的连接位置，通过胶纸贴在一起上流水线的，因此也可以称之为胶纸)、污点(污渍)、印刷问题(是指某些口罩是有图案和印花的，会存在印花错误)和口罩布料折痕。在在线运行期间，整个系统是端到端的。它以获取的图像为输入，模型首先移除不相关的背景，不相关的背景是指图像中拍到的除口罩布以外的区域。通过通道剪枝和深度可分离卷积实现高效率的图像特征提取。然后，在检测架构中融入了关于随机和认知不确定性的贝叶斯方法。随机不确定性源于训练数据本身的随机性。在口罩布瑕疵检测的背景下，这可能包括因口罩布料的自然变异、生产过程中的微小差异、或是图像采集过程中的噪声引起的不确定性。随机不确定性通常是不可减少的，因为它反映了数据本身的固有变异。认知不确定性源自于模型参数的不确定性，是关于模型对于数据的理解和推断过程中的不确定性。在口罩布瑕疵检测中，认知不确定性可能源于模型对特定类型瑕疵识别的不确定性，或是模型对新材料和新类型瑕疵的适应能力。贝叶斯方法包括：在backbone的中间特征层添加dropout层用于模拟贝叶斯的后验分布效果，来获得关于特征的不同输出，这些输出的分布提供了对预测不确定性的一个近似，其中预测结果的变异性反映了模型的不确定性；修改检测头，在模型输出包含边界框的坐标、类别和置信度的基础上新增各项参数的均值和方差表 示对预测不确定的度量；调整损失函数，新增负对数似然损失，惩罚过高的不确定性和不准确的预测。具体的，新增了检测头的负对数似然损失函数，用来计算贝叶斯概率损失。作用有二：第一惩罚过高的不确定性(因为如果模型对自己的预测很不确定，损失会增大)。第二惩罚不准确的预测(预测误差大会增加损失)。
本公开实施例优化口罩布料瑕疵检测系统的瑕疵类别、位置和不确定性的预测过程。改进检测头的输出为概率分布，新增贝叶斯损失，例如负对数似然损失，来增加对于不确定性结果的惩罚，从而优化反向传播。在backbone中间特征层增加dropout来模拟模型参数的不确定性，实现贝叶斯后验分布。数据不确定性是由数据噪声、标注方差、样本质量(例如光照，分辨率，对比度)决定的。通过上述改进，实现模型对其预测的不确定性进行估计，从而模型提高其鲁棒性和可靠性。这些创新的实施对于提高口罩布料瑕疵检测系统的准确性和效率较为重要，特别是在高速生产环境中。通过这些技术的结合，能够实现对高速运动口罩布料的有效检测，同时提高了系统的鲁棒性和可靠性。
数据服务层是口罩布瑕疵检测系统中的信息处理和决策支持中心，负责整合算法层的输出并执行后续的数据管理和决策制定。数据服务层包括瑕点和图像数据存储、瑕点数据聚合、瑕点数据传输、自动决策制定模块。图像和瑕点数据存储负责收集并保存，所有通过图像采集系统获取的原始图像数据以及算法层处理后的结果数据(包括瑕点的位置、瑕点的瑕疵类型、每个瑕点的不确定性、疵点区域的图像特征等中的一种或多种)，通过高效的数据组织和索引机制，确保数据的可查询性和可访问性。其中瑕点数据分析会对检测到的各类瑕疵数据进行统一整合和分析，将分散的瑕疵信息汇总成直观的报告，以便于监控和评估生产质量。瑕点数据传输，确保检测到的瑕疵信息能够快速且准确地传递到生产线的下游环节或质量控制部门，为实时响应和处理瑕疵提供支持。自动决策制定，是基于聚合分析的瑕疵数据，结合生产线的实际情况，本模块通过智能决策算法分析瑕疵点检出的分布情况和不确定性概率，生成报告，帮助生产者了解目前口罩生产出现哪种瑕疵点的情况最多，严重瑕疵点出现的频率等，让生产者从更宏观的角度了解生产工艺的缺陷，以优化生产过程。
数据服务层负责存储、分析和传输检测数据，并根据瑕疵的严重程度实施相应的处理策略。此外，数据服务层还增强了人工检查的效率，通过瑕疵边框的粗细直观显示瑕点等级/瑕疵等级，并提供瑕点信息/瑕疵信息的检索功能，支持质量追踪和生产流程的优化。
本公开实施例涉及一种高速生产环境下的口罩布瑕疵检测系统。该系统集成了高性能图像捕捉硬件(相机和光源)、先进的深度学习算法(包括上述实施例中提及的深度学习瑕疵检测模型)以及全面的数据服务层，以实现对高速移动口罩布料的实时瑕疵检测和分类。整体而言，本公开提供了一种既高效又可靠的解决方案，以满足快速生产线上对于高质量口罩的需求。
本公开的实施例提供的口罩布瑕疵检测系统，通过设置图像采集组件和算法层以及数据服务组件，能够在不影响口罩布生产流水线速度的前提下，对经过的口罩布进行高效、准确的瑕疵检测，并在检测到瑕疵时，自动进行打标，极大地提高了生产效率和产品质量。通过人工和自动化检测的结合，本系统也确保了高水平的检测可靠性。例如结合贝叶斯方法会给出预测出的每个瑕点一个不确定性的评分(例如参考上述公式(2))，如果某个瑕点的不确定性的评分较高，则会将该瑕点发送给人工来打标。
本公开实施例中的“高速”是指满足全局快门，最大帧率例如超过60fps。高速例如是指，在流水线上以500个/min的口罩生产速度去拍摄口罩布，图像不出现动态模糊，相机帧采样间隔不会丢失口罩布区域信息。
根据本公开的一个实施方面，提供了一种口罩布瑕疵检测系统，该系统包括：图像采集组件，包含高速面阵相机，条形光源以及相机和光源支架。高速面阵相机用于采集口罩布表面图像，光源用于突出瑕疵和背景的视觉差异；高性能工控机组件，包含一台装配了高性能GPU的工控机和一台显示器，工控机用于实时检测相机采集的图像中的瑕疵，显示器安装于口罩生产线尾端，方便工人确认口罩瑕疵的图像信息；报警组件，安装在口罩生产线尾端，当检测到瑕疵时，报警器触发提示工人及时处理；瑕点识别算法组件，通过分析瑕点特性和图像特性，对模型进行改进，优化其计算速度和精度。其中，图像特性可以包括图像中的背景的变化情况，被检测物体的表面纹理变化情况，周围环境光对图像的干扰情况等中的一种或多种。涉及图像特性的处理就是调整相机曝光、移除图像背景。
本公开实施例中，瑕点特性就是瑕点的锚框分布特性，瑕疵点特征类间欧几里得距离。例如，分析瑕疵点的锚框分布特性，瑕点特征类间欧几里得距离。根据锚框分布特性进行聚类，生成最优先验锚框，根据瑕点特征类间欧几里得距离，进行排序比对，优化瑕疵点分类精度。
本公开实施例中，分析疵点特征/瑕点特性的两种方式：
一种是生成最优先验锚框：例如通过k-means对标注的疵点锚框进行聚类，找出最优的先验锚框，从而分析疵点的长宽比例分布情况。
另一种是hard sample挖掘和优化：首先根据锚框的分布，找出不同比例的疵点，例如对于长宽比1：1的疵点，对于疵点A，计算A和长宽比1：1的其他类疵点的欧几里得距离。然后找出距离最近的不同类疵点标记为N(Negative)，找出距离最远的同类疵点标记为P(Positive)。那么这3个疵点样本就构成了一个hard sample triplet(三元组)。通过构筑多个不同长宽比例疵点的这种hard sample triplets最终构建一个hard sample dataset(数据集)。然后针对hard sample dataset将模型进行深度训练，使模型对这种hard sample更加拟合，从而达到更好的检测效果。即挑选最难分辨的疵点，进行针对性优化。
条形光源的优点是功率大，光照强度高。因为相机要拍摄高度运动物体，曝光时 间不能太长，所以需要足够强的光源来提供亮度。选择的条形光源的照明宽度和长度足够覆盖相机拍摄面积。
数据服务组件，对所有流水线上检测的瑕点进行记录和统计，实时更新到数据库，进行数据分析，帮助工人分析瑕疵成因，从而优化工艺。
本公开的目的在于提供一种口罩布瑕疵检测系统，该系统旨在至少在一定程度上解决现有技术在口罩布瑕疵检测中所遇到的诸如检测精度不足等问题。本系统通过引入先进的图像采集和处理技术，显著提高了瑕疵识别的准确性和效率，从而改善了相关技术领域的工艺流程。
本公开实施例提供了一整套在线高速疵点检测系统，其是一种针对具体业务场景(处于生产流水线上高速运动的口罩布)的系统性改进和创新，对模型进行整体轻量化，并在模型中引入贝叶斯估计，通过AIGC技术增强训练样本，通过量化学习解决hard sample问题使模型能够更好的收敛，通过binning提高图像采集质量，设计相机自动曝光算法，以及设计人机交互逻辑优化疵点决策过程。
在示例性实施例中，本公开实施例还提供了一种瑕疵检测系统，包括：图像拍摄设备，用于采集处于生产过程中的待检测对象的所述待处理图像；处理器和用于存储计算机程序的存储器；当所述处理器执行所述计算机程序时，实现上述任意实施例中所述方法中的步骤。
在示例性实施例中，所述存储器包括用于存储瑕疵检测程序(实现上述算法层的功能)的存储器和用于存储服务数据程序(实现上述数据服务层的功能)的存储器。
在以上描述中，智能缺陷检测系统和方法等被列为优选示例。本领域的技术人员将明白，在不脱离本发明的范围和精神的情况下，可以进行修改，包括添加和/或替换。可以省略具体细节，以免混淆本发明；然而，本公开的目的是使本领域的技术人员能够实践本文的教导，而无需进行过多的实验。
图16是本公开一示例性实施例的瑕疵检测装置的结构示意图。如图16所示，本公开实施例提供的瑕疵检测装置1500包括图像获得模块1510和图像处理模块1520。
图像获得模块1510可以用于获得待处理图像。
图像处理模块1520可以用于通过深度学习瑕疵检测模型中的特征提取模块处理所述待处理图像，所述特征提取模块包括中间特征层，所述中间特征层中包括丢弃层，所述丢弃层用于对所述中间特征层处理所述待处理图像后提取的特征进行随机丢弃，以使得所述待处理图像经过所述特征提取模块处理后获得经过随机丢弃处理后的多个特征。
图像处理模块可以还用于通过所述深度学习瑕疵检测模型中的检测头对经过随机丢弃处理后的多个特征进行处理，获得所述待处理图像中的瑕疵的多个类别预测结果、多个位置预测结果。
图像处理模块可以还用于根据所述瑕疵的多个类别预测结果和多个位置预测结 果，获得所述待处理图像中的所述瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性。
其中，图像处理模块1520还用于：根据所述第一瑕疵的多个第一类别预测结果和多个第一位置预测结果，获得所述第一瑕疵的第一类别检测结果、第一位置检测结果、第一类别不确定性和第一位置不确定性；根据所述第二瑕疵的多个第二类别预测结果和多个第二位置预测结果，获得所述第二瑕疵的第二类别检测结果、第二位置检测结果、第二类别不确定性和第二位置不确定性。
在示例性实施例中，图像获得模块1510还用于：从图像采集组件中的图像拍摄设备获得所述待处理图像，所述图像采集组件用于获取处于生产过程中的待检测对象的所述待处理图像。
其中，瑕疵检测装置1500还可以包括：速度获得模块，可以用于获得所述待检测对象的当前生产速度和所述深度学习瑕疵检测模型对所述待处理图像的当前处理速度。图像处理模块1520还可以用于根据所述当前生产速度和所述当前处理速度调整用于所述深度学习瑕疵检测模型处理所述待处理图像的并行线程或并行进程数量。
在示例性实施例中，图像获得模块1510还用于：从图像采集组件中的图像拍摄设备获得所述待处理图像；若所述待处理图像的灰度值和灰度分布不满足预设条件，则发送控制信号调整所述图像拍摄设备的曝光参数；从调整曝光参数后的所述图像拍摄设备获得新的待处理图像，直到获得满足所述预设条件的所述待处理图像。
在示例性实施例中，瑕疵检测装置1500还可以包括：发送模块，用于若所述瑕疵的类别不确定性和/或位置不确定性大于不确定阈值，则发送针对所述瑕疵的提示信息。
在示例性实施例中，瑕疵检测装置1500还可以包括：数据处理模块，用于统计所述待处理图像中检测出的各个瑕疵的像素数量信息；根据各个瑕疵的像素数量信息确定各个瑕疵的瑕疵等级；根据各个瑕疵的瑕疵等级，执行相应的处理。
在示例性实施例中，数据处理模块还可以用于：在所述待处理图像中，将处于不同瑕疵等级的瑕疵分别用具有不同粗细程度的边框标识。
在示例性实施例中，瑕疵检测装置1500还可以包括：数据处理模块，可以用于保存所述待处理图像以及所述待处理图像中检测出的瑕疵的类别检测结果、位置检测结果、类别不确定性、位置不确定性、瑕疵等级、瑕疵区域图像、瑕疵区域的图像特征、检出时间和检出序号；接收并根据输入的瑕疵区域的图像特征、检出时间和检出序号中的一项或多项，检索相应的瑕疵信息。
在示例性实施例中，瑕疵检测装置1500还可以包括：模型训练模块，可以用于获得目标训练数据集，所述目标训练数据集中包括目标样本图像及所述目标样本图像中的目标样本瑕疵的标注类别和标注位置；将所述目标样本图像输入至所述深度学习瑕疵检测模型，获得所述目标样本瑕疵的多个类别预测结果和多个位置预测结果；根 据所述目标样本瑕疵的多个类别预测结果和多个位置预测结果获得所述目标样本瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性；根据所述目标样本瑕疵的标注类别及类别检测结果、标注位置及位置检测结果、以及类别不确定性和位置不确定性构建损失函数以训练所述深度学习瑕疵检测模型。
在示例性实施例中，模型训练模块还可以用于：获取初始样本图像及所述初始样本图像中的初始样本瑕疵的标注类别和标注位置；对所述初始样本图像进行随机裁剪、旋转、缩放、光照变化模拟、瑕疵合成中的任意一项或多项处理，生成扩增样本图像及所述扩展增益图像中的扩展样本瑕疵的标注类别和标注位置。其中，所述目标样本图像包括所述初始样本图像和所述扩增样本图像。
在示例性实施例中，模型训练模块还可以用于：提取所述初始样本图像中的初始样本瑕疵；将所述初始样本瑕疵与扩增背景图像合成，生成所述扩增样本图像；和/或，通过深度学习生成模型学习所述初始样本图像，生成所述扩增样本图像。
在示例性实施例中，模型训练模块还可以用于：获取候选样本图像及所述候选样本图像中的候选样本瑕疵的标注类别和标注位置；通过所述特征提取模块获得所述候选样本图像的输出特征；基于各个候选样本图像的输出特征，获得各个候选样本图像之间的特征距离；选择标注类别相同的候选样本图像中特征距离大于第一距离阈值的候选样本图像，和/或，标注类别不同的候选样本图像中特征距离小于第二距离阈值的候选样本图像，作为所述目标训练数据集中的目标样本图像。
在示例性实施例中，模型训练模块还可以用于在所述深度学习瑕疵检测模型的训练过程中：提取所述深度学习瑕疵检测模型中的每个卷积层的权重张量；获得所述权重张量中每个通道的欧几里得范数；对每个卷积层根据其通道的欧几里得范数进行排序，根据剪枝阈值或剪枝比例，对相应的通道进行剪枝。
图16实施例的其它内容可参考上述其它实施例。
图17示意性示出了根据本公开的一实施例的电子设备1600的示意性结构图。图17所示的电子设备1600包括处理器1610，处理器1610可以从存储器中调用并运行计算机程序，以实现本公开实施例中的方法。
可选地，如图17所示，电子设备1600还可以包括存储器1620。其中，处理器1610可以从存储器1620中调用并运行计算机程序，以实现本公开实施例中的方法。
其中，存储器1620可以是独立于处理器1610的一个单独的器件，也可以集成在处理器1610中。
可选地，如图17所示，电子设备1600还可以包括收发器1630，处理器1610可以控制该收发器1630与其他设备进行通信，具体地，可以向其他设备发送信息或数据，或接收其他设备发送的信息或数据。
可选地，处理器1610、存储器1620以及收发器1630之间可以通过通信总线1640实现相互之间的双向通信。
应理解，本公开实施例的处理器可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法实施例的各步骤可以通过处理器中的硬件的集成逻辑电路或者软件形式的指令完成。
上述的处理器可以是通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现成可编程门阵列(Field Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本公开实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本公开实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器，处理器读取存储器中的信息，结合其硬件完成上述方法的步骤。
可以理解，本公开实施例中的存储器可以是易失性存储器或非易失性存储器，或可包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(Read-Only Memory，ROM)、可编程只读存储器(Programmable ROM，PROM)、可擦除可编程只读存储器(Erasable PROM，EPROM)、电可擦除可编程只读存储器(Electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(Random Access Memory，RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(Static RAM，SRAM)、动态随机存取存储器(Dynamic RAM，DRAM)、同步动态随机存取存储器(Synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(Double Data Rate SDRAM，DDR SDRAM)、增强型同步动态随机存取存储器(Enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(SynchlinkDRAM，SLDRAM)和直接内存总线随机存取存储器(Direct Rambus RAM，DR RAM)。应注意，本文描述的系统和方法的存储器旨在包括但不限于这些和任意其它适合类型的存储器。应理解，上述存储器为示例性但不是限制性说明。
本公开实施例还提供了一种计算机可读存储介质，用于存储计算机程序。该计算机程序使得计算机执行本公开实施例的各个方法中的相应流程，为了简洁，在此不再赘述。本公开实施例还提供了一种计算机程序产品，包括计算机程序指令。该计算机程序指令使得计算机执行本公开实施例的各个方法中的相应流程，为了简洁，在此不再赘述。本公开实施例还提供了一种计算机程序。当该计算机程序在计算机上运行时，使得计算机执行本公开实施例的各个方法中的相应流程，为了简洁，在此不再赘述。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功 能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本公开的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。在本公开所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。
工业实用性
本公开适用于人工智能技术领域，用以解决相关技术中瑕疵预测不准确且无法输出类别不确定性和位置不确定性的问题，达到快速有效的预测瑕疵的类别和位置、并评估预测结果的类别不确定性和位置不确定性的效果。

Claims (38)

  1. 一种瑕疵检测方法，包括：
    获得待处理图像；
    通过深度学习瑕疵检测模型中的特征提取模块处理所述待处理图像，所述特征提取模块包括中间特征层，所述中间特征层中包括丢弃层，所述丢弃层用于对所述中间特征层处理所述待处理图像后提取的特征进行随机丢弃，以使得所述待处理图像经过所述特征提取模块处理后获得经过随机丢弃处理后的多个特征；
    通过所述深度学习瑕疵检测模型中的检测头对经过随机丢弃处理后的多个特征进行处理，获得所述待处理图像中的瑕疵的多个类别预测结果、多个位置预测结果；
    根据所述瑕疵的多个类别预测结果和多个位置预测结果，获得所述待处理图像中的所述瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性。
  2. 如权利要求1所述的方法，其中，所述中间特征层包括第一跨阶段卷积块，所述丢弃层包括第一丢弃层；
    其中，所述第一跨阶段卷积块包括第一跨阶段输入层、第一卷积层、第一中间层、第二卷积层、第一融合层、第三卷积层和所述第一丢弃层；
    所述第一卷积层用于将所述第一跨阶段输入层输入的第一跨阶段输入特征图进行卷积处理，向所述第一中间层输入第一卷积特征图；
    所述第一中间层用于对所述第一卷积特征图进行处理，向所述第一融合层输入第一待融合特征图；
    所述第二卷积层用于对所述第一跨阶段输入特征图进行卷积处理，向所述第一融合层输入第二待融合特征图；
    所述第一融合层用于对所述第一待融合特征图和所述第二待融合特征图进行融合处理，向所述第三卷积层输入第一融合特征图；
    所述第三卷积层用于对所述第一融合特征图进行卷积处理，向所述第一丢弃层输入第二卷积特征图；
    所述第一丢弃层用于对所述第二卷积特征图中的特征进行随机丢弃，获得多个第一丢弃特征图，所述待处理图像经过所述特征提取模块处理后获得经过随机丢弃处理后的多个特征包括多个所述第一丢弃特征图。
  3. 如权利要求2所述的方法，其中，所述中间特征层还包括第二跨阶段卷积块和第三跨阶段卷积块，所述丢弃层还包括第二丢弃层和第三丢弃层；
    其中，所述第二跨阶段卷积块的输入端连接所述第一跨阶段卷积块的输出端，以分别接收多个所述第一丢弃特征图，并对多个所述第一丢弃特征图进行处理，通过所述第二丢弃层向所述第三跨阶段卷积块输入多个第二丢弃特征图；
    所述第三跨阶段卷积块的输入端连接所述第二跨阶段卷积块的输出端，以分别接收多 个所述第二丢弃特征图，并对多个所述第二丢弃特征图进行处理，通过所述第三丢弃层输出多个第三丢弃特征图。
  4. 如权利要求3所述的方法，其中，所述深度学习瑕疵检测模型还包括特征金字塔模块，所述特征金字塔模块包括第一空间金字塔池化块、第二空间金字塔池化块和第三空间金字塔池化块；
    所述第一空间金字塔池化块连接所述第一跨阶段卷积块，用于接收多个所述第一丢弃特征图；
    所述第二空间金字塔池化块连接所述第二跨阶段卷积块，用于接收多个所述第二丢弃特征图，并分别对多个所述第二丢弃特征图进行处理，获得多个第二空间池化特征图；
    所述第三空间金字塔池化块连接所述第三跨阶段卷积块，用于接收多个所述第三丢弃特征图，并分别对多个所述第三丢弃特征图进行处理，获得多个第三空间池化特征图；
    所述第一空间金字塔池化块还连接所述第二空间金字塔池化块，用于接收多个所述第二空间池化特征图，根据多个所述第一丢弃特征图和多个所述第二空间池化特征图获得多个第一空间池化特征图；
    所述第二空间金字塔池化块还连接所述第三空间金字塔池化块，用于接收多个所述第三空间池化特征图，根据多个所述第二丢弃特征图和多个所述第三空间池化特征图获得多个第二空间池化特征图。
  5. 如权利要求4所述的方法，其中，所述检测头包括第一深度可分离卷积块、第二深度可分离卷积块和第三深度可分离卷积块；所述瑕疵包括第一瑕疵、第二瑕疵和第三瑕疵；
    所述第一深度可分离卷积块与所述第一空间金字塔池化块连接，用于接收多个所述第一空间池化特征图，并分别对多个所述第一空间池化特征图进行处理，获得所述第一瑕疵的多个第一类别预测结果和多个第一位置预测结果；
    所述第二深度可分离卷积块与所述第二空间金字塔池化块连接，用于接收多个所述第二空间池化特征图，并分别对多个所述第二空间池化特征图进行处理，获得所述第二瑕疵的多个第二类别预测结果和多个第二位置预测结果；
    所述第三深度可分离卷积块与所述第三空间金字塔池化块连接，用于接收多个所述第三空间池化特征图，并分别对多个所述第三空间池化特征图进行处理，获得所述第三瑕疵的多个第三类别预测结果和多个第三位置预测结果；
    其中，根据所述瑕疵的多个类别预测结果和多个位置预测结果，获得所述待处理图像中的所述瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性，包括：
    根据所述第一瑕疵的多个第一类别预测结果和多个第一位置预测结果，获得所述第一瑕疵的第一类别检测结果、第一位置检测结果、第一类别不确定性和第一位置不确定性；
    根据所述第二瑕疵的多个第二类别预测结果和多个第二位置预测结果，获得所述第二瑕疵的第二类别检测结果、第二位置检测结果、第二类别不确定性和第二位置不确定性；
    根据所述第三瑕疵的多个第三类别预测结果和多个第三位置预测结果，获得所述第三 瑕疵的第三类别检测结果、第三位置检测结果、第三类别不确定性和第三位置不确定性。
  6. 如权利要求2所述的方法，其中，所述第一中间层包括第一中间输入层、第一深度可分离卷积层和第一累加层；
    所述第一中间输入层用于接收所述第一卷积层输出的第一卷积特征图，并将所述第一卷积特征图分别输入至所述第一深度可分离卷积层和所述第一累加层；
    所述第一深度可分离卷积层用于对所述第一卷积特征图进行深度可分离卷积处理，向所述第一累加层输入第一深度分离卷积特征图；
    所述第一累加层对所述第一卷积特征图和所述第一深度分离卷积特征图进行累加处理，获得所述第一待融合特征图。
  7. 如权利要求6所述的方法，其中，所述第一深度可分离卷积层包括第一深度分离输入层、第一深度卷积层、第一点卷积层和第一激活函数层；
    所述第一深度分离输入层用于接收所述第一中间输入层输出的第一卷积特征图，并将所述第一卷积特征图分别输入至所述第一深度卷积层和所述第一激活函数层；
    所述第一深度卷积层用于对所述第一卷积特征图进行深度卷积处理，并向所述第一点卷积层输入第一深度卷积特征图；
    所述第一点卷积层用于对所述第一深度卷积特征图进行点卷积处理，并向所述第一激活函数层输入第一点卷积特征图；
    所述第一激活函数层用于对所述第一点卷积特征图和所述第一卷积特征图进行处理，获得所述第一深度分离卷积特征图。
  8. 如权利要求1所述的方法，其中，所述特征提取模块还包括与所述中间特征层连接的注意力块；
    所述注意力块用于对所述待处理图像进行处理，获得所述待处理图像的下采样特征图，并将所述下采样特征图输入至所述中间特征层。
  9. 如权利要求1所述的方法，其中，所述深度学习瑕疵检测模型还包括背景移除模块；
    所述背景移除模块用于移除所述待处理图像中的背景信息，并将移除所述背景信息之后的所述待处理图像输入至所述特征提取模块。
  10. 如权利要求1所述的方法，其中，获得待处理图像，包括：
    从图像采集组件中的图像拍摄设备获得所述待处理图像，所述图像采集组件用于获取处于生产过程中的待检测对象的所述待处理图像；
    其中，所述方法还包括：
    获得所述待检测对象的当前生产速度和所述深度学习瑕疵检测模型对所述待处理图像的当前处理速度；
    根据所述当前生产速度和所述当前处理速度调整用于所述深度学习瑕疵检测模型处理所述待处理图像的并行线程或并行进程数量。
  11. 如权利要求1所述的方法，其中，获得待处理图像，包括：
    从图像采集组件中的图像拍摄设备获得所述待处理图像；
    若所述待处理图像的灰度值和灰度分布不满足预设条件，则发送控制信号调整所述图像拍摄设备的曝光参数；
    从调整曝光参数后的所述图像拍摄设备获得新的待处理图像，直到获得满足所述预设条件的所述待处理图像。
  12. 如权利要求1所述的方法，其中，还包括：
    若所述瑕疵的类别不确定性和/或位置不确定性大于不确定阈值，则发送针对所述瑕疵的提示信息。
  13. 如权利要求1所述的方法，其中，还包括：
    统计所述待处理图像中检测出的各个瑕疵的像素数量信息；
    根据各个瑕疵的像素数量信息确定各个瑕疵的瑕疵等级；
    根据各个瑕疵的瑕疵等级，执行相应的处理。
  14. 如权利要求13所述的方法，其中，根据各个瑕疵的瑕疵等级，执行相应的处理，包括：
    在所述待处理图像中，将处于不同瑕疵等级的瑕疵分别用具有不同粗细程度的边框标识。
  15. 如权利要求1所述的方法，其中，还包括：
    保存所述待处理图像以及所述待处理图像中检测出的瑕疵的类别检测结果、位置检测结果、类别不确定性、位置不确定性、瑕疵等级、瑕疵区域图像、瑕疵区域的图像特征、检出时间和检出序号；
    接收并根据输入的瑕疵区域的图像特征、检出时间和检出序号中的一项或多项，检索相应的瑕疵信息。
  16. 如权利要求1所述的方法，其中，还包括：
    获得目标训练数据集，所述目标训练数据集中包括目标样本图像及所述目标样本图像中的目标样本瑕疵的标注类别和标注位置；
    将所述目标样本图像输入至所述深度学习瑕疵检测模型，获得所述目标样本瑕疵的多个类别预测结果和多个位置预测结果；
    根据所述目标样本瑕疵的多个类别预测结果和多个位置预测结果获得所述目标样本瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性；
    根据所述目标样本瑕疵的标注类别及类别检测结果、标注位置及位置检测结果、以及类别不确定性和位置不确定性构建损失函数以训练所述深度学习瑕疵检测模型。
  17. 如权利要求16所述的方法，其中，所述损失函数包括第一损失函数和第二损失函数；
    当所述目标样本瑕疵的标注类别和类别检测结果的差异越大，标注位置和位置检测结果的差异越大时，所述第一损失函数的取值越大；
    当所述目标样本瑕疵的类别不确定性和所述位置不确定性越大时，所述第二损失函数的取值越大。
  18. 如权利要求17所述的方法，其中，所述第一损失函数L1如下：
    MSE是表示边界框回归计算，yi是第i个目标样本瑕疵的标注位置，是预测的第i个目标样本瑕疵的位置检测结果，N是目标样本瑕疵的数量，N为大于或等于1的正整数，i为大于或等于1且小于或等于N的正整数；CE表示多元交叉熵计算，表示预测的类别检测结果中类别c的类别置信度；sc表示标注类别中类别c的类别置信度，C表示类别数量，C为大于或等于1的正整数，c为大于或等于1且小于或等于C的正整数；BCE表示二元交叉熵计算，opre表示针对目标样本瑕疵预测的目标置信度，o表示目标样本瑕疵中标注的目标的置信度。
  19. 如权利要求17所述的方法，其中，所述第二损失函数L2如下：
    μi是预测的第i个目标样本瑕疵的位置检测结果，yi是第i个目标样本瑕疵的标注位置；是预测的第i个目标样本瑕疵的位置不确定性；N是目标样本瑕疵的数量，N为大于或等于1的正整数，i为大于或等于1且小于或等于N的正整数；μc表示预测的类别c的类别检测结果；sc表示标注类别中类别c的类别置信度，表示预测的类别c的类别不确定性，C表示类别数量，C为大于或等于1的正整数，c为大于或等于1且小于或等于C的正整数。
  20. 如权利要求16所述的方法，其中，获得目标训练数据集，包括：
    获取初始样本图像及所述初始样本图像中的初始样本瑕疵的标注类别和标注位置；
    对所述初始样本图像进行随机裁剪、旋转、缩放、光照变化模拟、瑕疵合成中的任意一项或多项处理，生成扩增样本图像及所述扩展增益图像中的扩展样本瑕疵的标注类别和标注位置；
    其中，所述目标样本图像包括所述初始样本图像和所述扩增样本图像。
  21. 如权利要求20所述的方法，其中，对所述初始样本图像进行瑕疵合成处理，生成扩增样本图像，包括：
    提取所述初始样本图像中的初始样本瑕疵；将所述初始样本瑕疵与扩增背景图像合成，生成所述扩增样本图像；和/或，
    通过深度学习生成模型学习所述初始样本图像，生成所述扩增样本图像。
  22. 如权利要求16所述的方法，其中，获得目标训练数据集，包括：
    获取候选样本图像及所述候选样本图像中的候选样本瑕疵的标注类别和标注位置；
    通过所述特征提取模块获得所述候选样本图像的输出特征；
    基于各个候选样本图像的输出特征，获得各个候选样本图像之间的特征距离；
    选择标注类别相同的候选样本图像中特征距离大于第一距离阈值的候选样本图像，和/或，标注类别不同的候选样本图像中特征距离小于第二距离阈值的候选样本图像，作为所述目标训练数据集中的目标样本图像。
  23. 如权利要求16所述的方法，其中，在所述深度学习瑕疵检测模型的训练过程中：
    提取所述深度学习瑕疵检测模型中的每个卷积层的权重张量；
    获得所述权重张量中每个通道的欧几里得范数；
    对每个卷积层根据其通道的欧几里得范数进行排序，根据剪枝阈值或剪枝比例，对相应的通道进行剪枝。
  24. 如权利要求16所述的方法，其中，所述目标样本图像包括具有不同材料、纹理、颜色、印花图案的口罩图像。
  25. 一种瑕疵检测装置，包括：
    图像获得模块，用于获得待处理图像；
    图像处理模块，用于通过深度学习瑕疵检测模型中的特征提取模块处理所述待处理图像，所述特征提取模块包括中间特征层，所述中间特征层中包括丢弃层，所述丢弃层用于对所述中间特征层处理所述待处理图像后提取的特征进行随机丢弃，以使得所述待处理图像经过所述特征提取模块处理后获得经过随机丢弃处理后的多个特征；
    所述图像处理模块还用于，通过所述深度学习瑕疵检测模型中的检测头对经过随机丢弃处理后的多个特征进行处理，获得所述待处理图像中的瑕疵的多个类别预测结果、多个位置预测结果；
    所述图像处理模块还用于，根据所述瑕疵的多个类别预测结果和多个位置预测结果，获得所述待处理图像中的所述瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性。
  26. 一种瑕疵检测系统，包括：
    图像采集组件，用于获取处于生产过程中的待检测对象的所述待处理图像；
    瑕疵检测装置，与所述图像采集组件连接，用于从所述图像采集组件获得所述待处理图像；其中，所述瑕疵检测装置包括：
    图像处理模块，用于通过深度学习瑕疵检测模型中的特征提取模块处理所述待处理图像，所述特征提取模块包括中间特征层，所述中间特征层中包括丢弃层，所述丢弃层用于对所述中间特征层处理所述待处理图像后提取的特征进行随机丢弃，以使得所述待处理图像经过所述特征提取模块处理后获得经过随机丢弃处理后的多个特征；通过所述深度学习瑕疵检测模型中的检测头对经过随机丢弃处理后的多个特征进行处理，获得所述待处理图像中的瑕疵的多个类别预测结果、多个位置预测结果；根据所述瑕疵的多个类别预测结果和多个位置预测结果，获得所述待处理图像中的所述瑕疵的类别检测结果、位置检测结果、类别不确定性和位置不确定性。
  27. 如权利要求26所述的系统，其中，所述图像采集组件包括：
    图像拍摄设备，所述图像拍摄设备包括面阵相机；
    其中，所述面阵相机用于采集原始待处理图像，并对所述原始待处理图像进行像素合并处理，以生成所述待处理图像。
  28. 如权利要求27所述的系统，其中，所述图像采集组件还包括：
    条形光源，位于所述图像拍摄设备的一侧，以预定角度斜向照射在生产线上移动的所述待检测对象上。
  29. 如权利要求28所述的系统，其中，所述预定角度为45度。
  30. 如权利要求26所述的系统，其中，所述系统包括工控机组件，所述工控机组件包括工控机，所述瑕疵检测装置部署在所述工控机上。
  31. 如权利要求30所述的系统，其中，所述工控机上还部署有数据服务组件；
    所述数据服务组件用于对所述生产过程中检测到的瑕疵信息进行记录和统计。
  32. 如权利要求26所述的系统，其中，还包括：
    报警组件，与所述瑕疵检测装置连接，用于在所述瑕疵检测装置检测到瑕疵时，接收所述瑕疵检测装置发送的触发信号，并根据所述触发信号发送报警信息。
  33. 如权利要求26所述的系统，其中，还包括：
    显示器，与所述瑕疵检测装置连接，用于显示所述瑕疵检测装置检测出的瑕疵信息。
  34. 一种瑕疵检测系统，包括：
    图像拍摄设备，用于采集处于生产过程中的待检测对象的所述待处理图像；
    处理器和用于存储计算机程序的存储器；当所述处理器执行所述计算机程序时，实现如上述权利要求1至24任一项所述方法中的步骤。
  35. 如权利要求34所述的系统，其中，所述存储器包括用于存储瑕疵检测程序的存储器和用于存储服务数据程序的存储器。
  36. 一种电子设备，包括：
    一个或多个处理器；
    存储器，配置为存储一个或多个程序，当所述一个或多个程序被所述一个或多个处理器执行时，使得所述电子设备实现权利要求1至24中任一项所述的方法。
  37. 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，当所述计算机程序在计算机上运行时，使得所述计算机执行权利要求1至24中任一项所述的方法。
  38. 一种计算机程序产品，包括计算机程序，该计算机程序被处理器执行时实现权利要求1至24中任一项所述的方法。
PCT/CN2024/107018 2023-06-15 2024-07-23 瑕疵检测方法及相关设备 Ceased WO2024255919A1 (zh)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US202363508275P 2023-06-15 2023-06-15
US63/508,275 2023-06-15
CN202410709277.5A CN119151853A (zh) 2023-06-15 2024-06-03 瑕疵检测方法及相关设备
CN202410709277.5 2024-06-03

Publications (1)

Publication Number Publication Date
WO2024255919A1 true WO2024255919A1 (zh) 2024-12-19

Family

ID=93804114

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/107018 Ceased WO2024255919A1 (zh) 2023-06-15 2024-07-23 瑕疵检测方法及相关设备

Country Status (2)

Country Link
CN (1) CN119151853A (zh)
WO (1) WO2024255919A1 (zh)

Cited By (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119444761A (zh) * 2025-01-13 2025-02-14 广州机智云物联网科技有限公司 基于生产环节瑕疵识别的质量检测模型训练方法
CN119478706A (zh) * 2025-01-14 2025-02-18 长春光华学院 一种基于卷积神经网络的植物叶片病害检测方法
CN119648691A (zh) * 2025-02-12 2025-03-18 江苏嘉通能源有限公司 一种基于视觉图像的涤纶面料缺陷检测方法及系统
CN119810102A (zh) * 2025-03-13 2025-04-11 无锡日联科技股份有限公司 多模型融合的pcb通孔缺陷检测方法、装置、设备及介质
CN119850601A (zh) * 2025-03-18 2025-04-18 江西智成飞桨科技有限公司 光伏电池板ai瑕疵检测方法及设备
CN119964155A (zh) * 2025-02-06 2025-05-09 清华大学 水生生物图像识别计数方法、装置、设备及介质
CN119991581A (zh) * 2025-01-09 2025-05-13 中国船舶集团有限公司第七〇九研究所 一种目标缺陷检测方法及装置
CN119992039A (zh) * 2025-04-15 2025-05-13 成都川油瑞飞科技有限责任公司 一种面向复杂电气仪表设备的状态指示灯定位方法及装置
CN120032226A (zh) * 2025-01-24 2025-05-23 中国科学院宁波材料技术与工程研究所 复杂环境下的轻量级高精度动力电池极柱检测方法
CN120125573A (zh) * 2025-04-27 2025-06-10 宝鸡市永盛泰钛业有限公司 基于多光谱图像融合的钛棒表面磨损形貌检测方法及系统
CN120133163A (zh) * 2025-05-15 2025-06-13 河南工学院 基于人工智能的自动化分拣方法及系统
CN120259287A (zh) * 2025-06-04 2025-07-04 华东交通大学 一种铁路轨道扣件缺陷检测方法、系统、装置及介质
CN120411618A (zh) * 2025-04-16 2025-08-01 内蒙古农业大学 一种基于复杂环境背景下的苹果叶部病害多分类方法
CN120411910A (zh) * 2025-04-21 2025-08-01 贵州装备制造职业学院 一种基于人工智能的车辆表面缺陷识别方法及系统
CN120635092A (zh) * 2025-08-14 2025-09-12 闽江学院 基于瑕疵判断模型的纺织服装瑕疵图案数据处理方法
CN120782712A (zh) * 2025-06-06 2025-10-14 浙江金汇华特种耐火材料有限公司 一种基于机器学习的耐火材料质量检测方法及系统
CN120953292A (zh) * 2025-10-20 2025-11-14 中国三峡建工(集团)有限公司 一种基于图像处理的高压电气设备表面缺陷识别方法
CN121033106A (zh) * 2025-10-28 2025-11-28 武汉科技大学 一种电力金具目标跟踪计数方法及系统
CN121052295A (zh) * 2025-11-05 2025-12-02 齐鲁工业大学(山东省科学院) 多工况刀具磨损监测模型的构建及其磨损监测方法、系统、设备及介质
CN121074562A (zh) * 2025-11-10 2025-12-05 安徽容知日新科技股份有限公司 一种托辊缺失检测模型训练方法与相关设备
CN121073905A (zh) * 2025-08-15 2025-12-05 宝纳生(深圳)科技有限公司 3d打印的异常检测方法、设备及存储介质
CN121075946A (zh) * 2025-11-05 2025-12-05 江苏道达智能科技有限公司 一种半导体瑕疵品的处理方法和系统
CN121169883A (zh) * 2025-09-19 2025-12-19 广东工业大学 基于纹理感知和异常检测的布匹表面瑕疵检测方法及装置
CN121213573A (zh) * 2025-11-28 2025-12-26 山东科技大学 一种基于任务对齐与动态类别加权的轨道缺陷检测方法

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119887788B (zh) * 2025-03-31 2025-07-15 中国海关科学技术研究中心 一种纺织品状态检测分析系统
CN119919430B (zh) * 2025-04-01 2025-07-18 宁德思客琦智能装备有限公司 一种用于智能制造的基于视觉的缺陷无监督识别分割方法

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109492556A (zh) * 2018-10-28 2019-03-19 北京化工大学 面向小样本残差学习的合成孔径雷达目标识别方法
CN110717493A (zh) * 2019-09-16 2020-01-21 浙江大学 一种基于深度学习的含堆叠字符的车牌识别方法
CN111002980A (zh) * 2019-12-10 2020-04-14 苏州智加科技有限公司 基于深度学习的道路障碍物轨迹预测方法和系统
CN115082416A (zh) * 2022-07-11 2022-09-20 心鉴智控(深圳)科技有限公司 镜片瑕疵检测方法、装置、设备及存储介质
US20220335966A1 (en) * 2021-04-16 2022-10-20 Ubtech North America Research And Development Center Corp System and method for multichannel speech detection
WO2023021208A1 (en) * 2021-08-20 2023-02-23 Five AI Limited Support tools for av testing

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109492556A (zh) * 2018-10-28 2019-03-19 北京化工大学 面向小样本残差学习的合成孔径雷达目标识别方法
CN110717493A (zh) * 2019-09-16 2020-01-21 浙江大学 一种基于深度学习的含堆叠字符的车牌识别方法
CN111002980A (zh) * 2019-12-10 2020-04-14 苏州智加科技有限公司 基于深度学习的道路障碍物轨迹预测方法和系统
US20220335966A1 (en) * 2021-04-16 2022-10-20 Ubtech North America Research And Development Center Corp System and method for multichannel speech detection
WO2023021208A1 (en) * 2021-08-20 2023-02-23 Five AI Limited Support tools for av testing
CN115082416A (zh) * 2022-07-11 2022-09-20 心鉴智控(深圳)科技有限公司 镜片瑕疵检测方法、装置、设备及存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
PANG FENGQIAN, ZHAO HONGFEI, KANG YINGYING: "Uncertainty estimation fused end-to-end video event detection algorithm", JOURNAL OF BEIJING UNIVERSITY OF AERONAUTICS AND ASTRONAUTICS, vol. 50, no. 12, 15 February 2023 (2023-02-15), CN , pages 3759 - 3770, XP093249332, ISSN: 1001-5965, DOI: 10.13700/j.bh.1001-5965.2022.0897 *

Cited By (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119991581A (zh) * 2025-01-09 2025-05-13 中国船舶集团有限公司第七〇九研究所 一种目标缺陷检测方法及装置
CN119444761A (zh) * 2025-01-13 2025-02-14 广州机智云物联网科技有限公司 基于生产环节瑕疵识别的质量检测模型训练方法
CN119478706B (zh) * 2025-01-14 2025-04-18 长春光华学院 一种基于卷积神经网络的植物叶片病害检测方法
CN119478706A (zh) * 2025-01-14 2025-02-18 长春光华学院 一种基于卷积神经网络的植物叶片病害检测方法
CN120032226A (zh) * 2025-01-24 2025-05-23 中国科学院宁波材料技术与工程研究所 复杂环境下的轻量级高精度动力电池极柱检测方法
CN119964155A (zh) * 2025-02-06 2025-05-09 清华大学 水生生物图像识别计数方法、装置、设备及介质
CN119648691A (zh) * 2025-02-12 2025-03-18 江苏嘉通能源有限公司 一种基于视觉图像的涤纶面料缺陷检测方法及系统
CN119810102A (zh) * 2025-03-13 2025-04-11 无锡日联科技股份有限公司 多模型融合的pcb通孔缺陷检测方法、装置、设备及介质
CN119850601A (zh) * 2025-03-18 2025-04-18 江西智成飞桨科技有限公司 光伏电池板ai瑕疵检测方法及设备
CN119992039A (zh) * 2025-04-15 2025-05-13 成都川油瑞飞科技有限责任公司 一种面向复杂电气仪表设备的状态指示灯定位方法及装置
CN120411618A (zh) * 2025-04-16 2025-08-01 内蒙古农业大学 一种基于复杂环境背景下的苹果叶部病害多分类方法
CN120411910A (zh) * 2025-04-21 2025-08-01 贵州装备制造职业学院 一种基于人工智能的车辆表面缺陷识别方法及系统
CN120125573A (zh) * 2025-04-27 2025-06-10 宝鸡市永盛泰钛业有限公司 基于多光谱图像融合的钛棒表面磨损形貌检测方法及系统
CN120133163A (zh) * 2025-05-15 2025-06-13 河南工学院 基于人工智能的自动化分拣方法及系统
CN120259287A (zh) * 2025-06-04 2025-07-04 华东交通大学 一种铁路轨道扣件缺陷检测方法、系统、装置及介质
CN120782712A (zh) * 2025-06-06 2025-10-14 浙江金汇华特种耐火材料有限公司 一种基于机器学习的耐火材料质量检测方法及系统
CN120635092A (zh) * 2025-08-14 2025-09-12 闽江学院 基于瑕疵判断模型的纺织服装瑕疵图案数据处理方法
CN121073905A (zh) * 2025-08-15 2025-12-05 宝纳生(深圳)科技有限公司 3d打印的异常检测方法、设备及存储介质
CN121169883A (zh) * 2025-09-19 2025-12-19 广东工业大学 基于纹理感知和异常检测的布匹表面瑕疵检测方法及装置
CN120953292A (zh) * 2025-10-20 2025-11-14 中国三峡建工(集团)有限公司 一种基于图像处理的高压电气设备表面缺陷识别方法
CN121033106A (zh) * 2025-10-28 2025-11-28 武汉科技大学 一种电力金具目标跟踪计数方法及系统
CN121052295A (zh) * 2025-11-05 2025-12-02 齐鲁工业大学(山东省科学院) 多工况刀具磨损监测模型的构建及其磨损监测方法、系统、设备及介质
CN121075946A (zh) * 2025-11-05 2025-12-05 江苏道达智能科技有限公司 一种半导体瑕疵品的处理方法和系统
CN121074562A (zh) * 2025-11-10 2025-12-05 安徽容知日新科技股份有限公司 一种托辊缺失检测模型训练方法与相关设备
CN121213573A (zh) * 2025-11-28 2025-12-26 山东科技大学 一种基于任务对齐与动态类别加权的轨道缺陷检测方法
CN121213573B (zh) * 2025-11-28 2026-02-24 山东科技大学 一种基于任务对齐与动态类别加权的轨道缺陷检测方法

Also Published As

Publication number Publication date
CN119151853A (zh) 2024-12-17

Similar Documents

Publication Publication Date Title
WO2024255919A1 (zh) 瑕疵检测方法及相关设备
CN113537106B (zh) 一种基于YOLOv5的鱼类摄食行为识别方法
CN110569747A (zh) 一种利用图像金字塔与Faster-RCNN快速统计大田水稻稻穗数的方法
Lv et al. A visual identification method for the apple growth forms in the orchard
CN112102224B (zh) 一种基于深度卷积神经网络的布匹疵点识别方法
Ji et al. Green apple detection method based on multidimensional feature extraction network model and transformer module
CN115099297A (zh) 一种基于改进型YOLO v5模型的大豆植株表型数据统计方法
CN105243356B (zh) 一种建立行人检测模型的方法及装置及行人检测方法
Li et al. Basic motion behaviour recognition of dairy cows based on skeleton and hybrid convolution algorithms
An et al. Fabric defect detection using deep learning: An Improved Faster R-approach
CN107480607A (zh) 一种智能录播系统中站立人脸检测定位的方法
CN106407908A (zh) 一种训练模型生成方法、人脸检测方法及装置
CN116448019A (zh) 建筑节能工程质量平面度智能检测装置及方法
CN116012789A (zh) 基于改进式目标检测模型的厨房行为实时监测方法及系统
Zhao et al. Development of an enhanced hybrid attention YOLOv8s small object detection method for phenotypic analysis of root nodules
Xu et al. Improved YOLOv8 garment sewing defect detection method based on attention mechanism
Bai et al. Video target detection of East Asian migratory locust based on the MOG2-YOLOv4 network
Shen et al. Multimodal image fusion based on point-wise mutual information
CN108073873A (zh) 基于高清智能摄像机的人脸检测与识别系统
CN119672426B (zh) 基于智能学习的rfid芯片图像异常检测方法及系统
CN119477791A (zh) 一种羊毛衫缺陷检测方法
CN118658042A (zh) 一种基于改进YOLOv8n网络的茶叶采摘时间预测方法
HK40114851A (zh) 瑕疵检测方法及相关设备
CN107403192A (zh) 一种基于多分类器的快速目标检测方法及系统
Su et al. Improvement of the yolov5 model in the optimization of the brown spot disease recognition algorithm of kidney bean. Plants 2023; 12: 3765

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24822864

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE