CN114842273B - Evaluation method, evaluation device and training method of PCB defect detection model - Google Patents
Evaluation method, evaluation device and training method of PCB defect detection model Download PDFInfo
- Publication number
- CN114842273B CN114842273B CN202210703079.9A CN202210703079A CN114842273B CN 114842273 B CN114842273 B CN 114842273B CN 202210703079 A CN202210703079 A CN 202210703079A CN 114842273 B CN114842273 B CN 114842273B
- Authority
- CN
- China
- Prior art keywords
- defect
- detection model
- defect detection
- test images
- defects
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000007547 defect Effects 0.000 title claims abstract description 367
- 238000001514 detection method Methods 0.000 title claims abstract description 129
- 238000011156 evaluation Methods 0.000 title claims abstract description 81
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000012549 training Methods 0.000 title claims abstract description 20
- 238000012360 testing method Methods 0.000 claims abstract description 145
- 238000013135 deep learning Methods 0.000 claims description 4
- 238000012854 evaluation process Methods 0.000 claims description 4
- 230000007246 mechanism Effects 0.000 claims description 4
- 230000001960 triggered effect Effects 0.000 claims description 3
- 238000007689 inspection Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 6
- 230000009471 action Effects 0.000 description 3
- 230000002950 deficient Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 238000013210 evaluation model Methods 0.000 description 1
- 231100000225 lethality Toxicity 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000013441 quality evaluation Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30141—Printed circuit board [PCB]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
The invention discloses an evaluation method, an evaluation device and a training method of a PCB defect detection model, wherein the evaluation method comprises the following steps: pre-establishing a test image set which comprises a plurality of test images with defect class labels; inputting a plurality of test images into a defect detection model to be evaluated to obtain defect prediction results which are in one-to-one correspondence with the test images and comprise defect types and corresponding probability values; classifying the defect prediction result according to the defect type in the obtained defect prediction result; outputting and displaying the test images predicted to be the same defect type according to the classification result, wherein the defect type labels of the test images and the probability values in the corresponding defect prediction results are configured to be viewable; and evaluating the capability of the defect detection model for identifying various defect types according to the defect type labels and classification results of the test images, and comparing the probability values corresponding to the test images under the same classification to evaluate the convergence of the defect detection model for identifying the defects.
Description
Technical Field
The invention relates to the field of model quality evaluation, in particular to an evaluation method, an evaluation device and a training method of a PCB defect detection model.
Background
In a typical AI training process, defective images are input into an AI model for identification and classification, the identified images are assigned a score that indicates their likelihood of being defective or non-defective (false positive defects), the image is considered to have real defects if the score is above a preset AI threshold, and false positive defects if the score is below the AI threshold.
The reasonable setting relationship of the AI threshold value is balanced between underkill and overkill, if the AI threshold value setting value is low, the actual false alarm defect can be wrongly identified as a real defect, and the over kill is caused by overlarge lethality; if the AI threshold setting is too high, the actual real defect may be erroneously identified as a false positive defect, resulting in underkill.
It can be said that the setting of the AI threshold has a close relationship with the accuracy of the model output result.
The above background disclosure is only for the purpose of assisting understanding of the inventive concept and technical solutions of the present invention, and does not necessarily belong to the prior art of the present patent application nor give technical teaching; the above background should not be used to assess the novelty and inventive aspects of the present application in the absence of express evidence that the above disclosure is published prior to the filing date of the present patent application.
Disclosure of Invention
The invention aims to provide an evaluation method, an evaluation device and a training method of a PCB defect detection model, which can effectively evaluate the recognition capability and the recognition convergence of the model.
In order to achieve the purpose, the technical scheme adopted by the invention is as follows:
an evaluation method of a PCB defect detection model is used for evaluating the defect prediction capability of a pre-trained defect detection model on a PCB image, wherein the defect prediction result of the defect detection model on the PCB image comprises defect types and corresponding probability values, and the evaluation method comprises the following steps:
pre-establishing a test image set comprising a plurality of test images, each test image having a defect category label;
inputting a plurality of test images in the test image set into a defect detection model to be evaluated to obtain defect prediction results corresponding to the test images one to one;
classifying the defect prediction result according to the defect type in the obtained defect prediction result;
outputting and displaying the test images predicted to be the same defect type according to the classification result, wherein the probability value of the defect type label of each test image and the corresponding defect prediction result is configured to be viewable;
and evaluating the capability of the defect detection model for identifying various defect types according to the defect category labels and classification results of the test images, and comparing the probability values corresponding to the test images under the same classification to evaluate the convergence of the defect detection model for identifying the type of defects.
Further, calculating the classification accuracy according to the defect category label and the classification result of the test image, wherein if the classification accuracy is lower than a preset accuracy threshold, the defect detection model has unqualified capability of identifying the defects; or,
and if the difference value between the maximum value and the minimum value of the probability values corresponding to the test images in the same category is greater than a preset difference threshold value, or if the variance of the probability values corresponding to the test images in the same category is greater than a preset variance threshold value, the defect detection model identifies that the convergence of the defects is unqualified.
Further, the evaluation method of the PCB defect detection model further comprises the following steps: and if the evaluation result is unqualified, retraining the defect detection module, and focusing the learning attention on the type of defects with unqualified recognition capability or convergence by the defect detection module.
Further, under the condition that the convergence of the defect detection model identifying the type of defect is qualified, according to the probability value corresponding to the classified test image and a preset rule, determining a demarcation value for defining a real defect and a false alarm defect.
Further, the demarcation score for defining true and false positive defects is determined by:
searching the minimum value of the probability values corresponding to the classified test images, and calculating by using the minimum value and a preset difference threshold value to obtain the division value; or,
calculating the average value of the probability values corresponding to the classified test images, and calculating the average value and a preset difference threshold value to obtain the demarcation value; or,
sorting the average values of the probability values corresponding to the classified test images, excluding the first probability values and the second probability values, calculating the average value of the residual probability values, and calculating by using the average value of the residual probability values and a preset difference threshold value to obtain the demarcation value;
wherein the preset difference threshold is a positive number, a negative number or zero.
Further, the manner in which the probability values in the defect category labels and the corresponding defect prediction results of the respective test images are configured to be viewable is as follows:
the probability value in the defect prediction result is displayed in a corresponding local area on the test image;
the defect type label of the test image is configured to be displayed in a popup window which appears after being triggered, and the triggering operation of the popup window comprises one or more of clicking the corresponding test image, multi-clicking the corresponding test image, right-clicking the corresponding test image, and stopping a cursor on the corresponding test image.
The invention discloses an evaluation device of a PCB defect detection model, which comprises the following modules:
a test sample module configured to create a test image set comprising a plurality of test images, each test image having a defect category label;
the prediction module is configured to input a plurality of test images in the test image set into a defect detection model to be evaluated to obtain defect prediction results corresponding to the test images one by one;
a classification module configured to classify the defect prediction result according to a defect type in the defect prediction result of the prediction module;
the display module is configured to output and display the test images predicted to be the same defect type according to the classification result of the classification module, and can display the defect type labels of the test images and the probability values in the corresponding defect prediction results;
the identification capability evaluation module is configured to evaluate the capability of the defect detection model for identifying various defect types according to the defect category labels and the classification results of the test images;
and the convergence evaluation module is configured to compare the probability values corresponding to the test images under the same classification so as to evaluate the convergence of the defect detection model for identifying the type of defects.
Further, the identification capability evaluation module calculates a classification accuracy according to the defect category label and the classification result of the test image, and if the classification accuracy is lower than a preset accuracy threshold, the evaluation result of the identification capability evaluation module is that the capability of the defect detection model for identifying the type of defect is unqualified;
the convergence evaluation module judges whether the difference value between the maximum value and the minimum value of the probability values corresponding to the test images in the same category is greater than a preset difference threshold value or not, or judges whether the variance of the probability values corresponding to the test images in the same category is greater than a preset variance threshold value or not, if yes, the evaluation result of the convergence evaluation module is that the convergence of the defect detection model for identifying the defects is unqualified.
Further, the evaluation device for the PCB defect detection model further includes a demarcation value determination module configured to determine a demarcation value for defining a real defect and a false positive defect according to a probability value corresponding to the classified test image and a preset rule under the condition that the convergence of the type of defect identified by the defect detection model is qualified according to the evaluation result of the convergence evaluation module.
According to another aspect of the present invention, a method for training a defect inspection model of a PCB is provided, wherein the defect inspection model which is currently trained is evaluated by using the above evaluation method, and if the capability of identifying all types of defects of the defect inspection model and the convergence of identifying all types of defects are evaluated to be qualified, the defect inspection model finishes training; and if not, retraining the defect detection model, and focusing the learning attention on the type of defects with unqualified recognition capability or convergence by the defect detection model.
Further, if the ability of the defect detection model for identifying the defects is evaluated to be unqualified, updating the learning sample library of the defects, and retraining the defect detection model by using the updated learning sample library;
and if the convergence of the defect detection model for identifying the type of defects is evaluated to be unqualified, updating or adjusting the scoring mechanism of the defect detection model.
Further, retraining the defect detection model comprises: and determining a test image with a defect type label inconsistent with the classification result, and performing feature learning on the test image with the defect type label inconsistent with the classification result by the defect detection model by using a deep learning algorithm to obtain an optimized model.
According to still another aspect of the present invention, there is provided a PCB defect detecting method, including the steps of:
inputting a PCB image to be detected into a defect detection model finishing training, wherein the defect detection model finishes evaluation by using the evaluation method;
the defect detection model outputs a defect prediction result of the PCB image to be detected, and if the probability value in the defect prediction result is higher than the demarcation value for defining real defects and false-alarm defects determined in the evaluation process, the detection result output by the defect detection model is the types of the real defects and the defects of the defect prediction result; otherwise, the detection result output by the defect detection model is a false alarm defect.
The technical scheme provided by the invention has the following beneficial effects:
a. classifying according to the prediction result of the test image, screening out the defect type of the evaluation result OK and the defect type of the evaluation result unqualified, and only retraining and learning the defect type of the evaluation result unqualified, so as to improve the efficiency of completing the training of the model;
b. the capability of the model is comprehensively evaluated from two aspects of defect identification capability and convergence, and the model is ensured to be subjected to high-quality and high-requirement evaluation;
c. an objective reference basis is provided for determining an AI score threshold value of an overkill and underkill boundary, and the identification accuracy of the model is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic flow chart of an evaluation method of a PCB defect inspection model according to an exemplary embodiment of the present invention;
FIG. 2 is a schematic diagram of a first type of interface for classifying a defect prediction result according to an exemplary embodiment of the present invention;
fig. 3 is a schematic diagram of a second interface for classifying a defect prediction result according to an exemplary embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in other sequences than those illustrated or described herein. Moreover, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, apparatus, article, or device that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or device.
The present invention proposes an evaluation tool in which software displays a defect image assigned with an AI score (predicted probability value), as shown in fig. 2 and 3, in which the result can be simulated by adjusting an AI threshold. This software can not only obtain statistical data about under-and over-kills, but also look at specific misrecognized images and create a new set of training data that is specific to the problematic (misrecognized) image. Such a set may be used to retrain an appropriately adapted AI model for more accurate recognition.
In an embodiment of the present invention, an evaluation method of a PCB defect detection model is provided, which is used for evaluating a defect prediction capability of a pre-trained defect detection model on a PCB image, where a defect prediction result of the defect detection model on the PCB image includes a defect type and a corresponding probability value, and as shown in fig. 1, the evaluation method includes:
pre-establishing a test image set comprising a plurality of test images, each test image having a defect category label;
inputting a plurality of test images in the test image set into a defect detection model to be evaluated to obtain defect prediction results (including defect types and corresponding probability values) corresponding to the test images one to one;
classifying the defect prediction result according to the defect type in the obtained defect prediction result;
outputting and displaying the test images predicted to be the same defect type according to the classification result, wherein the probability value of the defect type label of each test image and the corresponding defect prediction result is configured to be viewable;
and evaluating the capability of the defect detection model for identifying various defect types according to the defect type labels and classification results of the test images, and comparing the probability values corresponding to the test images under the same classification to evaluate the convergence of the defect detection model for identifying the type of defects.
Specifically, for example, the test image set includes 1000 test images labeled as short and 800 test images labeled as open, and the 1800 test images are input into the defect detection model to obtain 1800 defect prediction results, for example, the defect prediction results corresponding to 990 test images labeled as short are short defect types and respective probability values, 6 of the remaining 10 are identified as open defect types, and 4 are identified as other defect types; for example, the defect prediction results corresponding to 750 test images labeled as open circuit are open circuit defect types and their respective probability values, 15 of the remaining 50 test images are identified as short circuit defect types, and 35 test images are identified as other defect types.
Then, for example, according to the short-circuit defect type in the preset result, the defect prediction result is classified to obtain 990 test images labeled as short-circuit and 15 test images labeled as open-circuit, and the display interfaces are shown in fig. 2 and fig. 3: the probability values in the defect prediction results are displayed in the corresponding local area (the upper left corner area in fig. 2 and 3) on the test image; the defect category label of the test image is configured to be displayed in a popup window (shown in fig. 2) which appears after being triggered, and the triggering operation of the popup window comprises one or more of clicking the corresponding test image, multi-clicking the corresponding test image, right-clicking the corresponding test image, and staying a cursor on the corresponding test image.
The manner in which the probability values are visually displayed on the test image enables the overall situation of the predicted probability values to be visually seen to determine a suitable demarcation score for defining real defects and false positive defects, and the specific manner of determination is explained in detail later on after the evaluation convergence is explained in detail.
In this embodiment, the evaluated indicators include the ability to identify defects and the convergence of identifying defects:
first, describing the ability to identify defects, the classification accuracy is calculated according to the defect classification labels and classification results of the test images, for example, as follows: there are 10 test images with short circuit label and 15 test images with open circuit label, which are recognized as errors, so the calculation formula of the classification accuracy can be: 100% - (10 + 15)/(990 + 15) ≈ 97.5%. If the accuracy threshold value is 98 percent, the capability of the secondary defect detection model for identifying the short-circuit type defects is unqualified, and if the accuracy threshold value is 97 percent, the capability of the secondary defect detection model for identifying the short-circuit type defects is qualified.
As for the convergence degree of the defect identification, for example, in the defect prediction results of 990 test images correctly classified as short circuits, the maximum value and the minimum value of the probability values are 0.9 and 0.5, although the defect type is correctly identified as short circuit, the overall estimated probability values are too different, or the variance of the 990 probability values is too large, which indicates that the overall estimated probability value distribution is more discrete, and the convergence degree of the defect detection model for identifying the defect is not good.
The evaluation of the PCB defect detection model is to optimize the model:
in one embodiment, the model is considered to pass the evaluation under the classification (defect type) only under the condition that the evaluation results of the two models are qualified. And if the evaluation result is unqualified, retraining the defect detection module, and focusing the learning attention on the type of defects with unqualified recognition capability or convergence by the defect detection module. This benefit is enormous: for example, if the capability and the convergence of identifying the defect of the short-circuit type are both up to standard, and the capability and/or the convergence of identifying the defect of the open-circuit type are not up to standard, the strategy of optimizing the model is to focus the learning attention on the image characteristics of identifying the open-circuit defect, and specifically, the following method may be adopted:
if the capacity of the evaluation model for identifying the open circuit defect is not qualified, updating the learning sample library for identifying the open circuit defect, and retraining the defect detection model by using the updated learning sample library; or extracting the test image with the defect type label inconsistent with the classification result in the evaluation process, and performing feature learning on the test image with the defect type label inconsistent with the classification result by the defect detection model by using a deep learning algorithm to obtain an optimized model. I.e. to look at a specific misrecognized image and create a new set of training data specific to the image in question (misrecognized). Such a set may be used to retrain an appropriately adapted AI model for more accurate recognition.
If the convergence of the defects identified by the defect detection model is not qualified by evaluating, updating or adjusting a scoring mechanism of the defect detection model, namely, adjusting the distribution proportion of the model parameters or the weight values, so that the predicted dispersion of the probability value is reduced.
Under the condition that the convergence of the defect detection model for identifying the type of defect is qualified, the boundary score for defining the real defect and the false alarm defect can be determined according to the probability value corresponding to the classified test image and a preset rule, and the following method can be specifically adopted:
searching the minimum value of the probability values corresponding to the classified test images, and calculating by using the minimum value and a preset difference threshold value to obtain the demarcation value; or,
calculating the average value of the probability values corresponding to the classified test images, and calculating the average value and a preset difference threshold value to obtain the demarcation value; or,
sorting the average values of the probability values corresponding to the classified test images, excluding the first probability values and the second probability values, calculating the average value of the remaining probability values, and calculating the operation of the average value of the remaining probability values and a preset difference threshold value to obtain the demarcation value;
wherein the preset difference threshold is a positive number, a negative number or zero.
The invention discloses an evaluation device of a PCB defect detection model, which comprises the following modules:
a test sample module configured to create a test image set comprising a plurality of test images, each test image having a defect category label;
the prediction module is configured to input a plurality of test images in the test image set into a defect detection model to be evaluated to obtain defect prediction results corresponding to the test images one by one;
a classification module configured to classify the defect prediction result according to a defect type in the defect prediction result of the prediction module;
the display module is configured to output and display the test images predicted to be the same defect type according to the classification result of the classification module, and can display the defect type labels of the test images and the probability values in the corresponding defect prediction results;
the identification capability evaluation module is configured to evaluate the capability of the defect detection model for identifying various defect types according to the defect category labels and the classification results of the test images;
and the convergence evaluation module is configured to compare the probability values corresponding to the test images under the same classification so as to evaluate the convergence of the defect detection model for identifying the type of defects.
Further, the identification capability evaluation module calculates a classification accuracy according to the defect category label and the classification result of the test image, and if the classification accuracy is lower than a preset accuracy threshold, the evaluation result of the identification capability evaluation module is that the capability of the defect detection model for identifying the type of defect is unqualified;
the convergence evaluation module judges whether the difference value between the maximum value and the minimum value of the probability values corresponding to the test images in the same category is greater than a preset difference threshold value or not, or judges whether the variance of the probability values corresponding to the test images in the same category is greater than a preset variance threshold value or not, if yes, the evaluation result of the convergence evaluation module is that the convergence of the defect detection model for identifying the defects is unqualified.
Further, the evaluation device for the PCB defect detection model further includes a demarcation value determination module configured to determine a demarcation value for defining a real defect and a false positive defect according to a probability value corresponding to the classified test image and a preset rule under the condition that the convergence of the type of defect identified by the defect detection model is qualified according to the evaluation result of the convergence evaluation module.
The embodiment of the evaluation device of the PCB defect detection model belongs to the same concept as the embodiment of the evaluation method, and the whole content of the embodiment of the evaluation method is incorporated into the embodiment of the evaluation device by reference.
According to another aspect of the present invention, a method for training a defect inspection model of a PCB is provided, wherein the defect inspection model which is currently trained is evaluated by using the above evaluation method, and if the capability of identifying all types of defects of the defect inspection model and the convergence of identifying all types of defects are evaluated to be qualified, the defect inspection model finishes training; and if not, retraining the defect detection model, and focusing the learning attention on the type of defects with unqualified recognition capability or convergence by the defect detection model.
If the capacity of the defect detection model for identifying the defects is evaluated to be unqualified, updating the learning sample library of the defects, and retraining the defect detection model by using the updated learning sample library; or determining a test image with the defect type label inconsistent with the classification result, and performing feature learning on the test image with the defect type label inconsistent with the classification result by the defect detection model by using a deep learning algorithm to obtain an optimized model.
And if the convergence of the defect detection model for identifying the type of defects is evaluated to be unqualified, updating or adjusting the scoring mechanism of the defect detection model.
According to still another aspect of the present invention, there is provided a PCB defect detecting method, including the steps of:
inputting a PCB image to be detected into a defect detection model finishing training, wherein the defect detection model finishes evaluation by using the evaluation method;
the defect detection model outputs a defect prediction result of the PCB image to be detected, and if the probability value in the defect prediction result is higher than the demarcation value for defining real defects and false-alarm defects determined in the evaluation process, the detection result output by the defect detection model is the types of the real defects and the defects of the defect prediction result; otherwise, the detection result output by the defect detection model is a false alarm defect. For example, comparing fig. 2 and 3, a lower score (probability value) in fig. 2 indicates that the image in fig. 2 has a greater likelihood of false positive defects, and a higher score (probability value) in fig. 3 indicates that the image in fig. 3 has a greater likelihood of true defects.
It should be noted that, in this document, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising one of 8230; \8230;" 8230; "does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.
The foregoing is directed to embodiments of the present application and it is noted that numerous modifications and adaptations may be made by those skilled in the art without departing from the principles of the present application and are intended to be within the scope of the present application.
Claims (13)
1. An evaluation method of a PCB defect detection model is used for evaluating the defect prediction capability of a pre-trained defect detection model on a PCB image, and the defect prediction result of the defect detection model on the PCB image comprises a defect type and a corresponding probability value, and is characterized in that the evaluation method comprises the following steps:
pre-establishing a test image set comprising a plurality of test images, each test image having a defect category label;
inputting a plurality of test images in the test image set into a defect detection model to be evaluated to obtain defect prediction results corresponding to the test images one to one;
classifying the defect prediction result according to the defect type in the obtained defect prediction result;
outputting and displaying the test images predicted to be the same defect type according to the classification result, wherein the probability value of the defect type label of each test image and the corresponding defect prediction result is configured to be viewable;
and evaluating the capability of the defect detection model for identifying various defect types according to the defect type labels and classification results of the test images, and comparing the probability values corresponding to the test images under the same classification to evaluate the convergence of the defect detection model for identifying the type of defects.
2. The evaluation method of the PCB defect detection model according to claim 1, wherein a classification accuracy is calculated according to a defect category label and a classification result of a test image, and if the classification accuracy is lower than a preset accuracy threshold, the defect detection model has an unqualified capability of identifying the type of defect; or,
and if the difference value between the maximum value and the minimum value of the probability values corresponding to the test images in the same category is greater than a preset difference threshold value, or if the variance of the probability values corresponding to the test images in the same category is greater than a preset variance threshold value, the defect detection model identifies that the convergence of the defects is unqualified.
3. The method for evaluating the PCB defect detection model of claim 1, further comprising: and if the evaluation result is unqualified, retraining the defect detection model, and focusing the learning attention on the type of defects with unqualified recognition capability or convergence by the defect detection model.
4. The method for evaluating the PCB defect detection model of claim 1, wherein under the condition that the convergence of the defect detection model for identifying the type of defect is qualified, the demarcation value for defining the real defect and the false alarm defect is determined according to the probability value corresponding to the classified test image and a preset rule.
5. The method for evaluating the PCB defect detection model of claim 4, wherein the demarcation score for defining real defects and false positive defects is determined by:
searching the minimum value of the probability values corresponding to the classified test images, and calculating by using the minimum value and a preset difference threshold value to obtain the demarcation value; or,
calculating the average value of the probability values corresponding to the classified test images, and calculating the average value and a preset difference threshold value to obtain the demarcation value; or,
sorting the average values of the probability values corresponding to the classified test images, excluding the first probability values and the second probability values, calculating the average value of the remaining probability values, and calculating the operation of the average value of the remaining probability values and a preset difference threshold value to obtain the demarcation value;
wherein the preset difference threshold is a positive number, a negative number or zero.
6. The method for evaluating the PCB defect detection model of claim 1, wherein the probability values in the defect category labels and the corresponding defect prediction results of the test images are configured to be viewable in the following ways:
the probability value in the defect prediction result is displayed in a corresponding local area on the test image;
the defect type label of the test image is configured to be displayed in a popup window which appears after being triggered, and the triggering operation of the popup window comprises one or more of clicking the corresponding test image, multi-clicking the corresponding test image, right-clicking the corresponding test image, and stopping a cursor on the corresponding test image.
7. An evaluation device of a PCB defect detection model is characterized by comprising the following modules:
a test sample module configured to create a test image set comprising a plurality of test images, each test image having a defect category label;
the prediction module is configured to input a plurality of test images in the test image set into a defect detection model to be evaluated to obtain defect prediction results corresponding to the test images one by one;
a classification module configured to classify the defect prediction result according to a defect type in the defect prediction result of the prediction module;
the display module is configured to output and display the test images predicted to be the same defect type according to the classification result of the classification module, and can display the defect type labels of the test images and the probability values in the corresponding defect prediction results;
the identification capability evaluation module is configured to evaluate the capability of the defect detection model for identifying various defect types according to the defect category labels and the classification results of the test images;
and the convergence evaluation module is configured to compare the probability values corresponding to the test images under the same classification so as to evaluate the convergence of the defect detection model for identifying the type of defects.
8. The evaluation device of the PCB defect detection model according to claim 7, wherein the recognition capability evaluation module calculates a classification accuracy according to the defect type label and the classification result of the test image, and if the classification accuracy is lower than a preset accuracy threshold, the evaluation result of the recognition capability evaluation module is that the capability of the defect detection model for recognizing the type of defect is not qualified;
the convergence evaluation module judges whether the difference value between the maximum value and the minimum value of the probability values corresponding to the test images in the same category is greater than a preset difference threshold value or not, or judges whether the variance of the probability values corresponding to the test images in the same category is greater than a preset variance threshold value or not, if yes, the evaluation result of the convergence evaluation module is that the convergence of the defect detection model for identifying the defects is unqualified.
9. The evaluation device of the PCB defect detection model of claim 7, further comprising a demarcation score determining module configured to determine a demarcation score for defining real defects and false-positive defects according to a probability value corresponding to a test image under the classification and a preset rule, when the convergence of the defect detection model identified the type of defects is qualified according to the evaluation result of the convergence evaluating module.
10. A training method of a PCB defect detection model is characterized in that the currently trained defect detection model is evaluated by the evaluation method of any one of claims 1 to 6, and if the capability of identifying all types of defects of the defect detection model and the convergence of identifying all types of defects are evaluated to be qualified, the training of the defect detection model is finished; and if not, retraining the defect detection model, and focusing the learning attention on the type of defects with unqualified recognition capability or convergence by the defect detection model.
11. The training method according to claim 10, wherein if the ability of the defect detection model for identifying the type of defect is evaluated to be unqualified, the learning sample library of the type of defect is updated, and the defect detection model is retrained by using the updated learning sample library;
and if the convergence of the defect detection model for identifying the type of defects is evaluated to be unqualified, updating or adjusting the scoring mechanism of the defect detection model.
12. The training method of claim 10, wherein retraining the defect detection model comprises: and determining a test image with the defect type label inconsistent with the classification result, and performing feature learning on the test image with the defect type label inconsistent with the classification result by the defect detection model by using a deep learning algorithm to obtain an optimized model.
13. A PCB defect detection method is characterized by comprising the following steps:
inputting a PCB image to be detected into a defect detection model ending training, wherein the defect detection model completes evaluation by using the evaluation method according to claim 4 or 5;
the defect detection model outputs a defect prediction result of the PCB image to be detected, and if the probability value in the defect prediction result is higher than the demarcation value for defining real defects and false-alarm defects determined in the evaluation process, the detection result output by the defect detection model is the types of the real defects and the defects of the defect prediction result; otherwise, the detection result output by the defect detection model is a false alarm defect.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210703079.9A CN114842273B (en) | 2022-06-21 | 2022-06-21 | Evaluation method, evaluation device and training method of PCB defect detection model |
| TW111147747A TWI834426B (en) | 2022-06-21 | 2022-12-13 | Evaluation method, evaluation device and training method for pcb defect detection model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210703079.9A CN114842273B (en) | 2022-06-21 | 2022-06-21 | Evaluation method, evaluation device and training method of PCB defect detection model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114842273A CN114842273A (en) | 2022-08-02 |
| CN114842273B true CN114842273B (en) | 2022-10-14 |
Family
ID=82573802
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210703079.9A Active CN114842273B (en) | 2022-06-21 | 2022-06-21 | Evaluation method, evaluation device and training method of PCB defect detection model |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN114842273B (en) |
| TW (1) | TWI834426B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116563278B (en) * | 2023-07-06 | 2023-11-17 | 宁德时代新能源科技股份有限公司 | Detection result display method, device, computer equipment and storage medium |
| CN117078674B (en) * | 2023-10-14 | 2024-01-05 | 中电鹏程智能装备有限公司 | Intelligent PCBA appearance defect detection method and detection system |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021109011A1 (en) * | 2019-12-04 | 2021-06-10 | 电子科技大学 | Intelligent capacitor internal defect detection method based on ultrasound image |
| KR20210157253A (en) * | 2020-06-19 | 2021-12-28 | 아주대학교산학협력단 | Apparatus and method for learning quality estimation model of welding, and estimation apparatus using the model |
| CN112381175A (en) * | 2020-12-05 | 2021-02-19 | 中国人民解放军32181部队 | Circuit board identification and analysis method based on image processing |
| CN112907562A (en) * | 2021-03-16 | 2021-06-04 | 中科海拓(无锡)科技有限公司 | MobileNet-based SMT defect classification algorithm |
| CN113450312A (en) * | 2021-06-03 | 2021-09-28 | 南通大学 | System and method for detecting flash defect based on ResNet64 network |
| CN113255590A (en) * | 2021-06-25 | 2021-08-13 | 众芯汉创（北京）科技有限公司 | Defect detection model training method, defect detection method, device and system |
-
2022
- 2022-06-21 CN CN202210703079.9A patent/CN114842273B/en active Active
- 2022-12-13 TW TW111147747A patent/TWI834426B/en active
Also Published As
| Publication number | Publication date |
|---|---|
| TWI834426B (en) | 2024-03-01 |
| TW202400991A (en) | 2024-01-01 |
| CN114842273A (en) | 2022-08-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114842273B (en) | Evaluation method, evaluation device and training method of PCB defect detection model | |
| EP3502966A1 (en) | Data generation apparatus, data generation method, and data generation program | |
| CN110991657A (en) | Abnormal sample detection method based on machine learning | |
| CN109946303A (en) | Check device and method | |
| JP2001230289A (en) | Fault analyzing method and system | |
| CN111242899B (en) | Image-based flaw detection method and computer-readable storage medium | |
| CN108304567B (en) | Method and system for identifying working condition mode and classifying data of high-voltage transformer | |
| CN109726730A (en) | Automatic optics inspection image classification method, system and computer-readable medium | |
| TWI826207B (en) | Pcb defect detection system and pcb defect detection method | |
| CN110794360A (en) | Method and system for predicting fault of intelligent electric energy meter based on machine learning | |
| TWI747686B (en) | A defect detection method and a defect detection device | |
| CN103824092A (en) | Image classification method for monitoring state of electric transmission and transformation equipment on line | |
| CN113111903A (en) | Intelligent production line monitoring system and monitoring method | |
| CN112529109A (en) | Unsupervised multi-model-based anomaly detection method and system | |
| TWI707299B (en) | Optical inspection secondary image classification method | |
| CN110751170A (en) | Panel quality detection method, system, terminal device and computer readable medium | |
| CN110335242A (en) | A kind of tire X-ray defect detection method based on multi-model fusion | |
| CN111191670A (en) | Neural network-based classification device and classification method | |
| CN111737993A (en) | Method for extracting health state of equipment from fault defect text of power distribution network equipment | |
| CN116189909A (en) | Clinical medicine discriminating method and system based on lifting algorithm | |
| CN110057588A (en) | The bearing incipient fault detection and diagnostic method and system merged based on singular value with characteristics of graph theory | |
| JP2020077158A (en) | Image processing device and image processing method | |
| CN114596296A (en) | High-sensitivity hot-rolled steel coil end surface defect identification system and method | |
| CN110728310B (en) | Target detection model fusion method and fusion system based on super-parameter optimization | |
| KR20210106038A (en) | Automatic apparatus for inspecting separator of secondary battery using ai technology based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |