CN113706477A - Defect type identification method, device, equipment and medium - Google Patents

Defect type identification method, device, equipment and medium Download PDF

Info

Publication number
CN113706477A
CN113706477A CN202110912056.4A CN202110912056A CN113706477A CN 113706477 A CN113706477 A CN 113706477A CN 202110912056 A CN202110912056 A CN 202110912056A CN 113706477 A CN113706477 A CN 113706477A
Authority
CN
China
Prior art keywords
defect
defect type
sample image
image
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110912056.4A
Other languages
Chinese (zh)
Other versions
CN113706477B (en
Inventor
陈晓炬
杜松
王邦军
杨怀宇
李磊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing Xurui Software Technology Co ltd
Original Assignee
Nanjing Xurui Software Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing Xurui Software Technology Co ltd filed Critical Nanjing Xurui Software Technology Co ltd
Priority to CN202110912056.4A priority Critical patent/CN113706477B/en
Publication of CN113706477A publication Critical patent/CN113706477A/en
Application granted granted Critical
Publication of CN113706477B publication Critical patent/CN113706477B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/0002Inspection of images, e.g. flaw detection
    • G06T7/0004Industrial image inspection
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/21Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20081Training; Learning
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20084Artificial neural networks [ANN]
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02PCLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30Computing systems specially adapted for manufacturing

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Biomedical Technology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Quality & Reliability (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Image Analysis (AREA)

Abstract

The embodiment of the application provides a defect type identification method, a defect type identification device, defect type identification equipment and a defect type identification medium. The method comprises the following steps: acquiring an image to be processed; inputting an image to be processed into a first network of a defect category identification model trained in advance, and determining a feature vector of the image to be processed, wherein the feature vector comprises a plurality of local features and/or the whole feature for representing the image to be processed; and inputting the feature vectors into a second network of the defect type model, and determining the defect type of the image to be processed, wherein the defect type comprises surface defects and internal defects. According to the defect classification method and device, the newly added defect classification and the old defect classification in the image to be processed can be distinguished conveniently, and therefore the accuracy of defect classification identification is improved.

Description

Defect type identification method, device, equipment and medium
Technical Field
The application belongs to the field of industrial vision, and particularly relates to a defect type identification method, device, equipment and medium.
Background
In industry, for example, in a production line working environment, new defect categories are continuously generated, because there may be a case where the similarity between the new defect categories and the old defect categories is large, and if the edge device is used, the stored sample images of the old defect categories are limited, it is easy to distinguish the new defect categories from the old defect categories when identifying the defect categories of the images, and thus the accuracy of identifying the defect categories cannot be improved.
In order to solve the above problems, the prior art generally includes two methods, one is a fine tuning method, which has a situation that there is no training for the old defect class sample, so that the trained model is catastrophically forgotten when identifying the old defect class. The other method is a joint training method, namely, a model is trained by using all samples containing the newly added defect type and the old defect type, but the phenomenon of unbalanced sample number exists between the newly added defect type sample and the old defect type sample, so that the method has limitation in identifying the old defect type, the training time is long, the training cost is high, and the method is not beneficial to being put into production. Therefore, the prior art still cannot improve the accuracy of defect type identification.
Disclosure of Invention
The embodiment of the application provides a defect type identification method, a defect type identification device, defect type identification equipment and a defect type identification medium, and the accuracy of defect type identification is improved.
In a first aspect, an embodiment of the present application provides a defect type identification method, where the method includes: in some embodiments of the first aspect, an image to be processed is acquired; inputting an image to be processed into a first network of a defect category identification model trained in advance, and determining a feature vector of the image to be processed, wherein the feature vector comprises a plurality of local features and/or the whole feature for representing the image to be processed; and inputting the feature vectors into a second network of the defect type model, and determining the defect type of the image to be processed, wherein the defect type comprises surface defects and internal defects.
In some embodiments of the first aspect, the first network comprises an adaptive aggregation network and the second network comprises an offset correction network.
In some embodiments of the first aspect, prior to inputting the image to be processed to the first network of pre-trained defect class identification models, the method further comprises: acquiring a training sample set, wherein the training sample set comprises a plurality of sample image groups, and each sample image group comprises a sample image and a corresponding label defect type thereof; and training a preset defect type identification model by using the sample image group in the training sample set to obtain the defect type identification model. In some embodiments of the first aspect, the training sample set includes a first training sample set and a second training sample set, where the first training sample set includes a plurality of first label defect class groups corresponding to a plurality of preset ratios, each first label defect class group includes a plurality of first sample image groups, and each first sample image group includes a first sample image and its corresponding first label defect class;
the second training sample set comprises a plurality of second label defect class groups, each second label defect class group comprises a plurality of second sample image groups with the same preset proportion, and each second sample image group comprises a second sample image and a corresponding second label defect class thereof.
In some embodiments of the first aspect, the first group of sample images is input to a first network in a preset defect class identification model, and a reference feature vector of each first sample image is determined, wherein the reference feature vector comprises a plurality of features for characterizing the entirety and/or a part of the first sample image; inputting the reference feature vector and the second sample image group into a second network in a preset defect type identification model, and determining respective reference defect types of the first sample image and the second sample image, wherein the reference defect types comprise a reference surface defect and a reference internal defect; determining a loss function value of a preset defect type identification model according to a reference defect type of a target sample image and a label defect type of the target sample image, wherein the target sample image is any one of sample image groups; and under the condition that the loss function value does not meet the training stopping condition, adjusting the model parameters of the defect type identification model, and training the defect type identification model after parameter adjustment by using the sample image group until the loss function value meets the preset training condition to obtain the defect type identification model.
In a second aspect, an embodiment of the present application provides a defect type identification apparatus, where the apparatus includes: the acquisition module is used for acquiring an image to be processed; the determining module is used for inputting the image to be processed into a first network of a defect category identification model trained in advance and determining a feature vector of the image to be processed, wherein the feature vector comprises a plurality of features which are used for representing the whole and/or local of the image to be processed; and the determining module is also used for inputting the feature vectors into a second network of the defect type model and determining the defect type of the image to be processed, wherein the defect type comprises surface defects and internal defects.
In some embodiments of the second aspect, the first network comprises an adaptive aggregation network and the second network comprises an offset correction network. In some embodiments of the second aspect, the obtaining module is configured to obtain a training sample set, where the training sample set includes a plurality of sample image groups, and each sample image group includes a sample image and a corresponding label defect category; the device further comprises a training module: and the training module is used for training a preset defect type identification model by utilizing the sample image group in the training sample set to obtain the defect type identification model.
In a third aspect, a defect category identifying apparatus is provided, including: a memory for storing computer program instructions; and the processor is used for reading and executing the computer program instructions stored in the memory so as to execute the defect type identification method provided by any optional implementation mode of the first aspect and the second aspect.
In a fourth aspect, a computer storage medium is provided, on which computer program instructions are stored, and the computer program instructions, when executed by a processor, implement the defect class identification method provided in any optional implementation manner of the first aspect and the second aspect.
According to the method and the device, after the image to be processed is obtained, the obtained image to be processed is input into the first network of the defect type identification model which is trained in advance to determine the feature vector of the image to be processed, and the feature vector is input into the second network of the defect type identification model which is trained in advance to obtain the defect type of the image to be processed, so that the newly added defect type and the old defect type in the image to be processed can be distinguished conveniently, and the accuracy of defect type identification is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly described below, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic flowchart of a training model in a defect classification identification method according to an embodiment of the present application;
FIG. 2 is a schematic flowchart of a training model in another defect classification identification method according to an embodiment of the present disclosure;
FIG. 3 is a schematic diagram of a model structure of a first network in a defect class identification model according to an embodiment of the present application;
FIG. 4 is a schematic diagram of training parameters of a first network in a defect class identification model according to an embodiment of the present disclosure;
fig. 5 is a schematic flowchart of a defect classification identifying method according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of a defect classification identifying apparatus according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of a defect type identification device according to an embodiment of the present application.
Detailed Description
Features and exemplary embodiments of various aspects of the present application will be described in detail below, and in order to make objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are intended to be illustrative only and are not intended to be limiting. It will be apparent to one skilled in the art that the present application may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the present application by illustrating examples thereof.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone.
In order to solve the problem that the identification accuracy of the defect type cannot be improved due to the fact that the newly added defect type and the old defect type are not easy to distinguish in the prior art, embodiments of the present application provide a defect type identification method, apparatus, device and medium.
It should be noted that, in the defect type identification method provided in the embodiment of the present application, the image to be processed needs to be processed by using the defect type identification model trained in advance, and therefore, the defect type identification model needs to be trained before the image to be processed is input to the first network of the defect type identification image trained in advance. Therefore, a specific implementation of the training method for the defect class identification model provided by the embodiment of the present application is first described below with reference to the drawings.
The embodiment of the application provides a training method of a defect type identification model, which can be specifically realized through the following steps:
firstly, a sample set is obtained.
The training sample set comprises a plurality of sample image groups, and each sample image group comprises a label defect type corresponding to a sample image.
In one embodiment, as shown in fig. 1, acquiring the training sample set specifically may include the following steps:
s110, a plurality of sample images are obtained.
The sample image may be an image currently acquired by a camera provided in the electronic device, or may be an image stored in the electronic device. Accordingly, the sample image may be acquired by, for example, a camera of the electronic device, or may be acquired directly from an image database of the electronic device. Among them, the electronic apparatus may be an apparatus having an image pickup function.
And S120, labeling label defect types corresponding to the sample images one by one.
Wherein the label defect category includes surface defects and internal defects. Wherein the surface defect is characterized as a defect appearing on the surface of the object, and the internal defect is characterized as a defect of the internal structure of the object. Taking the example of obtaining a plurality of sample images of the liquid crystal panel, the label defect type may be, for example, a surface defect such as a scratch defect or a scratch defect appearing on the surface of the liquid crystal panel, or an internal defect such as a color particle defect or a bubble defect existing in the internal structure of the liquid crystal panel.
In one example, the label defect type of the labeled sample image can be equipment labeling or manual labeling.
S130, determining a training sample set according to the obtained sample images and the labeled label defect types corresponding to the sample images.
The training sample set comprises a plurality of sample image groups, and each sample image group comprises a sample image and a label defect type corresponding to the sample image. Specifically, each acquired sample image and the corresponding label defect type labeled by the manual labeling or the equipment labeling are combined to obtain a plurality of sample image groups, so as to determine a training sample set.
In addition, in order to achieve the effects of easily distinguishing the newly added defect type from the old defect type and improving the accuracy of defect type identification, the training sample set may include all samples of the newly added defect type and part of the old defect type samples.
Therefore, label defect types corresponding to the sample images one by one are obtained by labeling the sample images. After each sample image and the corresponding label defect type are obtained, the plurality of sample images and the corresponding label defect types are further integrated to obtain a training set, so that subsequent model training is facilitated, and further the relevant models can be accurately obtained.
In one embodiment, the set of training samples may be subdivided into a first training set and a second training set. The first training sample set comprises a plurality of first label defect type groups which correspond to a plurality of preset proportions one by one, each first label defect type group comprises a plurality of first sample image groups, and each first sample image group comprises a first sample image and a first label defect type corresponding to the first sample image;
the second training sample set comprises a plurality of second label defect class groups, each second label defect class group comprises a plurality of second sample image groups with the same preset proportion, and each second sample image group comprises a second sample image and a corresponding second label defect class thereof.
In one example, taking the liquid crystal panel image as an example, the method is as follows: 2: the sample images with label defect types of scratch defect, scratch defect and bubble defect are obtained according to the preset proportion of 3, and may be, for example, 100 sample images with label defect types of scratch defect, 200 sample images with label defect types of scratch defect, and 300 sample images with label defect types of bubble defect. The 100 sample images with the label defect types as scratch defects may be grouped into a first label defect type, and each label defect type may be a sample image with a scratch defect and a corresponding label defect type, that is, a scratch defect, and may be regarded as a first sample group. The 200 sample images with label defects as scratch defects may be grouped for another first defect category, and so on.
In another embodiment, for example, taking the liquid crystal panel image as an example, sample images of the label defect types of scratch defect, scratch defect and bubble defect can be respectively obtained according to the same preset proportion. For example, 100 sample images in which the label defect category is scratch defect, and 100 sample images in which the label defect category is bubble defect may be cited. The determining method of the second label defect type grouping and the second sample group may be based on the determining method of the first label defect type grouping and the first sample group, and is not described herein again.
In addition, in one embodiment, the second training sample set may obtain the sample image of each second label defect category from the first training sample set according to the same preset ratio.
Therefore, the first network used for training the preset defect type identification model can be based on the first training sample set, namely the unbalanced training sample set, and the second network of the preset defect type identification model can be subjected to parameter estimation based on the second training sample set, namely the balanced training sample set, so that the preset defect type identification model can be trained better, a more accurate defect type identification model can be obtained, and the accuracy of defect type identification can be improved.
And secondly, training a preset defect type identification model by utilizing the sample image group in the training sample set to obtain the defect type identification model.
As shown in fig. 2, this step may include the steps of:
s210, inputting the first sample image group into a first network in a preset defect type identification model, and determining a reference feature vector of the first sample image.
The first sample image group includes a first sample image and a first label defect type corresponding to the first sample image, the related sample images include the first sample image, and an acquisition mode of the first sample image is consistent with an acquisition mode of the sample image, which is not described herein again. The first network in the default defect class identification model may be a network that effectively maintains the stability and plasticity of defect class identification, i.e., effectively identifies new defect classes, and may include an adaptive aggregation network, for example. The reference feature vector includes a plurality of features that characterize the entirety, and/or a part, of the first sample image. The reference feature vector may include, for example, contrast, brightness, etc. of the entire sample image, and a plurality of features of contrast, brightness, size, position, color, etc. of the target defect.
Specifically, the first sample image group is input into a first network in a preset defect type identification model, and a plurality of features are extracted from the first sample image through the first network to determine a reference feature vector of the first sample image.
Therefore, the first network in the preset defect type identification model, such as the self-adaptive aggregation network, can be used for accurately distinguishing the newly added defect type from the old defect type under the condition that the similarity between the newly added defect type and the old defect type is larger, and further improving the accuracy of defect type identification. For example, the degree of similarity between scratch defects and scratch defects is relatively high, and the degree of similarity between bubble defects and color particle defects is relatively high. Therefore, the difference between defect types can be effectively amplified through the self-adaptive aggregation network, so that the defect types such as scratch defects, bubble defects, color particle defects and the like can be accurately distinguished.
In particular, the structure and principles of the adaptive aggregation network are described in detail below. It should be noted that the adaptive aggregation network includes a two-layer feature extraction structure and an adaptive aggregation weight learning strategy. The double-layer feature extraction structure of the self-adaptive aggregation network comprises a stable module, a plastic module and a neuron-level scaling weight. As shown in fig. 3, each residual block in the network may be split into two blocks: one is a plastic block, the parameters of which are fully discemable, usingThe other is a stable block, the parameter part of which is fixed and is used for maintaining the identification of the old defect type. Wherein x is a characteristic diagram, alphasTo stabilize the adaptive weight of the block, alphapIs the adaptive weight of the moldable mass. Since there are fewer parameters that can be learned in the stable block, but more parameters can be learned in the plastic block. Let p and s represent the network parameters of the mouldable and stable blocks, respectively. p contains all convolution weights, while s contains only scalable weights at the neuron level.
Scalable weighting applicable to network model θ of old defect classesbaseI.e. the model obtained from the previous training of 0-i-1 old defect classes. Network model θ due to old defect classbaseIs an existing model, the number of network parameters s is much smaller than the number of p. For example, when at θbaseWhen 3 × 3 neurons are used, the number of networkable parameters s is only 1/(3 × 3) of the number of full-net parameters.
For the stable block, the scaling weights of the neuron level involved in the adaptive aggregation network are learned in the 0 th stage, and the learned network parameters are frozen in other N stages. In these N stages, the weight parameters within the block are scaled at the neuron level using a small set of scaling weights.
To preserve the internal structural pattern of the neuron and to adapt the knowledge of the entire block to the newly added class data. Suppose that the k layer of the network contains R weights, let be
Figure BDA0003204020380000081
Abbreviated as Wk. For WkThe stable block needs to learn R scaling weights, denoted as Sk. Meanwhile, let the input characteristic diagram and output characteristic diagram of the k-th layer be X respectivelyk-1And XkWill zoom in/out the weight SkAct on WkWhich can be represented by the following formula (1):
Xk=(Wk⊙Sk)Xk-1 (1)
wherein, element-wise multiplication.
Assuming that the overall network has K layers in common, the scaling weights for all neuron levels can be expressed by equation (2):
Figure BDA0003204020380000091
and also in fig. 3, is the feature extraction and aggregation process of the network, which spans all residual layers in the adaptive neural network.
Note the book
Figure BDA0003204020380000092
A feature extraction transformation function representing the residual block at k layer. Given a batch of training data sets x[0]After passing through the k-th layer residual block, respectively passing through the stable block and the plastic block, the characteristic diagram is shown in the following formula (3):
Figure BDA0003204020380000093
wherein,
Figure BDA0003204020380000094
respectively showing the characteristic diagrams of the stable block and the plastic block at the k layer,
Figure BDA0003204020380000095
the feature extraction transformation functions of the stable block and the plastic block at the k-th layer are respectively.
Order to
Figure BDA0003204020380000096
And
Figure BDA0003204020380000097
the aggregate weights of the stable blocks and the plastic blocks in the k-th layer are represented, respectively. The output characteristic of the k-th layer is thus plotted as shown in equation (4):
Figure BDA0003204020380000098
in addition, it is to be understood that the adaptive aggregation network needs to optimize two sets of learnable parameters at each incremental stage: (a) neuron-level scaling weights of the stable blocks and convolution weights of the shapeable blocks; (b) and a feature set weight α. The former belongs to the network weight parameter and the latter belongs to the hyper-parameter. In the present invention, we express the whole optimization process as a two-layer optimization process.
That is, in the adaptive aggregation network, the network parameters [ s, p ] are trained based on the aggregation weight α as a hyper-parameter. The aggregation weight α, in turn, may be updated at the time of the temporary fixed network parameter s, p. Thus, the optimality of s, p imposes a constraint on α and vice versa.
Ideally, in the ith increment phase, the goal of model training and learning is to learn the optimal aggregation weight α and network parameters [ s, p ], so that the classification loss of the training sample set is minimized. Currently, the training set for the adaptive aggregation network is the first set of training samples. The combined penalty of the two-layer optimization strategy can thus be shown as equation (5) and equation (6) below:
Figure BDA0003204020380000101
Figure BDA0003204020380000102
wherein L (-) represents a loss function, which can be a cross-entropy loss, ε0:i-1∪εiRepresenting a first set of training samples. Alpha is alphaiIs the ith incremental stage aggregation weight. si,,piThe network parameters of the stable block and the plastic block in the ith increment stage are respectively.
Figure BDA0003204020380000103
Are estimates of the network parameters of the stable block and the plastic block, respectively, at the i-th incremental stage.
In the double-layer optimization process, the process of training the aggregation weight alpha by using the temporary fixed network parameters [ s, p ] can be called as an uplink stage; the process of [ s, p ] trained by the aggregation weight α is called a downlink stage.
As shown in FIG. 4, in the process of training the aggregation weights, a second training sample set is used for adaptive learning and updatingiFor balancing the stable block and the plastic block; in training network parameters, a first training sample set is used to train feature extracted network parameters [ s ]i,pi]。
Therefore, the parameters of the adaptive aggregation network can be trained based on the training set, and the more accurate adaptive aggregation network can be obtained.
And S220, inputting the reference feature vector and the second sample image group into a second network in a preset defect type identification model, and determining the reference defect type of the first sample image and the reference defect type of the second sample image.
The second network may be a network in which the defect difference is weakened by a linear structure in a case where the difference between the newly added defect type and the old defect type is large, and may be an offset correction network, for example. The reference defect category includes a reference surface defect and a reference internal defect. Taking the example of obtaining an image of a liquid crystal panel, the reference defect type included in the image may be, for example, a reference surface defect appearing on the surface of an object, such as a scratch defect or a scratch defect. Or a color particle defect, a bubble defect, or the like, which is a reference internal defect that occurs in the internal structure of the object.
Specifically, the reference feature vector and the second sample image group are input into a second network in a preset defect type identification model, parameters in the second network are estimated through the input second sample image group, and the second network subjected to parameter estimation can identify the respective reference defect types of the first sample image and the second sample image based on a plurality of features of the first sample image contained in the reference feature vector and a plurality of features of the second sample image acquired in the process of training the second network.
Therefore, the difference between the defect types can be weakened through the second network of the defect type identification model, so that the defect type identification inaccuracy caused by the fact that the first network tends to the defect type with large sample data volume under the condition that the difference between the newly added defect type and the old defect type is large is avoided, and the defect type identification accuracy is improved.
Further, it can be known that, under the condition that the difference between the newly added defect type and the old defect type is small and difficult to distinguish, the difference between the newly added defect type and the old defect type is amplified through the characteristic enhancement characteristic of the first network of the defect type identification model, so as to achieve the purpose of accurately distinguishing the newly added defect type from the old defect type. And the condition of inaccurate identification caused by the first network under the condition that the difference between the newly added defect type and the old defect type is large is made up through the second network of the defect type identification model, so that the newly added defect type and the old defect type are accurately distinguished, and the accuracy of defect type identification is improved.
In addition, it should be noted that the first network and the second network of the preset defect type identification model are connected according to a classifier full connection layer. Specifically, after a first network of a preset defect type identification model outputs a reference feature vector, a bias correction network added after a classifier is fully connected with a layer is used for defect type identification. The training process of the bias correction network is to freeze the adaptive aggregation network and the classifier after the training of the infrastructure is completed to estimate the bias parameters using a second set of training samples.
Since the data sample size of the old class is small, the bias correction network is designed as a simple and parameter-less linear model to correct the resulting bias in the adaptive aggregation network. Correcting (i, …, i + m) of the newly added defect type by keeping the output logic (0, …, i-1) of the old defect type and applying a linear model, wherein i and m are positive integers, as shown in the following formula (7):
Figure BDA0003204020380000111
wherein, OutkOutput logic characterized as an offset correction network, alpha and beta being offset parameters for the newly added defect class, OkIs the output of class k. The deviation parameters (α, β) are shared by all newly added defect classes, allowing them to be estimated by the validation set, i.e. the second set of training samples. When the bias parameters are optimized, the adaptive aggregation network and the classifier are frozen. The classification loss can be calculated using the softmax function to optimize the bias parameter, as shown in equation (8) below:
Figure BDA0003204020380000112
wherein, deltay=kThe value range of the preset coefficient of the kth defect category is [ -1, 1]。
And S230, determining a loss function value of a preset defect type identification model according to the reference defect type of the target sample image and the label defect type of the target sample image.
Wherein the target sample image is any one of the sample image groups. Specifically, a loss function value of a preset defect type model is determined based on a reference defect type finally obtained from the target sample image and a label defect type manually labeled before.
And S240, under the condition that the loss function value does not meet the training stopping condition, adjusting the model parameters of the defect type identification model, and training the defect type identification model after the parameters are adjusted by utilizing the sample image group to obtain the defect type identification model.
In order to obtain the trained defect type identification model, under the condition that the loss function value does not meet the training stop condition, the model parameters of the defect type identification model are adjusted, the defect type identification model after parameter adjustment is trained by using the sample image group until the loss function meets the training stop condition, and the accurate defect type identification model is obtained.
Based on the defect type identification model obtained in the above embodiment, the present application further provides a specific embodiment of the defect type identification method, which is specifically described in detail with reference to fig. 5.
Fig. 5 is a flowchart illustrating a defect type identification method according to an embodiment of the present application.
As shown in fig. 5, the main implementation of the method is a defect type identification model or the defect type identification method may include the following steps:
and S510, acquiring an image to be processed.
The image to be processed may be an image currently acquired by a camera provided in the electronic device, or may be an image to be processed stored in the electronic device. Correspondingly, the method for acquiring the image to be processed can be acquired by a camera of the electronic equipment or can be directly acquired from an image library of the electronic equipment. Among them, the electronic apparatus is characterized as an apparatus having an image pickup function.
S520, inputting the image to be processed into a first network of a defect type identification model trained in advance, and determining a feature vector of the image to be processed.
The feature vector comprises a plurality of features which are used for representing the whole of the image to be processed and/or the local part. The feature vector may include, for example, contrast, brightness, etc. of the entire sample image, and a plurality of features of contrast, brightness, size, location, color, etc. of the target defect.
Therefore, the newly added defect type and the old defect type can be accurately distinguished under the condition that the similarity between the newly added defect type and the old defect type is large according to the first network in the defect type identification model trained in advance, and the accuracy of defect type identification is improved. For example, the degree of similarity between scratch defects and scratch defects is relatively high, and the degree of similarity between bubble defects and color particle defects is relatively high. Thus, the difference between defect classes can be effectively amplified through the first network to accurately distinguish between a newly added defect class and an old defect class
S530, inputting the characteristic vector into a second network of the defect type identification model, and determining the defect type of the image to be processed.
Wherein the defect classes include surface defects and internal defects. Surface defects are characterized as defects appearing on the surface of the object, and internal defects are characterized as defects of the internal structure of the object.
In one embodiment, taking the example of obtaining multiple sample images of the liquid crystal panel, the defect type may be a surface defect such as a scratch defect or a scratch defect appearing on the surface of the liquid crystal panel, or an internal defect such as a color particle defect or a bubble defect existing in the internal structure of the liquid crystal panel.
In some embodiments, the first network of defect class identification models comprises an adaptive aggregation network and the second network of defect class identification models comprises a bias correction network.
Therefore, after the image to be processed is obtained, the obtained image to be processed is input into the first network of the defect type identification model which is trained in advance to determine the feature vector of the image to be processed, and the feature vector is input into the second network of the defect type identification model which is trained in advance to obtain the defect type of the image to be processed, so that the new defect type and the old defect type in the image to be processed can be distinguished conveniently, and the accuracy of defect type identification is improved.
Based on the same inventive concept, the embodiment of the application also provides a defect type identification device. The description will be made with reference to FIG. 6
Fig. 6 is a schematic structural diagram of a defect classification identifying apparatus according to an embodiment of the present application.
As shown in fig. 6, the defect type identifying apparatus 600 may include: an acquisition module 610 and a determination module 620.
An obtaining module 610, configured to obtain an image to be processed;
a determining module 620, configured to input the image to be processed into a first network of a pre-trained defect class identification model, and determine a feature vector of the image to be processed, where the feature vector includes a plurality of features used for characterizing the whole and/or local of the image to be processed;
the determining module 620 is further configured to input the feature vectors into a second network of the defect type model, and determine defect types of the image to be processed, where the defect types include surface defects and internal defects.
In some embodiments, the first network comprises an adaptive aggregation network and the second network comprises an offset correction network.
In some embodiments, the acquisition module is configured to acquire a training sample set, where the training sample set includes a plurality of sample image groups, and each sample image group includes a sample image and a corresponding label defect category thereof;
the device further comprises a training module:
and the training module is used for training a preset defect type identification model by utilizing the sample image group in the training sample set to obtain the defect type identification model.
In some embodiments, the training sample set includes a first training sample set and a second training sample set, where the first training sample set includes a plurality of first label defect category groups corresponding to a plurality of preset ratios, each first label defect category group includes a plurality of first sample image groups, and each first sample image group includes a first sample image and its corresponding first label defect category;
the second training sample set comprises a plurality of second label defect class groups, each second label defect class group comprises a plurality of second sample image groups with the same preset proportion, and each second sample image group comprises a second sample image and a corresponding second label defect class thereof.
In some embodiments, the training module is specifically configured to:
for each sample image group, the following steps are respectively executed:
inputting the first sample image group into a first network in a preset defect type identification model, and determining a reference feature vector of each first sample image, wherein the reference feature vector comprises a plurality of features for representing the whole and/or local first sample images;
inputting the reference feature vector and the second sample image group into a second network in a preset defect type identification model, and determining respective reference defect types of the first sample image and the second sample image, wherein the reference defect types comprise a reference surface defect and a reference internal defect;
determining a loss function value of a preset defect type identification model according to a reference defect type of a target sample image and a label defect type of the target sample image, wherein the target sample image is any one of sample image groups;
and under the condition that the loss function value does not meet the training stopping condition, adjusting the model parameters of the defect type identification model, and training the defect type identification model after parameter adjustment by using the sample image group until the loss function value meets the preset training condition to obtain the defect type identification model.
According to the method and the device, after the image to be processed is obtained, the obtained image to be processed is input into the first network of the defect type identification model which is trained in advance to determine the feature vector of the image to be processed, and the feature vector is input into the second network of the defect type identification model which is trained in advance to obtain the defect type of the image to be processed, so that the newly added defect type and the old defect type in the image to be processed can be distinguished conveniently, and the accuracy of defect type identification is improved.
Each module in the defect type identification apparatus provided in the embodiment of the present application may implement the method steps in the embodiment shown in fig. 5, and may achieve the corresponding technical effect, and for brevity, no further description is given here.
Fig. 7 is a schematic structural diagram of a defect type identification device according to an embodiment of the present application.
As shown in fig. 7, the defect classification device 700 in the present embodiment includes an input device 701, an input interface 702, a central processing unit 703, a memory 704, an output interface 705, and an output device 706. The input interface 702, the central processing unit 703, the memory 704, and the output interface 705 are connected to each other via a bus 710, and the input device 701 and the output device 706 are connected to the bus 710 via the input interface 702 and the output interface 705, respectively, and further connected to other components of the defect type identification device 700.
Specifically, the input device 701 receives input information from the outside, and transmits the input information to the central processor 703 through the input interface 702; the central processor 703 processes input information based on computer-executable instructions stored in the memory 704 to generate output information, stores the output information temporarily or permanently in the memory 704, and then transmits the output information to the output device 706 through the output interface 705; the output device 706 outputs the output information to the outside of the defect class identification device 700 for use by the user.
In one embodiment, the defect class identifying apparatus 700 shown in fig. 7 includes: a memory 704 for storing programs; the processor 703 is configured to execute the program stored in the memory to perform the method of the embodiment shown in fig. 5 provided by the embodiment of the present application.
Embodiments of the present application further provide a computer-readable storage medium having computer program instructions stored thereon; the computer program instructions, when executed by a processor, implement the method of the embodiment shown in fig. 5 provided by the embodiments of the present application.
It is to be understood that the present application is not limited to the particular arrangements and instrumentality described above and shown in the attached drawings. A detailed description of known methods is omitted herein for the sake of brevity. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications and additions or change the order between the steps after comprehending the spirit of the present application.
The functional blocks shown in the above structural block diagrams may be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic Circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, plug-in, function card, or the like. When implemented in software, the elements of the present application are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine-readable medium or transmitted by a data signal carried in a carrier wave over a transmission medium or a communication link. A "machine-readable medium" may include any medium that can store or transfer information. Examples of machine-readable media include electronic circuits, semiconductor Memory devices, Read-Only memories (ROMs), flash memories, erasable ROMs (eroms), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, Radio Frequency (RF) links, and so forth. The code segments may be downloaded via computer networks such as the internet, intranet, etc.
It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above-described steps, that is, the steps may be performed in the order mentioned in the embodiments, may be performed in an order different from the order in the embodiments, or may be performed simultaneously.
As will be apparent to those skilled in the art, for convenience and brevity of description, the specific working processes of the systems, modules and units described above may refer to corresponding processes in the foregoing method embodiments, and are not described herein again. It should be understood that the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present application, and these modifications or substitutions should be covered within the scope of the present application.

Claims (10)

1. A defect classification identifying method, comprising:
acquiring an image to be processed;
inputting the image to be processed into a first network of a defect class identification model trained in advance, and determining a feature vector of the image to be processed, wherein the feature vector comprises a plurality of features used for representing the whole and/or local of the image to be processed;
inputting the feature vectors into a second network of the defect type model, and determining the defect type of the image to be processed, wherein the defect type comprises surface defects and internal defects.
2. The method of claim 1, wherein the first network comprises an adaptive aggregation network and the second network comprises a bias correction network.
3. The method of claim 1, wherein prior to inputting the image to be processed into the first network of pre-trained defect class identification models, the method further comprises:
acquiring a training sample set, wherein the training sample set comprises a plurality of sample image groups, and each sample image group comprises a sample image and a label defect type corresponding to the sample image;
and training a preset defect type identification model by using the sample image group in the training sample set to obtain the defect type identification model.
4. The method according to claim 3, wherein the training sample set comprises a first training sample set and a second training sample set, wherein the first training sample set comprises a plurality of first label defect class groups corresponding to a plurality of preset ratios, each of the first label defect class groups comprises a plurality of first sample image groups, and each of the first sample image groups comprises a first sample image and a corresponding first label defect class thereof;
the second training sample set comprises a plurality of second label defect category groups, each of the second label defect category groups comprises a plurality of second sample image groups with the same preset proportion, and each of the second sample image groups comprises a second sample image and a corresponding second label defect category.
5. The method according to claim 4, wherein the training a preset defect type recognition model by using the sample image group in the training sample set to obtain a trained defect type recognition model comprises:
for each sample image group, the following steps are respectively executed:
inputting the first sample image group into a first network in a preset defect type identification model, and determining a reference feature vector of each first sample image, wherein the reference feature vector comprises a plurality of features for representing the whole and/or local parts of the first sample image;
inputting the reference feature vector and a second sample image group into a second network in a preset defect type identification model, and determining respective reference defect types of the first sample image and the second sample image, wherein the reference defect types comprise a reference surface defect and a reference internal defect;
determining a loss function value of the preset defect type identification model according to a reference defect type of a target sample image and a label defect type of the target sample image, wherein the target sample image is any one of the sample image groups;
and under the condition that the loss function value does not meet the training stopping condition, adjusting the model parameters of the defect type identification model, and training the defect type identification model after parameter adjustment by using the sample image group until the loss function value meets the preset training condition to obtain the defect type identification model.
6. A defect classification apparatus, comprising:
the acquisition module is used for acquiring an image to be processed;
the determining module is used for inputting the image to be processed into a first network of a defect category identification model trained in advance and determining a feature vector of the image to be processed, wherein the feature vector comprises a plurality of features which are used for representing the whole and/or local of the image to be processed;
the determining module is further configured to input the feature vector into a second network of the defect type model, and determine a defect type of the image to be processed, where the defect type includes a surface defect and an internal defect.
7. The apparatus of claim 6, wherein the first network comprises an adaptive aggregation network and the second network comprises a bias correction network.
8. The method of claim 6, comprising:
the acquisition module is used for acquiring a training sample set, the training sample set comprises a plurality of sample image groups, and each sample image group comprises a sample image and a label defect type corresponding to the sample image;
the apparatus further comprises a training module:
and the training module is used for training a preset defect type identification model by utilizing the sample image group in the training sample set to obtain the defect type identification model.
9. A defect class identification device, characterized in that the device comprises: a processor and a memory storing computer program instructions;
the processor reads and executes the computer program instructions to implement the defect class identification method of any one of claims 1 to 5.
10. A computer storage medium having computer program instructions stored thereon which, when executed by a processor, implement the defect class identification method of any one of claims 1-5.
CN202110912056.4A 2021-08-10 2021-08-10 Defect category identification method, device, equipment and medium Active CN113706477B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110912056.4A CN113706477B (en) 2021-08-10 2021-08-10 Defect category identification method, device, equipment and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110912056.4A CN113706477B (en) 2021-08-10 2021-08-10 Defect category identification method, device, equipment and medium

Publications (2)

Publication Number Publication Date
CN113706477A true CN113706477A (en) 2021-11-26
CN113706477B CN113706477B (en) 2024-02-13

Family

ID=78652065

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110912056.4A Active CN113706477B (en) 2021-08-10 2021-08-10 Defect category identification method, device, equipment and medium

Country Status (1)

Country Link
CN (1) CN113706477B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115984235A (en) * 2023-01-31 2023-04-18 苏州大学 Wafer map mixed defect mode identification method and system based on image segmentation
WO2024020994A1 (en) * 2022-07-29 2024-02-01 宁德时代新能源科技股份有限公司 Training method and training device for defect detection model of battery cell

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109583489A (en) * 2018-11-22 2019-04-05 中国科学院自动化研究所 Defect classifying identification method, device, computer equipment and storage medium
CN109671071A (en) * 2018-12-19 2019-04-23 南京市测绘勘察研究院股份有限公司 A kind of underground piping defect location and grade determination method based on deep learning
CN110097543A (en) * 2019-04-25 2019-08-06 东北大学 Surfaces of Hot Rolled Strip defect inspection method based on production confrontation network
US20200357109A1 (en) * 2018-07-02 2020-11-12 Beijing Baidu Netcom Science Technology Co., Ltd. Method for detecting display screen quality, apparatus, electronic device and storage medium
CN112036517A (en) * 2020-11-05 2020-12-04 中科创达软件股份有限公司 Image defect classification method and device and electronic equipment
CA3053894A1 (en) * 2019-07-19 2021-01-19 Inspectorio Inc. Defect prediction using historical inspection data
CN113155851A (en) * 2021-04-30 2021-07-23 西安交通大学 Copper-clad plate surface defect visual online detection method and device based on deep learning
US20210232872A1 (en) * 2020-01-27 2021-07-29 Kla Corporation Characterization System and Method With Guided Defect Discovery

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200357109A1 (en) * 2018-07-02 2020-11-12 Beijing Baidu Netcom Science Technology Co., Ltd. Method for detecting display screen quality, apparatus, electronic device and storage medium
CN109583489A (en) * 2018-11-22 2019-04-05 中国科学院自动化研究所 Defect classifying identification method, device, computer equipment and storage medium
CN109671071A (en) * 2018-12-19 2019-04-23 南京市测绘勘察研究院股份有限公司 A kind of underground piping defect location and grade determination method based on deep learning
CN110097543A (en) * 2019-04-25 2019-08-06 东北大学 Surfaces of Hot Rolled Strip defect inspection method based on production confrontation network
CA3053894A1 (en) * 2019-07-19 2021-01-19 Inspectorio Inc. Defect prediction using historical inspection data
US20210232872A1 (en) * 2020-01-27 2021-07-29 Kla Corporation Characterization System and Method With Guided Defect Discovery
CN112036517A (en) * 2020-11-05 2020-12-04 中科创达软件股份有限公司 Image defect classification method and device and electronic equipment
CN113155851A (en) * 2021-04-30 2021-07-23 西安交通大学 Copper-clad plate surface defect visual online detection method and device based on deep learning

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
沈宗礼;余建波;: "基于迁移学习与深度森林的晶圆图缺陷识别", 浙江大学学报(工学版), no. 06 *
范涛;朱青;王耀南;周显恩;刘远强;: "空瓶检测机器人瓶底缺陷检测方法研究", 电子测量与仪器学报, no. 09 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024020994A1 (en) * 2022-07-29 2024-02-01 宁德时代新能源科技股份有限公司 Training method and training device for defect detection model of battery cell
CN115984235A (en) * 2023-01-31 2023-04-18 苏州大学 Wafer map mixed defect mode identification method and system based on image segmentation

Also Published As

Publication number Publication date
CN113706477B (en) 2024-02-13

Similar Documents

Publication Publication Date Title
CN113706477B (en) Defect category identification method, device, equipment and medium
CN111563706A (en) Multivariable logistics freight volume prediction method based on LSTM network
WO2021089013A1 (en) Spatial graph convolutional network training method, electronic device and storage medium
CN111861909B (en) Network fine granularity image classification method
CN109034175B (en) Image processing method, device and equipment
CN111160538B (en) Method and system for updating margin parameter value in loss function
CN111239137A (en) Grain quality detection method based on transfer learning and adaptive deep convolution neural network
CN114964313A (en) RVM-based fiber optic gyroscope temperature compensation method
CN110598753A (en) Defect identification method based on active learning
CN116992779B (en) Simulation method and system of photovoltaic energy storage system based on digital twin model
CN114491028A (en) Small sample text classification method based on regularization meta-learning
CN115146761A (en) Defect detection model training method and related device
CN116129219A (en) SAR target class increment recognition method based on knowledge robust-rebalancing network
CN114386482A (en) Image classification system and method based on semi-supervised incremental learning
CN110705689B (en) Continuous learning method and device capable of distinguishing features
CN115794805B (en) Method for supplementing measurement data of medium-low voltage distribution network
CN114826690B (en) Intrusion detection method and device based on edge cloud environment
US20240020531A1 (en) System and Method for Transforming a Trained Artificial Intelligence Model Into a Trustworthy Artificial Intelligence Model
CN114646328A (en) Method, device, equipment and medium for determining path information
CN112598082B (en) Method and system for predicting generalized error of image identification model based on non-check set
CN114942031A (en) Visual positioning method, visual positioning and mapping method, device, equipment and medium
CN111929585B (en) Battery charge state calculating device, method, server and medium
CN113222020B (en) Domain increment learning method based on data conversion and knowledge distillation
CN110309988A (en) A kind of Methods of electric load forecasting based on grey relational grade and support vector machines
CN113283804B (en) Training method and system of risk prediction model

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant