TW202449740A - Defect detection model training method, defect image classification method, device and electronic equipment - Google Patents

Abstract

The present application discloses a defect detection model training method and apparatus, and an electronic device. The method comprises: obtaining a training sample set and first defect label information corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set comprises a defect image and a template image corresponding to the defect image, and the first defect label information comprises first category information and first position information for each defect in the defect image; determining a target loss function; utilizing a stochastic gradient descent method, and by means of the training sample set, the defect label information, and the target loss function, training a model to be trained, and obtaining a target defect detection model, wherein the target defect detection model is used for determining defect position information and defect category information in an image containing defects to be detected.

Description

Defect detection model training method, defect image classification method, device and electronic equipment

This invention claims priority to a Chinese patent application filed with the China Patent Office on February 17, 2023, with application number 202310184844.5 and application name “Training method, device and electronic device for defect detection model”, the entire contents of which are incorporated by reference in this invention.

The present invention relates to the field of image recognition, and more particularly to a method, device and electronic equipment for training a defect detection model.

In the related technology, when using the flawless grain image to identify and classify the defects of the grain image to be inspected, the template image is usually not used to assist in the identification. The method usually adopted is to directly fuse the grain image to be inspected and the flawless grain image, and then determine the defect type and defect location in the image to be inspected based on the fused image. Although this method uses the flawless image to realize the defect identification and classification of the image to be inspected, the amount of calculation is large when fusing the image and identifying the fused image, and the accuracy is relatively low.

Currently, no effective solution has been proposed to the above problems.

The embodiments of the present invention provide a defect detection model training method, apparatus and electronic equipment to at least solve the technical problems of large amount of calculation and low accuracy caused by directly fusing defect-free images and images to be detected when performing defect identification and classification on grain images in related technologies.

According to one aspect of an embodiment of the present invention, a method for training a defect detection model is provided, comprising: obtaining a training sample set and a first defect label message corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label message includes a first category message and a first position message of each defect in the defect image; determining a target loss function, wherein the target loss function includes a defect category loss function and a defect position loss function; adopting random gradient descent The method trains the model to be trained by using a training sample set, defect label information and a target loss function to obtain a target defect detection model, wherein the target defect detection model is used to determine the defect position information and defect category information in the defect image to be detected, the model to be trained is used to extract the first image feature of the defect image and the second image feature of the template image, and outputs the second label information of the defect image according to the first image feature and the second image feature, and the second label information includes the second category information and the second position information of the defects in each defect image determined by the model to be trained.

Optionally, the random gradient descent method is used to train the model to be trained through the training sample set, the defect label information and the target loss function, and the steps of obtaining the target defect detection model include: the first step, respectively inputting the defect image and the template image into the model to be trained, and obtaining the second label information output by the model to be trained; the second step, inputting the first category information and the second category information into the defect category loss function to obtain the first loss function value, and inputting the first position information and the second position information into the defect position loss function to obtain the second loss function value, and storing it. The first loss function value and the second loss function value are stored; in the third step, the model parameters of the model to be trained are adjusted by the first loss function value and the second loss function value using the stochastic gradient descent method; in the fourth step, all the stored first loss function values and the second loss function values are obtained, and when the first loss function value and the second loss function value meet the preset conditions, the adjusted model to be trained is determined to be the target defect detection model, otherwise jump to the first step, wherein the preset conditions include that the sum of a continuous preset number of first loss function values and the second loss function values are all in the target value interval.

Optionally, the step of inputting the defect image and the template image into the model to be trained respectively and obtaining the second label information output by the model to be trained includes: inputting the defect image and the template image into the backbone network of the model to be trained, wherein the backbone network is configured to extract image features from the defect image and the template image, thereby obtaining a defect feature image and a template feature image; and obtaining the second label information output by the model to be trained based on the defect feature image and the template feature image.

Optionally, the model to be trained also includes a feature fusion layer and a feature pyramid layer, wherein the feature fusion layer is configured to obtain a first target feature image based on the defect feature image and the template feature image, and input the first target feature image into the feature pyramid layer; the feature pyramid layer is configured to output a second label information based on the first target feature image.

Optionally, the feature fusion layer includes a first feature fusion layer, wherein the first feature fusion layer is configured to perform a differential operation on the defect feature image and the template feature image to obtain a differential feature image, wherein the pixel value of any pixel point in the differential feature image is equal to the absolute value of the difference between the defect pixel point and the template pixel point corresponding to any pixel point, the defect pixel point is the pixel point corresponding to any pixel point in the defect feature image, and the template pixel point is the pixel point corresponding to any pixel point in the template feature image; channel stitching processing is performed on the differential feature image and the defect feature image to obtain a second target feature image; and convolution processing is performed on the second target feature image through target convolution check to obtain the first target feature image.

Optionally, the feature fusion layer includes a second feature fusion layer, wherein the second feature fusion layer is configured to perform convolution processing on the defect feature image and the template feature image respectively through a target convolution kernel, and generate a first target feature image based on the defect feature image and the template feature image after the convolution processing, wherein the pixel value of any pixel point in the first target feature image is equal to the sum of the pixel values of the defect pixel point and the template pixel point corresponding to the any pixel point.

Optionally, the step of determining that the adjusted model to be trained is the target defect detection model includes: obtaining a verification sample set and a third defect label information corresponding to each group of samples in the verification sample set, wherein each group of samples in the verification sample set includes a defect image and a template image corresponding to the defect image, and the third defect label information includes third category information and third position information of defects at various locations in the defect image; when there are multiple adjusted models to be trained, determining the average precision mean of each adjusted model to be trained in the multiple adjusted models to be trained through the verification sample set and the third defect label information; and determining the adjusted model to be trained with the largest average precision mean as the target defect detection model.

Optionally, the step of obtaining a training sample set includes: obtaining a first defect image and a first template image corresponding to the first defect image; performing geometric transformation processing on the first defect image and the first template image using the same processing method, and using the first defect image after the geometric transformation processing as a defect image sample, and using the first template image after the combined transformation processing as a template image sample, wherein the processing method includes at least one of the following: random cropping, image flipping, and image stitching.

Optionally, the category information includes a defect category number of the defect, and the location information includes location information of a defect box corresponding to the defect.

According to another aspect of the embodiment of the present invention, a defect image classification method is also provided, including: determining a defect image to be detected, and a template image corresponding to the defect image to be detected; inputting the defect image to be detected and the template image into a target defect detection model to obtain defect location information and defect category information output by the target defect detection model, wherein the target defect detection model is used to extract a third image feature of the defect image to be detected and a fourth image feature of the template image, and output defect location information and defect category information based on the three image features and the fourth image feature; determining a preset number of target defects in the defect image to be detected that are closest to the center point of the defect image to be detected based on the defect location information; determining the category confidence of each target defect in the preset number of target defects, and determining the defect category of the target defect with the largest category confidence as the defect category of the defect image to be detected.

According to another aspect of the embodiment of the present invention, a training device for a defect detection model is provided, comprising: an input module, configured to obtain a training sample set and a first defect label message corresponding to each set of samples in the training sample set, wherein each set of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label message includes a first category message and a first position message of each defect in the defect image; The first processing module is configured to determine a target loss function, wherein the target loss function includes a defect category loss function and a defect position loss function; the second processing module is configured to use a random gradient descent method to train a model to be trained using a training sample set, defect label information and a target loss function to obtain a target defect detection model, wherein the target defect detection model is configured to determine defect location information and defect category information in a defect image to be detected.

According to another aspect of the embodiment of the present invention, a non-volatile storage medium is provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute a defect model training method or a defect image classification method.

According to another aspect of an embodiment of the present invention, an electronic device is provided. The electronic device includes a memory and a processor. The processor is configured to run a program stored in the memory. When the program is running, a training method for a defect detection model or a defect image classification method is executed.

In the embodiment of the present invention, a training sample set and a first defect label information corresponding to each group of samples in the training sample set are obtained, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes a first category information and a first position information of defects at various locations in the defect image; a target loss function is determined, wherein the target loss function includes a defect category loss function and a defect position loss function; a random gradient descent method is used to train a model to be trained through the training sample set, the defect label information and the target loss function to obtain a target defect detection model, wherein the target defect detection model is used to determine the defect location information and the defect category information in the defect image to be detected, and the model to be trained is used to extract the defect image. The method comprises the following steps: a first image feature and a second image feature of a template image, and outputting a second label information of the defect image based on the first image feature and the second image feature, wherein the second label information includes second category information and second position information of defects at various locations in the defect image to be determined by the training model. By adopting a training sample set including defect images and template images and defect label information to train a defect detection model, the purpose of obtaining a defect detection model that can utilize the template image in the defect recognition process is achieved, thereby achieving the technical effect of utilizing the template image to assist in classification recognition in the process of defect recognition and classification of grain images, thereby solving the technical problem of poor recognition and classification results caused by not utilizing the template image in the related technology when performing defect recognition and classification on grain images.

In order to enable a person with ordinary knowledge in the technical field to better understand the solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below in combination with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by a person with ordinary knowledge in the technical field without making progressive labor should belong to the scope of protection of the present invention.

It should be noted that the terms "first", "second", etc. in the specification and patent application of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or apparatus.

With the development of semiconductor device technology, more and more processes are used to manufacture semiconductor devices. The main purpose is to make circuits and electronic components in the grain, such as transistors, capacitors and logic switches, etc. For example, by oxidation and chemical vapor deposition on the surface of the grain, followed by coating, exposure, development, etching, ion implantation, metal sputtering and other steps, and finally completing several layers of circuit and component processing and manufacturing on the grain. However, since each process in the manufacturing process has a certain degree of complexity, each process flow may produce some unexpected structures at different layers when treating the grain, and these structures will cause the circuit on the chip to not work properly. This structure is usually called a grain defect.

In order to eliminate grain defects, the chip manufacturing process will arrange a grain defect detection step after many key processes to monitor the key processes and ensure their correctness. However, due to the extremely complex process of chip manufacturing and the wide variety of grain defects, there is currently no unified classification method. Currently, AOI (Automated Optical Inspection) usually uses traditional vision, i.e. image differential (target grain and gold grain) or threshold segmentation detection algorithms to locate defects on grains. However, due to the complex background of grains, the various processes, and the different styles of grains, although the use of traditional visual algorithms can achieve defect location, it is almost impossible to manually extract defect features for classification. Often, after formulating a series of rules (defect patterns and their defect shapes and sizes, grayscale values of defect location areas, and defect signal strength, etc.), the classification results obtained after screening are not accurate, and there are many over-inspections and mis-inspections. Therefore, most of the time, grain defect classification relies heavily on the original manual re-inspection method. Manual classification operations have the disadvantages of slow speed, poor consistency/reliability, and are easily affected by external factors, which makes grain defect classification have certain technical bottlenecks.

In addition, with the rapid research and development of deep learning algorithms and hardware equipment in recent years, especially the iterative optimization of neural networks in the fields of target classification, target detection, and target segmentation, the automated defect detection/classification in the semiconductor field has increasingly used deep learning algorithm technology to achieve higher and higher detection/classification accuracy.

However, related technologies rely on feature extraction technology for defect identification and classification, and defects on grains with complex backgrounds are only identified by a number of manually selected features, and the resulting model classification performance is limited. Convolutional neural networks can automatically learn to extract features suitable for classification, improving the detection and classification performance. However, in actual production, the grain background is complex, the textures are numerous, the defect features are not obvious, and the features of different types of defects are similar and difficult to distinguish. How to ignore the useless background and accurately extract the features of the defective area of interest is the key difficulty of grain defect detection and classification. Therefore, there is a problem of not being able to efficiently and accurately determine the fault type in the grain image and classify the fault. In order to solve this problem, the present invention provides a related solution in the embodiment, which is described in detail below.

According to an embodiment of the present invention, a method embodiment of a defect detection model training method is provided. It should be noted that the steps shown in the flowchart in the figure can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

The method embodiment provided by the embodiment of the present invention can be executed in a mobile terminal, a computer terminal or a similar computing device. FIG1 shows a hardware structure block diagram of a computer terminal (or mobile device) configured to implement a training method for a defect detection model. As shown in FIG1 , a computer terminal 10 (or mobile device 10) may include one or more (shown in the figure as 102a, 102b, ..., 102n) processors 102 (processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input/output port (I/O port), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network port, a power supply and/or a camera. A person skilled in the art may understand that the structure shown in FIG. 1 is only for illustration and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 may also include more or fewer components than those shown in FIG. 1, or have a configuration different from that shown in FIG. 1.

It should be noted that the one or more processors 102 and/or other data processing circuits mentioned above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination. In addition, the data processing circuit may be a single independent processing module, or may be fully or partially integrated into any of the other components in the computer terminal 10 (or mobile device). As involved in the embodiments of the present invention, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to a port).

The memory 104 can be configured to store software programs and modules of application software, such as the program instruction/data storage device corresponding to the defect detection model training method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the defect detection model training method of the above-mentioned application program is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The transmission module 106 is configured to receive or send data via a network. A specific example of the network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network interface controller (NIC), which can be connected to other network devices through a base station to communicate with the network. In one example, the transmission module 106 may be a radio frequency (RF) module, which is configured to communicate with the network wirelessly.

The display may be, for example, a touch screen liquid crystal display (LCD), which may enable a user to interact with a user interface of the computer terminal 10 (or mobile device).

In the above operating environment, the embodiment of the present invention provides a method for training a defect detection model, as shown in FIG2, the method includes the following steps: Step S202, obtaining a training sample set and a first defect label message corresponding to each set of samples in the training sample set, wherein each set of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label message includes a first category message and a first position message of defects in each location in the defect image; In the technical solution provided in step S202, the defect image and the template image in the sample image are shown in FIG3b. As can be seen from FIG3b, the template image and the defect image correspond to the same position in the grain image, and the only difference is that there is no defect or flaw in the template image.

As an optional implementation, after determining the position information of the defect image in the grain image, a corresponding template image can be obtained from the golden grain image shown in FIG. 3a according to the position information, wherein the golden grain image refers to an image without any defects or flaws. The golden grain image is usually stored in the AOI inspection equipment.

In addition, the first category information described in step S202 includes the category number of the defect. Specifically, the defect categories and corresponding category numbers that need to be identified in the die image can be determined in advance. For example, assuming that the defect categories that need to be identified include eight categories, including opening dirt, line dirt, opening damage, line etching, plastic surface damage, poor development, line bridge, and edge pressing, the corresponding category numbers can be {0, 1, 2, 3, 4, 5, 6, 7} in sequence.

The first position information in step S202 includes the coordinate information of the defect frame corresponding to the defect in the image. Specifically, tools such as labelImg can be used to mark all defects in the defect image with defect frames, and in the process of marking, images that cannot be analyzed and marked can be filtered out.

Optionally, the first defect label information can be saved in the form of a label file. Each line in the label file records the relevant information of a defect in the defect image, and the format is: [x0, y0, x1, y1, category label], where x0 represents the horizontal coordinate of the upper left corner of the defect box, y0 represents the vertical coordinate of the upper left corner of the defect box, x1 represents the horizontal coordinate of the lower right corner of the defect box, and y1 represents the vertical coordinate of the lower right corner of the defect box. Since the sides of the defect box are parallel to one side of the defect image, only the two coordinates of the upper left corner and the lower right corner of the defect box are needed to determine the position of the defect box, and then determine the position of the defect. It should be noted that the coordinates of the upper left corner and the lower right corner of the defect box are the coordinates of these two points in the defect plane rectangular coordinate system, and the defect plane rectangular coordinate system is a plane rectangular coordinate system established in the defect image.

As an optional implementation method, in order to realize efficient training of the defect detection model with limited grain defect data, the training sample set acquisition method shown in FIG4 can also be used to process the training sample set. As shown in FIG4, the following steps are included: Step S402, obtaining a first defect image and a first template image corresponding to the first defect image; Step S404, geometrically transforming the first defect image and the first template image using the same processing method, and using the first defect image after the geometric transformation as a defect image sample, and using the first template image after the combined transformation as a template image sample, wherein the processing method includes at least one of the following: random cropping, image flipping, and image splicing.

Specifically, the cropping scale range of random cropping can be set by yourself, for example, it can be set to 70% to 130% of the original image size, and the pixel value of the blank part of the cropped image can be set to zero. When flipping, multiple rows or columns of pixels in the image can be flipped horizontally or vertically according to the preset probability. Image stitching can be processed by R-Stitch stitching and other methods.

After sorting out the sample data in this way, the training batches in the training process are indirectly increased, the problem of unbalanced category distribution is solved, the diversity of samples is increased, the occurrence of overfitting in the training process is effectively prevented, and the robustness of the model to the size of the images to be tested is improved.

In step S204, a target loss function is determined, wherein the target loss function includes a defect category loss function and a defect position loss function; in the technical solution provided in step S204, the defect category loss function may be a cross entropy loss function, and the defect position loss function may be a smoothed L1 loss function.

Step S206, using the random gradient descent method, the model to be trained is trained through the training sample set, defect label information and target loss function to obtain the target defect detection model, wherein the target defect detection model is used to determine the defect position information and defect category information in the defect image to be detected, and the model to be trained is used to extract the first image feature of the defect image and the second image feature of the template image, and output the second label information of the defect image based on the first image feature and the second image feature, and the second label information includes the second category information and the second position information of the defects in each defect image determined by the model to be trained.

In the technical solution provided in step S206, the specific process of training the target defect detection model is shown in Figure 5, including the following steps: Step S502, respectively input the defect image and the template image into the model to be trained, and obtain the second label information output by the model to be trained, wherein the second label information includes the second category information and the second position information of each defect in the defect image determined by the model to be trained; Step S504, input the first category information and the second category information into the defect category loss function to obtain the first loss function value, and input the first position information and the second position information into the defect position loss function to obtain the second loss function value, and store the first loss function value and the second loss function value; Step S506, using the random gradient descent method, adjust the model parameters of the model to be trained through the first loss function value and the second loss function value; It should be noted that the official pre-trained weights can be loaded as the initial model parameters of the model to be trained. Since the official pre-trained weights are trained on the open source dataset ImageNet containing a large number of sample images, the use of pre-trained weights can enable the model to be trained to learn the characteristics of the target bottom layer better, greatly alleviate the overfitting problem caused by too few samples in the training sample set, and can speed up the model convergence speed.

In step S508, all the first loss function values and the second loss function values stored are obtained. If the first loss function value and the second loss function value meet the preset conditions, the adjusted model to be trained is determined to be the target defect detection model, otherwise jump to step S502.

The preset condition may be that both the first loss function value and the second loss function value no longer fluctuate. For example, the difference between any two first loss function values in a preset number of consecutive first loss function values is less than a first preset difference, and the difference between any two second loss function values in a preset number of consecutive second loss function values is less than a second preset difference.

Specifically, the common cross entropy loss is used as the loss calculation function for the category, the smooth L1 loss is used as the loss calculation function for the true coordinate frame, and the classic stochastic gradient descent (SGD) is used as the optimizer for network back propagation. The training set is batch-imported into the above-mentioned model to be trained, and the output (second label information) is calculated by the forward propagation process of the neural network. The loss function value is calculated by comparing it with the true value (first label information), and then the model parameters are updated through the back propagation process (gradient descent method). The training is repeated until the loss function value remains stable and no longer changes significantly.

In the technical solution provided in step S502, the specific method for extracting the second label information is shown in Figure 6, including the following steps: Step S602, input the defect image and the template image into the backbone network of the model to be trained, wherein the backbone network is configured to extract image features from the defect image and the template image, thereby obtaining the defect feature image and the template feature image; Step S604, obtain the second label information output by the model to be trained based on the defect feature image and the template feature image.

Optionally, the training model provided in the embodiment of the present invention is compared with the model in the related art, because the defect image and the template image need to be input simultaneously, so there will be two backbone networks with the same structure to extract the image features of the defect image and the template image simultaneously. The structure of the backbone network can select appropriate parameter quantities and network types according to the existing hardware memory conditions, such as VGG series, ResNet series, RegNet series, Efficient series, MobileNet series, etc.

The model to be trained provided in the embodiment of the present invention also includes a feature fusion layer and a feature pyramid layer, wherein the feature fusion layer is configured to obtain a first target feature image based on the defect feature image and the template feature image, and input the first target feature image into the feature pyramid layer; the feature pyramid layer is configured to output a second label message based on the first target feature image.

Specifically, as an optional implementation, as shown in FIG7 , the feature fusion layer includes a first feature fusion layer. The first feature fusion layer is configured to perform a differential operation on the defect feature image and the template feature image to obtain a differential feature image, wherein the pixel value of any pixel point in the differential feature image is equal to the absolute value of the difference between the defect pixel point and the template pixel point corresponding to any pixel point, the defect pixel point is the pixel point corresponding to any pixel point in the defect feature image, and the template pixel point is the pixel point corresponding to any pixel point in the template feature image; the differential feature image and the defect feature image are channel stitched to obtain a second target feature image; the second target feature image is convoluted by the target convolution check to obtain the first target feature image.

In some embodiments, the structure of the model to be trained may also be shown in FIG8. As can be seen from FIG8, the feature fusion layer includes a second feature fusion layer, wherein the second feature fusion layer is configured to perform convolution processing on the defect feature image and the template feature image respectively through a target convolution kernel, and generate the first target feature image according to the defect feature image and the template feature image after the convolution processing, wherein the pixel value of any pixel point in the first target feature image is equal to the sum of the pixel values of the defect pixel point and the template pixel point corresponding to the any pixel point.

The models to be trained shown in Figures 7 and 8 can both efficiently use the information of the template image to locate the defects or flaws in the defect image. The difference is that the model shown in Figure 7 first obtains the difference image between the defect image and the template image, then performs channel splicing on the difference image and the defect image (that is, the addition on the image channel, which is equivalent to expanding to twice the number of original channels of the feature map), and then performs channel reduction processing through a 1×1 convolution kernel and sends it to the subsequent feature pyramid layer. The solution embodied in Figure 8 is to directly superimpose the defect image and the template image, and then send it to the subsequent feature pyramid layer.

In order to ensure the training effect, multiple models to be trained can be trained at the same time, and after the training is completed, the validation set is input into each model to calculate the mAP (mean Average Precision), and the model with the highest mAP is selected as the final target model. The specific process is shown in Figure 9, which includes the following steps:

In step S902, a verification sample set and a third defect label information corresponding to each group of samples in the verification sample set are obtained, wherein each group of samples in the verification sample set includes a defect image and a template image corresponding to the defect image, and the third defect label information includes third category information and third position information of defects at various locations in the defect image; it should be noted that the verification sample set can adopt the same processing method as the training sample set to perform geometric transformation processing on the images in the verification sample set.

Step S904, when there are multiple adjusted models to be trained, determine the average precision mean of each of the multiple adjusted models to be trained by using the verification sample set and the third defect label information; Step S906, determine the adjusted model to be trained with the largest average precision mean as the target defect detection model.

By obtaining a training sample set and a first defect label information corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes the first category information and the first position information of each defect in the defect image; determining a target loss function, wherein the target loss function includes a defect category loss function and a defect position loss function; using a random gradient descent method, the model to be trained is trained through the training sample set, the defect label information and the target loss function, and a target defect detection model is obtained, wherein the target defect detection model The model is set to determine the defect location information and defect category information in the defect image to be detected, and the grain defect detection and classification based on template image feature matching is realized. The grain background information obtained in the previous inspection process is fully utilized. The template image and the defect image captured from the defect-free golden die are used as the input of the network model, and the feature image after feature extraction is matched and fused. This method efficiently uses the information of the template image and avoids the interference of the complex background circuit board. In addition, due to the translation invariance of the neural network, the impact caused by slight misalignment of the template image and the defect image is effectively reduced, reducing the dependence on data preprocessing. Compared with traditional visual detection algorithms and neural network models with a single input network, this method not only solves the problem that traditional visual detection algorithms rely too much on manually extracted features when classifying grain defects in complex backgrounds, but also solves the problem that a single input network ignores the golden grain information unique to AOI equipment and cannot reasonably use the existing grain background information to extract defect features.

In addition, although the model to be detected provided in the embodiment of the present invention can simultaneously input the defect image and the template image, since the two images use the same backbone network, the number of model parameters is almost not increased, thereby ensuring the inference speed of the model.

In addition, the embodiments of the present invention adopt pre-training weight fine-tuning and online random geometric transformation, splicing and other data enhancement methods, which solve the problem of unbalanced category distribution without increasing the amount of data (data storage space), increase the diversity of samples, effectively prevent the occurrence of over-fitting during network training, and improve the robustness of the model to the size of the image to be detected.

Moreover, the feature matching fusion model structure of the present invention is applicable to the transformation of any neural network, including classification, detection, segmentation, etc., and is not limited to the field of grain defect detection, but is applicable to the field of defect detection in all complex background scenes, and has strong versatility.

The embodiment of the present invention provides a defect image classification method. FIG10 is a flow chart of the defect image classification method. As shown in FIG10, the method includes the following steps: Step S1002, determining the defect image to be detected and the template image corresponding to the defect image to be detected; Step S1004, inputting the defect image to be detected and the template image into the target defect detection model, obtaining the defect location information and defect category information output by the target defect detection model, wherein the target defect detection model is used to extract the third image feature of the defect image to be detected and the fourth image feature of the template image, and outputting the defect location information and defect category information based on the third image feature and the fourth image feature; Step S1006, determining the preset number of target defects in the defect image to be detected that are closest to the center point of the defect image to be detected based on the defect location information; Step S1008, determine the category confidence of each target defect among the preset number of target defects, and determine the defect category of the target defect with the largest category confidence as the defect category of the defect image to be detected.

Specifically, after obtaining the positions and category labels of all defects in the defect image, the category with the largest category confidence in the topk (k can be any value, such as 3) detection box closest to the center point of the image is selected as the final defect category of the current image.

Since the defect images obtained in actual production may contain multiple defects of different types and locations, a simple classification network often cannot obtain accurate categories of the area of interest (a single-classification model may not converge, and a multi-classification model cannot determine the category location). In the embodiment of the present invention, a target detection model is designed to first locate and classify the defects in the defect image, and then the final category is determined by the distance from the area of interest and the category confidence, thereby ensuring the normal convergence of the model and obtaining accurate category labels.

The embodiment of the present invention also provides a practical application process of model training and image classification. Figure 11 is a schematic diagram of the model training and classification process. As shown in Figure 11, first in the model training process, after obtaining the defect image, the AOI equipment can capture the corresponding template image in the gold Die (grain) image according to the defect image, and generate a label file for the defect image. Then the defect image and the template image can be processed by geometric changes, R-stitch splicing, etc. After the processing is completed, the processed images are divided into a training set and a verification set according to a certain ratio. Among them, the training set is used to train the model to be trained. During the training process, the loss function can be constructed to determine whether the training process is completed, and the model parameters can be updated by the back propagation method. Specifically, the training can be completed when the function value of the loss function remains stable. When using the validation set to validate the trained model, the validation set can be input into the trained model, and the mAP corresponding to each model can be calculated, and the model with the highest mAP can be determined as the best model obtained in the end.

When applying the best model to the task of image classification, the AOI device will first extract the corresponding template image from the golden die image according to the image to be inspected, and input the template image and the image to be inspected into the best model. The best model will determine the location and type of all defects in the image to be inspected, and determine the category of the image to be inspected based on the categories of the defect frames closest to the center of the image to be inspected. Specifically, the defect category corresponding to the defect frame with the highest confidence among these defect frames can be used as the category of the image to be inspected.

The embodiment of the present invention provides a training device for a defect detection model, and FIG12 is a schematic diagram of the structure of the device. As shown in FIG12, the device includes: an input module 120, which is configured to obtain a training sample set and a first defect label message corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label message includes a first category message and a first position message of each defect in the defect image; a first processing module 122, which is configured to determine a target loss function, wherein the target loss function includes a defect category loss function and a defect position loss function; a second processing module 124, which is configured to adopt The model to be trained is trained by the random gradient descent method through the training sample set, defect label information and target loss function to obtain the target defect detection model, wherein the target defect detection model is set to determine the defect position information and defect category information in the defect image to be detected, and the model to be trained is set to extract the first image feature of the defect image and the second image feature of the template image, and output the second label information of the defect image according to the first image feature and the second image feature, and the second label information includes the second category information and the second position information of the defects in each defect image determined by the model to be trained.

In some embodiments of the present invention, the second processing module 124 uses a random gradient descent method to train the model to be trained through a training sample set, a defect label message and a target loss function, and the steps of obtaining the target defect detection model include: a first step, inputting the defect image and the template image into the model to be trained respectively, and obtaining the second label message output by the model to be trained, wherein the second label message includes the second category information and the second position information of each defect in the defect image determined by the model to be trained; a second step, inputting the first category information and the second category information into the defect category loss function to obtain the first loss function value, and inputting the first position information and the second position information into In the defect position loss function, the second loss function value is obtained, and the first loss function value and the second loss function value are stored; in the third step, the random gradient descent method is used to adjust the model parameters of the model to be trained through the first loss function value and the second loss function value; in the fourth step, all the stored first loss function values and the second loss function values are obtained, and when the difference between any two first loss function values in a continuous preset number of first loss function values is less than the first preset difference, and the difference between any two second loss function values in a continuous preset number of second loss function values is less than the second preset difference, the adjusted model to be trained is determined to be the target defect detection model, otherwise jump to the first step.

In some embodiments of the present invention, the second processing module 124 inputs the defect image and the template image into the model to be trained respectively, and the step of obtaining the second label information output by the model to be trained includes: inputting the defect image and the template image into the backbone network of the model to be trained, wherein the backbone network is configured to extract image features from the defect image and the template image, thereby obtaining a defect feature image and a template feature image; and obtaining the second label information output by the model to be trained based on the defect feature image and the template feature image.

In some embodiments of the present invention, the model to be trained also includes a feature fusion layer and a feature pyramid layer, wherein the feature fusion layer is configured to obtain a first target feature image based on the defect feature image and the template feature image, and input the first target feature image into the feature pyramid layer; the feature pyramid layer is configured to output a second label information based on the first target feature image.

In some embodiments of the present invention, the feature fusion layer includes a first feature fusion layer, wherein the first feature fusion layer is configured to perform a differential operation on the defect feature image and the template feature image to obtain a differential feature image, wherein the pixel value of any pixel point in the differential feature image is equal to the absolute value of the difference between the defect pixel point and the template pixel point corresponding to any pixel point, the defect pixel point is the pixel point corresponding to any pixel point in the defect feature image, and the template pixel point is the pixel point corresponding to any pixel point in the template feature image; channel stitching processing is performed on the differential feature image and the defect feature image to obtain a second target feature image; and convolution processing is performed on the second target feature image through target convolution check to obtain the first target feature image.

In some embodiments of the present invention, the feature fusion layer includes a second feature fusion layer, wherein the second feature fusion layer is configured to perform convolution processing on the defect feature image and the template feature image respectively through a target convolution kernel, and generate a first target feature image based on the defect feature image and the template feature image after the convolution processing, wherein the pixel value of any pixel point in the first target feature image is equal to the sum of the pixel values of the defect pixel point and the template pixel point corresponding to the any pixel point.

In some embodiments of the present invention, the step of the second processing module 124 determining that the adjusted model to be trained is the target defect detection model includes: obtaining a verification sample set and a third defect label message corresponding to each group of samples in the verification sample set, wherein each group of samples in the verification sample set includes a defect image and a template image corresponding to the defect image, and the third defect label message includes third category information and third position information of defects at various locations in the defect image; when there are multiple adjusted models to be trained, determining the average precision mean of each of the multiple adjusted models to be trained through the verification sample set and the third defect label message; and determining the adjusted model to be trained with the largest average precision mean as the target defect detection model.

In some embodiments of the present invention, the step of the input module 120 obtaining a training sample set includes: obtaining a first defect image and a first template image corresponding to the first defect image; performing geometric transformation processing on the first defect image and the first template image using the same processing method, and using the first defect image after the geometric transformation processing as a defect image sample, and using the first template image after the combined transformation processing as a template image sample, wherein the processing method includes at least one of the following: random cropping, image flipping, and image stitching.

In some embodiments of the present invention, the category information includes the defect category number of the defect, and the location information includes the location information of the defect box corresponding to the defect.

It should be noted that each module in the above-mentioned defect detection model training device can be a program module (for example, a set of program instructions to implement a certain specific function) or a hardware module. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

The embodiment of the present invention also provides a non-volatile storage medium. The non-volatile storage medium stores a program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the following defect detection model training method: obtain a training sample set and a first defect label message corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label message includes a first category message and a first position message of defects at various locations in the defect image; determine a target loss function, wherein the target loss function includes a defect category loss function and a defect position loss function ; The random gradient descent method is adopted to train the model to be trained through the training sample set, defect label information and target loss function to obtain the target defect detection model, wherein the target defect detection model is set to determine the defect position information and defect category information in the defect image to be detected, and the model to be trained is set to extract the first image feature of the defect image and the second image feature of the template image, and output the second label information of the defect image according to the first image feature and the second image feature, and the second label information includes the second category information and the second position information of the defects in each defect image determined by the model to be trained.

In some embodiments of the present invention, when the program is running, the device where the non-volatile storage medium is located can also be controlled to execute the following defect image classification method: determine the defect image to be detected and the template image corresponding to the defect image to be detected; input the defect image to be detected and the template image into the target defect detection model to obtain the defect location information and defect category information output by the target defect detection model, wherein the target defect detection model is set to extract the defect image to be detected. The third image feature and the fourth image feature of the template image are used, and defect location information and defect category information are output based on the third image feature and the fourth image feature; a preset number of target defects in the defect image to be detected that are closest to the center point of the defect image to be detected are determined based on the defect location information; a category confidence of each target defect in the preset number of target defects is determined, and the defect category of the target defect with the largest category confidence is determined as the defect category of the defect image to be detected.

The embodiment of the present invention also provides an electronic device, which includes a memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the program executes the following defect detection model training method when it is run: obtaining a training sample set and a first defect label message corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label message includes a first category message and a first position message of defects at various locations in the defect image; determining a target loss function, wherein the target loss function includes a defect category loss function. The method adopts a random gradient descent method to train the model to be trained through the training sample set, the defect label information and the target loss function to obtain the target defect detection model, wherein the target defect detection model is set to determine the defect position information and the defect category information in the defect image to be detected, and the model to be trained is set to extract the first image feature of the defect image and the second image feature of the template image, and output the second label information of the defect image according to the first image feature and the second image feature, and the second label information includes the second category information and the second position information of the defects in each defect image determined by the model to be trained.

In some embodiments of the present invention, the following defect image classification method can also be executed when the program is running: determine a defect image to be detected and a template image corresponding to the defect image to be detected; input the defect image to be detected and the template image into a target defect detection model to obtain defect location information and defect category information output by the target defect detection model, wherein the target defect detection model is configured to extract a third image feature of the defect image to be detected and a fourth image feature of the template image, and output defect location information and defect category information based on the three image features and the fourth image feature; determine a preset number of target defects in the defect image to be detected that are closest to the center point of the defect image to be detected based on the defect location information; determine the category confidence of each target defect in the preset number of target defects, and determine the defect category of the target defect with the largest category confidence as the defect category of the defect image to be detected.

In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

In the several embodiments provided by the present invention, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There can be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some ports, units or modules, which can be electrical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

In addition, each functional unit in each embodiment of the present invention can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional unit.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the relevant technology, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage media include: flash drives, read-only memory (ROM), random access memory (RAM), removable hard drives, magnetic disks or optical disks, and other media that can store program code.

The above is only the best implementation of the present invention. It should be pointed out that ordinary technicians in this technical field can make some improvements and modifications without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention. [Industrial Applicability]

The defect detection model training method, device and electronic equipment provided by the embodiments of the present invention are applied in the field of image recognition. In the embodiment of the present invention, a training sample set and a first defect label information corresponding to each group of samples in the training sample set are obtained, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes a first category information and a first position information of defects at various locations in the defect image; a target loss function is determined, wherein the target loss function includes a defect category loss function and a defect position loss function; a target defect detection model is obtained by training a model to be trained using the training sample set, the defect label information and the target loss function, wherein the target defect detection model is used to determine the defect location information and the defect category information in the defect image to be detected, and the model to be trained is used to extract the defect image. The method comprises the following steps: a first image feature and a second image feature of a template image, and outputting a second label information of the defect image based on the first image feature and the second image feature, wherein the second label information includes second category information and second position information of defects at various locations in the defect image to be determined by the training model. By adopting a training sample set including defect images and template images and defect label information to train a defect detection model, the purpose of obtaining a defect detection model that can utilize the template image in the defect recognition process is achieved, thereby achieving the technical effect of utilizing the template image to assist in classification recognition in the process of defect recognition and classification of grain images, thereby solving the technical problem of poor recognition and classification results caused by not utilizing the template image in the related technology when performing defect recognition and classification on grain images.

10: Computer terminal 102: Processor 104: Memory 106: Transmission module 120: Input module 122: First processing module 124: Second processing module S202~S206, S402, S404, S502~S508, S602, S604, S902~S906, S1002~S1008: Steps

The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation on the present invention. In the diagram: Figure 1 is a schematic diagram of the structure of a computer terminal according to an embodiment of the present invention; Figure 2 is a schematic diagram of the process of a training method for a defect detection model according to an embodiment of the present invention; Figure 3a is a schematic diagram of a gold grain image according to an embodiment of the present invention; Figure 3b is a schematic diagram of a defect image and a template image according to an embodiment of the present invention; Figure 4 is a schematic diagram of the process of a method for obtaining a training sample set according to an embodiment of the present invention; Figure 5 is a schematic diagram of the process of a training method for training a defect detection model based on a training sample set and defect label information according to an embodiment of the present invention; Figure 6 is a schematic diagram of the process of a method for obtaining a second label information according to an embodiment of the present invention; Figure 7 is a schematic diagram of the model structure of a defect detection model according to an embodiment of the present invention; Figure 8 is a schematic diagram of the model structure of another defect detection model according to an embodiment of the present invention; Figure 9 is a schematic diagram of the process of a screening method of a defect detection model according to an embodiment of the present invention; Figure 10 is a schematic diagram of the process of a defect image classification method according to an embodiment of the present invention; Figure 11 is a schematic diagram of the process of a model training and classification process according to an embodiment of the present invention; Figure 12 is a schematic diagram of the structure of a training device for a defect model according to an embodiment of the present invention.

S202~S206: Steps

Claims (13)

A method for training a defect detection model, comprising: Obtaining a training sample set and a first defect label message corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label message includes a first category message and a first position message of defects at various locations in the defect image; Determining a target loss function, wherein the target loss function includes a defect category loss function and a defect position loss function; The random gradient descent method is adopted to train the model to be trained through the training sample set, the defect label information and the target loss function to obtain the target defect detection model, wherein the target defect detection model is used to determine the defect position information and defect category information in the defect image to be detected, and the model to be trained is used to extract the first image feature of the defect image and the second image feature of the template image, and output the second label information of the defect image according to the first image feature and the second image feature, and the second label information includes the second category information and the second position information of the defects in each part of the defect image determined by the model to be trained. The training method of the defect detection model as described in claim 1, wherein the random gradient descent method is used to train the model to be trained through the training sample set, the defect label information and the target loss function, and the step of obtaining the target defect detection model includes: The first step is to input the defect image and the template image into the model to be trained respectively, and obtain the second label information output by the model to be trained; The second step is to input the first category information and the second category information into the defect category loss function to obtain a first loss function value, and input the first position information and the second position information into the defect position loss function to obtain a second loss function value, and store the first loss function value and the second loss function value; The third step is to use the stochastic gradient descent method to adjust the model parameters of the model to be trained through the first loss function value and the second loss function value; The fourth step is to obtain all the stored first loss function values and the second loss function values, and when the first loss function value and the second loss function value meet the preset conditions, determine that the adjusted model to be trained is the target defect detection model, otherwise jump to the first step, wherein the preset conditions include that the sum of the first loss function value and the second loss function value of a continuous preset number are all in the target value interval. The training method of the defect detection model as described in claim 2, wherein the step of inputting the defect image and the template image into the model to be trained respectively and obtaining the second label information output by the model to be trained comprises: Inputting the defect image and the template image into the backbone network of the model to be trained, wherein the backbone network is configured to extract image features from the defect image and the template image, thereby obtaining a defect feature image and a template feature image; Obtaining the second label information output by the model to be trained based on the defect feature image and the template feature image. The training method of the defect detection model as described in claim 3, wherein the model to be trained further includes a feature fusion layer and a feature pyramid layer, The feature fusion layer is configured to obtain a first target feature image based on the defect feature image and the template feature image, and input the first target feature image into the feature pyramid layer; The feature pyramid layer is configured to output the second label information based on the first target feature image. The training method of the defect detection model as described in claim 4, wherein the feature fusion layer includes a first feature fusion layer, The first feature fusion layer is configured to perform a differential operation on the defect feature image and the template feature image to obtain a differential feature image, wherein the pixel value of any pixel point in the differential feature image is equal to the absolute value of the difference between the defect pixel point and the template pixel point corresponding to the any pixel point, the defect pixel point is the pixel point in the defect feature image corresponding to the any pixel point, and the template pixel point is the pixel point in the template feature image corresponding to the any pixel point; channel stitching processing is performed on the differential feature image and the defect feature image to obtain a second target feature image; the second target feature image is convoluted by target convolution to obtain the first target feature image. The training method of the defect detection model as described in claim 4, wherein the feature fusion layer includes a second feature fusion layer, The second feature fusion layer is configured to perform convolution processing on the defect feature image and the template feature image respectively through a target convolution kernel, and generate the first target feature image according to the defect feature image and the template feature image after the convolution processing, wherein the pixel value of any pixel point in the first target feature image is equal to the sum of the pixel values of the defect pixel point and the template pixel point corresponding to the any pixel point. The defect detection model training method as described in claim 2, wherein the step of determining that the adjusted model to be trained is the target defect detection model includes: Obtaining a verification sample set and a third defect label message corresponding to each group of samples in the verification sample set, wherein each group of samples in the verification sample set includes a defect image and a template image corresponding to the defect image, and the third defect label message includes a third category message and a third position message of defects at various locations in the defect image; When there are multiple adjusted models to be trained, determining the average precision mean of each of the multiple adjusted models to be trained through the verification sample set and the third defect label message; Determining the adjusted model to be trained with the largest average precision mean as the target defect detection model. The training method of the defect detection model as described in claim 1, wherein the step of obtaining the training sample set includes: Obtaining a first defect image and a first template image corresponding to the first defect image; Performing geometric transformation processing on the first defect image and the first template image using the same processing method, and using the first defect image processed by the geometric transformation as the defect image sample, and using the first template image processed by the geometric transformation as the template image sample, wherein the processing method includes at least one of the following: random cropping, image flipping, and image splicing. A defect detection model training method as described in claim 1, wherein the category information includes a defect category number of the defect, and the location information includes location information of a defect box corresponding to the defect. A defect image classification method includes: Determining a defect image to be detected and a template image corresponding to the defect image to be detected; Inputting the defect image to be detected and the template image into a target defect detection model to obtain defect location information and defect category information output by the target defect detection model, wherein the target defect detection model is used to extract the third image feature of the defect image to be detected and the fourth image feature of the template image, and output the defect location information and the defect category information according to the third image feature and the fourth image feature; Determining a preset number of target defects in the defect image to be detected that are closest to the center point of the defect image to be detected according to the defect location information; Determining the category confidence of each target defect in the preset number of target defects, and determining the defect category of the target defect with the largest category confidence as the defect category of the defect image to be detected. A training device for a defect detection model, comprising: An input module, configured to obtain a training sample set and a first defect label message corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label message includes a first category message and a first position message of defects at various locations in the defect image; A first processing module, configured to determine a target loss function, wherein the target loss function includes a defect category loss function and a defect position loss function; The second processing module is configured to use a random gradient descent method to train the model to be trained through the training sample set, the defect label information and the target loss function to obtain a target defect detection model, wherein the target defect detection model is configured to determine the defect location information and defect category information in the defect image to be detected, and the model to be trained is configured to extract the first image feature of the defect image and the second image feature of the template image, and output the second label information of the defect image according to the first image feature and the second image feature, and the second label information includes the second category information and the second location information of the defects in each location in the defect image determined by the model to be trained. A non-volatile storage medium, wherein a program is stored in the non-volatile storage medium, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the defect detection model training method as described in any one of claim items 1 to 9, or the defect image classification method as described in claim item 10. An electronic device comprises: a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein when the program is run, the defect detection model training method described in any one of claim items 1 to 9 or the defect image classification method described in claim item 10 is executed.

Images