JP2022548438A - Defect detection method and related apparatus, equipment, storage medium, and computer program product - Google Patents

Abstract

The present application discloses a defect detection method, related apparatus, equipment and storage medium. The defect detection method is to perform a first target detection on the image to be detected and obtain a target detection result, the target detection result being a first class target and a first class in the image to be detected. obtaining an image area containing the target of the first class based on the first position information; performing defect detection on the image area; obtaining defect detection results for the target. According to the above solution, it is possible to reduce the defect detection omission rate and the false detection rate.

Description

(Cross reference to related applications)
This application claims priority from a Chinese patent application with application number 202010837709.2 filed on Aug. 19, 2020, the entire content of which is incorporated herein by reference.

The present application relates to the field of image processing, but is not limited to it, and more particularly to defect detection methods and related apparatus, equipment, storage media, and computer program products.

Currently, in order to reinforce the protection of equipment in the environment, it is generally necessary to perform fault detection on the equipment in the environment to ensure that defects in the equipment are immediately discovered, and to perform further maintenance.

In the related art, equipment defect detection involves taking an image of the equipment and manually inspecting the image to determine if a defect is present. However, when performing manual defect detection, detection omissions or erroneous detections are likely to occur due to human carelessness. Therefore, how to perform defect detection in order to reduce the defect detection omission rate and improve the accuracy of defect detection is now an extremely important issue.

Embodiments of the present application provide at least defect detection methods and related apparatus, apparatus, storage media, and computer program products.

Embodiments of the present application provide a defect detection method. The method is performing a first target detection on an image to be detected to obtain a target detection result, the target detection result comprising a first class target and a first class target in the image to be detected. obtaining an image region including the first class target based on the first position information; performing defect detection on the image region to detect the first class target; and obtaining defect detection results associated with.

Therefore, first, the position of the first class target in the image to be detected is detected, and then defect detection is performed on the image region containing the first class target. By performing defect detection directly on the image area where the target is located, the area of defect detection can be reduced and the false positive and false positive rates can be reduced.

In some embodiments of the present application, performing defect detection on an image region and obtaining defect detection results for a first class of targets may include performing a second target detection on an image region, performing a first defect detection, obtaining a result, the first defect detection result comprising information of at least one first defect present in the target of the first class; and/or performing classification on the image region; Obtaining a second defect detection result, the second defect detection result including information of a second defect to which the first class of targets belongs.

Therefore, by performing detection on an image region containing the first class targets using a plurality of methods, different detection methods can be used for different first class targets, making defect detection more objective. can have. The resulting defect detection results are more accurate. or the final defect detection result of the image to be detected obtained by performing different detection schemes on the same first class target and then combining each different defect detection scheme is more accurate, Further reduce the false positive rate.

In some embodiments of the present application, performing defect detection on an image region and obtaining defect detection results for a first class of targets may include: if the first class target is a first subclass target, the image region performing a second target detection on the image region to obtain a first defect detection result; and performing the step of obtaining defect detection results.

Therefore, using different defect detection schemes for different target objects makes defect detection more flexible. Compared to using the same detection scheme for all types of targets, it is more purposeful and results in more accurate defect detection results.

In some embodiments of the present application, the first defect detection result includes location information of various first defects and a first probability belonging to the first defect; After obtaining the defect detection result, the method respectively determines the defect areas of each kind of first defect in the image to be detected according to the position information of each kind of first defect, and determines the overlapping area between the different defect areas. and filtering first defects whose first probability satisfies a first filtering condition.

Therefore, obtaining position information in an image region containing the probability of the first defect in the first subclass of targets and the first subclass of targets, and corresponding to the first defect in the image to be detected based on the position information. By finding the areas to be processed and de-duplicating the overlapping parts, one first defect corresponds to one defect area and the problem of multiple subsequent processes for one defect is reduced. , to improve the accuracy rate of the final output. By removing regions that satisfy the filtering conditions, the number of regions to be subsequently processed can be reduced and processing efficiency can be improved.

In some embodiments of the present application, the first filtering condition is that the first probability is lower than the first probability threshold, and/or performing de-duplication on overlapping regions between different defect regions. involves performing non-maximal suppression on overlap regions between different defect regions.

Therefore, by removing those with low defect probabilities, many unnecessary processing areas are reduced and processing efficiency is improved. Deduplication using the non-maximal value suppression method can leave defective areas with a higher defect probability, thereby improving the processing efficiency and at the same time improving the accuracy rate of defect detection.

In some embodiments of the present application, the second defect detection result includes a second probability that the target of the first class belongs to the second defect, and performing classification on the image region to obtain the second defect detection result. performs a classification on the image region to obtain the probability that the target of the first class belongs to each of the given defects, and adds the various given probabilities that the target of the first class belongs to the second defect. obtaining a second probability.

Therefore, the defect probability of the target of the first class is determined by summing the probabilities of a plurality of types of predetermined defects. The defect probabilities of the first class targets are made more accurate by broadly considering the likelihood that the first class targets belong to various given defects.

In some embodiments of the present application, performing first target detection on the image to be detected and obtaining target detection results utilizes a first region detection network of the defect detection model to detect Performing a first target detection on the image to obtain a target detection result; Performing a second target detection on the image region to obtain the first defect detection result comprises detecting a second region of the defect detection model using a network to perform second target detection on an image area to obtain a first defect detection result, wherein the depth of the second area detection network is shallower than the depth of the first area detection network. Classifying the image region to obtain the second defect detection result includes classifying the image region to obtain the second defect detection result using a classification network of defect detection models. Including getting.

Therefore, by taking into account the feature that different defects may be present in each different component, and performing stepwise detection using different network models, the false detection rate is reduced when using a single detection network. To some extent, the problem of high At the same time, since the depth of the network model used for the second target detection is shallower than the depth of the network model used for the first target detection, the amount of computation in the second target detection process is reduced and the detection efficiency is improved. can be done.

In some embodiments of the present application, the target detection result further includes a second class of targets and second position information of the second class of targets in the image to be detected; After performing one target detection and obtaining the target detection result, the method determines the second class of targets as the third defect in the image to be detected, and converts the second class of targets and their second position information to the second It further includes making the defect detection result related to the target of the class.

Therefore, one defect detection model can detect a plurality of types of targets by making the second class targets, which should not exist, the third defects, thereby improving the adaptability of the defect detection model. .

In some embodiments of the present application, after obtaining the defect detection results, the method includes filtering defects satisfying a second filtering condition in the defect detection results to obtain a final defect detection result for the image to be detected. Including further.

Therefore, by filtering unsatisfied conditions, the number of apparently unwarranted defects is reduced and the final defect detection results obtained are more accurate.

In some embodiments of the present application, filtering the defects that satisfy the second filtering condition in the defect detection results includes obtaining the dimensions of the defects by using the position information of the defects in the defect detection results, and obtaining the dimensions of the defects in the defect detection results. does not meet a predetermined dimensional condition and/or filtering defects whose probability in the defect detection result is lower than a second probability threshold.

Therefore, by filtering those whose dimensions or probabilities are unsatisfactory, apparently unwarranted defects are reduced and the final defect detection results obtained are more accurate.

In some embodiments of the present application, after obtaining the final defect detection result of the image to be detected, the method performs: Further including outputting information for each defect.

Therefore, outputting the information of each defect in the final defect detection result according to the descending order of the probability of each defect makes the output result clearer and contributes to subsequent further observation.

In some embodiments of the present application, obtaining an image region containing a target of the first class based on the first positional information includes: expanding the target area outward by a predetermined multiple in the image to be detected; and extracting the outwardly expanded target area from the image to be detected. , obtaining an image region containing the first class of targets.

Thus, by extending the target region outward by a predetermined factor in the image to be detected, some background information can be left after obtaining one containing the target region corresponding to the first class of targets. , the accuracy of defect detection for first class targets can be improved.

In some embodiments of the present application, before performing first target detection on the image to be detected and obtaining target detection results, the method includes scaling the initial image to obtain an image of a predetermined size. , compressing the pixel values of the scaled initial image to a predetermined pixel value range; and performing a normalization process on the compressed initial image to obtain the image to be detected.

Therefore, by operating on the dimensions or pixels of the initial image obtained, a uniform image style is obtained and the robustness of the input image is improved to some extent.

In some embodiments herein, the first target detection and defect detection are performed by a defect detection model, the defect detection model at least obtaining a first sample set, the first sample set comprising at least one a sample image, wherein the sample image is annotated with true defect information about the target, the target comprising a first class of targets; and obtaining defect detection results for the target, and using the true defect information and defect detection results to adjust the parameters of the defect detection model.

Therefore, the accuracy of the defect detection model trained by the above method is higher, thereby performing the first target detection and defect detection, making the detection result more accurate.

In some embodiments of the present application, the defect detection model is jointly trained by a plurality of graphics processors, each graphics processor obtains a different first sample set, and the defect detection model is used to obtain a first Performing detection on the sample images in the sample set to obtain defect detection results relative to the target includes: synchronizing batch normalization layers of defect detection models in each graphics processor; Using the model to perform detection on a sample image to obtain defect detection results for a target, and using real defect information and defect detection results to adjust the parameters of the defect detection model. , determining an average loss value of the defect detection model based on all defect detection results and true defect information; and using the average loss value to adjust parameters of the defect detection model.

Therefore, by synchronizing the batch normalization layers of the defect detection model on multiple graphics processors, the global first sample set can be used for normalization, corresponding to an increase in batch size. This reduces the impact of using multiple graphics processors. Also, training the defect detection model with multiple graphics processors speeds up the training.

In some embodiments of the present application, the defect detection model is used to perform detection on the sample images in the first sample set, and before obtaining defect detection results for the target, the method performs detection on the second sample set. pre-training a defect detection model using the first sample set; pre-training a defect detection model using the first sample set; The learning rate used in the training process gradually increases from a first learning rate lower than the predetermined learning rate to the predetermined learning rate, and the sample images in the first sample set are at least It is obtained by preprocessing and obtaining a sample image, and the preprocessing method includes at least one of scale conversion, color conversion, horizontal flip, vertical flip, rotation, and cropping.

Therefore, before formally training the defect detection model, first initialize the parameters of the defect detection model by pre-training the defect detection model using the second set of samples, or Prior to formal training, we initialize the learning rate and perform preprocessing on the original images to improve the adaptability of the defect detection model.

Embodiments of the present application provide a defect detection apparatus. The apparatus comprises a first target detection module configured to perform a first target detection on an image to be detected and obtain a target detection result, the target detection result comprising a first class of targets and detections a first target detection module comprising first position information of a first class target in an image to be scanned; and based on the first position information, configured to obtain an image region containing the first class target. An image area acquisition module and a defect detection module configured to perform defect detection on the image area and obtain defect detection results for a first class of targets.

An embodiment of the present application provides an electronic device. The electronic device comprises a memory and a processor, the processor configured to execute program instructions stored in the memory to perform some or all of the steps of the defect detection method.

Embodiments of the present application provide a computer-readable storage medium. The computer-readable storage medium stores program instructions which, when executed by a processor, cause the processor to perform some or all of the steps of the defect detection method.

An embodiment of the present application provides a computer program product. The computer program product includes a non-transitory computer-readable storage medium storing a computer program, which when read and executed by a computer, causes a computer to perform a part of the methods described in the embodiments herein. Or run all steps. The computer program product may be a software installation package.

In an embodiment of the present application, first, the position of the first class target in the image to be detected is detected, and then defect detection is performed on the image region containing the first class target. By performing defect detection directly on the image area where the target is located, the area of defect detection can be reduced and the false positive and false positive rates can be reduced.

It is to be understood that the general descriptions above and the detailed descriptions that follow are exemplary and explanatory only and are not restrictive.

4 is a flow chart illustrating a defect detection method according to an embodiment of the present application; It is a schematic diagram showing the state of the defect detection method according to the embodiment of the present application. FIG. 3 is a schematic diagram showing an image to be detected in a defect detection method according to an embodiment of the present application; FIG. 2 is a schematic diagram showing an image area in a defect detection method according to an embodiment of the present application; 1 is a schematic diagram showing an implementation flow of a defect detection method according to an embodiment of the present application; FIG. FIG. 4 is a schematic diagram illustrating an implementation flow of an image preprocessing module according to an embodiment of the present application; FIG. 3 is a schematic diagram illustrating a network structure of primary member detection modules according to an embodiment of the present application; FIG. 4 is a schematic diagram showing an implementation flow of vibration damper classification recognition according to an embodiment of the present application; FIG. 4 is a schematic diagram showing a network structure of a secondary insulator self-destruction detection module according to an embodiment of the present application; 1 is a schematic diagram showing the structure of a defect detection device according to an embodiment of the present application; FIG. 1 is a schematic diagram showing the structure of an electronic device according to an embodiment of the present application; FIG. 1 is a schematic diagram showing the structure of a computer-readable storage medium according to an embodiment of the present application; FIG.

The drawings attached hereto are taken into the specification and constitute a part of the specification, show the embodiments compatible with the application, and are used to interpret the technical solution of the application together with the specification.

Hereinafter, the solutions of the embodiments of the present application will be described in detail with reference to the drawings of the specification.

In the following description, specific details such as specific system architectures, interfaces, techniques, etc. are presented in order to better understand the embodiments of the present application, and are for the purpose of interpreting the present application only, and not limiting the present application. not for

As used herein, the term “and/or” is used to describe a related relationship between related objects, and indicates that there are three types of relationships. For example, A and/or B represents three cases: only A is present, A and B are present at the same time, and only B is present. In this specification, the character "/" generally indicates that the contextual object is "or". Also, "plurality" in this specification means two or more than two. Also, as used herein, the term "at least one" represents any one of the plurality or any combination of at least two of the plurality. For example, including at least one of A, B, and C means including any one or more elements selected from the set consisting of A, B, and C.

In order to better understand the defect detection methods provided in the embodiments of the present application, the following first describes the defect detection methods in the related art.

Taking fault detection in distribution grid transmission lines as an example, there are more than one million kilometers of high-voltage transmission lines in our country. It is necessary to periodically conduct patrol inspection, curing and repair for these tracks. However, most of the high-voltage transmission lines are laid in unmanned areas, and in the case of patrol inspections of the distribution network, aerial photography of the transmission lines is taken by airplane, and the aerial video is observed with the naked eye. conduct an inspection. To improve efficiency, the related art employs image processing, machine learning based component defect detection solutions. In this solution, the detection failure rate is high, and the requirements for video shooting are high. In related art, drones fitted with high-definition cameras are used to take aerial photographs of power lines to improve the quality of the overall collection and reduce costs. However, because defect detection is still performed manually, false positive and false positive rates are still high. As deep learning models have made breakthroughs in the computer vision domain, performance indicators for each basic task in computer vision have improved significantly. All models are aimed at tasks involving academic datasets, and there is no detection and recognition solution designed solely for power patrols. The scale of the aerial image taken by the drone is extremely large, and the insulator self-destruction area, vibration damper, bird's nest, etc. detected during patrol inspections of the power distribution network are all small targets, so abnormal defects can be detected directly. Targeted detection results in a large number of false positives and negative detections, and the detection effectiveness is extremely low.

The defect detection method in the above related art has the following problems. 1) The training process is cumbersome and has a high false positive rate and false positives. 2) Test speed is slow because CPU is used for inference. 3) We have not refined the algorithm for power scenes. 4) It is only possible to independently detect and recognize one type of component or defect, such as, for example, insulator self-destruction, vibration damper rusting, and the like.

Referring to FIG. 1a, FIG. 1a is a flowchart illustrating a defect detection method according to an embodiment of the present application. The method may include the following steps.

In step S11, first target detection is performed on the image to be detected to obtain a target detection result, the target detection result includes the first class target and the first target of the first class target in the image to be detected. Contains location information.

In some embodiments of the present application, the image to be detected is first acquired before performing first target detection on the image to be detected. Referring to FIG. 1b, FIG. 1b is a schematic diagram illustrating the state of a defect detection method according to an embodiment of the present application. As shown in FIG. 1b, the image acquisition device acquires an image to be detected of each object to be detected in a specific environment. For example, an image to be detected containing targets of the first class is acquired. The image acquisition equipment then transmits the image to be detected to the defect detection equipment. Here, the defect detection device performs defect detection on the image to be detected transmitted from the image acquisition device according to the methods described in the embodiments of the present application. A first class of targets is the equipment to be serviced. If the device really exists, it can be a first class target for the embodiments of this application. For example, a traffic light at an intersection in a city, a sewer manhole cover, and so on. An image to be detected, which may include a traffic light or a manhole cover, may be acquired by the image acquisition equipment, and then an image to be detected, which may include the traffic light or the manhole cover, may be transferred to the defect detection equipment. input, and obtain the defect detection result of the signal lamp.

For example, in a power grid transmission line, it may be necessary to perform inspections on various pieces of equipment on the line. In this case, it is possible to acquire an aerial photographed image of each piece of track equipment on the track by a method such as photography. For example, an aerial image of an insulator string or a vibration damper, or an aerial image of a bird's nest is obtained. Subsequently, the acquired aerial image is input to the defect detection device as an image to be detected. The defect detection equipment can then perform defect detection on the image to be detected according to the methods described in the embodiments herein. Here, the image acquisition device and the defect detection device may be integrated as one device, or may be a plurality of distributed devices, and are not specifically limited here.

In some embodiments of the present application, the initial image can be scaled to an image of a predetermined size before performing primary target detection on the image to be detected. Here, the predetermined size includes a width-to-height ratio of (1.1 to 1.7):1 and/or one side of the small dimension being greater than 1000. The width to height ratio used in the examples of this application is 1.5:1 and the minimum side is 1200. Subsequently, the pixel values of the initial image after scaling are compressed to a predetermined pixel value range. For example, the pixel values of the initial image after scaling are compressed from 0-255 to 0-1. Subsequently, normalization processing is performed on the initial image after compression to obtain the image to be detected. Here, the method of normalization processing includes normalizing the initial image after compression using the mean value mean and the variance std of the ImageNet data set. This is as shown in the following formula (1-1).

where mean is the mean value of the dataset, std is the variance of the dataset, and pixelvalue is the value of each pixel in the image.

Of course, in some embodiments of the present application, other data sets may be used, such as, for example, the COCO (Common Objects in Context) data set. Further, the normalization method is not limited to the method of performing normalization by calling another data set, and other commonly used normalization methods may be used. For example, all pixel points in the initial image are traversed, the maximum and minimum values are recorded, and the initial image is normalized using the maximum and minimum values as parameters. Also, normalization may be performed using the sigmoid function. Accordingly, embodiments of the present application do not specifically limit the normalization scheme. First, the input robustness can be improved by performing the above-described preprocessing on the acquired images and unifying the style of the images.

In an embodiment of the present application, a first region detection network of a defect detection model performs first target detection on an image to be detected to obtain a target detection result.

Here, the defect detection model may be trained by the following steps.

First, obtain a first sample set, wherein the first sample set includes at least one sample image, the sample image annotated with true defect information associated with a target, where the target is a first It includes one class of targets and may of course include other classes of targets, such as, for example, a second class of targets. The sample images in the first sample set may contain targets of the first class as well as targets of other classes. Here, the true defect information of the target may be information indicating that the target is normal and has no defect, or information indicating that the target of the first class has a specific defect. There may be. That is, in the embodiments of the present application, a defect detection model can be trained using normal sample images and abnormal sample images with defects to obtain a highly adaptive defect detection model. The sample images in the first sample set may be obtained by preprocessing the original images. The preprocessing method may be at least one of scale conversion, color conversion, horizontal flip, vertical flip, rotation, and trimming. In some embodiments, in the process of scaling the original image, the length and width of the original image are converted by a predetermined ratio. For example, scale conversion is performed while maintaining the aspect ratio of the original image at 1.5:1. Of course, in some embodiments of the present application, other aspect ratios may be maintained or any conversion may be performed. Therefore, the embodiments of the present application do not specifically limit the aspect ratio in the case of scale conversion of the original image. The color conversion may be at least one of luminance, saturation, and chromaticity conversion. For example, by transforming all the luminance, saturation and chromaticity of the original image, the same true defect information including the same target is input to the defect detection model in various formats to train the defect detection model. It enhances the adaptability of the defect detection model and improves the accuracy of defect detection. In some embodiments, the various preprocessing schemes described above may be used alone in the network at various stages of the defect detection model, or may be used in the network at all stages.

The defect detection model is then used to perform detection on the sample images in the first sample set to obtain defect detection results for the target. Here, the defect detection model may be jointly trained by multiple graphics processors, and the first sample set obtained by each graphics processor is different. The training speed is doubled by using multiple graphics processors to perform detection on the defect detection model. In some embodiments, the batch normalization layers of the defect detection models in each graphics processor are synchronized, and the batch normalization in each graphics processor utilizes the synchronized defect detection models to perform detection on the sample images, and the target Obtain defect detection results. One of the graphics processors collects the mean and variance of the batch normalization layers of the defect detection models in the other graphics processors, and by computation, obtains the total mean and variance. Subsequently, the graphics processor returns the resulting mean and variance to the batch normalization layers of the defect detection models in the remaining graphics processors, and by synchronizing the batch normalization layers of the defect detection models in the plurality of graphics processors. , the global first set of samples can be used to perform normalization, corresponding to an increase in batch size. This reduces the impact of using multiple graphics processors.

Finally, the real defect information and defect detection results are used to adjust the parameters of the defect detection model. In some embodiments, based on all defect detection results and true defect information, determine an average loss value for the defect detection model, and then use the average loss value to adjust the parameters of the defect detection model. do. By synchronously training a defect detection model with a plurality of graphics processors, the parameters of the defect detection models in all the graphics processors after training are matched. Training speed is increased by synchronously training a defect detection model with multiple graphics processors as compared to a single graphics processor.

In some embodiments of the present application, the defect detection model is used to perform detection on sample images in a first sample set, and a second sample set is used to obtain defect detection results for a target. , the defect detection model may be pre-trained, and/or the defect detection model may be pre-trained using the first sample set. Here, the learning rate used in the preliminary training process gradually increases from a first learning rate lower than a predetermined learning rate to a predetermined learning rate. For example, if the predetermined learning rate is 0.6, in pre-training a first learning rate of 0.3 is gradually increased to 0.4, followed by a constant increase to the predetermined learning rate of 0.6. can be done. Here, the second sample set may be a public training set. For example, it may be the general COCO training set and the ImageNet dataset. The samples used for preliminary training may be the same as the samples used for formal training. That is, in pre-training, we still use the first sample set. In some embodiments, after pre-training with a second set of samples to initialize the parameters, pre-training is performed using the first set of samples to continuously adjust the learning rate. The process of constantly adjusting the learning rate through pre-training also amounts to further optimizing the parameters of the defect detection model. Therefore, in the case of formal training, training can be performed with suitable parameters. In some embodiments, the first sample set used for the pre-training process may still be the first sample set obtained after the above preprocessing. This can also improve the adaptability of the defect detection model and make the trained defect detection model more accurate. In some embodiments, a gradual linear or exponential increase from a first learning rate to a predetermined learning rate corresponds to initialization of the learning rate. Thereby, the defect detection model can be trained at a suitable learning rate, and the training effect can be improved.

Here, the first region detection network in the embodiments of the present application includes a Faster R-CNN (Region with Convolutional Neural Networks) network, such as ResNet50. Of course, in other embodiments, other neural network models, such as the SSD (Single Shot MultiBox Detector) network model, can be used to perform the first target detection on the image to be detected, and , the depth is not limited to ResNet50, but may be ResNet101, ResNet200, etc., and is not specifically limited here.

The first area detection network used in the examples of this application is the ResNet50 network. A feature map of the image to be detected is extracted and obtained by inputting the image to be detected into the header network of the first region detection network. Here, the size of the feature map is 1/16 the size of the image to be detected. Subsequently, candidate frames are extracted from the feature map by Region Proposal Networks (RPN). A candidate window is a region where a first class target may exist. Here, the extraction of the candidate frame is actually the extraction of the pixel values of the four vertices of the candidate frame. Subsequently, the feature map of each candidate frame is obtained by processing the candidate frame and the feature map through a pooling layer. The steps specifically include: The information of the four coordinates of the candidate frame is sent to the pooling layer, the coordinates of the four vertices of the candidate frame are mapped to the feature map in the pooling layer, and the feature vector of each candidate frame is obtained by the fully connected layer. Here, these feature vectors represent feature information of candidate frames. Subsequently, frame regression is performed on the feature vector, classification is performed according to a predetermined class, and finally a target detection result is obtained. As can be seen, the first target detection is to detect whether the image to be detected contains the first class of targets. If there is a target of the first class, further obtain first position information of the target of the first class in the image to be detected. In addition, the reliability of the first class targets can also be obtained. The confidence is the probability that a detected first class target is truly a first class target.

The target detection result includes a first class target and first position information of the first class target in the image to be detected and/or a probability that the candidate frame belongs to the first class target. In some embodiments, the first position information includes position information in the image to be detected of candidate frames corresponding to targets of the first class. The first position information is position information of the four vertices of the candidate frame corresponding to the target of the first class. Here, the first class of targets may further include a first subclass of targets and a second subclass of targets. Here, the targets of the first subclass, the targets of the second subclass and the targets of the first class may have the same attribute, although they may not have a relationship of affiliation. For example, a first class of targets may be an instrument, a first subclass of targets may be another instrument, and a second subclass of targets may be another instrument. The same attribute is being able to tell if there is a defect in the equipment by taking a picture and then analyzing the corresponding image. The probability of belonging to the first class of targets may further comprise the probability of belonging to the first subclass of targets and/or the probability of belonging to the second subclass of targets.

In some embodiments of the present application, the target detection result is second position information of the first class target and the second class target in the image to be detected and/or the probability of belonging to the second class target. further includes The second class of targets here may be targets that should not exist but exist. Its existence is flawed. Of course, in some embodiments, once the image to be detected is obtained after processing the initial image, the target detection results obtained after performing the first target detection on the image to be detected may include the first A first positional information in the initial image of the target of the class and/or the target of the second class is included. By further setting targets of the second class, when the same algorithm is used to detect defect conditions in a plurality of different target objects, multiple algorithms are used for different parts or devices in the same image. Therefore, it is not necessary to perform defect detection for each device, which greatly simplifies the operation steps for defect detection.

By way of example, the initial image may be an image of power lines taken aerially by a drone, or a graphic frame cropped from a high-definition video. First, processing is performed on the processed image. The processing includes scaling the initial image. The initial image is scaled to be a 1200*1800 image, then the pixel values of the initial image after scaling are compressed to a predetermined pixel value range of 0-1, and the mean and variance std of the ImageNet data set are is used to perform a normalization operation on the initial image after compression to obtain the image to be detected. Then, input the image to be detected into the ResNet50 network, obtain the probability that the target object in each candidate frame in the image to be detected corresponds to each predetermined target class, and obtain the probability that the target object in each candidate frame in the image to be detected corresponds to each predetermined target class. is selected as the class of the target object in the candidate pane, and at the same time the position information of each candidate pane is output. Here, in transmission lines, common members include vibration dampers, insulator strings, i.e. targets of the second class of extrinsic defects whose presence is a defect when they should not exist, i.e. May also contain bird nests. Here, in the embodiments of the present application, the vibration damper and the insulator string are the targets of the first class, and the bird's nest is the target of the second class. That is, after inputting the image to be detected to ResNet 50, the output target detection results include vibration dampers, insulator strings or bird nests, and their corresponding position information and associated probabilities in the image to be detected. may contain. The probability here is the probability that the candidate frame belongs to the vibration damper, the insulator string, or the bird's nest.

In some embodiments of the present application, after obtaining the target detection result, the second class target is determined as the third defect in the image to be detected, and the second class target and the second position information are The defect detection result related to the target of For example, if the second class target is a bird's nest and the bird's nest has been detected from the image to be detected, determine that the bird's nest is the third defect in the image to be detected, and Obtaining second position information of the nest of the nest in the image to be detected. Therefore, the final target detection result includes the bird's nest and the second location information of the bird's nest.

In step S12, an image region containing a first class target is obtained based on the first position information.

In an embodiment of the present application, a target area corresponding to a first class of targets is determined from the image to be detected based on the first position information. Specifically, it includes determining a region corresponding to the first class target from the image to be detected, that is, determining a candidate frame corresponding to the first class target. Subsequently, the target area is extended outward by a predetermined factor in the image to be detected. Here, the method of expanding outward includes leaving the center point of the target area as it is and increasing the length and width of the target area by 1.1 to 1.5 times that of the original target area. Here, in the embodiment of the present application, the length and width of the target area are set to about 1.2 times the original target area. If the boundary of the target region exceeds the boundary of the image to be detected, leave the part in to be detected. Subsequently, the outwardly extended target region is extracted from the image to be detected to obtain the image region containing the targets of the first class. Specifically, it involves cropping the outwardly extended target region from the image to be detected.

Referring to FIGS. 2 and 3 at the same time, FIG. 2 is a schematic diagram showing an image to be detected in the defect detection method according to an embodiment of the present application, and FIG. 1 is a schematic diagram showing the . As shown in FIGS. 2 and 3, each image 1 to be detected contains targets 100 of the first class. Here, the first target detection network obtains the position information of the first class target 100, and then, based on the position information of the first class target 100, the image area 110 including the first class target 100 is detected. get. To more clearly explain the relationship between the image area 110 and the candidate frames 101, the candidate frames 101 of the first class target 100 are shown in the image region of FIG. When acquiring the image area 110 including the target 100 of the first class from the image 1 to be detected, when acquiring the image area 110 including the target 100 of the first class, the candidate frame 101 of the target 100 of the first class is expanded outward at a constant ratio in the image 1 to be detected, the area of the resulting image area 100 is larger than the area of the original location area of the candidate frame 101 .

In some embodiments of the present application, the detected image is obtained by performing scaling, pixel value compression and normalization processing on the initial image, based on the first position information, the first class Determining a target region corresponding to the first class target in the initial image based on first position information of the first class target in the initial image, i.e., determining the position of the candidate frame corresponding to the first class target in the initial image, expanding the target region outward by a predetermined multiple in the initial image, extracting the outwardly expanded target region from the initial image; The goal is to obtain an image region containing one class of targets. Of course, since the image to be detected corresponds to the initial image, in some other embodiments of the present application, the detected image may undergo a series of scaling, pixel value compression and normalization operations on the initial image. The specific step of obtaining the image region containing the first class target based on the first position information, even if obtained in may include extracting the Thus, from the image to be detected, an image region containing the targets of the first class may be obtained, and from the initial image, an image region of the targets of the first class may be obtained, where specifically Not limited.

Thus, by extending the target region outward in the image to be detected by a predetermined multiple, some background information can be left after obtaining the target region corresponding to the first class of targets, and the first The accuracy of defect detection for a class of targets can be improved.

In some embodiments of the present application, after obtaining an image region containing the first class of targets, it is necessary to preprocess the image region, ie adjust the dimensions of the image region. Here, the aspect ratio of the image area can be adjusted to 1:1. If the image regions were obtained from the initial image, then it is necessary to perform normalization on the image regions. If it is obtained from the image to be detected after normalization processing, normalization may not be performed here.

For example, based on the first position information of the insulator string in the image to be detected, determine the corresponding candidate frame in the image to be detected of the insulator string, and move the candidate frame outward one step in the image to be detected. The image region containing the insulator can be obtained by dilating by a factor of .2 and subsequently extracting the outwardly dilated candidate frame from the image to be detected.

In step S13, defect detection is performed on the image area to obtain defect detection results related to the targets of the first class.

Methods for performing defect detection on image areas include at least one of the following. One approach is to perform a second target detection on the image area to obtain a first defect detection result, where the first defect detection result is at least one defect present in the first class of targets. Contains information on the first defect. Another way is to perform classification on the image area to obtain a second defect detection result, where the second defect detection result includes information of the second defect to which the target of the first class belongs. include. Here, the second target detection is to perform further target detection on the first class targets. What is obtained in the first target detection is the position information of the first class target, and the defect information of the first class target is not directly detected. Thus, secondary target detection provides intentional defect detection for targets of the first class. The classification for the image area is to classify the first class targets in the image area according to a predetermined defect class, and calculate the probability information that the first class target belongs to each defect class.

Detecting an image region containing the first class targets by multiple schemes makes the defect detection more objective by using different detection schemes for different first class targets. be able to. The resulting defect detection results are more accurate. Alternatively, by performing different detection schemes on the same first class target, and then combining the results obtained by each different defect detection scheme, the final defect information of the image to be detected is obtained, and the defect Make detection results more accurate.

wherein if the target of the first class is the target of the first subclass, performing second target detection on the image area to obtain a first defect detection result; If so, performing classification on the image region to obtain a second defect detection result.

Using different defect detection schemes for different target objects makes defect detection more flexible. Compared to using the same detection scheme for all types of targets, using different defect detection schemes for different target objects is more purposeful and results in more accurate defect detection results.

For example, a given insulator string is a target of a first subclass and a given vibration damper is a target of a second subclass. Therefore, when it is detected that the target of the first class is the insulator string, the second target detection is performed on the image area corresponding to the insulator string to obtain the first defect detection result. If the first class of target detected from the image to be detected is a vibration damper, classify for the image region corresponding to the vibration damper to obtain a second defect detection result. If the detected targets of the first class include the insulator string and also the vibration damper, the second target detection is performed on the image region corresponding to the insulator string, and the image region corresponding to the vibration damper is subjected to the second target detection. Classify. In some embodiments, defect detection for both may occur simultaneously. That is, the second target detection and classification may be performed simultaneously. Of course, in some embodiments, targets of the same first subclass are subjected to the second target detection and image region classification respectively, and then the results obtained by the two processing schemes are combined, and finally , a first defect detection result for a first subclass of targets.

The first defect detection result includes location information of various first defects and first probabilities belonging to the first defects. Here, the position information of the first defect is the position of the first defect in the image region containing the target of the first class, and the probability of belonging to the first defect is the reliability of the first defect obtained by detection. Yes, ie the probability that the first defect certainly belongs to the first defect. For example, by performing the second target detection on the image area including the insulator string, it is detected that there is a possibility that the insulator self-destruction occurred at position A in the insulator string. Here, the defect of insulator self-destruction belongs to the first defect. The A position in the insulator string belongs to the second position information of the insulator self-destruction, and the reliability of the insulator self-destruction is the first probability belonging to the insulator self-destruction.

In an embodiment of the present application, if the targets of the first class are the targets of the first subclass, the detection network used is the second region detection network when performing the second target detection on the image region, wherein , the depth of the second area detection network is shallower than the depth of the first area detection network. For example, if the first area detection network is ResNet50, the second area detection network may be ResNet18. Here, since both the network model used for detecting the second target and the network model used for detecting the first target are Faster R-CNN networks, the specific process of detecting the second target will not be described here. The difference from the first target detection is that in the first target detection, detection is performed on an image to be detected that may contain different first class targets, and the image to be detected has the first class determine whether the target is included, obtain a first position of the target of the first class, but in the second target detection, detect only the image region where the target of the first subclass is located, and the detection is Specifically, it includes detecting the probability that a defect exists in a first subclass of targets.

For example, after inputting the image region including the insulator string to the second region detection network ResNet 18, it is detected whether or not the insulator string has self-destructed. output the probability information of

In embodiments of the present application, after performing a second target detection on the image area and obtaining a first defect detection result, the first defect detection result may be processed. Here, it includes respectively determining the defect areas of the various first defects in the image to be detected according to the positional information of the various first defects. Since the position information obtained by the second target detection is the position information of the first defect in the image region containing the target of the first class, the mapping method determines the defect in the image to be detected of the first defect You also need to find an area. Subsequently, a predetermined process is performed on the defect area. Here, the predetermined processing is at least one of performing duplicate elimination processing on overlapping regions between different defect regions, and filtering first defects whose first probability satisfies a first filtering condition. including one. Here, the method of performing de-duplication processing on overlapping regions between different defect regions may further include performing non-maximal value suppression on overlapping regions between different defect regions. When the area of the overlapping regions reaches a predetermined area, obtaining a first probability of the defect regions overlapping each other, comparing the respective probabilities, leaving the defect regions with a high probability and discarding the defect regions with a low probability. Here, the first filtering condition is that the first probability is lower than the first probability threshold. That is, if the first defect probability is lower than the first probability threshold, remove the defect region corresponding to the first defect in the portion. Also, the position information and the first probability of the first defect are deleted from the first defect detection result at the same time. That is, the final first defect detection result is determined based on the defect area obtained only by the predetermined processing.

Obtaining position information in an image region containing the probability of the first defect in the first subclass of targets and the first subclass of targets, and based on the position information, the region corresponding to the first defect in the image to be detected. By finding and de-duplicating the overlapping parts, one first defect corresponds to one defect area, reducing the problem of multiple subsequent processes for one defect, and the final to improve the accuracy rate of typical output. By removing regions that satisfy the filtering conditions, the number of regions to be subsequently processed can be reduced and processing efficiency can be improved.

In some embodiments of the present application, removing those with low defect probabilities reduces many unnecessary processing areas and improves processing efficiency. Deduplication using the non-maximal value suppression method can leave defective areas with a higher defect probability, thereby improving the processing efficiency and at the same time improving the accuracy rate of defect detection.

If the first class target is a second subclass target, the second defect detection result includes a second probability that the first class target belongs to the second defect. In some embodiments, the second defect here refers to whether there is a defect in the second subclass of targets. Here, the second probability is the probability that a defect exists in the second subclass. Of course, in other embodiments, the second defect may include multiple different sub-defects. The second probability is the probability that a target of the second subclass belongs to each subdefect. The process of classifying on image regions includes classifying on image regions to obtain probabilities that targets of the first class belong to various given defects, i.e. targets of the second subclass belong to various given defects. , including obtaining the probabilities belonging to . For example, if a target in the first class is a target in a second subclass, such as vibration dampers, classification is performed for image regions containing vibration dampers, and the vibration dampers are normal vibration dampers, rusted vibration dampers, and vibration dampers, respectively. Get the probabilities belonging to damper torsion, vibration damper failure, and vibration damper dropout. For example, the probability that it belongs to a normal vibration damper is 0.1, the probability that it belongs to a rusted vibration damper is 0, the probability that it belongs to a vibration damper torsion is 0.2, and the probability that it belongs to a broken vibration damper is 0. .5 with a probability of belonging to the vibration damper dropout of 0. In this case, it can be obtained that the vibration damper has the highest probability of belonging to the given defect, vibration damper failure. In some embodiments of the present application, the probability allows the vibration damper to be attributed to the vibration damper failure. In an embodiment of the present application, after obtaining the probabilities that the targets of the first class belong to various predetermined defects, the probabilities of the various predetermined defects are added to obtain a second probability that the targets of the first class belong to the second defects. get That is, by adding the four types of defect probabilities of the vibration damper, it is known that the probability that the vibration damper has a defect is 0.7. The second defect in this case is a defect that exists in the vibration damper, and the second probability is the probability that the vibration damper is defective. Determining the defect probability for the first class of targets by summarizing a plurality of predetermined probabilities. The probabilities of the first class targets belonging to various given targets are broadly taken into account to make the defect probabilities of the first class targets more accurate.

In the process of performing classification on image regions, the network used is a classification network. Here, the depth of the classification network may be shallower than the depth of the first area detection network. The classification network in this case may similarly be ResNet18, or of course networks with other depths. For example, if the first region detection network is ResNet101, the classification network may be ResNet50. Of course, in some embodiments, the depth of the classification network may be the same as the depth of the first region detection network. Therefore, the selection of the classification network is not specifically limited here.

The problem of high false negatives when using a single detection network or classifier is fixed by performing stepwise detection using different network models, taking into account the different defect characteristics of each different component. can be reduced to the extent of At the same time, since the depth of the network model used for the second target detection is shallower than the depth of the network model used for the first target detection, the amount of computation in the second target detection process is reduced and the detection efficiency is improved. can be done.

In some embodiments of the present application, after obtaining defect detection results, defects in the defect detection results that satisfy a second filtering condition are filtered. Here, the defect detection result here may be a combination of the first defect detection result, the second defect detection result, and the defect detection result related to the target of the second class. The position information of each defect in the defect detection results is used to obtain the dimensions of the defects, and the defects whose dimensions in the defect detection results do not satisfy a predetermined dimension condition are filtered. For example, filter out insulator self-destruct defect areas or bird's nest areas and abnormal vibration damper locations that are too large or too small in size. Alternatively, filters defects whose probability in the defect detection result is lower than a second probability threshold. Specifically, different thresholds are set for targets of different classes. For example, a threshold of 0.3 is set for targets of the first subclass of the first target, a threshold of 0.2 is set for targets of the second subclass, and a threshold of 0.2 is set for targets of the second subclass. etc. Subsequently, the defects in the defect detection results are filtered based on a predetermined threshold. Of course, the defect size and defect probability in the defect detection results can be filtered simultaneously to reduce the number of defects in the defect detection results. Finally, information on each defect in the defect detection result is output according to the descending order of the probability of each defect in the final defect detection result. Here, the output defect information includes position information and probability of the defect in the image to be detected. Or directly output the location information and corresponding probabilities of the defects in the initial image.

In some embodiments of the present application, the form of output may include outputting, in the form of text, the position and probability of the defect in the image to be detected and/or the initial image; marking all defects in the image to be detected and/or the initial image and filling in the respective probability in a marker box, or each defect in the image to be detected and/or the initial image It may involve directly outputting the corresponding image regions and specifying the probabilities of defects in the image regions.

By filtering the defects in the defect detection results, the filtered defect detection results contain as little information as possible in normal cases or exclude clearly unreasonable situations, thereby improving the defect detection results. It improves the rationality of detection and reduces the false positive rate.

In the above solution, first, the position of the first class target in the image to be detected is detected, and then the defect detection is performed on the image area containing the first class target. By performing defect detection directly on the image area where the target is located, the area of defect detection can be reduced and the false positive and false positive rates can be reduced.

In the above methods of specific embodiments, the description order of each step means a strict execution order and does not limit the implementation process in any way, and the actual execution order of each step depends on its function and possible intrinsic logic. It should be understood by those skilled in the art that Here, the execution body of the defect detection method may be a defect detection device. For example, the defect detection method may be performed by a terminal device, server or other processing device. Here, the terminal equipment includes user equipment (UE), mobile equipment, user terminal, terminal, cellular telephone, cordless telephone, personal digital assistant (PDA), handheld equipment, computing equipment, vehicle-mounted equipment. , wearable devices, and the like. In some embodiments, the defect detection method may be implemented by a processor invoking computer readable instructions stored in memory.

An exemplary application of the embodiments of the present application to a real application scene is described below. Embodiments of the present application provide a multi-class fault detection method in power grid transmission lines based on deep learning. A deep learning model is used to perform two-stage detection and classification on large-scale input images. The method can simultaneously perform abnormal defect recognition such as insulator self-destruction detection, vibration damper defect detection, and bird's nest detection. As shown in FIG. 4a, the implementation flow of the method mainly consists of an RGB image input module 410, an image preprocessing module 420, a primary member detection module 430, a secondary vibration damper classification recognition module 440, a secondary insulator self-destruction detection module 450. , a self-destruct detection post-processing module 460 and an abnormal defect screening module 470 . Each module will be specifically described below.

1) RGB image input module. The module can take data output captured by different drone cameras and obtain RGB images or high-definition video captured by the drone aerially. For high-definition video, the image frames can be cut into corresponding RGB images.

2) Image preprocessing module. As shown in FIG. 4b, the module can perform unified processing on the RGB image to obtain the image to be input used for defect detection. The treatment process includes: a) Image scaling 421: Scales the input high-definition RGB image so that the image has a size of 1200×1800. b) Pixel Value Compression 422: Compresses the pixel values of the scaled image from 0-255 to 0-1. c) Pixel value normalization 423: Using the mean value mean and the variance std of the ImageNet data set, the pixel value compressed image is normalized according to the above equation (1-1).

3) Primary Member Detection Module. The input of the module is a normalized RGB image. Target detection network Faster R-CNN detects two classes of primary members: bird's nest, insulator string, and vibration damper, three classes of candidate regions: insulator string region, vibration damper region, and bird's nest region, and Obtain a score corresponding to each candidate region. The main network structure of the module uses the ResNet50 header network 431 as the backbone network of the detection network to extract the C4 layer feature map 432, as shown in Fig. 4c. The detection process of the primary member by the module includes the following. A C4 layer feature map 432 is obtained by processing the RGB image through a ResNet50 header network 431 . The size of the feature map is 1/16 of the input image. The ResNet50 header network 431 is constructed by stacking at least one convolutional layer, at least one normalization layer and subsampling layer, and performs forward operations in the form of skip connections. RPN-based candidate frame extraction 433 is performed on the C4 layer feature map 432, and the extracted candidate frame is an area where the target may exist. The candidate frames and the C4 layer feature map are processed by RoI pooling layer 434 to obtain a feature map for each candidate region, and further processed by at least one fully connected layer 435 to obtain a feature vector for each candidate region. The feature vector of the candidate area represents feature information of the candidate area. Finally, frame coordinate regression 436 and multi-class classification 437 are performed on the feature vector of each candidate region to obtain the final detection result, and the position regions and corresponding output the probability that Here, the bird's nest area is directly fed into the abnormal defect screening module, the insulator chain area is fed into the secondary self-destruction detection module for self-destruction detection, and the vibration damper area is fed into the secondary vibration damper classification recognition module for vibration Classify the damper class.

4) Secondary vibration damper classification recognition module. This module uses the vibration damper region detected by the primary member detection module as an input, and mainly uses a ResNet18 classification network to classify the vibration damper region trimmed from the original image into multiple abnormal types (classification class is also normal class). ), and finally matching the probabilities of the defect classes to obtain a normal vibration damper and a different abnormal vibration damper. The module classifies vibration dampers into five classes according to the form of the vibration dampers: normal vibration dampers, rusted vibration dampers, vibration damper torsion, vibration damper failure, and vibration damper dropout. Here, the latter four classes are defect classes. The process of vibration damper classification recognition by this module consists of four sub-parts: trimming the vibration damper region from the original image 441, image preprocessing 442, multi-classification by ResNet 18 443, and defect class probability matching 444, as shown in Fig. 4d. Including processes. For vibration damper regions detected from a single-stage member whose probability exceeds a first specified threshold, first crop the vibration damper region from the original image, then image for each cropped vibration damper region sub-image. Pre-treat and retain scale. Each sub-image is then input into a ResNet18 classification network for five classifications, and softmax calculates the probability values of the five vibration damper configuration classes. Finally, the defect class probabilities are matched, ie, the probability values of the vibration damper defects of the four classes are added to give the probability that the vibration damper is defective.

5) Secondary insulator self-destruction detection module. This module takes the insulator string region detected by the primary member detection module as input, mainly by Faster R-CNN detection network, performs self-explosion region detection for each insulator string region sub-image, and detects the possibility of self-explosion. Get a certain insulator. The main network structure of the module is shown in Fig. 4e, using a ResNet18 header network 451 as the backbone network of the detection network to reduce the amount of computation. First, trim the insulator string regions whose probability exceeds a second specified threshold, and perform image preprocessing on each resulting sub-image to preserve scale and obtain an RGB image for detection. Each RGB image is processed by a ResNet18 header network 451 to detect the self-destruct region of the insulator string to obtain a C4 layer feature map 452 . Candidate frame extraction 453 by RPN is performed on the C4 layer feature map 452 . The extracted candidate frame is a region in which continuous insulator self-destruction may exist. The candidate frame and the C4 layer feature map are processed by the RoI pooling layer 454 to obtain a feature map for each candidate region, and further processed by at least one fully connected layer 455 to obtain a feature vector for each candidate region. . Finally, frame coordinate regression 456 and multi-class classification 457 are performed on the feature vector of each candidate region to obtain the final detection result and output the self-destruct region of the insulator string and the corresponding probability value.

6) Self-destruct detection post-processing module. This module maps the self-destruction detection result in the insulator continuous region sub-image to the original image result, performs overlap suppression processing on the self-destruction region detected repeatedly a plurality of times, and obtains the position information of the insulator self-destruction region in the original image. . The processing process of the module is divided into the following three steps. a) Mapping the coordinates of the self-destruct region in the insulator sub-image onto the original image region. b) suppress overlapping self-destruct regions; c) Filter out self-destruct box with low score.

7) Abnormal defect sorting module. The module sorts out the bird's nest area, abnormal vibration damper, and insulator self-destruction detection recognition results to obtain a final abnormal defect detection result. The processing process of the module is divided into the following three steps. a) Filter the bird's nest area and the self-explosion area according to their dimensions to remove detection windows that are too large and too small. b) For bird's nest, insulator self-destruction and abnormal vibration damper, according to the respective class threshold, delete the results lower than the corresponding threshold. c) All remaining detection results are ordered globally according to the descending order of scores as output results.

In the embodiments of the present application, high-definition images taken by drones are used as input, and a lightweight deep learning detection and classification module is introduced to detect multiple defects such as insulator self-destruction, vibration dampers, and bird nest attachment. Anomalies are recognized in an integrated manner. It has the following beneficial effects: 1) In the grid fault detection algorithms in the related art, mainly using image processing and machine learning methods, it is necessary to construct hand-crafted features for the target a priori, which is not robust and accurate; The degree is low, and the influence of the shooting environment is large. In the embodiments of the present application, features are actively learned by the network based on a deep learning model. It has a wider application scene, stronger robustness, and higher accuracy. 2) The two-stage detection-detection, detection-classification algorithm proposed for the transmission line patrol inspection scene in the examples of the present application is suitable for high-definition images taken by drones, and the insulator self-destruction , solves the problem that the high-definition image detection and recognition effect of the single network detection or classifier in the related art is low by fully considering the defect feature of relying on the insulator string. 3) In the related art, one algorithm can only detect and recognize defects in one type of component. In order to detect multiple kinds of defects, it is necessary to repeatedly input the original image to different algorithms, which is inefficient. In the embodiments of the present application, multiple types of defects are detected and recognized by one algorithm, and the detection efficiency is greatly improved on the premise of ensuring accuracy.

Please refer to FIG. 5, which is a schematic diagram showing the structure of a defect detection apparatus according to an embodiment of the present application. The defect detection apparatus 30 comprises a first target detection module 31, an image area acquisition module 32, and a defect detection module 33, wherein the first target detection module 31 performs first target detection on an image to be detected. to obtain a target detection result, the target detection result including a first class of targets and first location information of the first class of targets in the image to be detected, and the region acquisition module 32 performs a first 1, the defect detection module 33 is configured to obtain an image region containing a first class target based on the position information, and performs defect detection on the image region to obtain a defect detection result related to the first class target; is configured to obtain

In the above solution, first, the position of the first class target in the image to be detected is detected, and then the defect detection is performed on the image area containing the first class target. By performing defect detection directly on the image area where the target is located, the area of defect detection can be reduced and the false positive and false positive rates can be reduced.

Here, the defect detection device 30 further comprises a pretreatment module (not shown).

In some embodiments of the present application, before the first target detection module 31 performs the first target detection on the image to be detected and obtains the target detection result, the preprocessing module scales the initial image, An image of a predetermined size is obtained, pixel values of the initial image after scaling are compressed to a predetermined pixel value range, normalization processing is performed on the compressed initial image, and an image to be detected is obtained. be done.

In the above solution, by performing processing on the dimensions or pixels of the initial image obtained, a uniform image style can be obtained and the robustness of the input image can be improved to a certain extent.

In some embodiments of the present application, the first target detection module 31 performs first target detection on the image to be detected and obtains the detection result using a first area detection network. performing a first target detection on the image to obtain a target detection result; performing a second target detection on the image region to obtain a first defect detection result comprising: a second region detection network; performing a second target detection on an image region to obtain a first defect detection result, wherein the depth of the second region detection network is greater than the depth of the first region detection network using Classifying the image region to obtain the second defect detection result comprises shallowly classifying the image region to obtain the second defect detection result using a classification network. include.

In the above solution, the stepwise detection using different network models, taking into account the different defect characteristics of each different component, results in a high false detection rate when using a single detection network. can be resolved to some extent. At the same time, since the depth of the network model used for the second target detection is shallower than the depth of the network model used for the first target detection, the amount of computation in the second target detection process is reduced and the detection efficiency is improved. can be done.

In some embodiments of the present application, the image region acquisition module 32 acquires an image region including a first class target based on the first location information to be detected based on the first location information. determining from the image a target region corresponding to the first class of targets; extending the target region outward by a predetermined multiple in the image to be detected; extracting the target region from the first class to obtain an image region containing the first class of targets.

In the above solution, leaving some background information after obtaining the one containing the target area corresponding to the first class of targets by extending the target area outward in the image to be detected by a predetermined factor. and improve the accuracy of defect detection for first class targets.

In some embodiments of the present application, defect detection module 33 is further configured to perform defect detection on the image area and obtain defect detection results for the first class of targets, comprising: to obtain a first defect detection result, wherein the first defect detection result includes information of at least one first defect present in the first class of targets; and/or performing classification on the image region to obtain a second defect detection result, wherein the second defect detection result includes information about the second defect to which the target of the first class belongs. including, including

In the above solution, by detecting the image area containing the first class targets by multiple methods, different detection methods are used for different first class targets, and the defect detection can be performed with higher accuracy. It can be purposeful. The resulting defect detection results are more accurate. or the final defect detection result of the image to be detected obtained by performing different detection schemes on the same first class target and then combining each different defect detection scheme is more accurate, Further reduce the false positive rate.

In some embodiments herein, defect detection module 33 is further configured to perform defect detection on the image area and obtain defect detection results for the first class of targets. performing a second target detection on the image region to obtain a first defect detection result if is a target of the first subclass; and if the target of the first class is a target of the second subclass: , performing classification on the image region to obtain a second defect detection result.

In the above solution, using different defect detection schemes for different target objects makes defect detection more flexible. Compared to using the same detection scheme for all types of targets, it is more purposeful and results in more accurate defect detection results.

In some embodiments of the present application, the first defect detection result includes location information of various first defects and a first probability belonging to the first defect, and the defect detection module 33 detects the second target for the image area. after obtaining the first defect detection result, according to the position information of the various first defects, respectively determine the defect areas of the various first defects in the image to be detected, and determine the overlap between the different defect areas At least one of performing a de-duplication process on the region and filtering first defects whose first probability satisfies a first filtering condition.

In the above solution, obtaining position information in an image region containing the probability of the first defect in the first subclass of targets and the first subclass of targets, and based on the position information, in the image to be detected, the first The problem of matching one first defect to one defect area by finding the area corresponding to the defect and performing de-duplication processing on the overlapping part, and performing multiple subsequent processes on one defect. and improve the accuracy rate of the final output. By removing regions that satisfy the filtering conditions, the number of regions to be subsequently processed can be reduced and processing efficiency can be improved.

In some embodiments of the present application, the first filtering condition is that the first probability is lower than the first probability threshold, and/or performing de-duplication on overlapping regions between different defect regions. involves performing non-maximal suppression on overlap regions between different defect regions.

In the above solution, by removing those with low defect probability, many unnecessary processing areas are reduced and the processing efficiency is improved. Deduplication using the non-maximal value suppression method can leave defective areas with a higher defect probability, thereby improving the processing efficiency and at the same time improving the accuracy rate of defect detection.

In some embodiments of the present application, the second defect detection result includes a second probability that the target in the first class belongs to the second defect, defect detection module 33 performing classification on the image region, the second defect Obtaining detection results includes: performing classification on the image region to obtain the probability that the first class of targets belong to various predetermined defects; and summing the various predetermined probabilities to obtain the obtaining a second probability belonging to the second defect.

In the above solution, the defect probability of the target of the first class is determined by aggregating the probabilities of a plurality of predetermined defects. The defect probabilities of the first class targets are made more accurate by broadly considering the likelihood that the first class targets belong to various given defects.

In some embodiments of the present application, the first target detection module 31 performs the first target detection on the image to be detected to obtain the target detection result using the first region detection network of the defect detection model. and performing a first target detection on the image to be detected to obtain a target detection result, and the defect detection module 33 performing a second target detection on the image area to obtain the first defect detection result. Obtaining a second target detection for the image region using a second region detection network of the defect detection model to obtain a first defect detection result, wherein the second region detection network is shallower than the depth of the first area detection network, performing classification on the image area to obtain a second defect detection result using the classification network of the defect detection model, Performing a classification on the image region to obtain a second defect detection result.

In the above solution, by taking into account the feature that each different component may have different defects, and detecting in stages using different network models, when using a single detection network: The problem of high false positive rate can be mitigated to some extent. At the same time, since the depth of the network model used for the second target detection is shallower than the depth of the network model used for the first target detection, the amount of computation in the second target detection process is reduced and the detection efficiency is improved. can be done.

In some embodiments of the present application, the target detection result further includes a second class of targets and second position information of the second class of targets in the image to be detected, and the first target detection module 31 detects After performing the first target detection on the image to be detected and obtaining the target detection result, further determining the second class target as the third defect in the image to be detected, and determining the second class target and its The second location information is configured to be defect detection results associated with a second class of targets.

In the above solution, by making the second class target that should not exist as the third defect, one defect detection model can detect a plurality of types of targets, and the adaptability of the defect detection model improve.

In some embodiments of the present application, after obtaining the defect detection results, the defect detection module 33 further filters defects that satisfy a second filtering condition in the defect detection results, and performs final defect detection of the image to be detected. Further including obtaining a result.

Filtering the unsatisfied conditions in the above solution reduces the number of apparently illegal defects and makes the final defect detection results more accurate.

In some embodiments of the present application, the detection module 33 filtering the defects satisfying the second filtering condition in the defect detection results includes using the defect location information in the defect detection results to obtain the dimensions of the defects, and the defect detection results. Filtering defects in the results whose dimensions do not meet a predetermined dimension condition and/or filtering defects whose probability in the defect detection result is lower than a second probability threshold.

In the above solution, by filtering those whose dimensions or probabilities do not meet the conditions, obviously unjustified defects are reduced and the finally obtained defect detection result is more accurate.

In some embodiments of the present application, after the defect detection module 33 obtains the final defect detection result of the image to be detected, further according to the descending order of the probability of each defect in the final defect detection result, the final It is configured to output information of each defect in the defect detection result.

In the above solution, outputting the information of each defect in the final defect detection result according to the descending order of the probability of each defect makes the output result clearer and contributes to subsequent further observation.

The defect detection apparatus further comprises a training module (not shown), the training module configured to train the defect detection model.

In some embodiments of the present application, the first target detection and defect detection are performed by a defect detection model, wherein the defect detection model at least obtains a first sample set, wherein: The sample set includes at least one sample image, the sample image annotated with true defect information about the target, the target including a first class of targets; performing detection on the sample images in the sample set to obtain target-related defect detection results; and using the true defect information and defect detection results to adjust the parameters of the defect detection model. was trained by

In the above solution, the accuracy of the defect detection model trained by the above method is higher, thereby performing the first target detection and defect detection, making the detection result more accurate.

In some embodiments of the present application, the defect detection model is jointly trained by multiple graphics processors, each graphics processor obtains a different first set of samples, and the training module utilizes the defect detection model to: , performing detection on the sample images in the first sample set to obtain defect detection results for the target includes: synchronizing the batch normalization layer of the defect detection model in each graphics processor; and synchronizing the batch normalization layer in each graphics processor. After using the defect detection model to perform detection on the sample image to obtain the defect detection results related to the target, the real defect information and the defect detection results are used to adjust the parameters of the defect detection model. determining an average loss value of a defect detection model based on all defect detection results and true defect information; using the average loss value to adjust parameters of the defect detection model; including.

In the above solution, by synchronizing the batch normalization layers of the defect detection model on multiple graphics processors, the global first sample set can be used for normalization, corresponding to an increase in batch size. This reduces the impact of using multiple graphics processors. Also, training the defect detection model with multiple graphics processors speeds up the training.

In some embodiments of the present application, the training module utilizes the defect detection model to perform detection on the sample images in the first sample set and to obtain the defect detection results for the target before performing detection on the second sample set. pre-training the defect detection model using the first sample set; and pre-training the defect detection model using the first sample set, wherein wherein the learning rate used in the pre-training process is from a first learning rate lower than the predetermined learning rate and gradually increases up to the predetermined learning rate, wherein the sample images in the first sample set are at least , preprocessing the original image to obtain a sample image, wherein the preprocessing method is at least one of scale conversion, color conversion, horizontal flip, vertical flip, rotation, and trimming. including one.

In the above solution, before formally training the defect detection model, first initialize the parameters of the defect detection model by pre-training the defect detection model using the second sample set; Or, before formal training, initializing the learning rate and preprocessing the original images to improve the adaptability of the defect detection model.

Referring to FIG. 5, FIG. 5 is a schematic diagram illustrating the structure of an electronic device according to an embodiment of the present application. The electronic device 40 comprises a memory 41 and a processor 42, the processor 42 being configured to execute program instructions stored in the memory 41 to implement the steps of the defect detection method embodiments described above. In one implementation, electronic device 40 may include, but is not limited to, a microcomputer, a server. Note that the electronic device 40 may include portable devices such as notebook computers and tablets, and is not limited to this.

Specifically, the processor 42 is configured to control itself and the memory 41 to implement the steps in any one of the defect detection method embodiments described above. The processor 42 may be called a CPU (Central Processing Unit). Processor 42 may be an integrated circuit chip with signal processing capabilities. Processor 42 may be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable gates. It may be an array, a discrete gate or transistor logic device, a discrete hardware component. A general-purpose processor may be a microprocessor, such as any conventional processor. Note that processor 42 may be co-embodied by an integrated circuit chip.

In the above solution, first, the position of the first class target in the image to be detected is detected, and then the defect detection is performed on the image area containing the first class target. By performing defect detection directly on the image area where the target is located, the area of defect detection can be reduced and the false positive and false positive rates can be reduced.

Referring to FIG. 6, FIG. 6 is a schematic diagram showing the structure of a computer-readable storage medium according to an embodiment of the present application. A computer readable storage medium 50 stores program instructions 501 executable by a processor, the program instructions 501 being configured to implement the steps of the above defect detection method embodiments.

Embodiments of the present application further provide a computer program product. The computer program product implements the method of any one of the above embodiments when executed by a processor. The computer program product may be implemented in hardware, software or a combination thereof. In some embodiments of the present application, the computer program product is embodied as a computer storage medium, and in some other embodiments of the present application, the computer program product is, for example, a Software Development Kit (SDK). ), etc., as a software product.

In the above solution, first, the position of the first class target in the image to be detected is detected, and then the defect detection is performed on the image area containing the first class target. By performing defect detection directly on the image area where the target is located, the area of defect detection can be reduced and the false positive and false positive rates can be reduced.

In some embodiments, the functions possessed by or the modules included in the apparatus provided by the embodiments of the present application may be configured to perform the methods described in the above method embodiments, the specific implementation of which includes: Reference can be made to the description of the embodiment of the above method, and for the sake of brevity, the detailed description is omitted here.

The description of each embodiment above emphasizes the differences between each embodiment, and the same or similar parts can be referred to each other, and the detailed description is omitted here for the sake of brevity.

It should be understood that the systems, devices and methods disclosed in some of the embodiments provided by the present invention can be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, for example, the division of the units is merely the division of logic functions, and other division schemes may be used in actual implementation. Also, for example, multiple units or components may be combined or incorporated into another system. Or some features may be ignored or not implemented. Also, the mutual couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some communication interface, device or unit, whether electrical, mechanical or Other forms are also possible.

Also, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist as a separate physical unit, or two or more units may be may be integrated in one unit. The integrated units may be implemented in the form of hardware or in the form of software functional units.

Note that integrated units may be implemented in the form of software functional units and stored in a computer readable storage medium when sold or used as stand-alone products. Based on this understanding, the technical solutions of the embodiments of the present application are essentially or part of the contribution to the prior art or part of the technical solutions are in the form of software products. , such computer software products may be stored in a storage medium, and may be stored in a single piece of computer equipment (such as a personal computer, server, or network device) or processor in accordance with the present application. It contains some instructions for performing all or part of the steps of the method described in each embodiment. The storage medium includes USB memory, removable hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, optical disk, and various other media capable of storing program code. including.

Embodiments of the present application provide a defect detection method, related apparatus, apparatus and storage medium. Here, the method is performing a first target detection on the image to be detected and obtaining a target detection result, the target detection result being a first class target and a first target in the image to be detected. obtaining an image region containing the first class target based on the first position information; performing defect detection on the image region; obtaining defect detection results for the targets of the . In the above means for solving the problem, the defect detection method provided by the embodiments of the present application performs defect detection on the image to be detected, thereby reducing the detection omission rate and the false detection rate.

An embodiment of the present application provides a computer program product. The computer program product includes a non-transitory computer-readable storage medium storing a computer program, which when read and executed by a computer, causes a computer to perform a part of the methods described in the embodiments herein. Or run all steps. The computer program product may be a software installation package.
For example, the present application provides the following items.
(Item 1)
A defect detection method comprising:
performing a first target detection on an image to be detected to obtain a target detection result, wherein the target detection result includes a first class target and the first class target in the image to be detected; including first location information for
obtaining an image region containing the first class of targets based on the first location information;
performing defect detection on the image area to obtain defect detection results for the first class of targets.
(Item 2)
performing defect detection on the image region and obtaining defect detection results for the first class of targets;
performing second target detection on the image area to obtain first defect detection results, wherein the first defect detection results are for at least one first defect present in the first class of targets; including information that
and/or performing classification on the image region to obtain a second defect detection result, wherein the second defect detection result includes information of a second defect to which the target of the first class belongs; characterized by including
The method of item 1.
(Item 3)
performing defect detection on the image area and obtaining defect detection results for the first class of targets;
performing a second target detection on the image region to obtain a first defect detection result if the first class of targets is a first subclass of targets;
performing classification on the image region to obtain a second defect detection result if the first class of targets is a second subclass of targets.
The method of item 2.
(Item 4)
The first defect detection result includes location information of various kinds of the first defects and a first probability belonging to the first defect, and performing second target detection on the image area to obtain a first defect detection result. After that, the method includes:
Defect areas of the various first defects in the image to be detected are determined according to the position information of the various first defects, and overlap elimination processing is performed on overlapping areas between the different defect areas. and
and filtering the first defects whose first probability satisfies a first filtering condition.
The method of item 2 or 3.
(Item 5)
the first filtering condition is that the first probability is lower than a first probability threshold; and/or
Performing de-duplication processing on overlapping areas between the different defect areas,
and performing non-maximal value suppression on overlapping regions between the different defect regions.
The method of item 4.
(Item 6)
wherein the second defect detection result includes a second probability that the target of the first class belongs to a second defect, and performing classification on the image region to obtain a second defect detection result includes:
performing a classification on the image region to obtain probabilities that the targets of the first class belong to various predetermined defects;
summing various predetermined probabilities to obtain a second probability that the first class target belongs to a second defect.
6. The method of any one of items 2-5.
(Item 7)
Performing first target detection on the image to be detected and obtaining a target detection result includes:
performing a first target detection on the image to be detected using a first region detection network of defect detection models to obtain a target detection result;
Performing second target detection on the image area to obtain a first defect detection result includes:
using a second area detection network of the defect detection model to perform a second target detection on the image area to obtain a first defect detection result, wherein the depth of the second area detection network is , shallower than the depth of the first area detection network,
Classifying the image region and obtaining a second defect detection result includes:
classifying the image region using the defect detection model classification network to obtain a second defect detection result.
7. The method of any one of items 2-6.
(Item 8)
The target detection result further includes a second class target and second position information of the second class target in the image to be detected, performing first target detection on the image to be detected, After obtaining the detection result, the method includes:
determining the second class target as a third defect in the image to be detected, and determining the second class target and its second position information as a defect detection result related to the second class target. characterized by further comprising
8. The method of any one of items 1-7.
(Item 9)
After obtaining the defect detection result, the method includes:
Filtering defects satisfying a second filtering condition in the defect detection result to obtain a final defect detection result of the image to be detected.
9. The method of any one of items 1-8.
(Item 10)
Filtering defects that satisfy a second filtering condition in the defect detection result includes:
Obtaining dimensions of the defects by using position information of the defects in the defect detection results, and filtering defects whose dimensions in the defect detection results do not satisfy a predetermined dimension condition; and/or
filtering the defects whose probability in the defect detection result is lower than a second probability threshold.
The method of item 9.
(Item 11)
After obtaining a final defect detection result for the image to be detected, the method includes:
Further comprising outputting information of each of the defects in the final defect detection result according to descending order of probability of each of the defects in the final defect detection result.
A method according to item 9 or 10.
(Item 12)
Obtaining an image region containing the first class of targets based on the first position information includes:
determining a target region corresponding to the first class of targets from the image to be detected based on the first position information;
expanding the target area outward in the image to be detected by a predetermined factor;
extracting the outwardly extended target region from the image to be detected to obtain an image region containing the first class of targets.
12. The method of any one of items 1-11.
(Item 13)
Before performing first target detection on the image to be detected and obtaining target detection results, the method comprises:
scaling the initial image to obtain an image of a given size;
compressing pixel values of the initial image after scaling to a predetermined pixel value range;
performing a normalization process on the compressed initial image to obtain the image to be detected.
13. The method of any one of items 1-12.
(Item 14)
The first target detection and defect detection are performed by a defect detection model, the defect detection model comprising at least:
obtaining a first sample set, said first sample set comprising at least one sample image, said sample image annotated with true defect information about a target, said target being a first class target; a step comprising
performing detection on sample images in the first sample set using the defect detection model to obtain defect detection results for the target;
and adjusting parameters of the defect detection model using the true defect information and the defect detection results.
14. The method of any one of items 1-13.
(Item 15)
wherein the defect detection model is jointly trained by a plurality of graphics processors, each graphics processor obtaining a different first sample set;
Using the defect detection model to perform detection on sample images in the first sample set to obtain defect detection results for the target,
synchronizing the batch normalization layers of the defect detection models in each of the graphics processors, and utilizing the defect detection models after synchronizing the batch normalization layers in each of the graphics processors to perform detections on the sample images to the targets; obtaining relevant defect detection results;
Adjusting parameters of the defect detection model using the true defect information and the defect detection results includes:
determining an average loss value of the defect detection model based on all the defect detection results and the true defect information;
and using the average loss value to adjust parameters of the defect detection model.
15. The method of item 14.
(Item 16)
Before performing detection on sample images in the first sample set using the defect detection model to obtain defect detection results for the target, the method includes:
pre-training the defect detection model using a second set of samples;
pre-training the defect detection model using a first sample set, wherein the learning rate used in the pre-training process is higher than a predetermined learning rate. gradually increases from a low first learning rate to the predetermined learning rate,
The sample images in the first sample set are at least
obtained by preprocessing an original image to obtain the sample image, wherein the preprocessing method is at least one of scale conversion, color conversion, horizontal flip, vertical flip, rotation, and trimming; characterized by including one
16. A method according to item 14 or 15.
(Item 17)
A defect detection device,
A first target detection module configured to perform a first target detection on an image to be detected and obtain a target detection result, the target detection result comprising a first class of targets and the detected a first target detection module comprising first location information of the first class of targets in the to-be-image;
an image region acquisition module configured to acquire an image region containing the first class of targets based on the first location information;
a defect detection module configured to perform defect detection on the image area and obtain defect detection results for the first class of targets.
(Item 18)
The defect detection module further performs a second target detection on the image area to obtain a first defect detection result, wherein the first defect detection result comprises at least one first defect detection present in the first class of targets. and/or performing classification on the image region to obtain a second defect detection result, the second defect detection result being information of a second defect to which the target of the first class belongs. characterized by being configured to include
18. Apparatus according to item 17.
(Item 19)
The defect detection module further performs a second target detection on the image region to obtain a first defect detection result if the first class of targets is a first subclass of targets; wherein if a target of a class is a target of a second subclass, performing classification on the image region to obtain a second defect detection result.
19. Apparatus according to item 18.
(Item 20)
the first defect detection result includes location information of various kinds of the first defects and a first probability belonging to the first defect, the defect detection module further performs a second target detection on the image area; After obtaining the first defect detection result,
Defect areas of the various first defects in the image to be detected are determined according to the position information of the various first defects, and overlap elimination processing is performed on overlapping areas between the different defect areas. and filtering said first defects whose first probability satisfies a first filtering condition.
20. Apparatus according to item 18 or 19.
(Item 21)
the first filtering condition is that the first probability is lower than a first probability threshold; and/or the defect detection module further performs non-maximum suppression for overlapping regions between the different defect regions. characterized by being configured to perform
21. Apparatus according to item 20.
(Item 22)
The second defect detection result includes a second probability that the first class target belongs to a second defect, and the defect detection module further performs a classification on the image region, wherein the first class target is obtaining probabilities of belonging to various pre-determined defects; adding the various pre-determined probabilities to obtain a second probability that the target of said first class belongs to a second defect;
22. Apparatus according to any one of items 18-21.
(Item 23)
the first target detection module is further configured to perform a first target detection on the image to be detected using a first region detection network of a defect detection model to obtain a target detection result;
The defect detection module further utilizes a second region detection network of the defect detection model to perform a second target detection on the image region to obtain a first defect detection result; The depth is shallower than the depth of the first region detection network and configured to utilize the defect detection model classification network to perform classification on the image region to obtain a second defect detection result. characterized by
23. Apparatus according to any one of items 18-22.
(Item 24)
The target detection result further includes a second class of targets and second position information of the second class of targets in the image to be detected, and the defect detection module further comprises: After performing one target detection and obtaining a target detection result, determining the second class target as a third defect in the image to be detected, and determining the second class target and its second position information as the configured to result in defect detection involving a second class of targets
24. Apparatus according to any one of items 17-23.
(Item 25)
The defect detection module is further configured to, after obtaining the defect detection result, filter defects satisfying a second filtering condition in the defect detection result to obtain a final defect detection result of the image to be detected. characterized by
25. Apparatus according to any one of items 17-24.
(Item 26)
The defect detection module further utilizes the position information of the defect in the defect detection result to obtain dimensions of the defect, filters defects whose dimensions in the defect detection result do not meet a predetermined dimension condition, and/or Alternatively, configured to filter the defects whose probability in the defect detection result is lower than a second probability threshold.
26. Apparatus according to item 25.
(Item 27)
The defect detection module further comprises, after obtaining a final defect detection result of the image to be detected, determining each defect in the final defect detection result according to a descending order of the probability of each defect in the final defect detection result. characterized in that it is configured to output information of the defect
27. Apparatus according to item 25 or 26.
(Item 28)
The image area acquisition module further determines a target area corresponding to the first class target from the image to be detected based on the first position information, and determines the target area from the image to be detected. and extracting the outwardly expanded target region from the image to be detected to obtain an image region containing the first class of targets. to be
28. Apparatus according to any one of items 17-27.
(Item 29)
The apparatus performs first target detection on the image to be detected, scales an initial image before obtaining a target detection result, obtains an image of a predetermined size, pixels of the initial image after scaling The method further comprises a preprocessing module configured to compress values to a predetermined pixel value range and perform a normalization operation on the compressed initial image to obtain the image to be detected.
29. Apparatus according to any one of items 17-28.
(Item 30)
The apparatus obtains a first sample set, the first sample set including at least one sample image, the sample image annotated with true defect information about a target, the target being a first class target. using the defect detection model to perform detection on sample images in the first sample set to obtain defect detection results for the target; using the true defect information and the defect detection results; and further comprising a training module configured to adjust parameters of the defect detection model.
30. Apparatus according to any one of items 17-29.
(Item 31)
wherein the defect detection model is jointly trained by a plurality of graphics processors, each graphics processor obtaining a different first sample set;
The training module further synchronizes a batch normalization layer of the defect detection model in each of the graphics processors, and utilizes the defect detection model after synchronization of the batch normalization layer in each of the graphics processors to detect defects on the sample images. obtaining defect detection results for the target; determining an average loss value of the defect detection model based on all the defect detection results and the true defect information; using the average loss value; configured to adjust parameters of the defect detection model
31. Apparatus according to item 30.
(Item 32)
The training module further performs detection on sample images in the first sample set using the defect detection model, and using a second sample set before obtaining defect detection results for the target. pre-training the defect detection model; and pre-training the defect detection model using a first sample set. and the learning rate used in the pretraining process gradually increases from a first learning rate lower than a predetermined learning rate to the predetermined learning rate,
The sample images in the first sample set are obtained by at least preprocessing an original image to obtain the sample images, and the preprocessing methods are scale conversion, color conversion, and horizontal inversion. , vertical flip, rotation, and cropping.
32. Apparatus according to item 30 or 31.
(Item 33)
An electronic device comprising a memory and a processor,
17. Electronic equipment, wherein the processor executes program instructions stored in the memory to perform the method of any one of items 1-16.
(Item 34)
17. A computer readable storage medium storing program instructions for, when executed by a processor, causing said processor to perform the method of any one of items 1 to 16.
(Item 35)
A computer comprising a non-transitory computer readable storage medium storing a computer program which, when read and executed by a computer, causes said computer to perform the method of any one of items 1 to 16. program product.

Claims (35)

A defect detection method comprising:
performing a first target detection on an image to be detected to obtain a target detection result, wherein the target detection result includes a first class target and the first class target in the image to be detected; including first location information for
obtaining an image region containing the first class of targets based on the first location information;
performing defect detection on the image area to obtain defect detection results for the first class of targets. performing defect detection on the image region and obtaining defect detection results for the first class of targets;
performing second target detection on the image area to obtain first defect detection results, wherein the first defect detection results are for at least one first defect present in the first class of targets; including information that
and/or performing classification on the image region to obtain a second defect detection result, wherein the second defect detection result includes information of a second defect to which the target of the first class belongs; 2. The method of claim 1, comprising: performing defect detection on the image area and obtaining defect detection results for the first class of targets;
performing a second target detection on the image region to obtain a first defect detection result if the first class of targets is a first subclass of targets;
performing classification on the image region to obtain a second defect detection result if the first class of targets is a second subclass of targets. The method described in . The first defect detection result includes location information of various kinds of the first defects and a first probability belonging to the first defect, and performing second target detection on the image area to obtain a first defect detection result. After that, the method includes:
Defect areas of the various first defects in the image to be detected are determined according to the position information of the various first defects, and overlap elimination processing is performed on overlapping areas between the different defect areas. and
4. A method according to claim 2 or 3, comprising at least one of: filtering said first defects for which a first probability satisfies a first filtering condition. the first filtering condition is that the first probability is lower than a first probability threshold; and/or
Performing de-duplication processing on overlapping areas between the different defect areas,
5. The method of claim 4, comprising performing non-maximal suppression on overlapping regions between the different defect regions. wherein the second defect detection result includes a second probability that the target of the first class belongs to a second defect, and performing classification on the image region to obtain a second defect detection result includes:
performing a classification on the image regions to obtain probabilities that the targets of the first class belong to various predetermined defects;
summing various predetermined probabilities to obtain a second probability that the target of the first class belongs to the second defect. Method. Performing first target detection on the image to be detected and obtaining a target detection result includes:
performing a first target detection on the image to be detected using a first region detection network of defect detection models to obtain a target detection result;
Performing second target detection on the image area to obtain a first defect detection result includes:
using a second area detection network of the defect detection model to perform a second target detection on the image area to obtain a first defect detection result, wherein the depth of the second area detection network is , shallower than the depth of the first area detection network,
Classifying the image region and obtaining a second defect detection result includes:
7. A method according to any one of claims 2 to 6, comprising performing a classification on the image region using the defect detection model classification network to obtain a second defect detection result. the method of. The target detection result further includes a second class target and second position information of the second class target in the image to be detected, performing first target detection on the image to be detected, After obtaining the detection result, the method includes:
determining the second class target as a third defect in the image to be detected, and determining the second class target and its second position information as a defect detection result related to the second class target. 8. A method according to any one of claims 1 to 7, further comprising: After obtaining the defect detection result, the method includes:
9. Further comprising filtering defects in the defect detection results satisfying a second filtering condition to obtain a final defect detection result for the image to be detected. The method described in . Filtering defects that satisfy a second filtering condition in the defect detection result includes:
Obtaining dimensions of the defects by using position information of the defects in the defect detection results, and filtering defects whose dimensions in the defect detection results do not satisfy a predetermined dimension condition; and/or
10. The method of claim 9, comprising filtering the defects whose probability in the defect detection result is below a second probability threshold. After obtaining a final defect detection result for the image to be detected, the method includes:
11. The method according to claim 9, further comprising: outputting information of each defect in the final defect detection result according to descending order of probability of each defect in the final defect detection result. the method of. Obtaining an image region containing the first class of targets based on the first position information includes:
determining a target region corresponding to the first class of targets from the image to be detected based on the first position information;
expanding the target area outward in the image to be detected by a predetermined factor;
extracting the outwardly extended target region from the image to be detected to obtain an image region containing the first class of targets. or the method described in paragraph 1. Before performing first target detection on the image to be detected and obtaining target detection results, the method comprises:
scaling the initial image to obtain an image of a given size;
compressing pixel values of the initial image after scaling to a predetermined pixel value range;
13. A method according to any one of the preceding claims, further comprising performing a normalization operation on the compressed initial image to obtain the image to be detected. The first target detection and defect detection are performed by a defect detection model, the defect detection model comprising at least:
obtaining a first sample set, said first sample set comprising at least one sample image, said sample image annotated with true defect information about a target, said target being a first class target; a step comprising
performing detection on sample images in the first sample set using the defect detection model to obtain defect detection results for the target;
and adjusting parameters of the defect detection model using the true defect information and the defect detection results. The method described in section. wherein the defect detection model is jointly trained by a plurality of graphics processors, each graphics processor obtaining a different first sample set;
Using the defect detection model to perform detection on sample images in the first sample set to obtain defect detection results for the target,
synchronizing the batch normalization layers of the defect detection models in each of the graphics processors, and utilizing the defect detection models after synchronizing the batch normalization layers in each of the graphics processors to perform detections on the sample images to the targets; obtaining relevant defect detection results;
Adjusting parameters of the defect detection model using the true defect information and the defect detection results includes:
determining an average loss value of the defect detection model based on all the defect detection results and the true defect information;
15. The method of claim 14, comprising using the average loss value to adjust parameters of the defect detection model. Before performing detection on sample images in the first sample set using the defect detection model to obtain defect detection results for the target, the method includes:
pre-training the defect detection model using a second set of samples;
pre-training the defect detection model using a first sample set, wherein the learning rate used in the pre-training process is higher than a predetermined learning rate. gradually increases from a low first learning rate to the predetermined learning rate,
The sample images in the first sample set are at least
obtained by preprocessing an original image to obtain the sample image, wherein the preprocessing method is at least one of scale conversion, color conversion, horizontal flip, vertical flip, rotation, and trimming; 16. A method according to claim 14 or 15, comprising one. A defect detection device,
A first target detection module configured to perform a first target detection on an image to be detected and obtain a target detection result, the target detection result comprising a first class of targets and the detected a first target detection module comprising first location information of the first class of targets in the to-be-image;
an image region acquisition module configured to acquire an image region containing the first class of targets based on the first location information;
a defect detection module configured to perform defect detection on the image area and obtain defect detection results for the first class of targets. The defect detection module further performs a second target detection on the image area to obtain a first defect detection result, wherein the first defect detection result comprises at least one first defect detection present in the first class of targets. and/or performing classification on the image region to obtain a second defect detection result, the second defect detection result being information of a second defect to which the target of the first class belongs. 18. The device of claim 17, wherein the device is configured to include: The defect detection module further performs a second target detection on the image region to obtain a first defect detection result if the first class of targets is a first subclass of targets; 19. The method of claim 18, wherein if a class of targets is a second subclass of targets, the method is configured to perform the step of performing classification on the image region to obtain a second defect detection result. equipment. the first defect detection result includes location information of various kinds of the first defects and a first probability belonging to the first defect, the defect detection module further performs a second target detection on the image area; After obtaining the first defect detection result,
Defect areas of the various first defects in the image to be detected are determined according to the position information of the various first defects, and overlap elimination processing is performed on overlapping areas between the different defect areas. and filtering the first defects for which a first probability satisfies a first filtering condition. Device. the first filtering condition is that the first probability is lower than a first probability threshold; and/or the defect detection module further performs non-maximum suppression for overlapping regions between the different defect regions. 21. Apparatus according to claim 20, characterized in that it is arranged to: The second defect detection result includes a second probability that the first class target belongs to a second defect, and the defect detection module further performs a classification on the image region, wherein the first class target is 18. Arranged to obtain probabilities of belonging to various pre-determined defects and to add the various pre-determined probabilities to obtain a second probability that the target of said first class belongs to a second defect. 22. The apparatus according to any one of 21. the first target detection module is further configured to perform a first target detection on the image to be detected using a first region detection network of a defect detection model to obtain a target detection result;
The defect detection module further utilizes a second region detection network of the defect detection model to perform a second target detection on the image region to obtain a first defect detection result; The depth is shallower than the depth of the first region detection network and configured to utilize the defect detection model classification network to perform classification on the image region to obtain a second defect detection result. 23. Apparatus according to any one of claims 18-22, characterized in that: The target detection result further includes a second class of targets and second position information of the second class of targets in the image to be detected, and the defect detection module further comprises: After performing one target detection and obtaining a target detection result, determining the second class target as a third defect in the image to be detected, and determining the second class target and its second position information as the 24. Apparatus according to any one of claims 17 to 23, characterized in that it is arranged to result in defect detection results involving a second class of targets. The defect detection module is further configured to, after obtaining the defect detection result, filter defects satisfying a second filtering condition in the defect detection result to obtain a final defect detection result of the image to be detected. 25. Apparatus according to any one of claims 17 to 24, characterized in that: The defect detection module further utilizes the position information of the defect in the defect detection result to obtain dimensions of the defect, filters defects whose dimensions in the defect detection result do not meet a predetermined dimension condition, and/or 26. The apparatus of claim 25, or configured to filter out defects whose probability in the defect detection result is lower than a second probability threshold. The defect detection module further comprises, after obtaining a final defect detection result of the image to be detected, determining each defect in the final defect detection result according to a descending order of the probability of each defect in the final defect detection result. 27. Apparatus according to claim 25 or 26, characterized in that it is arranged to output information of said defects. The image area acquisition module further determines a target area corresponding to the first class target from the image to be detected based on the first position information, and determines the target area from the image to be detected. and extracting the outwardly expanded target region from the image to be detected to obtain an image region containing the first class of targets. 28. Apparatus according to any one of claims 17-27. The apparatus performs first target detection on the image to be detected, scales an initial image before obtaining a target detection result, obtains an image of a predetermined size, pixels of the initial image after scaling The method further comprises a preprocessing module configured to compress values to a predetermined pixel value range and perform a normalization operation on the compressed initial image to obtain the image to be detected. 29. Apparatus according to any one of claims 17-28. The apparatus obtains a first sample set, the first sample set including at least one sample image, the sample image annotated with true defect information about a target, the target being a first class target. using the defect detection model to perform detection on sample images in the first sample set to obtain defect detection results for the target; using the true defect information and the defect detection results; 30. Apparatus according to any one of claims 17 to 29, further comprising a training module configured to adjust parameters of the defect detection model. wherein the defect detection model is jointly trained by a plurality of graphics processors, each graphics processor obtaining a different first sample set;
The training module further synchronizes a batch normalization layer of the defect detection model in each of the graphics processors, and utilizes the defect detection model after synchronization of the batch normalization layer in each of the graphics processors to detect defects on the sample images. obtaining defect detection results for the target; determining an average loss value of the defect detection model based on all the defect detection results and the true defect information; using the average loss value; 31. The apparatus of claim 30, configured to adjust parameters of the defect detection model. The training module further performs detection on sample images in the first sample set using the defect detection model, and using a second sample set before obtaining defect detection results for the target. pre-training the defect detection model; and pre-training the defect detection model using a first sample set. and the learning rate used in the pretraining process gradually increases from a first learning rate lower than a predetermined learning rate to the predetermined learning rate,
The sample images in the first sample set are obtained by at least preprocessing an original image to obtain the sample images, and the preprocessing methods are scale conversion, color conversion, and horizontal inversion. 32. Apparatus according to claim 30 or 31, comprising at least one of: , vertical flip, rotation, cropping. An electronic device comprising a memory and a processor,
17. Electronic equipment, wherein the processor executes program instructions stored in the memory to perform the method of any one of claims 1-16. A computer readable storage medium storing program instructions for, when executed by a processor, causing said processor to perform the method of any one of claims 1 to 16. a non-transitory computer readable storage medium storing a computer program which, when read and executed by a computer, causes said computer to perform the method of any one of claims 1 to 16, computer program product.

Images