WO2022036953A1 - 缺陷检测方法和相关装置、设备、存储介质、计算机程序产品 - Google Patents

缺陷检测方法和相关装置、设备、存储介质、计算机程序产品 Download PDF

Info

Publication number
WO2022036953A1
WO2022036953A1 PCT/CN2020/136251 CN2020136251W WO2022036953A1 WO 2022036953 A1 WO2022036953 A1 WO 2022036953A1 CN 2020136251 W CN2020136251 W CN 2020136251W WO 2022036953 A1 WO2022036953 A1 WO 2022036953A1
Authority
WO
WIPO (PCT)
Prior art keywords
target
defect
defect detection
image
detection result
Prior art date
Application number
PCT/CN2020/136251
Other languages
English (en)
French (fr)
Inventor
徐子豪
费敬敬
杨凯
吴立威
Original Assignee
上海商汤智能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海商汤智能科技有限公司 filed Critical 上海商汤智能科技有限公司
Priority to KR1020217037051A priority Critical patent/KR20220023335A/ko
Priority to JP2021567817A priority patent/JP2022548438A/ja
Publication of WO2022036953A1 publication Critical patent/WO2022036953A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • the present application relates to, but is not limited to, the field of image processing, and in particular, relates to a defect detection method and related devices, equipment, storage media, and computer program products.
  • defect detection of equipment is based on photographing an image of the equipment, and manually inspecting the image to determine whether there is a defect.
  • manual defect detection can easily lead to missed or wrong detection due to human negligence. Therefore, how to perform defect detection to reduce the defect missed detection rate and improve the accuracy of the defect detection is a very critical issue at present.
  • Embodiments of the present application provide at least one defect detection method and related apparatus, equipment, storage medium, and computer program product.
  • An embodiment of the present application provides a defect detection method, including: performing a first target detection on an image to be detected, and obtaining a target detection result, wherein the target detection result includes a first type of target and the first type of target in the first target of the to-be-detected image. position information; based on the first position information, obtain an image area containing the first type of target; perform defect detection on the image area to obtain a defect detection result about the first type of target.
  • the defect detection is reduced. area, which can reduce the missed detection rate and false detection rate.
  • performing defect detection on an image area to obtain a defect detection result about a first type of target includes: performing a second target detection on an image area to obtain a first defect detection result, wherein the first defect
  • the detection result includes information of at least one first defect existing on the first type of target; and/or, classifying the image area to obtain a second defect detection result, wherein the second defect detection result includes the first type of target to which the first defect belongs.
  • the image area including the first type of targets can be detected in a variety of ways, and different detection methods can be used for different first-type targets, so that the defect detection is more targeted, and the obtained defect detection results are more accurate or
  • the same first type of target adopts different detection methods, and then the different defect detection methods can be integrated to obtain a more accurate defect detection result of the final image to be detected, which further reduces the false detection rate.
  • performing defect detection on the image area to obtain a defect detection result about the first type of target includes: in the case that the first type of target is the first sub-category target, performing the first step on the image area. Two-target detection, the step of obtaining the first defect detection result; when the first type of target is the second sub-category target, the step of classifying the image area to obtain the second defect detection result is performed.
  • the defect detection is more flexible, and compared to using the same detection method for all types of targets, it is more targeted, and the defect detection results are more accurate.
  • the first defect detection result includes position information of each first defect and a first probability belonging to the first defect; after the second target detection is performed on the image area to obtain the first defect detection result , the method further includes at least one of the following: according to the position information of each first defect, respectively determining the defect area of each first defect on the image to be detected, and performing deduplication processing on the overlapping area between different defect areas; The first defects whose first probability satisfies the first filter condition are filtered.
  • the first filtering condition is that the first probability is lower than the first probability threshold; and/or, performing de-duplication processing on overlapping areas between different defect areas, including: The overlapping area of is subjected to non-maximum suppression.
  • Deduplication is carried out by means of non-maximum value suppression, and the defect areas with higher defect probability can be retained, which improves the processing efficiency. At the same time, it can also guarantee the accuracy of defect detection.
  • the second defect detection result includes a second probability that the first type of object belongs to the second defect; classifying the image area to obtain the second defect detection result includes: classifying the image area to obtain The probability that the first type of target belongs to each preset defect; the probability of each preset defect is added to obtain the second probability that the first type of target belongs to the second defect.
  • the defect probability of the first-type target is determined, and the possibility that the first-type target belongs to various preset defects is widely considered, so that the defect probability of the first-type target is more accurate.
  • performing the first target detection on the image to be detected to obtain the target detection result includes: using the first area detection network of the defect detection model to perform the first target detection on the image to be detected to obtain the target detection result;
  • Performing the second target detection on the image area to obtain the first defect detection result including: using the second area detection network of the defect detection model to perform the second target detection on the image area to obtain the first defect detection result; wherein, the second area detection network
  • the problem of high missed detection rate in a single detection network can be alleviated to a certain extent.
  • the second target detection uses The depth of the network model is shallower than that of the network model used for the first target detection, which can reduce the amount of calculation in the second target detection process and improve the detection efficiency.
  • the target detection result further includes the second type of target and the second position information of the second type of target in the to-be-detected image.
  • the method It also includes: determining the second type of target as the third defect in the image to be detected, and using the second type of target and its second position information as the defect detection result about the second type of target.
  • a defect detection model can detect multiple targets, and the adaptability of the defect detection model is enhanced.
  • the method further includes: filtering the defects satisfying the second filtering condition in the defect detection result, so as to obtain the final defect detection result of the image to be detected.
  • filtering the defects that satisfy the second filter condition in the defect detection result includes: obtaining the size of the defect by using the position information of the defect in the defect detection result, and filtering the size in the defect detection result that does not meet the preset size condition and/or, filtering defects whose probability is lower than the second probability threshold in the defect detection result.
  • the method further includes: outputting each defect in the final defect detection result in descending order of probability of each defect in the final defect detection result defect information.
  • acquiring an image area containing a first type of target based on the first position information includes: determining a target area corresponding to the first type of target from an image to be detected based on the first position information; The area is expanded outward by a preset multiple in the image to be detected; the expanded target area is extracted from the image to be detected to obtain an image area containing the first type of target.
  • the method before performing the first target detection on the image to be detected and obtaining the target detection result, the method further includes: scaling the initial image to obtain an image of a preset size; The value is compressed to the preset pixel value range; the compressed initial image is normalized to obtain the image to be detected.
  • the first object detection and defect detection are performed by a defect detection model; wherein, the defect detection model is obtained by training at least the following steps: acquiring a first sample set, wherein the first sample set At least one sample image is included, and the sample image is marked with real defect information about the target, and the target includes the first type of target; the sample image in the first sample set is detected by the defect detection model, and the defect detection result about the target is obtained; Defect information and defect detection results, adjust the parameters of the defect detection model.
  • the accuracy of the defect detection model obtained by training in the above manner is higher, so that the first target detection and the defect detection are performed so that the detection results are more prepared.
  • the defect detection model is jointly trained by multiple graphics processors, and the first sample set obtained by each graphics processor is different; the defect detection model is used to analyze the samples in the first sample set.
  • the image is detected to obtain the defect detection results about the target, including: synchronizing the batch normalization layer of the defect detection model in each graphics processor, and using the synchronized defect detection model of the batch normalization layer in each graphics processor.
  • Detect the sample images to obtain the defect detection results about the target use the real defect information and defect detection results to adjust the parameters of the defect detection model, including: determining the average loss value of the defect detection model based on all defect detection results and real defect information; Using the average loss value, adjust the parameters of the defect detection model.
  • the method before the defect detection model is used to detect the sample images in the first sample set, and the defect detection result about the target is obtained, the method further includes at least one of the following steps: using the second sample set to detect The defect detection model is pre-trained; the defect detection model is pre-trained by using the first sample set, wherein the learning rate used in the process of pre-warming training: starting from the first learning rate lower than the preset learning rate, and gradually increase to a preset learning rate; wherein, the sample images in the first sample set are obtained by at least the following steps: preprocessing the original image to obtain a sample image, wherein the preprocessing methods include scale transformation, color transformation, One or more of horizontal flip, vertical flip, rotation, and crop.
  • the parameters of the defect detection model are initialized by pre-training the defect detection model with the second sample set before the formal training of the defect detection model, or the learning rate is initialized before the formal training, and by The original image is preprocessed to improve the adaptability of the defect detection model.
  • An embodiment of the present application provides a defect detection device, including: a first target detection module, configured to perform a first target detection on an image to be detected to obtain a target detection result, wherein the target detection result includes a first type of target and a first type of target The target is in the first position information of the image to be detected; the image area acquisition module is configured to obtain an image area containing the first type of target based on the first position information; the defect detection module is configured to perform defect detection on the image area, and obtain information about the first type of target. Defect detection results for a class of objects.
  • An embodiment of the present application provides an electronic device, including a memory and a processor, where the processor is configured to execute program instructions stored in the memory, so as to implement some or all of the steps of the foregoing defect detection method.
  • An embodiment of the present application provides a computer-readable storage medium, which stores program instructions, and when the program instructions are executed by a processor, implements some or all of the steps of the above-mentioned defect detection method.
  • An embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, the implementation of the present application is implemented. Some or all of the steps of the methods described in the examples.
  • the computer program product may be a software installation package.
  • the reduction of The area of small defect detection can reduce the missed detection rate and false detection rate.
  • 1a is a schematic flowchart of a defect detection method provided by an embodiment of the present application.
  • FIG. 1b is a schematic state diagram of a defect detection method provided by an embodiment of the present application.
  • FIG. 2 is a schematic diagram of an image to be detected in a defect detection method provided by an embodiment of the present application
  • FIG. 3 is a schematic diagram of an image area in a defect detection method provided by an embodiment of the present application.
  • FIG. 4a is a schematic diagram of an implementation flowchart of a defect detection method provided by an embodiment of the present application.
  • FIG. 4b is a schematic diagram of an implementation flow of an image preprocessing module provided by an embodiment of the present application.
  • 4c is a schematic diagram of a network structure of a first-level component detection module provided by an embodiment of the present application.
  • Fig. 4d is a schematic diagram of the implementation process of classification and identification of anti-vibration hammers provided by an embodiment of the present application;
  • 4e is a schematic diagram of a network structure of a secondary insulator self-explosion detection module provided by an embodiment of the present application.
  • FIG. 5 is a schematic structural diagram of a defect detection device provided by an embodiment of the present application.
  • FIG. 6 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • FIG. 7 is a schematic structural diagram of a computer-readable storage medium provided by an embodiment of the present application.
  • the defect detection method in the above-mentioned related art has the following problems: 1) the training process is cumbersome, and the missed detection rate and the false detection rate are high; 2) the CPU is used for inference, and the test speed is slow; 3) the algorithm is not improved for the power scene; 4) Only a single component or defect can be detected and identified, such as self-explosion of insulators, corrosion of anti-vibration hammers, etc.
  • FIG. 1a is a schematic flowchart of a defect detection method provided by an embodiment of the present application. The method may include the following steps:
  • Step S11 performing first target detection on the image to be detected to obtain a target detection result, wherein the target detection result includes the first type of target and the first position information of the first type of target in the to-be-detected image.
  • FIG. 1b is a schematic state diagram of a defect detection method provided by an embodiment of the present application.
  • the image acquisition device collects the to-be-detected images of each object to be detected in a specific environment, for example, the to-be-detected images containing the first type of objects, and then the image acquisition device transmits the to-be-detected images to the defect detection device,
  • the defect detection device performs defect detection on the image to be detected transmitted from the image acquisition device according to the method described in the embodiment of the present application.
  • the first type of target is the equipment that needs to be overhauled.
  • a traffic light, a manhole cover, etc. at an intersection in a city can acquire images to be inspected that may contain traffic lights or manhole covers through the image acquisition device, and then input the images to be inspected that may contain traffic lights or manhole covers to the defect detection equipment, and then Obtain the defect detection results of traffic lights.
  • aerial pictures of each line equipment in the line can be obtained by taking pictures, such as aerial pictures of insulator strings or anti-vibration hammers, or Aerial image of the bird's nest. Then, the obtained aerial image is used as the image to be detected, and is input into the defect detection device, and then the defect detection device can perform defect detection on the image to be detected according to the method described in the embodiment of the present application.
  • the image acquisition device and the defect detection device may be integrated into one device, or may be multiple separate devices, which are not specified here.
  • the initial image before performing the first target detection on the image to be detected, the initial image may be scaled to an image of a preset size.
  • the preset size includes that the ratio of width to height is (1.1-1.7):1 and/or the side with the smaller size is greater than 1000.
  • the ratio of width to height used in the embodiments of the present application is 1.5:1, and the minimum side is 1200.
  • the pixel values of the scaled initial image are compressed to a predetermined range of pixel values. For example, the pixel values of the scaled initial image are compressed from 0-255 to 0-1.
  • the normalization processing method includes using the mean and variance std of the ImageNet dataset to normalize the compressed initial image, as shown in the following formula (1-1):
  • mean is the mean of the dataset
  • std is the variance of the dataset
  • pixelvalue is the value of each pixel in the image.
  • COCO Common Objects in Context
  • the normalization method is not necessarily limited to normalization by calling other data sets.
  • other normalization methods commonly used such as traversing the pixel values of all pixels in the initial image, and recording the maximum and minimum values, and normalizing the initial image by using the maximum and minimum values as parameters
  • the processing may also be normalized by using a sigmoid function. Therefore, the embodiment of the present application does not specifically limit the normalization method.
  • the first target detection is performed on the image to be detected by the first area detection network of the defect detection model, and the target detection result is obtained.
  • the defect detection model can be trained by the following steps:
  • the first sample set includes at least one sample image, and the sample image is marked with real defect information about the target.
  • the target here includes the first type of target, of course, it can also include other types of targets. goals, such as second-class goals, and so on.
  • the sample images in the first sample set may only contain the first type of targets and other categories of targets, wherein, the real defect information of the target may indicate that the target is normal without defects, or the first type of target has specific information. defect information. That is to say, in the embodiment of the present application, the defect detection model can be trained by using normal sample images and abnormal defective sample images to obtain a defect detection model with strong adaptability.
  • the sample images in the first sample set can be obtained by preprocessing the original images, and the preprocessing method can be one or more of scale transformation, color transformation, horizontal flip, vertical flip, rotation, and cropping kind.
  • the length and width of the original image are transformed at a preset ratio, for example, the aspect ratio value of the original image is scaled while maintaining the ratio of 1.5:1.
  • other aspect ratios or arbitrary transformations may also be maintained. Therefore, the aspect ratios of the scale transformation of the original image are not specified in the embodiments of the present application.
  • the transformation of color can be one or more of brightness, saturation and chroma transformation, for example, the brightness, saturation and chroma of an original image are transformed, so that the same real defect information containing the same target can pass through
  • Different forms are input into the defect detection model, and the defect detection model is trained, so that the defect detection model can enhance the adaptability and improve the accuracy of defect detection.
  • the above-mentioned various preprocessing methods can be used individually in the networks of different stages of the defect detection model, or can be used in the networks of all stages.
  • the defect detection model is used to detect the sample images in the first sample set, and the defect detection result about the target is obtained.
  • the defect detection model can be jointly trained by multiple graphics processors, and the first sample set obtained by each graphics processor is different. By using multiple graphics processors to detect the defect detection model, the training is doubled speed.
  • the batch normalization layers of the defect detection models in each graphics processor are synchronized, and the sample images are inspected using the batch-normalized synchronized defect detection models in each graphics processor to obtain information about the target. Defect detection results.
  • One of the GPUs collects the mean and variance of the batch normalization layers of the defect detection models in the other GPUs to calculate an overall mean and variance, and then the GPU returns the calculated mean and method to the rest of the
  • the batch normalization layer of the defect detection model in the GPU by synchronizing the batch normalization layer of the defect detection model on multiple GPUs, makes it possible to use the global first sample set for normalization, which is equivalent to increasing The batch size is reduced, thereby reducing the impact of using multiple GPUs.
  • the parameters of the defect detection model are adjusted.
  • an average loss value of the defect detection model is determined based on all defect detection results and real defect information, and then the parameters of the defect detection model are adjusted using the average loss value.
  • the defect detection model is trained synchronously through multiple graphics processors, so that the parameters of the defect detection models in all graphics processors after training are consistent. Compared with a single graphics processor, using multiple graphics processors to synchronously detect defects The model is trained to speed up the progress of the training.
  • the second sample set before using the defect detection model to detect the sample images in the first sample set to obtain the defect detection result about the target, the second sample set may also be used to pre-train the defect detection model, and/or using the first sample set to perform warm-up training on the defect detection model, wherein the learning rate used in the warm-up training process starts from a first learning rate lower than a preset learning rate and gradually increases to the preset learning rate. For example, if the preset learning rate is 0.6, in the warm-up training, the first learning rate can be gradually increased from 0.3 to 0.4, and then continuously increased to the preset learning rate of 0.6.
  • the second sample set can be a public training set, for example, can be a common COCO training set and ImageNet data set and so on.
  • the samples used in the warm-up training can be the same as the samples used in the formal training, that is, the first sample set is still used in the warm-up training.
  • the first sample set is used to perform warm-up training to continuously adjust the learning rate.
  • the It can be regarded as a further optimization of the parameters of the defect detection model, so that the training can be carried out on better parameters during formal training.
  • the first sample set used in the warm-up training process may still be the first sample set obtained after the above preprocessing, which can also improve the adaptability of the defect detection model, so that the defects obtained by training The detection accuracy of the detection model is higher.
  • gradually increasing linearly or exponentially from the first learning rate to the preset learning rate is equivalent to initializing the learning rate, so that the defect detection model can be trained from an optimal learning rate, so that the training is effective. Better results.
  • the first region detection network in the embodiment of the present application includes a Faster R-CNN (Region with Convolutional Neural Networks) network, such as ResNet50.
  • ResNet50 a Faster R-CNN (Region with Convolutional Neural Networks) network
  • other neural network models can also be used to perform the first target detection on the image to be detected, such as an SSD (Single Shot MultiBox Detector) network model, etc., and the depth need not be limited to ResNet50, but can also be ResNet101 , ResNet200, etc., no specific restrictions are made here.
  • the first area detection network used in the embodiment of the present application is the ResNet50 network.
  • a feature map of the image to be detected is extracted and obtained, wherein the size of the feature map is the size of the to-be-detected image.
  • the candidate frame refers to the area where the first type of target may exist, wherein the extraction of the candidate frame is essentially It is to extract the pixel values of the four vertices of the candidate frame, and then pass the candidate frame and the feature map together through the pooling layer to obtain the feature map of each candidate frame.
  • This step specifically includes sending the information of the four coordinates of the candidate frame to the pooling layer.
  • the coordinates of the four vertices of the candidate frame are mapped to the feature map, and then the feature vector of each candidate frame is obtained through the fully connected layer, where these feature vectors represent the feature information of the candidate frame, and then the feature vector The vector performs bounding regression and is classified according to the preset category, and finally the target detection result is obtained.
  • the first target detection also refers to detecting whether the image to be detected contains the first type of target. If there is a first type of target, the first position information of the first type of target in the to-be-detected image is further obtained. It is also possible to further learn the confidence of the first type of target, that is, the probability that the detected first type of target really belongs to the first type of target.
  • the target detection result includes the first type of target and the first position information of the first type of target in the image to be detected, and/or the probability that the candidate frame belongs to the first type of target.
  • the first position information includes position information of the candidate frame corresponding to the first type of target in the image to be detected.
  • the first position information refers to the position information of the four vertices of the candidate frame corresponding to the first type of target.
  • the first type of target may further include a first sub-category target and a second sub-category target. Wherein, the first subcategory target and the second category subtarget here may not belong to the first category target, but may have the same attributes.
  • the first type of target can be a device
  • the first sub-target can be another device
  • the second type of sub-target can be other devices.
  • the same attribute is that you can take pictures and then analyze the corresponding images. It is possible to know if the device is defective.
  • the probability of belonging to the first category of objects may further include the probability of belonging to the first sub-category of objects and/or the probability of belonging to the second sub-category of objects.
  • the target detection result further includes the second type of target and the second position information of the second type of target in the image to be detected, and/or the probability of belonging to the second type of target.
  • the second type of target here can be a target that should not exist, and its existence is a defect.
  • the target detection result obtained after the first target detection on the to-be-detected image includes the first type of target and/or the second type of target.
  • the initial image is an image of a power transmission line photographed by a drone, or a graphic frame cut from a high-definition video.
  • the initial image is processed first, including scaling the initial image to a 1200*1800 image. , and then compress the pixel value of the initial image after scaling to a preset pixel value range between 0-1, and then use the mean and variance std of the ImageNet dataset to normalize the compressed initial image to get image to be detected. Then, input the image to be detected into the ResNet50 network to obtain the probability of the target object in each candidate frame in the image to be detected corresponding to each preset target category, and select the preset target category with the highest probability as the category of the target object in the candidate frame.
  • the position information of each candidate box will be output.
  • common components include anti-vibration hammers, insulator strings, and may also include external defects, that is, they should not exist.
  • the anti-vibration hammer and the insulator string are temporarily included in the first category of targets, and the bird's nest is included in the second category of targets. That is, after inputting the image to be detected into ResNet50, the output target detection result may include anti-vibration hammer, insulator or bird's nest, as well as their corresponding position information and related probability in the image to be detected.
  • the probability here refers to the probability that the candidate frame belongs to the anti-vibration hammer, the insulator string or the bird's nest.
  • the second type of target is determined as the third defect in the image to be detected, and the second type of target and the second position information are used as the information about the second type of target.
  • Defect detection results For example, the second type of target is a bird's nest. If a bird's nest is detected in the image to be detected, the bird's nest is determined to be the third defect in the image to be detected, and the second position information of the bird's nest in the image to be detected is obtained. Therefore, the final target detection result contains the bird's nest and the second position information of the bird's nest.
  • Step S12 Based on the first position information, acquire an image area including a first type of object.
  • the target area corresponding to the first type of target is determined from the image to be detected. Specifically, it includes determining, from the image to be detected, an area corresponding to the object of the first type, that is, determining a candidate frame corresponding to the object of the first type. Then, the target area is expanded outward by a preset multiple in the image to be detected. The outward expansion method includes that the center point of the target area remains unchanged, and the length and width of the target area are changed to 1.1 to 1.5 times the original target area. 1.2 times the area. If the boundary of the target area exceeds the boundary of the image to be detected, the part in the image to be detected is reserved. Then, the expanded target area is extracted from the image to be detected, so as to obtain an image area containing the first type of target. Specifically, it includes cropping out the target area after the external expansion from the image to be detected.
  • FIG. 2 is a schematic diagram of an image to be detected in a defect detection method provided by an embodiment of the present application
  • FIG. 3 is a schematic diagram of an image area in a defect detection method provided by an embodiment of the present application.
  • the image 1 to be detected includes the first type of target 100 respectively, wherein the position information of the first type of target 100 is obtained through the first target detection network, and then based on the position of the first type of target 100 information, obtain the image area 110 containing the first type of object 100, in order to more clearly illustrate the relationship between the image area 110 and the candidate frame 101, the candidate frame 101 of the first type of object 100 is shown in the image area of FIG. 3 .
  • the candidate frame 101 of the first-type of object 100 is expanded by a certain proportion in the to-be-detected image 1. Therefore, the obtained image area 110 is larger than the area where the original candidate frame 101 is located.
  • the detection image is obtained by scaling, compressing pixel values, and normalizing the initial image
  • acquiring the image area containing the first type of target refers to the The first position information of the first type of target in the initial image, determine the target area corresponding to the first type of target in the initial image, that is, determine the position of the candidate frame corresponding to the first type of target in the initial image, and set the target area in the initial image.
  • the image is expanded by a preset multiple, and the expanded target area is extracted from the initial image to obtain an image area containing the first type of target.
  • the specific step of acquiring the image area containing the first type of target may also include extracting the image area of the first type of target from the image to be detected. Therefore, the image area containing the first type of target may be acquired in the image to be detected or the image area of the first type of target may be acquired in the initial image, which is not specifically limited here.
  • the image area after acquiring the image area containing the first type of target, the image area needs to be preprocessed, that is, the size of the image area is adjusted, wherein the aspect ratio of the image area can be adjusted to 1: 1. If the image area is obtained from the initial image, the image area needs to be normalized. If it is obtained from the image to be detected after normalization processing, the normalization may not be performed here.
  • the candidate frame corresponding to the insulator string in the image to be inspected For example, based on the first position information of the insulator string in the image to be inspected, determine the candidate frame corresponding to the insulator string in the image to be inspected, expand the candidate frame in the image to be inspected by 1.2 times, and then extract the frame from the image to be inspected After expanding the candidate frame, the image area containing the insulator can be obtained.
  • Step S13 Perform defect detection on the image area to obtain defect detection results for the first type of targets.
  • the method of performing defect detection on the image area includes at least one of the following methods.
  • One is to perform a second target detection on the image area to obtain a first defect detection result, wherein the first defect detection result includes at least one object existing on the first type of target.
  • One is to classify the image area to obtain the second defect detection result, wherein the second defect detection result includes the information of the second defect of the first type of target.
  • the second target detection refers to the further target detection of the first type of target, the position information of the first type of target is obtained in the first target detection, and the defect information of the first type of target is not directly detected. Therefore, the second target detection performs targeted defect detection for the first category.
  • Classifying the image area refers to classifying the first-type objects in the image area according to preset defect categories, and calculating the probability information that the first-type objects belong to each defect category.
  • the image area including the first type of objects is detected in a variety of ways, and different detection methods can be used for different first-type objects, so that the defect detection is more targeted and the detection result is more accurate.
  • different detection methods are used for the same first-type target, and then the results obtained by different defect detection methods can be synthesized to obtain the defect information of the final image to be detected, so that the defect detection results are more accurate.
  • the step of performing the second target detection on the image area to obtain the first defect detection result and in the case of the first type of target being the second sub-category , and perform the step of classifying the image area to obtain the second defect detection result.
  • the defect detection is more flexible. Compared with using the same detection method for all types of targets, it is more targeted to use different defect detection methods for different target objects. , making the detection results more accurate.
  • the preset insulator string is the target of the first subtype
  • the preset anti-vibration hammer is the target of the second subtype. Therefore, when it is detected that the target of the first category is the insulator string, the image area corresponding to the insulator string is subjected to the second Target detection is performed to obtain a first defect detection result.
  • the image area corresponding to the anti-vibration hammer is classified to obtain a second defect detection result.
  • the second target detection is performed on the image area corresponding to the insulator string respectively, and the image area corresponding to the anti-vibration hammer is classified.
  • the defect detection for both can be performed simultaneously, that is, the second target detection and classification are performed simultaneously.
  • the second target detection and the image area classification can also be performed for the same first sub-category target respectively, and then the results obtained by the two processing methods are combined to finally obtain the first sub-category target. Defect detection results.
  • the first defect detection result includes position information of each first defect and a first probability belonging to the first defect.
  • the position information of the first defect refers to the position of the first defect in the image area containing the first type of target, and the probability of belonging to the first defect refers to the detected confidence of the first defect, that is, the first defect.
  • the defect really belongs to the probability of being the first defect.
  • the second target detection is performed on the image area containing the insulator string, and it is detected that the insulator self-explosion may occur at the A position on the insulator string.
  • the second position information belonging to the self-explosion of the insulator, the confidence of the self-explosion of the insulator is the first probability of belonging to the self-explosion of the insulator.
  • the detection network used in the second target detection for the image area is the second area detection network, wherein the depth ratio of the second area detection network
  • the depth of the first region detection network is shallow.
  • the first region detection network is ResNet50
  • the second region detection network can be ResNet18.
  • the first target detection will detect the to-be-detected images that may contain different first-type targets, determine whether the to-be-detected images contain the first-type targets, and obtain the first In the second target detection, only the image area where the target of the first sub-category is located is detected, which specifically includes detecting the probability that the target of the first sub-category has defects.
  • the output will be the position information of the insulator self-explosion and the self-explosion in the insulator string. probability information.
  • the first defect detection result may be processed. It includes determining the defect area of each first defect on the image to be inspected according to the position information of each first defect. Because the position information obtained through the second target detection is the position information of the first defect on the image area containing the first type of target, it is also necessary to find the defect area of the first defect on the image to be inspected by means of mapping. Next, preset processing is performed on the defect area, wherein the preset processing includes at least one of the following, that is, performing deduplication processing on overlapping areas between different defect areas, and filtering first defects whose first probability satisfies the first filtering condition.
  • the manner of performing de-duplication processing on the overlapping regions between different defect regions may further include performing non-maximum value suppression on the overlapping regions between different defect regions.
  • the first filter condition is that the first probability is lower than the first probability threshold. That is, when the probability of the first defect is lower than the first probability threshold, the defect area corresponding to this part of the first defect is eliminated.
  • the position information and the first probability of the first defect are deleted from the first defect detection result, that is, the final first defect detection result is determined based on only the defect area obtained after the preset processing.
  • the overlapping Deduplication processing is carried out on the part of the first defect, so that a first defect corresponds to a defect area, which reduces the problem of multiple follow-up processing of a defect, and improves the accuracy of the final output.
  • the number of areas to be processed increases processing efficiency.
  • many unnecessary processing areas are reduced by eliminating the ones with low defect probability, and the processing efficiency is improved.
  • the defect area can not only improve the processing efficiency, but also ensure the accuracy of defect detection.
  • the second defect detection result includes a second probability that the first type of target belongs to the second defect.
  • the second defect here refers to whether the target of the second subclass has defects, wherein the second probability is the probability that the second subclass has defects.
  • the second defect may include multiple different sub-defects, and the second probability is the probability that the second sub-class target belongs to each sub-defect.
  • the process of classifying the image area includes: classifying the image area to obtain the probability that the first type of target belongs to each preset defect, that is, the probability that the second sub-type target belongs to each preset defect.
  • the image area containing the anti-vibration hammer is classified, and it is obtained that the anti-vibration hammer belongs to normal anti-vibration hammer, anti-vibration hammer rusted, and anti-vibration hammer.
  • the probability of torsion, damage to the anti-vibration hammer, and the falling off of the anti-vibration hammer for example, the probability of belonging to the normal anti-vibration hammer is 0.1, the probability of belonging to the anti-vibration hammer is 0, the probability of belonging to the anti-vibration hammer torsion is 0.2, and the probability of belonging to the anti-vibration hammer is damaged.
  • the probability is 0.5, and the probability that the anti-vibration hammer falls off is 0. At this time, it can be concluded that the anti-vibration hammer belongs to the preset defect, and the probability of damage to the anti-vibration hammer is the highest.
  • the anti-vibration hammer belongs to the damage of the anti-vibration hammer through this probability.
  • the probability of each preset defect is added to obtain the second probability that the first type of target belongs to the second defect.
  • the four defect probabilities of the anti-vibration hammer are added together, and the probability that the anti-vibration hammer is defective is 0.7.
  • the second defect at this time is that the anti-vibration hammer is defective, and the second probability is the probability that the anti-vibration hammer is defective.
  • the network used is a classification network.
  • the depth of the classification network can also be shallower than that of the first area detection network.
  • the classification network can also be ResNet18, and of course other deep networks.
  • the first area detection network is ResNet101
  • the classification network It can also be ResNet50, of course, in some embodiments, the depth of the classification network can be the same as the depth of the first region detection network. Therefore, the selection of the classification network is not specifically limited here.
  • the problem of single network detection or the high missed detection rate of the classifier can be solved to a certain extent.
  • the network used in the second target detection The depth of the model is shallower than that of the network model used for the first target detection, which can reduce the amount of calculation in the second target detection process and improve the detection efficiency.
  • the defects satisfying the second filter condition in the defect detection result are filtered.
  • the defect detection result here may be a combination of the first defect detection result, the second defect detection result, and the defect detection result about the second type of target.
  • the defects whose size does not meet the preset size condition in the defect detection result are filtered. For example, filter out the defective area where the size of the insulator is too large or too small, or the area of the bird's nest and the area where the abnormal anti-vibration hammer is located. Or filter defects whose probability is lower than the second probability threshold in the defect detection result.
  • the first sub-category target of the first target is set to a threshold of 0.3
  • the second sub-category target is set to 0.2
  • the second type of target is set to 0.2
  • the defects in the defect detection result are filtered according to the preset threshold.
  • the size of the defect and the probability of the defect in the defect detection result can also be filtered at the same time, so as to reduce the number of defects in the defect detection result.
  • the information of each defect in the defect detection result is output according to the probability of each defect in the final defect detection result in descending order.
  • the output defect information includes the position information and probability of the defect in the image to be detected, or the position information and corresponding probability of the defect in the initial image are directly output.
  • the output form may include outputting the position and probability of defects in the image to be inspected and/or the initial image in the form of characters, and may also use an annotation method to directly record all defects in the image to be inspected. It can be identified in the image and/or the initial image, and the respective probabilities are written in the identification box, or the corresponding image area of each defect in the to-be-detected image and/or the initial image can be directly output, and written in the image area. probability of a defect.
  • the filtered defect detection results include as little information as possible in normal situations or exclude obvious unreasonable situations, thereby improving the rationality of the defect detection results, and also This reduces the false detection rate.
  • the defect detection is reduced. area, which can reduce the missed detection rate and false detection rate.
  • the execution subject of the defect detection method may be a defect detection apparatus.
  • the defect detection method may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a user equipment (User Equipment, UE), a mobile device, a user terminal , terminals, cellular phones, cordless phones, personal digital processing (Personal Digital Assistant, PDA), handheld devices, computing devices, in-vehicle devices, wearable devices, etc.
  • the defect detection method may be implemented by a processor invoking computer-readable instructions stored in a memory.
  • the embodiment of the present application proposes a method for detecting multi-category defects in power grid transmission lines based on deep learning.
  • a deep learning model is used to perform two-stage detection and classification of large-scale input pictures. This method can jointly perform insulator self-explosion detection, Anti-vibration hammer defect detection, bird's nest detection and other abnormal defect identification.
  • the implementation process of the method mainly includes the following functional modules: RGB image input module 410, image preprocessing module 420, primary component detection module 430, secondary anti-vibration hammer classification and identification module 440, secondary insulator self-explosion detection module 450 , a post-processing module 460 for self-explosion detection, and an abnormal defect screening module 470 .
  • RGB image input module 410 image preprocessing module 420
  • primary component detection module 430 primary component detection module 430
  • secondary anti-vibration hammer classification and identification module 440 secondary insulator self-explosion detection module 450
  • a post-processing module 460 for self-explosion detection mainly includes the following functional modules:
  • RGB image input module This module can obtain data output from different drone cameras, and obtain RGB images or high-definition video of drone aerial photography. For high-definition video, it can be divided into image frames to obtain corresponding RGB images.
  • Image preprocessing module refer to Figure 4b, this module can uniformly process RGB images to obtain images to be input that can be used for defect detection.
  • the processing process includes: a) Image scaling 421: scaling the input high-definition RGB images , scaled to an image of size 1200 ⁇ 1800; b) Pixel value compression 422: Compress the scaled image pixel values from 0-255 to 0-1; c) Pixel value normalization 423: Use the mean of the ImageNet dataset The mean and variance std are normalized and calculated according to the aforementioned formula (1-1) for the image after the compressed pixel values.
  • First-level component detection module The input of this module is the normalized RGB image, and the target detection network Faster R-CNN detects three types of first-level components: bird's nest, insulator string, and anti-vibration hammer, and obtains the insulator string area, anti-vibration hammer, etc. There are three types of candidate areas, vibrating hammer area and bird's nest area, and the corresponding score of each candidate area.
  • the main network structure of this module is shown in Figure 4c.
  • the ResNet50 head network 431 is used as the backbone network of the detection network, and the C4 layer feature map 432 is extracted.
  • the first-level component detection process of this module includes: the RGB image passes through the ResNet50 head network 431 to obtain the C4 layer feature map 432, and the size of the feature map is 1/16 of the input image.
  • the ResNet50 head network 431 is continuously stacked by at least one convolutional layer.
  • C4 layer feature map 432 extracts candidate frame 433 through RPN, and the extracted candidate frame is the area where the target may exist, and the candidate frame and The feature map of the C4 layer passes through the RoI pooling layer 434 to obtain the feature map of each candidate region, and then passes through at least one fully connected layer 435 to obtain the feature vector of each candidate region, and the feature vector of the candidate region represents the feature information of the candidate region; Finally, perform frame coordinate regression 436 and multi-category classification 437 on the feature vector of each candidate area to obtain the final detection result, and output the original image position area and corresponding probability value of the bird's nest, insulator string, and anti-vibration hammer, among which the bird's nest area directly It is sent to the abnormal defect screening module, the insulator string area is sent to the secondary self-explosion detection module for self-explosion detection, and the anti-vibration hammer area is sent to the secondary
  • Two-level anti-vibration hammer classification and identification module takes the anti-vibration hammer area detected by the first-level component detection module as input, and mainly uses the ResNet18 classification network to classify the abnormal type of the anti-vibration hammer area cut from the original image (classification).
  • the categories also include normal categories), and finally the probability of defect categories is integrated to obtain normal anti-vibration hammers and different abnormal anti-vibration hammers.
  • the module divides the anti-vibration hammer into the following five categories according to the shape of the anti-vibration hammer: normal anti-vibration hammer, anti-vibration hammer corrosion, anti-vibration hammer torsion, anti-vibration hammer damage, anti-vibration hammer falling off, of which the last four categories are defect categories .
  • the process of classifying and identifying anti-vibration hammers by this module is shown in Figure 4d, including four sub-processes of cropping the anti-vibration hammer area 441 from the original image, image preprocessing 442, ResNet18 multi-classification 443, and defect category probability integration 444.
  • the anti-vibration hammer area detected by the first-stage component whose probability exceeds the first specific threshold is first cropped from the original image; then, image preprocessing is performed on each cropped anti-vibration hammer area sub-image to maintain the scale Consistent; then input each sub-image into the ResNet18 classification network for five classifications, and use softmax to calculate the probability values of the five anti-vibration hammer shape categories; finally, the probability of defect categories is integrated, that is, the four categories belong to the anti-vibration hammer defects. and, as the probability of the anti-vibration hammer being defective.
  • Secondary insulator self-explosion detection module takes the insulator string area detected by the primary component detection module as input, and mainly uses the Faster R-CNN detection network to detect the self-explosion area of each insulator string area sub-map, and obtains the possibility of self-explosion. insulator.
  • the main network structure of this module is shown in Figure 4e.
  • the ResNet18 head network 451 is used as the backbone network of the detection network, which can reduce the amount of calculation. First, cut out the insulator string region whose probability exceeds the second specific threshold. The image is preprocessed to keep the scale consistent, and the RGB image for detection is obtained.
  • Each RGB image is passed through the ResNet18 head network 451 to detect the self-explosion area of the insulator string, and the C4 layer feature map 452 is obtained; C4 layer feature map 452
  • the candidate frame 453 is extracted through the RPN, and the extracted candidate frame is the area where the insulator string may explode.
  • the candidate frame and the C4 layer feature map are passed through the RoI pooling layer 454 to obtain the feature map of each candidate area, and then through at least one fully connected layer. 455 to obtain the feature vector of each candidate region; finally, perform frame coordinate regression 456 and multi-class classification 457 on the feature vector of each candidate region to obtain the final detection result, and output the self-explosion region of the insulator string and the corresponding probability value.
  • Self-explosion detection post-processing module This module maps the self-explosion detection results in the sub-map of the insulator string area back to the original image results, and performs overlapping suppression processing on the self-explosion areas that have been repeatedly detected to obtain the position of the insulator self-explosion area in the original image. information.
  • the processing of this module can be divided into the following three steps: a) The coordinates of the self-explosion area in the sub-image of the insulator string are mapped back to the original image area; b) The overlapping self-explosion area is suppressed; c) The self-explosion area frame with a lower score is filtered.
  • Abnormal defect screening module This module screens the detection and identification results of bird's nest area, abnormal anti-vibration hammer, and insulator self-explosion to obtain the final abnormal defect detection result.
  • the processing process of this module can be divided into the following three steps: a) filter the bird's nest area and the self-explosion area according to the size, and remove the detection frame that is too large and too small; b) filter the bird's nest, insulator self-explosion and abnormal anti-vibration hammer according to their respective categories Threshold, remove the results lower than the corresponding threshold; c) Sort all the remaining detection results according to the score from high to low as the output result.
  • a lightweight deep learning detection and classification model is introduced to jointly identify several defect abnormalities such as insulator self-explosion, anti-vibration hammer, and attached bird's nest, with the following Beneficial effects: 1)
  • the methods of image processing and machine learning are mainly used, and manual features need to be constructed a priori for the target, which has poor robustness and low accuracy, and is greatly affected by the shooting environment.
  • the application scenarios are wider, the robustness is stronger, and the accuracy is higher; 2)
  • a two-stage detection is proposed in combination with the circuit inspection scene-
  • the detection, detection-classification algorithm is suitable for high-definition images of UAV aerial photography, and fully considers the defect characteristics of insulator self-explosion relying on the detected insulator strings, which can solve the problem of single network detection or classifier in related technologies.
  • the defect detection device 30 includes a first target detection module 31, an image area acquisition module 32, a defect detection module 33, and a first target detection module 31, which is configured to perform a first target detection on an image to be detected to obtain a target detection result, wherein the target detection
  • the result includes the first type of target and the first position information of the first type of target in the to-be-detected image
  • the area acquisition module is configured to acquire an image area containing the first type of target based on the first position information
  • the defect detection module 33 is configured as Defect detection is performed on the image area to obtain defect detection results for the first type of targets.
  • the defect detection is reduced. area, which can reduce the missed detection rate and false detection rate.
  • the defect detection device 30 further includes a preprocessing module (not shown in the figure).
  • the first target detection module 31 performs first target detection on the image to be detected, and before obtaining the target detection result, the preprocessing module is configured to scale the initial image to an image of a preset size; The pixel value of the initial image is compressed to a preset pixel value range; the compressed initial image is normalized to obtain the image to be detected.
  • the first target detection module 31 performs the first target detection on the image to be detected to obtain the detection result, including: using the first area detection network to perform the first target detection on the image to be detected to obtain the detection result;
  • the second target detection is performed on the image area to obtain the first defect detection result, which includes: using the second area detection network to perform the second target detection on the image area to obtain the first defect detection result; wherein, the depth of the second area detection network is greater than that of the third area detection network.
  • the depth of the first area detection network is shallow; classifying the image area to obtain the second defect detection result includes: classifying the image area by using the classification network to obtain the second defect detection result.
  • the above scheme by considering the characteristics of different defects in different parts and using different network models for detection, can solve the problem of high missed detection rate in a single detection network to a certain extent.
  • the network used for the second target detection The depth of the model is shallower than that of the network model used for the first target detection, which can reduce the amount of calculation in the second target detection process and improve the detection efficiency.
  • the image area acquisition module 32 acquires an image area containing the first type of target based on the first position information, including: determining, from the to-be-detected image, corresponding to the first type of target based on the first position information target area; expand the target area in the image to be detected by a preset multiple; extract the expanded target area in the image to be detected to obtain an image area containing the first type of target.
  • the defect detection module 33 is further configured to perform defect detection on the image area to obtain a defect detection result about the first type of target, including: performing second target detection on the image area to obtain the first defect detection As a result, the first defect detection result includes information of at least one first defect existing on the first type of target; and/or the image area is classified to obtain a second defect detection result, wherein the second defect detection result Including the information of the second defect to which the first type of object belongs.
  • the image area including the first type of target is detected in various ways, and different detection methods can be used for different first-type targets, so that the defect detection is more targeted, and the obtained defect detection results are more accurate or Different detection methods are used for the same first-type target, and then different defect detection methods can be integrated to obtain a more accurate defect detection result of the final image to be detected, which further reduces the false detection rate.
  • the defect detection module 33 is further configured to perform defect detection on the image area to obtain defect detection results about the first type of objects, including: in the case that the first type of objects is the first sub-type of objects , perform the step of performing the second target detection on the image area to obtain the first defect detection result; in the case that the first type of target is the second sub-type target, perform the step of classifying the image area to obtain the second defect detection result .
  • the above scheme by using different defect detection methods for different target objects, makes defect detection more flexible, and is more targeted than using the same detection method for all types of targets, making the defect detection results more accurate.
  • the first defect detection result includes position information of each first defect and a first probability of belonging to the first defect; the defect detection module 33 is further configured to perform second target detection on the image area, and obtain After the first defect detection result, it also includes at least one of the following: according to the position information of each first defect, determine the defect area of each first defect on the to-be-detected image, and determine the overlapping area between different defect areas. Perform deduplication processing; filter the first defect whose first probability satisfies the first filtering condition.
  • the overlapping part is deduplicated, so that a first defect corresponds to a defect area, which reduces the problem of multiple subsequent processing of a defect, and improves the accuracy of the final output.
  • the first filtering condition is that the first probability is lower than the first probability threshold; and/or, performing de-duplication processing on overlapping areas between different defect areas, including: The overlapping area of is subjected to non-maximum suppression.
  • the above scheme reduces many unnecessary processing areas and improves processing efficiency by eliminating the ones with low defect probability.
  • Deduplication is carried out by means of non-maximum value suppression, which can retain defect areas with higher defect probability, and improve the processing efficiency. Efficiency can also ensure the accuracy of defect detection.
  • the second defect detection result includes a second probability that the first type of target belongs to the second defect; the defect detection module 33 classifies the image area to obtain the second defect detection result, including: Perform classification to obtain the probability that the first type of target belongs to each preset defect; add the probabilities of each preset defect to obtain the second probability that the first type of target belongs to the second defect.
  • the above solution combines the probabilities of multiple preset defects to determine the defect probability of the first-type target, and extensively considers the possibility that the first-type target belongs to various preset defects, so that the defect probability of the first-type target is more accurate.
  • the first target detection module 31 performs the first target detection on the image to be detected to obtain the target detection result, including: using the first area detection network of the defect detection model to perform the first target detection on the image to be detected, obtaining the target detection result; the defect detection module 33 performs second target detection on the image area to obtain the first defect detection result, including: using the second area detection network of the defect detection model to perform second target detection on the image area to obtain the first defect The detection result; wherein, the depth of the second area detection network is shallower than that of the first area detection network; classifying the image area to obtain the second defect detection result, including: classifying the image area by using the classification network of the defect detection model, A second defect detection result is obtained.
  • the above scheme by considering the characteristics that different components may have different defects, uses different network models for detection, which can alleviate the problem of high missed detection rate in a single detection network to a certain extent.
  • the second target detection uses The depth of the network model of the first target detection is shallower than that of the network model used for the first target detection, which can reduce the amount of calculation in the second target detection process and improve the detection efficiency.
  • the target detection result further includes the second type of target and the second position information of the second type of target in the to-be-detected image.
  • the first target detection module 31 performs the first target detection on the to-be-detected image to obtain After the target detection result, it is further configured to: determine the second type of target as the third defect in the image to be detected, and use the second type of target and its second position information as the defect detection result about the second type of target.
  • one defect detection model can detect multiple targets, thereby enhancing the adaptability of the defect detection model.
  • the defect detection module 33 is further configured to: filter the defects satisfying the second filter condition in the defect detection result, so as to obtain the final defect detection result of the image to be detected.
  • the defect detection module 33 filters the defects that satisfy the second filter condition in the defect detection result, including: obtaining the size of the defect by using the position information of the defect in the defect detection result, and filtering the defect in the defect detection result that does not satisfy the size Defects with preset size conditions; and/or, filtering defects whose probability is lower than a second probability threshold in the defect detection result.
  • the defect detection module 33 is further configured to: output the final defect in descending order of the probability of each defect in the final defect detection result Information for each defect in the inspection results.
  • the defect detection device includes a training module (not shown in the figure), and the training module is configured to train a defect detection model.
  • the first target detection and the defect detection are performed by a defect detection model; wherein, the defect detection model is obtained by training the training module by at least performing the following steps: acquiring a first sample set, wherein the first The sample set includes at least one sample image, the sample image is marked with real defect information about the target, and the target includes the first type of target; the sample image in the first sample set is detected by using the defect detection model, and the defect detection result about the target is obtained. ; using real defect information and defect detection results to adjust the parameters of the defect detection model.
  • the accuracy of the defect detection model obtained by training in the above manner is higher, so that the first target detection and the defect detection are performed, so that the detection results are more prepared.
  • the defect detection model is jointly trained by multiple graphics processors, and the first sample set obtained by each graphics processor is different; the training module uses the defect detection model to analyze the first sample set. Detect the sample images of the target to obtain defect detection results about the target, including: synchronizing the batch normalization layer of the defect detection model in each graphics processor, and using the synchronized defect detection of the batch normalization layer in each graphics processor.
  • the model detects the sample images and obtains the defect detection results about the target; uses the real defect information and defect detection results to adjust the parameters of the defect detection model, including: determining the average loss of the defect detection model based on all defect detection results and real defect information value; use the average loss value to adjust the parameters of the defect detection model.
  • the above scheme by synchronizing the batch normalization layers of the defect detection model on multiple graphics processors, makes it possible to use the global first sample set for normalization, which is equivalent to increasing the batch size, thereby reducing the need for multiple graphics processing. processor impact, and training the defect detection model with multiple GPUs speeds up the training.
  • the training module further includes at least one of the following steps before using the defect detection model to detect the sample images in the first sample set to obtain defect detection results about the target: using the second sample set Pre-training the defect detection model; using the first sample set to pre-train the defect detection model, wherein the learning rate used in the warm-up training process is: starting from the first learning rate lower than the preset learning rate , and gradually increase to the preset learning rate; wherein, the sample images in the first sample set are obtained by at least the following steps: preprocessing the original images to obtain sample images, wherein the preprocessing methods include scale transformation, color transformation , one or more of Flip Horizontal, Flip Vertical, Rotate, and Crop.
  • the parameters of the defect detection model are initialized by using the second sample set to pre-train the defect detection model before the formal training of the defect detection model, or the learning rate is initialized before the formal training, and the The original image is preprocessed to improve the adaptability of the defect detection model.
  • FIG. 5 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • the electronic device 40 includes a memory 41 and a processor 42, and the processor 42 is configured to execute the program instructions stored in the memory 41, so as to implement the steps of the above embodiments of the defect detection method.
  • the electronic device 40 may include but is not limited to : a microcomputer, a server, and the electronic device 40 may also include mobile devices such as a notebook computer and a tablet computer, which are not limited herein.
  • the processor 42 is configured to control itself and the memory 41 to implement the steps in any of the above-mentioned embodiments of the defect detection method.
  • the processor 42 may also be referred to as a CPU (Central Processing Unit, central processing unit).
  • the processor 42 may be an integrated circuit chip with signal processing capability.
  • the processor 42 may also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • the processor 42 may be jointly implemented by an integrated circuit chip.
  • the defect detection is reduced. area, which can reduce the missed detection rate and false detection rate.
  • FIG. 6 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present application.
  • the computer-readable storage medium 50 stores program instructions 501 that can be executed by the processor, and the program instructions 501 are used to implement the steps of the foregoing defect detection method embodiments.
  • Embodiments of the present disclosure further provide a computer program product, which implements any one of the methods in the foregoing embodiments when the computer program product is executed by a processor.
  • the computer program product can be implemented in hardware, software or a combination thereof.
  • the computer program product is embodied as a computer storage medium, and in other embodiments of the present disclosure, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.
  • the defect detection is reduced. area, which can reduce the missed detection rate and false detection rate.
  • the functions or modules included in the apparatus provided in the embodiments of the present application may be configured to execute the methods described in the above method embodiments, and the specific implementation may refer to the descriptions in the above method embodiments. No longer.
  • the disclosed method and apparatus may be implemented in other manners.
  • the device implementations described above are only illustrative.
  • the division of modules or units is only a logical function division. In actual implementation, there may be other divisions.
  • units or components may be combined or integrated. to another system, or some features can be ignored, or not implemented.
  • the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.
  • the integrated unit if implemented as a software functional unit and sold or used as a stand-alone product, may be stored in a computer-readable storage medium.
  • the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, and the computer software products are stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the methods of the various embodiments of the present application.
  • the aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .
  • Embodiments of the present application provide a defect detection method, related devices, equipment, and storage media, wherein the method includes: performing a first target detection on an image to be detected, and obtaining a target detection result, wherein the target detection result includes a first type of target and a first target detection result. First position information of a type of target in the image to be detected; based on the first position information, an image area containing the first type of target is acquired; defect detection is performed on the image area to obtain a defect detection result about the first type of target.
  • the defect detection of the image to be inspected can reduce the rate of missed detection of defects and the rate of false detection of defects.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

一种缺陷检测方法和相关装置、设备、存储介质，缺陷检测方法包括：对待检测图像进行第一目标检测，得到目标检测结果，其中，目标检测结果包括第一类目标以及第一类目标在待检测图像的第一位置信息（S11）；基于第一位置信息，获取包含第一类目标的图像区域（S12）；对图像区域进行缺陷检测，得到关于第一类目标的缺陷检测结果（S13）。上述方法能够降低缺陷漏检率和误检率。

Description

缺陷检测方法和相关装置、设备、存储介质、计算机程序产品
相关申请的交叉引用
本申请基于申请号为202010837709.2、申请日为2020年08月19日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此引入本申请作为参考。
技术领域
本申请涉及但不限于图像处理领域，特别是涉及一种缺陷检测方法和相关装置、设备、存储介质、计算机程序产品。
背景技术
目前，为了提高对环境中设备的保护，经常需要对环境中的设备进行缺陷检测，以保证及时发现设备缺陷，进而进行维护。
相关技术中，设备的缺陷检测都是基于拍摄设备图像，并对该图像进行人工检查，以确定是否存在缺陷。然而，人工进行缺陷检测，极容易因为人为疏忽而存在漏检或错检的情况。因此，如何进行缺陷检测以降低缺陷漏检率，提高缺漏检测准确性，是目前极为关键的课题。
发明内容
本申请实施例至少提供一种缺陷检测方法和相关装置、设备、存储介质、计算机程序产品。
本申请实施例提供了一种缺陷检测方法，包括：对待检测图像进行第一目标检测，得到目标检测结果，其中，目标检测结果包括第一类目标以及第一类目标在待检测图像的第一位置信息；基于第一位置信息，获取包含第一类目标的图像区域；对图像区域进行缺陷检测，得到关于第一类目标的缺陷检测结果。
因此，通过先检测第一类目标在待检测图像中的位置，然后再对包含第一类目标的图像区域进行缺陷检测，由于直接针对目标所在的图像区域进行缺陷检测，即减小缺陷检测的区域，可降低漏检率和误检率。
在本申请的一些实施例中，对图像区域进行缺陷检测，得到关于第一类目标的缺陷检测结果，包括：对图像区域进行第二目标检测，得到第一缺陷检测结果，其中，第一缺陷检测结果包括第一类目标上存在的至少一种第一缺陷的信息；和/或，对图像区域进行分类，得到第二缺陷检测结果，其中，第二缺陷检测结果包括第一类目标所属第二缺陷的信息。
因此，通过多种方式对包括第一类目标的图像区域进行检测，可针对不同的第一类目标采用不同的检测方式，使得缺陷检测更具有针对性，从而得到的缺陷检测结果更准确或者为同一个第一类目标采用不同的检测方式然后可以综合各个不同的缺陷检测方式得出最终待检测图像的缺陷检测结果更准确，进一步减低误检率。
在本申请的一些实施例中，对图像区域进行缺陷检测，得到关于第一类目标的缺陷检测结果，包括：在第一类目标为第一子类目标的情况下，执行对图像区域进行第二目标检测，得到第一缺陷检测结果的步骤；在第一类目标为第二子类目标的情况下，执行对图像区域进行分类，得到第二缺陷检测结果的步骤。
因此，通过对不同的目标对象采用不同的缺陷检测方式，使得缺陷检测更加灵活，相比对所有种类的目标都采用同一种检测方式来讲，更有针对性，使得缺陷检测结果更准确。
在本申请的一些实施例中，第一缺陷检测结果包括每种第一缺陷的位置信息以及属于第一缺陷的第一概率；在对图像区域进行第二目标检测，得到第一缺陷检测结果之后，方法还包括以下至少一者：依据每种第一缺陷的位置信息，分别确定每种第一缺陷在待检测图像上的缺陷区域，并对不同缺陷区域之间的重叠区域进行去重处理；过滤第一概率满足第一过滤条件的第一缺陷。
因此，通过获取第一子类目标中第一缺陷的概率以及在包含第一子类目标的图像区域中位置信息，并且基于该位置信息找到第一缺陷在待检测图像中上对应的区域，并对重叠的部分进行去重处理，使得一个第一缺陷对应一个缺陷区域，减少出现对一个缺陷进行多次后续处理的问题，提高最后输出的准确率，而通过将满足过滤条件的区域剔除，能够减少后续处理的区域数量，提高处理效率。
在本申请的一些实施例中，第一过滤条件为第一概率低于第一概率阈值；和/或，对不同缺陷区域之间的重叠区域进行去重处理，包括：对不同缺陷区域之间的重叠区域进行非极大值抑制。
因此，通过将缺陷概率低的剔除，减少了很多不必要的处理区域，提高了处理效率，通过非极大值抑制的方式进行去重，能够保留缺陷概率更大的缺陷区域，在提高处理效率的同时还能保障缺陷检测的准确率。
在本申请的一些实施例中，第二缺陷检测结果包括第一类目标属于第二缺陷的第二概率；对图像区域进行分类，得到第二缺陷检测结果，包括：对图像区域进行分类，得到第一类目标属于每种预设缺陷的概率；将每种预设缺陷的概率相加，得到第一类目标属于第二缺陷的第二概率。
因此，综合多种预设缺陷的概率，确定该第一类目标的缺陷概率，广泛考虑第一类目标属于各种预设缺陷的可能性，使得第一类目标的缺陷概率更精确。
在本申请的一些实施例中，对待检测图像进行第一目标检测，得到目标检测结果，包括：利用缺陷检测模型的第一区域检测网络对待检测图像进行第一目标检测，得到目标检测结果；对图像区域进行第二目标检测，得到第一缺陷检测结果，包括：利用缺陷检测模型的第二区域检测网络对图像区域进行第二目标检测，得到第一缺陷检测结果；其中，第二区域检测网络的深度比第一区域检测网络的深度浅；对图像区域进行分类，得到第二缺陷检测结果，包括：利用缺陷检测模型的分类网络对图像区域进行分类，得到第二缺陷检测结果。
因此，通过考虑各个不同部件可能存在不同缺陷的特点，利用不同的网络模型分段进行检测，能够在一定程度上减轻了单一检测网络出现漏检率高的问题，同时，第二目 标检测使用的网络模型深度比第一目标检测使用的网络模型深度浅，能够减少第二目标检测过程中的计算量，提高检测效率。
在本申请的一些实施例中，目标检测结果还包括第二类目标以及第二类目标在待检测图像的第二位置信息，在对待检测图像进行第一目标检测，得到目标检测结果之后，方法还包括：将第二类目标确定为待检测图像中的第三缺陷，并将第二类目标及其第二位置信息作为关于第二类目标的缺陷检测结果。
因此，通过将不应该存在的第二类目标设置为第三缺陷，使得一个缺陷检测模型能够针对多种目标进行检测，加强了缺陷检测模型的适应性。
在本申请的一些实施例中，在得到缺陷检测结果之后，方法还包括：过滤缺陷检测结果中满足第二过滤条件的缺陷，以得到待检测图像的最终缺陷检测结果。
因此，通过过滤不满足条件的缺陷，减少了明显不合理的缺陷，使得最后得到的缺陷检测结果更精确。
在本申请的一些实施例中，过滤缺陷检测结果中满足第二过滤条件的缺陷，包括：利用缺陷检测结果中缺陷的位置信息得到缺陷的尺寸，过滤缺陷检测结果中尺寸不满足预设尺寸条件的缺陷；和/或，过滤缺陷检测结果中概率低于第二概率阈值的缺陷。
因此，通过过滤掉尺寸或概率不满足条件的，减少了明显不合理的缺陷，使得最后得到的缺陷检测结果更准确。
在本申请的一些实施例中，在得到待检测图像的最终缺陷检测结果之后，方法还包括：按照最终缺陷检测结果中每个缺陷的概率从高到低的顺序，输出最终缺陷检测结果中每个缺陷的信息。
因此，通过按照每个缺陷的概率从高到低的顺序输出最终缺陷检测结果中每个缺陷的信息，使得输出结果更整洁，便于后续进一步的观察。
在本申请的一些实施例中，基于第一位置信息，获取包含第一类目标的图像区域，包括：基于第一位置信息，从待检测图像中确定第一类目标对应的目标区域；将目标区域在待检测图像中向外扩预设倍数；在待检测图像中提取经外扩后的目标区域，以得到包含第一类目标的图像区域。
因此，通过将目标区域在待检测图像中向外扩预设倍数，使得能够在获取包含第一类目标对应的目标区域之后使得能够保留一些背景信息，能够提升对第一类目标缺陷检测的准确度。
在本申请的一些实施例中，对待检测图像进行第一目标检测，得到目标检测结果之前，方法还包括：将初始图像进行缩放，得到预设大小的图像；将经缩放后的初始图像的像素值压缩至预设像素值范围；对经压缩的初始图像进行归一化处理，得到待检测图像。
因此，通过将获取到的初始图像的尺寸或像素进行处理，得到统一图像的样式，在一定程度上提高了输入图像的鲁棒性。
在本申请的一些实施例中，第一目标检测和缺陷检测是由缺陷检测模型执行的；其中，缺陷检测模型至少由以下步骤训练得到：获取第一样本集，其中，第一样本集包括至少一个样本图像，样本图像标注有关于目标的真实缺陷信息，目标包括第一类目标； 利用缺陷检测模型对第一样本集中的样本图像进行检测，得到关于目标的缺陷检测结果；利用真实缺陷信息和缺陷检测结果，调整缺陷检测模型的参数。
因此，通过按照上述方式训练得到的缺陷检测模型的准确度更高，从而执行第一目标检测以及缺陷检测使得检测结果更准备。
在本申请的一些实施例中，缺陷检测模型是由多个图形处理器共同训练得到的，每个图形处理器获取的第一样本集不同；利用缺陷检测模型对第一样本集中的样本图像进行检测，得到关于目标的缺陷检测结果，包括：对每个图形处理器中的缺陷检测模型的批标准化层进行同步，并利用每个图形处理器中批标准化层同步后的缺陷检测模型对样本图像进行检测，得到关于目标的缺陷检测结果；利用真实缺陷信息和缺陷检测结果，调整缺陷检测模型的参数，包括：基于所有缺陷检测结果与真实缺陷信息，确定缺陷检测模型的平均损失值；利用平均损失值，调整缺陷检测模型的参数。
因此，通过同步多个图形处理器上中缺陷检测模型的批标准化层，使得可以使用全局的第一样本集进行归一化，相当于增大了批量大小，从而减少因为使用多个图形处理器的影响，并且通过多个图形处理器对缺陷检测模型进行训练加快了训练的速度。
在本申请的一些实施例中，在所利用缺陷检测模型对第一样本集中的样本图像进行检测，得到关于目标的缺陷检测结果之前，方法还包括以下至少一个步骤：利用第二样本集对缺陷检测模型进行预训练；利用第一样本集对缺陷检测模型进行预热训练，其中，预热训练的过程中采用的学习率为：从低于预设学习率的第一学习率开始，并逐步增大至预设学习率；其中，第一样本集中的样本图像至少由以下步骤得到：对原始图像进行预处理，得到样本图像，其中，预处理的方式包括尺度变换、颜色变换、水平翻转、竖直翻转、旋转、裁剪中的一种或多种。
因此，通过在正式对缺陷检测模型训练之前，先利用第二样本集对缺陷检测模型进行预训练使得对缺陷检测模型的参数进行了初始化，或在正式训练之前对学习率进行初始化，以及通过对原始图像进行预处理，提高了缺陷检测模型的适应性。
本申请实施例提供了一种缺陷检测装置，包括：第一目标检测模块，配置为对待检测图像进行第一目标检测，得到目标检测结果，其中，目标检测结果包括第一类目标以及第一类目标在待检测图像的第一位置信息；图像区域获取模块，配置为基于第一位置信息，获取包含第一类目标的图像区域；缺陷检测模块，配置为对图像区域进行缺陷检测，得到关于第一类目标的缺陷检测结果。
本申请实施例提供了一种电子设备，包括存储器和处理器，处理器配置为执行存储器中存储的程序指令，以实现上述缺陷检测方法的部分或全部步骤。
本申请实施例提供了一种计算机可读存储介质，其上存储有程序指令，程序指令被处理器执行时实现上述缺陷检测方法的部分或全部步骤。
本申请实施例提供一种计算机程序产品，其中，所述计算机程序产品包括存储了计算机程序的非瞬时性计算机可读存储介质，所述计算机程序被计算机读取并执行时，实现如本申请实施例中所描述的方法的部分或全部步骤。该计算机程序产品可以为一个软件安装包。
本申请实施例中，通过先检测第一类目标在待检测图像中的位置，然后再对包含第 一类目标的图像区域进行缺陷检测，由于直接针对目标所在的图像区域进行缺陷检测，即减小缺陷检测的区域，可降低漏检率和误检率。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，而非限制本申请。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，这些附图示出了符合本申请的实施例，并与说明书一起用于说明本申请实施例的技术方案。
图1a是本申请实施例提供的一种缺陷检测方法的流程示意图；
图1b是本申请实施例提供的一种缺陷检测方法的状态示意图；
图2是本申请实施例提供的一种缺陷检测方法中待检测图像的示意图；
图3是本申请实施例提供的一种缺陷检测方法中图像区域的示意图；
图4a是本申请实施例提供的一种缺陷检测方法的实现流程示意图；
图4b是本申请实施例提供的一种图像预处理模块的实现流程示意图；
图4c是本申请实施例提供的一级部件检测模块的网络结构示意图；
图4d是本申请实施例提供的一种防振锤分类识别的实现流程示意图；
图4e是本申请实施例提供的二级绝缘子自爆检测模块的网络结构示意图；
图5是本申请实施例提供的一种缺陷检测装置的结构示意图；
图6是本申请实施例提供的一种电子设备的结构示意图；
图7是本申请实施例提供的一种计算机可读存储介质的结构示意图。
具体实施方式
下面结合说明书附图，对本申请实施例的方案进行详细说明。
以下描述中，为了说明而不是为了限定，提出了诸如特定系统结构、接口、技术之类的具体细节，以便透彻理解本申请实施例。
本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。此外，本文中的“多”表示两个或者多于两个。另外，本文中术语“至少一种”表示多种中的任意一种或多种中的至少两种的任意组合，例如，包括A、B、C中的至少一种，可以表示包括从A、B和C构成的集合中选择的任意一个或多个元素。
为了更好地理解本申请实施例提供的缺陷检测方法，下面先对相关技术中的缺陷检测方法进行说明。
以电网传输线路中的缺陷检测为例，我国具有超过百万公里的高压输电线路，这些线路需要定期的巡查、养护和维修，但由于大多数高压输电线架设在无人区域，电网巡检都是通过飞机航拍输电线路，人工肉眼观测航拍视频的方式进行巡检。为了提升效率，在相关技术中采用基于图像处理、机器学习的部件缺陷检测方案，但该方案漏检率较高，对拍摄视频的要求也较高。相关技术中，可以采用安装高清摄像头的无人机接航拍输电线路，使得整体的采集拍摄质量得到提升，成本也得到降低。但是由于仍然采用人工进 行缺陷检测，导致漏检率和误报率依旧较高。随着深度学习模型在计算机视觉领域取得突破，计算机视觉各项基本任务的性能指标大幅提高。但所有模型都更针对学术数据集相关任务，并没有针对电力巡检场景单独设计的检测识别方案。由于无人机航拍图像尺度很大，而电网巡检中需要检测的绝缘子自爆区域、防振锤、鸟巢等都属于很小的目标，所以直接将异常缺陷当做检测目标进行检测会存在大量误检和漏检，检测效果很差。
上述相关技术中的缺陷检测方法存在如下问题：1)训练流程繁琐，漏检率和误检率高；2)使用CPU进行推理，测试速度慢；3)算法没有针对电力场景进行改良；4)只能单一地对一种部件或缺陷进行检测识别，如：绝缘子自爆、防振锤锈蚀等。
请参阅图1a，图1a是本申请实施例提供的一种缺陷检测方法的流程示意图。该方法可以包括如下步骤：
步骤S11：对待检测图像进行第一目标检测，得到目标检测结果，其中，目标检测结果包括第一类目标以及第一类目标在待检测图像的第一位置信息。
在本申请的一些实施例中，在对待检测图像进行第一目标检测之前，先获取待检测图像。参见图1b，图1b是本申请实施例提供的一种缺陷检测方法的状态示意图。如图1b所示，图像采集设备在特定环境中采集各待检测对象的待检测图像，例如，包含第一类目标的待检测图像，然后图像采集设备通过将待检测图像传输给缺陷检测设备，其中缺陷检测设备按照本申请实施例描述的方法对图像采集设备传输过来的待检测图像进行缺陷检测。第一类目标也就是需要对其进行检修的设备，只要设备真实存在，都可以成为本申请实施例的第一类目标。例如，城市中某路口的红绿灯、下水道井盖等等，通过图像采集设备即可获取到可能包含红绿灯或井盖的待检测图像，然后将可能包含红绿灯或井盖的待检测图像输入到缺陷检测设备，进而得到红绿灯的缺陷检测结果。
例如，在电网传输线路中，可能需要对线路中的各种设备进行检修，此时可以通过拍照等方式获取线路中各个线路设备的航拍图，例如绝缘子串或者防振锤的航拍图，亦或者鸟巢的航拍图。然后将获取到的航拍图作为待检测图像，输入缺陷检测设备中，然后缺陷检测设备即可根据本申请实施例所描述的方法对待检测图像进行缺陷检测。其中，图像采集设备和缺陷检测设备可以集成为一个设备，也可以是分开的多个设备，此处不做具体规定。
在本申请的一些实施例中，在对待检测图像进行第一目标检测之前，可将初始图像进行缩放到预设大小的图像。其中，预设大小包括宽和高的比例为(1.1～1.7):1和/或尺寸较小的一边大于1000。本申请实施例中采用的宽和高的比例为1.5:1，最小边为1200。接着将经过缩放后的初始图像的像素值压缩到预设像素值范围。例如将经过缩放后的初始图像的像素值从0-255压缩到0-1。然后对经压缩之后的初始图像进行归一化处理，得到待检测图像。其中，归一化处理的方式包括使用ImageNet数据集的均值mean和方差std对压缩之后的初始图像进行归一化处理，如下公式(1-1)所示：
Figure PCTCN2020136251-appb-000001
其中，mean为数据集的均值，std为数据集的方差，pixelvalue为图像中各像素的值。
当然，在本申请一些实施例中也可采用其他的数据集，例如COCO(Common Objects  in Context)数据集等，以及归一化的方式也不必限定于通过调用其他数据集的方式进行归一化，还可以平常使用的其他归一化方式，例如遍历初始图像中全部像素点的像素值，并将最大值和最小值记录下来，通过最大值和最小值作为参数，对初始图像进行归一化处理，还可以是利用Sigmoid函数进行归一化，因此，对于归一化的方式本申请实施例不做具体限定。通过先对获取到的图像进行上述预处理，统一图像的样式，能够提高输入的鲁棒性。
本申请实施例中，通过缺陷检测模型的第一区域检测网络对待检测图像进行第一目标检测，得到目标检测结果。
其中，缺陷检测模型可以由以下步骤训练得到：
首先，获取第一样本集，其中，第一样本集包括至少一个样本图像，而样本图像标注有关于目标的真实缺陷信息，这里的目标包括第一类目标，当然还可包括其他类别的目标，例如第二类目标等等。第一样本集中的样本图像可以只包含第一类目标同时包含其他类别的目标，其中，目标的真实缺陷信息可以表示该目标是正常的没有缺陷的信息，也可以是第一类目标具有特定缺陷的信息。也就是说，本申请实施例中，可以利用正常的样本图像以及异常的有缺陷的样本图像对缺陷检测模型进行训练，得到适应性较强的缺陷检测模型。其中，第一样本集中的样本图像可以通过对原始图像进行预处理得到，其中，预处理的方式可以是尺度变换、颜色变换、水平翻转、竖直翻转、旋转、裁剪中的一种或多种。在一些实施例中，原始图像的尺度变换过程中，原始图像的长宽呈预设比例进行变换，例如原始图像的长宽比值在保持1.5:1的比例的情况下进行尺度变换。当然在本申请一些实施例中，也可保持其他的长宽比例或者任意变换均可，因此，原始图像的尺度变换的长宽比例本申请实施例不做具体规定。其中，颜色的变换可以是亮度、饱和度、色度变换中的一种或多种，例如将一种原始图像的亮度以及饱和度、色度都变换，使得包含相同目标的同一真实缺陷信息通过不同的形式输入到缺陷检测模型中，对缺陷检测模型进行训练，使得缺陷检测模型能够增强适应性以及提高缺陷检测的准确性。在一些实施例中，上述各种预处理方式可以在缺陷检测模型不同阶段的网络中单独使用，也可以在所有阶段的网络中使用。
其次，利用缺陷检测模型对第一样本集中的样本图像进行检测，得到关于目标的缺陷检测结果。其中，缺陷检测模型可以由多个图形处理器共同训练得到的，每个图形处理器获取的第一样本集不同，通过采用多个图形处理器对缺陷检测模型进行检测使得成倍加快了训练的速度。在一些实施例中，对每个图形处理器中的缺陷检测模型的批标准化层进行同步，并利用每个图形处理器中批标准化同步后的缺陷检测模型对样本图像进行检测，得到关于目标的缺陷检测结果。其中一个图形处理器收集其他图形处理器中的缺陷检测模型的批标准化层的均值和方差，从而计算得到一个总的均值和方差，然后这个图形处理器再将计算得到的均值和方法返回至其余的图形处理器中的缺陷检测模型的批标准化层，通过同步多个图形处理器上中缺陷检测模型的批标准化层，使得可以使用全局的第一样本集进行归一化，相当于增大了批量大小，从而减少因为使用多个图形处理器的影响。
最后，利用真实缺陷信息和缺陷检测结果，调整缺陷检测模型的参数。在一些实施 例中，基于所有缺陷检测结果与真实缺陷信息，确定缺陷检测模型的平均损失值，然后利用平均损失值，调整缺陷检测模型的参数。通过多张图形处理器同步对缺陷检测模型进行训练使得训练之后的所有图形处理器中的缺陷检测模型的参数一致，相比单个图形处理器来讲，使用多张图形处理器同步进行对缺陷检测模型进行训练加快了训练的进度。
在本申请的一些实施例中，在利用缺陷检测模型对第一样本集中的样本图像进行检测，得到关于目标的缺陷检测结果之前，还可利用第二样本集对缺陷检测模型进行预训练，和/或利用第一样本集对缺陷检测模型进行预热训练，其中，预热训练的过程中采用的学习率为：从低于预设学习率的第一学习率开始，并逐步增大至预设学习率。例如，预设学习率为0.6，则预热训练中，可以从第一学习率为0.3逐步增加到0.4，然后不断增加到预设学习率0.6。其中，第二样本集可以是公开训练集，例如，可以是常见的COCO训练集以及ImageNet数据集等。预热训练中使用的样本可以是与正式训练使用的样本相同，即在预热训练中仍然使用第一样本集。在一些实施例中，经过第二样本集的预训练对参数进行初始化之后，再利用第一样本集进行预热训练不断调整学习率，在预热训练不断调整学习率的过程中，也可以算得上是对缺陷检测模型的参数进行进一步的优化，使得正式训练的时候能够在较优的参数上进行训练。在一些实施例中，在预热训练过程中使用的第一样本集仍然可以是经过上述预处理之后得到的第一样本集，也是能够提高缺陷检测模型的适应性，使得训练得到的缺陷检测模型的检测精度更高。在一些实施例中，从第一学习率逐步线性增长或指数型增长到预设学习率，相当于对学习率进行初始化，使得能够从一个较优的学习率对缺陷检测模型进行训练使得训练的效果更好。
其中，本申请实施例中的第一区域检测网络包括Faster R-CNN(Region with Convolutional Neural Networks)网络，例如ResNet50。当然，在其他实施例中还可以采用其他的神经网络模型对待检测图像进行第一目标检测，例如SSD(Single Shot MultiBox Detector)网络模型等等，且深度也可不必限定为ResNet50，还可以是ResNet101、ResNet200等等，此处不做具体限定。
本申请实施例中采用的第一区域检测网络为ResNet50网络，通过将待检测图像输入第一区域检测网络的头部网络，提取得到待检测图像的特征图，其中，特征图的大小为待检测图像大小的十六分之一，接着通过区域候选网络(Region Proposal Networks，RPN)从特征图中提取候选框，候选框指的是可能存在第一类目标的区域，其中，提取候选框实质上就是提取候选框四个顶点的像素值，然后将候选框与特征图一起经过池化层得到每一个候选框的特征图，该步骤具体包括将候选框的四个坐标的信息送入池化层，在池化层中将候选框的四个顶点坐标映射到特征图上，再经过全连接层得到每个候选框的特征向量，其中，这些特征向量表示了候选框的特征信息，然后对特征向量进行边框回归以及按照预设类别进行分类，最后得到目标检测结果。由此可见，第一目标检测也就是指的是对待检测图像中是否包含第一类目标进行检测，若存在第一类目标则进一步获取第一类目标在待检测图像中的第一位置信息，还可进一步得知第一类目标的置信度，也就是检测到的第一类目标真正属于第一类目标的概率。
其中，目标检测结果包括第一类目标以及第一类目标在待检测图像中的第一位置信 息，和/或候选框属于第一类目标的概率。在一些实施例中，第一位置信息包括第一类目标对应候选框在待检测图像中的位置信息。第一位置信息指的是第一类目标对应候选框四个顶点的位置信息。其中，第一类目标还可包括第一子类目标以及第二子类目标。其中，这里的第一子类目标以及第二类子目标与第一类目标之间可以没有所属关系，但是可以具有相同属性。例如，第一类目标可以是一种设备，而第一子类目标可以是另外一种设备，第二类子目标却可以是其他的设备，相同的属性就是可以通过拍照，然后分析对应图像即可知道设备是否具有缺陷。属于第一类目标的概率还可进一步包括属于第一子类目标的概率和/或属于第二子类目标的概率。
在本申请的一些实施例中，目标检测结果还包括第二类目标以及第二类目标在待检测图像中的第二位置信息，和/或属于第二类目标的概率。其中，这里的第二类目标可以是本不应该存在而存在的目标，它的存在即是一种缺陷。当然，在一些实施例中，如果对初始图像进行了处理之后才得到了待检测图像，那么在对待检测图像进行第一目标检测之后得到的目标检测结果中包括第一类目标和/或第二类目标在初始图像中的第一位置信息。通过进一步设置第二类目标，能够使得同一套算法检测多个不同目标对象的缺陷状况，不需要运用多套算法对同一张图像中的不同部件或设备分别进行缺陷检测，很大程度上简化了缺陷检测的操作步骤。
举例说明，初始图像为无人机航拍的输电线路的图像，或者说通过在高清视频中切割出来的图形帧，先将初始图像进行处理，包括对初始图像进行缩放，缩放到1200*1800的图像，然后将经过缩放之后的初始图像的像素值压缩到预设像素值范围即0-1之间，然后使用ImageNet数据集的均值mean和方差std对压缩后的初始图像进行归一化运算，得到待检测图像。然后将待检测图像输入ResNet50网络获取待检测图像中每个候选框内目标对象对应每个预设目标类别的概率，选取概率最高的预设目标类别作为候选框的中目标对象的类别，同时还会输出每个候选框的位置信息。其中，在输电线路中，常见的部件包括防振锤、绝缘子串，也可能包括外来缺陷也即是本不应该存在而存在，其存在就是一种缺陷的第二类目标，即鸟巢。其中，本申请实施例中，暂将防振锤以及绝缘子串纳入第一类目标，将鸟巢纳入第二类目标。即，通过将待检测图像输入ResNet50之后，输出的目标检测结果可能包括防振锤、绝缘子或鸟巢，以及各自在待检测图像中对应的的位置信息以及相关概率。这里的概率指的是，候选框属于防振锤、绝缘子串或鸟巢的概率。
在本申请的一些实施例中，在得到目标检测结果之后，将第二类目标确定为待检测图像中的第三缺陷，并将第二类目标以及第二位置信息作为关于第二类目标的缺陷检测结果。例如，第二类目标为鸟巢，若在待检测图像中检测到了鸟巢，则确定鸟巢为待检测图像中的第三缺陷，获取鸟巢在待检测图像中的第二位置信息，所以最后目标检测结果中则包含鸟巢以及鸟巢的第二位置信息。
步骤S12：基于第一位置信息，获取包含第一类目标的图像区域。
本申请实施例中，基于第一位置信息，从待检测图像中确定第一类目标对应的目标区域。具体包括通过从待检测图像中确定包含第一类目标对应的区域，也就是确定第一类目标对应候选框。接着将目标区域在待检测图像中向外扩预设倍数。其中，外扩的方 式包括目标区域的中心点不变，目标区域的长和宽变为原目标区域的1.1～1.5倍，其中，本申请实施例采用将目标区域的长和宽变为原目标区域的1.2倍左右。若目标区域的边界超出待检测图像的边界，则保留待检测图像中的部分。然后在待检测图像中提取经外扩后的目标区域，以得到包含第一类目标的图像区域。具体包括将外扩之后的目标区域从待检测图像中裁剪出来。
同时参见图2及图3，图2是本申请实施例提供的一种缺陷检测方法中待检测图像的示意图，图3是本申请实施例提供的一种缺陷检测方法中图像区域的示意图。如图2和图3所示，待检测图像1中分别包括了第一类目标100，其中，通过第一目标检测网络得到第一类目标100的位置信息，然后基于第一类目标100的位置信息，获取包含第一类目标100的图像区域110，为了更清楚地说明图像区域110以及候选框101之间的关系，在图3的图像区域中示出了第一类目标100的候选框101。因为从待检测图像1中获取包含第一类目标100的图像区域110时，将第一类目标100的候选框101在待检测图像1中进行了一定比例的外扩，因此，得到的图像区域110是比原本候选框101所在区域的面积要大。
在本申请的一些实施例中，若检测图像是由初始图像经过缩放、压缩像素值以及归一化处理得到时，则基于第一位置信息，获取包含第一类目标的图像区域指的是基于第一类目标在初始图像中的第一位置信息，在初始图像中确定第一类目标对应的目标区域，即确定初始图像中第一类目标对应的候选框的位置，并将目标区域在初始图像中外扩预设倍数，在初始图像中提取外扩后的目标区域，以得到包含第一类目标的图像区域。当然，因为待检测图像与初始图像对应，所以在本申请的另一些实施例中，即使检测图像是由初始图像经过缩放、压缩像素值以及归一化一系列处理得到的，基于第一位置信息，获取包含第一类目标的图像区域的具体步骤也可包括从待检测图像中提取第一类目标的图像区域。因此，在待检测图像中获取包含第一类目标的图像区域还是在初始图像中获取第一类目标的图像区域都可以，此处不做具体限定。
因此，通过将目标区域在待检测图像中向外扩预设倍数，使得能够在获取包含第一类目标对应的目标区域之后使得能够保留一些背景信息，能够提升对第一类目标缺陷检测的准确度。
在本申请的一些实施例中，在获取包含第一类目标的图像区域之后，需要对图像区域进行预处理，即调整图像区域的尺寸，其中，可以将图像区域的长宽比例调整为1:1。如果图像区域是从初始图像中获得，则需要对图像区域进行归一化，如果是从经过归一化处理之后的待检测图像中获得，则此处可以不再进行归一化。
例如，基于绝缘子串在待检测图像中的第一位置信息，确定绝缘子串在待检测图像中对应的候选框，将候选框在待检测图像中向外扩1.2倍，然后从待检测图像中提取外扩之后的候选框，即可得到包含绝缘子的图像区域。
步骤S13：对图像区域进行缺陷检测，得到关于第一类目标的缺陷检测结果。
对图像区域进行缺陷检测的方式包括以下至少一种，一种是对图像区域进行第二目标检测，得到第一缺陷检测结果，其中，第一缺陷检测结果包括第一类目标上存在的至少一种第一缺陷的信息，另一种是对图像区域进行分类，得到第二缺陷检测结果，其中， 第二缺陷检测结果包括第一类目标所述第二缺陷的信息。其中，第二目标检测指的是对第一类目标进行进一步的目标检测，在第一目标检测中得到的是第一类目标的位置信息，并没有直接检测出第一类目标的缺陷信息，因此，第二目标检测则对第一类进行针对性的缺陷检测。对图像区域进行分类指的就是将图像区域中第一类目标按照预设的缺陷类别进行分类，计算第一类目标属于各个缺陷类别的概率信息。
通过多种方式对包括第一类目标的图像区域进行检测，可针对不同的第一类目标采用不同的检测方式，使得缺陷检测更具有针对性，从而检测结果更准确。或者说为同一个第一类目标采用不同的检测方式然后可以综合各个不同的缺陷检测方式得到的结果得出最终待检测图像的缺陷信息，使得缺陷检测结果更准确。
其中，在第一类目标为第一子类目标的情况下，执行对图像区域进行第二目标检测，得到第一缺陷检测结果的步骤，而在第一类目标为第二子类的情况下，执行对图像区域进行分类，得到第二缺陷检测结果的步骤。
通过对不同的目标对象采用不同的缺陷检测方式，使得缺陷检测更加灵活，相比对所有种类的目标都采用同一种检测方式来讲，针对不同的目标对象采用不同的缺陷检测方式更有针对性，使得检测结果更准确。
例如，预设绝缘子串为第一子类目标，预设防振锤为第二子类目标，因此，当检测出第一类目标为绝缘子串时，则对绝缘子串对应的图像区域进行第二目标检测，以得到第一缺陷检测结果，当待检测图片中检测出的第一类目标为防振锤时，则对防振锤对应的图像区域进行分类，以得到第二缺陷检测结果。而当检测出第一类目标既包括绝缘子串也包括防振锤时，则分别对绝缘子串对应的图像区域进行第二目标检测，对防振锤对应的图像区域进行分类，一些实施例中，对二者的缺陷检测可同时进行，即同时进行第二目标检测和分类。当然，在一些实施例中，也可针对同一第一子类目标分别进行第二目标检测以及图像区域分类，然后综合两种处理方式得到的结果，最终得出对第一子类目标的第一缺陷检测结果。
第一缺陷检测结果包括每种第一缺陷的位置信息以及属于第一缺陷的第一概率。其中，第一缺陷的位置信息指的是第一缺陷在包含第一类目标的图像区域中的位置，属于第一缺陷的概率指的是检测得到的第一缺陷的置信度，也就是第一缺陷真正属于是第一缺陷的概率。例如，对包含绝缘子串的图像区域进行第二目标检测，检测到绝缘子串上的A位置可能出现了绝缘子自爆，其中，绝缘子自爆这一缺陷即属于第一缺陷，而绝缘子串上的A位置则属于绝缘子自爆的第二位置信息，绝缘子自爆的置信度即为属于绝缘子自爆的第一概率。
本申请实施例中，当第一类目标为第一子类目标时，对图像区域进行第二目标检测中，使用的检测网络为第二区域检测网络，其中，第二区域检测网络的深度比第一区域检测网络的深度要浅。例如，当第一区域检测网络为ResNet50时，第二区域检测网络可以是ResNet18。其中，因为第二目标检测中使用的网络模型和第一目标检测中使用的网络模型都是Faster R-CNN网络，因此，第二目标检测具体的过程此处不再赘述。与第一目标检测不同的是第一目标检测中会针对可能包含不同的第一类目标的待检测图像进行检测，判断待检测图像中是否包含第一类目标以及获取第一类目标的第一位置， 而第二目标检测中仅针对第一子类目标所在的图像区域进行检测，具体包括检测第一子类目标存在缺陷的概率。
例如，将包含绝缘子串的图像区域输入第二区域检测网络ResNet18中后，会对绝缘子串中是否出现了绝缘子自爆进行检测，若存在则输出的将是绝缘子串中出现绝缘子自爆的位置信息以及自爆的概率信息。
本申请实施例中，在对图像区域进行第二目标检测，得到第一缺陷检测结果之后，可以对第一缺陷检测结果进行处理。其中包括依据每种第一缺陷的位置信息，分别确定每种第一缺陷在待检测图像上的缺陷区域。因为通过第二目标检测获得的位置信息是第一缺陷在包含第一类目标的图像区域上的位置信息，还需要通过映射的方式找到第一缺陷在待检测图像上的缺陷区域。接着对缺陷区域进行预设处理，其中，预设处理包括以下至少一者，即对不同缺陷区域之间的重叠区域进行去重处理，过滤第一概率满足第一过滤条件的第一缺陷。其中，对不同缺陷区域之间的重叠区域进行去重处理的方式又可进一步包括对不同缺陷区域之间的重叠区域进行非极大值抑制。当重叠区域的面积达到预设面积时，获取互相重叠的缺陷区域的第一概率，并对各自的概率进行比较，保留概率较大的缺陷区域，舍弃概率较小的缺陷区域。其中，第一过滤条件为第一概率低于第一概率阈值。即当第一缺陷概率低于第一概率阈值时，则剔除掉这部分第一缺陷对应的缺陷区域。并且同时将第一缺陷的位置信息及第一概率从第一缺陷检测结果中删除，也就是基于仅预设处理后得到的缺陷区域确定最终的第一缺陷检测结果。
通过获取第一子类目标中第一缺陷的概率以及在包含第一子类目标的图像区域中位置信息，并且基于该位置信息找到第一缺陷在待检测图像中上对应的区域，并对重叠的部分进行去重处理，使得一个第一缺陷对应一个缺陷区域，减少出现对一个缺陷进行多次后续处理的问题，提高最后输出的准确率，而通过将满足过滤条件的区域剔除，能够减少后续处理的区域数量，提高处理效率。
在本申请的一些实施例中，通过将缺陷概率低的剔除，减少了很多不必要的处理区域，提高了处理效率，通过非极大值抑制的方式进行去重，能够保留缺陷概率更大的缺陷区域，在提高处理效率的同时还能保障缺陷检测的准确率。
当第一类目标为第二子类目标时，第二缺陷检测结果包括第一类目标属于第二缺陷的第二概率。在一些实施例中，这里的第二缺陷指的就是第二子类目标是否存在缺陷，其中，第二概率也就是第二子类存在缺陷的概率。当然在其他实施例中，第二缺陷可以包括多个不同的子缺陷，第二概率为第二子类目标属于各个子缺陷的概率。对图像区域进行分类的过程包括：对图像区域进行分类，得到第一类目标属于每种预设缺陷的概率，也即是第二子类目标属于每种预设缺陷的概率。例如，当第一类目标为第二子类目标，例如防振锤时，对包含防振锤的图像区域进行分类，得到防振锤分别属于正常防振锤、防振锤锈蚀、防振锤扭转、防振锤损坏、防振锤脱落的概率，例如属于正常防振锤的概率为0.1，属于防振锤锈蚀的概率为0，属于防振锤扭转的概率为0.2属于防振锤损坏的概率为0.5，属于防振锤脱落的概率为0。此时，可以得出防振锤属于预设缺陷防振锤损坏的概率最高。在本申请的一些实施例中，可以通过这个概率认定防振锤属于防振锤损坏。本申请实施例中在得出第一类目标属于每种预设缺陷的概率之后，将每种预设缺陷 的概率相加，得到第一类目标属于第二缺陷的第二概率。即将防振锤的四种缺陷概率进行相加，得出此防振锤有缺陷的概率为0.7。此时的第二缺陷即为防振锤存在缺陷，第二概率则为防振锤有缺陷的概率。通过综合多种预设缺陷的概率，确定该第一类目标的缺陷概率，广泛考虑第一类目标属于各种预设缺陷的可能性，使得第一类目标的缺陷概率更精确。
在对图像区域进行分类的过程中，使用的网络为分类网络。其中，该分类网络的深度也可比第一区域检测网络的深度浅，此时的分类网络同样可以是ResNet18，当然还可以是其他深度的网络，例如当第一区域检测网络为ResNet101时，分类网络还可以是ResNet50，当然在一些实施例中，分类网络的深度可与第一区域检测网络的深度相同。因此关于分类网络的选择，此处不做具体限定。
通过考虑各个不同部件的不同缺陷的特点，利用不同的网络模型分段进行检测，能够在一定程度上解决单一网络检测或分类器出现漏检率高的问题，同时，第二目标检测使用的网络模型深度比第一目标检测使用的网络模型深度浅，能够减少第二目标检测过程中的计算量，提高检测效率。
在本申请的一些实施例中，在得到缺陷检测结果之后，过滤缺陷检测结果中满足第二过滤条件的缺陷。其中，这里的缺陷检测结果可以是第一缺陷检测结果和第二缺陷检测结果以及关于第二类目标的缺陷检测结果的组合。利用缺陷检测结果中各个缺陷的位置信息得到的缺陷的尺寸，过滤缺陷检测结果中尺寸不满足预设尺寸条件的缺陷。例如，过滤掉尺寸过大或者过小的绝缘子自爆的缺陷区域或者鸟巢的区域以及异常防振锤所在区域。或者过滤缺陷检测结果中概率低于第二概率阈值的缺陷。具体包括可对不同类的目标设置不同的阈值，例如，第一目标的第一子类目标设置阈值为0.3，第二子类目标设置为0.2，第二类目标设置为0.2等等。然后根据预设阈值对缺陷检测结果中的缺陷进行过滤。当然，也可同时对缺陷检测结果中缺陷的尺寸以及缺陷的概率进行过滤，减少缺陷检测结果中缺陷的数量。最后，按照最终缺陷检测结果中每个缺陷的概率从高到底的顺序输出缺陷检测结果中每个缺陷的信息。其中，输出的缺陷的信息包括缺陷在待检测图像中的位置信息和概率，或者直接输出缺陷在初始图像中的位置信息和相应的概率。
在本申请的一些实施例中，输出的形式可以包括以字符的形式输出缺陷在待检测图像和/或初始图像中的位置和概率，还可以是利用标注的方式将所有的缺陷直接在待检测图像和/或初始图像中标识出来，并在标识框内写出各自的概率，或者还可以是直接输出各个缺陷在待检测图像和/或初始图像中对应的图像区域，并在图像区域中写明缺陷的概率。
通过对缺陷检测结果中的缺陷进行过滤使得过滤之后的缺陷检测结果中尽可能少包含属于正常情形下的信息或者说排除明显不合理的情形，从而提高了缺陷检测结果的合理性，并且还会使得误检率下降。
上述方案，通过先检测第一类目标在待检测图像中的位置，然后再对包含第一类目标的图像区域进行缺陷检测，由于直接针对目标所在的图像区域进行缺陷检测，即减小缺陷检测的区域，可降低漏检率和误检率。
本领域技术人员可以理解，在具体实施方式的上述方法中，各步骤的撰写顺序并不意味着严格的执行顺序而对实施过程构成任何限定，各步骤的具体执行顺序应当以其功能和可能的内在逻辑确定。其中缺陷检测方法的执行主体可以是缺陷检测装置，例如，缺陷检测方法可以由终端设备或服务器或其它处理设备执行，其中，终端设备可以为用户设备(User Equipment，UE)、移动设备、用户终端、终端、蜂窝电话、无绳电话、个人数字处理(Personal Digital Assistant，PDA)、手持设备、计算设备、车载设备、可穿戴设备等。在一些实施例中，该缺陷检测方法可以通过处理器调用存储器中存储的计算机可读指令的方式来实现。
下面，将说明本申请实施例在一个实际的应用场景中的示例性应用。本申请实施例提出了一种基于深度学习的电网传输线路中多类别缺陷检测的方法，使用深度学习模型对大尺度的输入图片进行两阶段的检测和分类，该方法可以联合进行绝缘子自爆检测、防振锤缺陷检测、鸟巢检测等异常缺陷识别。参见图4a，该方法的实现流程主要包括如下功能模块：RGB图像输入模块410、图像预处理模块420、一级部件检测模块430、二级防振锤分类识别模块440、二级绝缘子自爆检测模块450、自爆检测后处理模块460以及异常缺陷筛选模块470。下面分别对各个模块进行具体说明：
1)RGB图像输入模块：该模块可以获取不同无人机摄像头拍摄的数据输出，得到无人机航拍的RGB图或高清视频，对于高清视频，可切分成图像帧得到相应的RGB图。
2)图像预处理模块：参见图4b，该模块可以对RGB图像做统一的处理，得到可用于缺陷检测的待输入图像，处理过程包括：a)图像缩放421：对输入的高清RGB图像进行缩放，缩放至大小为1200×1800的图像；b)像素值压缩422：将缩放后的图像像素值从0-255压缩到0-1；c)像素值归一化423：使用ImageNet数据集的均值mean和方差std对压缩像素值后的图像按照前述公式(1-1)进行归一化计算。
3)一级部件检测模块：该模块的输入为归一化后的RGB图像，通过目标检测网络Faster R-CNN检测鸟巢、绝缘子串、防振锤三类一级部件，得到绝缘子串区域、防振锤区域和鸟巢区域三类候选区域，以及每一候选区域对应的得分。该模块的主要网络结构如图4c所示，使用ResNet50头部网络431作为检测网络的骨干网络，提取得到C4层特征图432。该模块进行一级部件检测的流程包括：RGB图像经过ResNet50头部网络431得到C4层特征图432，特征图大小为输入图像的1/16，ResNet50头部网络431由不断堆叠至少一个卷积层、至少一个归一化层、下采样层组成，以跳层连接的方式进行前向运算；C4层特征图432经过RPN提取候选框433，提取的候选框为可能存在目标的区域，候选框与C4层特征图一起经过RoI池化层434得到每一个候选区域的特征图，再经过至少一个全连接层435得到每个候选区域的特征向量，候选区域的特征向量表征了候选区域的特征信息；最后对每个候选区域的特征向量进行框坐标回归436和多类别分类437，得到最终的检测结果，输出鸟巢、绝缘子串、防振锤的原图位置区域和相应的概率值，其中鸟巢区域直接送入异常缺陷筛选模块，绝缘子串区域送入二级自爆检测模块进行自爆检测，防振锤区域送入二级防振锤分类识别模块进行防振锤类别的分类。
4)二级防振锤分类识别模块：该模块以一级部件检测模块检测的防振锤区域作为输入，主要通过ResNet18分类网络对原图剪裁的防振锤区域进行异常类型的多分类(分 类的类别也包含正常类)，最后对缺陷类别概率进行整合，得到正常的防振锤及不同异常的防振锤。该模块依据防振锤的形态将防振锤划分为以下五类：正常防振锤、防振锤锈蚀、防振锤扭转、防振锤损坏、防振锤脱落，其中后四类为缺陷类别。该模块进行防振锤分类识别的流程如图4d所示，包括从原图裁剪防振锤区域441、图像预处理442、ResNet18多分类443、缺陷类别概率整合444四个子流程。对概率超过第一特定阈值的一阶段部件检出的防振锤区域，首先从原图裁剪防振锤区域；然后，对每一个裁剪出的防振锤区域子图进行图像预处理，保持尺度一致；之后将每一个子图像输入ResNet18分类网络进行五分类，使用softmax计算得到五种防振锤形态类别的概率值；最后进行缺陷类别概率整合，即将四类属于防振锤缺陷的概率值求和，作为防振锤有缺陷的概率。
5)二级绝缘子自爆检测模块：该模块以一级部件检测模块检测的绝缘子串区域作为输入，主要通过Faster R-CNN检测网络对每个绝缘子串区域子图进行自爆区域检测，得到可能包含自爆的绝缘子。该模块的主要网络结构如图4e所示，使用ResNet18头部网络451作为检测网络的骨干网络，可以减少计算量，首先裁剪出概率超过第二特定阈值的绝缘子串区域，对得到的每一子图进行图像预处理，保持尺度一致，得到用于检测的RGB图像，将每一个RGB图像经过ResNet18头部网络451对绝缘子串的自爆区域进行检测，得到C4层特征图452；C4层特征图452经过RPN提取候选框453，提取的候选框为可能存在绝缘子串自爆的区域，候选框与C4层特征图一起经过RoI池化层454得到每个候选区域的特征图，再经过至少一个全连接层455得到每个候选区域的特征向量；最后对每个候选区域的特征向量进行框坐标回归456和多类别分类457，得到最终的检测结果，输出绝缘子串的自爆区域和相应的概率值。
6)自爆检测后处理模块：该模块将绝缘子串区域子图中的自爆检测结果映射回原图结果，并对多次重复检测的自爆区域进行重叠抑制处理，得到原图中绝缘子自爆区域的位置信息。该模块的处理过程可以分为以下三步：a)绝缘子串子图像中的自爆区域坐标映射回原图区域；b)对重叠的自爆区域进行抑制；c)过滤得分较低的自爆区域框。
7)异常缺陷筛选模块：该模块对鸟巢区域、异常防振锤、绝缘子自爆的检测识别结果进行筛选得到最终的异常缺陷检测结果。该模块的处理过程可以分为以下三步：a)对鸟巢区域和自爆区域按照尺寸进行过滤，剔除掉过大和过小的检测框；b)对鸟巢、绝缘子自爆和异常防振锤按照各自类别的阈值，剔除掉低于对应阈值的结果；c)对所有保留下来的检测结果按照得分从高到低整体排序作为输出结果。
本申请实施例中，使用无人机航拍高分辨率图作为输入，引入轻量化的深度学习检测和分类模型，对绝缘子自爆、防振锤、附着鸟巢等几种缺陷异常进行联合识别，具有以下有益效果：1)相关技术中的电网缺陷检测算法中，主要利用图像处理和机器学习的方法，对目标需要先验构造手工特征，鲁棒性差，准确度低，且受拍摄环境影响很大，本申请实施例中基于深度学习模型，通过网络自主学习特征，应用场景更广，鲁棒性更强，准确度更高；2)本申请实施例中，结合电路巡检场景提出两阶段检测-检测、检测-分类算法适合无人机航拍的高清图像，且充分考虑绝缘子自爆依托在检测到的绝缘子串上的缺陷特点，可以解决相关技术中单一网络检测或分类器对高清图像检测识别效果差的问题；3)相关技术中一套算法只能检测识别一种部件的缺陷，要进行多种缺陷检 测需要不断重复输入原图到不同算法中，效率低。本申请实施例中将几种缺陷整合到一套算法中进行检测识别，保证准确性的前提下，大幅提高检测效率。
请参阅图5，图5是本申请实施例提供的一种缺陷检测装置的结构示意图。缺陷检测装置30包括第一目标检测模块31、图像区域获取模块32、缺陷检测模块33，第一目标检测模块31，配置为对待检测图像进行第一目标检测，得到目标检测结果，其中，目标检测结果包括第一类目标以及第一类目标在待检测图像的第一位置信息；区域获取模块，配置为基于第一位置信息，获取包含第一类目标的图像区域；缺陷检测模块33，配置为对图像区域进行缺陷检测，得到关于第一类目标的缺陷检测结果。
上述方案，通过先检测第一类目标在待检测图像中的位置，然后再对包含第一类目标的图像区域进行缺陷检测，由于直接针对目标所在的图像区域进行缺陷检测，即减小缺陷检测的区域，可降低漏检率和误检率。
其中，缺陷检测装置30还包括预处理模块(图未示)。
在本申请的一些实施例中，第一目标检测模块31对待检测图像进行第一目标检测，得到目标检测结果之前，预处理模块配置为将初始图像进行缩放到预设大小的图像；将经缩放后的初始图像的像素值压缩至预设像素值范围；对经压缩的初始图像进行归一化处理，得到待检测图像。
上述方案，通过先对获取到的图像进行上述预处理，统一图像的样式，能够提高输入的鲁棒性。
在本申请的一些实施例中，第一目标检测模块31对待检测图像进行第一目标检测，得到检测结果，包括：利用第一区域检测网络对待检测图像进行第一目标检测，得到检测结果；对图像区域进行第二目标检测，得到第一缺陷检测结果，包括：利用第二区域检测网络对图像区域进行第二目标检测，得到第一缺陷检测结果；其中，第二区域检测网络的深度比第一区域检测网络的深度浅；对图像区域进行分类，得到第二缺陷检测结果，包括：利用分类网络对图像区域进行分类，得到第二缺陷检测结果。
上述方案，通过考虑各个不同部分的不同缺陷的特点，利用不同的网络模型分段进行检测，能够在一定程度上解决单一检测网络出现漏检率高的问题，同时，第二目标检测使用的网络模型深度比第一目标检测使用的网络模型深度浅，能够减少第二目标检测过程中的计算量，提高检测效率。
在本申请的一些实施例中，图像区域获取模块32基于第一位置信息，获取包含第一类目标的图像区域，包括：基于第一位置信息，从待检测图像中确定第一类目标对应的目标区域；将目标区域在待检测图像中向外扩预设倍数；在待检测图像中提取经外扩后的目标区域，以得到包含第一类目标的图像区域。
上述方案，通过将目标区域在待检测图像中向外扩预设倍数，使得能够在获取包含第一类目标对应的目标区域之后使得能够保留一些背景信息，能够提升对第一类目标缺陷检测的准确度。
在本申请的一些实施例中，缺陷检测模块33还配置为对图像区域进行缺陷检测，得到关于第一类目标的缺陷检测结果，包括：对图像区域进行第二目标检测，得到第一缺陷检测结果，其中，第一缺陷检测结果包括第一类目标上存在的至少一种第一缺陷的 信息；和/或，对图像区域进行分类，得到第二缺陷检测结果，其中，第二缺陷检测结果包括第一类目标所属第二缺陷的信息。
上述方案，通过多种方式对包括第一类目标的图像区域进行检测，可针对不同的第一类目标采用不同的检测方式，使得缺陷检测更具有针对性，从而得到的缺陷检测结果更准确或者为同一个第一类目标采用不同的检测方式然后可以综合各个不同的缺陷检测方式得出最终待检测图像的缺陷检测结果更准确，进一步减低误检率。
在本申请的一些实施例中，缺陷检测模块33还配置为对图像区域进行缺陷检测，得到关于第一类目标的缺陷检测结果，包括：在第一类目标为第一子类目标的情况下，执行对图像区域进行第二目标检测，得到第一缺陷检测结果的步骤；在第一类目标为第二子类目标的情况下，执行对图像区域进行分类，得到第二缺陷检测结果的步骤。
上述方案，通过对不同的目标对象采用不同的缺陷检测方式，使得缺陷检测更加灵活，相比对所有种类的目标都采用同一种检测方式来讲，更有针对性，使得缺陷检测结果更准确。
在本申请的一些实施例中，第一缺陷检测结果包括每种第一缺陷的位置信息以及属于第一缺陷的第一概率；缺陷检测模块33还配置为对图像区域进行第二目标检测，得到第一缺陷检测结果之后，还包括以下至少一者：依据每种第一缺陷的位置信息，分别确定每种第一缺陷在待检测图像上的缺陷区域，并对不同缺陷区域之间的重叠区域进行去重处理；过滤第一概率满足第一过滤条件的第一缺陷。
上述方案，通过获取第一子类目标中第一缺陷的概率以及在包含第一子类目标的图像区域中位置信息，并且基于该位置信息找到第一缺陷在待检测图像中上对应的区域，并对重叠的部分进行去重处理，使得一个第一缺陷对应一个缺陷区域，减少出现对一个缺陷进行多次后续处理的问题，提高最后输出的准确率，而通过将满足过滤条件的区域剔除，能够减少后续处理的区域数量，提高处理效率。
在本申请的一些实施例中，第一过滤条件为第一概率低于第一概率阈值；和/或，对不同缺陷区域之间的重叠区域进行去重处理，包括：对不同缺陷区域之间的重叠区域进行非极大值抑制。
上述方案，通过将缺陷概率低的剔除，减少了很多不必要的处理区域，提高了处理效率，通过非极大值抑制的方式进行去重，能够保留缺陷概率更大的缺陷区域，在提高处理效率的同时还能保障缺陷检测的准确率。
在本申请的一些实施例中，第二缺陷检测结果包括第一类目标属于第二缺陷的第二概率；缺陷检测模块33对图像区域进行分类，得到第二缺陷检测结果，包括：对图像区域进行分类，得到第一类目标属于每种预设缺陷的概率；将每种预设缺陷的概率相加，得到第一类目标属于第二缺陷的第二概率。
上述方案，综合多种预设缺陷的概率，确定该第一类目标的缺陷概率，广泛考虑第一类目标属于各种预设缺陷的可能性，使得第一类目标的缺陷概率更精确。
在本申请的一些实施例中，第一目标检测模块31对待检测图像进行第一目标检测，得到目标检测结果，包括：利用缺陷检测模型的第一区域检测网络对待检测图像进行第一目标检测，得到目标检测结果；缺陷检测模块33对图像区域进行第二目标检测，得 到第一缺陷检测结果，包括：利用缺陷检测模型的第二区域检测网络对图像区域进行第二目标检测，得到第一缺陷检测结果；其中，第二区域检测网络的深度比第一区域检测网络的深度浅；对图像区域进行分类，得到第二缺陷检测结果，包括：利用缺陷检测模型的分类网络对图像区域进行分类，得到第二缺陷检测结果。
上述方案，通过考虑各个不同部件可能存在不同缺陷的特点，利用不同的网络模型分段进行检测，能够在一定程度上减轻了单一检测网络出现漏检率高的问题，同时，第二目标检测使用的网络模型深度比第一目标检测使用的网络模型深度浅，能够减少第二目标检测过程中的计算量，提高检测效率。
在本申请的一些实施例中，目标检测结果还包括第二类目标以及第二类目标在待检测图像的第二位置信息，第一目标检测模块31在对待检测图像进行第一目标检测，得到目标检测结果之后，还配置为：将第二类目标确定为待检测图像中的第三缺陷，并将第二类目标及其第二位置信息作为关于第二类目标的缺陷检测结果。
上述方案，通过将不应该存在的第二类目标设置为第三缺陷，使得一个缺陷检测模型能够针对多种目标进行检测，加强了缺陷检测模型的适应性。
在本申请的一些实施例中，缺陷检测模块33在得到缺陷检测结果之后，还配置为：过滤缺陷检测结果中满足第二过滤条件的缺陷，以得到待检测图像的最终缺陷检测结果。
上述方案，通过过滤不满足条件的缺陷，减少了明显不合理的缺陷，使得最后得到的缺陷检测结果更精确。
在本申请的一些实施例中，缺陷检测模块33过滤缺陷检测结果中满足第二过滤条件的缺陷，包括：利用缺陷检测结果中缺陷的位置信息得到缺陷的尺寸，过滤缺陷检测结果中尺寸不满足预设尺寸条件的缺陷；和/或，过滤缺陷检测结果中概率低于第二概率阈值的缺陷。
上述方案，通过过滤掉尺寸或概率不满足条件的，减少了明显不合理的缺陷，使得最后得到的缺陷检测结果更准确。
在本申请的一些实施例中，缺陷检测模块33在得到待检测图像的最终缺陷检测结果之后，还配置为：按照最终缺陷检测结果中每个缺陷的概率从高到低的顺序，输出最终缺陷检测结果中每个缺陷的信息。
上述方案，通过按照每个缺陷的概率从高到低的顺序输出最终缺陷检测结果中每个缺陷的信息，使得输出结果更整洁，便于后续进一步的观察。
其中，缺陷检测装置包括训练模块(图未示)，训练模块配置为训练缺陷检测模型。
在本申请的一些实施例中，第一目标检测和缺陷检测是由缺陷检测模型执行的；其中，缺陷检测模型由训练模块至少执行以下步骤训练得到：获取第一样本集，其中，第一样本集包括至少一个样本图像，样本图像标注有关于目标的真实缺陷信息，目标包括第一类目标；利用缺陷检测模型对第一样本集中的样本图像进行检测，得到关于目标的缺陷检测结果；利用真实缺陷信息和缺陷检测结果，调整缺陷检测模型的参数。
上述方案，通过按照上述方式训练得到的缺陷检测模型的准确度更高，从而执行第一目标检测以及缺陷检测使得检测结果更准备。
在本申请的一些实施例中，缺陷检测模型是由多个图形处理器共同训练得到的，每个图形处理器获取的第一样本集不同；训练模块利用缺陷检测模型对第一样本集中的样本图像进行检测，得到关于目标的缺陷检测结果，包括：对每个图形处理器中的缺陷检测模型的批标准化层进行同步，并利用每个图形处理器中批标准化层同步后的缺陷检测模型对样本图像进行检测，得到关于目标的缺陷检测结果；利用真实缺陷信息和缺陷检测结果，调整缺陷检测模型的参数，包括：基于所有缺陷检测结果与真实缺陷信息，确定缺陷检测模型的平均损失值；利用平均损失值，调整缺陷检测模型的参数。
上述方案，通过同步多个图形处理器上中缺陷检测模型的批标准化层，使得可以使用全局的第一样本集进行归一化，相当于增大了批量大小，从而减少因为使用多个图形处理器的影响，并且通过多个图形处理器对缺陷检测模型进行训练加快了训练的速度。
在本申请的一些实施例中，训练模块在所利用缺陷检测模型对第一样本集中的样本图像进行检测，得到关于目标的缺陷检测结果之前，还包括以下至少一个步骤：利用第二样本集对缺陷检测模型进行预训练；利用第一样本集对缺陷检测模型进行预热训练，其中，预热训练的过程中采用的学习率为：从低于预设学习率的第一学习率开始，并逐步增大至预设学习率；其中，第一样本集中的样本图像至少由以下步骤得到：对原始图像进行预处理，得到样本图像，其中，预处理的方式包括尺度变换、颜色变换、水平翻转、竖直翻转、旋转、裁剪中的一种或多种。
上述方案，通过在正式对缺陷检测模型训练之前，先利用第二样本集对缺陷检测模型进行预训练使得对缺陷检测模型的参数进行了初始化，或在正式训练之前对学习率进行初始化，以及通过对原始图像进行预处理，提高了缺陷检测模型的适应性。
请参阅图5，图5是本申请实施例提供的一种电子设备的结构示意图。电子设备40包括存储器41和处理器42，处理器42配置为执行存储器41中存储的程序指令，以实现上述缺陷检测方法实施例的步骤，在一个实施场景中，电子设备40可以包括但不限于：微型计算机、服务器，此外，电子设备40还可以包括笔记本电脑、平板电脑等移动设备，在此不做限定。
具体而言，处理器42配置为控制其自身以及存储器41以实现上述任一缺陷检测方法实施例中的步骤。处理器42还可以称为CPU(Central Processing Unit，中央处理单元)。处理器42可能是一种集成电路芯片，具有信号的处理能力。处理器42还可以是通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。另外，处理器42可以由集成电路芯片共同实现。
上述方案，通过先检测第一类目标在待检测图像中的位置，然后再对包含第一类目标的图像区域进行缺陷检测，由于直接针对目标所在的图像区域进行缺陷检测，即减小缺陷检测的区域，可降低漏检率和误检率。
请参阅图6，图6为本申请实施例提供的一种计算机可读存储介质的结构示意图。计算机可读存储介质50存储有能够被处理器运行的程序指令501，程序指令501用于实 现上述缺陷检测方法实施例的步骤。
本公开实施例还提供一种计算机程序产品，该计算机程序产品被处理器执行时实现前述实施例的任意一种方法。该计算机程序产品可以通过硬件、软件或其结合的方式实现。在本公开的一些实施例中，所述计算机程序产品体现为计算机存储介质，在本公开的另一些实施例中，计算机程序产品体现为软件产品，例如软件开发包(Software Development Kit，SDK)等等。
上述方案，通过先检测第一类目标在待检测图像中的位置，然后再对包含第一类目标的图像区域进行缺陷检测，由于直接针对目标所在的图像区域进行缺陷检测，即减小缺陷检测的区域，可降低漏检率和误检率。
在一些实施例中，本申请实施例提供的装置具有的功能或包含的模块可以配置为执行上文方法实施例描述的方法，其具体实现可以参照上文方法实施例的描述，为了简洁，这里不再赘述。
上文对各个实施例的描述倾向于强调各个实施例之间的不同之处，其相同或相似之处可以互相参考，为了简洁，本文不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的方法和装置，可以通过其它的方式实现。例如，以上所描述的装置实施方式仅仅是示意性的，例如，模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性、机械或其它的形式。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)或处理器(processor)执行本申请各个实施方式方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
工业实用性
本申请实施例提供了一种缺陷检测方法和相关装置、设备、存储介质，其中方法包括：对待检测图像进行第一目标检测，得到目标检测结果，其中，目标检测结果包括第一类目标以及第一类目标在待检测图像的第一位置信息；基于第一位置信息，获取包含第一类目标的图像区域；对图像区域进行缺陷检测，得到关于第一类目标的缺陷检测结果。上述方案，根据本申请实施例提供的缺陷检测方法对待检测图像进行缺陷检测，能够降低缺陷漏检率和误检率。

Claims (35)

  1. 一种缺陷检测方法，包括：
    对待检测图像进行第一目标检测，得到目标检测结果，其中，所述目标检测结果包括第一类目标以及所述第一类目标在所述待检测图像的第一位置信息；
    基于所述第一位置信息，获取包含所述第一类目标的图像区域；
    对所述图像区域进行缺陷检测，得到关于所述第一类目标的缺陷检测结果。
  2. 根据权利要求1所述的方法，其中，所述对所述图像区域进行缺陷检测，得到关于所述第一类目标的缺陷检测结果，包括：
    对所述图像区域进行第二目标检测，得到第一缺陷检测结果，其中，所述第一缺陷检测结果包括所述第一类目标上存在的至少一种第一缺陷的信息；
    和/或，对所述图像区域进行分类，得到第二缺陷检测结果，其中，所述第二缺陷检测结果包括所述第一类目标所属第二缺陷的信息。
  3. 根据权利要求2所述的方法，其中，所述对所述图像区域进行缺陷检测，得到关于所述第一类目标的缺陷检测结果，包括：
    在所述第一类目标为第一子类目标的情况下，执行所述对所述图像区域进行第二目标检测，得到第一缺陷检测结果的步骤；
    在所述第一类目标为第二子类目标的情况下，执行所述对所述图像区域进行分类，得到第二缺陷检测结果的步骤。
  4. 根据权利要求2或3所述的方法，其中，所述第一缺陷检测结果包括每种所述第一缺陷的位置信息以及属于所述第一缺陷的第一概率；在所述对所述图像区域进行第二目标检测，得到第一缺陷检测结果之后，所述方法还包括以下至少一者：
    依据每种所述第一缺陷的位置信息，分别确定每种所述第一缺陷在所述待检测图像上的缺陷区域，并对不同所述缺陷区域之间的重叠区域进行去重处理；
    过滤第一概率满足第一过滤条件的所述第一缺陷。
  5. 根据权利要求4所述的方法，其中，所述第一过滤条件为所述第一概率低于第一概率阈值；和/或，
    所述对不同所述缺陷区域之间的重叠区域进行去重处理，包括：
    对不同所述缺陷区域之间的重叠区域进行非极大值抑制。
  6. 根据权利要求2至5任一项所述的方法，其中，所述第二缺陷检测结果包括所述第一类目标属于第二缺陷的第二概率；所述对所述图像区域进行分类，得到第二缺陷检测结果，包括：
    对所述图像区域进行分类，得到所述第一类目标属于每种预设缺陷的概率；
    将每种预设缺陷的概率相加，得到所述第一类目标属于第二缺陷的第二概率。
  7. 根据权利要求2至6任一项所述的方法，其中，所述对待检测图像进行第一目标检测，得到目标检测结果，包括：
    利用缺陷检测模型的第一区域检测网络对待检测图像进行第一目标检测，得到目标检测结果；
    所述对所述图像区域进行第二目标检测，得到第一缺陷检测结果，包括：
    利用所述缺陷检测模型的第二区域检测网络对所述图像区域进行第二目标检测，得到第一缺陷检测结果；其中，所述第二区域检测网络的深度比第一区域检测网络的深度浅；
    所述对所述图像区域进行分类，得到第二缺陷检测结果，包括：
    利用所述缺陷检测模型的分类网络对所述图像区域进行分类，得到第二缺陷检测结果。
  8. 根据权利要求1-7任一项所述的方法，其中，所述目标检测结果还包括第二类目标以及第二类目标在所述待检测图像的第二位置信息，在所述对待检测图像进行第一目标检测，得到目标检测结果之后，所述方法还包括：
    将所述第二类目标确定为所述待检测图像中的第三缺陷，并将所述第二类目标及其所述第二位置信息作为关于所述第二类目标的缺陷检测结果。
  9. 根据权利要求1-8任一项所述的方法，其中，在得到所述缺陷检测结果之后，所述方法还包括：
    过滤所述缺陷检测结果中满足第二过滤条件的缺陷，以得到所述待检测图像的最终缺陷检测结果。
  10. 根据权利要求9所述的方法，其中，所述过滤所述缺陷检测结果中满足第二过滤条件的缺陷，包括：
    利用所述缺陷检测结果中所述缺陷的位置信息得到所述缺陷的尺寸，过滤所述缺陷检测结果中所述尺寸不满足预设尺寸条件的缺陷；和/或，
    过滤所述缺陷检测结果中概率低于第二概率阈值的所述缺陷。
  11. 根据权利要求9或10所述的方法，其中，在所述得到所述待检测图像的最终缺陷检测结果之后，所述方法还包括：
    按照所述最终缺陷检测结果中每个所述缺陷的概率从高到低的顺序，输出所述最终缺陷检测结果中每个所述缺陷的信息。
  12. 根据权利要求1-11任一项所述的方法，其中，所述基于所述第一位置信息，获取包含所述第一类目标的图像区域，包括：
    基于所述第一位置信息，从所述待检测图像中确定所述第一类目标对应的目标区域；
    将所述目标区域在所述待检测图像中向外扩预设倍数；
    在所述待检测图像中提取经外扩后的所述目标区域，以得到包含所述第一类目标的图像区域。
  13. 根据权利要求1-12任一项所述的方法，其中，所述对待检测图像进行第一目标检测，得到目标检测结果之前，所述方法还包括：
    将初始图像进行缩放，得到预设大小的图像；
    将经缩放后的所述初始图像的像素值压缩至预设像素值范围；
    对经压缩的所述初始图像进行归一化处理，得到所述待检测图像。
  14. 根据权利要求1至13任一项所述的方法，其中，所述第一目标检测和缺陷检测是由缺陷检测模型执行的；其中，所述缺陷检测模型至少由以下步骤训练得到：
    获取第一样本集，其中，所述第一样本集包括至少一个样本图像，所述样本图像标注有关于目标的真实缺陷信息，所述目标包括第一类目标；
    利用所述缺陷检测模型对所述第一样本集中的样本图像进行检测，得到关于所述目标的缺陷检测结果；
    利用所述真实缺陷信息和所述缺陷检测结果，调整所述缺陷检测模型的参数。
  15. 根据权利要求14所述的方法，其中，所述缺陷检测模型是由多个图形处理器共同训练得到的，每个所述图形处理器获取的第一样本集不同；
    所述利用所述缺陷检测模型对所述第一样本集中的样本图像进行检测，得到关于所述目标的缺陷检测结果，包括：
    对每个所述图形处理器中的所述缺陷检测模型的批标准化层进行同步，并利用每个所述图形处理器中所述批标准化层同步后的缺陷检测模型对所述样本图像进行检测，得到关于所述目标的缺陷检测结果；
    所述利用所述真实缺陷信息和所述缺陷检测结果，调整所述缺陷检测模型的参数，包括：
    基于所有所述缺陷检测结果与所述真实缺陷信息，确定所述缺陷检测模型的平均损失值；
    利用所述平均损失值，调整所述缺陷检测模型的参数。
  16. 根据权利要求14或15所述的方法，其中，在所述利用所述缺陷检测模型对所述第一样本集中的样本图像进行检测，得到关于所述目标的缺陷检测结果之前，所述方法还包括以下至少一个步骤：
    利用第二样本集对所述缺陷检测模型进行预训练；
    利用第一样本集对所述缺陷检测模型进行预热训练，其中，所述预热训练的过程中采用的学习率为：从低于预设学习率的第一学习率开始，并逐步增大至所述预设学习率；
    其中，所述第一样本集中的样本图像至少由以下步骤得到：
    对原始图像进行预处理，得到所述样本图像，其中，所述预处理的方式包括尺度变换、颜色变换、水平翻转、竖直翻转、旋转、裁剪中的一种或多种。
  17. 一种缺陷检测装置，包括：
    第一目标检测模块，配置为对待检测图像进行第一目标检测，得到目标检测结果，其中，所述目标检测结果包括第一类目标以及所述第一类目标在所述待检测图像的第一位置信息；
    图像区域获取模块，配置为基于所述第一位置信息，获取包含所述第一类目标的图像区域；
    缺陷检测模块，配置为对所述图像区域进行缺陷检测，得到关于所述第一类目标的缺陷检测结果。
  18. 根据权利要求17所述的装置，其中，所述缺陷检测模块还配置为：对所述图像区域进行第二目标检测，得到第一缺陷检测结果，其中，所述第一缺陷检测结果包括所述第一类目标上存在的至少一种第一缺陷的信息；和/或，对所述图像区域进行分类，得到第二缺陷检测结果，其中，所述第二缺陷检测结果包括所述第一类目标所属第二缺 陷的信息。
  19. 根据权利要求18所述的装置，其中，所述缺陷检测模块还配置为：在所述第一类目标为第一子类目标的情况下，执行所述对所述图像区域进行第二目标检测，得到第一缺陷检测结果的步骤；在所述第一类目标为第二子类目标的情况下，执行所述对所述图像区域进行分类，得到第二缺陷检测结果的步骤。
  20. 根据权利要求18或19所述的装置，其中，所述第一缺陷检测结果包括每种所述第一缺陷的位置信息以及属于所述第一缺陷的第一概率；所述缺陷检测模块还配置为：在所述对所述图像区域进行第二目标检测，得到第一缺陷检测结果之后，还包括以下至少一者：依据每种所述第一缺陷的位置信息，分别确定每种所述第一缺陷在所述待检测图像上的缺陷区域，并对不同所述缺陷区域之间的重叠区域进行去重处理；过滤第一概率满足第一过滤条件的所述第一缺陷。
  21. 根据权利要求20所述的装置，其中，所述第一过滤条件为所述第一概率低于第一概率阈值；和/或，所述缺陷检测模块还配置为：对不同所述缺陷区域之间的重叠区域进行非极大值抑制。
  22. 根据权利要求18至21任一项所述的装置，其中，所述第二缺陷检测结果包括所述第一类目标属于第二缺陷的第二概率；所述缺陷检测模块还配置为：对所述图像区域进行分类，得到所述第一类目标属于每种预设缺陷的概率；将每种预设缺陷的概率相加，得到所述第一类目标属于第二缺陷的第二概率。
  23. 根据权利要求18至22任一项所述的装置，其中，所述第一目标检测模块还配置为：利用缺陷检测模型的第一区域检测网络对待检测图像进行第一目标检测，得到目标检测结果；
    所述缺陷检测模块还配置为：利用所述缺陷检测模型的第二区域检测网络对所述图像区域进行第二目标检测，得到第一缺陷检测结果；其中，所述第二区域检测网络的深度比第一区域检测网络的深度浅；利用所述缺陷检测模型的分类网络对所述图像区域进行分类，得到第二缺陷检测结果。
  24. 根据权利要求17-23任一项所述的装置，其中，所述目标检测结果还包括第二类目标以及第二类目标在所述待检测图像的第二位置信息，所述缺陷检测模块还配置为：在所述对待检测图像进行第一目标检测，得到目标检测结果之后，将所述第二类目标确定为所述待检测图像中的第三缺陷，并将所述第二类目标及其所述第二位置信息作为关于所述第二类目标的缺陷检测结果。
  25. 根据权利要求17-24任一项所述的装置，其中，所述缺陷检测模块还配置为：在得到所述缺陷检测结果之后，过滤所述缺陷检测结果中满足第二过滤条件的缺陷，以得到所述待检测图像的最终缺陷检测结果。
  26. 根据权利要求25所述的装置，其中，所述缺陷检测模块还配置为：利用所述缺陷检测结果中所述缺陷的位置信息得到所述缺陷的尺寸，过滤所述缺陷检测结果中所述尺寸不满足预设尺寸条件的缺陷；和/或，过滤所述缺陷检测结果中概率低于第二概率阈值的所述缺陷。
  27. 根据权利要求25或26所述的装置，其中，所述缺陷检测模块还配置为：在所述得到所述待检测图像的最终缺陷检测结果之后，按照所述最终缺陷检测结果中每个 所述缺陷的概率从高到低的顺序，输出所述最终缺陷检测结果中每个所述缺陷的信息。
  28. 根据权利要求17-27任一项所述的装置，其中，所述图像区域获取模块还配置为：基于所述第一位置信息，从所述待检测图像中确定所述第一类目标对应的目标区域；将所述目标区域在所述待检测图像中向外扩预设倍数；在所述待检测图像中提取经外扩后的所述目标区域，以得到包含所述第一类目标的图像区域。
  29. 根据权利要求17-28任一项所述的装置，其中，所述装置还包括：预处理模块，配置为：在所述对待检测图像进行第一目标检测，得到目标检测结果之前，将初始图像进行缩放，得到预设大小的图像；将经缩放后的所述初始图像的像素值压缩至预设像素值范围；对经压缩的所述初始图像进行归一化处理，得到所述待检测图像。
  30. 根据权利要求17至29任一项所述的装置，其中，所述装置还包括：训练模块，配置为：获取第一样本集，其中，所述第一样本集包括至少一个样本图像，所述样本图像标注有关于目标的真实缺陷信息，所述目标包括第一类目标；利用所述缺陷检测模型对所述第一样本集中的样本图像进行检测，得到关于所述目标的缺陷检测结果；利用所述真实缺陷信息和所述缺陷检测结果，调整所述缺陷检测模型的参数。
  31. 根据权利要求30所述的装置，其中，所述缺陷检测模型是由多个图形处理器共同训练得到的，每个所述图形处理器获取的第一样本集不同；
    所述训练模块还配置为：对每个所述图形处理器中的所述缺陷检测模型的批标准化层进行同步，并利用每个所述图形处理器中所述批标准化层同步后的缺陷检测模型对所述样本图像进行检测，得到关于所述目标的缺陷检测结果；基于所有所述缺陷检测结果与所述真实缺陷信息，确定所述缺陷检测模型的平均损失值；利用所述平均损失值，调整所述缺陷检测模型的参数。
  32. 根据权利要求30或31所述的装置，其中，所述训练模块还配置为：在所述利用所述缺陷检测模型对所述第一样本集中的样本图像进行检测，得到关于所述目标的缺陷检测结果之前，执行以下至少一个步骤：利用第二样本集对所述缺陷检测模型进行预训练；利用第一样本集对所述缺陷检测模型进行预热训练，其中，所述预热训练的过程中采用的学习率为：从低于预设学习率的第一学习率开始，并逐步增大至所述预设学习率；
    其中，所述第一样本集中的样本图像至少由以下步骤得到：对原始图像进行预处理，得到所述样本图像，其中，所述预处理的方式包括尺度变换、颜色变换、水平翻转、竖直翻转、旋转、裁剪中的一种或多种。
  33. 一种电子设备，其特征在于，包括存储器和处理器，所述处理器配置为执行所述存储器中存储的程序指令，以实现权利要求1至16任一项所述的方法。
  34. 一种计算机可读存储介质，其上存储有程序指令，其特征在于，所述程序指令被处理器执行时实现权利要求1至16任一项所述的方法。
  35. 一种计算机程序产品，所述计算机程序产品包括存储了计算机程序的非瞬时性计算机可读存储介质，所述计算机程序被计算机读取并执行时，实现如权利要求1至16任一项所述的方法。
PCT/CN2020/136251 2020-08-19 2020-12-14 缺陷检测方法和相关装置、设备、存储介质、计算机程序产品 WO2022036953A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
KR1020217037051A KR20220023335A (ko) 2020-08-19 2020-12-14 결함 검출 방법 및 관련 장치, 기기, 저장 매체, 컴퓨터 프로그램 제품
JP2021567817A JP2022548438A (ja) 2020-08-19 2020-12-14 欠陥検出方法及び関連装置、機器、記憶媒体、並びにコンピュータプログラム製品

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010837709.2 2020-08-19
CN202010837709.2A CN112001902A (zh) 2020-08-19 2020-08-19 缺陷检测方法和相关装置、设备、存储介质

Publications (1)

Publication Number Publication Date
WO2022036953A1 true WO2022036953A1 (zh) 2022-02-24

Family

ID=73473665

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/136251 WO2022036953A1 (zh) 2020-08-19 2020-12-14 缺陷检测方法和相关装置、设备、存储介质、计算机程序产品

Country Status (5)

Country Link
JP (1) JP2022548438A (zh)
KR (1) KR20220023335A (zh)
CN (1) CN112001902A (zh)
TW (1) TWI770757B (zh)
WO (1) WO2022036953A1 (zh)

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114359281A (zh) * 2022-03-17 2022-04-15 南方电网数字电网研究院有限公司 基于层次化集成学习的电力部件识别方法和装置
CN114820561A (zh) * 2022-05-16 2022-07-29 奥蒂玛光学科技（深圳）有限公司 一种异常区域检测方法、自动光学检测设备以及存储介质
CN115131307A (zh) * 2022-06-23 2022-09-30 腾讯科技（深圳）有限公司 物品缺陷检测方法及相关装置
CN115272330A (zh) * 2022-09-28 2022-11-01 深圳先进技术研究院 基于电池表面图像的缺陷检测方法、系统及相关设备
CN115272249A (zh) * 2022-08-01 2022-11-01 腾讯科技（深圳）有限公司 一种缺陷检测方法、装置、计算机设备和存储介质
CN115526892A (zh) * 2022-11-29 2022-12-27 南方电网数字电网研究院有限公司 基于三维重建的图像缺陷去重检测方法和装置
CN115564773A (zh) * 2022-11-18 2023-01-03 南方电网数字电网研究院有限公司 基于元学习的小样本图像缺陷检测方法、装置和设备
CN115631193A (zh) * 2022-12-06 2023-01-20 北京矩视智能科技有限公司 基于注意力机制的工件缺陷检测方法、装置及存储介质
CN116385952A (zh) * 2023-06-01 2023-07-04 华雁智能科技(集团)股份有限公司 配网线路小目标缺陷检测方法、装置、设备及存储介质
CN116449875A (zh) * 2023-06-16 2023-07-18 拓恒技术有限公司 一种无人机巡检方法及系统
CN116539619A (zh) * 2023-04-19 2023-08-04 广州里工实业有限公司 产品缺陷检测方法、系统、装置及存储介质
US11763549B1 (en) * 2022-07-29 2023-09-19 Contemporary Amperex Technology Co., Limited Method and apparatus for training cell defect detection model
CN117078689A (zh) * 2023-10-17 2023-11-17 沈阳宏远电磁线股份有限公司 一种基于机器视觉的线缆缺陷识别方法及系统
CN117253179A (zh) * 2023-11-20 2023-12-19 合肥中科类脑智能技术有限公司 配电线路鸟巢检测方法、存储介质和电子设备
WO2023241352A1 (zh) * 2022-06-14 2023-12-21 中兴通讯股份有限公司 基于交互的编码方法、编码装置及可读存储介质
CN117437188A (zh) * 2023-10-17 2024-01-23 广东电力交易中心有限责任公司 一种用于智慧电网的绝缘子缺陷检测系统
CN117576042A (zh) * 2023-11-22 2024-02-20 魅杰光电科技(上海)有限公司 晶圆缺陷检测方法、系统、电子设备以及存储介质
CN117934462A (zh) * 2024-03-21 2024-04-26 高视科技(苏州)股份有限公司 用于调整检测参数的方法、电子设备及存储介质

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112001902A (zh) * 2020-08-19 2020-11-27 上海商汤智能科技有限公司 缺陷检测方法和相关装置、设备、存储介质
CN112634254A (zh) * 2020-12-29 2021-04-09 北京市商汤科技开发有限公司 绝缘子缺陷检测方法及相关装置
CN112926438B (zh) * 2021-02-22 2024-04-05 深圳中科飞测科技股份有限公司 检测方法及装置、检测设备和存储介质
CN112926437B (zh) * 2021-02-22 2024-06-11 深圳中科飞测科技股份有限公司 检测方法及装置、检测设备和存储介质
CN112926439A (zh) * 2021-02-22 2021-06-08 深圳中科飞测科技股份有限公司 检测方法及装置、检测设备和存储介质
CN113139540B (zh) * 2021-04-02 2022-11-01 北京邮电大学 背板检测方法及设备
CN113205512B (zh) * 2021-05-26 2023-10-24 北京市商汤科技开发有限公司 图像异常检测方法、装置、设备及计算机可读存储介质
CN113222989A (zh) * 2021-06-09 2021-08-06 联仁健康医疗大数据科技股份有限公司 一种图像分级方法、装置、存储介质及电子设备
CN114298992A (zh) * 2021-12-21 2022-04-08 北京百度网讯科技有限公司 一种视频帧去重方法、装置、电子设备及存储介质
CN116087345B (zh) * 2023-04-06 2023-06-13 广东电网有限责任公司揭阳供电局 复合绝缘子的轴向缺陷长度的计算方法、装置及介质
CN117495846B (zh) * 2023-12-27 2024-04-16 苏州镁伽科技有限公司 图像检测方法、装置、电子设备及存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20180115368A (ko) * 2017-04-12 2018-10-23 주식회사 포스코 선재의 표면 흠 자동 맵핑 장치 및 방법
CN109785288A (zh) * 2018-12-17 2019-05-21 广东电网有限责任公司 基于深度学习的输电设备缺陷检测方法及系统
CN111161246A (zh) * 2019-12-30 2020-05-15 歌尔股份有限公司 一种产品缺陷检测方法、装置与系统
CN111242943A (zh) * 2020-01-22 2020-06-05 腾讯科技（深圳）有限公司 图像处理方法、装置、图像处理设备及存储介质
CN111462109A (zh) * 2020-04-17 2020-07-28 广东电网有限责任公司 一种耐张线夹的缺陷检测方法、装置、设备及存储介质
CN112001902A (zh) * 2020-08-19 2020-11-27 上海商汤智能科技有限公司 缺陷检测方法和相关装置、设备、存储介质

Family Cites Families (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4255065B2 (ja) * 2003-08-07 2009-04-15 財団法人電力中央研究所 画像処理による電線異常検出方法および装置およびプログラム並びに電線点検用画像の作成方法
JP2011257303A (ja) * 2010-06-10 2011-12-22 Olympus Corp 画像取得装置、欠陥修正装置および画像取得方法
JP5442549B2 (ja) * 2010-07-16 2014-03-12 一般財団法人電力中央研究所 電線の異常検出方法、異常検出装置、および異常検出プログラム
CN103530620A (zh) * 2013-10-30 2014-01-22 华北电力大学 一种输电线路杆塔鸟巢的识别方法
CN106226050B (zh) * 2016-07-15 2019-02-01 北京航空航天大学 一种基于卷积神经网络的tfds故障图像自动识别方法
CN107392901A (zh) * 2017-07-24 2017-11-24 国网山东省电力公司信息通信公司 一种用于输电线路部件智能自动识别的方法
CN108389197B (zh) * 2018-02-26 2022-02-08 上海赛特斯信息科技股份有限公司 基于深度学习的输电线路缺陷检测方法
CN108648169B (zh) * 2018-03-20 2020-11-10 中国科学院自动化研究所 高压输电塔绝缘子缺陷自动识别的方法及装置
CN108597053A (zh) * 2018-04-25 2018-09-28 北京御航智能科技有限公司 基于图像数据和神经网络的杆塔和通道目标识别与缺陷诊断方法
CN110619619A (zh) * 2018-06-04 2019-12-27 杭州海康威视数字技术股份有限公司 一种缺陷检测方法、装置及电子设备
JP7028091B2 (ja) * 2018-07-11 2022-03-02 日本製鉄株式会社 表面欠陥検出装置及び表面欠陥検出方法
CN108921846A (zh) * 2018-07-17 2018-11-30 北京航空航天大学 一种基于灰度图像和深度图像结合的钢轨踏面缺陷识别方法
JP2020034413A (ja) * 2018-08-30 2020-03-05 東ソー株式会社 欠陥検出方法
CN109685930A (zh) * 2018-12-19 2019-04-26 深圳供电局有限公司 电力部件巡检方法、装置、计算机设备和存储介质
CN109829893B (zh) * 2019-01-03 2021-05-25 武汉精测电子集团股份有限公司 一种基于注意力机制的缺陷目标检测方法
CN109949296A (zh) * 2019-03-21 2019-06-28 北京中飞艾维航空科技有限公司 一种输电线路缺陷识别方法、装置及存储介质
CN110378222B (zh) * 2019-06-14 2023-10-27 南瑞集团有限公司 一种输电线路防震锤目标检测与缺陷识别方法及装置
CN110349145B (zh) * 2019-07-09 2022-08-16 京东方科技集团股份有限公司 缺陷检测方法、装置、电子设备以及存储介质
CN110517265B (zh) * 2019-09-04 2022-03-01 北京理工大学 一种产品表面缺陷的检测方法、装置及存储介质
CN110555842A (zh) * 2019-09-10 2019-12-10 太原科技大学 一种基于锚点集优化的硅片图像缺陷检测方法
CN110796107A (zh) * 2019-11-04 2020-02-14 南京北旨智能科技有限公司 电力巡检图像缺陷识别方法和系统、电力巡检无人机
CN110969620A (zh) * 2019-12-19 2020-04-07 中国农业大学 一种磁瓦波纹类缺陷的检测方法及装置
CN111260055B (zh) * 2020-01-13 2023-09-01 腾讯科技（深圳）有限公司 基于三维图像识别的模型训练方法、存储介质和设备
CN111353413B (zh) * 2020-02-25 2022-04-15 武汉大学 一种输电设备低漏报率缺陷识别方法
CN111415330A (zh) * 2020-02-27 2020-07-14 苏州杰锐思智能科技股份有限公司 基于深度学习的铜箔外观缺陷检测方法
CN111507958B (zh) * 2020-04-15 2023-05-26 全球能源互联网研究院有限公司 目标检测方法、检测模型的训练方法及电子设备

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20180115368A (ko) * 2017-04-12 2018-10-23 주식회사 포스코 선재의 표면 흠 자동 맵핑 장치 및 방법
CN109785288A (zh) * 2018-12-17 2019-05-21 广东电网有限责任公司 基于深度学习的输电设备缺陷检测方法及系统
CN111161246A (zh) * 2019-12-30 2020-05-15 歌尔股份有限公司 一种产品缺陷检测方法、装置与系统
CN111242943A (zh) * 2020-01-22 2020-06-05 腾讯科技（深圳）有限公司 图像处理方法、装置、图像处理设备及存储介质
CN111462109A (zh) * 2020-04-17 2020-07-28 广东电网有限责任公司 一种耐张线夹的缺陷检测方法、装置、设备及存储介质
CN112001902A (zh) * 2020-08-19 2020-11-27 上海商汤智能科技有限公司 缺陷检测方法和相关装置、设备、存储介质

Cited By (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114359281A (zh) * 2022-03-17 2022-04-15 南方电网数字电网研究院有限公司 基于层次化集成学习的电力部件识别方法和装置
CN114820561A (zh) * 2022-05-16 2022-07-29 奥蒂玛光学科技（深圳）有限公司 一种异常区域检测方法、自动光学检测设备以及存储介质
WO2023241352A1 (zh) * 2022-06-14 2023-12-21 中兴通讯股份有限公司 基于交互的编码方法、编码装置及可读存储介质
CN115131307A (zh) * 2022-06-23 2022-09-30 腾讯科技（深圳）有限公司 物品缺陷检测方法及相关装置
US11763549B1 (en) * 2022-07-29 2023-09-19 Contemporary Amperex Technology Co., Limited Method and apparatus for training cell defect detection model
CN115272249A (zh) * 2022-08-01 2022-11-01 腾讯科技（深圳）有限公司 一种缺陷检测方法、装置、计算机设备和存储介质
WO2024066035A1 (zh) * 2022-09-28 2024-04-04 深圳先进技术研究院 基于电池表面图像的缺陷检测方法、系统及相关设备
CN115272330A (zh) * 2022-09-28 2022-11-01 深圳先进技术研究院 基于电池表面图像的缺陷检测方法、系统及相关设备
CN115272330B (zh) * 2022-09-28 2023-04-18 深圳先进技术研究院 基于电池表面图像的缺陷检测方法、系统及相关设备
CN115564773A (zh) * 2022-11-18 2023-01-03 南方电网数字电网研究院有限公司 基于元学习的小样本图像缺陷检测方法、装置和设备
CN115526892A (zh) * 2022-11-29 2022-12-27 南方电网数字电网研究院有限公司 基于三维重建的图像缺陷去重检测方法和装置
CN115631193A (zh) * 2022-12-06 2023-01-20 北京矩视智能科技有限公司 基于注意力机制的工件缺陷检测方法、装置及存储介质
CN116539619B (zh) * 2023-04-19 2024-05-10 广州里工实业有限公司 产品缺陷检测方法、系统、装置及存储介质
CN116539619A (zh) * 2023-04-19 2023-08-04 广州里工实业有限公司 产品缺陷检测方法、系统、装置及存储介质
CN116385952B (zh) * 2023-06-01 2023-09-01 华雁智能科技(集团)股份有限公司 配网线路小目标缺陷检测方法、装置、设备及存储介质
CN116385952A (zh) * 2023-06-01 2023-07-04 华雁智能科技(集团)股份有限公司 配网线路小目标缺陷检测方法、装置、设备及存储介质
CN116449875B (zh) * 2023-06-16 2023-09-05 拓恒技术有限公司 一种无人机巡检方法及系统
CN116449875A (zh) * 2023-06-16 2023-07-18 拓恒技术有限公司 一种无人机巡检方法及系统
CN117078689A (zh) * 2023-10-17 2023-11-17 沈阳宏远电磁线股份有限公司 一种基于机器视觉的线缆缺陷识别方法及系统
CN117437188A (zh) * 2023-10-17 2024-01-23 广东电力交易中心有限责任公司 一种用于智慧电网的绝缘子缺陷检测系统
CN117078689B (zh) * 2023-10-17 2024-01-30 沈阳宏远电磁线股份有限公司 一种基于机器视觉的线缆缺陷识别方法及系统
CN117437188B (zh) * 2023-10-17 2024-05-28 广东电力交易中心有限责任公司 一种用于智慧电网的绝缘子缺陷检测系统
CN117253179A (zh) * 2023-11-20 2023-12-19 合肥中科类脑智能技术有限公司 配电线路鸟巢检测方法、存储介质和电子设备
CN117253179B (zh) * 2023-11-20 2024-02-02 合肥中科类脑智能技术有限公司 配电线路鸟巢检测方法、存储介质和电子设备
CN117576042A (zh) * 2023-11-22 2024-02-20 魅杰光电科技(上海)有限公司 晶圆缺陷检测方法、系统、电子设备以及存储介质
CN117934462A (zh) * 2024-03-21 2024-04-26 高视科技(苏州)股份有限公司 用于调整检测参数的方法、电子设备及存储介质

Also Published As

Publication number Publication date
TWI770757B (zh) 2022-07-11
CN112001902A (zh) 2020-11-27
KR20220023335A (ko) 2022-03-02
TW202208832A (zh) 2022-03-01
JP2022548438A (ja) 2022-11-21

Similar Documents

Publication Publication Date Title
WO2022036953A1 (zh) 缺陷检测方法和相关装置、设备、存储介质、计算机程序产品
US11676390B2 (en) Machine-learning model, methods and systems for removal of unwanted people from photographs
WO2017148265A1 (zh) 一种单词的分割方法和装置
WO2018233038A1 (zh) 基于深度学习的车牌识别方法、装置、设备及存储介质
WO2018145470A1 (zh) 一种图像检测方法和装置
CN111695514B (zh) 一种基于深度学习的雾天下车辆检测方法
US9070005B2 (en) Image processing apparatus, image processing method, non-transitory computer-readable medium, and image processing system for detection of target cells using image feature determination
JP2014531097A (ja) マルチレイヤ連結成分をヒストグラムと共に用いるテキスト検出
CN110909724B (zh) 一种多目标图像的缩略图生成方法
WO2021139197A1 (zh) 一种图像处理方法及装置
WO2020151299A1 (zh) 黄色禁停线识别方法、装置、计算机设备及存储介质
CN112614136A (zh) 一种红外小目标实时实例分割方法及装置
CN113781421A (zh) 基于水下的目标识别方法、装置及系统
WO2023024766A1 (zh) 物体尺寸识别方法、可读存储介质及物体尺寸识别系统
CN112329881A (zh) 车牌识别模型训练方法、车牌识别方法及装置
CN113673584A (zh) 一种图像检测方法及相关装置
CN111935479A (zh) 一种目标图像确定方法、装置、计算机设备及存储介质
CN114445651A (zh) 一种语义分割模型的训练集构建方法、装置及电子设备
CN112434647A (zh) 一种人脸活体检测方法
CN113076860A (zh) 一种野外场景下的鸟类检测系统
CN115512207A (zh) 一种基于多路特征融合及高阶损失感知采样的单阶段目标检测方法
CN112308061B (zh) 一种车牌字符识别方法及装置
CN112070041B (zh) 一种基于cnn深度学习模型的活体人脸检测方法和装置
CN114445916A (zh) 一种活体检测方法、终端设备及存储介质
CN114821582A (zh) 基于深度学习的ocr识别方法

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2021567817

Country of ref document: JP

Kind code of ref document: A

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20950154

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 11.07.2023)

122 Ep: pct application non-entry in european phase

Ref document number: 20950154

Country of ref document: EP

Kind code of ref document: A1