WO2020156409A1

WO2020156409A1 - Data processing method, defect detection method, computing apparatus, and storage medium

Info

Publication number: WO2020156409A1
Application number: PCT/CN2020/073704
Authority: WO
Inventors: 李虹杰; 魏溪含; 陈想; 陈岩
Original assignee: 阿里巴巴集团控股有限公司
Priority date: 2019-02-02
Filing date: 2020-01-22
Publication date: 2020-08-06
Also published as: TW202040511A; CN111598825A; CN111598825B

Abstract

Embodiments of the present application provide a data processing method, a defect detection method, a computing apparatus, and a storage medium. The method in the embodiments of the present application comprises: determining a predicted type and a predicted detection box of a pixel according to an image feature, the predicted type indicating whether the pixel is a part of a defect, and the predicted target box indicating a defect position if the pixel is a part of a defect; determining a type loss between the predicted type of the pixel and an actual type, determining a detection box loss between the predicted detection box of the pixel and an actual detection box; and determining a total loss of the pixel accordingly, generating a defect detection model according to the total loss, and performing defect detection on an image awaiting prediction according to the detection model. The invention enables highly accurate and efficient segmentation for defect detection, and achieves comprehensive automated segmentation for defect detection, thereby reducing labor costs, improving product manufacturing efficiency, and adding product value.

Description

Data processing method, defect detection method, computing device and storage medium

This application claims the priority of the Chinese patent application filed on February 02, 2019 with the application number 201910107275.8 and the invention title "data processing method, defect detection method, computing device and storage medium", the entire content of which is incorporated by reference In this application.

Technical field

This application relates to the field of computer technology, and in particular to a data processing method, a defect detection method, a computing device, and a storage medium.

Background technique

With the development of modern large-scale industry, high-speed production efficiency and low-speed inspection efficiency have formed a huge contradiction. However, in industrial project inspection, a basic requirement is to qualitatively determine whether the product has defects, and to prevent defective products from leaving the workshop or going out. The next production line. At present, in some industrial production, many companies use manual inspection to test the quality of products. Traditional manual visual inspection is a completely subjective evaluation method, which has disadvantages such as large subjective factors, poor real-time performance, and low efficiency. It can no longer meet the testing needs of manufacturing companies.

Summary of the invention

Various aspects of the present application provide a data processing method, a defect detection method, a computing device, and a storage medium, which are used to detect defects more accurately in an all-round and automated manner and improve production efficiency.

An embodiment of the present application provides a data processing method, including: acquiring features of at least one picture, and determining a prediction type and a prediction detection frame of a pixel in the picture according to the feature, where the prediction type reflects whether the pixel is a defect, The prediction target frame reflects the defect position of the pixel when it has a defect; determines the type loss between the predicted type and the true type of the pixel, and determines the detection frame loss between the predicted detection frame and the real detection frame of the pixel; according to Pixel type loss and detection frame loss, determine the total loss of pixels, and generate a defect detection model based on the total loss.

An embodiment of the present application also provides a defect detection method, including: obtaining at least one picture to be predicted, determining the prediction type and the prediction detection frame of the pixel in the picture to be predicted; for a picture to be predicted, according to the prediction type The pixels are aggregated to generate a pixel area, and the predicted detection frame of the pixel area is determined according to the predicted detection frame of the pixel; the defect of the picture to be predicted is determined according to the predicted detection frame of the pixel area and/or the pixel area.

An embodiment of the present application also provides a defect detection method, including: acquiring features of at least one picture, and determining a prediction type and a prediction detection frame of pixels in the picture according to the features, the prediction type reflecting whether the pixel is a defect Situation, the prediction target frame reflects the defect position of the pixel when the pixel has a defect; the type loss between the predicted type and the true type of the pixel is determined, and the detection frame loss between the predicted detection frame and the real detection frame of the pixel is determined ; According to the pixel type loss and the detection frame loss, determine the total loss of the pixel, according to the total loss, generate a defect detection model; according to the generated defect detection model, determine the prediction type of the pixel in the picture to be predicted and the prediction detection frame For a picture to be predicted, the pixels are aggregated according to the prediction type to generate a pixel area, and the prediction detection frame of the pixel area is determined according to the prediction detection frame of the pixel; according to the prediction of the pixel area and/or pixel area Check the frame to determine the defects of the picture to be predicted.

An embodiment of the present application also provides a defect detection system, including: a first computing device and a second computing device; the first computing device acquires the characteristics of at least one picture, and determines the pixels in the picture according to the characteristics A prediction type and a prediction detection frame, the prediction type reflects whether the pixel is a defect, the prediction target frame reflects the location of the defect when the pixel has a defect; the type loss between the predicted type and the true type of the pixel is determined, And determine the detection frame loss between the predicted detection frame of the pixel and the real detection frame; determine the total loss of the pixel according to the pixel type loss and the detection frame loss, and generate a defect detection model based on the total loss; the second The computing device determines the prediction type and the prediction detection frame of the pixel in the picture to be predicted according to the detection model of the generated defect; for a picture to be predicted, the pixels are aggregated according to the prediction type to generate a pixel area, and the pixel is detected according to the prediction The frame determines the prediction detection frame of the pixel area; and determines the defect of the picture to be predicted according to the prediction detection frame of the pixel area and/or the pixel area.

An embodiment of the present application also provides a computing device, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program for: obtaining at least one picture Feature, determine the prediction type and prediction detection frame of the pixel in the picture according to the feature, the prediction type reflects whether the pixel is a defect, the prediction target frame reflects the location of the defect when the pixel has a defect; determining the location of the pixel The type loss between the prediction type and the real type, and the detection frame loss between the predicted detection frame and the real detection frame for determining the pixel; according to the pixel type loss and the detection frame loss, the total loss of the pixel is determined, according to the total Loss, generating a defect detection model.

The embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by one or more processors, the one or more processors will cause the one or more processors to implement the steps in the above-mentioned defect detection method.

An embodiment of the present application also provides a computing device, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program, and is used to: obtain at least one to be predicted Picture, determine the prediction type and prediction detection frame of the pixel in the picture to be predicted; for a picture to be predicted, the pixels are aggregated according to the prediction type to generate a pixel area, and the pixel area is determined according to the prediction detection frame of the pixel Predictive detection frame; according to the pixel area and/or the predicted detection frame of the pixel area, determine the defect of the picture to be predicted.

An embodiment of the present application also provides a computing device, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program for: obtaining at least one picture Feature, determine the prediction type and prediction detection frame of the pixel in the picture according to the feature, the prediction type reflects whether the pixel is a defect, the prediction target frame reflects the location of the defect when the pixel has a defect; determining the location of the pixel The type loss between the prediction type and the real type, and the detection frame loss between the predicted detection frame and the real detection frame for determining the pixel; according to the pixel type loss and the detection frame loss, the total loss of the pixel is determined, according to the total Loss, generate a defect detection model; according to the generated defect detection model, determine the prediction type and prediction detection frame of the pixel in the picture to be predicted; for a picture to be predicted, aggregate pixels according to the prediction type to generate a pixel area, and According to the predicted detection frame of the pixel, the predicted detection frame of the pixel area is determined; and the defect of the picture to be predicted is determined according to the predicted detection frame of the pixel area and/or the pixel area.

In the embodiment of the present application, the prediction type of the pixel and the prediction detection frame are determined according to the picture characteristics, the prediction type reflects whether the pixel is a defect, and the prediction target frame reflects the defect position of the pixel when it has a defect; the prediction type of the pixel is determined and true The type loss between the types, and the detection frame loss between the predicted detection frame and the real detection frame to determine the pixel, so as to determine the total loss of the pixel, based on the total loss, generate a defect detection model, and perform the predicted image according to the detection model Defect detection can perform defect detection segmentation with high accuracy and high speed, and realize the segmentation of defect detection in an all-round way, thereby reducing labor costs, improving product production efficiency, and creating value for products.

Description of the drawings

The drawings described here are used to provide a further understanding of the application and constitute a part of the application. The exemplary embodiments and descriptions of the application are used to explain the application and do not constitute an undue limitation of the application. In the attached picture:

FIG. 1 is a schematic structural diagram of a defect detection system according to an exemplary embodiment of the application;

2 is a schematic flowchart of a data processing method according to an exemplary embodiment of this application;

3 is a schematic flowchart of a defect detection method provided by an exemplary embodiment of the application;

4 is a schematic flowchart of a defect detection method provided by another exemplary embodiment of this application;

Fig. 5 is a schematic diagram of a defect provided by an exemplary embodiment of the application;

FIG. 6 is a schematic diagram of a defect provided by another exemplary embodiment of the application;

FIG. 7 is a schematic diagram of obtaining difficult negative samples according to another exemplary embodiment of this application;

FIG. 8 is a schematic structural diagram of a data processing device provided by an exemplary embodiment of this application;

FIG. 9 is a schematic structural diagram of a defect detection device provided by an exemplary embodiment of the application;

10 is a schematic structural diagram of a defect detection device provided by another exemplary embodiment of this application;

FIG. 11 is a schematic structural diagram of a computing device provided by an exemplary embodiment of this application;

FIG. 12 is a schematic structural diagram of a computing device provided by another exemplary embodiment of this application;

FIG. 13 is a schematic structural diagram of a computing device provided by another exemplary embodiment of this application.

detailed description

In order to make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely in conjunction with specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

In industrial defect detection, a basic requirement is to qualitatively determine whether there is a defect, and to prevent defective products from leaving the workshop or the next production line. The next step is to detect the location and category of the flaws, so that the flaws can be quickly located for subsequent processing and repair. When detecting the defect location, the defect can be segmented at the pixel level, so that the area of the defect can be determined and it is convenient for customers to grade the defect.

Product manufacturers can achieve a high recall rate by detecting defects and avoid defective products from leaving the production line, causing greater losses. At the same time, a high accuracy rate (low false alarm rate) is required, otherwise it will cause two aspects: in the production process, it will cause a lot of unnecessary re-inspection, rework, and downstream customer complaints, which will cause customer production costs to rise; In the delivery project, due to the wrong report problem, the product rating is low, and the product is sold at a low price, causing loss of customer revenue.

The embodiment of the application obtains the detection result of the defect by directly performing the pixel classification and the determination of the detection frame on the feature layer.

The following describes in detail the technical solutions provided by the embodiments of the present application with reference to the accompanying drawings.

Fig. 1 is a schematic structural diagram of a defect detection system provided by an exemplary embodiment of the application. As shown in FIG. 1, the detection system 100 includes: a first computing device 101 and a second computing device 102.

Among them, the first computing device 101 may be a stand-alone server or a server array, or a cloudized service virtual machine VM.

The second computing device 102 may also be a stand-alone server or a server array, or a cloudized service virtual machine VM.

Among them, the first computing device 101 refers to a computing device that can generate a defect detection model, and the computing device refers to a device that can provide model training services in a network virtual environment. In terms of physical implementation, the computing device can be any device that can provide computing services, respond to service requests, and perform processing, such as a conventional server, cloud server, cloud host, virtual center, and so on. The composition of the computing device mainly includes a processor, a hard disk, a memory, a system bus, etc., and is similar to a general computer architecture.

The second computing device 102 refers to a computing device that can perform defect detection on a picture, and the computing device refers to a device that can provide computing processing services in a network virtual environment. In terms of physical implementation, the computing device can be any device that can provide computing services, respond to service requests, and perform processing, such as a conventional server, cloud server, cloud host, virtual center, and so on. The composition of the computing device mainly includes a processor, a hard disk, a memory, a system bus, etc., and is similar to a general computer architecture.

In the example of this application, the first computing device 101 obtains the characteristics of at least one picture, and determines the prediction type and the prediction detection frame of the pixel in the picture according to the characteristics. The prediction type reflects whether the pixel is a defect, and the prediction target frame reflects the pixel The position of the flaw when there is a flaw; determine the type loss between the predicted type and the true type of the pixel, and determine the detection frame loss between the predicted detection frame of the pixel and the real detection frame; according to the pixel type loss and the detection frame loss, Determine the total loss of pixels, and generate a defect detection model based on the total loss.

The second computing device 102 receives the detection model sent by the first computing device 101, and the second computing device 102 obtains at least one picture to be predicted, inputs it into the detection model, and obtains the prediction type and the prediction detection frame of the pixel in the picture to be predicted ; For a picture to be predicted, the second computing device 102 aggregates pixels according to the prediction type to generate a pixel area, and determines the prediction detection frame of the pixel area according to the prediction detection frame of the pixel; according to the prediction detection of the pixel area and/or the pixel area Box to determine the defects of the picture to be predicted.

In some examples, the detection system may also include a terminal 103.

The terminal 103 may be any device with a certain computing capability, for example, it may be a smart phone, a notebook, a personal computer (PC), etc. The basic structure of the terminal 103 includes: at least one processor. The number of processors depends on the configuration and type of the terminal 103. The terminal 103 may also include a memory. The memory may be volatile, such as RAM, or non-volatile, such as read-only memory (ROM), flash memory, etc., or may include both Kind of type. The memory usually stores an operating system (Operating System, OS), one or more application programs, and may also store program data. In addition to the processing unit and memory, the terminal 103 also includes some basic configurations, such as a network card chip, an IO bus, a camera, and audio and video components. Optionally, the terminal 103 may also include some peripheral devices, such as a keyboard, a mouse, a stylus, and a printer. Other peripheral devices are well known in the art and will not be described in detail here.

In some examples, the terminal 103 sends the picture to be predicted to the second computing device 102, and may receive the detection result of the defect in the picture to be predicted returned by the second computing device 102.

In the foregoing embodiment, the first computing device 101 can be connected to the second computing device 102, and the second computing device 102 can be connected to the terminal 103, and the network connection can be a wired network connection.

It should be noted that the embodiment of the present application can also use only one computing device to train the model and directly perform defect detection based on the generated model. The computing device can be a stand-alone server or a server array, or a cloud-based service virtualization.机VM.

In some instances, the first computing device 101 may also obtain at least one picture to be predicted from the terminal 103 and input it into the detection model to obtain the prediction type and the prediction detection frame of the pixel in the picture to be predicted; For a picture to be predicted, the pixels are aggregated according to the prediction type to generate a pixel area, and the prediction detection frame of the pixel area is determined according to the prediction detection frame of the pixel; the prediction detection frame of the pixel area and/or the pixel area is determined defect.

In some instances, the second computing device 102 may also obtain the characteristics of at least one picture, and determine the prediction type and the prediction detection frame of the pixel in the picture according to the characteristics. The prediction type reflects whether the pixel is a defect, and the prediction target frame reflects the pixel. The position of the flaw when there is a flaw; determine the type loss between the predicted type and the true type of the pixel, and determine the detection frame loss between the predicted detection frame of the pixel and the real detection frame; according to the pixel type loss and the detection frame loss, Determine the total loss of pixels, and generate a defect detection model based on the total loss.

In the following, in conjunction with the method embodiments, the model generation process of the first computing device 101 or the second computing device 102 and the defect detection process of the first computing device 101 or the second computing device 102 can be described in detail.

FIG. 2 is a schematic flowchart of a data processing method according to an exemplary embodiment of the application. The method 200 provided by the embodiment of the present application is executed by a computing device, and the method 200 includes the following steps:

201: Obtain the feature of at least one picture, and determine the prediction type and the prediction detection frame of the pixel in the picture according to the feature, the prediction type reflects whether the pixel is a defect, and the prediction target frame reflects the defect position of the pixel when it has a defect.

202: Determine the type loss between the predicted type and the true type of the pixel, and determine the detection frame loss between the predicted detection frame and the true detection frame of the pixel.

203: Determine the total loss of pixels according to the type loss of pixels and the loss of detection frame, and generate a defect detection model according to the total loss.

The following is a detailed description of the above steps:

Among them, characteristics refer to the corresponding (essential) characteristics or characteristics of a certain type of object that are different from other types of objects, or a collection of these characteristics and characteristics. Features are data that can be extracted through measurement or processing. For an image (or picture), each image has its own characteristics that can be distinguished from other types of images, such as color characteristics, texture characteristics, shape characteristics, and spatial relationship characteristics.

The prediction type refers to the prediction classification of whether a pixel is a defect. When the pixel is a defect, the type of the defect belongs to. The final prediction type of the pixel can be determined by the prediction probability.

The prediction target frame refers to the prediction rule frame where the defect of the pixel belongs on the picture, such as a rectangular frame.

In some instances, the way to obtain the characteristics of the picture may be: obtaining the characteristics of the picture through a convolutional neural network. For example, input a picture into a convolutional neural network to obtain various features of the picture.

Among them, convolutional neural network refers to a type of feedforward neural network that includes convolutional calculations and has a deep structure. It includes a feature extractor composed of a convolutional layer, a pooling layer, and a fully connected layer.

In some examples, determining the prediction type and prediction detection frame of the pixel in the picture according to the feature includes: processing the feature according to the full convolutional neural network model to determine the prediction type and prediction detection frame of the pixel in the picture.

Among them, a fully convolutional neural network (also called a fully convolutional neural network model) refers to a neural network improved on the basis of a convolutional neural network, which can convert the fully connected layer in the convolutional neural network into a convolutional layer Neural network.

In some instances, processing the features according to the fully convolutional neural network model to determine the prediction type and prediction detection frame of the pixels in the picture includes: processing the features according to the fully convolutional neural network model to obtain at least one of the pixels The prediction probability of the prediction type; the prediction type with the largest prediction probability is selected as the prediction type of the pixel; the prediction detection frame corresponding to the prediction type with the largest prediction probability is used as the prediction detection frame of the pixel.

For example, the computing device can obtain multiple pictures from other computing device nodes, normalize the multiple pictures, and input the processed pictures into a convolutional neural network (also called a convolutional neural network model) , Get the feature or feature layer of the processed picture. The computing device is processing the feature or feature layer of the processed picture according to the full convolutional neural network (also called the full convolutional neural network model), classifying each pixel on the corresponding picture, and obtaining the prediction type of the pixel At the same time, the prediction detection frame corresponding to each prediction type is determined according to the full convolutional neural network. For example, the prediction probability of a pixel of A picture without defect is 0.2, the prediction probability of a defect is 0.6, and the prediction probability of a defect is 0.6. The prediction probability of is 0.3, the prediction type with the highest prediction probability is selected as the prediction type of the pixel, that is, the prediction type of a pixel is a defect, and the rectangular frame coordinates (predx1, predx2, predy1, predy2) are used as the coordinates of the prediction detection frame of pixel a.

It should be noted that for the prediction detection frame as a rectangular frame, its coordinates are composed of two points with the same abscissa and two points with the same ordinate, so for simple illustration, the coordinates of the rectangular block can be written as (pred x1, pred x2, pred y1, pred y2).

In addition, the coordinates mentioned above are all coordinates in the picture.

Among them, the type loss refers to the degree of difference between the predicted type used for evaluation and the true type, for example, the degree of different types. Among them, the true type refers to a true classification of whether a pixel belongs to a defect, and each pixel belongs to only one true type, and the true type is known data.

The detection frame loss refers to the degree of difference between the predicted detection frame used for evaluation and the real detection frame, for example, the degree of different coordinate positions. Among them, the real detection frame refers to a real regular frame where the defect of the pixel belongs on the picture, such as a rectangular frame, etc., and each pixel has only one real detection frame, and the real detection frame is known data.

In some instances, determining the type loss between the predicted type and the true type of the pixel includes: determining the type loss of the pixel under the true type according to the true type of the pixel and the predicted probability of the predicted type that matches the true type.

In some instances, the type loss between the predicted type and the true type of the pixel can be determined by the following formula 1):

Among them, Loss_per_pixel _o is the type loss, M is the total number of true types, y _o,c is the value corresponding to whether the prediction type of the pixel o is the true type c, and p _o,c is the predicted probability of the pixel o belonging to the prediction type c.

It should be noted that the total number of true types refers to the total number of true types involved in all pixels of multiple pictures. For example, there are 7 types involved, including no defect, a defect, b defect, c defect, d defect, e Flaws and f flaws. When the prediction type of the pixel o is the same as the true type, y _o,c is 1, and when the prediction type of the pixel o is different from the true type, y _o,c is 0. c belongs to M and can be 1-7.

For example, according to the above, taking pixel a as an example, the defect type of pixel a is a defect, that is, a defect type. For a pixel, its type loss=0 (c=1)-1*log0.6 (c=2)+0(c=3)+0(c=4)+0(c=5)+0(c=6)+0(c=7)=-log 0.6.

It should be noted that c=1 means when the real type is a flawless type, c=2 means when the real type is a flawed type, and so on until c=7, which will not be repeated.

In some instances, determining the detection frame loss between the predicted detection frame of the pixel and the real detection frame includes: determining the relative predicted coordinates of the predicted detection frame for the corresponding pixel according to the predicted coordinates of the predicted detection frame in the corresponding picture; The relative real coordinates of the real detection frame to the corresponding pixels; determine the coordinate distance between the relative predicted coordinates and the relative real coordinates; determine the loss of the detection frame according to the coordinate distance.

Among them, the predicted coordinates refer to the coordinate positions corresponding to the predicted detection frame obtained through the full convolutional neural network described above, for example, the rectangular frame coordinates (pred x1, pred x2, pred y1, pred y2) corresponding to the predicted detection frame.

The real coordinates of the real detection frame in the corresponding picture are the coordinate positions of the pixel real detection frame, for example, the rectangular frame coordinates (x1, x2, y1, y2) corresponding to the real detection frame, and the real coordinates are known.

The relative prediction coordinate refers to the coordinate position where the detection frame is predicted based on the pixel.

The relative real coordinate refers to the coordinate position of the real detection frame based on the pixel.

In some instances, the relative prediction coordinates can be determined as follows: suppose the position of the prediction detection frame is (pred x1, pred x2, pred y1, pred y2), the current pixel coordinates are (x, y), and the picture size is set to (w ,h), w is the width of the picture, and h is the height of the picture. The relative prediction coordinates (also called regression target) are ((x-pred x1)/w,(y- pred y1)/h,(pred x2- x)/w,(pred y2-y)/h).

It should be noted that the method for determining the relative real coordinates is the same as the method for determining the relative predicted coordinates, and will not be repeated here. For example, the relative real coordinates can be ((x-x1)/w, (y-y1)/h, (x2-x)/w, (y2-y)/h).

In addition, the relative real coordinates only exist under the premise that the corresponding pixel has a defect, and the relative prediction coordinates only exist under the premise that the corresponding pixel prediction has a defect. When the pixel has a relative prediction coordinate, it may not have a relative real coordinate. At this time, the relative real coordinate is 0.

In some instances, the detection frame loss can be determined by formula 2):

Among them, Loss _det is the loss of the detection frame, and x is the coordinate distance.

For example, according to the above, for pixel a, x is ((x-pred x1)/w,(y-pred y1)/h,(pred x2-x)/w,(pred y2-y)/ h) Coordinate distance from ((x-x1)/w, (y-y1)/h, (x2-x)/w, (y2-y)/h). When |x| is less than 1, the detection frame loss is 0.5x ² , otherwise when |x| is greater than or equal to 1, the detection frame loss is |x|-0.5.

Among them, the total loss refers to the detection degree of the difference between the predicted flaw and the real flaw used for evaluation.

In some instances, determining the total loss of pixels according to the type loss of the pixel and the loss of the detection frame includes: determining the type loss sum of the pixel according to the type loss of the pixel; determining the detection frame loss sum of the pixel according to the loss of the detection frame of the pixel ; Determine the total loss based on the type loss sum and the detection frame loss sum.

In some instances, the pixel type loss and Loss _cls can be determined by the following equation 3):

Among them, N is the total number of pixels in multiple pictures.

Therefore, the type loss and Loss _cls are the sum of the type loss of all pixels in multiple pictures.

In some instances, the method of determining the sum of the detection frame losses of the pixels may be: the sum of the detection frame losses of the pixels with the predicted detection frame in multiple pictures.

In some examples, determining the total loss according to the sum of the type loss and the sum of the detection frame loss includes: determining the total loss of the type loss and the sum of the detection frame loss according to a weighted sum algorithm.

In some instances, the total loss Loss can be determined by the following equation 4):

among them,

Is the weight coefficient, and the total Loss _det is the sum of the loss of the detection frame.

In order to reduce the false alarm rate, it is necessary to mine negative samples that are prone to false alarms. Such samples may exist in images without defects, so negative samples need to be selectively screened. Since positive samples have been participating in training, they will not be affected.

In some examples, the method 200 further includes: for any picture, obtaining difficult negative samples from normal pixels that are not defective in the picture. The difficult negative samples refer to negative samples with predetermined quality; for any picture, Defective pixels in the picture are taken as positive samples; among them, determining the type loss sum of pixels according to the type loss of pixels includes: determining the type loss sum of difficult negative samples in at least one picture; determining the type loss sum of at least one picture The type loss sum of the positive sample, the type loss sum of the difficult negative sample and the type loss sum of the positive sample, are used as the type loss of the corresponding picture.

Among them, the positive sample refers to the pixel belonging to the defect.

Negative samples refer to normal pixels that are not blemishes.

Difficult negative samples refer to high-quality negative samples, which are easily detected as defective pixels.

In some examples, obtaining difficult negative samples from normal pixels that are not defective in the picture includes: taking normal pixels that are not defective in each picture as negative samples, and determining the type loss of negative samples; The type loss is sorted from large to small, and two negative samples corresponding to the largest loss of adjacent types are selected, the negative sample that is sorted later is used as the critical point, and the negative sample sorted before the critical point is used as the difficult negative sample.

For example, according to the foregoing, select any picture from multiple pictures as an example for description. For picture A with multiple negative samples, calculate the type loss of each negative sample according to formula 1), and obtain The type loss of multiple pixels is sorted from large to small, and the two negative samples corresponding to the largest adjacent loss gap are selected from this ranking. As shown in Figure 7, it can be seen that the maximum adjacent loss gap is between 0.7 and 0.3 The difference of 0.4, at this time, the negative sample corresponding to 0.7 is the 200th negative sample in the sorting, and the negative sample corresponding to 0.3 is the 201st negative sample in the sorting, then the 201st negative sample will be selected as the critical point, then 0.3 For the corresponding gap point, select the first 200 negative samples as difficult negative samples.

It should be noted that the selection of difficult negative samples is performed for each picture, and multiple difficult negative samples corresponding to multiple pictures are obtained.

In some instances, the type loss sum of difficult negative samples can be determined by the following formula 5) and formula 6):

Among them, weight _o is the weight of pixel o, and gaploss is the gap point gappoint.

Among them, the type loss of the _{negative sample} and Loss _{negative sample are} calculated as the following formula 6):

Among them, the negative sample Loss_per_pixel _o is the type loss of the negative sample of pixel o.

It should be noted that formula 5) and formula 6) are for a picture. Each of the multiple pictures can be processed according to formula 5) and formula 6), and the type loss sum of the difficult negative samples can be obtained.

The Loss _{negative samples} of multiple pictures and the type loss of each positive sample of the multiple pictures are summed to obtain Loss _cls corresponding to the multiple pictures.

It should be noted that, in order to ensure the convergence of algorithm training, if the number of defective pixel samples is N for images containing defects, the number of negative samples participating in training is at least N and at most 5N. For images without flaws, the number of difficult negative samples can be determined according to the above method. In order to ensure the convergence of the algorithm, the threshold of the number of difficult negative samples can be set. If the size of any picture is (w, h), select the difficult negative The number of samples is at most (w*h)/r, where r is the ratio value. For pictures containing defects, the number of difficult negative samples is also determined according to the above method.

In some instances, generating a defect detection model based on the total loss includes: updating the detection parameters of the detection model according to the total loss, and performing iterative training of the model until the iterative training stop condition is met, and generating the detection model.

Among them, the detection parameters are parameters used to detect defects, such as parameters in a convolutional neural network, which can be used to obtain features of a picture, or parameters of a fully convolutional neural network, which can be used to determine the prediction type of pixels in the picture and Predict the detection frame.

For example, according to the foregoing, the parameters of the fully convolutional neural network are adjusted according to the total loss, and the new total loss is continuously determined after the above steps, and the parameters of the fully convolutional neural network are adjusted according to the total loss, and the full volume is continuously performed Iteration of the product neural network, until the total loss is maintained within a threshold range, or the number of training times reaches the threshold, the training can be stopped and a trained detection model, such as a fully convolutional neural network model, can be obtained.

In the embodiment of the present application, the pixel value is directly segmented through the fully convolutional neural network, and the prediction detection frame is regressed at the same time, so as to avoid the problem that the defect aspect ratio gap is too large and cannot be covered, and realizes multi-scale and large deformation Defect detection; by dynamically screening the number of negative samples, it ensures that positive samples participate in training and difficult negative samples are trained. At the same time, in the industrial defect detection scene, flawless pictures are far more than flawed pictures, making full use of pictures without flaws Negative sample training is carried out, and the areas that are likely to cause false alarms will participate in the training, maintaining a balance between positive and negative ratios, and reducing false alarms without reducing the model recall rate.

FIG. 3 is a schematic flowchart of a defect detection method provided by another exemplary embodiment of the application. The method 300 provided in the embodiment of the present application is executed by a computing device, and the method 300 includes the following steps:

301: Obtain at least one picture to be predicted, and determine the prediction type and the prediction detection frame of the pixel in the picture to be predicted.

302: For a picture to be predicted, the pixels are aggregated according to the prediction type to generate a pixel area, and the prediction detection frame of the pixel area is determined according to the prediction detection frame of the pixel.

303: Determine the defect of the picture to be predicted according to the pixel area and/or the predicted detection frame of the pixel area.

The following is a detailed description of the above steps:

In some examples, determining the prediction type and prediction detection frame of the pixel in the picture to be predicted includes: inputting at least one picture to be predicted into the generated defect detection model to obtain the prediction type and prediction of the pixel in the picture to be predicted Check box.

For example, according to the foregoing, the computing device can obtain multiple to-be-predicted pictures from the terminal, normalize the multiple to-be-predicted pictures, and input the processed pictures into the generated defect detection model, such as In the fully convolutional neural network model, forward propagation is performed to obtain the prediction type and the prediction detection frame of the pixel in the picture to be predicted.

Among them, the pixel area refers to an area composed of multiple pixels.

In some instances, aggregating pixels according to the prediction type to generate a pixel area includes: determining the defect type of each pixel according to the predicted probability of the prediction type; aggregating pixels according to the defect type to generate an aggregate area of pixels of the same defect type, as Pixel area.

In some examples, the method 300 further includes: using the same defect type as the defect type of the pixel area; wherein, determining the predicted detection frame of the pixel area according to the predicted detection frame of the pixel includes: selecting the defect type of the pixel area The predicted detection frame of the pixel with the highest predicted probability is used as the predicted detection frame of the pixel area.

For example, according to the foregoing, the computing device processes the predicted probability of the pixel by means of the softmax function (normalized exponential function) in the generated flaw detection model to obtain the prediction type corresponding to the maximum predicted probability of each pixel. As the defect type of each pixel; the area is aggregated according to the defect type of each pixel, and the pixels of the same defect type are aggregated to generate a pixel area. If the defect type of the pixel in the pixel area is defect c, then The defect type of the pixel area is also defect c, so that the semantic segmentation result of the defect is obtained, as shown in Fig. 5, where the aggregation area is an elliptical area in a rectangular frame. The pixel with the highest predicted probability of defect c in the pixel area is selected, and the predicted detection frame of this pixel is used as the predicted detection frame of the pixel area.

It should be noted that the semantic segmentation result refers to segmentation according to the semantics of the image, such as image content as the semantics.

In addition, the defect type of the pixel area can also be determined directly through the fully convolutional neural network model to realize semantic segmentation.

In some examples, determining the defect of the picture to be predicted according to the predicted detection frame of the pixel area and/or the pixel area includes: when the defect type of the pixel area belongs to the first type, taking the defect type of the pixel area as the picture to be predicted The type of flaw included, the prediction detection frame of the pixel area is used as the prediction detection frame of the flaw contained in the picture to be predicted; when some pixels in the pixel area exceed the prediction detection frame, some pixels are removed from the pixel area and the remaining pixels are composed The area of is used as the pixel area of the defect contained in the picture to be predicted.

Among them, the first type may be block defects, as shown in FIG. 5, an oval area.

For example, according to the foregoing, when the defect type of the pixel area is defect a, and the defect a is a type of block defect, then the corresponding picture to be predicted contains block defects, and the computing device directly detects the prediction of the pixel area The frame is used as the predictive detection frame of the blocky defect, and at the same time, the pixels in the pixel area beyond the predicted detection frame are removed, and the area composed of the remaining pixels in the pixel area is taken as the pixel area of the blocky defect.

In some examples, determining the defect of the picture to be predicted according to the prediction detection frame of the pixel area and/or the pixel area includes: when the defect type of the pixel area belongs to the second type, using the defect type of the pixel area as the picture to be predicted The types of defects included are the pixel area as the pixel area of the defect contained in the picture to be predicted, and the smallest bounding rectangle of the pixel area is taken as the prediction detection frame of the defect contained in the picture to be predicted.

Among them, the first type may be linear defects, as shown in FIG. 6, several linear defects.

For example, according to the foregoing, when the defect type of the pixel area is defect d, and the defect d is a type of linear defect, the corresponding image to be predicted contains linear defects, and the computing device uses the method of openCV's minAreaRect function to find The minimum circumscribed rectangular frame of the linear defect is used as the output result of the prediction detection frame, and the pixel area is directly used as the pixel area of the linear defect.

It should be understood that the defect types corresponding to the block defects and the linear defects are known, so it can be determined whether the defect is a linear defect or a block defect according to different types of defects.

In the embodiment of the present application, two types of common defects in industrial inspection, block defects and linear defects, adopt different post-processing methods, so that the confidence of the detection result of the block defect is higher, and the confidence of the semantic segmentation result of the linear defect is higher.

FIG. 4 is a schematic flowchart of yet another defect detection method provided by another exemplary embodiment of the application. The method 400 provided in the embodiment of the present application is executed by a computing device, and the method 400 includes the following steps:

401: Obtain the feature of at least one picture, and determine the prediction type and the prediction detection frame of the pixel in the picture according to the feature, the prediction type reflects whether the pixel is a defect, and the prediction target frame reflects the defect position of the pixel when it has a defect.

402: Determine the type loss between the predicted type and the true type of the pixel, and determine the detection frame loss between the predicted detection frame and the true detection frame of the pixel.

403: Determine the total loss of pixels according to the loss of the pixel type and the loss of the detection frame, and generate a defect detection model based on the total loss.

404: Determine the prediction type and the prediction detection frame of the pixel in the picture to be predicted according to the detection model of the generated defect.

405: For a picture to be predicted, the pixels are aggregated according to the prediction type to generate a pixel area, and the prediction detection frame of the pixel area is determined according to the prediction detection frame of the pixel.

406: Determine the defect of the picture to be predicted according to the pixel area and/or the predicted detection frame of the pixel area.

It should be noted that the specific implementation of the steps in the method 400 provided in the foregoing embodiment has been described in detail above, and will not be repeated here.

FIG. 8 is a schematic structural diagram of a data processing device provided by another exemplary embodiment of this application. The data processing 800 can be applied to a computing device. The data processing 800 includes an acquisition module 801, a determination module 802, and a generation module 803. The functions of each module are described in detail below:

The obtaining module 801 is used to obtain the characteristics of at least one picture, and determine the prediction type and the prediction detection frame of the pixel in the picture according to the characteristics, the prediction type reflects whether the pixel is a defect, and the prediction target frame reflects the defect of the pixel when it has a defect position.

The determining module 802 is used to determine the type loss between the predicted type and the true type of the pixel, and the detection frame loss between the predicted detection frame and the true detection frame of the pixel;

The generating module 803 is used to determine the total loss of pixels according to the type loss of the pixels and the loss of the detection frame, and generate a defect detection model according to the total loss.

In some examples, the generating module 803 includes: a first determining unit for determining the type loss of the pixel according to the type loss of the pixel; and a second determining unit for determining the detection frame loss of the pixel according to the loss of the detection frame of the pixel和; The third determining unit is used to determine the total loss according to the type loss sum and the detection frame loss sum.

In some examples, the obtaining module 801 is used to obtain difficult negative samples from normal pixels that are not defective in the picture for any picture. The difficult negative samples refer to negative samples with predetermined quality; for any picture, The pixels belonging to the defect in the picture are regarded as positive samples.

The first determining unit is used to determine the type loss sum of the difficult negative samples in at least one picture; determine the type loss sum of the positive samples in at least one picture, and compare the type loss sum of the difficult negative samples with the positive sample The type loss sum is used as the type loss of the corresponding picture.

In some examples, the acquisition module 801 includes: a fourth determining unit, used to take normal pixels that are not defective in each picture as a negative sample, and determine the type loss of the negative sample; The type loss is sorted from large to small, and two negative samples corresponding to the largest loss of adjacent types are selected, the negative sample that is sorted later is used as the critical point, and the negative sample sorted before the critical point is used as the difficult negative sample.

In some examples, the acquisition module 801 is configured to process the features according to the fully convolutional neural network model, and determine the prediction type and the prediction detection frame of the pixel in the picture.

In some examples, the determining module 802 is configured to determine the type loss of the pixel under the true type according to the true type of the pixel and the predicted probability of the predicted type that matches the true type.

In some instances, the type loss between the predicted type and the true type of the pixel is determined by the following formula 1):

In some examples, the determining module 802 includes: a fifth determining unit, configured to determine the relative predicted coordinates of the predicted detection frame to the corresponding pixel according to the predicted coordinates of the predicted detection frame in the corresponding picture; an acquiring unit, configured to obtain the real The relative real coordinates of the detection frame to the corresponding pixels; the fifth determining unit is used to determine the coordinate distance between the relative predicted coordinates and the relative real coordinates; the fifth determining unit is used to determine the loss of the detection frame according to the coordinate distance.

In some instances, the detection frame loss is determined by formula 2):

In some instances, the generation module 803 is used to update the detection parameters of the detection model according to the total loss, and perform iterative training of the model until the iterative training stop condition is met, and then the detection model is generated.

In some examples, the generating module 803 is configured to determine the total loss of the sum of the type loss and the sum of the detection frame loss according to the weighted sum algorithm.

In some examples, the obtaining module 801 is configured to process features according to the fully convolutional neural network model to obtain the prediction probability of at least one prediction type of the pixel; select the prediction type with the largest prediction probability as the prediction type of the pixel; The prediction detection frame corresponding to the prediction type with the highest probability is used as the prediction detection frame of the pixel.

FIG. 9 is a schematic structural diagram of a defect detection device provided by another exemplary embodiment of the application. The detection device 900 can be applied to a computing device. The detection device 900 includes an acquisition module 901, a generation module 902, and a determination module 903. The functions of each module are described in detail below:

The obtaining module 901 is configured to obtain at least one picture to be predicted, and determine the prediction type and the prediction detection frame of the pixel in the picture to be predicted.

The generating module 902 is configured to aggregate pixels according to the prediction type for a picture to be predicted to generate a pixel area, and determine the predicted detection frame of the pixel area according to the predicted detection frame of the pixel.

The determining module 903 is configured to determine the defect of the picture to be predicted according to the pixel area and/or the predicted detection frame of the pixel area.

In some examples, the generating module 902 includes: a determining unit for determining the defect type of each pixel according to the predicted probability of the prediction type; a generating unit for aggregating pixels according to the defect type to generate an aggregation of pixels of the same defect type Area, as the pixel area.

In some examples, the device 900 further includes: a selection module for taking the same defect type as the defect type of the pixel area; wherein, the generating module 902 is used for selecting the pixel with the highest predicted probability of the defect type in the pixel area. The prediction detection frame is used as the prediction detection frame of the pixel area.

In some examples, the determining module 903 includes: a selection unit, configured to, when the defect type of the pixel area belongs to the first type, use the defect type of the pixel area as the defect type contained in the picture to be predicted, and detect the prediction of the pixel area The frame is used as the prediction detection frame of the defect contained in the picture to be predicted; the removal unit is used to remove some pixels from the pixel area when some pixels in the pixel area exceed the prediction detection frame, and use the area composed of the remaining pixels as the prediction detection frame The image contains the pixel area of the defect.

In some examples, the determining module 903 is configured to, when the defect type of the pixel area belongs to the second type, use the defect type of the pixel area as the defect type contained in the picture to be predicted, and the pixel area as the defect contained in the picture to be predicted In the pixel area of, the smallest bounding rectangular frame of the pixel area is used as the prediction detection frame of the defect contained in the picture to be predicted.

In some examples, the obtaining module 901 is configured to input at least one picture to be predicted into the generated defect detection model to obtain the prediction type and the prediction detection frame of the pixel in the picture to be predicted.

FIG. 10 is a schematic structural frame diagram of another defect detection device provided by another exemplary embodiment of the application. The detection device 1000 can be applied to a computing device. The detection device 1000 includes: an acquisition module 1001, a determination module 1002, and a generation module 1003. The functions of each module are described in detail below:

The obtaining module 1001 is used to obtain the characteristics of at least one picture, and determine the prediction type and the prediction detection frame of the pixel in the picture according to the characteristics, the prediction type reflects whether the pixel is a defect, and the prediction target frame reflects the defect of the pixel when it has a defect position.

The determining module 1002 is used to determine the type loss between the predicted type and the true type of the pixel, and the detection frame loss between the predicted detection frame and the true detection frame of the pixel.

The generating module 1003 is used to determine the total loss of pixels according to the loss of the pixel type and the loss of the detection frame, and generate a defect detection model according to the total loss.

The determining module 1002 is used to determine the prediction type and the prediction detection frame of the pixel in the picture to be predicted according to the detection model of the generated defect.

The generating module 1003 is configured to aggregate pixels according to the prediction type for a picture to be predicted to generate a pixel area, and determine the predicted detection frame of the pixel area according to the predicted detection frame of the pixel;

The determining module 1002 is used to determine the defect of the picture to be predicted according to the pixel area and/or the predicted detection frame of the pixel area.

The internal functions and structure of the data processing device 800 shown in FIG. 8 are described above. In a possible design, the structure of the data processing device 800 shown in FIG. 8 can be implemented as a computing device. As shown in FIG. 11, the computing device 1100 may include: a memory 1101 and a processor 1102;

The memory 1101 is used to store computer programs;

The processor 1102 is used to execute computer programs for:

Obtain the characteristics of at least one picture, determine the prediction type of the pixel in the picture and the prediction detection frame according to the characteristics, the prediction type reflects whether the pixel is a defect, and the prediction target frame reflects the defect location of the pixel when it has a defect; determine the prediction of the pixel The type loss between the type and the real type, and the detection frame loss between the predicted detection frame and the real detection frame of the determined pixel; according to the type loss of the pixel and the detection frame loss, the total loss of the pixel is determined, and the defect is generated according to the total loss The detection model.

In some examples, the processor 1102 is specifically configured to: determine the type loss sum of the pixel according to the type loss of the pixel; the second determining unit is configured to determine the detection frame loss sum of the pixel according to the detection frame loss of the pixel; third The determination unit is used to determine the total loss according to the type loss sum and the detection frame loss sum.

In some examples, the processor 1102 is specifically configured to: for any picture, obtain difficult negative samples from normal pixels that are not defective in the picture, where the difficult negative samples refer to negative samples with predetermined quality; for any picture , And take the defective pixels in the picture as positive samples.

The processor 1102 is specifically configured to: determine the type loss sum of the difficult negative samples in at least one picture; determine the type loss sum of the positive samples in at least one picture, and compare the type loss sum of the difficult negative samples with the positive samples The type loss sum, as the type loss of the corresponding picture.

In some examples, the processor 1102 is specifically configured to: take normal pixels that are not defective in each picture as negative samples, and determine the type loss of the negative samples; sort the type loss of the negative samples from large to small, and The two negative samples corresponding to the largest loss of adjacent types are selected, the negative sample that is sorted later is used as the critical point, and the negative sample that is sorted before the critical point is used as the difficult negative sample.

In some examples, the processor 1102 is specifically configured to process the features according to the fully convolutional neural network model, and determine the prediction type of the pixel in the picture and the prediction detection frame.

In some examples, the processor 1102 is specifically configured to determine the type loss of the pixel under the true type according to the true type of the pixel and the predicted probability of the prediction type that matches the true type.

In some examples, the processor 1102 is specifically configured to: determine the relative predicted coordinates of the predicted detection frame for the corresponding pixel according to the predicted coordinates of the predicted detection frame in the corresponding picture; obtain the relative real coordinates of the real detection frame for the corresponding pixel; Determine the coordinate distance between the relative predicted coordinate and the relative real coordinate; determine the detection frame loss according to the coordinate distance.

In some instances, the detection frame loss is determined by formula 2):

In some examples, the processor 1102 is specifically configured to: update the detection parameters of the detection model according to the total loss, and perform iterative training of the model until the iterative training stop condition is met, and then generate the detection model.

In some examples, the processor 1102 is specifically configured to determine the type loss sum and the total loss of the detection frame loss sum according to the weighted sum algorithm.

In some examples, the processor 1102 is specifically configured to: process features according to the full convolutional neural network model to obtain the prediction probability of at least one prediction type of the pixel; select the prediction type with the largest prediction probability as the prediction type of the pixel; The prediction detection frame corresponding to the prediction type with the largest prediction probability is used as the prediction detection frame of the pixel.

In addition, the embodiment of the present invention provides a computer storage medium. When a computer program is executed by one or more processors, the one or more processors are caused to implement the steps of the data processing method in the method embodiment in FIG. 2.

The internal functions and structure of the detection device 900 shown in FIG. 9 are described above. In a possible design, the structure of the detection device 900 shown in FIG. 9 can be implemented as a computing device. As shown in FIG. 12, the computing device 1200 It may include: a memory 1201 and a processor 1202;

The memory 1201 is used to store computer programs;

The processor 1202 is used to execute computer programs for:

Obtain at least one picture to be predicted, determine the prediction type of the pixel in the picture to be predicted and the prediction detection frame; for a picture to be predicted, aggregate pixels according to the prediction type to generate a pixel area, and determine the pixel according to the prediction detection frame of the pixel Predictive detection frame of the area: Determine the defect of the picture to be predicted according to the predicted detection frame of the pixel area and/or pixel area.

In some examples, the processor 1102 is specifically configured to: determine the defect type of each pixel according to the predicted probability of the prediction type; aggregate pixels according to the defect type, and generate an aggregated area of pixels of the same defect type as the pixel area.

In some instances, the processor 1102 is further configured to: use the same defect type as the defect type of the pixel area; wherein, the processor 1102 is specifically configured to: select the prediction probability of the pixel with the highest prediction probability of the defect type in the pixel area The detection frame serves as the prediction detection frame of the pixel area.

In some examples, the processor 1102 is specifically configured to: when the defect type of the pixel area belongs to the first type, use the defect type of the pixel area as the defect type contained in the picture to be predicted, and use the predicted detection frame of the pixel area as the defect type. The prediction detection frame of the defect contained in the picture to be predicted; when some pixels in the pixel area exceed the prediction detection frame, some pixels are removed from the pixel area, and the area composed of the remaining pixels is used as the pixel area of the defect contained in the picture to be predicted.

In some examples, the processor 1102 is specifically configured to: when the defect type of the pixel area belongs to the second type, use the defect type of the pixel area as the defect type contained in the picture to be predicted, and use the pixel area as the picture to be predicted. In the pixel area of the defect, the smallest bounding rectangle of the pixel area is used as the prediction detection frame of the defect contained in the picture to be predicted.

In some examples, the processor 1102 is specifically configured to: input at least one picture to be predicted into the generated defect detection model to obtain the prediction type and the prediction detection frame of the pixel in the picture to be predicted.

In addition, the embodiment of the present invention provides a computer storage medium. When the computer program is executed by one or more processors, the one or more processors are caused to implement the steps of the defect detection method in the method embodiment in FIG. 3.

The internal functions and structure of the apparatus 1000 shown in FIG. 10 are described above. In a possible design, the structure of the apparatus 1000 shown in FIG. 10 may be implemented as a computing device. As shown in FIG. 13, the computing device 1300 may include : Memory 1301 and processor 1302;

The memory 1301 is used to store computer programs;

The processor 1302 is used to execute a computer program for:

Memory, used to store computer programs;

The processor is used to execute computer programs for:

Obtain the characteristics of at least one picture, determine the prediction type of the pixel in the picture and the prediction detection frame according to the characteristics, the prediction type reflects whether the pixel is a defect, and the prediction target frame reflects the defect location of the pixel when it has a defect; determine the prediction of the pixel The type loss between the type and the real type, and the detection frame loss between the predicted detection frame and the real detection frame of the determined pixel; according to the type loss of the pixel and the detection frame loss, the total loss of the pixel is determined, and the defect is generated according to the total loss According to the detection model of the generated defect, determine the prediction type of the pixel in the picture to be predicted and the prediction detection frame; for a picture to be predicted, the pixels are aggregated according to the prediction type, and the pixel area is generated, and the detection frame is predicted according to the pixel , Determine the predicted detection frame of the pixel area; determine the defect of the picture to be predicted according to the predicted detection frame of the pixel area and/or the pixel area.

In addition, an embodiment of the present invention provides a computer storage medium. When a computer program is executed by one or more processors, the one or more processors are caused to implement the steps of the defect detection method in the method embodiment in FIG. 4.

In addition, in some of the processes described in the above-mentioned embodiments and drawings, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may be performed out of the order in which they appear in this document or performed in parallel The sequence numbers of operations, such as 201, 202, 203, etc., are only used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions of "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, nor do they limit the "first" and "second" Are different types.

The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the objectives of the solutions of the embodiments. Those of ordinary skill in the art can understand and implement without creative work.

Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course, it can also be implemented by a combination of hardware and software. Based on this understanding, the above technical solutions essentially or the part that contributes to the prior art can be embodied in the form of computer products, and the present invention can be used in one or more computer usable storage containing computer usable program codes. The form of a computer program product implemented on a medium (including but not limited to disk storage, CD-ROM, optical storage, etc.).

The present invention is described with reference to flowcharts and/or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and/or block in the flowchart and/or block diagram, and the combination of processes and/or blocks in the flowchart and/or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable multimedia data processing equipment to generate a machine, so that the instructions are executed by the processor of the computer or other programmable multimedia data processing equipment A device for realizing the functions specified in one flow or multiple flows in the flowchart and/or one block or multiple blocks in the block diagram is generated.

These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable multimedia data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device. The instruction device realizes the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

These computer program instructions can also be loaded on a computer or other programmable multimedia data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, which can be executed on the computer or other programmable equipment. The instructions provide steps for implementing functions specified in a flow or multiple flows in a flowchart and/or a block or multiple blocks in a block diagram.

In a typical configuration, the computing device includes one or more processors (CPU), input/output interfaces, network interfaces, and memory.

The memory may include non-permanent memory in a computer readable medium, random access memory (RAM) and/or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer readable media.

Computer-readable media includes permanent and non-permanent, removable and non-removable media, and information storage can be realized by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical storage, Magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media can be used to store information that can be accessed by computing devices. According to the definition in this article, computer-readable media does not include transitory media, such as modulated data signals and carrier waves.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions recorded in the foregoing embodiments are modified, or some of the technical features are equivalently replaced; these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

A data processing method, characterized by comprising:

Obtain the feature of at least one picture, determine the prediction type of the pixel in the picture and the prediction detection frame according to the feature, the prediction type reflects whether the pixel is a defect, and the prediction target frame reflects the defect of the pixel when it has a defect position;

Determine the type loss between the predicted type and the true type of the pixel, and determine the detection frame loss between the predicted detection frame and the true detection frame of the pixel;

According to the pixel type loss and the detection frame loss, the total loss of the pixel is determined, and the defect detection model is generated according to the total loss.
The method of claim 1, wherein determining the total loss of pixels according to the type loss of the pixels and the loss of the detection frame comprises:

According to the pixel type loss, determine the pixel type loss sum;

Determine the sum of the pixel detection frame loss according to the pixel detection frame loss;

According to the type loss sum and the detection frame loss sum, the total loss is determined.
The method of claim 2, wherein the method further comprises:

For any picture, obtain difficult negative samples from normal pixels that are not defective in the picture, and difficult negative samples refer to negative samples with predetermined quality;

Regarding any picture, use the defective pixels in the picture as a positive sample;

Wherein, determining the type loss sum of the pixel according to the type loss of the pixel includes:

Determine the type loss sum of the difficult negative samples in at least one picture;

Determine the type loss sum of the positive samples in at least one picture, and use the type loss sum of the difficult negative samples and the type loss sum of the positive samples as the type loss of the corresponding picture.
The method according to claim 3, wherein the obtaining difficult negative samples from normal pixels that are not defective in the picture comprises:

Taking normal pixels that are not defective in each picture as a negative sample, and determining the type loss of the negative sample;

Sort the type loss of the negative samples from large to small, and select the two negative samples corresponding to the largest difference in loss between adjacent types, take the negative samples that are ranked later as the critical point, and sort the negative samples before the critical point. The sample serves as the difficult negative sample.
The method according to claim 1, wherein the determining the prediction type and the prediction detection frame of the pixel in the picture according to the characteristic comprises:

According to the fully convolutional neural network model, the feature is processed to determine the prediction type and the prediction detection frame of the pixel in the picture.
The method according to claim 1, wherein the determining the type loss between the predicted type and the true type of the pixel comprises:

According to the true type of the pixel and the predicted probability of the predicted type consistent with the true type, the type loss of the pixel under the true type is determined.
The method according to claim 1, wherein the detection frame loss between the predicted detection frame and the real detection frame of the determined pixel comprises:

Determine the relative prediction coordinates of the prediction detection frame for the corresponding pixels according to the prediction coordinates of the prediction detection frame in the corresponding picture;

Obtain the relative real coordinates of the real detection frame to the corresponding pixels;

Determining the coordinate distance between the relative predicted coordinate and the relative real coordinate;

The loss of the detection frame is determined according to the coordinate distance.
The method according to claim 1, wherein the generating a defect detection model according to the total loss comprises:

According to the total loss, the detection parameters of the detection model are updated, and the iterative training of the model is performed until the iterative training stop condition is met, and the detection model is generated.
The method according to claim 2, wherein the determining the total loss according to the type loss sum and the detection frame loss sum comprises:

According to a weighted sum algorithm, the total loss of the sum of the type loss and the sum of the detection frame loss is determined.
The method according to claim 5, wherein the processing the feature according to the fully convolutional neural network model to determine the prediction type and the prediction detection frame of the pixel in the picture comprises:

According to the fully convolutional neural network model, processing the feature to obtain the prediction probability of at least one prediction type of the pixel;

Select the prediction type with the largest prediction probability as the prediction type of the pixel;

The prediction detection frame corresponding to the prediction type with the largest prediction probability is used as the prediction detection frame of the pixel.
A defect detection method, which is characterized in that it includes:

Obtain at least one picture to be predicted, and determine the prediction type and the prediction detection frame of the pixel in the picture to be predicted;

For a picture to be predicted, aggregate pixels according to the prediction type to generate a pixel area, and determine the predicted detection frame of the pixel area according to the predicted detection frame of the pixel;

According to the pixel area and/or the predicted detection frame of the pixel area, the defect of the picture to be predicted is determined.
The method according to claim 11, wherein the aggregating pixels according to the prediction type to generate a pixel area comprises:

Determine the defect type of each pixel according to the prediction probability of the prediction type;

According to the defect type, the pixels are aggregated to generate an aggregated area of the pixels of the same defect type as the pixel area.
The method of claim 12, wherein the method further comprises:

Taking the same defect type as the defect type of the pixel area;

Wherein, the determining the predicted detection frame of the pixel area according to the predicted detection frame of the pixel includes:

The predicted detection frame of the pixel with the highest predicted probability of the defect type in the pixel area is selected as the predicted detection frame of the pixel area.
The method according to claim 11, wherein the determining the defect of the picture to be predicted according to the pixel area and/or the prediction detection frame of the pixel area comprises:

When the defect type of the pixel area belongs to the first type, use the defect type of the pixel area as the defect type contained in the picture to be predicted, and use the prediction detection frame of the pixel area as the prediction detection frame of the defect contained in the picture to be predicted;

When part of the pixels in the pixel area exceeds the prediction detection frame, the part of pixels is removed from the pixel area, and the area composed of the remaining pixels is taken as the pixel area of the defect contained in the picture to be predicted.
The method according to claim 11, wherein the determining the defect of the picture to be predicted according to the pixel area and/or the prediction detection frame of the pixel area comprises:

When the defect type of the pixel area belongs to the second type, the defect type of the pixel area is taken as the defect type contained in the picture to be predicted, the pixel area is taken as the pixel area of the defect contained in the picture to be predicted, and the smallest pixel area is The circumscribed rectangular frame is used as the prediction detection frame of the defect contained in the picture to be predicted.
The method according to claim 11, wherein the determining the prediction type and the prediction detection frame of the pixel in the picture to be predicted comprises:

Input at least one picture to be predicted into the generated defect detection model to obtain the prediction type and the prediction detection frame of the pixel in the picture to be predicted.
A defect detection method, which is characterized in that it includes:

Obtain the feature of at least one picture, determine the prediction type of the pixel in the picture and the prediction detection frame according to the feature, the prediction type reflects whether the pixel is a defect, and the prediction target frame reflects the defect of the pixel when it has a defect position;

Determine the type loss between the predicted type and the true type of the pixel, and determine the detection frame loss between the predicted detection frame and the true detection frame of the pixel;

Determine the total loss of pixels according to the type loss of the pixels and the loss of the detection frame, and generate a defect detection model according to the total loss;

According to the detection model of the generated defect, determine the prediction type of the pixel in the picture to be predicted and the prediction detection frame;

For a picture to be predicted, aggregate pixels according to the prediction type to generate a pixel area, and determine the predicted detection frame of the pixel area according to the predicted detection frame of the pixel;

According to the pixel area and/or the predicted detection frame of the pixel area, the defect of the picture to be predicted is determined.
A defect detection system, characterized by comprising: a first computing device and a second computing device;

The first computing device obtains the characteristics of at least one picture, and determines the prediction type and the prediction detection frame of the pixel in the picture according to the characteristics, the prediction type reflects whether the pixel is a defect, and the prediction target frame reflects The defect position of the pixel when there is a defect;

Determine the type loss between the predicted type and the true type of the pixel, and determine the detection frame loss between the predicted detection frame and the true detection frame of the pixel;

Determine the total loss of pixels according to the type loss of the pixels and the loss of the detection frame, and generate a defect detection model according to the total loss;

The second computing device determines the prediction type and the prediction detection frame of the pixel in the picture to be predicted according to the detection model of the generated defect;

For a picture to be predicted, aggregate pixels according to the prediction type to generate a pixel area, and determine the predicted detection frame of the pixel area according to the predicted detection frame of the pixel;

According to the pixel area and/or the predicted detection frame of the pixel area, the defect of the picture to be predicted is determined.
A computing device, characterized by comprising a memory and a processor;

The memory is used to store computer programs;

The processor is configured to execute the computer program for:

Obtain the feature of at least one picture, determine the prediction type of the pixel in the picture and the prediction detection frame according to the feature, the prediction type reflects whether the pixel is a defect, and the prediction target frame reflects the defect of the pixel when it has a defect position;

Determine the type loss between the predicted type and the true type of the pixel, and determine the detection frame loss between the predicted detection frame and the true detection frame of the pixel;

According to the pixel type loss and the detection frame loss, the total loss of the pixel is determined, and the defect detection model is generated according to the total loss.
A computer-readable storage medium storing a computer program, wherein when the computer program is executed by one or more processors, it causes the one or more processors to implement the method of any one of claims 1-10 Steps in.
A computing device, characterized by comprising a memory and a processor;

The memory is used to store computer programs;

The processor is configured to execute the computer program for:

Obtain at least one picture to be predicted, and determine the prediction type and the prediction detection frame of the pixel in the picture to be predicted;

For a picture to be predicted, aggregate pixels according to the prediction type to generate a pixel area, and determine the predicted detection frame of the pixel area according to the predicted detection frame of the pixel;

According to the pixel area and/or the predicted detection frame of the pixel area, the defect of the picture to be predicted is determined.
A computer-readable storage medium storing a computer program, wherein when the computer program is executed by one or more processors, it causes the one or more processors to implement the method described in any one of claims 11-16 Steps in.
A computing device, characterized by comprising a memory and a processor;

The memory is used to store computer programs;

The processor is configured to execute the computer program for:

Obtain the feature of at least one picture, determine the prediction type of the pixel in the picture and the prediction detection frame according to the feature, the prediction type reflects whether the pixel is a defect, and the prediction target frame reflects the defect of the pixel when it has a defect position;

Determine the type loss between the predicted type and the true type of the pixel, and determine the detection frame loss between the predicted detection frame and the true detection frame of the pixel;

Determine the total loss of pixels according to the type loss of the pixels and the loss of the detection frame, and generate a defect detection model according to the total loss;

According to the detection model of the generated defect, determine the prediction type of the pixel in the picture to be predicted and the prediction detection frame;

For a picture to be predicted, aggregate pixels according to the prediction type to generate a pixel area, and determine the predicted detection frame of the pixel area according to the predicted detection frame of the pixel;

According to the pixel area and/or the predicted detection frame of the pixel area, the defect of the picture to be predicted is determined.
A computer-readable storage medium storing a computer program, wherein the computer program is executed by one or more processors to cause the one or more processors to implement the steps in the method of claim 17.