CN114299066A

CN114299066A - Defect detection method and device based on salient feature pre-extraction and image segmentation

Info

Publication number: CN114299066A
Application number: CN202210200679.3A
Authority: CN
Inventors: 黄必清; 徐荣阁
Original assignee: Tsinghua University
Current assignee: Tsinghua University
Priority date: 2022-03-03
Filing date: 2022-03-03
Publication date: 2022-04-08
Anticipated expiration: 2042-03-03
Also published as: CN114299066B

Abstract

The application relates to a method, a device, a computer device, a storage medium and a computer program product for defect detection based on salient feature pre-extraction and image segmentation. The method comprises the following steps: and acquiring an image to be detected, processing the image to be detected, segmenting the gray change characteristic map according to an image segmentation threshold, and screening pixel points contained in the image to be detected for multiple times to obtain a target defect significance characteristic map and a defect subgraph of the image to be detected. Therefore, the data are input into a pixel point defect probability prediction model which is trained in advance, and a defect detection result image of the image to be detected is obtained. The method provided by the invention realizes the preliminary screening of the high-resolution image by preprocessing and extracting the significant characteristics of the image to be detected on the glass surface, extracts effective defect characteristics while improving the detection efficiency, assists the detection process and ensures the high fineness and the high efficiency of the detection of the defects on the glass surface.

Description

Defect detection method and device based on salient feature pre-extraction and image segmentation

Technical Field

The application relates to the technical field of glass detection, in particular to a defect detection method and device based on saliency feature pre-extraction and image segmentation.

Background

With the rapid development of advanced manufacturing technology and the continuous improvement of the living standard of people, the demand of the society on product diversification and individuation is urgent day by day, and the requirements on the production efficiency and the quality of the product are continuously upgraded. Because the manufacturing industry has strict requirements on the surface quality of products, and any tiny defects, impurities, deformation and the like can have important influence on the aesthetic property and even the safety of the products, the effective detection of the surface quality of the products is a problem to be solved urgently.

In the related art, the product image is analyzed through an artificial intelligence algorithm, and then the glass surface defects are positioned and classified to form a novel solution. However, since the image detection algorithm in the related art has a limit in its own calculation capability, it is impossible to detect defects in a high-resolution image of the surface of a glass product.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a defect detection method, apparatus, computer device, computer readable storage medium and computer program product based on salient feature pre-extraction and image segmentation, which can accurately detect defect location points in high-resolution images.

In a first aspect, the present application provides a defect detection method based on salient feature pre-extraction and image segmentation. The method comprises the following steps:

the method comprises the steps of obtaining an image to be detected, processing the image to be detected to obtain a gray level change characteristic diagram of the image to be detected, wherein the gray level change characteristic diagram comprises pixel values of all pixel points in the image to be detected, and the image to be detected is a high-resolution image of the glass surface.

Carrying out mean value calculation on pixel values corresponding to the pixel points meeting the preset threshold value screening condition to obtain an image segmentation threshold value;

extracting a region corresponding to a pixel point of which the pixel value is greater than or equal to the image segmentation threshold value in the gray level change characteristic map to obtain a target defect significance characteristic map of the image to be detected;

according to the coordinates of the target defect saliency characteristic graph, the image to be detected is cut to obtain a defect subgraph corresponding to the target defect saliency characteristic graph;

and inputting the target defect significance characteristic graph and the defect subgraph into a pixel point defect probability prediction model trained in advance to obtain a defect detection result image of the image to be detected.

In one embodiment, the extracting a region corresponding to a pixel point whose pixel value is greater than or equal to the image segmentation threshold value in the gray-scale change feature map to obtain a target defect saliency feature map of the image to be detected includes:

determining the pixel value of a pixel point of which the pixel value is smaller than an image segmentation threshold value in the gray level change characteristic image as a target value to obtain a first defect significance characteristic image corresponding to the image to be detected;

according to a preset morphological open operation algorithm, carrying out noise point removal processing on the first defect significance characteristic diagram to obtain a second defect significance characteristic diagram of the image to be detected;

and removing the region corresponding to the pixel point with the pixel value as the target value in the second defect significance characteristic diagram to obtain the target defect significance characteristic diagram of the image to be detected.

In one embodiment, before the step of inputting the target defect saliency feature map and the defect subgraph into a pixel defect probability prediction model trained in advance, the method further includes:

and performing image enhancement processing on the defect subgraph to obtain the processed defect subgraph, wherein the image enhancement processing comprises one or more of image random rotation processing, image displacement processing, image scaling processing, image shearing processing and image turning processing.

In one embodiment, the defect detection result image of the image to be detected includes defect probability values of a plurality of pixel points in the image to be detected, and the method further includes:

and carrying out binarization processing on the plurality of pixel points in the image to be detected according to a preset binarization segmentation threshold value and defect probability values of the plurality of pixel points in the image to be detected to obtain a defect image of the image to be detected.

In one embodiment, the method further comprises:

acquiring training data, wherein the training data comprises a sample defect significance characteristic diagram of a sample image, a sample defect subgraph and a sample defect detection result image;

inputting the sample defect significance characteristic graph and the sample defect subgraph into a pixel point defect probability prediction model to be trained to obtain a predicted defect detection result image;

calculating a loss value according to a sample defect probability value of a plurality of pixel points contained in the sample defect detection result image and a predicted defect probability value of a plurality of pixel points contained in the predicted defect detection result image by a preset loss function;

and updating the network parameters of the pixel defect probability prediction model to be trained according to the loss value, and returning to the step of executing the training data acquisition until the loss value meets the preset training completion condition to obtain the trained pixel defect probability prediction model.

In one embodiment, the inputting the sample defect saliency feature map and the sample defect subgraph into a pixel defect probability prediction model to be trained to obtain a predicted defect detection result image includes:

performing channel cascade processing on the sample defect significance characteristic graph and the sample defect subgraph to obtain a sample spliced image;

extracting the feature vectors of the sample spliced image through a first number of encoders to obtain a feature map of the sample image;

and performing convolution operation on the feature maps of the sample images through a second number of decoders to obtain the prediction defect probability value of each pixel point in the sample images, and combining to obtain the prediction defect detection result images of the sample images.

In a second aspect, the present application further provides a defect detection apparatus based on salient feature pre-extraction and image segmentation. The device comprises:

the acquisition module is used for acquiring an image to be detected and processing the image to be detected to obtain a gray level change characteristic diagram of the image to be detected, the gray level change characteristic diagram contains pixel values of all pixel points in the image to be detected, and the image to be detected is a high-resolution image of the glass surface.

The calculation module is used for carrying out mean value calculation on pixel values corresponding to the pixel points meeting the preset threshold value screening condition to obtain an image segmentation threshold value;

the extraction module is used for extracting a region corresponding to a pixel point with a pixel value larger than or equal to the image segmentation threshold value in the gray scale change characteristic map to obtain a target defect significance characteristic map of the image to be detected;

the cutting module is used for cutting the image to be detected according to the coordinates of the target defect saliency characteristic graph to obtain a defect subgraph corresponding to the target defect saliency characteristic graph;

and the detection module is used for inputting the target defect significance characteristic graph and the defect subgraph into a pixel point defect probability prediction model trained in advance to obtain a defect detection result image of the image to be detected.

In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the following steps when executing the computer program:

In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of:

In a fifth aspect, the present application further provides a computer program product. The computer program product comprising a computer program which when executed by a processor performs the steps of:

The defect detection method, apparatus, computer device, storage medium and computer program product based on salient feature pre-extraction and image segmentation described above, the method comprising: and acquiring an image to be detected, processing the image to be detected, segmenting the gray change characteristic map according to an image segmentation threshold, and screening pixel points contained in the image to be detected for multiple times to obtain a target defect significance characteristic map and a defect subgraph of the image to be detected. Therefore, the data are input into a pixel point defect probability prediction model which is trained in advance, and a defect detection result image of the image to be detected is obtained. The method provided by the invention realizes the preliminary screening of the high-resolution image by preprocessing and extracting the significant characteristics of the image to be detected on the glass surface, extracts effective defect characteristics while improving the detection efficiency, assists the detection process and ensures the high fineness and the high efficiency of the detection of the defects on the glass surface.

Drawings

FIG. 1 is a schematic flow chart illustrating a defect detection method based on salient feature pre-extraction and image segmentation in one embodiment;

FIG. 2 is a statistical histogram of gray pixel values in one embodiment;

FIG. 3 is a flowchart illustrating the steps of determining a target defect saliency map in one embodiment;

FIG. 4 is a schematic flow chart of the training step in one embodiment;

FIG. 5 is a schematic flow chart diagram illustrating the steps of the model internal computation in one embodiment;

FIG. 6 is a schematic diagram of an encoder in one embodiment;

FIG. 7 is a block diagram of a decoder in one embodiment;

FIG. 8 is a flowchart illustrating a defect detection method based on salient feature pre-extraction and image segmentation in another embodiment;

FIG. 9A is a diagram illustrating an input image of a defect detection method based on salient feature pre-extraction and image segmentation in one embodiment;

FIG. 9B is a diagram illustrating an output image of a defect detection method based on salient feature pre-extraction and image segmentation, according to an embodiment;

FIG. 10 is a block diagram of a defect detection apparatus based on salient feature pre-extraction and image segmentation in one embodiment;

FIG. 11 is a block diagram of a defect detection apparatus based on salient feature pre-extraction and image segmentation in another embodiment;

FIG. 12 is a diagram illustrating an internal structure of a computer device according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

With the rapid development of advanced manufacturing technology and the continuous improvement of the living standard of people, the demand of the society on product diversification and individuation is urgent day by day, the requirements on the production efficiency and the quality of the product are continuously upgraded, and the production management mode of the traditional manufacturing industry faces huge challenges. With the development of science and technology, the Chinese manufacturing industry meets continuous development opportunities, and intelligent manufacturing becomes an important support way for the competitiveness improvement of the Chinese equipment manufacturing industry and the upgrading and reconstruction of the traditional manufacturing industry. The intelligent detection technology aims to detect the quality of a product in the manufacturing process through an intelligent algorithm, and simultaneously record and feed back data, so that powerful support is provided for the intelligent decision of subsequent links such as product sorting, repair and testing. The manufacturing industry has strict requirements on the product quality, particularly the surface quality, and any tiny defects, impurities, deformation and the like can have important influence on the product aesthetic property and even the safety, so that the effective detection on the product quality is a problem to be solved urgently. Meanwhile, the application of intelligent technology to replace traditional manual work is also a necessary trend for realizing efficient and accurate detection. Therefore, for the visual detection module in intelligent manufacturing, the core value of the visual detection module provides technical support for high-precision manufacturing industry, the intelligent detection of manufacturing defects which is driven by big data and takes an artificial intelligence technology as the core is realized, the labor force is effectively saved, and meanwhile, the production efficiency and the product quality are improved.

The surface defects of the product are directly related to the appearance of the product, and the influence on the quality of the product is self-evident. However, due to the influence of the manufacturing environment and the process flow, typical surface defects such as stains, cracks, collisions, scratches, irregular shapes and the like are frequently found on various production lines and are one of the defect types with the highest occurrence rate in the manufacturing process. The detection of the surface defects of products in the traditional manufacturing industry is usually finished manually, and some high-precision products such as glass, films and the like need technical personnel to be equipped with a professional detection device, so that the consumption of human resources is huge, and the automatic detection and the intelligent detection of the surface defects are always urgent requirements of the traditional manufacturing industry. Under the background of big data era and intelligent manufacturing, the artificial intelligence algorithm is used for analyzing the product images of glass and the like, and further positioning and classifying the surface defects become a novel solution.

The existing image detection algorithm can not realize high-efficiency and accurate detection aiming at the defects of the glass surface, and is particularly difficult when the image resolution is very high.

In an embodiment, as shown in fig. 1, a defect detection method based on salient feature pre-extraction and image segmentation is provided, and this embodiment is illustrated by applying this method to a terminal, it can be understood that this method may also be applied to a server, and may also be applied to a system including a terminal and a server, and is implemented by interaction between the terminal and the server, where the terminal may be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, internet of things devices, and portable wearable devices, and the internet of things devices may be smart speakers, smart televisions, smart air conditioners, smart car-mounted devices, and the like. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device and the like, and the server can be realized by an independent server or a server cluster formed by a plurality of servers. In this embodiment, the defect detection method based on salient feature pre-extraction and image segmentation includes the following steps:

and 102, acquiring an image to be detected, and processing the image to be detected to obtain a gray level change characteristic diagram of the image to be detected.

The image to be detected comprises original pixel values of a plurality of pixel points in the image of the object to be detected, and can be represented by I; the processing of the image to be detected can be mean filtering processing of the image to be detected; the mean filtering may be two-dimensional gaussian mean filtering; the gray-scale change characteristic diagram comprises pixel values of all pixel points in the image to be detected after mean value filtering, the pixel values are pixel gray values, the object to be detected can be glass, and the image to be detected can be a high-resolution image of the surface of the glass.

In this embodiment, the terminal obtains an image to be detected of an object to be detected, and performs two-dimensional gaussian mean filtering on a pixel value of each pixel point included in the image to be detected to obtain a mean filtered image. In this way, the terminal performs difference processing on the image to be detected and the mean value filtering image, and calculates the gray level change characteristic diagram of the image to be detected. In an actual application scenario, the gray scale change characteristic map corresponding to the detected image can be calculated by the following formula:

F=I-G(I)，

wherein the content of the first and second substances,Fis a characteristic diagram of the gray-scale variation,Iin order to detect the image to be detected,Gfor mean filtering (e.g., two-dimensional gaussian filtering),G (I)the method is to obtain a mean value filtering image after mean value filtering is carried out on an image to be detected.

And 104, performing mean value calculation on pixel values corresponding to the pixel points meeting the preset threshold value screening condition to obtain an image segmentation threshold value.

The preset threshold screening condition may be that, within pixel values of a plurality of pixels included in the image to be detected, a first number of pixels having a largest pixel value, for example, the first number may be ten percent of the number of pixels included in the image to be detected, the number of pixels included in the image to be detected may be 100, and then the corresponding first number may be 10.

In this embodiment, the terminal screens each pixel point included in the gray-scale change characteristic map according to a preset threshold screening condition, and extracts a first number of pixel points that satisfy the preset threshold screening condition. In this way, the terminal may perform an average processing on the first number of pixel values of the first number of pixel points to obtain an image segmentation threshold.

In one example, the terminal counts the pixel points of which the pixel values are within 0-255 in the gray scale change characteristic image, and the statistical result is a gray scale histogram which can represent the defect significance characteristics of the image to be detected. The histogram in a specific implementation may be as shown in fig. 2, where an abscissa (Intensity) of the histogram represents a pixel gray-scale value, and an ordinate (N) represents the number of pixels. The terminal can extract the first ten percent of pixels with the maximum pixel gray value from the pixel point set with the pixel gray value not being zero in the gray variation characteristic image according to the statistical information represented by the gray histogram, the pixels serve as the pixels meeting the preset threshold screening condition, the pixel values of the pixels meeting the preset threshold screening condition are subjected to mean value calculation, and the image segmentation threshold is determined.

And 106, extracting the corresponding region of the pixel point with the pixel value larger than or equal to the image segmentation threshold value in the gray level change characteristic map to obtain a target defect significance characteristic map of the image to be detected.

Specifically, the terminal can segment the gray-scale change feature map based on the image segmentation threshold to obtain a target defect significance feature map of the image to be detected. The specific segmentation process may be: and traversing the pixel values of all the pixel points contained in the gray scale change characteristic map by the terminal, and setting the pixel values of the pixel points with the pixel values smaller than the image segmentation threshold value to be zero (zero gray value) to obtain a first defect significance characteristic map. In this way, the terminal may traverse the pixel values of the pixels included in the first defect saliency feature map again, extract the regions corresponding to the pixels whose pixel values are not zero, or extract the regions corresponding to the pixels whose pixel values are greater than or equal to the image segmentation threshold, to obtain the target defect saliency feature map of the image to be detected.

And 108, cutting the image to be detected according to the coordinates of the target defect saliency characteristic graph to obtain a defect subgraph corresponding to the target defect saliency characteristic graph.

Specifically, the terminal determines a position to be cut on the image to be detected according to coordinates of each pixel point contained in the target defect significance characteristic diagram in the image to be detected, and cuts the image to be detected according to the position to be cut to obtain a defect subgraph corresponding to the target defect significance characteristic diagram. The defect subgraph is a partial image of the image to be detected, wherein defects can exist in the partial image. In the case that the image to be detected is a high-resolution glass surface image, the defect subgraph represents a partial image which may contain defects in the high-resolution glass surface image.

And 110, inputting the target defect significance characteristic graph and the defect subgraph into a pixel point defect probability prediction model trained in advance to obtain a defect detection result image of the image to be detected.

Specifically, the pre-trained pixel defect probability prediction model is used for predicting the defect probability of a plurality of pixels contained in the image to be detected according to the target defect significance characteristic graph and the defect subgraph to obtain the defect probability of each pixel contained in the image to be detected. Therefore, the terminal can take the image containing the defect probability of each pixel point as a defect detection result image of the image to be detected.

In the defect detection method based on the salient feature pre-extraction and the image segmentation, an image to be detected is obtained and processed, a gray change feature map is segmented according to an image segmentation threshold, and pixel points contained in the image to be detected are screened for multiple times to obtain a target defect salient feature map and a defect subgraph of the image to be detected. Therefore, the data are input into a pixel point defect probability prediction model which is trained in advance, and a defect detection result image of the image to be detected is obtained. The defect detection method provided by the invention realizes the preliminary screening of the high-resolution image by preprocessing the image to be detected and extracting the significant characteristics, improves the detection efficiency, extracts the effective defect characteristics, assists the detection process and ensures the high fineness and the high efficiency of the detection of the defects on the surface of the glass. Therefore, the method provided by the invention can classify the product image pixel by pixel while ensuring the detection time efficiency, and accurately segment the defect area to meet the requirement of industrial automatic detection. The high-precision defect positioning can meet the quality detection requirement of high-quality products, and has great significance for subsequent links of product quality control, intelligent sorting, product repair and the like in the intelligent manufacturing process.

In one embodiment, as shown in fig. 3, the specific processing procedure of step 108 "extracting a region corresponding to a pixel point whose pixel value is greater than or equal to an image segmentation threshold value in a gray-scale change feature map to obtain a target defect saliency feature map of an image to be detected" includes:

step 202, determining the pixel value of the pixel point with the pixel value smaller than the image segmentation threshold value in the gray scale change characteristic image as a target value, and obtaining a first defect significance characteristic image corresponding to the image to be detected.

The target value may be used to distinguish whether the pixel value of the pixel point is greater than or equal to the image segmentation threshold or whether the pixel value of the pixel point is smaller than the image segmentation threshold.

Specifically, the terminal traverses pixel values of all pixel points included in the gray scale change feature map, and sets the pixel value of the pixel point of which the pixel value is smaller than the image segmentation threshold as a target value to obtain a first defect saliency feature map. For example, the target value may be zero (zero gray value). In this way, the pixel value of each pixel included in the first defect saliency map includes the target value, is greater than the image segmentation threshold, and is equal to the image segmentation threshold.

And 204, performing noise point removal processing on the first defect significance characteristic diagram according to a preset morphological open operation algorithm to obtain a second defect significance characteristic diagram of the image to be detected.

Specifically, for the first defect saliency feature map, the terminal may perform noise removal processing, so as to further improve accuracy and precision of image detection. The specific process of the terminal performing noise removal processing may include: and carrying out corrosion treatment and expansion treatment on the first defect significance characteristic diagram through a preset morphological open operation algorithm to obtain a second defect significance characteristic diagram.

And step 206, removing the area corresponding to the pixel point with the pixel value as the target value in the second defect saliency characteristic map to obtain the target defect saliency characteristic map of the image to be detected.

Specifically, the terminal may traverse the pixel values of the pixel points included in the second defect saliency feature map again, extract the regions corresponding to the pixel points whose pixel values are not zero, that is, extract the regions corresponding to the pixel points whose pixel values are greater than or equal to the image segmentation threshold, and obtain the target defect saliency feature map of the image to be detected.

In this embodiment, the target defect saliency characteristic map is obtained by removing the gray level change characteristic map, and the accuracy and precision of image detection can be improved on the premise of ensuring the detection efficiency.

In one embodiment, before the step 110 "inputting the target defect saliency feature map and the defect subgraph into the pixel defect probability prediction model trained in advance", the defect detection method based on the saliency feature pre-extraction and the image segmentation further includes:

and carrying out image enhancement processing on the defect subgraph to obtain the processed defect subgraph.

The image enhancement processing comprises one or more of image random rotation processing, image displacement processing, image scaling processing, image shearing processing and image turning processing, and the processing modes do not distinguish the execution sequence.

In an example, the process of randomly rotating the image of the defect subgraph by the terminal may be that the terminal determines a target rotation angle according to a random strategy in an angle set, the angle set includes all angle values between 0 degree and 360 degrees, and the target rotation angle is any angle value between 0 degree and 360 degrees. Therefore, the terminal can rotate the defect subgraph by the target rotation angle to obtain the processed defect subgraph.

The process of the terminal performing the image displacement processing on the defect subgraph may be that the terminal first obtains the size information of the defect subgraph, including the length value in the horizontal direction and the length value in the vertical direction, so that the terminal can calculate the displacement distance according to the size information, and then perform the displacement processing on the defect subgraph in the horizontal direction or the vertical direction according to the displacement distance. The displacement distance may be a ten percent horizontal length value or a ten percent vertical length value. For example, the horizontal displacement distance or the vertical displacement distance is in the range of 0% to 10% of the image size.

The process of the terminal performing image scaling on the defect subgraph may be that the terminal first obtains size information of the defect subgraph, including a length value in a horizontal direction and a length value in a vertical direction, so that the terminal may perform enlargement processing or reduction processing on the defect subgraph. The terminal can perform amplification processing or reduction processing on the defect subgraph within a preset range, wherein the preset range is 0% to 5% of size information of the defect subgraph.

The process of the terminal performing the image clipping processing on the defect subgraph may be that the terminal first obtains the size information of the defect subgraph, including the length value in the horizontal direction and the length value in the vertical direction, so that the terminal may clip the defect subgraph. The terminal can cut the defect subgraph within a preset range, wherein the preset range is 0% to 5% of the size information of the defect subgraph.

The process of the terminal for turning over the image of the defect subgraph can be that the terminal turns over the defect subgraph in the horizontal direction or in the vertical direction according to a preset random strategy. In another example, the terminal may randomly perform horizontal translation or vertical translation within a certain coordinate range on the defect subgraph; randomly rotating an input image within a certain angle range; carrying out random zooming within a certain magnification range on an input image; the input image is randomly flipped horizontally or vertically.

In this embodiment, by performing image enhancement processing on the defective subgraph, parameters of the image to be detected can be further adjusted (reduced), and the image detection efficiency is improved.

In one embodiment, the defect detection result image of the image to be detected includes defect probability values of a plurality of pixel points in the image to be detected. Correspondingly, the defect detection method based on the salient feature pre-extraction and the image segmentation further comprises the following steps:

Specifically, the pre-trained pixel defect probability prediction model is used for predicting the defect probability of a plurality of pixels contained in the image to be detected according to the target defect significance characteristic diagram and the defect subgraph to obtain the defect probability of each pixel contained in the image to be detected and obtain a defect detection result image of the image to be detected. Therefore, the terminal can carry out binarization processing according to the defect probability of each pixel point contained in the defect detection result image of the image to be detected and a preset probability threshold value to obtain the defect image of the image to be detected. The specific process of binarization may be: and the terminal carries out condition judgment on the defect probability of each pixel point, takes the pixel points with the defect probability greater than or equal to a preset probability threshold as defect pixel points, and takes the pixel points with the defect probability less than the preset probability threshold as normal pixel points. Therefore, the terminal can take the image formed by each defective pixel point as the defect detection result image of the image to be detected.

In an example, the preset probability threshold may be 0.5, or other parameter values determined according to an actual application scenario, or a preset probability threshold obtained in response to an input operation of a user in the input operation.

In one embodiment, as shown in fig. 4, the defect detection method based on salient feature pre-extraction and image segmentation further includes:

step 302, training data is obtained.

The training data comprises a sample defect significance characteristic graph of a sample image, a sample defect subgraph and a sample defect detection result image. The training data includes a plurality of sample images, which may be, for example, a plurality of grayscale images of the high-resolution glass surface, a sample defect subgraph of each grayscale image, and a sample defect detection result image thereof.

And 304, inputting the sample defect significance characteristic graph and the sample defect subgraph into a pixel point defect probability prediction model to be trained to obtain a predicted defect detection result image.

The pixel defect probability prediction model to be trained can be a semantic segmentation model based on deep learning, and the model comprises an encoder module and a decoder module. The encoder module can comprise a plurality of encoders, and each encoder comprises a stride convolution layer, a Batch Normalization layer and an activation function layer; similarly, the decoder module may include a plurality of channel cascade units and a plurality of decoders, each of the decoders includes a convolutional layer, a Batch Normalization layer, an activation function layer, and an upsampling layer, an upsampling method used in the upsampling layer may be bilinear interpolation, an input size of each encoder is different from an input size, and an input size of each decoder is different from an input size.

Specifically, a terminal acquires a sample image in training data, processes the sample image to obtain a sample defect saliency feature map and a sample defect subgraph corresponding to the sample image, performs channel cascade processing on the sample image, inputs the obtained multichannel image with defect features into a first encoder in an encoder module, calculates the feature map according to the multichannel image with defect features through learnable convolution kernel in the first encoder, and each pixel position in the feature map is represented by a feature vector; for the ith encoder, the input is a profile of the output of the (i-1) th encoder.

The terminal inputs the feature map output by the last encoder into a first decoder in a decoder module, the output of the first decoder is the decoded feature map, the input of the ith decoder is the output of the (i-1) th decoder and the result (spliced according to the feature channels) of the feature map of the corresponding scale encoder after channel cascade processing, the input of the last decoder is the output of the previous decoder, and the output of the last decoder is the predicted defect probability of each pixel point position of the sample image, namely the single-channel score map.

And step 306, calculating a loss value according to the sample defect probability values of a plurality of pixel points contained in the sample defect detection result image and the predicted defect probability values of a plurality of pixel points contained in the predicted defect detection result image by presetting a loss function.

Specifically, the loss value may be calculated according to a preset loss function, a sample defect probability value of a plurality of pixel points, and a predicted defect probability value of a plurality of pixel points included in the predicted defect detection result image, where the preset loss function may be a local loss function, and specifically, the loss value may be calculated by the following formulaLoss：

Wherein the content of the first and second substances,nrepresenting the total number of pixel points,p _iis shown asiThe probability value of the predicted defect of each pixel point,y _iis shown asiThe sample defect probability values of the individual pixel points (the sample defect probability value at the defect point position is 1, the sample defect probability value at the non-defect point position is 0),logis represented by a natural numbereA base logarithm operation;αandγis a constant parameter, set toα=0.75，γ=2。

And 308, updating the network parameters of the pixel defect probability prediction model to be trained according to the loss value, and returning to the step of acquiring training data until the loss value meets the preset training completion condition to obtain the trained pixel defect probability prediction model.

The preset training completion condition may be that a loss function corresponding to the loss value has converged, or that the number of iterations of the training data has reached a target number, or the like. For example, the target number of times may be 1 × 10⁵~2×10⁵And the like, the target times are not particularly limited in the embodiment of the present invention.

Specifically, according to the loss value, calculating new network parameters of the pixel defect probability prediction model to be trained, and then updating the pixel defect probability prediction model to be trained to obtain an updated pixel defect probability prediction model. Then, the terminal re-inputs the training data into the updated pixel defect probability prediction model, and re-executes the steps of the method of the embodiment until the calculated loss value meets the preset training completion condition, so as to obtain the trained pixel defect probability prediction model.

Optionally, the input of the pixel defect probability prediction model to be trained is a gray image of the surface of the high-resolution glass, and the data type is a single-channel matrix of uint 8; and outputting a glass surface subimage obtained by preliminary positioning and a corresponding defect significance characteristic image, namely a defect image of the image to be detected.

In this embodiment, through the setting of the model and the training process of the model, the recognition performance of the model can be improved and the calculation amount of training and the calculation amount of recognition can be reduced on the premise of ensuring the accuracy of feature extraction.

In an embodiment, as shown in fig. 5, a specific processing procedure of step 304 "inputting the sample defect saliency feature map and the sample defect subgraph into a pixel defect probability prediction model to be trained to obtain a predicted defect detection result image" includes:

and step 402, performing channel cascade processing on the sample defect significance characteristic graph and the sample defect subgraph to obtain a sample spliced image.

Specifically, the pixel defect probability prediction model to be trained may include a first channel cascade module, where the first channel cascade module is configured to receive a sample defect significance feature map and a sample defect subgraph input by a terminal, and perform channel cascade on the two images to obtain a sample spliced image. For example, a channel cascade refers to two would have a width of: (w) High, high (h) A channel (a)c) The images of 3 dimensions are stitched according to the channel dimensions.

In one example, the sample defect subgraph may be a defect subgraph of a high resolution glass surface image, and the size information may be 224 × 224 × 1 (width × height × number of channels); the size information of the sample defect saliency map may be 224 × 224 × 1 (width × height × number of channels). In this way, the terminal performs channel cascade on the two images through the first channel cascade module to obtain 1 image of 224 × 224 × 2 (width × height × number of channels).

And step 404, performing feature vector extraction processing on the sample spliced image through a first number of encoders to obtain a feature map of the sample image.

Wherein, the structure diagram of each encoder may be as shown in fig. 6, and includes a first input subunit(s) ((Input features) A first convolution calculation subunit, a second convolution calculation subunit, a third convolution calculation subunit and a first output subunit (Out features). The first convolution calculation subunit comprises a 1 × 1 convolution (1 × 1)Conv) Layer, normalization: (BN) Layer and activation function: (ReLU) Layer, second convolutionThe calculation subunit comprises a 3 × 3 packet convolution (3 × 3)Group Conv) A layer, a normalization layer, and an activation function layer, and a third convolution calculating subunit comprising a 1 × 1 convolution (1 × 1)Conv) Layer, normalization layer. In which a 3 × 3 packet convolution (3 × 3)Group Conv) The layers are grouped convolution calculations with convolution kernels of 3 x 3 size, and the normalization layer is obtained byBatch NormalizationThe function is normalized, 1 × 1 convolution (1 × 1)Conv) The layers are convolution calculated with convolution kernel of 1 × 1 size and the activation function layer is formed byReLUThe function performs activation processing.

Specifically, the terminal inputs the sample-stitched image first into the first input subunit in the first encoder. In the first encoder, a terminal firstly calculates a sample spliced image through a first convolution calculation subunit, the first convolution calculation subunit inputs a calculation result to a second convolution calculation subunit, the second convolution calculation subunit calculates the sample spliced image, and inputs the calculation result to a third convolution calculation subunit to obtain a first convolution calculation result. And the terminal takes the sum of the first convolution calculation result and the sample splicing image as an output result of the first encoder.

In one example, the calculation of the sum of the first convolution calculation result and the sample stitched image may be: and adding the first convolution calculation result and the sample splicing image according to the corresponding channel position to obtain a sum. The calculation process of the other encoders is similar to that of the first encoder, and is not described herein.

In one example, the first input subunit inputs the sample stitched image to a first convolution calculation subunit where the sample stitched image would first be convolved by 1 × 1 (1 × 1)Conv) Layer, carrying out convolution calculation with convolution kernel of 1 × 1 size to obtain a first initial convolution calculation result; the initial convolution calculation result is input into the normalization layer according toBatch NormalizationNormalizing the function to obtain a normalized result, inputting the normalized result into the activation function layer, and performing normalization according to the normalized resultReLUThe function activates the normalization result to obtain a second initial convolution meterAnd calculating a result. That is, the output of the first convolution operator unit is the second initial convolution calculation result.

In this way, the terminal inputs the second initial convolution calculation result to the second convolution calculation subunit, and the second initial convolution calculation result is sequentially convoluted by 3 × 3 packets (3 × 3)Group Conv) And the convolution layer, the normalization layer and the activation function layer obtain a third initial convolution calculation result. And the terminal inputs the third initial convolution calculation result into a third convolution calculation subunit, and sequentially performs convolution calculation and normalization processing on the 1 × 1 convolution layer on the third initial convolution calculation result to obtain a first convolution calculation result. And the terminal takes the sum of the first convolution calculation result and the sample splicing image as an output result of the first encoder.

And 406, performing convolution operation on the feature map of the sample image through a second number of decoders to obtain the predicted defect probability value of each pixel point in the sample image, and combining to obtain a predicted defect detection result image of the sample image.

Wherein, the structure diagram of each decoder may be as shown in fig. 7, and includes a second input subunit(s) ((s))Input features) A fourth convolution calculation subunit, a fifth convolution calculation subunit, a sixth convolution calculation subunit, and an upsampling subunit (a)UpSampling) And a second output subunit (Out features). The fourth convolution calculation subunit comprises a 3 × 3 packet convolution (3 × 3)Group Conv) Layer, normalization: (BN) Layer and activation function: (ReLU) Layer, 3 x 3 packet convolution (3 x 3)Group Conv) The layers are grouped convolution calculations with convolution kernels of 3 x 3 size, and the normalization layer is obtained byBatch NormalizationThe function is normalized by activating the function layerReLUThe function performs activation processing. The fifth convolution calculation subunit includes a 1 × 1 convolution (1 × 1)Conv) Layer, normalization layer and activation function layer, 1 × 1 convolution (1 × 1)Conv) The layer is subjected to convolution calculation with a convolution kernel of 1 × 1 size, and the fifth convolution calculation subunit has the same structure as the sixth convolution subunit.

Specifically, the terminal inputs the feature map of the sample image output by the encoder module into a first decoder, in the first decoder, the feature map of the sample image passes through a fourth convolution calculation subunit and a fifth convolution calculation subunit in sequence to obtain a second convolution calculation result, and the feature map of the sample image is also directly input into a sixth convolution calculation subunit to obtain a third convolution calculation result. And the terminal inputs the sum of the third convolution calculation result and the second convolution calculation result to the upper sampling subunit to obtain an output result of the first decoder.

Optionally, the calculation process of the sum of the first convolution calculation result and the second convolution calculation result may be: and adding the two convolution calculation results according to the corresponding channel positions to obtain a sum. The calculation process of the other decoders is similar to that of the first decoder, and is not described herein again. The specific calculation processes of the fourth convolution calculation subunit, the fifth convolution calculation subunit and the sixth convolution calculation subunit are similar to the calculation processes of the first convolution calculation subunit, the second convolution calculation subunit and the third convolution calculation subunit, and are not described herein again.

In this embodiment, the input of each encoder is a defect region sub-graph (sample defect significance feature graph and sample defect sub-graph) or a feature graph output by the last encoder, and the output is a feature graph (each pixel position is represented by a feature vector) calculated by a learnable convolution kernel. The encoder performs layer-by-layer feature extraction on the original image to generate a layer-by-layer shallow-to-deep feature map. Each decoder decodes the deep features by convolution operations to predict the probability of each pixel location being a defect while restoring the shallow features. The cascade operation is to make the decoder receive coding features of different levels and different scales, so as to improve the accuracy of the detail of the result.

The practical use process of the trained pixel defect probability prediction model may be described in detail below with reference to a specific embodiment, as shown in fig. 8:

the terminal can apply the pixel point defect probability prediction model to the actual application scene of the glass product surface defect detection. The specific detection process may be: performing the processes of step 102 to step 108 on an image to be detected (a high-resolution glass surface image) to perform feature pre-extraction and defect clipping processing to obtain a glass surface subgraph (a defect subgraph) which possibly contains defects and a defect feature map (a target defect significance feature map) corresponding to the subgraph; the terminal carries out channel cascade processing on the obtained glass surface subgraph (defect subgraph) which possibly contains defects and a corresponding defect feature map (target defect significance feature map) to obtain a multi-channel image with defect features, the multi-channel image (spliced image) is input to a trained pixel point defect probability prediction model to obtain a defect detection result image (single-channel score map) of the image to be detected, and the result image comprises defect probability values of a plurality of pixel points in the detected image. The terminal can carry out binarization processing on a plurality of pixel points in the image to be detected based on a preset binarization segmentation threshold value and defect probability values of the plurality of pixel points in the image to be detected, so as to obtain a defect image (defect segmentation result) of the image to be detected.

The high resolution glass side image is shown in FIG. 9A and the defect image of the image to be detected is shown in FIG. 9B. For convenient display, a large-area non-defective area is cut out, an intermediate process is omitted, and a display result is a final result obtained by restoring the detection result of the subgraph to the corresponding position of the original graph. As can be seen from the results, the detection model can segment the fine and complex defects on the glass surface very accurately. Meanwhile, in the detection process, the high-resolution image is cut through simple feature pre-extraction operation, so that the image size of an input detection model is greatly reduced, and the convolution operation amount of a neural network is also greatly reduced, for example, the input image size is 7168 × 498 and is about 3.6 × 10⁶Pixel, and the extracted defect subgraph is about 256 × 256, about 6.5 × 10⁴The number of pixels and subgraphs is about 10, and the number of image pixels is in direct proportion to the amount of convolution calculation, so that the amount of detection calculation is reduced by about 5 times, and the amount of calculation required by feature pre-extraction is small and almost negligible. In addition, the surface defects of the glass product tend to be small in area and have a defect surfaceThe smaller the product ratio is, the larger the calculation amount saved by the method is, so that the detection efficiency is obviously improved, the real-time detection requirement of an industrial production line is met, and the method has very high practical value.

In an example, the trained pixel defect probability prediction model may include an encoder and a decoder, the encoder module may include a plurality of encoders, the decoder module may include a plurality of decoders and a plurality of channel cascade units, and specific parameter information of each encoder, each decoder, and each channel cascade unit may be as shown in table 1 below:

TABLE 1

It should be understood that, although the steps in the flowcharts related to the embodiments as described above are sequentially displayed as indicated by arrows, the steps are not necessarily performed sequentially as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the flowcharts related to the embodiments described above may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the execution order of the steps or stages is not necessarily sequential, but may be rotated or alternated with other steps or at least a part of the steps or stages in other steps.

Based on the same inventive concept, the embodiment of the present application further provides a defect detection apparatus based on salient feature pre-extraction and image segmentation, for implementing the above-mentioned defect detection method based on salient feature pre-extraction and image segmentation. The implementation scheme for solving the problem provided by the apparatus is similar to the implementation scheme described in the above method, so that the specific limitations in one or more embodiments of the defect detection apparatus based on salient feature pre-extraction and image segmentation provided below can be referred to the limitations in the above defect detection method based on salient feature pre-extraction and image segmentation, and are not described herein again.

In one embodiment, as shown in fig. 10, there is provided a defect detecting apparatus 600 based on salient feature pre-extraction and image segmentation, including: an obtaining module 601, a calculating module 602, an extracting module 603, a clipping module 604 and a detecting module 605, wherein:

the acquiring module 601 is configured to acquire an image to be detected, and process the image to be detected to obtain a gray level change feature map of the image to be detected, where the gray level change feature map includes pixel values of each pixel point in the image to be detected;

a calculating module 602, configured to perform mean calculation on pixel values corresponding to pixel points meeting a preset threshold screening condition to obtain an image segmentation threshold;

an extracting module 603, configured to extract a region corresponding to a pixel point whose pixel value is greater than or equal to an image segmentation threshold in the gray-scale change feature map, to obtain a target defect saliency feature map of the image to be detected;

the cropping module 604 is configured to crop an image to be detected according to the coordinates of the target defect saliency feature map to obtain a defect sub-map corresponding to the target defect saliency feature map;

and the detection module 605 is configured to input the target defect saliency feature map and the defect subgraph into a pixel defect probability prediction model trained in advance, so as to obtain a defect detection result image of the image to be detected.

In one embodiment, the extraction module is specifically configured to:

In one embodiment, the apparatus further comprises:

and the image processing module is used for carrying out image enhancement processing on the defect subgraph to obtain the processed defect subgraph, wherein the image enhancement processing comprises one or more of image random rotation processing, image displacement processing, image scaling processing, image shearing processing and image overturning processing.

In one embodiment, the apparatus further comprises:

and the binarization processing module is used for carrying out binarization processing on the plurality of pixel points in the image to be detected according to a preset binarization segmentation threshold value and defect probability values of the plurality of pixel points in the image to be detected so as to obtain a defect image of the image to be detected.

In one embodiment, the apparatus further comprises:

the training data acquisition module is used for acquiring training data, and the training data comprises a sample defect significance characteristic diagram of a sample image, a sample defect subgraph and a sample defect detection result image;

the input module is used for inputting the sample defect significance characteristic graph and the sample defect subgraph into a pixel point defect probability prediction model to be trained to obtain a predicted defect detection result image;

the loss value calculation module is used for calculating a loss value according to the sample defect probability values of a plurality of pixel points contained in the sample defect detection result image and the predicted defect probability values of a plurality of pixel points contained in the predicted defect detection result image through a preset loss function;

and the updating module is used for updating the network parameters of the pixel defect probability prediction model to be trained according to the loss values and returning to the step of acquiring the training data until the loss values meet the preset training completion conditions to obtain the trained pixel defect probability prediction model.

In one embodiment, the input module is specifically configured to:

The modules in the defect detection device based on salient feature pre-extraction and image segmentation can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In another embodiment, the present invention further provides a glass defect detecting apparatus 1000 for pre-extracting salient features and segmenting an image, which is described in detail below with reference to fig. 11. The device comprises the following components: the image acquisition device 100, the image processing and detection device 200, the defect visualization device 400, the defect alarm device 300 and the data storage and management device 500.

The image acquisition device 100 is used for acquiring surface images of glass products by adopting an industrial linear array camera, comprises a conveyor belt, a light source and the industrial linear array camera and outputs high-resolution glass product surface gray level images; the image processing and detecting device 200 uses the image collected by the image collecting device 100 as input through the program instructionGPUAndCPUperforming feature pre-extraction related calculation processing on the image, then performing defect detection calculation, and finally outputting a defect detection result; the defect detection result is transmitted to the defect visualization device 400 and the data storage and management deviceIs placed in 500. The defect visualization device is used for visually displaying the defect detection result, including the position, type and shape of the defect. The visualization device 400 is controlled by the detection device 200, usingLEDThe liquid crystal display completes the display. The data storage and management device 500 stores and manages the detection output result, and can realize reading and query at any time. The device is controlled by a computer program, and a storage medium is a mechanical hard disk; the defect detection result is input into the defect alarm device 300 through a decision (alarm or no alarm) to give an alarm when defective substandard products appear in the glass industrial production line. The alarm device 300 is also controlled by the detection device 200, and the output is divided into two forms of audio and light, which are respectively completed by using a loudspeaker and an LED alarm lamp.

In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 12. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing the detection data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a defect detection method based on salient feature pre-extraction and image segmentation.

Those skilled in the art will appreciate that the architecture shown in fig. 12 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is further provided, which includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the steps of the above method embodiments.

In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.

In an embodiment, a computer program product is provided, comprising a computer program which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. The non-volatile memory may include a read-only memory (R-ROM)Read-Only Memory，ROM) Magnetic tape, floppy disk, flash memory, optical memory, high-density embedded nonvolatile memory, and resistance change memoryReRAM) Magnetic change memory (1)Magnetoresistive Random Access Memory，MRAM) Ferroelectric memory device (Ferroelectric Random Access Memory，FRAM) Phase change memory devicePhase Change Memory， PCM) Graphene memory, etc. The volatile memory may comprise random access memory (Random Access Memory， RAM) Or external cache memory, etc. By way of illustration and not limitation,RAMcan be in various forms, such as static random access memory (Static Random Access Memory，SRAM) Or dynamic random access memory (Dynamic Random Access Memory，DRAM) And the like. The databases referred to in various embodiments provided herein may include at least one of relational and non-relational databases.The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing based data processing logic devices, etc., without limitation.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A defect detection method based on salient feature pre-extraction and image segmentation is characterized by comprising the following steps:

acquiring an image to be detected to obtain a gray level change characteristic map of the image to be detected, wherein the gray level change characteristic map comprises pixel values of all pixel points in the image to be detected, and the image to be detected is a high-resolution image of the glass surface;

2. The method according to claim 1, wherein the extracting a region corresponding to a pixel point with a pixel value greater than or equal to the image segmentation threshold value from the gray-scale change feature map to obtain a target defect saliency feature map of the image to be detected comprises:

3. The method of claim 1, wherein before the step of inputting the target defect saliency feature map and the defect sub-map into a pre-trained pixel point defect probability prediction model, the method further comprises:

4. The method of claim 1, wherein the defect detection result image of the image to be detected comprises defect probability values of a plurality of pixel points in the image to be detected, and the method further comprises:

5. The method of claim 1, further comprising:

6. The method of claim 5, wherein the step of inputting the sample defect significance feature map and the sample defect subgraph into a pixel defect probability prediction model to be trained to obtain a predicted defect detection result image comprises:

7. A defect detection device based on salient feature pre-extraction and image segmentation, the device comprising:

the system comprises an acquisition module, a processing module and a display module, wherein the acquisition module is used for acquiring an image to be detected to obtain a gray change characteristic map of the image to be detected, the gray change characteristic map comprises pixel values of all pixel points in the image to be detected, and the image to be detected is a high-resolution image of the glass surface;

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 6.

9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the method of any one of claims 1 to 6 when executed by a processor.