CN112767354A - Defect detection method, device and equipment based on image segmentation and storage medium - Google Patents
Defect detection method, device and equipment based on image segmentation and storage medium Download PDFInfo
- Publication number
- CN112767354A CN112767354A CN202110073392.4A CN202110073392A CN112767354A CN 112767354 A CN112767354 A CN 112767354A CN 202110073392 A CN202110073392 A CN 202110073392A CN 112767354 A CN112767354 A CN 112767354A
- Authority
- CN
- China
- Prior art keywords
- image
- defect
- target
- neural network
- preset neural
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/047—Probabilistic or stochastic networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20076—Probabilistic image processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
Abstract
The invention discloses a defect detection method based on image segmentation, which comprises the following steps: receiving a training image containing a target defect, and carrying out pixel labeling on the target defect to obtain a defect image; performing feature extraction on the defect image through a preset neural network to obtain a target feature layer, and performing average pooling of different scales on the target feature layer to obtain pooling results of different scales; performing feature fusion on the pooling results of different scales to obtain a target feature map, and calculating consistency loss according to the target feature map; and according to the consistency loss, carrying out iterative processing on the preset neural network, and adjusting the parameters of the preset neural network after the iterative processing so as to finish the training of the preset neural network. The invention also discloses a defect detection device, equipment and a storage medium based on image segmentation. The invention improves the defect compatibility problem in detection, increases the integrity of the defect, reduces the defect classification error and improves the stability and generalization of the neural network.
Description
Technical Field
The present invention relates to the field of image segmentation, and in particular, to a method, an apparatus, a device, and a storage medium for detecting defects based on image segmentation.
Background
With the rapid development of the deep learning technology, the image segmentation technology is applied to a plurality of fields such as medical treatment, transportation, industry and the like, especially defect detection branches are widely applied to cloth flaw detection, workpiece surface quality detection, aerospace field and the like, the types, shapes and sizes of defects have very large randomness in an industrial scene, and when the defect detection problem is solved by the mainstream image segmentation network at present, the defect detection is limited to be inflexible in detection scale, and the detection precision of a large target is difficult to be compatible in the same scene, and the detection effect of a small target cannot be omitted. In defect classification, due to the difference of defect scales, the misjudgment of the types of cavities, discontinuities and edges in large-scale defect detection is also a troublesome problem faced by the current defect detection.
Disclosure of Invention
The invention mainly aims to provide a defect detection method, a defect detection device, defect detection equipment and a storage medium based on image segmentation, and aims to solve the technical problem that the detection accuracy is low when the existing image segmentation technology is used for detecting defects.
In addition, in order to achieve the above object, the present invention further provides a defect detection method based on image segmentation, including the steps of:
receiving a training image containing a target defect, and carrying out pixel labeling on the target defect to obtain a defect image;
performing feature extraction on the defect image through a preset neural network to obtain a target feature layer, and performing average pooling of different scales on the target feature layer to obtain pooling results of different scales;
performing feature fusion on the pooling results of different scales to obtain a target feature map, and calculating consistency loss according to the target feature map;
and according to the consistency loss, carrying out iterative processing on the preset neural network, and adjusting the parameters of the preset neural network after the iterative processing so as to finish the training of the preset neural network.
Optionally, after the step of receiving a training image containing a target defect, and performing pixel labeling on the target defect to obtain a defect image, the method includes:
performing data enhancement processing on the defect image to obtain a transformed image, and establishing a consistent relation between the defect image and the transformed image;
judging whether the defect image and the transformation image meet the input requirement of a preset neural network or not;
if the defect image and the transformation image do not meet the input requirement of the preset neural network, performing normalization processing and interpolation processing on the defect image and the transformation image to obtain a target image;
the step of extracting the features of the defect image through a preset neural network to obtain a target feature layer comprises the following steps:
and performing feature extraction on the target image through a preset neural network to obtain a target feature layer.
Optionally, after the step of determining whether the defect image and the transformed image meet the input requirement of the preset neural network, the method includes:
if the defect image and the transformation image meet the input requirement of the preset neural network, taking the defect image and the transformation image as a first image;
the step of extracting the features of the defect image through a preset neural network to obtain a target feature layer comprises the following steps:
and performing feature extraction on the first image through a preset neural network to obtain a target feature layer.
Optionally, the step of performing feature extraction on the defect image through a preset neural network to obtain a target feature layer, and performing average pooling of different scales on the target feature layer to obtain pooling results of different scales includes:
acquiring a target scale of the target feature layer and a scale set corresponding to the target scale, wherein the scale set at least comprises two different scales;
and according to different scales in the scale set, performing average pooling on the target feature layer to obtain pooling results of different scales.
Optionally, the step of performing feature fusion on the pooling results of different scales to obtain a target feature map, and calculating a consistency loss according to the target feature map includes:
and calculating a target difference value between the target feature map and the target feature layer, and taking the target difference value as consistency loss.
Optionally, the step of performing iterative processing on the preset neural network according to the consistency loss, and adjusting parameters of the preset neural network after the iterative processing to complete training of the preset neural network includes:
acquiring a target convolution kernel used when the target characteristic layer is subjected to average pooling of different scales, and taking the target convolution kernel as a parameter of the preset neural network;
adjusting the target convolution kernel, and returning to the step of performing average pooling of different scales on the target characteristic layer to obtain pooling results of different scales, so as to obtain target consistency loss;
and if the target consistency loss is greater than or equal to a preset threshold value, returning to the step of adjusting the target convolution kernel until the target consistency loss is less than the preset threshold value, and finishing the training of the preset neural network.
Optionally, after the step of performing iterative processing on the preset neural network according to the consistency loss and adjusting the parameters of the preset neural network after the iterative processing to complete the training of the preset neural network, the method includes:
inputting a detection image containing a first defect to a trained preset neural network;
acquiring a probability chart output by the trained preset neural network;
and determining the defect type of the first defect according to the probability map.
In order to achieve the above object, the present invention also provides an image segmentation-based defect detection apparatus, including:
the pixel labeling module is used for receiving a training image containing a target defect and performing pixel labeling on the target defect to obtain a defect image;
the characteristic extraction and pooling module is used for extracting characteristics of the defect image through a preset neural network to obtain a target characteristic layer, and performing average pooling of different scales on the target characteristic layer to obtain pooling results of different scales;
the consistency loss calculation module is used for performing feature fusion on the pooling results of different scales to obtain a target feature map, and calculating consistency loss according to the target feature map;
and the neural network training module is used for performing iterative processing on the preset neural network according to the consistency loss and adjusting the parameters of the preset neural network after the iterative processing so as to finish the training of the preset neural network.
Further, to achieve the above object, the present invention also provides an image segmentation based defect detecting apparatus including: the defect detection method comprises a memory, a processor and a defect detection program based on image segmentation, wherein the defect detection program based on image segmentation is stored on the memory and can run on the processor, and when the program is executed by the processor, the steps of the defect detection method based on image segmentation are realized.
In addition, in order to achieve the above object, the present invention further provides a storage medium having stored thereon a defect detection program based on image segmentation, which when executed by a processor implements the steps of the defect detection method based on image segmentation as described above.
The embodiment of the invention provides a defect detection method, a defect detection device, defect detection equipment and a storage medium based on image segmentation. In the embodiment of the invention, a defect detection program based on image segmentation receives a training image containing a target defect, carries out pixel labeling on the target defect to obtain a defect image, carries out feature extraction on the defect image through a preset neural network to obtain a target feature layer, further carries out average pooling of different scales on the target feature layer to obtain pooling results of different scales, then carries out feature fusion on the pooling results of different scales to obtain a target feature map, calculates consistency loss according to the target feature map, carries out iterative processing on the preset neural network, adjusts parameters of the preset neural network after the iterative processing to finally complete training of the preset neural network, and leads the neural network to obtain richer feature representations in the training process and pass through the consistency loss through the training of the preset neural network, the consistency expression of the defects is enhanced, so that the defect compatibility problem in the detection process is improved, the completeness of the defects is increased, the defect classification errors are reduced, and the stability and the generalization of the neural network are improved.
Drawings
Fig. 1 is a schematic hardware structure diagram of an embodiment of a defect detection apparatus based on image segmentation according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a defect detection method based on image segmentation according to a first embodiment of the present invention;
FIG. 3 is a flowchart illustrating a defect detection method based on image segmentation according to a second embodiment of the present invention;
FIG. 4 is a functional block diagram of an embodiment of a defect detection apparatus based on image segmentation.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In the following description, suffixes such as "module", "component", or "unit" used to denote elements are used only for facilitating the explanation of the present invention, and have no specific meaning in itself. Thus, "module", "component" or "unit" may be used mixedly.
The defect detection terminal (also called terminal, equipment or terminal equipment) based on image segmentation in the embodiment of the invention can be a PC (personal computer), and can also be a mobile terminal equipment with a display function, such as a smart phone, a tablet computer, a portable computer and the like.
As shown in fig. 1, the terminal may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display screen (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (e.g., a magnetic disk memory). The memory 1005 may alternatively be a storage device separate from the processor 1001.
Optionally, the terminal may further include a camera, a Radio Frequency (RF) circuit, a sensor, an audio circuit, a WiFi module, and the like. Such as light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor that may adjust the brightness of the display screen according to the brightness of ambient light, and a proximity sensor that may turn off the display screen and/or the backlight when the mobile terminal is moved to the ear. As one of the motion sensors, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally, three axes), detect the magnitude and direction of gravity when the mobile terminal is stationary, and can be used for applications (such as horizontal and vertical screen switching, related games, magnetometer attitude calibration), vibration recognition related functions (such as pedometer and tapping) and the like for recognizing the attitude of the mobile terminal; of course, the mobile terminal may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, and an infrared sensor, which are not described herein again.
Those skilled in the art will appreciate that the terminal structure shown in fig. 1 is not intended to be limiting and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
As shown in fig. 1, a memory 1005, which is a storage medium, may include therein an operating system, a network communication module, a user interface module, and a defect detection program based on image segmentation.
In the terminal shown in fig. 1, the network interface 1004 is mainly used for connecting to a backend server and performing data communication with the backend server; the user interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; and the processor 1001 may be configured to invoke an image segmentation based defect detection program stored in the memory 1005, which when executed by the processor implements the operations in the image segmentation based defect detection method provided by the embodiments described below.
Based on the hardware structure of the equipment, the embodiment of the defect detection method based on image segmentation is provided.
Referring to fig. 2, in a first embodiment of the defect detection method based on image segmentation of the present invention, the defect detection method based on image segmentation includes:
and step S10, receiving a training image containing the target defect, and performing pixel labeling on the target defect to obtain a defect image.
The defect detection method based on image segmentation in the embodiment is applied to the defect detection device based on image segmentation, wherein the defect detection device based on image segmentation can be a personal computer.
The target defect in the present embodiment refers to a defect of a product, and specifically, an application scenario of the defect detection method based on image segmentation in the present embodiment is product defect detection, by taking a photograph of a product having a defect to obtain a training image containing a target defect, it can be known that pictures containing the same kind of product defect are classified into one, the pixel-level labeling of target defects in the training image is performed to determine objects in the training image, and following the process of name tagging, pixel-level labeling can be segmented based on pixel color, that is, the color of the pixel in the target defect is different from that of the pixel in the surrounding image, and the pixel-level labeling of the target defect can make the selection of the object (target defect) more convenient and faster, and the pixel labeling is also the most important work before the image training.
And step S20, extracting the features of the defect image through a preset neural network to obtain a target feature layer, and performing average pooling of different scales on the target feature layer to obtain pooling results of different scales.
Specifically, since the color is susceptible to light, before the feature extraction of the defect image, the defect image may be subjected to gray scale processing to accelerate the feature extraction speed, it can be understood that there are many methods for feature extraction, for example, scale-invariant feature transformation, the feature extraction is to scan the image to obtain feature points (salient points that do not change due to external factors) represented in the image, for example, corner points, edge points, and points with different intensities from surrounding images, and after the feature points are found, points with edge effects may be removed to retain key points (for example, corner points when the image is subjected to rotation transformation). The method for removing the edge effect points has the advantages that the anti-noise capability and the stability are enhanced, then the characteristic points are represented by numerical values, characteristic extraction of different scales is carried out on the defect image, a plurality of characteristic layers can be obtained, each characteristic layer has different characteristic matrixes, the embodiment takes the characteristic layer with the middle scale, namely the target characteristic layer, and carries out average pooling of different scales on the target characteristic layer to obtain pooling results of different scales, specifically, the scales of the target characteristic layer, the half scale of the target characteristic layer, the third scale of the target characteristic layer and the like can be selected according to different scales, the scale size of a convolution kernel used when carrying out average pooling of different scales can also be the scale of the target characteristic layer, the half scale of the target characteristic layer, the third scale of the target characteristic layer and the like, the pooling of the target feature layer with different scales is averaged, and then the pooling result with different scales is obtained, wherein the averaging pooling means that the average value of the feature values in the size of the convolution kernel used for averaging the pooling is also a matrix, and the obtained pooling result is also a matrix, for example, the target feature layer is a 50 × 50 feature matrix, the size of the convolution kernel used for averaging the pooling is one half of the target feature layer, namely, a 25 × 25 grid, and the result of averaging the pooling of the target feature layer using the convolution kernel is a2 × 2 matrix.
And step S30, performing feature fusion on the pooling results of different scales to obtain a target feature map, and calculating consistency loss according to the target feature map.
It can be understood that fusing features of different scales is a method for improving segmentation performance, and as can be known, low-level features have higher resolution, contain more position and detail information, but have lower semantic property and more noise, and high-level features have higher semantic information, but have lower resolution and poorer perception capability for details, the feature fusion includes early fusion and late fusion, higher-dimensional features or composite features may be obtained after feature fusion is performed on pooling results of different scales, a target feature map is obtained by optimizing the correlation between corresponding features between different pooling results, and as can be known, different defect images corresponding to the same product defect and differences in features are obtained by calculating the loss of consistency between the target feature map and the defect images.
And step S40, according to the consistency loss, carrying out iterative processing on the preset neural network, and adjusting the parameters of the preset neural network after the iterative processing to finish the training of the preset neural network.
Performing feature fusion on pooling results of different scales to obtain consistency loss, wherein the consistency loss represents differences generated by identifying the same type of defects and features of different scales, and the smaller the obtained consistency loss is, the better the preset neural network is, on the basis, performing iterative processing on the preset neural network according to the obtained consistency loss, specifically, performing iterative processing on the preset neural network if the obtained consistency loss is smaller than a preset expected value, and optimizing the preset neural network by adjusting parameters of the preset neural network after the iterative processing to enhance the consistency expression of the same defect features, so that the obtained defect integrity is stronger, and defects internal cavities and edge classification errors are reduced, in the embodiment, the parameters of the preset neural network comprise convolution kernels used when performing average pooling of different scales on a target feature layer, and iteration times and the like, wherein when the intention consistency loss obtained after the iteration is carried out for a plurality of times reaches a preset expected value, or the iteration times reach a certain number, the training of the preset neural network is stopped so as to finish the training of the preset neural network.
Specifically, steps subsequent to step S10 include:
step a1, performing data enhancement processing on the defect image to obtain a transformed image, and establishing a consistent relation between the defect image and the transformed image.
A2, judging whether the defect image and the transformation image meet the input requirements of a preset neural network;
step a3, if the defect image and the transformation image do not meet the input requirement of the preset neural network, performing normalization processing and interpolation processing on the defect image and the transformation image to obtain a target image.
Specifically, the step S20 is a step of refining:
step a4, extracting the features of the target image through a preset neural network to obtain a target feature layer.
It can be known that, because the probability of defects in a product is low, the obtained defect atlas is relatively single, and more training data is needed for training a preset neural network, the embodiment performs data enhancement on a training image to increase the number of training images and facilitate enhancing the consistency between defects of the same type, the method for performing data enhancement on the training image includes rotating, adjusting brightness, transforming contrast, randomly cropping, mirroring, distorting and the like on the defect image to obtain different images (i.e., the transformed image in the embodiment) corresponding to the same defect image, and the purpose of establishing the consistent relationship between the defect image and the transformed image is to enable the preset neural network to better identify that the defect image and the transformed image are different images corresponding to the same defect type, after the transformed image is obtained by enhancing the class similarity between the defect features, the purpose of performing normalization processing and interpolation processing on the defect image and the transformed image is to make the defect image and the transformed image meet the input requirement of a preset neural network.
Specifically, the steps after step a2 include:
and b1, if the defect image and the transformation image meet the input requirement of the preset neural network, taking the defect image and the transformation image as a first image.
Specifically, the step S20 is a step of refining:
and b2, performing feature extraction on the first image through a preset neural network to obtain a target feature layer.
It should be noted that the preset neural network in this embodiment has a certain input requirement, and the input requirement may be set to perform normalization and unification processing on the defect image input into the preset neural network, specifically, when the defect image corresponding to the same defect type input into the preset neural network includes the defect image and the corresponding transformed image, since the transformed image may undergo data enhancement processing such as random cropping and the like, the transformed image and the defect image have different sizes, the transformed image needs to be enlarged to the same size as the defect image, and when the image is enlarged, normalization processing and interpolation processing need to be performed on the pixel missing problem caused in the image enlargement process, so that the defect image and the transformed image meet the input requirement of the preset neural network, for example, the input requirement of the preset neural network in this embodiment is, the pixel size of the input defect image is a multiple of 16, when the defect image or the transformation image does not meet the requirement, normalization processing and interpolation processing are required to be carried out on the defect image or the transformation image, and when the image after normalization processing and interpolation processing meets the requirement, feature extraction is carried out on the defect image and the transformation image through a preset neural network so as to obtain a target feature layer.
Specifically, the step of step S40 refinement includes:
and c1, acquiring a target convolution kernel used when the target characteristic layer is subjected to average pooling in different scales, and taking the target convolution kernel as a parameter of the preset neural network.
And c2, adjusting the target convolution kernel, and returning to the step of performing average pooling of different scales on the target feature layer to obtain pooling results of different scales, so as to obtain target consistency loss.
And c3, if the target consistency loss is greater than or equal to a preset threshold, returning to the step of adjusting the target convolution kernel until the target consistency loss is less than the preset threshold, and finishing the training of the preset neural network.
It should be noted that, after feature fusion is performed on pooling results of different scales, a feature map is obtained, where the obtained feature map has multiple levels, and the target feature map in this embodiment refers to a last-layer feature map in the above-mentioned multiple-level feature map, and then, a consistency loss is calculated according to the target feature map, where the obtained consistency loss represents different defect images corresponding to the same product defect, and a difference in features, where a parameter of the neural network is preset in this embodiment, and includes a convolution kernel used when performing average pooling of different scales on the target feature layer, that is, a target convolution kernel in this embodiment, and the calculated consistency loss can be changed by adjusting the target convolution kernel, and it is determined whether the adjustment of the target convolution kernel is accurate by obtaining a change situation of the consistency loss before and after adjusting the target convolution kernel, and it is known that the obtained consistency loss is smaller, and (4) indicating that the adjustment direction of the target convolution kernel is accurate, circularly iterating in the way until the obtained consistency loss is less than a preset threshold value, stopping iteration, and finishing the training of the preset neural network.
Specifically, steps subsequent to step S40 include:
and d1, inputting the detection image containing the first defect into the trained preset neural network.
And d2, acquiring the probability chart output by the trained preset neural network.
And d3, determining the defect type of the first defect according to the probability map.
Therefore, after the training of the preset neural network is completed, the defect detection may be performed on the unknown picture through the trained preset neural network, where the unknown picture may or may not include a defect, and after an image to be subjected to the defect detection (i.e., the detection image including the first defect in this embodiment) is obtained, the detection image including the first defect is input to the trained preset neural network, unlike the training image, the detection image does not need to be subjected to pixel labeling, instead, after the trained preset neural network is input, the detection image is subjected to feature extraction through the preset neural network to obtain a target feature layer, then the target feature layer is pooled to obtain a pooled result, then the pooled result is subjected to feature fusion to obtain a feature map, and finally, the feature map is subjected to linear operation and logistic regression operation, obtaining a probability map, wherein the obtained probability map characterizes the probability of which defect type the detected image belongs to, that is, the probability map contains the defect type and the probability of the detected image belonging to each defect type, and the defect type corresponding to the highest probability value is taken, that is, the defect type of the first defect in the embodiment.
In this embodiment, a defect detection program based on image segmentation receives a training image containing a target defect, performs pixel labeling on the target defect to obtain a defect image, performs feature extraction on the defect image through a preset neural network to obtain a target feature layer, further performs average pooling of different scales on the target feature layer to obtain pooling results of different scales, performs feature fusion on the pooling results of different scales to obtain a target feature map, calculates a consistency loss according to the target feature map, performs iterative processing on the preset neural network, adjusts parameters of the preset neural network after the iterative processing to finally complete training of the preset neural network, and performs training on the preset neural network to obtain richer feature representations in the neural network training process and enhance consistency expression of the defect through the consistency loss, therefore, the defect compatibility problem in the detection process is improved, the completeness of the defect is increased, the defect classification errors are reduced, and the stability and the generalization of the neural network are further improved.
Further, referring to fig. 3, on the basis of the above embodiment of the present invention, a second embodiment of the defect detection method based on image segmentation of the present invention is proposed.
This embodiment is a step of the first embodiment, which is a refinement of step S20, and the difference between this embodiment and the above-described embodiment of the present invention is:
step S21, obtaining a target scale of the target feature layer and a scale set corresponding to the target scale, where the scale set includes at least two different scales.
And step S22, performing average pooling on the target feature layer according to different scales in the scale set to obtain pooling results of different scales.
It should be noted that, the target scale in this embodiment refers to a scale of the target feature layer, and the scale included in the scale set corresponding to the target scale may be, a scale of the target feature layer (i.e., a target scale), a half scale of the target feature layer (i.e., a half target scale), a third scale of the target feature layer (i.e., a third target scale), and the like, in this embodiment, different scales correspond to different convolution kernels, and these convolution kernels are used in the process of performing average pooling on the target feature layer according to different scales in the scale set, specifically, when performing average pooling on the target feature layer at a third target scale, a convolution kernel corresponding to a third target scale needs to be used, for example, the target feature layer is a 60 × 60 feature matrix, and the size of the convolution kernel used in the average pooling is one third of the target feature layer, i.e. a 20 x 20 grid, the result of averaging pooling the target feature layer using the convolution kernel results in a3 x 3 matrix, and the matrix obtained after averaging pooling is the pooling result for one third of the target scale.
Specifically, the step of step S30 refinement includes:
and e1, calculating a target difference value between the target feature map and the target feature layer, and taking the target difference value as consistency loss.
Therefore, the category loss between the defect features can be obtained by calculating the target difference between the target feature map and the target feature layer, and the result is represented by a numerical value.
In the embodiment, the target feature layer is subjected to average pooling in different scales, so that the consistency expression of the defects is enhanced, the defect compatibility problem in the detection process is solved, the completeness of the defects is improved, the defect classification errors are reduced, and the stability and the generalization of the neural network are improved.
In addition, referring to fig. 4, an embodiment of the present invention further provides an image segmentation-based defect detection apparatus, including:
the pixel labeling module 10 is configured to receive a training image containing a target defect, and perform pixel labeling on the target defect to obtain a defect image;
the feature extraction and pooling module 20 is configured to perform feature extraction on the defect image through a preset neural network to obtain a target feature layer, and perform average pooling of different scales on the target feature layer to obtain pooling results of different scales;
a consistency loss calculation module 30, configured to perform feature fusion on the pooling results of different scales to obtain a target feature map, and calculate consistency loss according to the target feature map;
and the neural network training module 40 is configured to perform iterative processing on the preset neural network according to the consistency loss, and adjust parameters of the preset neural network after the iterative processing, so as to complete training of the preset neural network.
Optionally, the apparatus for detecting defects based on image segmentation includes:
the data enhancement module is used for performing data enhancement processing on the defect image to obtain a transformed image and establishing a consistent relation between the defect image and the transformed image;
the judging module is used for judging whether the defect image and the transformation image meet the input requirement of a preset neural network or not;
the normalization and interpolation module is used for performing normalization processing and interpolation processing on the defect image and the transformation image to obtain a target image if the defect image and the transformation image do not meet the input requirement of the preset neural network;
the consistency loss calculation module 30 includes:
and the first feature extraction unit is used for extracting features of the target image through a preset neural network to obtain a target feature layer.
Optionally, the apparatus for detecting defects based on image segmentation includes:
the first image determining module is used for taking the defect image and the transformation image as first images if the defect image and the transformation image meet the input requirements of the preset neural network;
the consistency loss calculation module 30 includes:
and the second feature extraction unit is used for extracting features of the first image through a preset neural network to obtain a target feature layer.
Optionally, the feature extraction and pooling module 20 includes:
a target scale obtaining unit, configured to obtain a target scale of the target feature layer and a scale set corresponding to the target scale, where the scale set includes at least two different scales;
and the average pooling unit is used for performing average pooling on the target feature layer according to different scales in the scale set to obtain pooling results of different scales.
Optionally, the consistency loss calculation module 30 includes:
and the target difference value calculating unit is used for calculating a target difference value between the target feature map and the target feature layer and taking the target difference value as consistency loss.
Optionally, the neural network training module 40 includes:
a target convolution kernel obtaining unit, configured to obtain a target convolution kernel used when performing average pooling of different scales on the target feature layer, and use the target convolution kernel as a parameter of the preset neural network;
the adjusting unit is used for adjusting the target convolution kernel and returning to the step of performing average pooling of different scales on the target characteristic layer to obtain pooling results of different scales so as to obtain target consistency loss;
and the circulating unit is used for returning to the step of adjusting the target convolution kernel if the target consistency loss is greater than or equal to a preset threshold value until the target consistency loss is less than the preset threshold value, and finishing the training of the preset neural network.
Optionally, the apparatus for detecting defects based on image segmentation includes:
the detection image input module is used for inputting a detection image containing a first defect to the trained preset neural network;
the probability map output module is used for acquiring a probability map output by the trained preset neural network;
and the defect type determining module is used for determining the defect type of the first defect according to the probability map.
Furthermore, an embodiment of the present invention further provides a storage medium, on which an image segmentation based defect detection program is stored, and the image segmentation based defect detection program, when executed by a processor, implements the operations in the image segmentation based defect detection method provided by the above embodiments.
The method executed by each program module can refer to each embodiment of the method of the present invention, and is not described herein again.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity/action/object from another entity/action/object without necessarily requiring or implying any actual such relationship or order between such entities/actions/objects; the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
For the apparatus embodiment, since it is substantially similar to the method embodiment, it is described relatively simply, and reference may be made to some descriptions of the method embodiment for relevant points. The above-described apparatus embodiments are merely illustrative, in that elements described as separate components may or may not be physically separate. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the invention. One of ordinary skill in the art can understand and implement it without inventive effort.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) as described above and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the defect detection method based on image segmentation according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A defect detection method based on image segmentation is characterized by comprising the following steps:
receiving a training image containing a target defect, and carrying out pixel labeling on the target defect to obtain a defect image;
performing feature extraction on the defect image through a preset neural network to obtain a target feature layer, and performing average pooling of different scales on the target feature layer to obtain pooling results of different scales;
performing feature fusion on the pooling results of different scales to obtain a target feature map, and calculating consistency loss according to the target feature map;
and according to the consistency loss, carrying out iterative processing on the preset neural network, and adjusting the parameters of the preset neural network after the iterative processing so as to finish the training of the preset neural network.
2. The method of claim 1, wherein the step of receiving a training image containing a target defect, labeling the target defect with pixels, and obtaining a defect image comprises:
performing data enhancement processing on the defect image to obtain a transformed image, and establishing a consistent relation between the defect image and the transformed image;
judging whether the defect image and the transformation image meet the input requirement of a preset neural network or not;
if the defect image and the transformation image do not meet the input requirement of the preset neural network, performing normalization processing and interpolation processing on the defect image and the transformation image to obtain a target image;
the step of extracting the features of the defect image through a preset neural network to obtain a target feature layer comprises the following steps:
and performing feature extraction on the target image through a preset neural network to obtain a target feature layer.
3. The image segmentation-based defect detection method according to claim 2, wherein the step of determining whether the defect image and the transformed image meet the input requirement of a preset neural network is followed by:
if the defect image and the transformation image meet the input requirement of the preset neural network, taking the defect image and the transformation image as a first image;
the step of extracting the features of the defect image through a preset neural network to obtain a target feature layer comprises the following steps:
and performing feature extraction on the first image through a preset neural network to obtain a target feature layer.
4. The image segmentation-based defect detection method according to claim 1, wherein the step of performing feature extraction on the defect image through a preset neural network to obtain a target feature layer, and performing average pooling of different scales on the target feature layer to obtain pooling results of different scales comprises:
acquiring a target scale of the target feature layer and a scale set corresponding to the target scale, wherein the scale set at least comprises two different scales;
and according to different scales in the scale set, performing average pooling on the target feature layer to obtain pooling results of different scales.
5. The image segmentation-based defect detection method according to claim 1, wherein the step of performing feature fusion on the pooling results of different scales to obtain a target feature map, and calculating a consistency loss according to the target feature map comprises:
and calculating a target difference value between the target feature map and the target feature layer, and taking the target difference value as consistency loss.
6. The method of claim 1, wherein the step of performing an iterative process on the pre-set neural network according to the consistency loss and adjusting parameters of the iteratively processed pre-set neural network to complete training of the pre-set neural network comprises:
acquiring a target convolution kernel used when the target characteristic layer is subjected to average pooling of different scales, and taking the target convolution kernel as a parameter of the preset neural network;
adjusting the target convolution kernel, and returning to the step of performing average pooling of different scales on the target characteristic layer to obtain pooling results of different scales, so as to obtain target consistency loss;
and if the target consistency loss is greater than or equal to a preset threshold value, returning to the step of adjusting the target convolution kernel until the target consistency loss is less than the preset threshold value, and finishing the training of the preset neural network.
7. The method for detecting defects based on image segmentation as claimed in claim 1, wherein the step of performing iterative processing on the preset neural network according to the consistency loss and adjusting parameters of the iteratively processed preset neural network to complete training of the preset neural network comprises:
inputting a detection image containing a first defect to a trained preset neural network;
acquiring a probability chart output by the trained preset neural network;
and determining the defect type of the first defect according to the probability map.
8. An image segmentation-based defect detection apparatus, comprising:
the pixel labeling module is used for receiving a training image containing a target defect and performing pixel labeling on the target defect to obtain a defect image;
the characteristic extraction and pooling module is used for extracting characteristics of the defect image through a preset neural network to obtain a target characteristic layer, and performing average pooling of different scales on the target characteristic layer to obtain pooling results of different scales;
the consistency loss calculation module is used for performing feature fusion on the pooling results of different scales to obtain a target feature map, and calculating consistency loss according to the target feature map;
and the neural network training module is used for performing iterative processing on the preset neural network according to the consistency loss and adjusting the parameters of the preset neural network after the iterative processing so as to finish the training of the preset neural network.
9. An image segmentation-based defect detection apparatus, characterized in that the image segmentation-based defect detection apparatus comprises: a memory, a processor and an image segmentation based defect detection program stored on the memory and executable on the processor, the image segmentation based defect detection program when executed by the processor implementing the steps of the image segmentation based defect detection method according to any one of claims 1 to 7.
10. A storage medium having stored thereon an image segmentation based defect detection program, which when executed by a processor implements the steps of the image segmentation based defect detection method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110073392.4A CN112767354A (en) | 2021-01-19 | 2021-01-19 | Defect detection method, device and equipment based on image segmentation and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110073392.4A CN112767354A (en) | 2021-01-19 | 2021-01-19 | Defect detection method, device and equipment based on image segmentation and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112767354A true CN112767354A (en) | 2021-05-07 |
Family
ID=75703281
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110073392.4A Pending CN112767354A (en) | 2021-01-19 | 2021-01-19 | Defect detection method, device and equipment based on image segmentation and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112767354A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113344857A (en) * | 2021-05-13 | 2021-09-03 | 深圳市华汉伟业科技有限公司 | Defect detection network training method, defect detection method and storage medium |
| CN113470024A (en) * | 2021-09-02 | 2021-10-01 | 深圳市信润富联数字科技有限公司 | Hub internal defect detection method, device, equipment, medium and program product |
| CN113706530A (en) * | 2021-10-28 | 2021-11-26 | 北京矩视智能科技有限公司 | Surface defect region segmentation model generation method and device based on network structure |
| CN117495884A (en) * | 2024-01-02 | 2024-02-02 | 湖北工业大学 | Steel surface defect segmentation method and device, electronic equipment and storage medium |
-
2021
- 2021-01-19 CN CN202110073392.4A patent/CN112767354A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113344857A (en) * | 2021-05-13 | 2021-09-03 | 深圳市华汉伟业科技有限公司 | Defect detection network training method, defect detection method and storage medium |
| CN113344857B (en) * | 2021-05-13 | 2022-05-03 | 深圳市华汉伟业科技有限公司 | Defect detection network training method, defect detection method and storage medium |
| CN113470024A (en) * | 2021-09-02 | 2021-10-01 | 深圳市信润富联数字科技有限公司 | Hub internal defect detection method, device, equipment, medium and program product |
| CN113706530A (en) * | 2021-10-28 | 2021-11-26 | 北京矩视智能科技有限公司 | Surface defect region segmentation model generation method and device based on network structure |
| CN117495884A (en) * | 2024-01-02 | 2024-02-02 | 湖北工业大学 | Steel surface defect segmentation method and device, electronic equipment and storage medium |
| CN117495884B (en) * | 2024-01-02 | 2024-03-22 | 湖北工业大学 | Steel surface defect segmentation method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109961009B (en) | Pedestrian detection method, system, device and storage medium based on deep learning | |
| US9697416B2 (en) | Object detection using cascaded convolutional neural networks | |
| CN112767354A (en) | Defect detection method, device and equipment based on image segmentation and storage medium | |
| CN111771226A (en) | Electronic device, image processing method thereof, and computer-readable recording medium | |
| CN112767366A (en) | Image recognition method, device and equipment based on deep learning and storage medium | |
| CN105046254A (en) | Character recognition method and apparatus | |
| CN109711508B (en) | Image processing method and device | |
| CN111340796B (en) | Defect detection method and device, electronic equipment and storage medium | |
| CN109559303B (en) | Method and device for identifying calcification points and computer-readable storage medium | |
| WO2022105569A1 (en) | Page direction recognition method and apparatus, and device and computer-readable storage medium | |
| CN111814905A (en) | Target detection method, target detection device, computer equipment and storage medium | |
| CN111598084B (en) | Defect segmentation network training method, device, equipment and readable storage medium | |
| CN108052869B (en) | Lane line recognition method, lane line recognition device and computer-readable storage medium | |
| CN111080665A (en) | Image frame identification method, device and equipment and computer storage medium | |
| CN115345895B (en) | Image segmentation method and device for visual detection, computer equipment and medium | |
| CN112116596A (en) | Training method of image segmentation model, image segmentation method, medium, and terminal | |
| CN115063473A (en) | Object height detection method and device, computer equipment and storage medium | |
| CN110955369B (en) | Focus judgment method, device and equipment based on click position and storage medium | |
| CN114677578A (en) | Method and device for determining training sample data | |
| CN109685069B (en) | Image detection method, device and computer readable storage medium | |
| CN112966757A (en) | Method and device for expanding training sample, storage medium and equipment | |
| CN114255493A (en) | Image detection method, face detection device, face detection equipment and storage medium | |
| CN110807452A (en) | Prediction model construction method, device and system and bank card number identification method | |
| CN114529912A (en) | Graphic verification code identification method and device, electronic equipment and readable storage medium | |
| Chen et al. | Design of gesture recognition system based on machine vision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |