CN114494260B

CN114494260B - Object defect detection method and device, computer equipment and storage medium

Info

Publication number: CN114494260B
Application number: CN202210401828.2A
Authority: CN
Inventors: 田倬韬; 王远; 易振彧; 刘枢; 吕江波; 沈小勇
Original assignee: Shenzhen Smartmore Technology Co Ltd
Current assignee: Shenzhen Smartmore Technology Co Ltd
Priority date: 2022-04-18
Filing date: 2022-04-18
Publication date: 2022-07-19
Anticipated expiration: 2042-04-18
Also published as: CN114494260A; WO2023201924A1

Abstract

The application relates to an object defect detection method, an object defect detection device, computer equipment and a storage medium. The method comprises the following steps: acquiring a characteristic diagram corresponding to an image to be detected; the image to be detected is an image obtained by shooting a target object; the target object is an object needing defect detection; carrying out region division on the feature map to obtain a first sub-feature map corresponding to each pixel point in the image to be detected; inputting the first sub-feature maps into first target region classifiers in one-to-one correspondence to obtain first defect type probabilities corresponding to the image to be detected; inputting the feature map into a target global classifier to obtain a second defect type probability corresponding to the image to be detected; the second defect type probability comprises the attribution probability of each pixel point in the image to be detected aiming at each defect type; and determining the defect type corresponding to the target object according to the first defect type probability and the second defect type probability. By adopting the method, the accuracy of identifying the defect type of the target object can be improved.

Description

Object defect detection method and device, computer equipment and storage medium

Technical Field

The present application relates to the field of machine vision technologies, and in particular, to a method and an apparatus for detecting object defects, a computer device, and a storage medium.

Background

The machine vision technology is a cross discipline in many fields such as artificial intelligence, computer science, image processing, pattern recognition and the like. Machine vision is to use a machine to replace human eyes for measurement and judgment, to acquire, process and calculate images from specific real objects, and finally to perform actual detection, control and application, and is widely applied to product defect detection in the manufacturing industry.

Defects of products in the production process often have certain randomness, namely the types and the shapes and the sizes of the defects are different, but the defect detection models in the existing machine vision detection only aim at a certain specific product or a certain specific type of defects to detect, and different product defect types cannot be accurately identified.

Therefore, the conventional technology has a problem that the recognition accuracy of the product defect is poor.

Disclosure of Invention

In view of the above, it is necessary to provide an object defect detection method, apparatus, computer device, computer readable storage medium and computer program product capable of improving the product defect identification accuracy.

In a first aspect, the present application provides a method for detecting object defects. The method comprises the following steps:

acquiring a characteristic diagram corresponding to an image to be detected; the image to be detected is an image obtained by shooting a target object; the target object is an object needing defect detection;

carrying out region division on the feature map to obtain a first sub-feature map corresponding to each pixel point in the image to be detected;

inputting each first sub-feature map into a first target region classifier in one-to-one correspondence to obtain a first defect type probability corresponding to the image to be detected; the first defect type probability comprises the attribution probability of each pixel point in the image to be detected aiming at each defect type;

inputting the feature map into a target global classifier to obtain a second defect type probability corresponding to the image to be detected; the second defect type probability comprises the attribution probability of each pixel point in the image to be detected aiming at each defect type;

and determining the defect type corresponding to the target object according to the first defect type probability and the second defect type probability.

In one embodiment, the inputting each of the first sub-feature maps into a one-to-one corresponding first target region classifier to obtain a first defect type probability corresponding to the image to be detected includes:

inputting each first sub-feature map into the first target region classifier in one-to-one correspondence to obtain defect type probability corresponding to each first sub-feature map;

splicing the first sub-feature maps according to the spatial sequence of the first sub-feature maps to obtain a spliced first sub-feature map;

determining the defect type probability corresponding to the feature map according to the defect type probability corresponding to the spliced first sub-feature map; the defect type probability corresponding to the feature map comprises the attribution probability of each pixel point in the feature map aiming at each defect type;

and determining the first defect type probability according to the defect type probability corresponding to the feature map.

In one embodiment, the determining the defect type corresponding to the target object according to the first defect type probability and the second defect type probability includes:

adding the first defect type probability and the second defect type probability to obtain a target defect type probability corresponding to the image to be detected; the target defect type probability comprises a target attribution probability of each pixel point in the image to be detected aiming at each defect type;

and determining the defect type corresponding to the target object according to the target attribution probability of each pixel point in the image to be detected aiming at each defect type.

In one embodiment, the determining the defect type corresponding to the target object according to the target attribution probability of each pixel point in the image to be detected for each defect type includes:

determining the maximum target attribution probability corresponding to each pixel point in the image to be detected in the target attribution probability of each pixel point in the image to be detected aiming at each defect type;

taking the defect type corresponding to each maximum target attribution probability as the defect type corresponding to each pixel point in the image to be detected;

and determining the defect type corresponding to the target object according to the defect type corresponding to each pixel point in the image to be detected.

In one embodiment, the method further comprises:

carrying out at least one time of region division on the feature map under different scales to obtain a second sub-feature map corresponding to the image to be detected under at least one scale; the size of the second sub-feature map is larger than that of the first sub-feature map and smaller than that of the feature map;

inputting each second sub-feature map into a one-to-one corresponding second target region classifier to obtain at least one third defect type probability corresponding to the image to be detected; the at least one third defect type probability comprises the attribution probability of each pixel point in the image to be detected aiming at each defect type under the corresponding scale;

and adding the first defect type probability, the second defect type probability and the at least one third defect type probability to obtain the target defect type probability.

In one embodiment, the method further comprises:

acquiring a region classifier to be trained; the region classifier to be trained comprises local classifiers corresponding to different scales;

training the region classifier to be trained through a gradient back propagation algorithm to obtain a trained first region classifier;

and/or the presence of a gas in the gas,

training the region classifier to be trained through an exponential moving average algorithm to obtain a second region classifier which is trained;

obtaining a target region classifier according to the first region classifier and/or the second region classifier; the target region classifier includes the first target region classifier, the target global classifier, and the second target region classifier.

In one embodiment, the target global classifier comprises a vector similarity determination module and a defect type probability determination module; inputting the feature map into a target global classifier to obtain a second defect type probability corresponding to the image to be detected, wherein the second defect type probability comprises the following steps:

inputting the feature map into the vector similarity determination module to obtain target similarity between feature vectors corresponding to pixel points in the image to be detected and feature vectors corresponding to defect types; the dimensionality of the characteristic vector corresponding to each pixel point in the image to be detected is equal to the dimensionality of the characteristic vector corresponding to each defect type;

inputting the target similarity to the defect type probability determination module to obtain the second defect type probability; the second defect type probability is obtained by inputting the target similarity to a normalized multi-classification function through the defect type probability determination module.

In a second aspect, the application further provides an object defect detecting device. The device comprises:

the acquisition module is used for acquiring a characteristic diagram corresponding to an image to be detected; the image to be detected is an image obtained by shooting a target object; the target object is an object needing defect detection;

the dividing module is used for carrying out region division on the feature map to obtain a first sub-feature map corresponding to each pixel point in the image to be detected;

the first input module is used for inputting the first sub-feature maps into first target region classifiers in one-to-one correspondence to obtain first defect type probabilities corresponding to the image to be detected; the first defect type probability comprises the attribution probability of each pixel point in the image to be detected aiming at each defect type;

the second input module is used for inputting the feature map into a target global classifier to obtain a second defect type probability corresponding to the image to be detected; the second defect type probability comprises the attribution probability of each pixel point in the image to be detected aiming at each defect type;

and the determining module is used for determining the defect type corresponding to the target object according to the first defect type probability and the second defect type probability.

In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the following steps when executing the computer program:

In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of:

In a fifth aspect, the present application further provides a computer program product. The computer program product comprising a computer program which when executed by a processor performs the steps of:

inputting the first sub-feature maps into first target region classifiers in one-to-one correspondence to obtain first defect type probabilities corresponding to the image to be detected; the first defect type probability comprises the attribution probability of each pixel point in the image to be detected aiming at each defect type;

The object defect detection method, the object defect detection device, the computer equipment, the storage medium and the computer program product can be applied to any depth semantic segmentation model, and a characteristic map corresponding to an image to be detected is obtained; the image to be detected is an image obtained by shooting a target object needing defect detection; then, carrying out region division on the feature map to obtain a first sub-feature map corresponding to each pixel point in the image to be detected; inputting each first sub-feature map into a first target region classifier in one-to-one correspondence to obtain a first defect type probability corresponding to an image to be detected; the first defect type probability comprises the attribution probability of each pixel point in the image to be detected aiming at each defect type; meanwhile, inputting the feature map into a target global classifier to obtain a second defect type probability corresponding to the image to be detected; the second defect type probability comprises the attribution probability of each pixel point in the image to be detected aiming at each defect type; determining a defect type corresponding to the target object according to the first defect type probability and the second defect type probability; therefore, the first sub-feature maps corresponding to the pixel points in the image to be detected are input into the first target area classifiers in one-to-one correspondence, so that each pixel point in the image to be detected has one corresponding first target area classifier to carry out defect type classification, and therefore the local detail information of the image to be detected can be fully and effectively utilized to carry out defect type identification on the target object in the image to be detected, and the first defect type probability corresponding to the image to be detected is obtained; the feature map is directly input into the target global classifier, so that the global information of the image to be detected can be fully utilized to identify the defect type of the target object, and the second defect type probability corresponding to the image to be detected is obtained; finally, determining the defect type corresponding to the target object according to the first defect type probability and the second defect type probability corresponding to the image to be detected, so that the situation that the defect type cannot be accurately identified due to the fact that local information of the image to be detected is excessively concerned when the defect type is identified only by using the first defect type probability and when the surface defect of the target object is large, the defect position cannot be effectively and completely positioned can be prevented; the situation that the defect type of the target object cannot be accurately identified due to the fact that the local information of the image to be detected cannot be sufficiently and effectively utilized when the defect type is identified only by using the second defect type probability can also be prevented; and the advantages of the two conditions are fully combined, and the precision of identifying the defect type of the target object is improved.

Drawings

FIG. 1 is a schematic flow chart illustrating a method for detecting object defects according to an embodiment;

FIG. 2 is a flowchart illustrating the step of obtaining a probability of a first defect type in one embodiment;

FIG. 3 is a schematic flowchart illustrating a method for detecting object defects according to another embodiment;

FIG. 4 is a block diagram of an object defect detecting apparatus according to an embodiment;

FIG. 5 is a diagram illustrating an internal structure of a computer device according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of and not restrictive on the broad application.

It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

In one embodiment, as shown in FIG. 1, an object defect detection method is provided that can be applied to any depth semantic segmentation model. The embodiment is illustrated by applying the method to the server, and it can be understood that the method can also be applied to the terminal, and can also be applied to a system comprising the terminal and the server, and is implemented by interaction between the terminal and the server. In this embodiment, the method includes the steps of:

and step S110, acquiring a characteristic diagram corresponding to the image to be detected.

The image to be detected is an image obtained by shooting a target object.

The target object is an object needing defect detection.

In the specific implementation, the server can acquire an image to be detected obtained by shooting a target object needing defect detection, input the image to be detected into a trained deep semantic segmentation model, and perform feature extraction on the image to be detected through a feature extraction function by a feature extraction module in the deep semantic segmentation model to output a feature map.

In practical application, the feature extraction function may be G, if the input image to be detected is I, the feature map output by the feature extraction module is X, and the calculation formula is as follows:

X=G（I）

wherein, the size of the characteristic diagram X is [ h, w, d ], h represents the height of the characteristic diagram, w represents the width of the characteristic diagram, and d represents the number of characteristic channels of the characteristic diagram.

And step S120, carrying out region division on the feature map to obtain a first sub-feature map corresponding to each pixel point in the image to be detected.

In specific implementation, the server may perform region division on the feature map to obtain a first sub-feature map corresponding to each pixel point in the image to be detected.

In practical application, if the feature map X is divided into H × W independent regions, the sub-feature map corresponding to each region is Xi, i ∈ [1,2, …, H × W ], the size of each Xi is [ kh, kw, d ], kh represents the height of the sub-feature map, kw represents the width of the sub-feature map, and d represents the number of feature channels of the sub-feature map; then H = H/kh, W = W/kw, and kh =1, kw =1 in the first sub-feature map Xi corresponding to each pixel point in the image to be detected.

Step S130, inputting each first sub-feature map into a first target region classifier corresponding to one, and obtaining a first defect type probability corresponding to the image to be detected.

The first defect type probability comprises the attribution probability of each pixel point in the image to be detected aiming at each defect type.

In practical application, the first target region classifier may also be named as a region-aware prototype.

In the specific implementation, the size of the target classifier P used by the deep semantic segmentation model in the scheme is [ H × W, n, d ], and n represents the number of defect types to be predicted, that is, there are n defect types. After the feature map is divided into H W independent areas, the area of the feature map where the defect type prediction is carried out by the target area classifier Pi (i belongs to {1,2, …, H W }) unique to each area in the target classifier P is an area with the size of [ kh, kw, d ]; wherein Pi is [1, n, d ].

The server inputs the first sub-feature maps into the first target area classifiers corresponding to one another, and the first defect type probability corresponding to the image to be detected can be obtained through the defect type probability corresponding to the first sub-feature maps output by the first target area classifiers, wherein the first defect type probability comprises the attribution probability of each pixel point in the image to be detected aiming at each defect type.

And step S140, inputting the feature map into a target global classifier to obtain a second defect type probability corresponding to the image to be detected.

And the second defect type probability comprises the attribution probability of each pixel point in the image to be detected aiming at each defect type.

In specific implementation, the server may directly input the feature map into a globally shared target global classifier to obtain a defect type probability output by the target global classifier for the feature map, so that a second defect type probability corresponding to the image to be detected may be obtained based on the defect type probability, where the second defect type probability includes an attribution probability of each pixel point in the image to be detected for each defect type.

And S150, determining the defect type corresponding to the target object according to the first defect type probability and the second defect type probability.

In the specific implementation, the server can determine the target defect type probability corresponding to the image to be detected according to the first defect type probability and the second defect type probability, wherein the target defect type probability comprises the target attribution probability of each pixel point in the image to be detected for each defect type, and therefore the server can determine the defect type corresponding to the target object based on the target attribution probability of each pixel point in the image to be detected for each defect type.

Specifically, the defect type corresponding to the target object may be a surface defect type of the target object, such as a crack, a silver streak, a groove, a ripple mark, and embrittlement.

The object defect detection method can be applied to any depth semantic segmentation model, and a characteristic image corresponding to an image to be detected is obtained; the image to be detected is an image obtained by shooting a target object needing defect detection; then, carrying out region division on the feature map to obtain a first sub-feature map corresponding to each pixel point in the image to be detected; inputting each first sub-feature map into a first target region classifier in one-to-one correspondence to obtain a first defect type probability corresponding to an image to be detected; the first defect type probability comprises the attribution probability of each pixel point in the image to be detected aiming at each defect type; meanwhile, inputting the feature map into a target global classifier to obtain a second defect type probability corresponding to the image to be detected; the second defect type probability comprises the attribution probability of each pixel point in the image to be detected aiming at each defect type; determining a defect type corresponding to the target object according to the first defect type probability and the second defect type probability; therefore, the first sub-feature maps corresponding to the pixel points in the image to be detected are input into the first target area classifiers in one-to-one correspondence, so that each pixel point in the image to be detected has one corresponding first target area classifier to carry out defect type classification, and therefore the local detail information of the image to be detected can be fully and effectively utilized to carry out defect type identification on the target object in the image to be detected, and the first defect type probability corresponding to the image to be detected is obtained; by directly inputting the feature map into the target global classifier, the global information of the image to be detected can be fully utilized to identify the defect type of the target object, and a second defect type probability corresponding to the image to be detected is obtained; finally, determining the defect type corresponding to the target object according to the first defect type probability and the second defect type probability corresponding to the image to be detected, so that the situation that the defect type cannot be accurately identified due to the fact that local information of the image to be detected is excessively concerned when the defect type is identified only by using the first defect type probability and when the surface defect of the target object is large, the defect position cannot be effectively and completely positioned can be prevented; the situation that the defect type of the target object cannot be accurately identified due to the fact that the local information of the image to be detected cannot be sufficiently and effectively utilized when the defect type is identified only by using the second defect type probability can also be prevented; and the advantages of the two conditions are fully combined, and the precision of identifying the defect type of the target object is improved.

In one embodiment, as shown in fig. 2, step S130 includes:

step S210, inputting each first sub-feature map into the first target region classifier corresponding to one another, and obtaining the defect type probability corresponding to each first sub-feature map.

In a specific implementation, in the process that the server inputs each first sub-feature map into the first target region classifiers corresponding to one to obtain the first defect type probabilities corresponding to the images to be detected, the server may input each first sub-feature map into the first target region classifiers corresponding to one to obtain the defect type probabilities corresponding to each first sub-feature map output by each first target region classifier.

In practical application, the probability of the defect type corresponding to each sub-feature map may be

，

Wherein the content of the first and second substances,

the attribution probability of each pixel point (in this embodiment, the first sub-feature graph) in the sub-feature graph for each defect type is represented, and the size is [ kh, kw, n ]]The softmax is a normalized multi-classification function, f is a similarity calculation function, and is used to determine the similarity between the d-dimensional feature vector corresponding to each sub-feature map (in this embodiment, each first sub-feature map) after the feature map is divided and the d-dimensional feature vector corresponding to each defect type, and the similarity calculation mode may be any vector similarity calculation mode, such as vector direct point multiplication, cosine similarity between vectors, and the like, which is not limited herein.

And step S220, splicing the first sub-feature maps according to the spatial sequence of the first sub-feature maps to obtain the spliced first sub-feature maps.

In a specific implementation, after the server obtains the probability of the defect type corresponding to each first sub-feature map output by each first target region classifier, the server may splice each first sub-feature map according to the spatial sequence of each first sub-feature map to obtain a spliced first sub-feature map.

And step S230, determining the defect type probability corresponding to the characteristic diagram according to the defect type probability corresponding to the spliced first sub-characteristic diagrams.

The defect type probability corresponding to the feature map comprises the attribution probability of each pixel point in the feature map aiming at each defect type.

In specific implementation, after the server obtains the spliced first sub-feature maps, the server may determine the defect type probability corresponding to the feature map according to the defect type probability corresponding to each spliced first sub-feature map, where the defect type probability corresponding to the feature map includes the attribution probability of each pixel point in the feature map for each defect type.

Step S240, determining a first defect type probability according to the defect type probability corresponding to the feature map.

In the specific implementation, because the feature map output by the deep semantic segmentation model is consistent with the width and the height of the input original image (i.e., the image to be detected), after the server obtains the probability of the defect type corresponding to the feature map, the server can determine the attribution probability of each pixel point in the image to be detected for each defect type according to the attribution probability of each pixel point in the feature map for each defect type, so as to obtain the first defect type probability.

According to the technical scheme of the embodiment, the defect type probability corresponding to each first sub-feature map is obtained by inputting each first sub-feature map into the first target region classifiers corresponding to one another; splicing the first sub-feature graphs according to the spatial sequence of the first sub-feature graphs to obtain spliced first sub-feature graphs; determining the defect type probability corresponding to the characteristic graph according to the defect type probability corresponding to each spliced first sub-characteristic graph; the defect type probability corresponding to the feature map comprises the attribution probability of each pixel point in the feature map aiming at each defect type; finally, determining the probability of the first defect type according to the probability of the defect type corresponding to the characteristic diagram; therefore, the first sub-feature maps corresponding to the pixel points in the image to be detected are input into the first target area classifiers in one-to-one correspondence, so that each pixel point in the image to be detected has one corresponding first target area classifier to classify the defect type, the defect type of the target object in the image to be detected can be identified by fully and effectively utilizing the local detail information of the image to be detected, and the local detail capturing capability in the defect type identification process of the target object is improved.

In one embodiment, determining the defect type corresponding to the target object according to the first defect type probability and the second defect type probability includes: adding the first defect type probability and the second defect type probability to obtain a target defect type probability corresponding to the image to be detected; and determining the defect type corresponding to the target object according to the target attribution probability of each pixel point in the image to be detected aiming at each defect type.

The target defect type probability comprises target attribution probability of each pixel point in the image to be detected aiming at each defect type.

In the specific implementation, in the process that the server determines the defect type corresponding to the target object according to the first defect type probability and the second defect type probability, the server can add the first defect type probability and the second defect type probability to obtain the target defect type probability corresponding to the image to be detected, wherein the target defect type probability comprises the target attribution probability of each pixel point in the image to be detected aiming at each defect type; and then, determining the defect type corresponding to the target object according to the target attribution probability of each pixel point in the image to be detected aiming at each defect type.

In practical applications, the probability of the first defect type may be

The second defect type probability may be

And the probability of the target defect type can be y, then:

according to the technical scheme of the embodiment, the target defect type probability corresponding to the image to be detected is obtained by adding the first defect type probability and the second defect type probability; the target defect type probability comprises target attribution probability of each pixel point in the image to be detected aiming at each defect type; determining a defect type corresponding to a target object according to the target attribution probability of each pixel point in the image to be detected aiming at each defect type; therefore, the situation that the defect type cannot be accurately identified due to the fact that local information of the image to be detected is excessively concerned when the defect on the surface of the target object is large when the defect on the surface of the target object is only identified by using the first defect type probability can be prevented; the defect type of the target object can be prevented from being accurately identified due to the fact that local information of the image to be detected cannot be fully and effectively utilized when the defect type is identified only by using the second defect type probability; and the advantages of the two conditions are fully combined, and the precision of identifying the defect type of the target object is improved.

In one embodiment, determining the defect type corresponding to the target object according to the target attribution probability of each pixel point in the image to be detected for each defect type includes: determining the maximum target attribution probability corresponding to each pixel point in the image to be detected in the target attribution probability of each pixel point in the image to be detected aiming at each defect type; taking the defect type corresponding to each maximum target attribution probability as the defect type corresponding to each pixel point in the image to be detected; and determining the defect type corresponding to the target object according to the defect type corresponding to each pixel point in the image to be detected.

In the specific implementation, in the process of determining the defect type corresponding to the target object according to the target attribution probability of each pixel point in the image to be detected for each defect type, the server can determine the maximum target attribution probability corresponding to each pixel point in the image to be detected in the target attribution probability of each pixel point in the image to be detected for each defect type, so that the defect type corresponding to each maximum target attribution probability can be used as the defect type corresponding to each pixel point in the image to be detected, and the defect type corresponding to each pixel point forming the target object in the image to be detected can be determined according to the defect type corresponding to each pixel point in the image to be detected, and further the defect type of the target object can be determined.

According to the technical scheme of the embodiment, the maximum target attribution probability corresponding to each pixel point in the image to be detected is determined in the target attribution probability of each pixel point in the image to be detected aiming at each defect type; taking the defect type corresponding to each maximum target attribution probability as the defect type corresponding to each pixel point in the image to be detected; determining a defect type corresponding to a target object according to the defect type corresponding to each pixel point in an image to be detected; in this way, the defect type to which the maximum target attribution probability corresponding to each pixel point in the image to be detected belongs is used as the defect type corresponding to each pixel point in the image to be detected; therefore, the defect type corresponding to the target object can be accurately determined according to the defect type corresponding to each pixel point forming the target object, and the identification accuracy of the defect type of the target object is improved.

In one embodiment, the method further comprises: carrying out at least one time of region division on the feature map under different scales to obtain a second sub-feature map corresponding to the image to be detected under at least one scale; inputting each second sub-feature map into a one-to-one corresponding second target region classifier to obtain at least one third defect type probability corresponding to the image to be detected; the at least one third defect type probability comprises the attribution probability of each defect type aiming at each pixel point in the image to be detected under the corresponding scale; and adding the first defect type probability, the second defect type probability and at least one third defect type probability to obtain a target defect type probability.

And the size of the second sub-feature map is larger than that of the first sub-feature map and smaller than that of the feature map.

In a specific implementation, in the process of determining the probability of the target defect type corresponding to the image to be detected, the server may further perform at least one time of region division on the feature map under different scales to obtain a second sub-feature map corresponding to the image to be detected under at least one scale, where the size of the second sub-feature map is larger than that of the first sub-feature map and smaller than that of the feature map.

In practical application, the size of each second sub-feature map Xi corresponding to at least one scale is [ kh, kw, d ], and the value of kh and kw may be 2,4,8, and the like, and the value of the value needs to be associated with the actual condition of the project, and needs to be larger than the size of the first sub-feature map (kh =1, kw = 1) and smaller than the size of the feature map (kh = h, kw = w), and the present solution does not specifically limit this.

Then, inputting the second sub-feature maps into one-to-one corresponding second target region classifiers to obtain the defect type probability output by the corresponding second target region classifiers under different scales

Obtaining at least one third defect type probability corresponding to the image to be detected, wherein the at least one third defect type probability comprises the attribution probability of each pixel point in the image to be detected aiming at each defect type under the corresponding scale; and finally, the server adds the first defect type probability, the second defect type probability and the corresponding third defect type probability under different scales to obtain the target defect type probability corresponding to the image to be detected.

According to the technical scheme of the embodiment, the second sub-feature map corresponding to the image to be detected under at least one scale is obtained by carrying out at least one time of region division under different scales on the feature map; the size of the second sub-feature map is larger than that of the first sub-feature map and smaller than that of the feature map; inputting each second sub-feature map into a one-to-one corresponding second target region classifier to obtain at least one third defect type probability corresponding to the image to be detected; the at least one third defect type probability comprises the attribution probability of each defect type aiming at each pixel point in the image to be detected under the corresponding scale; adding the first defect type probability, the second defect type probability and at least one third defect type probability to obtain a target defect type probability; in this way, the feature map is divided in multiple scales, the divided sub-feature maps are input into the second target region classifiers which correspond to one another, the corresponding third defect type probabilities of the feature map under different scales are obtained through the output results of the second target region classifiers, and the third defect type probabilities are added with the first defect type probabilities which fully utilize the local information of the feature map and the second defect type probabilities which fully utilize the global information of the feature map to obtain the target defect type probabilities; therefore, semantic content of the classifier can be changed according to the change of the content of the feature maps of different areas, feature information on the feature maps of different sizes can be fully extracted and fused, feature details of a target object in an image to be detected are enriched, the defect type of the target object can be accurately identified through the probability of the target defect type corresponding to the image to be detected, and the self-adaptive perception capability of the classifier on different areas of each image to be detected is enhanced.

In one embodiment, the method further comprises: acquiring a region classifier to be trained; training the region classifier to be trained through a gradient back propagation algorithm to obtain a trained first region classifier; and/or training the region classifier to be trained through an exponential moving average algorithm to obtain a second region classifier which is trained; and obtaining a target region classifier according to the first region classifier and/or the second region classifier.

The region classifier to be trained comprises local classifiers corresponding to different scales.

The target area classifier comprises a first target area classifier, a target global classifier and a second target area classifier.

In specific implementation, the server may further obtain corresponding to-be-trained region classifiers of the feature map under different scales, train the to-be-trained region classifier through a gradient back propagation algorithm, and obtain a trained first region classifier, where the first region classifier includes d-dimensional feature vector representations corresponding to the defect types.

In practical applications, the first region classifier may be

Defining the region classifier to be trained as a learnable parameter, and enabling the network to optimize automatically to obtain a trained first region classifier

. In each iteration of the training, the training sequence,

is updated by gradient back-propagation.

The server can also train the region classifier to be trained through an Exponential Moving Average (Exponential Moving Average) algorithm to obtain a second region classifier which is trained

The second region classifier includes d-dimensional feature vector representations corresponding to each defect type.

In the practical application of the method, the air conditioner,

the update process of (2) is as follows:

wherein ti represents the ith [ kh, kw, d ] in the sample feature map]The defect type sample label corresponding to the region Xi has a ti size [ kh, kw, n]A one hot vector (one mask) including n defect types; m represents a feature processing function that functions to process Xi to a size of [1, n, d ] by mask pooling]The feature vector of (2);

the predicted value output by the training at this time is shown,

representing a region classifier obtained by exponential moving average corresponding to the ith region in the sample feature map; γ is the weight of the moving average and may be 0.999.

Finally, the server can obtain a target region classifier according to the first region classifier and/or the second region classifier; the target region classifier includes a first target region classifier, a target global classifier, and a second target region classifier.

In practical application, the first zone can be usedDomain classifier

Obtaining a target region classifier Pi, i.e. Pi =

(ii) a The target area classifier Pi, i.e. Pi =, may also be obtained from the second area classifier

(ii) a Preferably, the target region classifier Pi may be obtained by a first region classifier and a second region classifier, i.e. the first region classifier and the second region classifier

。

In practical applications, the target region classifier may be a last layer 1 × 1 convolutional layer or a fully connected layer in the target depth semantic segmentation model.

According to the technical scheme of the embodiment, the corresponding region classifiers to be trained of the characteristic diagram under different scales are obtained; training the region classifier to be trained through a gradient back propagation algorithm to obtain a trained first region classifier; and/or training the region classifier to be trained through an exponential moving average algorithm to obtain a second region classifier which is trained; obtaining a target region classifier according to the first region classifier and/or the second region classifier; therefore, the target area classifier can be obtained through training by various methods, the advantages of the various methods are combined, the accuracy of the target area classifier in identifying the defect types can be improved, and meanwhile, the method for obtaining the target area classifier is more diversified.

In one embodiment, the target global classifier includes a vector similarity determination module and a defect type probability determination module; inputting the feature map into a target global classifier to obtain a second defect type probability corresponding to the image to be detected, wherein the second defect type probability comprises the following steps: inputting the feature map into a vector similarity determination module to obtain target similarity between feature vectors corresponding to all pixel points in the image to be detected and feature vectors corresponding to all defect types; and inputting the target similarity into a defect type probability determining module to obtain a second defect type probability.

And the dimensionality of the characteristic vector corresponding to each pixel point in the image to be detected is equal to the dimensionality of the characteristic vector corresponding to each defect type.

And the second defect type probability is obtained by inputting the target similarity to the normalized multi-classification function through the defect type probability determining module.

In specific implementation, the target global classifier comprises a vector similarity determining module and a defect type probability determining module; in the process that the server inputs the feature map into the target global classifier to obtain the probability of the second defect type corresponding to the image to be detected, the server can input the feature map into the vector similarity determining module to obtain the target similarity between the feature vector corresponding to each pixel point in the image to be detected and the feature vector corresponding to each defect type; the dimensionality of the feature vector corresponding to each pixel point in the image to be detected is equal to the dimensionality of the feature vector corresponding to each defect type; inputting the target similarity into a defect type probability determining module to obtain a second defect type probability; and the second defect type probability is obtained by inputting the target similarity to the normalized multi-classification function through the defect type probability determining module.

In practical application, the target global classifier may be C, the feature map may be X, and the probability of the second defect type is

Then:

wherein f represents a similarity calculation function for determining the target similarity between the d-dimensional feature vector corresponding to each pixel point in the feature map and the d-dimensional feature vector corresponding to each defect type, and the similarity calculation mode may be any vector similarity calculation mode, such as direct point multiplication of vectors, cosine similarity between vectors, and the like, and is not limited herein; softmax is a normalized multi-classification function.

Wherein, when the last layer 1x1 convolution layer of the target depth semantic segmentation model is regarded as a classifier, the size of the target global classifier C is [ n, d ]]Representing each defect type of the n defect types by a d-dimensional feature vector;

has a size of [ h, w, n]And representing the attribution probability of each pixel point aiming at each defect type.

According to the technical scheme of the embodiment, the feature map is input to the vector similarity determining module, so that the target similarity between the feature vector corresponding to each pixel point in the image to be detected and the feature vector corresponding to each defect type is obtained; inputting the target similarity into a defect type probability determining module to obtain a second defect type probability; therefore, when the surface defect of the target object is large, the defect position can be effectively and completely positioned through the target global classifier, and the defect type corresponding to the target object is accurately identified through the output second defect type probability.

In another embodiment, as shown in fig. 3, an object defect detecting method is provided, which is described by taking the method as an example for being applied to a server, and includes the following steps:

and step S302, acquiring a characteristic diagram corresponding to the image to be detected.

And step S304, carrying out region division on the feature map to obtain a first sub-feature map corresponding to each pixel point in the image to be detected.

Step S306, inputting each first sub-feature map into the first target region classifier corresponding to one, and obtaining the defect type probability corresponding to each first sub-feature map.

And step S308, splicing the first sub-feature maps according to the spatial sequence of the first sub-feature maps to obtain the spliced first sub-feature maps.

And S310, determining the defect type probability corresponding to the characteristic diagram according to the defect type probability corresponding to the spliced first sub-characteristic diagrams.

Step S312, determining the probability of the first defect type corresponding to the image to be detected according to the probability of the defect type corresponding to the characteristic diagram.

Step S314, inputting the feature map into a vector similarity determination module to obtain the target similarity between the feature vector corresponding to each pixel point in the image to be detected and the feature vector corresponding to each defect type.

And step S316, inputting the target similarity into a defect type probability determining module to obtain a second defect type probability corresponding to the image to be detected.

And step S318, adding the first defect type probability and the second defect type probability to obtain the target attribution probability of each pixel point in the image to be detected aiming at each defect type.

And step S320, determining the defect type corresponding to the target object according to the target attribution probability of each pixel point in the image to be detected aiming at each defect type.

It should be noted that, the specific limitations of the above steps can be referred to the above specific limitations of the object defect detection method.

It should be understood that, although the steps in the flowcharts related to the embodiments as described above are sequentially displayed as indicated by arrows, the steps are not necessarily performed sequentially as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the flowcharts related to the embodiments described above may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the execution order of the steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a part of the steps or stages in other steps.

Based on the same inventive concept, the embodiment of the present application further provides an object defect detecting apparatus for implementing the object defect detecting method. The implementation scheme for solving the problem provided by the apparatus is similar to the implementation scheme described in the method, so the specific limitations in one or more embodiments of the object defect detecting apparatus provided below may refer to the limitations on an object defect detecting method in the foregoing, and are not described herein again.

In one embodiment, as shown in fig. 4, there is provided an object defect detecting apparatus including: an obtaining module 410, a dividing module 420, a first input module 430, a second input module 440, and a determining module 450, wherein:

an obtaining module 410, configured to obtain a feature map corresponding to an image to be detected; the image to be detected is an image obtained by shooting a target object; the target object is an object needing defect detection.

And the dividing module 420 is configured to perform region division on the feature map to obtain a first sub-feature map corresponding to each pixel point in the image to be detected.

A first input module 430, configured to input each of the first sub-feature maps into a first target region classifier corresponding to one another, so as to obtain a first defect type probability corresponding to the image to be detected; the first defect type probability comprises the attribution probability of each pixel point in the image to be detected aiming at each defect type.

The second input module 440 is configured to input the feature map to a target global classifier, so as to obtain a second defect type probability corresponding to the image to be detected; and the second defect type probability comprises the attribution probability of each pixel point in the image to be detected aiming at each defect type.

A determining module 450, configured to determine a defect type corresponding to the target object according to the first defect type probability and the second defect type probability.

In one embodiment, the first input module 430 is specifically configured to input each of the first sub-feature maps into the first target region classifiers in one-to-one correspondence, so as to obtain a defect type probability corresponding to each of the first sub-feature maps; splicing the first sub-feature maps according to the spatial sequence of the first sub-feature maps to obtain a spliced first sub-feature map; determining the defect type probability corresponding to the feature map according to the defect type probability corresponding to the spliced first sub-feature map; the defect type probability corresponding to the feature map comprises the attribution probability of each pixel point in the feature map aiming at each defect type; and determining the first defect type probability according to the defect type probability corresponding to the feature map.

In one embodiment, the determining module 450 is specifically configured to add the first defect type probability and the second defect type probability to obtain a target defect type probability corresponding to the image to be detected; the target defect type probability comprises a target attribution probability of each pixel point in the image to be detected aiming at each defect type; and determining the defect type corresponding to the target object according to the target attribution probability of each pixel point in the image to be detected aiming at each defect type.

In one embodiment, the determining module 450 is specifically configured to determine a maximum target attribution probability corresponding to each pixel point in the image to be detected in the target attribution probabilities of each pixel point in the image to be detected for each defect type; taking the defect type corresponding to each maximum target attribution probability as the defect type corresponding to each pixel point in the image to be detected; and determining the defect type corresponding to the target object according to the defect type corresponding to each pixel point in the image to be detected.

In one embodiment, the apparatus further comprises: the second sub-feature map acquisition module is used for carrying out at least one time of region division on the feature map under different scales to obtain a second sub-feature map corresponding to the image to be detected under at least one scale; the size of the second sub-feature map is larger than that of the first sub-feature map and smaller than that of the feature map; the third input module is used for inputting each second sub-feature map into a one-to-one corresponding second target region classifier to obtain at least one third defect type probability corresponding to the image to be detected; the at least one third defect type probability comprises the attribution probability of each defect type aiming at each pixel point in the image to be detected under the corresponding scale; and the adding module is used for adding the first defect type probability, the second defect type probability and the at least one third defect type probability to obtain the target defect type probability.

In one embodiment, the apparatus further comprises: the classifier acquisition module is used for acquiring a region classifier to be trained; the region classifier to be trained comprises local classifiers corresponding to different scales; the first training module is used for training the region classifier to be trained through a gradient back propagation algorithm to obtain a trained first region classifier; and/or the second training module is used for training the region classifier to be trained through an exponential moving average algorithm to obtain a trained second region classifier; a target region classifier determining module, configured to obtain a target region classifier according to the first region classifier and/or the second region classifier; the target region classifier includes the first target region classifier, the target global classifier, and the second target region classifier.

In one embodiment, the target global classifier comprises a vector similarity determination module and a defect type probability determination module; the second input module 440 is specifically configured to input the feature map to the vector similarity determination module, so as to obtain a target similarity between a feature vector corresponding to each pixel point in the image to be detected and a feature vector corresponding to each defect type; the dimensionality of the characteristic vector corresponding to each pixel point in the image to be detected is equal to the dimensionality of the characteristic vector corresponding to each defect type; inputting the target similarity to the defect type probability determination module to obtain the second defect type probability; the second defect type probability is obtained by inputting the target similarity to a normalized multi-classification function through the defect type probability determination module.

The modules in the object defect detecting device can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent of a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer equipment is used for storing the image defect detection processing data to be detected. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement an object defect detection method.

Those skilled in the art will appreciate that the architecture shown in fig. 5 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is further provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method embodiments when executing the computer program.

In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.

In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, carries out the steps in the method embodiments described above.

It should be noted that, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high-density embedded nonvolatile Memory, resistive Random Access Memory (ReRAM), Magnetic Random Access Memory (MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (PCM), graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others. The databases referred to in various embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing based data processing logic devices, etc., without limitation.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A method for detecting defects in an object, the method comprising:

inputting each first sub-feature map into a first target region classifier corresponding to one another, and obtaining a first defect type probability corresponding to the image to be detected according to the defect type probability corresponding to each first sub-feature map output by each first target region classifier; the first defect type probability comprises the attribution probability of each pixel point in the image to be detected aiming at each defect type; the first target region classifier and the first sub-feature map are equal in number;

2. The method according to claim 1, wherein obtaining the probability of the type of the defect corresponding to the image to be detected through the probability of the type of the defect corresponding to each of the first sub-feature maps output by each of the first target region classifiers comprises:

splicing the first sub-feature maps according to the spatial sequence of the first sub-feature maps, and determining the defect type probability corresponding to the spliced first sub-feature maps according to the defect type probability corresponding to the first sub-feature maps output by the first target region classifier;

3. The method of claim 1, wherein determining the defect type corresponding to the target object according to the first defect type probability and the second defect type probability comprises:

4. The method according to claim 3, wherein the determining the defect type corresponding to the target object according to the target attribution probability of each pixel point in the image to be detected for each defect type comprises:

5. The method of claim 3, further comprising:

performing at least one time of region division on the feature map under different scales to obtain a second sub-feature map corresponding to the image to be detected under at least one scale; the size of the second sub-feature map is larger than that of the first sub-feature map and smaller than that of the feature map;

inputting each second sub-feature map into a one-to-one corresponding second target region classifier to obtain at least one third defect type probability corresponding to the image to be detected; the at least one third defect type probability comprises the attribution probability of each defect type aiming at each pixel point in the image to be detected under the corresponding scale;

6. The method of claim 5, further comprising:

and/or the presence of a gas in the gas,

7. The method of claim 1, wherein the target global classifier comprises a vector similarity determination module and a defect type probability determination module; inputting the feature map into a target global classifier to obtain a second defect type probability corresponding to the image to be detected, wherein the second defect type probability comprises the following steps:

8. An object defect detection apparatus, characterized in that the apparatus comprises:

the first input module is used for inputting each first sub-feature map into a first target region classifier corresponding to one another, and obtaining a first defect type probability corresponding to the image to be detected according to the defect type probability corresponding to each first sub-feature map output by each first target region classifier; the first defect type probability comprises the attribution probability of each pixel point in the image to be detected aiming at each defect type; the first target region classifier and the first sub-feature map are equal in number;

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 7.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.