CN112801047B

CN112801047B - Defect detection method and device, electronic equipment and readable storage medium

Info

Publication number: CN112801047B
Application number: CN202110293621.3A
Authority: CN
Inventors: 刘文龙; 高斌斌
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-03-19
Filing date: 2021-03-19
Publication date: 2021-08-17
Anticipated expiration: 2041-03-19
Also published as: CN112801047A

Abstract

The application relates to the technical field of image processing, and discloses a defect detection method and device based on a computer vision technology, an electronic device and a readable storage medium, wherein the defect detection method comprises the following steps: acquiring an image to be detected and a standard image of a target to be detected, extracting a characteristic to be detected of the image to be detected, and extracting a standard characteristic of the standard image; fusing the to-be-detected features and the standard features to obtain first fused features; determining a region of interest feature in the first fused feature; and detecting defect information in the image to be detected based on the region-of-interest feature and the first fusion feature. The defect detection method provided by the application can effectively improve the detection precision of the defect information of the image of the article.

Description

Defect detection method and device, electronic equipment and readable storage medium

Technical Field

The present application relates to the field of defect detection technologies, and in particular, to a defect detection method, a defect detection apparatus, an electronic device, and a readable storage medium.

Background

With the development of artificial intelligence and computer vision technology, machine vision technology is applied more and more in industrial scenes, and occupies a higher and higher position. For example, the machine vision technology can be applied to quality control links in industrial production processes, and a mode of detecting image defects of articles in an image recognition mode is developed.

At present, for a method for detecting defects of an article image, characteristics of an area of interest in which defects may exist in an image of a target to be detected are determined first, and then defect information in the characteristics of the area of interest is further determined.

Disclosure of Invention

The purpose of the present application is to solve at least one of the above technical drawbacks, and to provide the following solutions:

in a first aspect, a defect detection method is provided, including:

acquiring an image to be detected and a standard image of a target to be detected, extracting the characteristic to be detected of the image to be detected, and extracting the standard characteristic of the standard image;

fusing the feature to be detected and the standard feature to obtain a first fused feature;

determining a region of interest feature in the first fused feature;

and detecting defect information in the image to be detected based on the region-of-interest feature and the first fusion feature.

In an alternative embodiment of the first aspect, fusing the feature to be detected with the standard feature to obtain a first fused feature, includes:

and fusing the to-be-detected features corresponding to each channel with the standard features corresponding to each channel respectively to obtain first fusion features.

In an optional embodiment of the first aspect, fusing the to-be-detected feature corresponding to each channel with the standard feature corresponding to each channel, respectively, to obtain a first fused feature, includes:

determining a first weight of the to-be-detected feature corresponding to each channel based on the fusion network, and determining a second weight of the standard feature corresponding to each channel;

and determining the weighted sum of the to-be-detected feature and the standard feature respectively corresponding to each channel based on the first weight and the second weight respectively corresponding to each channel to obtain a first fusion feature.

In an optional embodiment of the first aspect, determining the region of interest feature in the first fused feature comprises:

extracting at least one candidate region in the first fusion feature;

and performing region feature aggregation on at least one candidate region to obtain a region-of-interest feature.

In an optional embodiment of the first aspect, performing region feature aggregation on the at least one candidate region to obtain a region-of-interest feature includes:

traversing at least one candidate region, and dividing each candidate region into at least one unit;

and determining the coordinate position of each unit through bilinear interpolation and performing maximum pooling operation to obtain the characteristics of the region of interest.

In an optional embodiment of the first aspect, detecting defect information in the image to be detected based on the region-of-interest feature and the first fused feature includes:

fusing the region-of-interest feature and the first fusion feature to obtain a second fusion feature;

and carrying out target detection on the second fusion characteristics to obtain defect information aiming at the target to be detected.

In an optional embodiment of the first aspect, fusing the region of interest feature and the first fused feature to obtain a second fused feature, includes:

performing dimension conversion on the first fusion characteristic to obtain a conversion characteristic; the dimensions of the conversion feature and the feature of the region of interest are the same;

pooling the conversion characteristics to obtain pooled characteristics; the scale of the pooled features is the same as the scale of the region of interest features;

a second fused feature is determined based on the pooled features and the region of interest features.

In an alternative embodiment of the first aspect, pooling the transformed features to obtain pooled features comprises:

and performing integral pooling on the conversion characteristics based on the preset characteristic blocks and the preset characteristic graph to obtain pooling characteristics.

In an alternative embodiment of the first aspect, the defect information comprises a defect location and a defect category; carrying out target detection on the second fusion characteristics to obtain defect information aiming at the target to be detected, wherein the defect information comprises the following steps:

and performing dense local regression on the second fusion characteristics in a mode of predicting a multipoint set by the convolutional layer to obtain the defect position and the defect type of the target to be detected.

In a second aspect, a defect detection apparatus is provided, including:

the extraction module is used for acquiring an image to be detected and a standard image of a target to be detected, extracting the characteristic to be detected of the image to be detected and extracting the standard characteristic of the standard image;

the fusion module is used for fusing the to-be-detected feature and the standard feature to obtain a first fusion feature;

a determination module for determining a region of interest feature in the first fused feature;

and the detection module is used for detecting the defect information in the image to be detected based on the region-of-interest feature and the first fusion feature.

In an optional embodiment of the second aspect, the fusion module is specifically configured to, when fusing the feature to be detected and the standard feature to obtain the first fused feature:

In an optional embodiment of the second aspect, the fusion module is specifically configured to, when fusing the to-be-detected feature corresponding to each channel with the standard feature corresponding to each channel to obtain the first fusion feature:

In an optional embodiment of the second aspect, the determining module, when determining the region of interest feature in the first fusion feature, is specifically configured to:

extracting at least one candidate region in the first fusion feature;

In an optional embodiment of the second aspect, when the determining module performs region feature aggregation on at least one candidate region to obtain a region-of-interest feature, the determining module is specifically configured to:

In an optional embodiment of the second aspect, when detecting defect information in the image to be detected based on the region-of-interest feature and the first fusion feature, the detection module is specifically configured to:

In an optional embodiment of the second aspect, the detection module is specifically configured to, when the region-of-interest feature and the first fusion feature are fused to obtain a second fusion feature:

In an optional embodiment of the second aspect, when the detection module pools the conversion feature to obtain a pooled feature, the detection module is specifically configured to:

In an alternative embodiment of the second aspect, the defect information comprises a defect location and a defect category; when the detection module performs target detection on the second fusion feature to obtain defect information for the target to be detected, the detection module is specifically configured to:

In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the defect detection method shown in the first aspect of the present application is implemented.

In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, which when executed by a processor, implements the defect detection method shown in the first aspect of the present application.

The beneficial effect that technical scheme that this application provided brought is:

the first fusion characteristic is obtained by fusing the characteristic to be detected and the standard characteristic, so that the contrast information between the image to be detected and the standard image can be obtained, even if the defect on the image is irregular in size distribution, the contrast information can be embodied in the first fusion characteristic, and then the defect information is detected based on the first fusion characteristic, so that the accuracy of defect detection can be effectively improved.

Furthermore, by determining the region-of-interest feature in the first fusion feature, the approximate local range where the defect information is located can be determined, and the detection efficiency is improved.

Furthermore, the region-of-interest feature can represent a local feature, the first fusion feature can represent a global feature, and the local feature and the global feature can be effectively combined by combining the region-of-interest feature and the first fusion feature, so that the accuracy of defect detection is further improved.

Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.

Drawings

The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

FIG. 1 is a schematic flow chart of a prior art scheme for defect detection;

fig. 2 is an application scenario diagram of a defect detection method according to an embodiment of the present application;

fig. 3 is a schematic flowchart of a defect detection method according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a defect detection scheme provided by an embodiment of the present application;

FIG. 5 is a flowchart illustrating a defect detection method according to an embodiment;

FIG. 6 is a schematic diagram of a defect detection scheme provided by an embodiment of the present application;

FIG. 7 is a schematic flowchart of a defect detection method in an example provided by an embodiment of the present application;

FIG. 8a is a schematic diagram illustrating the effect of defect detection in the prior art;

FIG. 8b is a schematic diagram illustrating the effect of defect detection according to the present application;

FIG. 9a is a schematic diagram illustrating the effect of defect detection in the prior art;

FIG. 9b is a schematic diagram illustrating the effect of defect detection according to the present application;

FIG. 10a is a schematic diagram illustrating the effect of defect detection using the prior art;

FIG. 10b is a schematic diagram illustrating the effect of defect detection according to the present application;

fig. 11 is a schematic structural diagram of a defect detection apparatus according to an embodiment of the present application;

fig. 12 is a schematic structural diagram of an electronic device for defect detection according to an embodiment of the present application.

Detailed Description

Reference will now be made in detail to the embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary only for the purpose of explaining the present application and are not to be construed as limiting the present application.

As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combinations of one or more of the associated listed items.

To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

Artificial Intelligence (AI) is a theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human Intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the realization method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making.

The artificial intelligence technology is a comprehensive subject and relates to the field of extensive technology, namely the technology of a hardware level and the technology of a software level. The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, blockchains, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

Computer Vision technology (CV) is a science for researching how to make a machine see, and further means that a camera and a Computer are used for replacing human eyes to perform machine Vision such as identification, tracking and measurement on a target, and further image processing is performed, so that the Computer processing becomes an image more suitable for human eyes to observe or is transmitted to an instrument to detect. As a scientific discipline, computer vision research-related theories and techniques attempt to build artificial intelligence systems that can capture information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, three-dimensional object reconstruction, 3D technologies, virtual reality, augmented reality, synchronous positioning, map construction, and other technologies, and also include common biometric technologies such as face recognition and fingerprint recognition.

Machine Learning (ML) is a multi-domain cross discipline, and relates to a plurality of disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and the like. The special research on how a computer simulates or realizes the learning behavior of human beings so as to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve the performance of the computer. Machine learning is the core of artificial intelligence, is the fundamental approach for computers to have intelligence, and is applied to all fields of artificial intelligence. Machine learning and deep learning generally include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and formal education learning.

The scheme provided by the embodiment of the application relates to an artificial intelligence defect detection technology, and is specifically explained by the following embodiment.

In modern industrial manufacturing, production efficiency is improved by introducing a flow line. But the complicated process inevitably causes the generation of product defects. However, these defects are often dependent on environmental conditions, and are generated probabilistically, and it is necessary to perform statistical analysis on the defects at a later stage. Therefore, the method is an essential link in the modern production process for detecting and diagnosing the defects of the finished product.

In the traditional method, an enterprise mostly adopts a manual observation mode to detect the defects of products. In this example, there is a problem that the detection cost (personnel cost) is large for the enterprise; for the staff, the staff has a large loss rate due to the problems of large working intensity and single working content because the defect area is small (difficult to detect); for the algorithm, the defect forms are different in size, and the oversize defect and the undersize defect can cause a certain degree of missed detection, so that the actual production line yield is influenced.

At present, Mask R-CNN (Mask Region-Convolutional Neural Networks) is generally adopted to detect defects of an article image.

As shown in fig. 1, after extracting features to be detected from each pixel point on an image to be detected through a neural network model (backbone), a Mask R-CNN selects an ROI (region of interest) region that may have a defect, and then performs frame selection fine adjustment and judgment on specific details of the defect (such as defect type) on the specific region, so as to finally obtain an accurate defect detection frame.

The above approach generates the detection results in an end-to-end manner, but the proposed method is designed against images in ImageNet (a large visualization database for visual object recognition software research). The recognition objects (defect areas) in the application are rich in expression form, and the defect size distribution is irregular on each picture. The method is different from the characteristics that the object body in ImageNet is clear and the region is single. Therefore, the direct use of the method can result in low recognition accuracy and unsatisfactory recognition.

The defect detection method, device, electronic equipment and computer-readable storage medium provided by the application aim to solve the technical problems in the prior art.

The following describes the technical solutions of the present application and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.

As shown in fig. 2, the defect detection method of the present application may be applied to the scene shown in fig. 2, specifically, the image acquisition device 201 acquires an image to be detected of an object to be detected, and sends the image to be detected to the server 202, the server 202 stores a standard image of the object to be detected, and the server 202 extracts a feature to be detected of the image to be detected and extracts a standard feature of the standard image; the server 202 fuses the to-be-detected features and the standard features to obtain first fused features; the server 202 determines the region-of-interest feature in the first fusion feature, detects defect information in the image to be detected based on the region-of-interest feature and the first fusion feature, and sends the defect information to the terminal 203.

In the scenario shown in fig. 2, the defect detection method may be performed in the server, or in another scenario, may be performed in the terminal. For example, the terminal can have image acquisition and defect detection functions, acquires an image to be detected and a standard image, and performs defect detection based on the image to be detected and the standard image.

Those skilled in the art will understand that the "terminal" used herein may be a Mobile phone, a tablet computer, a PDA (Personal Digital Assistant), an MID (Mobile Internet Device), etc.; a "server" may be implemented as a stand-alone server or as a server cluster comprised of multiple servers.

A possible implementation manner is provided in the embodiment of the present application, and as shown in fig. 3, a defect detection method is provided, which may be applied to a terminal or a server, and may include the following steps:

step S301, an image to be detected and a standard image of a target to be detected are obtained, the characteristic to be detected of the image to be detected is extracted, and the standard characteristic of the standard image is extracted.

The image to be detected may be an acquired real-time image for the target to be detected, and the image to be detected may include defect information for the target to be detected.

Taking the target to be detected as the mobile phone camera bracket accessory as an example, the image to be detected may include the defect information of pressure damage, sticking, material shortage or dirt and the like of the mobile phone camera bracket accessory.

Wherein the standard image is an image containing no defective information for the object to be detected.

Specifically, the server or the terminal for performing defect detection may acquire the image to be detected of the target to be detected from the image acquisition device, and the server or the terminal for performing defect detection may pre-store a standard image for the target to be detected.

Specifically, the process of extracting the features to be detected of the image to be detected and extracting the standard features of the standard image can be understood as the process of extracting the depth semantic features of the image.

In this embodiment, an HRNet (high resolution retention network) may be used to extract the feature to be detected of the image to be detected and the standard feature of the standard image, that is, both the image to be detected and the standard image are input into the HRNet to perform feature extraction, so as to obtain the feature to be detected and the standard feature.

Specifically, the HRNet is used for feature extraction, so that high resolution can be kept, and small target detection is facilitated.

In other embodiments, other networks may also be used for feature extraction, for example, a Residual Network (ResNet) is used for feature extraction, and a Residual block inside the Residual Network uses jump connection, so that the problem of gradient disappearance caused by increasing depth in a deep neural Network is alleviated, and the accuracy of feature extraction can be improved; the Incep network can also be used for feature extraction, and can also avoid the phenomenon that the network is over-fitted and the gradient disappears when the network has a deep depth.

Step S302, fusing the feature to be detected and the standard feature to obtain a first fused feature.

Specifically, the difference between the feature to be detected and the standard feature may be compared to obtain a first fusion feature, the first fusion feature may be obtained by performing weighted summation between the feature to be detected and the standard feature, and the first fusion feature may be obtained by performing weighted summation between the feature to be detected and the standard feature in a channel-by-channel manner.

The specific process of determining the first fusion characteristic will be described in detail below.

Step S303, determining the region-of-interest feature in the first fusion feature.

The region of interest is a region selected from image features, the region is a key point concerned by image feature analysis, the region is defined for further processing, and the ROI is used for defining the key region, so that the processing time can be reduced, and the accuracy can be improved.

Specifically, the feature of the region of interest may include defect information for the target to be detected, the feature of the region of interest may be determined first, and then the defect information is further detected according to the feature of the region of interest, and a process of specifically determining the feature of the region of interest will be described in detail below.

And step S304, detecting defect information in the image to be detected based on the region-of-interest feature and the first fusion feature.

In some embodiments, defect information in an image to be detected can be detected directly based on the region-of-interest features.

In other embodiments, the feature of the region of interest is local information, and the feature of the region of interest and the first fusion feature may be further fused, and the defect information is detected based on the fused feature by combining the local information and the global information.

The detection method specific to the defect information will be described in detail below.

As shown in fig. 4, extracting the feature to be detected of the image to be detected, extracting the standard feature of the standard image, and fusing the feature to be detected and the standard feature to obtain a first fused feature; determining a region of interest feature in the first fused feature; and detecting defect information in the image to be detected based on the region-of-interest feature and the first fusion feature.

In the above embodiment, the first fusion feature is obtained by fusing the feature to be detected and the standard feature, so that the comparison information between the image to be detected and the standard image can be obtained, even if the defect size distribution on the image is irregular, the comparison information can be embodied in the first fusion feature, and then the defect information is detected based on the first fusion feature, so that the accuracy of defect detection can be effectively improved.

In addition, by determining the region-of-interest feature in the first fusion feature, the approximate local range where the defect information is located can be determined, and the detection efficiency is improved; the region-of-interest feature can represent a local feature, the first fusion feature can represent a global feature, and the local feature and the global feature can be effectively combined by combining the region-of-interest feature and the first fusion feature, so that the accuracy of defect detection is further improved.

A specific process of obtaining the first fusion characteristics will be described below with reference to specific embodiments.

A possible implementation manner is provided in the embodiment of the present application, the fusing the feature to be detected and the standard feature in step S302 to obtain a first fused feature may include:

In this embodiment, the features to be detected and the standard features may be fused in a channel-by-channel manner.

Specifically, the to-be-detected feature corresponding to each channel is fused with the standard feature corresponding to each channel, so as to obtain a first fusion feature, which may include:

(1) determining a first weight of the to-be-detected feature corresponding to each channel based on the fusion network, and determining a second weight of the standard feature corresponding to each channel;

(2) and determining the weighted sum of the to-be-detected feature and the standard feature respectively corresponding to each channel based on the first weight and the second weight respectively corresponding to each channel to obtain a first fusion feature.

Specifically, the converged network may be converged by using the following formula:

（1）

in the above formula, the first and second carbon atoms are,

representing a first fused feature;

representing the feature to be detected;

representing a standard feature; n represents the number of channels of the characteristics to be detected and the standard characteristics; for example, N may be 256;

is a first weight value;

is the second weight value;

and

the feature dimension of the standard feature is equal to the dimension of the feature to be detected and the standard feature; in addition to this, the present invention is,

and

are learnable parameters of the converged network. The method has the advantages that the fusion network can independently learn the feature fusion mode, and meanwhile, the weighting feature fusion mode can achieve the purpose of automatically screening features.

In a specific implementation process, a fusion network can be obtained by training first, the feature to be detected and the standard feature are input into the fusion network, the fusion network determines a first weight of the feature to be detected and a second weight of the standard feature respectively, and the feature to be detected and the standard feature are fused based on the first weight and the second weight to obtain a second fusion feature.

The above embodiments illustrate a specific process of acquiring the first fusion feature, and a specific process of determining the region of interest feature will be further described below with reference to the drawings and the embodiments.

A possible implementation manner is provided in the embodiment of the present application, and the determining the region-of-interest feature in the first fusion feature in step S303 may include:

(1) at least one candidate region in the first fused feature is extracted.

Specifically, an ROI Region that may have defects may be extracted using an RPN (Region-selective Network).

(2) And performing region feature aggregation on at least one candidate region to obtain a region-of-interest feature.

Specifically, performing region feature aggregation on at least one candidate region to obtain a region-of-interest feature may include:

a. traversing at least one candidate region, and dividing each candidate region into at least one unit;

b. and determining the coordinate position of each unit through bilinear interpolation and performing maximum pooling operation to obtain the characteristics of the region of interest.

Specifically, the ROI feature can be obtained by using ROI Align (a region feature aggregation algorithm). The idea of ROI Align includes: and (3) canceling the quantization operation, and obtaining an image numerical value on a pixel point with the coordinate as a floating point number by using a bilinear interpolation method, so that the whole feature aggregation process is converted into a continuous operation. In a specific algorithm operation, the ROI Align does not simply complement the coordinate points on the boundary of the candidate region and then pool the coordinate points, but redesigns the following flow:

1) and traversing each candidate region, and keeping the floating point number boundary not to be quantized.

2) The candidate region is partitioned into k x k units, the boundaries of each unit are not quantized.

3) Fixed four coordinate positions are calculated in each cell, the values of the four positions are calculated by a bilinear interpolation method, and then the maximum pooling operation is performed.

Some explanation of the third point of the above steps is given here: this fixed position refers to a position determined in each rectangular unit (bin) according to a fixed rule.

For example, if the number of sampling points is 1, it is the center point of the cell. If the number of samples is 4, then the cell is divided equally into 4 small squares and their respective center points. Obviously, the coordinates of these sampling points are usually floating point numbers, so it is necessary to use an interpolation method to obtain the pixel values.

In the above embodiment, by determining the region-of-interest feature in the first fusion feature, the approximate local range where the defect information is located can be determined, and then the defect information is detected based on the region-of-interest feature and the first fusion feature, so that the defect detection efficiency can be effectively improved.

The above embodiments illustrate specific determination processes of the region of interest features, and the specific processes of detecting defect information will be further described below with reference to the drawings and the embodiments.

As shown in fig. 5, the step S304 of detecting defect information in an image to be detected based on a region-of-interest feature and a first fusion feature may include:

and S410, fusing the region-of-interest feature and the first fusion feature to obtain a second fusion feature.

Specifically, the fusing the region of interest feature and the first fusion feature in step S410 to obtain a second fusion feature may include:

(1) performing dimension conversion on the first fusion characteristic to obtain a conversion characteristic; the transformed features are the same dimension as the region of interest features.

Specifically, the first fusion feature may be dimension-converted by a convolution operation, i.e., conv operation.

(2) Pooling the conversion characteristics to obtain pooled characteristics; the dimensions of the pooled features are the same as the dimensions of the region of interest features.

Specifically, pooling the conversion features to obtain pooled features may include:

(3) A second fused feature is determined based on the pooled features and the region of interest features.

Specifically, the pooled feature and the region-of-interest feature may be multiplied to obtain a second fusion feature.

In this embodiment, the second fusion characteristic may be determined using the following formula:

（2）

in the above equation, the conv operation (convolution operation) aims to transform the first fused feature dimension to ensure that the first fused feature dimension and the feature dimension of the region of interest are equal;

the purpose of the function is to pool the first fusion feature globally in blocks so that the first fusion feature and the feature scale size of the region of interest remain the same;

representing a feature of interest;

representing the first fused feature.

In particular, the method comprises the following steps of,

the function implementation can be RoiPool (pooling of interest) or RoiAlign, and in the application, an integral RoiPool can be adopted, and compared with the former two, pooling is performed through a global pooling or linear interpolation mode, and integral pooling accuracy is higher. Given a feature block bin and a feature map

Performing pooling operation by calculating a second order integral, specifically as follows:

（3）

in the above formula (1)x ₁ ,y ₁）（x ₂ ，y ₂) Representing the upper left and lower right coordinates of bin, respectively.

And step S420, carrying out target detection on the second fusion characteristics to obtain defect information aiming at the target to be detected.

Wherein the defect information may include a defect location and a defect type;

specifically, the step S420 of performing target detection on the second fusion feature to obtain defect information for the target to be detected may include:

Specifically, at least one defect type can be predicted for each second fusion feature, so that the defect type of the target to be detected is determined; and simultaneously predicting a candidate position frame corresponding to the defect position so as to determine the defect position of the target to be detected.

As shown in fig. 6, in this embodiment, a feature to be detected of an image to be detected is extracted, a standard feature of a standard image is extracted, and the feature to be detected and the standard feature are fused to obtain a first fusion feature; determining a region of interest feature in the first fused feature; fusing the region of interest feature and the first fusion feature to obtain a second fusion feature; and detecting the defect information in the image to be detected based on the second fusion characteristic.

In order to better understand the above defect detection method, as shown in fig. 7, an example of the defect detection method of the present invention is explained in detail as follows:

in one example, the defect detection method provided by the present application may include the following steps:

1) performing paired inputs: inputting an image to be detected and a standard image into a backbone (neural network model);

2) extracting the to-be-detected features of the to-be-detected image based on the backbone, and extracting the standard features of the standard image;

3) and (3) fusion of contrast characteristics: determining a first weight of the feature to be detected based on the fusion network, and determining a second weight of the standard feature; performing channel-division fusion on the to-be-detected feature and the standard feature based on the first weight and the second weight to obtain a first fusion feature;

5) self-adaptive global feature fusion: determining a region of interest feature in the first fused feature; the RPN network can be adopted to extract interesting features; performing ROI Align operation on the region-of-interest feature and the first fusion feature to perform fusion to obtain a second fusion feature;

7) dense local regression: combining the mask branch and the bbox branch to obtain a defect position and a defect type aiming at the target to be detected; specifically, a mask branch in the graph is used for predicting at least one defect category for each second fusion feature, and a bbox branch is used for predicting a candidate position frame corresponding to the defect position.

According to the defect detection method, the first fusion characteristic is obtained by fusing the to-be-detected characteristic and the standard characteristic, the comparison information between the to-be-detected image and the standard image can be obtained, even if the defect size distribution on the image is irregular, the defect size distribution can be reflected in the first fusion characteristic, and then the defect information is detected based on the first fusion characteristic, so that the accuracy of defect detection can be effectively improved.

To further illustrate the effectiveness of the defect detection method of the present application, further description will be provided below in conjunction with experimental data and the accompanying drawings.

As shown in fig. 8a and 8b, fig. 8a illustrates defect detection using a conventional Mask R-CNN method, fig. 8b illustrates a defect detection method of the present application, and it can be seen from comparison between fig. 8a and 8b that the defect detection method of the present application has a higher detection and identification capability for micro defects compared to the prior art.

As shown in fig. 9a and 9b, fig. 9a illustrates defect detection using a conventional Mask R-CNN method, fig. 9b illustrates a defect detection method of the present application, and it can be seen from comparison between fig. 9a and 9b that the defect detection method of the present application has better defect boundary fitting effect than the conventional method.

As shown in fig. 10a and 10b, fig. 10a illustrates defect detection using a conventional Mask R-CNN method, fig. 10b illustrates a defect detection method of the present application, and it can be seen from comparison between fig. 10a and 10b that the defect detection method of the present application has a stronger detection and identification capability for defects such as large defects (e.g., missing material) compared to the prior art.

Compared with the fusion modes such as addition, subtraction, fixed value weighting and the like, the defect detection method has stronger generalization capability by adopting the fusion mode of fusing network autonomous learning parameters; the main defect area can be extracted for analysis and judgment, the calculation amount is reduced, and the accuracy is improved; the method can better apply the image data of the defect-free accessories in the actual production line, and improve the performance of the model.

A possible implementation manner is provided in the embodiment of the present application, and as shown in fig. 11, a defect detection apparatus 110 is provided, where the defect detection apparatus 110 may include: an extraction module 1101, a fusion module 1102, a determination module 1103, and a detection module 1104, wherein,

the extraction module 1101 is configured to obtain an image to be detected and a standard image of a target to be detected, extract a feature to be detected of the image to be detected, and extract a standard feature of the standard image;

the fusion module 1102 is configured to fuse the feature to be detected and the standard feature to obtain a first fusion feature;

a determining module 1103 for determining a region of interest feature in the first fused feature;

and the detection module 1104 is configured to detect defect information in the image to be detected based on the region-of-interest feature and the first fusion feature.

In the embodiment of the present application, a possible implementation manner is provided, and when the fusion module 1102 fuses the feature to be detected and the standard feature to obtain a first fusion feature, the fusion module is specifically configured to:

In the embodiment of the present application, a possible implementation manner is provided, and the fusion module 1102 is specifically configured to fuse the to-be-detected feature corresponding to each channel with the standard feature corresponding to each channel, to obtain a first fusion feature:

In the embodiment of the present application, a possible implementation manner is provided, and when determining the region of interest feature in the first fusion feature, the determining module 1103 is specifically configured to:

extracting at least one candidate region in the first fusion feature;

In an embodiment of the present application, a possible implementation manner is provided, and when performing region feature aggregation on at least one candidate region to obtain a region of interest feature, the determining module 1103 is specifically configured to:

The embodiment of the present application provides a possible implementation manner, and when detecting defect information in an image to be detected based on a region-of-interest feature and a first fusion feature, the detecting module 1104 is specifically configured to:

In the embodiment of the present application, a possible implementation manner is provided, and when the detection module 1104 fuses the feature of the region of interest and the first fusion feature to obtain a second fusion feature, the detection module is specifically configured to:

The embodiment of the present application provides a possible implementation manner, and the detection module 1104 is specifically configured to, when pooling the conversion features and obtaining the pooled features:

The embodiment of the application provides a possible implementation mode, and the defect information comprises a defect position and a defect type; when the detection module 1104 performs target detection on the second fusion feature to obtain defect information for the target to be detected, the detection module is specifically configured to:

According to the defect detection device, the first fusion characteristic is obtained by fusing the characteristic to be detected and the standard characteristic, the contrast information between the image to be detected and the standard image can be obtained, even if the defect size distribution on the image is irregular, the contrast information can be reflected in the first fusion characteristic, and then the defect information is detected based on the first fusion characteristic, so that the accuracy of defect detection can be effectively improved.

The image defect detection apparatus of the embodiment of the present disclosure can execute the image defect detection method provided by the embodiment of the present disclosure, and the implementation principle is similar, the actions executed by each module in the image defect detection apparatus of the embodiments of the present disclosure correspond to the steps in the image defect detection method of the embodiments of the present disclosure, and for the detailed function description of each module of the image defect detection apparatus, reference may be specifically made to the description in the corresponding image defect detection method shown in the foregoing, and details are not repeated here.

Based on the same principle as the method shown in the embodiments of the present disclosure, embodiments of the present disclosure also provide an electronic device, which may include but is not limited to: a processor and a memory; a memory for storing computer operating instructions; and the processor is used for executing the defect detection method shown in the embodiment by calling the computer operation instruction. Compared with the prior art, the defect detection method can effectively improve the accuracy of defect detection.

In an alternative embodiment, there is provided an electronic device, as shown in fig. 12, an electronic device 4000 shown in fig. 12 including: a processor 4001 and a memory 4003. Processor 4001 is coupled to memory 4003, such as via bus 4002. Optionally, the electronic device 4000 may further comprise a transceiver 4004. In addition, the transceiver 4004 is not limited to one in practical applications, and the structure of the electronic device 4000 is not limited to the embodiment of the present application.

The Processor 4001 may be a CPU (Central Processing Unit), a general-purpose Processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other Programmable logic device, a transistor logic device, a hardware component, or any combination thereof. Which may implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the disclosure. The processor 4001 may also be a combination that performs a computational function, including, for example, a combination of one or more microprocessors, a combination of a DSP and a microprocessor, or the like.

Bus 4002 may include a path that carries information between the aforementioned components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus 4002 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 12, but this is not intended to represent only one bus or type of bus.

The Memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical Disc storage, optical Disc storage (including Compact Disc, laser Disc, optical Disc, digital versatile Disc, blu-ray Disc, etc.), a magnetic Disc storage medium or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to these.

The memory 4003 is used for storing application codes for executing the scheme of the present application, and the execution is controlled by the processor 4001. Processor 4001 is configured to execute application code stored in memory 4003 to implement what is shown in the foregoing method embodiments.

Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 12 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

The present application provides a computer-readable storage medium, on which a computer program is stored, which, when running on a computer, enables the computer to execute the corresponding content in the foregoing method embodiments. Compared with the prior art, the defect detection method can effectively improve the accuracy of defect detection.

It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.

The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

Embodiments of the present application provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device realizes the following when executed:

determining a region of interest feature in the first fused feature;

Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present disclosure may be implemented by software or hardware. The name of the module does not in some cases constitute a limitation of the module itself, and for example, the detection module may also be described as a "module for detecting defect information".

The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.

Claims

1. A method of defect detection, comprising:

acquiring an image to be detected and a standard image of a target to be detected, extracting a characteristic to be detected of the image to be detected, and extracting a standard characteristic of the standard image;

fusing the to-be-detected features and the standard features to obtain first fused features;

determining a region of interest feature in the first fused feature; the region-of-interest feature is used for representing a local feature of contrast information between the image to be detected and the standard image, and the region-of-interest is a region which is defined from the first fusion feature and possibly comprises defect information aiming at the target to be detected;

fusing the region of interest feature and the first fusion feature to obtain a second fusion feature;

performing target detection on the second fusion characteristics to obtain defect information aiming at the target to be detected;

wherein the fusing the region of interest feature and the first fusion feature to obtain a second fusion feature comprises:

performing dimension conversion on the first fusion characteristic to obtain a conversion characteristic; the dimensions of the conversion feature and the region of interest feature are the same;

pooling the conversion characteristics to obtain pooled characteristics; the dimensions of the pooled features are the same as the dimensions of the region of interest features;

determining the second fused feature based on the pooled features and the region of interest features.

2. The defect detection method of claim 1, wherein the fusing the features to be detected with the standard features to obtain first fused features comprises:

and fusing the to-be-detected features corresponding to each channel with the standard features corresponding to each channel respectively to obtain the first fusion features.

3. The defect detection method according to claim 2, wherein the fusing the to-be-detected features corresponding to each channel with the standard features corresponding to each channel to obtain the first fused features comprises:

and determining the weighted sum of the to-be-detected feature and the standard feature respectively corresponding to each channel based on the first weight and the second weight respectively corresponding to each channel to obtain the first fusion feature.

4. The defect detection method of claim 1, wherein said determining the region-of-interest feature in the first fused feature comprises:

extracting at least one candidate region in the first fusion feature;

and performing region feature aggregation on the at least one candidate region to obtain the region-of-interest feature.

5. The defect detection method of claim 4, wherein the performing the region feature clustering on the at least one candidate region to obtain the region-of-interest feature comprises:

traversing the at least one candidate region, and dividing each candidate region into at least one unit;

6. The method of claim 1, wherein pooling the transformed features to obtain pooled features comprises:

and performing integral pooling on the conversion characteristics based on a preset characteristic block and a preset characteristic diagram to obtain the pooling characteristics.

7. The defect detection method of claim 1, wherein the defect information includes a defect location and a defect category; the performing target detection on the second fusion feature to obtain defect information for the target to be detected includes:

8. A defect detection apparatus, comprising:

a determination module for determining a region of interest feature in the first fused feature; the region-of-interest feature is used for representing a local feature of contrast information between the image to be detected and the standard image, and the region-of-interest is a region which is defined from the first fusion feature and possibly comprises defect information aiming at the target to be detected;

the detection module is used for fusing the region-of-interest feature and the first fusion feature to obtain a second fusion feature; performing target detection on the second fusion characteristics to obtain defect information aiming at the target to be detected;

the detection module is specifically configured to, when fusing the region-of-interest feature and the first fusion feature to obtain a second fusion feature:

9. The defect detection apparatus according to claim 8, wherein the fusion module is specifically configured to, when fusing the to-be-detected feature and the standard feature to obtain a first fused feature:

10. The defect detection device according to claim 9, wherein the fusion module is specifically configured to, when fusing the to-be-detected feature corresponding to each channel with the standard feature corresponding to each channel to obtain the first fusion feature:

11. The defect detection apparatus of claim 8, wherein the determining module, when determining the region-of-interest feature of the first fused feature, is specifically configured to:

extracting at least one candidate region in the first fusion feature;

12. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the defect detection method of any of claims 1-7 when executing the program.

13. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the defect detection method of any one of claims 1 to 7.