CN117495846B

CN117495846B - Image detection method, device, electronic equipment and storage medium

Info

Publication number: CN117495846B
Application number: CN202311811816.8A
Authority: CN
Inventors: 徐海俊; 韩晓
Original assignee: Suzhou Mega Technology Co Ltd
Current assignee: Suzhou Mega Technology Co Ltd
Priority date: 2023-12-27
Filing date: 2023-12-27
Publication date: 2024-04-16
Anticipated expiration: 2043-12-27
Also published as: CN117495846A

Abstract

The embodiment of the application provides an image detection method, an image detection device, electronic equipment and a storage medium. The method comprises the following steps: acquiring an image to be detected; performing defect detection on the image to be detected by using a preset defect detection algorithm to determine a first defect detection result, wherein the first defect detection result comprises position information of a defect area in the image to be detected; acquiring a defect-free image from the image to be detected based on the first defect detection result; training the defect detection model with the defect-free image to obtain a trained defect detection model; performing defect detection on the target image at least by using the trained defect detection model to determine a second defect detection result, wherein the second defect detection result comprises position information of a defect area in the target image, and the target image is an image to be detected or a defective image obtained from the image to be detected based on the first defect detection result. The method is beneficial to improving the accuracy and the robustness of image detection.

Description

Image detection method, device, electronic equipment and storage medium

Technical Field

The present application relates to the technical field, and more particularly, to an image detection method, an image detection apparatus, an electronic device, and a storage medium.

Background

The traditional industrial product surface quality detection is performed manually by a quality inspector, but the manual detection method has the defects of low precision, low efficiency, high labor intensity and the like. Automatic quality inspection of industrial product surfaces using machine vision techniques has become a very important tool for manufacturing to improve product quality. The quality detection system based on machine vision generally has the advantages of high precision, high efficiency, high continuous detection speed, non-contact measurement and the like.

In the related art, it is common to directly detect whether defects exist on the surface of an industrial product by using a visual detection algorithm. However, when the threshold is relatively low, the conventional visual detection algorithm is easy to judge the normal industrial product as a defective industrial product. Thereby, the accuracy of the detection result is affected.

Disclosure of Invention

The present application has been made in view of the above-described problems. The application provides an image detection method, an image detection device, an electronic device and a storage medium.

According to an aspect of the present application, there is provided an image detection method including: acquiring an image to be detected; performing defect detection on the image to be detected by using a preset defect detection algorithm to determine a first defect detection result, wherein the first defect detection result comprises position information of a defect area in the image to be detected; acquiring a defect-free image from the image to be detected based on the first defect detection result; training the defect detection model with the defect-free image to obtain a trained defect detection model; performing defect detection on the target image at least by using the trained defect detection model to determine a second defect detection result, wherein the second defect detection result comprises position information of a defect area in the target image, and the target image is an image to be detected or a defective image obtained from the image to be detected based on the first defect detection result.

According to the technical scheme, the defect-free image is obtained from the image to be detected through the first defect detection result obtained based on the preset defect detection algorithm, and the defect detection model obtained based on the defect-free image training is used for detecting the defect of the target image, so that the accuracy and the robustness of image detection are improved. Meanwhile, the scheme does not need to additionally collect images when a defect detection model is trained, and is beneficial to improving the efficiency of image detection.

Illustratively, performing defect detection on the target image using at least the trained defect detection model to determine a second defect detection result includes: extracting sub-images to be detected, which correspond to the positions of at least one target image one by one, from the target image; performing defect detection on at least part of the extracted at least one sub-image to be detected by using a trained defect detection model to determine sub-defect detection results which are in one-to-one correspondence with each sub-image to be detected in the at least part of sub-images to be detected, wherein the sub-defect detection results comprise position information of sub-defect areas in the corresponding sub-images to be detected and/or areas of the sub-defect areas; the second defect detection result comprises sub-defect detection results which are in one-to-one correspondence with at least part of the sub-images to be detected.

According to the technical scheme, the sub-images to be detected are extracted from the target image, and the trained defect detection model is utilized to detect defects of at least part of the sub-images to be detected in the extracted at least one sub-image to be detected, so that sub-defect detection results corresponding to at least part of the sub-images to be detected in a one-to-one mode can be obtained. In short, the scheme is helpful for accurately determining the defect detection result of the target image.

Illustratively, performing defect detection on at least a portion of the at least one sub-image to be detected using the trained defect detection model includes: and performing defect detection on the sub-image to be detected, of which the intersection exists with the defect area indicated in the first defect detection result, in the at least one sub-image to be detected by using the trained defect detection model.

In the scheme of the example, only the sub-image to be detected with the defect determined by the first defect detection result is required to be input into the trained defect detection model for defect detection, so that the calculation amount is reduced, and the defect detection efficiency is improved.

Illustratively, performing defect detection on at least a portion of the at least one sub-image to be detected using the trained defect detection model includes: and performing defect detection on the sub-image to be detected, of which the fluctuation degree of the pixel value is larger than the preset fluctuation degree, in at least one sub-image to be detected by using the trained defect detection model.

In the scheme of the example, only the sub-image to be detected with high image noise is required to be input into the trained defect detection model for defect detection, so that the calculation amount is reduced, and the defect detection efficiency is improved.

Illustratively, performing defect detection on the target image using at least the trained defect detection model to determine a second defect detection result, further comprising: for each sub-image to be detected in the remaining sub-images to be detected, taking a first defect detection result corresponding to the sub-image to be detected as a sub-defect detection result corresponding to the sub-image to be detected; or, performing defect detection on the remaining sub-images to be detected by using the trained defect detection model so as to determine sub-defect detection results corresponding to the remaining sub-images to be detected one by one; the remaining sub-images to be detected are other sub-images to be detected except for the at least part of sub-images to be detected in the at least one sub-image to be detected, and the second defect detection result further comprises sub-defect detection results corresponding to the remaining sub-images to be detected in a one-to-one correspondence.

In the above technical solution, the first defect detection result may be directly used to determine the sub-defect detection results corresponding to the remaining sub-images to be detected, or the trained defect detection model may be used to determine the sub-defect detection results corresponding to the remaining sub-images to be detected. The method for determining the sub-defect detection results corresponding to the remaining sub-images to be detected by directly utilizing the first defect detection results is beneficial to avoiding repeated calculation and improving the defect detection efficiency. The trained defect detection model is utilized to determine the sub-defect detection results corresponding to the remaining sub-images to be detected, so that the sub-defect detection results corresponding to the remaining sub-images to be detected can be determined more accurately.

Illustratively, the preset defect detection algorithm is to detect defects based on differences between at least one image region in the image to be detected and the standard image.

According to the technical scheme, the defect detection is directly carried out based on the difference between each of at least one image area in the image to be detected and the standard image, so that the first defect detection result of the image to be detected can be accurately and rapidly obtained.

Illustratively, the pixels of each of the at least one image area are in one-to-one correspondence with the pixels of the standard image, and the preset defect detection algorithm comprises the following operations: for each image area in at least one image area, calculating a pixel value difference value between the pixel value of each pixel in the image area and the pixel value of the corresponding pixel in the standard image, wherein the pixel difference value between the image area and the standard image is represented by the pixel value difference value; and determining the region where the corresponding pixel value difference value in the image region is larger than a preset difference value threshold value as a defect region in the image to be detected.

The technical scheme directly carries out defect detection based on the pixel value difference value between the pixel value of each pixel in the image area and the pixel value of the corresponding pixel in the standard image, has simple calculation, is beneficial to improving the calculation efficiency, and has better accuracy.

Illustratively, the image to be detected and the standard image are product images of a product to be detected, each of the at least one image area and the standard image each comprise a single product to be detected, the preset defect detection algorithm further comprising the operations of: acquiring a plurality of sample images containing a single product to be detected, wherein the width and the height of the sample images are consistent; calculating a median value or a mean value of pixel values of pixels located at the same image coordinates in the acquired plurality of sample images; an image having a pixel value equal to the median or mean of the pixel values is determined as a standard image.

In the above technical solution, an image with a pixel value equal to the median value of the pixel values or the average value of the pixel values is determined as a standard image, so that the first defect detection result of the image to be detected can be determined more accurately based on the standard image.

Illustratively, the image to be inspected includes a plurality of identical areas to be inspected, and acquiring a defect-free image from the image to be inspected based on a first defect inspection result includes: extracting a plurality of target product images from the image to be detected, wherein each target product image comprises a single area to be detected; and determining a target product image which does not contain the defect area indicated by the first defect detection result in the plurality of target product images as a defect-free image.

In this example scheme, the image to be detected contains a plurality of products to be detected that are identical to each other. And each region where the product to be detected is located corresponds to one target product image. According to the scheme, whether each target product image has a defect or not (namely whether the target product image is a defect image or not) is determined by utilizing the first defect detection result, and a defect-free image can be acquired more accurately, so that more accurate samples are provided for training of a defect detection model in the subsequent step, and the detection precision of the defect detection model is improved.

Illustratively, the target image is a defective image, and when performing the step of acquiring a non-defective image from the image to be inspected based on the first defect detection result, the method further comprises: and determining a target product image including a defect area indicated by the first defect detection result from the plurality of target product images as a defective image.

In this example aspect, it may be determined whether the target product image is a defective image directly based on the first defect detection result. In the subsequent step, the trained defect detection model can be utilized to perform secondary judgment on the defective image, so that the accuracy of the image detection method can be further improved.

Illustratively, the number of the images to be detected is plural, different images to be detected respectively contain the same product to be detected, and the defect-free image is obtained from the images to be detected based on the first defect detection result, including: and determining the images to be detected, which do not contain the defect area indicated by the first defect detection result, in the plurality of images to be detected as non-defective images.

According to the technical scheme, the defect-free images can be accurately obtained from the images to be detected, so that accurate samples can be provided for training of the defect detection model in the subsequent steps, and the detection precision of the defect detection model can be improved.

Illustratively, the target image is a defective image, and when performing the step of acquiring a non-defective image from the image to be inspected based on the first defect detection result, the method further comprises: and determining the images to be detected, which contain the defect area indicated by the first defect detection result, in the plurality of images to be detected as defective images.

In the scheme of the example, the defective image in the plurality of images to be detected can be directly determined, and in the subsequent step, the trained defect detection model can be utilized to perform secondary judgment on the defective image, so that the accuracy of the image detection method can be further improved.

Illustratively, training the defect detection model with the defect-free image includes: extracting sample sub-images corresponding to at least one target image position on the defect-free image; for each of the at least one target image position, training the defect detection model with a sample sub-image corresponding to the target image position to obtain a trained defect detection model corresponding to the target image position.

In the above technical solution, the defect detection model is trained by using the sample sub-images of the same target image position, so as to obtain the trained defect detection model corresponding to each target image position. The scheme is helpful to obtain a defect detection model with higher detection precision.

According to another aspect of the present application, there is provided an image detection apparatus including: the first acquisition module is used for acquiring an image to be detected; the first detection module is used for carrying out defect detection on the image to be detected by using a preset defect detection algorithm so as to determine a first defect detection result, wherein the first defect detection result comprises position information of a defect area in the image to be detected; the second acquisition module is used for acquiring a defect-free image from the image to be detected based on the first defect detection result; the training module is used for training the defect detection model by utilizing the defect-free image so as to obtain a trained defect detection model; and the second detection module is used for carrying out defect detection on the target image at least by utilizing the trained defect detection model so as to determine a second defect detection result, wherein the second defect detection result comprises position information of a defect area in the target image, and the target image is an image to be detected or a defective image obtained from the image to be detected based on the first defect detection result.

According to a further aspect of the application there is provided an electronic device comprising a processor and a memory, wherein the memory has stored therein computer program instructions which, when executed by the processor, are adapted to carry out the image detection method as described above.

According to still another aspect of the present application, there is provided a storage medium having stored thereon program instructions for executing the above-described image detection method when executed.

Drawings

The above and other objects, features and advantages of the present application will become more apparent from the following more particular description of embodiments of the present application, as illustrated in the accompanying drawings. The accompanying drawings are included to provide a further understanding of embodiments of the application and are incorporated in and constitute a part of this specification, illustrate the application and together with the embodiments of the application, and not constitute a limitation to the application. In the drawings, like reference numerals generally refer to like parts or steps.

FIG. 1 shows a schematic flow chart of an image detection method according to one embodiment of the application;

FIG. 2 shows a schematic diagram of an image detection method according to one embodiment of the application;

FIG. 3 is a schematic diagram of an image detection method according to another embodiment of the present application;

FIG. 4 shows a schematic diagram of an image to be detected according to one embodiment of the application;

fig. 5 shows a schematic block diagram of an image detection apparatus according to an embodiment of the present application; and

Fig. 6 shows a schematic block diagram of an electronic device according to an embodiment of the application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be apparent that the described embodiments are only some embodiments of the present application and not all embodiments of the present application, and it should be understood that the present application is not limited by the example embodiments described herein. Based on the embodiments of the application described in the present application, all other embodiments that a person skilled in the art would have without inventive effort shall fall within the scope of the application.

Traditional industrial product surface quality detection is performed manually by a quality inspector. Taking a wafer as an example, in the semiconductor packaging process, at least four times of optical inspection are needed, and a person performs spot inspection or full inspection under different multiplying powers by using a microscope, so that the phenomena of low efficiency, tiredness of vision of the person, defect omission inspection and the like exist. In order to improve the detection efficiency, the quality detection of the whole process section in the semiconductor packaging process is usually performed by adopting a machine vision technology at present. In the related art, it is common to directly detect whether defects exist on the surface of an industrial product by using a visual detection (MV) algorithm. In some embodiments of the related art, the MV algorithm generally establishes a standard template of a product to be detected based on a sample image in advance, then makes a difference between the image to be detected containing the product to be detected and the standard template image, and determines that a defect exists in the image to be detected when the difference is greater than a preset threshold. However, the preset threshold is generally set by a user, and when the preset threshold is relatively small, a normal industrial product is easily judged as a defective industrial product, and when the preset threshold is relatively large, a defective industrial product is easily judged as a normal industrial product. Thereby, the accuracy of the detection result is affected,

In order to at least partially solve the above-mentioned problems, an embodiment of the present application provides an image detection method. Fig. 1 shows a schematic flow chart of an image detection method according to an embodiment of the application. As shown in fig. 1, the image detection method 100 may include the following steps S110, S120, S130, S140, and S150.

In step S110, an image to be detected is acquired.

The image to be detected according to the embodiment of the present application may be an image of any object to be detected as a defect. In other words, a target object to be defect detected may be included in the image to be detected. The target object to be defect detected may be any suitable object, including but not limited to, metal, glass, paper, electronic components, and the like, which have strict requirements on the appearance and have clear indicators, and the like, and the present application is not limited thereto.

The image to be detected may be a black-and-white image or a color image, for example. The image to be detected may be an image of any size or resolution size, for example. Alternatively, the image to be detected may be an image satisfying a preset resolution requirement. In one example, the image to be detected may be a black and white image having a 512 x 512 pixel size. The requirements for the image to be detected may be set based on the actual detection requirements, hardware conditions of the image capturing device, requirements for the input image by a defect detection algorithm (for example, a preset defect detection algorithm hereinafter), and the like, which is not limited by the present application.

The image to be detected may be an original image acquired by the image acquisition device, for example. According to the embodiment of the application, any existing or future image acquisition mode can be adopted to acquire the image to be detected. For example, the image to be detected may be acquired by an image acquisition device in a machine vision detection system, such as an illumination device, a lens, a high-speed camera, and an image acquisition card that are matched with the detection environment and the object to be detected.

In another example, the image to be detected may be an image after the preprocessing operation is performed on the original image.

The preprocessing operation may be any preprocessing operation that can meet the needs of the subsequent image detection step, and may include all operations that facilitate image detection of the image to be detected, such as improving the visual effect of the image, improving the sharpness of the image, or highlighting certain features in the image. Optionally, the preprocessing operation may include denoising operations such as filtering, and may also include adjustment of image parameters such as adjustment of image enhancement gray scale, contrast, and brightness. Alternatively, the preprocessing operation may include pixel normalization processing of the image to be detected. For example, each pixel of the image to be detected may be divided by 255 so that the pixel of the preprocessed image to be detected is in the range of 0-1. This helps to improve the efficiency of subsequent image detection.

Illustratively, the preprocessing operations may also include cropping images, deleting images, and the like. For example, the original image may be cut to the size required by the defect detection algorithm, and the original image that does not satisfy the image quality requirement may be deleted to obtain an image to be detected that satisfies the image quality requirement, and the like.

The number of images to be detected may be 1 or a plurality of images, for example. Alternatively, the number of images to be detected is 1, for example, only one image to be detected is acquired at a time. Alternatively, the number of images to be detected may be plural, for example, 10, 500, and the plural images to be detected may be acquired at one time, and then defect detection is performed on the plural images to be detected using a preset defect detection algorithm described below.

In step S120, defect detection is performed on the image to be detected by using a preset defect detection algorithm to determine a first defect detection result, where the first defect detection result includes location information of a defect region in the image to be detected.

Illustratively, the defect region is a region in the image to be detected in which the target object has a defect, which may be a partial region in the image to be detected. It is easy to understand that the normal region and the defective region of the target object in the image to be detected may have different forms, and the defective region may be detected based on the different forms thereof, such as gray scale, texture, and the like. For example, for an example in which the image to be detected is a metal image of some kind, the defective area may be an area showing scratches on the metal object. For an example in which the image to be detected is a glass image, the defective region may be a region showing bubbles, impurities, cracks, and the like in the glass.

Alternatively, the preset defect detection algorithm may be any defect detection algorithm that is present or developed in the future. For example, the algorithm may determine whether a defect exists in the image to be detected based on a comparison result between the image to be detected and the standard template image. In some embodiments, an image of the target object may be acquired in advance as a standard template image. And extracting feature vectors of the image to be detected and the standard template image respectively to obtain the feature vectors of the image to be detected and the feature vectors of the standard template image, and carrying out similarity calculation on the two feature vectors to obtain a first defect detection result. In other embodiments, template matching may be performed according to the image to be detected and the standard template image, so as to obtain the first defect detection result. In still other embodiments, the subtraction operation may be performed on the image to be detected and the standard template image, so as to obtain a pixel difference corresponding to each pixel position in the image. Then, the first defect detection result is obtained by comparing each pixel difference with a pixel difference threshold.

Alternatively, the position information of the defective area in the image to be detected may be a relative position between the defective area and the image to be detected. Alternatively, the position information of the defective region in the image to be detected may also be coordinates of the defective region in the image to be detected. For example, coordinates of four corner points of the target frame where the defective area is located may be mentioned.

In step S130, a non-defective image is acquired from the image to be inspected based on the first defect detection result.

Alternatively, the defect-free image may include a partial region in the image to be detected. In some embodiments, the image to be inspected may be a wafer image, each wafer image including a plurality of die (die) units thereon. The partial region may be an image region corresponding to each die unit on the wafer image. In this embodiment, step S120 may be performed for each wafer image to determine whether each die unit on the wafer image is defective. When a certain die unit has a defect, taking an image area where the die unit is positioned as a defective image; and when the image area where the die unit is positioned has no defect, taking the image area where the die unit is positioned as a defect-free image. The above division is merely an example, and other division may be adopted to divide a partial region on the image to be detected. For example, a preset number of die units may be used as a region, the Shan Zhangjing circles are divided into regions of the same size in advance, then the region is used as a processing unit, and when no defect exists in the die units in the region, the region is determined to be a defect-free image, and a trained defect detection model is obtained.

Alternatively, the number of images to be detected may be plural. The same number of target objects (e.g., wafers) may be included in the plurality of images to be inspected. After the step S120 is performed on the images to be detected, a first defect detection result corresponding to each image to be detected may be obtained. Then, the image to be detected without defects can be taken as a defect-free image according to the first defect detection result corresponding to each image to be detected. In some embodiments, all defect-free images in the image to be detected may be acquired. In other embodiments, only partially defect-free images of the image to be inspected may be acquired.

After obtaining the non-defective image, step S140 may be performed based on the acquired non-defective image.

In step S140, the defect detection model is trained using the defect-free image, and a trained defect detection model is obtained.

Alternatively, the defect-free image may be divided into a plurality of sample sub-images, each training the same or different defect detection models. In one embodiment, the same defect detection model may be trained based on multiple sample sub-images located at different positions of the defect-free image. In another embodiment, the defect detection model may be trained based on sample sub-images located at each location of the non-defective image, respectively, to obtain trained defect detection models that correspond one-to-one to the respective locations of the non-defective image.

Alternatively, the defect detection model may be any defect detection model that may be used for defect detection, existing or developed in the future, and the present application is not limited to a specific model type. Alternatively, the defect detection model may be a neural network model for defect detection. For example, it may be a target detection model based on deep learning, a semantic segmentation model, or the like. Alternatively, the defect detection model may be trained based on any open source anomaly detection algorithm. For example, the open source anomaly detection algorithm may be, for example, patchcore algorithm, CFA algorithm, pyramidFlow algorithm, or the like.

In step S150, defect detection is performed on the target image at least using the trained defect detection model to determine a second defect detection result, where the second defect detection result includes location information of a defect region in the target image, and the target image is an image to be detected or a defective image obtained from the image to be detected based on the first defect detection result.

In short, the defective area detected in step S130 may be caused by the over-inspection. Based on this, the embodiment of the present application may perform the second inspection on the first inspection result using S150, that is, perform the second defect inspection on the defect area detected in step S130 using the trained defect inspection model to determine the second defect inspection result.

Similar to the non-defective image, the defective image may be a partial region of the image to be detected, or may be the entire image to be detected. For example, when the number of the images to be detected is plural, if it is determined that a defect exists in the images to be detected according to the first defect detection result, the images to be detected are determined to be defective images.

Fig. 2 shows a schematic diagram of an image detection method according to an embodiment of the application. As shown in fig. 2, first, defect detection is performed on an image to be detected by using a preset defect detection algorithm to determine a first defect detection result. Then, inputting the target image in the image to be detected into the trained defect detection model to obtain a second defect detection result, thereby obtaining the position and the area of the defect region in the image to be detected. In this embodiment, the target image is a defective image obtained from the image to be detected based on the first defect detection result.

Illustratively, step S150 of performing defect detection on the target image at least using the trained defect detection model to determine the second defect detection result may specifically include the steps of: extracting sub-images to be detected, which correspond to the positions of at least one target image one by one, from the target image; performing defect detection on at least part of the extracted at least one sub-image to be detected by using a trained defect detection model to determine sub-defect detection results which are in one-to-one correspondence with each sub-image to be detected in the at least part of sub-images to be detected, wherein the sub-defect detection results comprise position information of sub-defect areas in the corresponding sub-images to be detected and/or areas of the sub-defect areas; the second defect detection result comprises sub-defect detection results which are in one-to-one correspondence with at least part of the sub-images to be detected.

Alternatively, the target image may comprise at least one region, each region corresponding to a target image location. For example, the target image may be divided into a plurality of image areas based on a preset division rule, each image area corresponding to a respective one of the target image positions. In this embodiment, each image area may be used as one sub-image to be detected, and then at least some of the sub-images to be detected in the plurality of sub-images to be detected are input into the trained defect detection model, so as to obtain sub-defect detection results corresponding to at least some of the sub-images to be detected in a one-to-one correspondence. In a specific embodiment, the areas of the plurality of image areas are the same size. For example, the area where each pixel in the target image is located may be regarded as one image area.

Alternatively, at least part of the sub-images to be detected may be all of the sub-images to be detected in the target image. Alternatively, at least part of the sub-image to be detected may be a specific part of the sub-image to be detected in the target image. For example, it may be a sub-image to be detected including the defective area determined based on the first defect detection result. For another example, the sub-image to be detected that does not include the defective area may be determined based on the first defect detection result. In a specific embodiment, when the threshold value corresponding to the preset defect detection algorithm is low, at least part of the sub-images to be detected may be the sub-images to be detected including the defect area determined based on the first defect detection result. Thereby helping to avoid overdriving.

Illustratively, performing defect detection on at least part of the sub-images to be detected in the at least one sub-image to be detected using the trained defect detection model may specifically include the steps of: and performing defect detection on the sub-image to be detected, of which the intersection exists with the defect area indicated in the first defect detection result, in the at least one sub-image to be detected by using the trained defect detection model.

In this example, at least part of the sub-images to be detected in the target image are sub-images to be detected that intersect with the defect region indicated in the first defect detection result. In other words, at least part of the sub-images to be detected are defective sub-images to be detected. In the scheme of the example, only the sub-image to be detected with the defect determined by the first defect detection result is required to be input into the trained defect detection model for defect detection, so that the calculation amount is reduced, and the defect detection efficiency is improved.

Illustratively, performing defect detection on at least part of the sub-images to be detected in the at least one sub-image to be detected using the trained defect detection model may specifically include the steps of: and performing defect detection on the sub-image to be detected, of which the fluctuation degree of the pixel value is larger than the preset fluctuation degree, in at least one sub-image to be detected by using the trained defect detection model.

Alternatively, the degree of fluctuation of the pixel value may be a difference between a maximum pixel value and a minimum pixel value in the sub-image to be detected. Alternatively, the average value of each pixel value in the sub-image to be detected may be calculated first, and then the difference between each pixel value in the sub-image to be detected and the pixel average value may be calculated. The degree of fluctuation of the pixel values can be represented by the average value of the difference values corresponding to the respective pixel values.

Alternatively, the preset fluctuation degree may be set as needed.

It will be appreciated that for a sub-image to be detected, the higher the degree of fluctuation of the pixel values in the sub-image to be detected, the higher the noise representing the sub-image to be detected, and the poorer the image quality of the sub-image to be detected. The lower the accuracy of the first defect detection result obtained based on the sub-image to be detected may be. In the scheme of the example, at least part of the sub-images to be detected are sub-images to be detected with high image noise. In other words, the scheme only needs to input the sub-image to be detected with high image noise into the trained defect detection model for defect detection, so that the calculation amount is reduced, and the defect detection efficiency is improved.

Illustratively, step S150 of performing defect detection on the target image at least using the trained defect detection model to determine a second defect detection result may further include the steps of: for each sub-image to be detected in the remaining sub-images to be detected, taking a first defect detection result corresponding to the sub-image to be detected as a sub-defect detection result corresponding to the sub-image to be detected; or performing defect detection on the remaining sub-images to be detected by using the trained defect detection model so as to determine sub-defect detection results corresponding to the remaining sub-images to be detected one by one; the remaining sub-images to be detected are other sub-images to be detected except at least part of the sub-images to be detected in at least one sub-image to be detected, and the second defect detection result further comprises sub-defect detection results corresponding to the remaining sub-images to be detected one by one.

Optionally, step S150, performing defect detection on the target image at least using the trained defect detection model to determine a second defect detection result, may further include the steps of: and for each sub-image to be detected in the rest sub-images to be detected, taking a first defect detection result corresponding to the sub-image to be detected as a sub-defect detection result corresponding to the sub-image to be detected. In this alternative embodiment, the sub-defect detection results corresponding to the remaining sub-images to be detected may be determined directly based on the first defect detection result. In other words, for the remaining sub-images to be detected, they may be subjected to defect detection based only on a preset defect detection algorithm. Therefore, repeated calculation can be avoided, and defect detection efficiency is improved.

Optionally, step S150, performing defect detection on the target image at least using the trained defect detection model to determine a second defect detection result, may further include the steps of: and performing defect detection on the remaining sub-images to be detected by using the trained defect detection model so as to determine sub-defect detection results corresponding to the remaining sub-images to be detected one by one. In this embodiment, the sub-defect detection results corresponding to the remaining sub-images to be detected are determined by the trained defect detection model, thereby contributing to improvement of the accuracy of defect detection.

Illustratively, step S140, training the defect detection model with the defect-free image, may include the steps of: extracting sample sub-images corresponding to at least one target image position on the defect-free image; for each of the at least one target image position, training the defect detection model with a sample sub-image corresponding to the target image position to obtain a trained defect detection model corresponding to the target image position.

Alternatively, the defect-free image may include at least one region, each region corresponding to one of the target image locations. For example, a wafer image is taken as an example, and the wafer image includes a plurality of die units thereon. In the case where it is determined that a region does not have a defect, for example, the upper left corner region in the wafer image does not have a region, each die cell in the region may be regarded as a non-defective image.

Alternatively, for each non-defective image, it may be divided into a plurality of image areas, each image area corresponding to one target image location. In this case, each image region may be determined as one sample sub-image, and then the defect detection model is trained using the sample sub-image of the target image location to obtain a trained defect detection model corresponding to the target image location.

Fig. 3 shows a schematic diagram of an image detection method according to another embodiment of the present application. As shown in fig. 3, first, defect detection is performed on an image to be detected by using a preset defect detection algorithm, so as to obtain a defect-free image in the image to be detected according to the obtained first defect detection result (in this embodiment, the image to be detected is a wafer image, and the defect-free image is an image area corresponding to die units on the wafer image). Then, a plurality of sub-images of the same size as the at least one target image may be extracted on the non-defective image in a preset step size by means of a sliding window. Then, for each target image position, the defect detection model may be trained using sample sub-images of the same target image position to obtain a trained defect detection model corresponding to that target image position. Thus, the defect detection model training is completed.

After the defect detection model training is completed, the target image may be defect detected using the trained defect detection model, as shown in fig. 3. In this embodiment, the target image may be first divided into sub-images to be detected in a one-to-one correspondence with at least one target image position in a preset step by means of a sliding window. And then, respectively utilizing the trained defect detection models corresponding to the at least one target image position to detect the defects of the sub-images to be detected corresponding to each target image position, thereby obtaining a second defect detection result and further determining the defect area in the target image.

The image to be detected comprises a plurality of areas to be detected, and the areas to be detected correspond to the positions of the target images one by one. Performing defect detection on the target image using at least the trained defect detection model to determine a second defect detection result, comprising: extracting sub-images to be detected, which correspond to the positions of a plurality of target images one by one, from the images to be detected; for each sub-image to be detected in the plurality of sub-images to be detected, performing defect detection on the sub-image to be detected by using a trained defect detection model corresponding to the target image position of the sub-image to be detected, so as to obtain a second sub-defect detection result corresponding to the sub-image to be detected; for any two identical sub-images to be detected in the plurality of sub-images to be detected, the defect detection models to be trained for defect detection are identical.

It will be appreciated that in this example scenario, a plurality of target image locations may be included on the image to be detected, each target image location corresponding to a respective region to be detected on the image to be detected. The plurality of regions to be detected may include the same region to be detected, and for the same region to be detected, the same trained defect detection model may be used for defect detection.

Alternatively, the same region to be detected in the image to be detected may be determined using a method such as normalized cross-correlation (NCC) matching.

Fig. 4 shows a schematic diagram of an image to be detected according to an embodiment of the application. As shown in fig. 4, the image to be detected includes a plurality of regions to be detected (i.e., circular holes in the figure) that are identical to each other. Therefore, the region where each round hole is located can be extracted as a sub-image to be detected, and then the defect detection is carried out on each sub-image to be detected by using the same trained defect detection model.

In this example, the same defect detection model can be adopted for the same sub-image to be detected, and thus, repeated model training is not required, contributing to further improvement of image detection efficiency.

Illustratively, the preset defect detection algorithm performs defect detection for differences between at least one image region in the image to be detected and the standard image.

Alternatively, the difference may be a difference between pixel values between each of at least one image region in the image to be detected and the standard image, or a difference between feature vectors between each of at least one image region in the image to be detected and the standard image. For example, feature vector extraction may be performed on the image to be detected and the standard template image, respectively, to obtain feature vectors of the image to be detected and feature vectors of the standard template image, and then similarity calculation may be performed on the two feature vectors, so as to determine differences between feature vectors of each of at least one image region in the image to be detected and the standard image.

Alternatively, the preset difference threshold may be set as desired. It will be appreciated that at higher values of the threshold value of the difference from the preset value, missed detection may occur. When the preset difference threshold is low, an over-test may occur. Therefore, the user can set the preset difference threshold according to actual needs. In a specific embodiment, the preset difference threshold may be set to a lower value. It is understood that in the subsequent step S150, defect detection may be performed only on a defective image obtained from the image to be detected based on the first defect detection result. In other words, even if the over-inspection occurs due to the lower preset difference threshold, the trained defect detection model can be used for performing the second judgment in the subsequent step, so that the over-inspection condition is eliminated, and the detection accuracy is improved.

In this embodiment, a plurality of equally sized sample images containing a single product to be tested may be first acquired. Then, a median or mean of pixel values of pixels located at the same image coordinates in the acquired plurality of sample images, which may be referred to as a standard pixel value at the corresponding image coordinates, may be calculated. Finally, a standard image may be generated based on the respective standard pixel values for each image coordinate.

Illustratively, the to-be-detected image includes a plurality of to-be-detected regions identical to each other, and the step S130 of acquiring a defect-free image from the to-be-detected image based on the first defect detection result may include the steps of: extracting a plurality of target product images from the image to be detected, wherein each target product image comprises a single area to be detected; and determining a target product image which does not contain the defect area indicated by the first defect detection result in the plurality of target product images as a defect-free image.

In this example scheme, the image to be detected contains a plurality of regions to be detected that are identical to each other. And each region where the product to be detected is located corresponds to one target product image. Taking a wafer image as an example, the wafer image comprises a plurality of identical die units, and an image area where each die unit is located is an area to be detected. According to the scheme, whether each target product image has a defect or not (namely whether the target product image is a defect image or not) is determined by utilizing the first defect detection result, and a defect-free image can be acquired more accurately, so that more accurate samples are provided for training of a defect detection model in the subsequent step, and the detection precision of the defect detection model is improved.

Illustratively, the target image is a defective image, and in performing the step of acquiring a non-defective image from the image to be inspected based on the first defect detection result, the method 100 may further include the steps of: and determining a target product image including a defect area indicated by the first defect detection result from the plurality of target product images as a defective image. In this example aspect, it may be determined whether the target product image is a defective image directly based on the first defect detection result. In a subsequent step (e.g., step S150), the trained defect detection model may be utilized to make a secondary determination of the defective image, whereby the accuracy of the image detection method may be further improved.

Illustratively, the number of the images to be detected is plural, different images to be detected respectively contain the same product to be detected, and the obtaining of the defect-free image from the images to be detected based on the first defect detection result may include the steps of: and determining the images to be detected, which do not contain the defect area indicated by the first defect detection result, in the plurality of images to be detected as non-defective images.

In the solution of the present example, the number of images to be detected is a plurality. After the step S120 is performed on each of the plurality of images to be detected, a first defect detection result corresponding to each of the images to be detected may be obtained. Then, the image to be detected without defects can be taken as a defect-free image according to the first defect detection result corresponding to each image to be detected.

Illustratively, the target image is a defective image, and in performing the step of acquiring a non-defective image from the image to be inspected based on the first defect detection result, the method 100 may further include the steps of: and determining the images to be detected, which contain the defect area indicated by the first defect detection result, in the plurality of images to be detected as defective images.

In this exemplary scheme, a defective image among the plurality of images to be detected may be directly determined, and in a subsequent step (e.g., step S150), the defective image may be secondarily judged using the trained defect detection model, whereby the accuracy of the image detection method may be further improved.

According to another aspect of the present application, there is provided an image detection apparatus. Fig. 5 shows a schematic block diagram of an image detection apparatus according to an embodiment of the present application. As shown in fig. 5, the image detection apparatus 500 may include a first acquisition module 510, a first detection module 520, a second acquisition module 530, a training module 540, and a second detection module 550.

A first acquiring module 510, configured to acquire an image to be detected.

The first detection module 520 is configured to detect a defect of the image to be detected by using a preset defect detection algorithm to determine a first defect detection result, where the first defect detection result includes location information of a defect area in the image to be detected.

A second acquiring module 530, configured to acquire a defect-free image from the image to be detected based on the first defect detection result.

The training module 540 is configured to train the defect detection model using the defect-free image to obtain a trained defect detection model.

And a second detection module 550, configured to perform defect detection on the target image at least using the trained defect detection model to determine a second defect detection result, where the second defect detection result includes location information of a defect area in the target image, and the target image is an image to be detected or a defective image obtained from the image to be detected based on the first defect detection result.

According to another aspect of the present application, an electronic device is provided. Fig. 6 shows a schematic block diagram of an electronic device according to an embodiment of the application. As shown in fig. 6, the control device 600 includes a processor 610 and a memory 620. The memory 620 has stored therein a computer program. The processor 610 is configured to execute a computer program to implement the image detection method 100.

In the alternative, the processor may comprise any suitable processing device having data processing capabilities and/or instruction execution capabilities. For example, the processor may be implemented using one or a combination of several of a Programmable Logic Controller (PLC), a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), a Micro Control Unit (MCU), and other forms of processing units.

According to yet another aspect of an embodiment of the present application, there is also provided a storage medium. The storage medium has stored therein a computer program/instruction which, when executed by a processor, implements the image detection method 100 described above. The storage medium may include, for example, read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, or any combination of the preceding. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

Those skilled in the art will understand the specific implementation schemes of the image detection apparatus, the electronic device, and the storage medium by reading the above description about the image detection method 100, and for brevity, the description is omitted here.

Although the illustrative embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the above illustrative embodiments are merely illustrative and are not intended to limit the scope of the present application thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the application. All such changes and modifications are intended to be included within the scope of the present application as set forth in the appended claims.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The application may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third, etc. do not denote any order. These words may be interpreted as names.

The foregoing description is merely illustrative of specific embodiments of the present application and the scope of the present application is not limited thereto, and any person skilled in the art can easily think about variations or substitutions within the scope of the present application. The protection scope of the application is subject to the protection scope of the claims.

Claims

1. An image detection method, comprising:

Acquiring an image to be detected, wherein the image to be detected comprises a target object to be detected for defect detection;

Performing defect detection on the image to be detected by using a preset defect detection algorithm to determine a first defect detection result, wherein the first defect detection result comprises position information of a defect area in the image to be detected;

Acquiring a defect-free image from the image to be detected based on the first defect detection result;

training a defect detection model using the defect-free image to obtain a trained defect detection model;

performing defect detection on a target image at least by using the trained defect detection model to determine a second defect detection result, wherein the second defect detection result comprises position information of a defect area in the target image, and the target image is the image to be detected or a defective image obtained from the image to be detected based on the first defect detection result;

Wherein the image to be detected includes a plurality of identical areas to be detected, and the obtaining a defect-free image from the image to be detected based on the first defect detection result includes:

Extracting a plurality of target product images from the image to be detected, wherein each target product image comprises a single region to be detected;

Determining a target product image which does not contain a defect area indicated by the first defect detection result in the target product images as the non-defective image;

Or alternatively, the first and second heat exchangers may be,

The number of the images to be detected is plural, different images to be detected respectively contain the same products to be detected, and the obtaining of the defect-free images from the images to be detected based on the first defect detection result includes:

And determining the images to be detected, which do not contain the defect area indicated by the first defect detection result, in the plurality of images to be detected as the non-defect images.

2. The image detection method according to claim 1, wherein the performing defect detection on the target image using at least the trained defect detection model to determine a second defect detection result comprises:

extracting sub-images to be detected, which correspond to the positions of at least one target image one by one, from the target image;

performing defect detection on at least part of the extracted at least one sub-image to be detected by using the trained defect detection model to determine sub-defect detection results corresponding to each sub-image to be detected in the at least part of sub-images to be detected, wherein the sub-defect detection results comprise position information of sub-defect areas in the corresponding sub-images to be detected and/or areas of the sub-defect areas;

The second defect detection result comprises sub-defect detection results which are in one-to-one correspondence with the at least part of sub-images to be detected.

3. The image detection method according to claim 2, wherein the defect detection of at least part of the at least one sub-image to be detected using the trained defect detection model comprises:

And performing defect detection on the sub-image to be detected, of which the intersection exists with the defect area indicated in the first defect detection result, in the at least one sub-image to be detected by utilizing the trained defect detection model.

4. The image detection method according to claim 2 or 3, wherein the defect detection of at least part of the at least one sub-image to be detected using the trained defect detection model comprises:

and performing defect detection on the sub-image to be detected, of which the pixel value fluctuation degree is larger than a preset fluctuation degree, in the at least one sub-image to be detected by using the trained defect detection model.

5. The image detection method according to claim 2 or 3, wherein the performing defect detection on the target image using at least the trained defect detection model to determine a second defect detection result further comprises:

For each sub-image to be detected in the remaining sub-images to be detected, taking a first defect detection result corresponding to the sub-image to be detected as a sub-defect detection result corresponding to the sub-image to be detected;

Or alternatively, the first and second heat exchangers may be,

Performing defect detection on the remaining sub-images to be detected by using the trained defect detection model so as to determine sub-defect detection results corresponding to the remaining sub-images to be detected one by one;

The remaining sub-images to be detected are other sub-images to be detected except for the at least part of sub-images to be detected in the at least one sub-image to be detected, and the second defect detection result further comprises sub-defect detection results corresponding to the remaining sub-images to be detected in a one-to-one correspondence.

6. An image detection method according to any one of claims 1-3, wherein the preset defect detection algorithm is a defect detection based on differences between each of at least one image area in the image to be detected and a standard image.

7. The image detection method according to claim 6, wherein the pixels of each of the at least one image area correspond one-to-one to the pixels of the standard image, and the preset defect detection algorithm includes the operations of:

for each of the at least one image area,

Calculating a pixel value difference between a pixel value of each pixel in the image area and a pixel value of a corresponding pixel in the standard image, wherein the pixel difference between the image area and the standard image is represented by the pixel value difference;

and determining the region where the pixel value difference value corresponding to the image region is greater than a preset difference value threshold value as a defect region in the image to be detected.

8. The image detection method according to claim 6, wherein the image to be detected and the standard image are product images of a product to be detected, each of the at least one image area and the standard image each contain a single product to be detected, the preset defect detection algorithm further comprising the operations of:

acquiring a plurality of sample images containing a single product to be detected, wherein the width and the height of the sample images are different;

calculating a median value or a mean value of pixel values of pixels located at the same image coordinates in the acquired plurality of sample images;

and determining an image with the pixel value equal to the median value of the pixel values or the average value of the pixel values as the standard image.

9. The image detection method according to any one of claims 1 to 3, wherein the image to be detected includes a plurality of areas to be detected that are identical to each other, the target image is the defective image, and when the step of acquiring a non-defective image from the image to be detected based on the first defect detection result is performed, the method further comprises:

And determining a target product image containing a defect area indicated by the first defect detection result in the target product images as the defective image.

10. The image detection method according to any one of claims 1 to 3, wherein the number of the images to be detected is plural, different ones of the images to be detected each contain the same product to be detected, the target image is the defective image, and when the step of acquiring a non-defective image from the images to be detected based on the first defect detection result is performed, the method further comprises:

And determining the images to be detected, which contain the defect areas indicated by the first defect detection results, in the plurality of images to be detected as the defective images.

11. The image detection method according to any one of claims 1 to 3, wherein the training of the defect detection model using the non-defective image includes:

extracting sample sub-images corresponding to at least one target image position one by one on the defect-free image;

For each of the at least one target image position,

And training the defect detection model by using the sample sub-image corresponding to the target image position to obtain a trained defect detection model corresponding to the target image position.

12. An image detection apparatus, comprising:

the first acquisition module is used for acquiring an image to be detected, wherein the image to be detected comprises a target object to be detected;

The first detection module is used for carrying out defect detection on the image to be detected by using a preset defect detection algorithm so as to determine a first defect detection result, wherein the first defect detection result comprises position information of a defect area in the image to be detected;

the second acquisition module is used for acquiring a defect-free image from the image to be detected based on the first defect detection result;

a training module for training a defect detection model using the defect-free image to obtain a trained defect detection model;

the second detection module is used for carrying out defect detection on a target image at least by utilizing the trained defect detection model so as to determine a second defect detection result, wherein the second defect detection result comprises position information of a defect area in the target image, and the target image is the image to be detected or a defective image obtained from the image to be detected based on the first defect detection result;

wherein the image to be detected includes a plurality of identical areas to be detected, and the second acquisition module includes:

The extraction submodule is used for extracting a plurality of target product images from the image to be detected, and each target product image comprises a single region to be detected;

A first determination sub-module configured to determine, as the non-defective image, a target product image that does not include a defective area indicated by the first defect detection result from among the plurality of target product images;

Or alternatively, the first and second heat exchangers may be,

The number of the images to be detected is multiple, different images to be detected respectively contain the same products to be detected, and the second acquisition module comprises:

And the second determining submodule is used for determining an image to be detected, which does not contain the defect area indicated by the first defect detection result, in the plurality of images to be detected as the defect-free image.

13. An electronic device comprising a processor and a memory, wherein the memory has stored therein computer program instructions which, when executed by the processor, are adapted to carry out the image detection method of any of claims 1-11.

14. A storage medium having stored thereon program instructions for performing the image detection method according to any of claims 1-11 when run.