TW202147254A - Method for labeling image - Google Patents

Abstract

A method for labeling an image comprises: obtaining a target image of a target object; generating a reconstructed image according to the target image and a reconstruction model, wherein the reconstruction model is trained with a plurality of reference images and a machine learning algorithm, each of the reference image is an image of a reference object whose defect level is in a tolerable range with an upper limit, and each of the reference objects is associated with the target object; performing a first difference algorithm and a second difference algorithm respectively according to the target image and the reconstructed image to generate a first image and a second image respectively; and performing a pixel-wise operation according to the first image and the second image to generate an output image, wherein the output image includes a mark of a defect of the target object.

Description

How to tag images

The present invention relates to the field of image processing, in particular to a method for marking defects of objects in an image.

Products such as laptops or tablets need to be checked and confirmed by quality control personnel before final shipment to customers. These quality managers check products for scratches, dents, or other defects based on a learned guideline. If the severity of the defect exceeds the range allowed by the specification, the computer product is deemed to be "failed"; otherwise, the computer product is deemed to be "passed" for defect detection.

To detect defects in the appearance of computer products, multiple surface images of the computer product can be collected, the defect types are marked on the images, and machine learning for defect detection in an Automatic Optical Inspection (AOI) machine is trained. or deep learning models. Traditionally, object detection and classification are performed in a supervised manner. In the case of supervised learning, in order to improve the detection accuracy, it is necessary to collect a large amount of training data with labels, including normal samples and defective samples.

More training data means more labeling work. However, the collection and labeling of training data is labor-intensive and can be quite difficult. For example, if a computer product manufacturer does not have the infrastructure for collecting big data (especially large amounts of image data) and outsources data collection and labelling tasks, the security, integrity and confidentiality of data may cause problems. great attention. More importantly, as the life cycle of computer products is shortened and product designs tend to be diverse, it is impractical to collect and mark defects in computer surface images with sufficient diversity. The surface of a computer product can be any color, and can have any texture and texture. Furthermore, surface defects are of many types, such as scratches, dents, stains, and the like. Surface defects of the same type can come in various shapes and sizes. More seriously, some surface defects are not easy to classify. Inconsistent labels will inevitably appear in the training data. Conventional methods need to correctly classify and label defect detections in the training data in order to have good accuracy. Therefore, it is difficult to collect a large amount of consistent marker data with sufficient variety. After collecting and labeling enough training images, the computer products corresponding to these training images may be close to the market.

In view of this, the present invention proposes a method for labeling images to meet the needs of a large amount of training data.

A method for labeling an image according to an embodiment of the present invention, the method includes: obtaining a target image of a target object; generating a reconstructed image according to the target image and a reconstruction model, wherein the reconstruction model adopts a plurality of reference images and a machine learning The algorithm is trained, each reference image is an image of a reference object and the defect degree of the reference object falls within an allowable range with an upper limit, and each reference object is associated with the target object; according to the target image and the reconstructed image respectively executing a first difference algorithm and a second difference algorithm to generate a first difference image and a second difference image, respectively; and performing a pixel scale operation according to the first difference image and the second difference image to generate an output image, wherein the output image includes A marker of a defect in the target object.

To sum up, the method for marking images proposed by the present invention is suitable for classifying or detecting original images of computer products. The present invention reduces the need for a large number of labeled images as training data. The reconstruction model proposed by the present invention is not over-generalized, so that some defects are regarded as texture patterns in normal regions, so the present invention can reduce the occurrence of false negatives. The present invention mimics human perception by highlighting only anomalies and ignoring complex contexts. This perceptual-attention-based approach effectively reduces false positives.

The above description of the present disclosure and the following description of the embodiments are used to demonstrate and explain the spirit and principle of the present invention, and provide further explanation of the scope of the patent application of the present invention.

The detailed features and advantages of the present invention are described in detail below in the embodiments, and the content is sufficient to enable any person skilled in the relevant art to understand the technical content of the present invention and implement it accordingly, and according to the content disclosed in this specification, the scope of the patent application and the drawings , any person skilled in the related art can easily understand the related objects and advantages of the present invention. The following examples further illustrate the viewpoints of the present invention in detail, but do not limit the scope of the present invention in any viewpoint.

The method for labeling an image provided by the present invention is suitable for detecting defects of the target object itself, and adding supplementary labels related to the defects in the target image with the target object. In one embodiment, the target object is the surface of a computer product, such as the top cover of a notebook computer, and the defect is scratches, dents or stains on the top cover. In another embodiment, the target object is a printed circuit board, and the defect is a missing part, a skewed part, or a wrong part of the part.

Please refer to FIG. 1 , which is a flowchart of a method for marking an image according to an embodiment of the present invention. Referring to step S0, a reconstruction model is generated according to a plurality of reference images and a machine learning algorithm. Each of the plurality of reference images is an image of a reference object, and the defect degree of the reference object falls within an allowable range with an upper limit value. Each reference object is associated with a target object. Specifically, the reference object is a qualified sample of the target object, or referred to as an allowable sample. The reference object is, for example, the top cover of a notebook computer. According to the specification requirements, the reference object has no defects, or the number and degree of defects in the reference object are within the allowable range. For example, please refer to Table 1, the allowable range includes, for example, the defect types of the first level and the second level, and the upper limit is the maximum boundary value defined in the second level (20 mm, 2 bars, 1 mm ² and /or 2) or, for example, includes the defect types of the first class, and the upper limit is the maximum boundary value defined in the first class (12 mm, 2 strips, 0.7 mm ² and/or 3). For the convenience of description, "the degree of defect is within the allowable range" is abbreviated as "without defect" hereinafter.

Form 1 Defect type first level second level third level scratches Length: 12 mm Pass: 2 Length: 20 mm Pass: 2 Length: 25 mm Pass: 1 Sag 0.5 mm ² ~ 0.7 mm ² pass: 3 0.5 mm ² ~ 1 mm ² Pass: 2 1 mm ² ~ 1.3 mm ² pass: 1

In one embodiment, the machine learning algorithm described in step S0 is an auto-encoder. In another embodiment, the machine learning algorithm described in step S0 is a one-class support vector machine (one-class support vector machine). The machine learning algorithm uses multiple reference images obtained by shooting multiple reference objects as training data and trains a reconstruction model. Reconstruction models, or generative models, are models used to describe qualified samples. After the pre-training of the reconstructed model in step S0 is completed, steps S1 to S5 shown in FIG. 1 are the actual operation stage.

Please refer to step S1 to obtain the target image. In this step S1, a target image of the target object is captured by, for example, a camera device. The target object is, for example, a top cover of a notebook computer or a printed circuit board. For ease of illustration, the target object has one or more defects that are outside the allowable range. However, after the method for marking images proposed by the present invention is executed, a situation of "the target object does not have defects" may also occur.

Please refer to step S2 to generate a reconstructed image according to the target image and the reconstructed model. For example, the camera device sends the target image obtained in step S1 to the processor. The processor generates a reconstruction image according to the target image and the reconstruction model. The reconstructed image can be considered a "defect-free" target image. The way of generating the reconstructed image from the reconstruction model includes, but is not limited to, selecting one of a plurality of candidate reconstructed images, generating a reconstructed image from a plurality of feature prototypes in a linear combination manner, or outputting the reconstructed image according to an image transformation function.

If the target object in the target image has defects, after the reconstructed image is generated in step S2, there is a reconstruction error between the reconstructed image and the target image. Please refer to step S3 and step S4. In step S3, the processor executes a first difference algorithm according to the target image and the reconstructed image to generate a first difference image, and in step S4, the processor executes a second difference algorithm according to the target image and the reconstructed image to generate a second difference image. In steps S3 and S4, the processor calculates the reconstruction error according to different comparison scales. Steps S3 and S4 can be executed simultaneously or sequentially, and the present invention does not limit the order in which the processor executes steps S3 and S4.

Please refer to FIG. 2 , which shows a detailed flowchart of step S3 in an embodiment of the present invention.

Please refer to step S31 and step S32. Step S31 is to generate a first feature map according to the target image and a neural network model. Step S32 is to generate a second feature map according to the reconstructed image and the neural network model. The first feature map and the second feature map respectively have one or more feature blocks, and these feature blocks represent parts of the feature map that need attention. The size of the feature block is, for example, a rectangular patch with a length and a width of 64 pixels, and the present invention does not limit the length and width of the patch. Feature maps can also be called deep features.

In one embodiment, the neural network model used in steps S31 and S32 is, for example, SqueezeNet. In other embodiments, the neural network model may be AlexNet or ResNet. In one embodiment, the neural network model is pre-trained with multiple images from a large visual database (eg, ImageNet) that are not associated with the target object. During training, each pixel of each image is used to extract a rectangular block (for example, a rectangular block with a length and width of 64 pixels) including the pixel as training data. In another embodiment, the neural network model is first trained with multiple images not associated with the target object, and then the neural network model is fine-tuned including multiple images associated with the target object, thereby improving feature extraction accuracy. The feature map output by the trained neural network model in the feature extraction stage has a feature identification strategy similar to that of human visual perception.

Please refer to step S33 to calculate the difference between the first feature map and the second feature map as a first difference image. For example, the first difference image is the subtraction of the first feature map and the second feature map. The first difference image is a perceptual attention map, which has the effect of imitating human comparison of image differences. To be more specific, when humans compare the reference image and the target image, they do not particularly notice whether there is a slight displacement or a slight difference between the two images, but can easily observe the difference between the feature blocks in the two images. The first disparity algorithm described in steps S31 to S33 calculates the reconstruction error at a coarse level from a block perspective.

In general, the autoencoder uses an L2 loss function or a structural similarity index (SSIM) to calculate the reconstruction error between the target image and the reconstructed image. However, these metrics are usually sensitive to slight overall changes, so the above metrics will not work well when alignments focus on texture pattern similarity rather than precise alignment. Even if the defect degree of the target object in the target image is not serious, if there is a small amount of displacement between the target image and the reconstructed image, the above-mentioned indicators may increase unnecessary reconstruction errors. Therefore, the present invention adopts the first difference algorithm introduced in steps S31 to S33 to emphasize the matching between higher-order structures and feature representations. In general, the first difference image generated by applying the first difference algorithm has the effect of emphasizing a region of interest (ROI) and reducing background noise.

Referring to step S4, a second difference algorithm is executed according to the target image and the reconstructed image to generate a second difference image. The second difference algorithm is that the processor calculates a relative error according to each pixel of the reconstructed image and the target image. The relative error value is, for example, the pixel-wise square error of each pixel in the two images or the pixel-wise absolute difference of each pixel. In this step S4 , the processor obtains the defect position of the target object in the target image by calculating at the pixel level.

Referring to step S5, a pixel-wise operation is performed according to the first difference image and the second difference image to generate a first output image. In one embodiment, the pixel scale operation is a bitwise multiplication. To be more specific, in step S5, for a certain position of the first difference image and the same position of the second difference image, if the processor determines that the pixel values of these two positions are both indicated as defects, the first output image will keep this position. Location flaws. If only one of the first difference image and the second difference image determines that the pixel value of a certain position indicates a defect, the first output image does not retain the defect at this position.

In one embodiment, after step S5 is completed, the processor may mark defects in the first output image according to whether each pixel in the first output image is indicated as a defect. In another embodiment, in order to further reduce false positives, after step S5 is performed, the processor may continue to perform step S6 to improve the marking accuracy.

Referring to step S6, a multi-threshold generating process is performed according to the first output image to generate a second output image and a marker, wherein the first threshold is greater than the second threshold. The first threshold is used to obtain pixels that may be defective, and the second threshold is used to extend these pixels that may be defective to surrounding pixels.

Please refer to FIG. 3 , which shows a flow chart of the multi-threshold generating procedure described in step S6 . Please refer to step S61 and step S62. In step S61, the processor performs a binarization process on the first output image according to the first threshold to generate a third output image, and in step S62, the processor performs a binarization process on the first output image according to the second threshold to generate a The fourth output image. Steps S61 and S62 process the first output image according to different thresholds. Steps S61 and S62 can be executed simultaneously or sequentially, and the present invention does not limit the sequence of executing steps S61 and S62. In one embodiment, the processor calculates the average value A and the standard deviation S (standard deviation) of the reconstruction errors between the plurality of reference images and the reconstructed images, and sets the first threshold as A+4S and the second threshold as A+4S for A+S.

Please refer to step S63 to select the defective block in the third output image. In particular, defective portions can be captured from the high threshold processed third output image. Referring to step S64, according to the position corresponding to the defective block in the fourth output image, it is determined whether the pixels around the position have defects so as to selectively expand the defective block. For example, if the coordinate of the center point of the selected defective block in the third output image is (123, 45), the processor searches for the surrounding pixels of the pixel whose coordinate is (123, 45) in the fourth output image, Including pixels with coordinates (122, 45), (124, 45), (123, 43), (123, 46), etc., and then judging whether these pixels of the fourth output image are defects. If the determination result is "Yes", the defective block and the surrounding pixels that are also defective are retained in the second output image. In one embodiment, step S64 may employ, for example, a flood fill algorithm to generate a connectivity graph including defective blocks.

According to the second output image generated in step S6, the processor determines the pixels belonging to defects, and further marks these defects. The multi-threshold generating procedure proposed in the present invention in step S6 can reduce false positive image markers.

4 to 11 are examples of images generated after the steps of FIGS. 1 to 3 are performed.

Please refer to FIG. 4 , which is an example of the target image obtained after step S1 is executed. The target object in FIG. 4 is a printed circuit board and a circuit component. It can be seen that the circuit component has three pins, and the middle pin is not correctly inserted into the socket on the circuit board.

Please refer to FIG. 5 , which is an example of a reconstructed image generated after step S2 is executed. It can be seen from FIG. 5 that each pin of the circuit element in the target object "without defects" is inserted into the socket.

Please refer to FIG. 6 , which is an example of the first difference image generated after step S3 is executed. It can be seen from FIG. 6 that the lower half of the first difference image has a white area, which is more recognizable than the upper half of the first difference image.

Please refer to FIG. 7 , which is an example of the second difference image generated after step S4 is executed. Figure 7 presents the reconstruction error in pixel scale, so more detail corresponding to Figure 5 can be seen from Figure 7 .

Please refer to FIG. 8 , which is an example of the first output image generated after step S5 is executed. The contrast between the defect portion and the periphery of the circuit element is higher in FIG. 8 than in FIGS. 6 and 7 .

Please refer to FIG. 9 , which is an example of the third output image generated after step S62 is executed. FIG. 9 is the result of performing the binarization process according to FIG. 8 and the second threshold.

Please refer to FIG. 10 , which is an example of the fourth output image generated after step S61 is executed. FIG. 10 is the result of performing the binarization process according to FIG. 8 and the first threshold. The location of the defect is evident from Figure 10.

Please refer to FIG. 11 , which is an example of the second output image and the mark generated after step S64 is executed. Labeled as the box in Figure 11 indicating the location of the defect.

Please refer to FIG. 12 , which is an example of manually marking defects in an image according to an embodiment of the present invention. It can be seen from Fig. 12 that the markers obtained by using the present invention are very close to the real results.

In practice, after executing the process shown in Figure 1, the obtained labeled images can provide, for example, a defect detection model implemented by a Region-based Convolutional Neural Network (R-CNN). . The area-based convolutional neural networks are, for example, Fast R-CNN, Faster R-CNN, Mask R-CNN, YOLO (You Only Look Once) or SSD (Single Shot Detection).

To sum up, the method for marking images proposed by the present invention is suitable for classifying or detecting original images of computer products. The present invention reduces the need for a large number of labeled images as training data. The reconstruction model proposed by the present invention is not over-generalized, so that some defects are regarded as texture patterns in normal regions, so the present invention can reduce the occurrence of false negatives. The present invention mimics human perception by highlighting only anomalies and ignoring complex contexts. This perceptual-attention-based approach effectively reduces false positives.

Although the present invention is disclosed in the foregoing embodiments, it is not intended to limit the present invention. Changes and modifications made without departing from the spirit and scope of the present invention belong to the scope of patent protection of the present invention. For the protection scope defined by the present invention, please refer to the attached patent application scope.

S0~S6: Steps S31~S33: Steps S61~S64: Steps

FIG. 1 is a flowchart illustrating a method for marking an image according to an embodiment of the present invention. FIG. 2 is a detailed flowchart of step S3 in an embodiment of the present invention. FIG. 3 is a flowchart illustrating a multi-threshold generation process according to an embodiment of the present invention. FIG. 4 is an example of a target image in an embodiment of the present invention. FIG. 5 is an example of a reconstructed image according to an embodiment of the present invention. FIG. 6 is an example of a first difference image in an embodiment of the present invention. FIG. 7 is an example of a second difference image in an embodiment of the present invention. FIG. 8 is an example of a first output image in an embodiment of the present invention. FIG. 9 is an example of a third output image in an embodiment of the present invention. FIG. 10 is an example of a fourth output image in an embodiment of the present invention. FIG. 11 is an example of a second output image in an embodiment of the present invention. FIG. 12 is an example of manually marking defects in an image according to an embodiment of the present invention.

S0~S6: Steps

Claims (9)

A method of marking an image, the method comprising: obtaining a target image of a target object; generating a reconstructed image according to the target image and a reconstruction model, wherein the reconstruction model adopts a plurality of reference images and a machine learning algorithm obtained from training, each of the reference images is an image of a reference object and the defect degree of the reference object falls within an allowable range with an upper limit, and each of the reference objects is associated with the target object; according to the target The image and the reconstructed image respectively execute a first difference algorithm and a second difference algorithm to generate a first difference image and a second difference image respectively; and execute a difference according to the first difference image and the second difference image Pixel scale operations are performed to generate an output image, wherein the output image includes an indication of a defect of the target object. The method of labeling an image of claim 1, wherein the reconstruction model is an autoencoder. The method for labeling an image as claimed in claim 1, wherein the first difference algorithm comprises: generating a first feature map according to the target image and a neural network model; generating a first feature map according to the reconstructed image and the neural network model two feature maps; and calculating a difference degree between the first feature map and the second feature map, wherein the first difference image includes the difference degree. The method for labeling an image as claimed in claim 3, wherein the neural network model is SqueezeNet. The method for labeling an image as claimed in claim 3, wherein the neural network model is trained on a plurality of images unrelated to the target object. The method for marking an image as claimed in claim 1, wherein the second difference algorithm comprises: calculating a relative error value according to each pixel of the reconstructed image and the target image. A method of labeling an image as claimed in claim 6, wherein the relative error value is a squared error or an absolute value error. The method of marking an image of claim 1, wherein the pixel scale operation is a bitwise multiplication. The method for marking an image as claimed in claim 1, wherein the output image is a first output image, and after the pixel scale operation is performed according to the first difference image and the second difference image to generate the output image, further comprising: : perform a binarization process on the first output image to generate a third output image and a fourth output image according to a first threshold and a second threshold, wherein the first threshold is greater than the second threshold; select the A defective block in the third output image; and according to a position corresponding to the defective block in the fourth output image, determining whether pixels around the position have defects to selectively extend the defective block.

Images