TWI808019B - Method for filtering surface type of object based on artificial neural network and system - Google Patents

Abstract

A method for filtering surface type of an object based on an artificial neural network includes: identifying, by a first detection module, an object image to classify the object image to one of a good product group and a first bad product group and outputting a marked image of the object image classified to the first bad product group, wherein the marked image is the object image having at least one defective mark; when the object image is classified to the first bad product group, capturing, corresponding to the at least one defective mark, at least one quasi-defective image from the marked image; and classifying, by a second detection module, the object image into one of a fuzzy group and a second bad product group according to the at least one quasi-defective image, wherein the second detection module classifies the object image into the fuzzy group when the detection result of all the quasi-defective image(s) is flawless.

Description

Screening Method and System for Object Surface Type Based on Artificial Neural Network

The present invention relates to the screening technology of the surface form of objects, in particular to a screening method of the surface form of objects based on artificial neural network.

Various safety protection measures are composed of many small structural objects, such as seat belts. If the strength of these small structural objects is insufficient, the protective effect of safety protection measures can be questioned.

During the manufacturing process of these structural objects, due to various reasons, such as collisions, process errors, mold defects, etc., there may be tiny defects on the surface, such as deformation, bruises, sand holes, broken columns, collapse, die drawing, foreign objects, burrs, etc. These tiny blemishes are not easy to detect. One of the traditional defect inspection methods is to manually observe the structural object to be inspected with naked eyes or touch it with both hands to determine whether the structural object has defects. However, the efficiency of manually detecting whether a structural object has defects is low, and misjudgment is very likely to occur, which will cause the yield rate of the structural object to be uncontrollable.

Another defect detection method is to detect through artificial neural network. However, the defect detection results of the traditional artificial neural network are only simply classified into good products and defective products, and there are still cases of misjudgment. In addition, the defect detection of traditional artificial neural network only considers a single parameter, and the accuracy is low.

In one embodiment, the present invention provides a method for screening object surface types based on artificial neural network. The method for screening the object surface type based on the artificial neural network includes: using the first detection module to identify the object image to classify the object image into one of the good product group and the first defective product, and output a marked image of the object image classified into the first defective product group, wherein the marked image is an object image with at least one defect mark; when the object image is classified into the first defective product group, at least one quasi-defective image is extracted from the marked image corresponding to at least one defect mark; The image is classified into one of the fuzzy group and the second defective product group, wherein the second detection module classifies the object image into the fuzzy group when at least one quasi-defective image is detected to be free of defects.

In an embodiment, the present invention further provides an artificial neural network-based object surface type detection system, including: a first detection module and a second detection module. The first detection module identifies an object image to classify the object image into one of the good product group and the first defective product group, and outputs a tagged image of the object image classified into the first defective product group, and extracts at least one quasi-defective image from the tagged image. The second detection module classifies the object image into one of the fuzzy group and the second defective product group according to the at least one quasi-defective image, wherein the object image is classified into the fuzzy group when no defect is detected in the at least one quasi-defective image.

To sum up, in any embodiment of the artificial neural network-based object surface type screening method and detection system, firstly, the first detection module classifies the object images according to the more stringent and easy-to-consistency defect standards, so as to reduce the misclassification caused by the omission of the calibration personnel or the different calibration standards. Afterwards, the second detection module is used to classify at least one quasi-defective image of the object images classified into the first defective product group, so as to rescue overkill objects. In this way, the calibration project of the object can be greatly simplified, the model defects caused by the calibration operation can be reduced, and the detection accuracy can be more stable. In addition, due to the multi-stage classification, each stage model (ie, the first detection module and the second detection module) is responsible for different tasks, from strict to wide, and from wide to subtle, making the division of labor of each stage model more specific and easier to converge. Furthermore, problems such as calibration problems, complex samples, and uneven defect categories can be solved, so that testing equipment using this screening method can be brought online earlier. In addition, because of strict killing first and then rescue, the operators who use the testing equipment using this screening method do not have to worry about the outflow of defective products, which improves the reliability of the testing equipment.

The detailed features and advantages of the present invention are described in detail below in the embodiments, the content of which is sufficient to enable any skilled person to understand the technical content of the present invention and implement it accordingly, and according to the content disclosed in this specification, the scope of the patent application and the drawings, any skilled person can easily understand the related objectives and advantages of the present invention.

To make the above objects, features and advantages of the embodiments of the present invention more comprehensible, a detailed description is given below in conjunction with the accompanying drawings.

FIG. 1 is a schematic block diagram of an embodiment of a detection system, and FIG. 2 is a schematic flowchart of an embodiment of a method for screening the surface type of an object. Please refer to FIG. 1 and FIG. 2 , the detection system 100 can implement any embodiment of the method for screening object surface types based on the artificial neural network 110 , 120 . In some implementation aspects, the detection system 100 can be implemented on a processor. In addition, the processor can be applied in a detection device.

In one embodiment, the detection system 100 includes two artificial neural networks (hereinafter respectively referred to as the first artificial neural network 110 and the second artificial neural network 120 ), and the second artificial neural network 120 is serially connected to the output of the first artificial neural network 110 . In other words, the first artificial neural network 110 is the first stage in the detection system 100 , and the second artificial neural network 120 is the second stage in the detection system 100 . Wherein, the first artificial neural network 110 has a first detection module 111 , and the second artificial neural network 120 has a second detection module 121 .

In some embodiments, in the learning phase, the first artificial neural network 110 (an untrained artificial neural network at this time) may use different training conditions to perform deep learning on the same or different multiple object images I11-I1N, so as to establish the first detection module 111 (an artificial neural network that has been trained at this time) for identifying and classifying the surface type of the object. Wherein, N is a positive integer greater than 1.

In some embodiments, the plurality of object images I11-I1N may be images of surfaces of the same object at the same relative position. In other words, the plurality of object images I11-I1N are surface images of the first object to the Nth object respectively, and the first object to the Nth object are the same kind of object. Here, the first artificial neural network 110 can receive a plurality of object images I11 - I1N with fixed image coordinate parameters. Wherein, the object images I11-I1N may be obtained by capturing images of surfaces of multiple objects. When the surface of the object has any surface type, the corresponding image positions in the object images I11-I1N of each object also have images of these surface types. In some embodiments, the surface pattern can be, for example, surface structures such as slots, cracks, bumps, sand holes, pores, impact marks, scratches, edges, and textures. Wherein, each surface structure is a three-dimensional fine structure, and its size is from submicron to micron. In other words, the longest side or longest diameter of the three-dimensional microstructure is between submicron to micron. Wherein, submicron refers to less than 1 micron, such as 0.1 micron to 1 micron. For example, the three-dimensional microstructure may be a microstructure of 300 nanometers to 6 micrometers. In some implementations, the object may be a seat belt bearing, and the object image thereof is the surface image of the seat belt bearing (the object image I11 shown in FIG. 7 ).

In some embodiments, in the learning phase, the second artificial neural network 120 (an artificial neural network that has not been trained at this time) can use different training conditions to perform deep learning on the same or different quasi-defective images, so as to establish a second detection module 121 (an artificial neural network that has been trained) for identifying whether an object has a defect and classifying it accordingly. Here, each quasi-defective image is an object image, such as a partial image in the object image I11 (such as quasi-defective images I510 - I519 shown in FIG. 10 ), and each quasi-defective image may or may not contain a defective tile. Wherein, the defect image block is formed by imaging corresponding to the defect on the surface of the object. In some embodiments, defects may include but not limited to deformation, bruise, sand hole, broken column, collapse, die, foreign matter, burrs, etc., or any other artificially defined defects on the surface.

In some implementation aspects, the training conditions may be, for example, different numbers of neural network layers, different neuron configurations, different preprocessing of input images, different neural network algorithms or any combination thereof. Wherein, image preprocessing may be feature enhancement, image cropping, data format conversion, image superposition or any combination thereof. In addition, the first detection module 111 of the first artificial neural network 110 and the second detection module 121 of the second artificial neural network 120 can be respectively implemented using a convolutional neural network (CNN) algorithm, but this case is not limited thereto.

In some embodiments, the first artificial neural network 110 of the detection system 100 receives at least one object image I11 - I1N (step S10 ). In some implementation aspects, the first artificial neural network 110 receives the object images I11 - I1N simultaneously or sequentially.

In some embodiments, the object images I11 - I1N of each object may be spliced from multiple perspective images of an object (ie, a set of perspective images V1 - VA ). Therefore, in an embodiment of step S10 , for example, an image processing module 112 may be used to respectively receive the multiple sets of perspective images V1-VA (step S11 ), and the image processing module 112 will output the multiple sets of perspective images V1-VA into multiple object images I11-I1N (step S12 ) to the first artificial neural network 110 . Wherein, A is a positive integer greater than 1.

In some implementations, each set of perspective images V1-VA is sequentially input to the image processing module 112 according to the capture order, so that the image processing module 112 can sequentially splice object images I11-I1N accordingly. The sizes of the images V1-VA of each viewing angle can be approximately the same to facilitate the splicing operation. For example, the dimensions (length x width) of each perspective image V1-VA are 800 (pixels) and 800 (pixels), and each set of perspective images V1-VA has a total of 7 x 8 pieces to form object images I11-I1N with a size of 5600 (pixels) x 6400 (pixels). In addition, the image processing module 112 can be implemented by using but not limited to a processor. In some implementation aspects, the image processing module 112 can be integrated into the first artificial neural network 110 . In other words, the first artificial neural network 110 may include an image processing module 112 and a first detection module 111 , and the first detection module 111 is serially connected to the output of the image processing module 112 , as shown in FIG. 1 .

After the first detection module 111 receives the object images I11-I1N (step S10), the first detection module 111 can identify each object image I11-I1N to classify the object images I11-I1N into the good product group G1 or the defective product group G2 (hereinafter referred to as the first defective product group G2), and output the tagged images I31, I37 of the object images I11, I17 classified into the first defective product group G2 (step S20 ). For example, the first detection module 111 of the first artificial neural network 110 classifies the object images I11, I17 into the first defective product group G2, classifies the object images I12-I16, I18-I1N into the good product group G1, and outputs the marked images I31, I37 of the object images I11, I17. In some embodiments, the marker images I31, I37 are the object images I11, I17 with the defect marker M1. For example, the marker image I31 is an object image I11 in which 10 defect images in the image are framed by 10 frame lines (ie, defect markers M1 ), as shown in FIG. 9 .

FIG. 3 is a schematic flowchart of an embodiment of step S20. Please refer to FIG. 1 to FIG. 3 , in an embodiment of step S20 , the first detection module 111 may respectively perform a defect prediction on each object image I11 - I1N to generate a corresponding defect heat map (step S21 ). For example, the first detection module 111 performs defect prediction on the object image I11 to form a defect heat map I6 of the object image I11 , as shown in FIG. 8 . Next, the first detection module 111 can perform defect screening on each defect heat map according to a defect standard (step S22 ). Wherein, please refer to FIG. 1 to FIG. 3 , in defect screening, if it is determined that the defect standard is met, the first detection module 111 will classify the object image, for example, classify the object images I11 and I17 into the first defective product group G2 (step S23 ). Conversely, in defect screening, if it is determined that the defect standard is not met, the first detection module 111 will classify the object image, for example, classify the object images I12-I16, I18-I1N into the good product group G1 (step S24). After the classification (which can be referred to as the first stage of classification) is completed, the first detection module 111 can generate the marked images I31, I37 according to the object images I11, I17 classified into the first defective product group G2 and their corresponding defect heat maps (step S25).

In some embodiments, please refer to FIG. 7 and FIG. 8 , the defect heat map I6 is a probability distribution map representing defects in the object image I11 . The defect heat map I6 includes a plurality of pixels, and each pixel has a defect score. Wherein, the blemish score is used to indicate the possibility of a blemish in the object image I11, and its value can be between 0 and 1. The higher the blemish score, the higher the probability that this location (the coordinate position of the pixel) is a blemish. Here, please refer to FIG. 1 , FIG. 7 and FIG. 8 , the defect score is determined by the machine based on the results in the learning stage, for example, the first detection module 111 predicts the pixels that may belong to the defect block in each object image I11-I1N for each object image I11-I1N according to the result in the learning stage, so as to give the corresponding defect score according to the possibility that each pixel in each object image I11-I1N belongs to the defect block, and generate each pixel according to its corresponding defect score The corresponding defect heat map displayed in the form of hot spots. In other words, the defect heat map I6 has the same image size as the object image I11. For example, the dimensions (length x width) of the defect heat map I6 and the object image I11 are both 5600 (pixels) x 6400 (pixels). Moreover, in the blemish heat map I6, brighter pixels indicate higher blemish scores and higher probability of blemishes (blocks) appearing in the same corresponding position of the object image I11, as shown in FIG. 8 . In some implementation aspects, the size (length x width) of the defect heat map of each object image I11-I1N may be 700 (pixels) x 800 (pixels), that is, it is composed of 560,000 digital arrays.

In some embodiments, the defect criteria may include but not limited to confidence threshold, area threshold and quantity threshold. FIG. 4 is a schematic flowchart of an embodiment of step S22. Please refer to FIG. 1 to FIG. 4, in an embodiment of step S22, in the defect screening of the defect heat map of each object image I11-I1N, the first detection module 111 can first filter out at least one defect pixel from the plurality of pixels in the defect heat map according to the confidence threshold (step S221). For example, the first detection module 111 can compare the defect score of each pixel with a confidence threshold, and filter out pixels whose defect scores are greater than or equal to the confidence threshold. Wherein, adjacent defective pixels may form a defective area. In some implementations, the first detection module 111 can set the value of each pixel whose defect score is less than the confidence threshold to 0, and set the value of each pixel whose defect score is greater than or equal to the confidence threshold to 1, so as to filter out the defect pixels (that is, the pixels whose value is set to 1).

After the screening, the first detection module 111 will count the number of blemishes whose area is greater than the area threshold (hereinafter referred to as the number of blemishes) (step S222 ). For example, the first detection module 111 can firstly calculate the area value of each defect area, and then compare the area value of each defect area with the area threshold to obtain the number of defects. Next, the first detection module 111 can compare the number of defects with the number threshold to determine whether the number of defects is greater than the number threshold (step S223 ). Wherein, when the number of defects is greater than or equal to the number threshold, the first detection module 111 determines that the defect standard is met (step S224 ), and proceeds to step S23 to classify the corresponding object images, such as object images I11 and I17, into the first defective product group G2. Conversely, when the number of defects is less than the quantity threshold, the first detection module 111 determines that the defect standard is not met (step S225 ), and proceeds to step S24 to classify the corresponding object images, such as object images I12-I16, I18-I1N, into the good product group G1.

In some implementation aspects, the confidence threshold, the area threshold and the quantity threshold of the flaw standard may be the combination with the highest accuracy calculated by using multiple test samples (for example, 100) for testing. For example, the confidence threshold can be [0.5, 0.6, 0.7, 0.8, 0.9], the area threshold can be [0, 3, 5, 7, 9], and the quantity threshold can be [0, 1, 2, 3], and there are 5*5*4 combinations, and a set of numerical combinations with the highest accuracy can be calculated after testing, but this case is not limited to this.

FIG. 5 is a schematic flowchart of an embodiment of step S25. Please refer to FIG. 1 to FIG. 5 , FIG. 7 and FIG. 8 , in an embodiment of step S25 , the first inspection module 111 can superimpose the object image I11 and the corresponding defect heat map I6 into a superimposed image (step S251 ). Here, the superimposed image includes at least one candidate defect block. Wherein, each candidate defect block refers to the part of the block corresponding to the adjacent defective pixels (which will form a defect) on the defect heat map I6 in the superimposed image on the object image I11 in the superimposed image. After obtaining the superimposed image, the first detection module 111 performs calculations according to the intersection candidate defect blocks in the superimposed image, and according to the calculation results, for example, frame each candidate defect block with a frame line to obtain a marked image I31 with at least one defect mark M1 (step S252 ). In some implementation aspects, frame lines may be set correspondingly according to the shape and range size of each candidate defect block. However, this application is not limited thereto. In some other embodiments, the frame line may have a fixed shape, such as a rectangle, a circle, an ellipse, etc., and at least encloses the defect candidate blocks.

In some embodiments, the number of candidate defect blocks may be the same as the number of defects on the defect heat map I6. But this case is not limited thereto. In some other embodiments, the number of candidate defect blocks may be less than the number of defects on the defect heat map I6. For example, the first detection module 111 can first find a predetermined number of defects from all the defects on the defect heat map I6 in the superimposed image, and frame the predetermined number of defects with frame lines to become the candidate defect blocks. Specifically, the first detection module 111 can first find out the 10 most likely defect parts from all the defects on the defect heat map I6 in the superimposed image, and only frame the 10 most likely defect parts with a frame line to become the candidate defect block. Here, the first detection module 111 can first calculate the total defect score of each defect, that is, the sum of the defect scores of all the pixels of the defect, and then sort the total defect scores from large to small, and take the top 10 defects in the total defect score after sorting as the 10 most likely defects.

Here, the object images I11 and I17 classified into the first defective product group G2 are predicted again by the second artificial neural network 120 . Before re-prediction, the second artificial neural network 120 first uses the labeled images I31, I37 of the object images I11, I17 to perform dimension reduction (dimension reduction) on the object images I11, I17, and then performs re-prediction with the dimensionality-reduced images. In this way, the amount of data subsequently input to the second detection module 121 for re-prediction can be greatly reduced. In addition, the learning of the second artificial neural network 120 in the learning phase can be made more focused.

Referring back to FIG. 1, FIG. 2, FIG. 7 and FIG. 9, after step 20, for the object images I11, I17 classified into the first defective product group G2, the image processing module 122 will extract at least one quasi-defective image I51s, I57 of each object image I11, I17 from the marked image I31, I37 according to at least one defect mark M1 in the mark image I31, I37 of each object image I11, I17 s (step S30).

FIG. 6 is a schematic flowchart of an embodiment of step S30. 1 to 6, FIG. 9 and FIG. 10, in an embodiment of step S30, for the object images I11 and I17 of the first defective product group G2 (the object image I11 is used as an example for description below), the image processing module 122 can first extract the partial images corresponding to each defect mark M1 from the mark image I31 according to at least one defect mark M1 in the mark image I31 (step S31). Here, each partial image covers at least one of the candidate defect blocks. In other words, the size of the partial image varies with the size of the defect candidate block (ie, the size of each defect in the object image I11 ). Therefore, after obtaining the partial images corresponding to each defect mark M1, the image processing module 122 can resize each partial image into quasi-defect images I510-I519 with a predetermined size (step S32). In other words, the number of defect marks M1 in the marker image I31 is the same as the number of the extracted quasi-defect images I510-I519. In some implementation aspects, the predetermined size (length x width) of each quasi-defective image I510 - I519 may be 128 (pixels) x 128 (pixels). In some other implementation aspects, the predetermined size (length x width) of each quasi-defective image I510 - I519 may be 800 (pixels) x 800 (pixels). It should be noted that the predetermined size of each quasi-defective image I510-I519 may depend on the design.

In some implementation aspects, the image processing module 122 can be implemented by using but not limited to a processor. In addition, in some implementation aspects, the image processing module 122 can be integrated into the second artificial neural network 120 . In other words, the second artificial neural network 120 may include an image processing module 122 and a second detection module 121 . Wherein, the second detection module 121 is serially connected to the output of the image processing module 122 , as shown in FIG. 1 . In other embodiments, the image processing module 122 may also be connected in series between the first artificial neural network 110 and the second artificial neural network 120 (not shown).

Referring back to FIG. 1 and FIG. 2 , for the object images I11 and I17 of the first defective product group G2, after obtaining at least one quasi-defective image I51s and I57s of the object images I11 and I17 (that is, the dimensionality reduction of the object images I11 and I17 is completed) (step S30), the detection system 100 can use the second detection module 121 of the second artificial neural network 120 to classify and correspond to the quasi-defective images I51s and I57s The object images I11 and I17 are sent to the fuzzy group G3 or the defective product group G4 (hereinafter referred to as the second defective product group G4) (step S40 ). Here, the second detection module 121 classifies the corresponding object image I17 into the fuzzy group G3 when all quasi-defective images I57s to which the object image I17 belongs are detected to be flawless (step S50 ). On the contrary, if any quasi-defective image I51s to which the object image I11 belongs is detected to be defective, the second detection module 121 classifies the corresponding object image I11 into the second defective product group G4 (step S60 ).

In an embodiment of step S40, for each object image I11, I17 of the first defective product group G2, the second detection module 121 can individually determine whether the confidence level of each quasi-defective image I51s, 157s (which can be called a defect confidence level) is less than a predetermined threshold, so as to choose to proceed to step S50 or step S60 according to the judgment result (which can be called the second stage of classification). Wherein, when the confidence levels of all the quasi-defect images I57s to which the object image I17 belongs are less than the predetermined threshold, the second detection module 121 will continue to perform step S50, ie classify the object image I17 into the fuzzy group G3. When the confidence level of any quasi-defective image I51s to which the object image I11 belongs (for example, any one of the quasi-defective images I510-I519 shown in FIG. 10 ) is greater than or equal to the predetermined threshold, the second detection module 121 proceeds to step S60, namely classifying the object image I11 into the second defective product group G4.

In some implementation aspects, the confidence level of each quasi-defect image I51s, 157s is predicted by the second detection module 121, and its value can be between 0 and 1. The predetermined threshold can be obtained through statistics in the learning phase, such as but not limited to 0.5.

In some implementations of step S40, the second detection module 121 can first output a score value of each object image I11, I17 according to the judgment result of the quasi-defective image I51s, 157s of each object image I11, I17, as the basis for its classification, and then classify each object image I11, I17 into the fuzzy group G3 or the second defective product group G4 according to the score value of each object image I11, I17. For example, assume that there are object images I11 and I17 classified into the first defective product group G2. When the judgment result of the object image I17 is that the confidence of each quasi-defective image I57s is less than the predetermined threshold and the judgment result of the object image I11 is that the confidence of any quasi-defective image I51s is greater than or equal to the predetermined threshold, the second detection module 121 can output a score value of 1 for the object image I17 and a score value of 0 for the object image I11, and then the second detection module 121 performs classification according to the score values of the object images I11 and I17. Here, the second detection module 121 can classify the object image I17 into the fuzzy group G3 because the score value of the object image I17 is 1, and classify the object image I11 into the second defective group G4 because the score value of the object image I11 is 0.

FIG. 11 is a schematic block diagram of an embodiment of the second artificial neural network in FIG. 1 . Please refer to FIG. 1 and FIG. 11 , in some embodiments, the second detection module 121 may include a classification unit 1211 and plural determination units 121A- 121X. The output of the classification unit 1211 may include a flawless class C1 and a plurality of flawed classes C21-C2Y. Here, the number of judging units 121A-121X is equal to the number of defect categories C21-C2Y, and each judging unit 121A-121X can be connected in series to an output of the classification unit 1211 corresponding to one of the plurality of defect categories C21-C2Y. In some implementation aspects, the plurality of defect categories C21-C2Y may include but not limited to deformation, bruise, sand hole, broken pillar, chipping, drawing mold, foreign matter, burr and so on.

FIG. 12 is a schematic flowchart of an embodiment of a method for screening the surface type of an object. Please refer to FIG. 1, FIG. 11 and FIG. 12. In an embodiment of step S40, the second detection module 121 can first use the classification unit 1211 to classify each quasi-defect image I51s, I57s of each object image I11, I17 (for example, each quasi-defect image I510-I1519 of the object image I11, as shown in FIG. 41) (may be referred to as the second stage of classification). Afterwards, the second detection module 121 uses the classifying unit 1211 to extract feature vectors for each of the flaw categories C21 - C2Y to obtain complex feature vectors of the multiple flaw categories C21 - C2Y (step S42 ).

In some implementations of step S41, after classifying the quasi-defect images I51s and 157s of the object images I11 and I17, the classifying unit 1211 may further generate score vectors for each category of the object images I11 and I17 (ie, the score vector of the flawless category C1 and the score vectors of the flaw categories C21-C2Y). Wherein, the score vector is used to represent the confidence level of this category. In addition, in some implementation aspects of step S42, the feature vector is a one-dimensional feature vector, and may include but not limited to defect attribute scores (for example, the score vector of step S41), defect size (for example, obtained from step S22), number of repetitions (for example, obtained from step S41), category, etc.

After obtaining the complex number of feature vectors, the second detection module 121 can use each of the judging units 121A-121X to judge whether each feature vector is smaller than the attribute threshold of the corresponding defect category C21-C2Y (step S43) (which can be called the third stage of classification). For example, the attribute threshold can be a confidence threshold, such as 0.5, and the judging unit 121A can judge whether the flaw attribute score in the feature vector of the flaw category C21 is smaller than the attribute threshold of the flaw category C21.

When all the judging units 121A-121X determine that the feature vectors of all the defect categories C21-C2Y of the object image I17 are smaller than the attribute thresholds corresponding to the defect categories C21-C2Y, the second detection module 121 may proceed to step S50 to classify the object image I17 into the fuzzy group G3. Conversely, when any of the judging units 121A-121X determine that the feature vector of the defect category C21-C2Y of the object image I11 is greater than or equal to the attribute threshold value corresponding to the defect category C21-C2Y (that is, the feature vector of any defect category C21-C2Y of the object image I11 is greater than or equal to the corresponding attribute threshold value), the second detection module 121 continues to execute step S60 to classify the object image I11 into the second defective product group G4 .

In some implementation aspects, since the input of each judging unit 121A-121X has compressed and extracted the feature dimension to a certain extent, each judging unit 121A-121X can be realized by using, for example but not limited to, a traditional machine learning algorithm. In addition, each judging unit 121A-121X can adopt a specific algorithm for the corresponding defect category C21-C2Y respectively. Furthermore, since each judging unit 121A-121X only needs to judge a single defect category, the feature space that needs to be learned by the algorithm of each judging unit 121A-121X can be reduced.

In some embodiments, the objects corresponding to the object images I12-I16, I18-I1N classified into the good product group G1 are good products, the objects corresponding to the object image I11 classified into the second defective product group G4 are defective products, and the objects corresponding to the object image I17 classified into the fuzzy group G3 need to be further manually identified to identify whether they are good products or defective products. In some implementations, the object image I17 classified into the fuzzy group G3 can be manually identified, and then the object image I17 can be used as training data for good or defective products to the detection system 100 according to the identification result, so as to improve the accuracy of the detection system 100 . Specifically, when the object image I17 is identified as a good product, in the learning phase, the object image I17 will be input into the detection system 100 as good product training data, so that the first artificial neural network 110 performs deep learning on the object image I17. When the object image I17 is identified as a defective product, in the learning phase, the object image I17 will be input into the detection system 100 as training data for defective products, so that the first artificial neural network 110 and the second artificial neural network 120 perform deep learning on the object image I17.

In some embodiments, the defect standard adopted by the first detection module 111 may be a more stringent judgment standard that is easy to achieve consistency, so as to reduce misclassifications caused by missed marks by calibration personnel or different calibration standards.

In some embodiments, in the learning phase, only the tiles of the defective parts (such as the quasi-defective images I510 - I519 ) in the object image I17 can be captured as the defective images (ie, training data) input to the second detection module 121 , so as to greatly reduce the influence of the image background on the training of the second detection module 121 . In addition, only one type of defect image can be input at a time to better control the proportion of various defects in training without the problem of uneven number of categories.

To sum up, in any embodiment of the artificial neural network-based object surface type screening method and detection system, it first uses the first detection module 111 to classify the object images I11-I1N according to the more stringent and easily consistent defect standards, so as to reduce the misclassification caused by the omission of the calibration personnel or different calibration standards. Afterwards, the second detection module 121 classifies at least one quasi-defective image I51s, I57s of the object images I11, I17 classified into the first defective product group G2, so as to rescue overkill objects. In this way, the calibration project of the object can be greatly simplified, the model defects caused by the calibration operation can be reduced, and the detection accuracy can be more stable. In addition, due to the multi-stage classification, the models in each stage (ie, the first detection module 111 and the second detection module 121) are responsible for different tasks, from strict to wide, and from wide to subtle, making the division of labor of each stage model more specific and easier to converge. Furthermore, problems such as calibration problems, complex samples, and uneven defect categories can be solved, so that testing equipment using this screening method can be brought online earlier. In addition, because of strict killing first and then rescue, the operators who use the testing equipment using this screening method do not have to worry about the outflow of defective products, which improves the reliability of the testing equipment.

Although the technical content of the present invention has been disclosed above with preferred embodiments, it is not intended to limit the present invention. Any modification and modification made by those skilled in the art without departing from the spirit of the present invention should be covered within the scope of the present invention. Therefore, the scope of protection of the present invention should be defined by the scope of the appended patent application.

100: Detection system 110: The first artificial neural network 111: The first detection module 112: Image processing module 120:Second artificial neural network 121: Second detection module 1211: Taxa 121A-121X: judging unit 122: Image processing module C1: Flawless category C21-C2Y: Defect categories G1: good product group G2: The first defective product group G3: fuzzy group G4: The second defective product group I11-I1N: Object Image I31, I37: Marking images I51s, I57s: quasi-defective images I510-I519: quasi-defective images I6: Defect heat map M1: Flaw Mark V1-VA: Perspective image S10-S60: Steps S11-S12: Steps S21-S25: steps S221-S225: Steps S251-S252: Steps S31-S32: Steps

FIG. 1 is a schematic block diagram of an embodiment of a detection system. FIG. 2 is a schematic flowchart of an embodiment of a method for screening the surface type of an object. FIG. 3 is a schematic flowchart of an embodiment of step S20. FIG. 4 is a schematic flowchart of an embodiment of step S22. FIG. 5 is a schematic flowchart of an embodiment of step S25. FIG. 6 is a schematic flowchart of an embodiment of step S30. FIG. 7 is a schematic diagram of an embodiment of an object image. FIG. 8 is a schematic diagram of an embodiment of a defect heat map. FIG. 9 is a schematic diagram of an embodiment of marking an image. FIG. 10 is a schematic diagram of an embodiment of a quasi-defective image. FIG. 11 is a schematic block diagram of an embodiment of the second artificial neural network in FIG. 1 . FIG. 12 is a schematic flowchart of an embodiment of a method for screening the surface type of an object.

100: Detection system

110: The first artificial neural network

111: The first detection module

112: Image processing module

120:Second artificial neural network

121: Second detection module

122: Image processing module

G1: good product group

G2: The first defective product group

G3: fuzzy group

G4: The second defective product group

I11-I1N: Object Image

I31,I37: mark images

I51s, I57s: quasi-defective images

V1-VA: Perspective image

Claims (19)

A method for screening object surface types based on an artificial neural network, comprising: using a first detection module to identify an object image to classify the object image into one of a good product group and a first defective product group, and output a tag image of the object image classified into the first defective product group, wherein the tag image is the object image with at least one defect mark; images, wherein each of the quasi-defective images is a partial image of the marked image; and using a second detection module to classify the object image into one of a fuzzy group and a second defective product group according to the at least one quasi-defective image, wherein the second detection module classifies the object image into the fuzzy group when the at least one quasi-defective image is detected to be flawless. The artificial neural network-based object surface type screening method described in Claim 1 further includes: receiving multiple perspective images of an object; and stitching the perspective images into the object image. The object surface type screening method based on artificial neural network as described in claim 1, wherein the first detection module is used to identify the object image to classify the object image into one of the good product group and the first defective product group, and the step of outputting the tagged image of the object image classified into the first defective product group includes: Using the first detection module to perform a defect prediction on the object image to generate a corresponding defect heat map; perform a defect screening on the defect heat map according to a defect standard; classify the object image into the first defective product group when it is determined in the defect screening that the defect standard is met, and generate the tagged image according to the defect heat map and the object images classified into the first defective product group; The method for screening object surface types based on artificial neural network as described in claim 3, wherein the defect heat map includes a plurality of pixels, each pixel has a defect score, the defect standard includes a confidence threshold, an area threshold and a quantity threshold, and wherein the step of performing the defect screening on the defect heat map according to the defect standard includes: selecting at least one defect pixel from the pixels according to the confidence threshold, wherein the defect pixel is the defect pixel whose defect score is greater than or equal to the confidence threshold pixels, and the at least one adjacent defect pixel forms a defect area; calculate the number of defects whose area is greater than the area threshold; determine whether the number of defects is greater than the number threshold; when the number of defects is greater than or equal to the number threshold, determine that the defect standard is met; and when the number of defects is less than the number threshold, determine that the defect standard is not met. The object surface type screening method based on artificial neural network as described in claim 3, wherein the step of generating the tagged image according to the blemish heat map and the object image classified into the first defective product group includes: superimposing the object image and the defect heat map into a composite image, wherein the composite image includes at least one candidate defect block; and computing the intersection of each of the candidate defect blocks in the composite image to obtain the marker image. The method for screening object surface types based on artificial neural network as described in claim 5, wherein the step of extracting the at least one quasi-defect image from the marked image corresponding to the at least one defect mark comprises: extracting a partial image corresponding to each of the defect marks from the marked image according to the at least one defect mark, wherein each of the partial images covers one of the at least one candidate defect block; and adjusting the size of each of the partial images to have a predetermined size of the quasi-defect image. The artificial neural network based object surface type screening method as described in claim 5, wherein each of the candidate defect blocks is a part of the block corresponding to at least one defect pixel adjacent to the defect heat map in the superimposed image in the object image in the superimposed image. The method for screening object surface types based on artificial neural network as described in claim 1, wherein the step of using the second detection module to classify the object image into one of the fuzzy group and the second defective product group according to the at least one quasi-defective image includes: using the second detection module to determine whether a confidence level of each quasi-defective image is less than a predetermined threshold value; when the confidence level of each quasi-defective image is less than the predetermined threshold value, using the second detection module to classify the object image into the fuzzy group; When the confidence level of the quasi-defective image is greater than or equal to the predetermined threshold, the second detection module is used to classify the object image into the second defective product group. The method for screening object surface types based on artificial neural network as described in Claim 1, wherein the step of using the second detection module to classify the object image into one of the fuzzy group and the second defective product group according to the at least one quasi-defective image includes: using a classification unit of the second detection module to classify the at least one quasi-defect image into one of a non-defect category or a plurality of defect categories; using the classification unit to extract plural feature vectors of the defect categories; Whether the feature vector of each of the defect categories is smaller than an attribute threshold corresponding to the defect category, wherein each of the judging units corresponds to one of the defect categories; when the judging units all determine that the feature vector of the corresponding defect category is smaller than the attribute threshold corresponding to the defect category, the object image is classified into the fuzzy group; A detection system for object surface type based on an artificial neural network, comprising: a first detection module, identifying an object image to classify the object image into one of a good product group and a first defective product group, and outputting a tagged image of the object image classified into the first defective product group, and extracting at least one quasi-defective image from the tagged image, wherein each of the quasi-defective images is a partial image of the tagged image; and a second detection module, according to the at least one quasi-defective image to classify the object image into One of a fuzzy group and a second defective product group, wherein the object image is classified into the fuzzy group when no defect is detected in the at least one quasi-defective image. The object surface type detection system based on artificial neural network as described in claim 10 further includes: a first image processing module, receiving multiple perspective images, splicing these perspective images into the object image, and outputting the object image. The object surface type detection system based on artificial neural network as described in claim 10, wherein the first detection module performs a defect prediction on the object image to generate a corresponding defect heat map, performs a defect screening on the defect heat map according to a defect standard, and classifies the object image into the first defective product group when it is determined in the defect screening that the defect standard is met, and generates the mark image according to the defect heat map and the object images classified into the first defective product group, and is judged in the defect screening. When the defect standard is not met, the object image is classified into the good product group. The object surface type detection system based on artificial neural network as described in claim 12, wherein the defect heat map includes a plurality of pixels, each pixel has a defect score, the defect standard includes a confidence threshold, an area threshold and a quantity threshold, the first detection module selects at least one defect pixel from the pixels according to the confidence threshold, wherein the defect pixel is the pixel whose defect score is greater than or equal to the confidence threshold, and the at least one adjacent defect pixel forms a defect area, the first detection module calculates the number of defects whose area of the defect is greater than the area threshold, and judges whether the number of defects is greater than the number threshold. When the number of defects is greater than or equal to the number threshold, it is determined that the defect standard is met; The object surface type detection system based on artificial neural network as described in claim 12, wherein the first detection module superimposes the object image and the defect heat map into a stack A combined image, wherein the combined image includes at least one candidate defect block, and the first detection module calculates the intersection of each of the candidate defect blocks in the combined image to obtain the marked image. The object surface type detection system based on artificial neural network as claimed in claim 14, wherein the marked image is the object image having at least one defect mark, wherein the detection system further includes: a second image processing module corresponding to the at least one defect mark to extract the at least one quasi-defect image from the mark image. The object surface type detection system based on artificial neural network as described in claim 15, wherein the second image processing module extracts a partial image corresponding to each of the defect marks from the marked image according to the at least one defect mark, wherein each of the partial images covers one of the at least one candidate defect block, and the second image processing module adjusts the size of each of the partial images into the quasi-defect image with a predetermined size. The object surface type detection system based on artificial neural network as described in claim 10, wherein the second detection module judges whether a degree of confidence of each of the quasi-defective images is less than a predetermined threshold value, and when the confidence degree of each of the quasi-defective images is less than the predetermined threshold value, the second detection module classifies the object image into the fuzzy group, and when the confidence degree of any of the quasi-defective images is greater than or equal to the predetermined threshold value, the second detection module classifies the object image into the second defective product group. The object surface type detection system based on artificial neural network as claimed in claim 10, wherein the second detection module further includes: a classification unit for classifying the at least one quasi-defective image into one of a flawless category or a plurality of flawed categories. The object surface type detection system based on artificial neural network as described in claim 18, wherein the classification unit extracts the plural feature vectors of the defect categories, and the second detection module further includes: a plurality of judgment units, respectively corresponding to the defect categories, respectively judge whether the feature vectors of the defect categories are smaller than an attribute threshold value corresponding to the defect category, and classify the object image into the fuzzy group when the judgment units all determine that the feature vector corresponding to the defect category is smaller than the attribute threshold value corresponding to the defect category , classifying the object image into the second defective product group when any of the judging units determines that the feature vector corresponding to the defect category is greater than or equal to the attribute threshold value corresponding to the defect category.