TW202035975A - Surface defect detection system and method thereof - Google Patents

Abstract

A surface defect detection method applied to a suface of an object is disclosed. The method includes obtaining an image of the surface, performing a deep learning algorithm by a computing device to set a plurality of bounding boxes in the image and to output a plurality of feature parameters associated with the plurality of bounding boxes, with each bounding box enclosing a possible defect of the surface, and performing a classifying algorithm by the computing device according to the bounding boxes and the feature parameters to determine whether the surface conforms to a specification.

Description

Surface defect detection system and method

The present invention relates to defect detection, in particular to a defect detection system and method for the surface of an object.

Before computers such as laptops or tablets are shipped, quality control personnel must check for surface defects. The quality control personnel checks whether there are scratches, dents or other surface defects defined in the specification documents. If the type and severity of the surface defect exceeds the range allowed in the specification, the computer is considered unqualified; on the contrary, if the type and severity of the surface defect falls within the range allowed in the specification, the computer "passes "Surface defect detection.

Traditionally, surface defect detection is performed by human inspectors. They read and follow the inspection specification documents to determine whether the inspected computer passes. This kind of surface defect inspection not only requires a lot of manpower, but there are three disadvantages of using manual inspectors.

The first is "inaccuracy." The human eye cannot make accurate measurements, especially in a very small range. Even when comparing two similar objects, the human eye may not find that one object is slightly smaller than the other. This is also reflected in the surface roughness, size and any other factors that need to be measured. Although the form of the inspection specification file is usually the size threshold, human vision still cannot be used as an accurate tool to complete the task.

The second is "unreliable". Judging certain surface defects defined in inspection specifications requires very fine vision. Detection can be very complicated, such as extremely small size; or very tricky, such as easily confused with computer surface texture. In addition, it is well known that human eyes can be deceived by illusions. Therefore, human visual inspection is not always reliable.

The third is "inconsistency". Humans are prone to fatigue or inattention. For example, when a manual inspector's shift is about to end, he or she may feel tired or out of focus, so the surface defect is not detected, and the defective computer is released as a normal product. The quality of computers coming out of the checkpoint changes over time. Different manual inspectors also have different judgment standards, which leads to changes in the quality of products coming out of the inspection station.

On the other hand, the traditional defect detection methods used in SMT production lines or printed circuit boards, such as template matching or computer vision, are technologies. However, the above technologies are prone to geometric registration errors, and it is difficult to update the configuration when the number of defect types increases, and when the defects cannot be accurately described in geometric language or rules, the above technology cannot respond. Although, manual inspectors can still find defects in misaligned samples, or defects that are not easily described by rules, such as surface contamination caused by fingerprints or dirt, and defects caused by wear or scratches. However, manual inspectors also have the disadvantages mentioned above. On the whole, neither manual inspectors, template matching, or computer vision technology can fully meet all requirements in the surface defect detection process.

In view of this, the present invention proposes a machine learning (ML)-based visual recognition system for computer surface defect detection. The invention includes systems and methods to achieve the goal of automatic surface defect detection and classification.

According to an embodiment of the present invention, a surface defect detection method is applicable to a surface of an object. The method includes: obtaining an image of the surface with a camera; and executing a deep learning algorithm with a computing device The method is to select a certain bounding box from the image and output a characteristic parameter related to the bounding box, wherein the bounding box contains a defect on the surface; and using the computing device according to the bounding box and the characteristic parameter A classification algorithm is executed to determine whether the surface meets a specification.

According to an embodiment of the present invention, a surface defect detection system is applicable to a surface of an object. The system includes: a camera device for obtaining an image of the surface; a computing device electrically connected The camera device and the computing device are used to execute a deep learning algorithm to select a bounding box from the image and output a characteristic parameter associated with the bounding box, wherein the bounding box contains a defect on the surface; The computing device is further used to execute a classification determination algorithm based on the bounding box and the characteristic parameter to determine whether the surface meets a specification; and a back-end processing device electrically connected to the computing device, the back-end processing device It is used to perform actions related to the surface according to the judgment result of the computing device.

With the above architecture, the deep learning-based system proposed by the present invention imitates the human visual system. As long as sample training images are provided, it can detect fuzzy defects that are difficult to describe by rules. In addition, surface defect detection based on machine learning is more capable of detecting a variety of increasing defects without sacrificing detection performance and speed.

The surface defect detection system and method based on deep learning proposed by the present invention can find subtle defects better than traditional computer vision methods. In addition, the present invention avoids the shortcomings of inaccuracy, unreliability and inconsistency of manual inspectors. In the actual production line, manual inspectors can tolerate a certain degree of defects according to the inspection specifications. When the present invention executes the allocation judgment algorithm, it can also learn from the data on the production line based on the machine learning mechanism to achieve the effect of customized judgment.

The defect detection and classification performed by the present invention can be directly performed on the production line without sending the image data to the cloud server for defect detection. Therefore, the delay in transmission is not counted. And since all detection and classification operations are performed locally, the cost of transmitting large image information and the security of image data are improved.

In the surface defect detection system and method disclosed in the present invention, the defects on the surface of the object are circled through the deep learning algorithm, and then the classification judgment algorithm determines whether the surface of the object can pass the surface defect test according to the specification file. The invention can detect various styles of surface defects to avoid the shortcomings of inaccuracy, unreliability and inconsistency caused by manual detection, and maintain a fast detection speed.

The above description of the content of the disclosure and the description of the following embodiments are used to demonstrate and explain the spirit and principle of the present invention, and to provide a further explanation of the patent application scope of the present invention.

The detailed features and advantages of the present invention are described in detail in the following embodiments, and the content is sufficient to enable anyone familiar with the relevant art to understand the technical content of the present invention and implement it accordingly, and according to the content disclosed in this specification, the scope of patent application and the drawings Anyone who is familiar with the relevant art can easily understand the related purpose and advantages of the present invention. The following examples further illustrate the viewpoints of the present invention in detail, but do not limit the scope of the present invention by any viewpoint.

The present invention is suitable for detecting defects on a surface LS of the object L. The surface LS is basically similar to a plane, but the plane may have a height difference within a specified range. In practice, the present invention can be used to detect the top cover (Top Cover) of the notebook computer, the palm rest (Palm rest, the inner surface of the notebook computer except the keyboard and the touch pad) or the touch panel of the tablet computer.

Please refer to FIG. 1, which illustrates a schematic structural diagram of a surface defect detection system 100 according to an embodiment of the present invention. The surface defect detection system 100 includes a camera device 10, a computing device 30, and a back-end processing device 50. The computing device 30 is electrically connected to the camera device 10 and the back-end processing device 50, as shown in FIG.

The imaging device 10 is used to obtain an image of the surface LS of the object L. In practice, a light emitting device can be added to form a uniform light field around the object L, so that the camera device 10 can obtain a clear image.

The computing device 30 determines whether the surface LS meets a specification according to the image of the surface LS. In other words, the computing device 30 determines whether the number of defects or the severity of defects on the surface LS is within the tolerance range defined by the specifications.

The back-end processing device 50 is used for performing actions related to the surface LS according to the determination result of the computing device 30. As shown in FIG. 1, the back-end processing device 50 is, for example, a screen, which can display the judgment result of the computing device 30 for inspection by an operator on the production line.

In detail, the judgment of the computing device 30 is divided into two stages: the first stage is defect detection based on deep learning, and the second stage is classification judgment based on machine learning. The output data of the first stage will be used as the input data of the second stage.

In the first stage, the computing device 30 executes a deep learning algorithm to select a plurality of bounding boxes from the image and output a plurality of characteristic parameters associated with these bounding boxes, and each bounding box contains a surface LS One defect. The type of the defect may include a scratch, abrasion, a dent, a stain, etc. on the surface LS.

The deep learning algorithm, for example, adopts a defect detection model obtained by pre-training a Region-based Convolutional Neural Network (R-CNN). The region-based convolutional neural network is for example: Fast R-CNN, Faster R-CNN, Mask R-CNN, YOLO (You Only Look Once) or SSD (Single Shot Detection). However, it should be noted that the above only lists examples that can be used for deep learning algorithms, and is not intended to limit the deep learning algorithms that can be used in the present invention. In an embodiment of the present invention, Faster R-CNN is used to train the defect detection model, which has a good performance in recognition speed and accuracy.

In an embodiment of the present invention, the bounding box is rectangular. However, if a deep learning algorithm that can circle the shape of the defect is used, the bounding box can also be irregular. The characteristic parameters include the area of the rectangular bounding box, the length of the diagonal, the confidence index and the defect type. The confidence index, for example, presents the degree of compliance with the defects in the bounding box in a percentage.

In the second stage, the computing device 30 executes a classification decision algorithm according to the bounding box and feature parameters output in the first stage to determine whether the surface LS meets a specification.

In one embodiment, the computing device 30 uses the number of bounding boxes and an area ratio as input data. The number of bounding boxes is the number of bounding boxes selected in the first stage. The area ratio is the ratio of the total area of the bounding box selected in the first stage to the area of the surface LS. However, the present invention is not limited to only use the above two dimensions of data to execute the classification decision algorithm.

The classification decision algorithm, for example, uses one of Decision Tree (Decision Tree), Support Vector Machine (SVM), and K-Nearest Neighbor (KNN) to be obtained through prior machine training The binary classification decision model of, which is used to determine whether the surface LS passes or fails. In an embodiment of the present invention, SVM is used to train the classification decision model.

On the production line, the computing device 30 uses, for example, the NVIDIA Jetson TX2 package, which can load the trained Faster R-CNN and SVM models from the cloud server. In the actual inspection, there is no need to connect to the server again, but to determine the defects such as scratches on the surface LS in real time on the production line, thus increasing the speed of defect determination and saving additional transmission of images to the cloud server Time and cost of the device. However, it should be noted that the computing device 30 of the present invention can also be the cloud server itself. The invention does not particularly limit the type of hardware of the computing device 30.

Please refer to FIG. 2, which illustrates the flow of a surface defect detection method according to an embodiment of the present invention. Please refer to step S1: the camera device 10 obtains an image of the surface LS of the object L.

Please refer to step S2: the computing device 30 executes a deep learning algorithm to select a certain bounding box from the image and output a characteristic parameter associated with the bounding box, where the bounding box contains a defect of the surface LS.

Please refer to FIG. 3, which is a schematic diagram of the deep learning algorithm marking the bounding boxes B1 to B4 and the exception area B5 on the image of the surface LS. The bounding frame B1 surrounds a scratch. The bounding box B2 surrounds a wear or a stain. It should be noted that the deep learning algorithm marks each defect separately, regardless of whether the marked bounding boxes overlap. For example, the overlapping bounding boxes B3 and B4 in Fig. 3 belong to two defects respectively.

Please refer to step S3: the computing device 30 executes a classification determination algorithm according to the bounding box and characteristic parameters to determine whether the surface LS meets a specification.

Please refer to step S4: the back-end processing device executes actions related to the surface LS according to the determination result of the computing device 30. For example, the judgment result is displayed on the screen, or the passing and non-passing surfaces LS are distinguished by the conveying device.

In another embodiment of the present invention, before acquiring the image in step S1, the method further includes: forming a uniform light field around the object L with a light emitting device. In this way, the influence of ambient light on the image obtained by the imaging device 10 on the surface LS can be reduced.

In another embodiment of the present invention, before the deep learning algorithm is executed in step S2, the method further includes: setting an exception area for the image by the computing device 30 (B5 of FIG. 3); and the selected bounding box and the exception area are not overlapping. The exception area B5 is, for example, the name of the brand on the notebook computer. The deep learning algorithm does not perform any defect detection for the exception area B5.

In another embodiment of the present invention, before executing the deep learning algorithm in step S2, the method further includes: obtaining a plurality of training images and corresponding training parameters by a server.

The sources of the training images include images of the surface LS with defects and images of the surface LS without defects. In the initial training, at least 500 defective training images are required, and at least 100 non-defective training images. This allows the deep learning algorithm to learn how to identify defects and their various types. It should be noted that the above-mentioned number of images is only an example, not a limitation of the present invention.

The training parameters include a sample bounding box labeled with a training image with a defect and a corresponding defect type label. The sample bounding box is a rectangle that contains the defect as a whole.

After obtaining the training images and corresponding training parameters as described above, the server can execute a deep learning algorithm based on the training data to generate a defect detection model.

In addition, the training images each have a label of the first image or the second image. The label of the first image represents that the image meets a specification; the label of the second image represents that the image does not meet the specification. For example, 100 images that originally did not have defects are generally marked as the first image. Among 500 images with defects, 150 images may be marked because of the small number of defects or the small sum of defect areas. Marked as the first image, the remaining 350 images are marked as the second image. Therefore, among all 600 training images, 250 images are labeled as the first image, and 350 images are labeled as the second image. The server executes a classification judgment algorithm based on these training images and their respective annotations to generate a classification judgment model.

The computing device 30 loads the trained defect detection model and the classification judgment model from the server, and executes the two models in step S2 and step S3 to perform surface defect detection on the surface LS.

In practice, the defect detection model can use the data used in actual detection as training data to update the model, thereby improving the accuracy of defect detection.

In addition, during the training process, the training image can be divided into multiple sub-images in advance to reduce the data throughput processed by the server during training.

In summary, the surface defect detection system and method disclosed in the present invention can accurately mark various types of defects on the surface of an object through a deep learning algorithm, and the classification judgment algorithm determines the defect according to the specification file. Whether the surface can pass the surface defect test. The invention can detect various styles of surface defects to avoid the shortcomings of inaccuracy, unreliability and inconsistency caused by manual detection, and maintain a fast detection speed.

Although the present invention is disclosed in the foregoing embodiments, it is not intended to limit the present invention. All changes and modifications made without departing from the spirit and scope of the present invention fall within the scope of patent protection of the present invention. For the scope of protection defined by the present invention, please refer to the attached patent scope.

100: Surface defect detection system 10: Camera 30: Computing device 50: back-end processing device L: Object LS: Object surface S1~S4: steps B1~B4: bounding box B5: Exceptional area

FIG. 1 is a structural diagram of a surface defect detection system according to an embodiment of the invention. 2 is a flowchart of a surface defect detection method according to an embodiment of the invention. Figure 3 shows the bounding box and exception area marked on the surface of the object.

S1~S4: steps

Claims (11)

A method for detecting surface defects is suitable for a surface of an object. The method includes: obtaining an image of the surface with a camera device; and executing a deep learning algorithm with a computing device to select a certain boundary from the image The frame and output are related to a characteristic parameter of the bounding box, where the bounding box contains a defect on the surface; and the computing device executes a classification decision algorithm according to the bounding box and the characteristic parameter to determine Whether the surface meets a specification. The surface defect detection method according to claim 1, wherein the deep learning algorithm is a region-based convolutional neural network (R-CNN). The surface defect detection method according to claim 2, wherein the convolutional neural network based on the region is Fast R-CNN, Faster R-CNN, Mask R-CNN or YOLO. The surface defect detection method according to claim 1, wherein the bounding frame is a rectangle, and the characteristic parameter includes an area of the rectangle, a diagonal length of the rectangle, a confidence index and a defect type, The defect types include a scratch, a wear, a dent and a stain on the surface. The surface defect detection method according to claim 1, wherein the classification judgment algorithm is one of Decision Tree, SVM, and KNN. The method for detecting surface defects according to claim 1, and before the computing device executes the classification and determination algorithm, further includes: selecting another bounding box from the image by the computing device and outputting the at least certain bounding box. At least one other characteristic parameter of the bounding box, wherein the at least certain bounding box contains at least one other defect of the surface; calculating the number of one of the bounding boxes selected by the deep learning algorithm; calculating by the deep learning algorithm The ratio of the total area of the bounding boxes selected by the method to the area of the surface; and the computing device executes the classification determination algorithm according to the number and the ratio to determine whether the surface meets the specification. The surface defect detection method according to claim 1, wherein before executing the deep learning algorithm, it further comprises: setting an exception area for the image by the computing device; and the selected bounding box and the exception area No overlap. The surface defect detection method according to claim 1, wherein before obtaining the image, it further comprises: forming a uniform light field around the object with a light emitting device. The surface defect detection method according to claim 1, before executing the deep learning algorithm, it further includes: obtaining a plurality of training images by a server, wherein the training images include images of the surface with the defect and images An image of the surface with the defect; each of the training images with the defect has a sample bounding box and a defect type. The sample bounding box includes a rectangle of the defect as a whole, and the defect type is associated with The defect; each of the training images further includes a first image annotation or a second image annotation, where the first image annotation represents that the training image meets the specification, and the annotation of the second image represents that the training image does not meet the Specifications; the server executes the deep learning algorithm according to the training images, the sample bounding boxes and the defect types to generate a defect detection model; the server according to the training images, the first The image annotation and the second image annotation execute the classification judgment algorithm to generate a classification judgment model; and load the defect detection model and the classification judgment model from the server by the computing device. A surface defect detection system is suitable for a surface of an object. The system includes: a camera device for obtaining an image of the surface; a computing device electrically connected to the camera device, the computing device for A deep learning algorithm is executed to select a certain bounding box from the image and output a characteristic parameter associated with the bounding box, where the bounding box contains a defect on the surface; the computing device is further used to determine The bounding box and the characteristic parameter execute a classification judgment algorithm to determine whether the surface meets a specification; and a back-end processing device electrically connected to the computing device, and the back-end processing device is used to determine the result of the computing device Perform actions associated with the surface. The surface defect detection system according to claim 10, wherein the deep learning algorithm is a region-based convolutional neural network; the classification judgment algorithm is one of Decision Tree, SVM, and KNN; and the The computing device is further used to select another at least certain bounding box from the image and output another at least one characteristic parameter associated with the at least certain bounding box, and execute the classification determination algorithm according to a quantity and a ratio; wherein the at least certain bounding box The frame contains at least one other defect of the surface; the quantity is related to the bounding boxes selected by the deep learning algorithm; and the ratio is related to the bounding boxes selected by the deep learning algorithm The sum of the area and the area of the surface.

Images