TW202141421A - Method and system of artificial intelligence automatic optical inspection - Google Patents

Abstract

A method and a system of an artificial intelligence automatic optic inspection are disclosed. The method includes the following steps: capturing a plurality of display panels through an optical imaging device and generating a plurality of training images; conducting an automatic encoding calculation to the plurality of training images and obtaining a panel standard image; capturing a test panel through an optical imaging device and generating a test image; conducting the automatic encoding calculation to the test image and obtaining a panel comparison image; comparing the difference value between the panel standard image and the panel comparison image. When the difference value exceeds the preset standard value, the test image is judged as an abnormal image. When the difference value does not exceed the preset standard value, the test image is judged as a normal image.

Description

Artificial intelligence automatic optical detection method and system

The present invention relates to an artificial intelligence automatic optical inspection method and system, and particularly relates to an artificial intelligence automatic optical inspection method and system executed by an algorithm of reference plane image comparison and defect classification and comparison.

In the manufacturing process of various panels, circuits, workpieces, etc. or when the product is completed, it can be inspected through Automated Optical Inspection (AOI) technology, and the digital image of the object to be measured is obtained through the camera, and then through the software Automatically calculate defects to achieve quality inspection, process verification and other effects. The automatic detection method of the device can replace the traditional manual visual detection method, which improves the detection efficiency and saves labor costs at the same time.

However, the technology of automated optical inspection usually uses the comparison between the captured image and the standard image. In actual operation, it is susceptible to various factors such as the testing environment, testing equipment, product category, product specification standards, etc., for example, the light source during testing. , Material area layer design, human operation error, etc., will affect the result of shooting images. Therefore, the judgment of the subsequent comparison will also be biased, and the abnormal situation cannot be correctly identified.

In view of this, although the current automatic optical inspection method can be used to automatically detect and identify the object to be tested, there is still the possibility of misjudgment in actual operation, and it is impossible to correctly identify the abnormality. Therefore, the inventor of the present invention considers and designs an artificial intelligence automatic optical inspection method and system to improve on the deficiencies of the existing technology, thereby enhancing the industrial application and utilization.

In view of the above-mentioned problems of the conventional technology, the purpose of the present invention is to provide an artificial intelligence automatic optical detection method and system, which can improve the detection accuracy and detection efficiency, and avoid the problem of misjudgment in the detection result.

According to the above objective, an embodiment of the present invention provides an artificial intelligence automatic optical inspection method, which includes the following steps: photographing a plurality of display panels through an optical imaging device, generating a plurality of training images, and storing the plurality of training images in a storage device; The processor accesses the storage device and performs automatic encoding operations of multiple training images to obtain the reference plane standard images of the multiple training images; the panel to be tested is captured by the optical imaging device to generate the image to be tested, and the image to be tested Stored in the storage device; through the processor accesses the storage device, the image to be tested is automatically encoded to generate the reference level comparison image of the test image; the difference between the reference level comparison image and the reference level standard image is compared by the processor If the difference value exceeds the preset standard value, the image to be tested is determined to be an abnormal image, and if the difference value does not reach the preset standard value, the image to be tested is determined to be a normal image.

In an embodiment of the present invention, the automatic encoding operation may include performing a complex-layer convolution network operation on a plurality of training images to generate a hidden layer image, and then pass the hidden layer image through a complex-layer deconvolution network operation to obtain a reference plane Standard image.

In the embodiment of the present invention, the number of layers of the complex-layer deconvolution network operation may be less than or equal to the number of layers of the complex-layer convolution network operation.

In the embodiment of the present invention, the artificial intelligence automatic optical inspection method may further include the following steps: the processor divides the normal image into a plurality of image detection areas to form a plurality of detection images; and the processor divides the plurality of detection images Perform image feature calculations to obtain image feature values of a plurality of detected images; the processor classifies the image feature values of the plurality of images to form an image classification model. The image classification model may include a plurality of surface defect classifications.

Another embodiment of the present invention provides an artificial intelligence automatic optical inspection system, which includes an optical imaging device, a storage device, and a processor. The optical imaging device shoots a plurality of display panels to generate a plurality of training images, and shoots the panel to be tested to generate the image to be tested. The storage device is connected to the optical imaging device to store a plurality of training images and images to be tested. The processor is connected to the storage device and executes a plurality of instructions to perform the following processing procedures: perform automatic coding operations of a plurality of training images to obtain reference plane standard images of a plurality of training images; perform automatic coding operations on the images to be tested to generate the images to be tested The reference plane comparison image; the judgment procedure is performed to compare the difference value between the reference plane comparison image and the reference plane standard image. If the difference value exceeds the preset standard value, the image to be tested is judged to be an abnormal image, if the difference value does not reach the preset value Standard value, judge the image to be tested as a normal image.

In the embodiment of the present invention, the processor may further execute the following processing procedures: perform a cutting procedure, divide the normal image into a plurality of image detection areas to form a plurality of detection images; perform image feature calculations on the plurality of detection images respectively to obtain The image feature values of a plurality of detected images; the image feature values of the plurality of images are classified to form an image classification model. The image classification model may include a plurality of surface defect classifications.

Continuing from the above, according to the artificial intelligence automatic optical inspection method and system disclosed in the embodiments of the present invention, the overall image of the panel can be automatically optically inspected, and the standard image of the reference plane can be obtained by the automatic coding algorithm, and then based on the standard image of the reference plane. Screening out abnormal images by comparison, avoiding the cost of inspection by continuing defect analysis of abnormal images. For normal images, it can further divide the detection area to perform feature value calculations to automatically determine whether there are defects in the detected images and the types of defects, so as to efficiently achieve the results of product inspection.

In order to understand the technical features, content and advantages of the present invention as well as the effects that can be achieved, the present invention is described in detail with the accompanying drawings and in the form of embodiment expressions as follows, and the figures used therein are only For the purpose of illustrating and supplementing the description, it may not be the true scale and precise configuration after the implementation of the present invention. Therefore, the scale and configuration relationship of the attached drawings should not be interpreted, and the scope of rights of the present invention in actual implementation should not be interpreted. Narrate.

In the drawings, the thickness or width of layers, films, panels, regions, light guides, etc. are exaggerated for clarity. Throughout the specification, the same reference numerals denote the same elements. It should be understood that when an element such as a layer, film, region, or substrate is referred to as being "on" or "connected to" another element, it can be directly on or connected to the other element, or Intermediate elements may also be present. Conversely, when an element is referred to as being "directly on" or "directly connected to" another element, there are no intervening elements. As used herein, "connection" can refer to a physical and/or electrical connection. Furthermore, "electrical connection" or "coupling" can mean that there are other elements between the two elements. In addition, it should be understood that although the terms “first”, “second”, and “third” may be used herein to describe various elements, components, regions, layers and/or parts, they are used to refer to an element, component , Region, layer and/or part are distinguished from another element, component, region, layer and/or part. Therefore, it is only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or its sequence relationship.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have meanings commonly understood by those skilled in the art to which the present invention belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meaning in the context of related technologies and the present invention, and will not be interpreted as idealized or excessive The formal meaning, unless explicitly defined as such in this article.

Please refer to Figure 1, which is a flowchart of an artificial intelligence automatic optical inspection method according to an embodiment of the present invention. As shown in the figure, the artificial intelligence automatic optical inspection method includes the following steps (S1~S5):

Step S1: Take a plurality of display panels with an optical image capturing device, generate a plurality of training images, and store the plurality of training images in a storage device. First of all, for display panels that need to be inspected, whether they are semi-finished products such as display units, substrates, and circuit boards in the process, or display devices that have completed the process steps, multiple display panels can be photographed with an optical imaging device in advance. Obtain the corresponding training image. The optical image capturing device described here includes various cameras, video cameras and other shooting devices, which are set on the production line of the production line or at the inspection position. After shooting the display panel, the image of the display panel is obtained as an artificial intelligence abnormal judgment. Training images, and store these training images in the storage device. The storage device may be the memory of the optical image capturing device or the memory of the computer that executes the calculation steps.

Step S2: Access the storage device by the processor, and perform an automatic encoding operation of a plurality of training images to obtain a reference plane standard image of the plurality of training images. After obtaining multiple training images, the processor in the computer device can execute the user's instructions to access these training images and perform automatic encoding operations on these images to obtain the reference plane standard images of these training images, which corresponds to The standard image of the reference plane of the display panel. The automatic encoding algorithm performed here involves performing a complex-layer convolutional network operation on the training image to generate a hidden layer image of the display panel, and then performing a complex-layer deconvolution network operation on the hidden layer image to generate a reference plane standard image.

The automatic encoding operation will be further described below. Please refer to Figure 2, which is a schematic diagram of the automatic encoding operation according to an embodiment of the present invention. As shown in the figure, the training image 21 is the surface image of the display panel captured by the optical image capturing device, including overall surface images such as display area and non-display area. In another embodiment, the training image 21 may also be a substrate surface image of a semi-finished display panel, including surface images such as light-emitting elements and circuit lines. The automatic encoding operation first performs the convolution network operation on the training image 21 through the set convolution kernel to obtain the characteristic data image of the training image. In this embodiment, the training image 21 is operated by a four-layer convolution network (W1~W4). ), the hidden layer image 22 is obtained. The number shown in the figure is the pixel size of the image, but the present disclosure is not limited to this. The training image can also have other layers, such as 5-layer convolutional network operations or more layers of convolutional network operations. To obtain the hidden layer image 22, the setting of the convolution kernel can also be determined according to the pixel size of the training image 21 to obtain the corresponding hidden layer image 22.

Next, the automatic encoding operation includes performing a deconvolution network operation on the hidden layer image 22. In this embodiment, the hidden layer image 22 passes through a four-layer deconvolution network operation (WT4~WT1) to obtain the original image 23. Use this as the reference plane standard image 25. Since the training image 21 shows the normal image selected first, the original image 23 should be obtained after the convolution network operation (W1~W4) and the deconvolution network operation (WT1~WT4) of the same number of layers. In addition, when the three-layer deconvolution network operation (WT4~WT2) with fewer layers than the convolution network operation (W1~W4) is performed, the feature layer image 24 can be obtained, and the feature layer image can also be used as a reference surface Standard image 25. In this embodiment, the reference level standard image 25 may include the original image 23 and the feature layer image 24 at the same time. In another embodiment, one of the original image 23 and the feature layer image 24 can be selected as the reference level standard image 25 . These reference plane standard images 25 and related parameters of the automatic encoding operation can be stored in the storage device 20.

Step S3: The panel to be tested is photographed by the optical image capturing device, an image to be tested is generated, and the image to be tested is stored in the storage device. According to the foregoing steps, the reference surface standard images 25 corresponding to various models and sizes are stored in the storage device 20. When there are panels to be tested of the same model and size to be tested, the surface image of the panel to be tested is captured by the optical orientation device. Generate images to be tested, which can be stored in a storage device.

Step S4: The processor accesses the storage device, and performs an automatic encoding operation on the image to be measured to generate a reference plane comparison image of the image to be measured. For the images to be tested, the processor in the computer device can execute user instructions, access these training images, and perform automatic encoding operations on these images. The automatic encoding operation here uses the same parameters as the parameters stored in the storage device, such as the size of the convolution kernel and the number of layers of the convolutional layer. After the image to be tested is automatically coded and calculated, the original image or feature map can be generated as the reference surface comparison image 25', and these reference surface standard images 25' are stored in the storage device 20.

Step S5: The processor compares the difference value between the reference plane comparison image and the reference plane standard image to determine whether the difference value exceeds a preset standard value. If the difference value exceeds the preset standard value, the image to be tested is judged to be an abnormal image NG; if the difference value does not reach the preset standard value, the image to be tested is judged to be a normal image G. When the storage device 20 stores the reference surface standard image 25 and the reference surface comparison image 25', the processor can execute the comparison procedure to compare the difference between the reference surface standard image 25 and the reference surface comparison image 25', and determine Whether the image to be tested is a normal image G. In this embodiment, the calculation of the difference value can be based on the type of the reference plane standard image 25. When the reference plane standard image 25 is the original image 23, the reference plane standard image 25 and the reference plane comparison image 25' can be used. Among them, the grayscale value of each pixel is subtracted to obtain the difference value. In another embodiment, when the reference level standard image 25 is the feature layer image 24, the difference value can be calculated based on the vector inner product of each pixel of the reference level standard image 25 and the reference level comparison image 25'. Further compare whether the difference value exceeds the preset standard value. If it is, it is determined that the image to be tested is abnormal image NG, and the manufacturing or inspection process needs to be re-examined for defects; if not, it is determined that the image to be tested is a normal image G, which can pass Inspection or further detailed defect classification inspection.

The above-mentioned artificial intelligence automatic optical inspection method mainly detects the overall image of the display panel, that is, before performing the detailed detection of each component position, the overall image is automatically detected to avoid the need for subsequent detection of abnormal images. The cost of testing. For example, when the detection machine is set to deviate or the lighting equipment is changed, the image captured by the optical image capturing device may not be the object to be detected, or the image brightness may be significantly different and the abnormality cannot be correctly identified. Therefore, through the artificial intelligence automatic optical detection method, abnormal images can be automatically picked out, the object to be tested can be re-detected or the detection process or device can be checked for abnormalities.

Please refer to FIG. 3, which is a flowchart of an artificial intelligence automatic optical inspection method according to another embodiment of the present invention. As shown in the figure, the artificial intelligence automatic optical inspection method includes the following steps (S1~S8):

Step S1 to step S5: The steps S1 to S5 here are the same as the processes described in the foregoing embodiment, and therefore, the same technical features will not be described repeatedly. The panel to be tested can determine whether the image captured by the optical imaging device is a normal image G through steps S1 to S5, and if so, continue the subsequent inspection process.

Step S6: The processor divides the normal image into a plurality of image detection areas to form a plurality of detection images. For the normal image G, it can be cut into multiple image detection areas according to the pixel size of the image, that is, the complete panel image is cut into multiple smaller inspection images, and the complete panel is inspected by the judgment of each inspection image Whether the image is defective.

Step S7: Perform image feature calculations on the plurality of detected images by the processor to obtain the image feature values of the plurality of detected images. For each detected image, the processor executes an image feature calculation program, respectively performs image feature calculation on each detected image, and obtains the image feature value of the detected image through the image feature calculation.

Step S8: The processor classifies the image feature values of the plurality of images to form an image classification model. The image classification model can include normal image classification and defect image classification. Normal image classification means that the production of panel circuits and components meets specifications. Defect image classification includes multiple surface defect classifications, such as ripples, missing corners, and cracks. When the feature value is When belonging to a specific defect image classification, it is judged that the detected image is defective, and an abnormal notification is generated.

In this embodiment, dividing the normal image into a plurality of image detection regions can not only subdivide each region to perform defect detection separately, but also avoid the situation that the defect ratio on the overall image is too low and cannot be revealed in the feature value. In addition, for different detection areas, the detection feature calculation method of the area can also be designed according to the product category, and the detection is performed according to the component type or the surface pattern type to further improve the accuracy of the detection.

Among the above-mentioned image feature calculation methods, the image feature calculation may include a plurality of convolutional layer calculations, pooling layer calculations, and activation layer calculations. Please refer to FIG. 4, which is a schematic diagram of image classification calculations according to an embodiment of the present invention. As shown in the figure, each detection image 31 can be converted into feature images of different levels through multi-layer convolutional layer operations, and then combined with pooling layer operations and activation layer operations to output the hidden features of the images to obtain feature images 32. The feature image 32 can calculate the corresponding image feature value, and then use the classifier to classify each feature image 32 according to different levels. For example, when the image feature value is less than the preset standard value, it is detected as normal and no defect, if it exceeds the preset standard value, it is defect A, and if it reaches a higher standard value, it is defect B. For different defects, the operator can rework the panel to be tested according to the process. If it exceeds the inspection standard and cannot be reworked, it may be scrapped.

In the entire process, because the overall image has been automatically detected, images with abnormal states can be initially screened, avoiding the waste of computing resources by performing related computing procedures on invalid images when performing defect classification and judging. For the images that meet the standards, the detection area is further used to identify whether it has defect characteristics, and the object to be tested is efficiently tested to eliminate various defective products and improve the production efficiency and quality of the process.

Please refer to FIG. 5, which is a schematic diagram of an artificial intelligence automatic optical inspection system according to an embodiment of the present invention. As shown in the figure, the artificial intelligence automatic optical inspection system 1 includes an optical image capturing device 41, a storage device 42 and a processor 43. In this embodiment, the optical image capturing device 41 can be a camera or a video camera. During the training phase, the panel image can be captured to generate training images, and during the inspection phase, the panel to be tested 40 can be captured to obtain the image to be tested. The above-mentioned image data can be uploaded to the memory storage in the storage device 42 through wireless network transmission, wireless communication transmission or general wired Internet. The memory can include read-only memory, flash memory, floppy disk or cloud data Library etc.

Then, the processor 43 of the artificial intelligence automatic optical inspection system 1 can be connected to the storage device 42 to access data in the storage device 42. In this embodiment, the processor 43 can include a central processing unit in a computer or a server, Image processors, microprocessors, etc., which may include multi-core processing units or a combination of multiple processing units. The processor 43 executes a plurality of instructions to execute the artificial intelligence automatic optical inspection program 44. In detail, the artificial intelligence automatic optical inspection program 44 includes automatic coding operation of training images, obtaining the reference plane standard image of the training image; performing the image to be tested Carry out automatic encoding operation to generate the reference plane comparison image of the image to be tested; perform the judgment procedure to compare the difference value between the reference plane comparison image and the reference plane standard image. If the difference value exceeds the preset standard value, the test image is judged to be abnormal For images, if the difference value does not reach the preset standard value, the image to be tested is judged to be a normal image. For the above detection procedure, please refer to the description of the foregoing embodiment, and the same technical features will not be described repeatedly.

In the process of automatic encoding operation, the processor 43 may perform a plurality of convolutional network operations to generate a hidden layer image, and then obtain a reference plane standard image through a plurality of deconvolutional network operations. When the number of layers of the convolutional network is the same as the number of layers of the deconvolutional network, the base-level standard image can be the original image of the training image. When the number of layers of the convolutional network is greater than that of the deconvolutional network When calculating the number of layers, the base-level standard image can be the feature layer image.

In another embodiment, after the aforementioned artificial intelligence automatic optical detection program 44 determines that the image to be tested is a normal image or an abnormal image, the processor 43 may further exclude the abnormal image, and perform the artificial intelligence automatic classification process on the normal image. 45. In detail, the artificial intelligence automatic classification program 45 includes a cutting process, which divides the normal image into a plurality of image detection areas to form a plurality of detection images; performs image feature calculations on each detection image to obtain the image feature value of the plurality of detection images ; Classify each image feature value to form an image classification model. The image classification includes normal surface images and defective surface images. The images to be tested by the artificial intelligence automatic optical inspection program 44 can be inspected for defects through automatic classification methods, and the types of defects can be determined by different classification models.

The above description is only illustrative, and not restrictive. Any equivalent modifications or alterations that do not depart from the spirit and scope of the present invention shall be included in the scope of the appended patent application.

1: Artificial intelligence automatic optical inspection system 20: storage device 21: Training images 22: Hidden layer image 23: Original image 24: Feature layer image 25: Datum standard image 25’: Datum comparison image 31: Detect image 32: Feature image 40: Panel to be tested 41: Optical imaging device 42: storage device 43: processor 44: Artificial intelligence automatic optical inspection program 45: Artificial intelligence automatic classification program A, B: Defects G: Normal image NG: abnormal image S1~S5, S6~S8: steps W1~W4: Convolutional network operation WT1~WT4: Deconvolution network operation

In order to make the technical features, content and advantages of the present invention and the effects that can be achieved more obvious, the present invention is described in detail in the form of embodiments with the accompanying drawings as follows: Figure 1 is a flowchart of an artificial intelligence automatic optical inspection method according to an embodiment of the present invention. Figure 2 is a schematic diagram of an automatic encoding operation according to an embodiment of the present invention. FIG. 3 is a flowchart of an artificial intelligence automatic optical inspection method according to another embodiment of the present invention. Figure 4 is a schematic diagram of an image classification operation according to an embodiment of the present invention. Figure 5 is a schematic diagram of an artificial intelligence automatic optical inspection system according to an embodiment of the present invention.

S1~S5: steps

Claims (10)

An artificial intelligence automatic optical detection method, which includes the following steps: Take a plurality of display panels through an optical imaging device to generate a plurality of training images, and store the plurality of training images in a storage device; A processor accesses the storage device and performs an automatic encoding operation of the plurality of training images to obtain a reference plane standard image of the plurality of training images; Shooting a panel to be tested by the optical imaging device, generating an image to be tested, and storing the image to be tested in the storage device; Accessing the storage device by the processor, performing the automatic encoding operation on the image to be measured, and generating a reference plane comparison image of the image to be measured; The processor compares a difference value between the reference level comparison image and the reference level standard image. If the difference value exceeds a preset standard value, it is determined that the image to be tested is an abnormal image. If the difference value does not reach The preset standard value determines that the image to be tested is a normal image. According to the artificial intelligence automatic optical inspection method described in item 1 of the scope of patent application, the automatic encoding operation includes performing complex layer convolutional network operations on the plurality of training images to generate a hidden layer image, and then the hidden layer image Obtain the standard image of the reference plane through a complex layer deconvolution network operation. The artificial intelligence automatic optical inspection method described in item 2 of the scope of patent application, wherein the number of layers of the complex-layer deconvolution network operation is less than or equal to the number of layers of the complex-layer convolution network operation. The artificial intelligence automatic optical inspection method described in item 1 of the scope of patent application further includes the following steps: Dividing the normal image into a plurality of image detection areas by the processor to form a plurality of detection images; Performing an image feature operation on the plurality of detection images by the processor to obtain an image feature value of the plurality of detection images; The image feature values of the plurality of images are classified by the processor to form an image classification model. The artificial intelligence automatic optical inspection method described in item 4 of the scope of patent application, wherein the image classification model includes a plurality of surface defect classifications. An artificial intelligence automatic optical inspection system, which includes: An optical image capturing device that shoots a plurality of display panels to generate a plurality of training images, and shoots a panel to be tested to generate an image to be tested; A storage device connected to the optical image capturing device to store the plurality of training images and the image to be tested; and A processor, connected to the storage device, executes a plurality of instructions to perform the following processing procedures: Performing an automatic encoding operation of one of the plurality of training images to obtain one of the reference plane standard images of the plurality of training images; Performing the automatic encoding operation on the image to be measured to generate a reference plane comparison image of the image to be measured; A judgment process is performed to compare the difference between the reference level comparison image and the reference level standard image. If the difference value exceeds a preset standard value, the image to be tested is judged to be an abnormal image, and if the difference value does not reach The preset standard value determines that the image to be tested is a normal image. The artificial intelligence automatic optical inspection system described in item 1 of the scope of patent application, wherein the automatic encoding operation includes performing complex layer convolutional network operations on the plurality of training images to generate a hidden layer image, and then storing the image layer The image is calculated by a multiple-layer deconvolution network to obtain the standard image of the reference plane. Such as the artificial intelligence automatic optical inspection system described in item 7 of the scope of patent application, wherein the number of layers of the complex-layer deconvolution network operation is less than or equal to the number of layers of the complex-layer convolution network operation. Such as the artificial intelligence automatic optical inspection system described in item 1 of the scope of patent application, the processor further implements the following processing procedures: Perform a cutting procedure to divide the normal image into a plurality of image detection areas to form a plurality of detection images; Performing an image feature operation on the plurality of detected images respectively to obtain an image feature value of the plurality of detected images; The image feature values of the plurality of images are classified to form an image classification model. For the artificial intelligence automatic optical inspection system described in item 9 of the scope of patent application, the image classification model includes a plurality of surface defect classifications.

Images