TW202223740A - Automatic inspection system and operation method thereof including a machine vision inspection device, a repair station, an artificial intelligence module and a processing platform - Google Patents

Abstract

An automatic inspection system includes a machine vision inspection device, a repair station connected to the machine vision inspection device, an artificial intelligence module, and a processing platform connected to the repair station and the artificial intelligence module. The machine vision inspection device can obtain an image of a to-be-inspected object and displays the image through an operation screen of the repair station. The artificial intelligence module retrieves the image of the to-be-inspected object for inference of classification. The processing platform automatically superimposes resultant data of the inference of classification of the image of the to-be-inspected object on the operation screen of the repair station, and intercepts manual review resultant data inputted into the repair station by an operator through a user interface for transmission to the artificial intelligence module as the data for subsequent retraining and learning of the automatic classification model.

Description

Automatic detection system and operation method thereof

The present invention provides a system and method, and more particularly, relates to an automatic detection system and a method of operation thereof.

Press, with the advancement of science and technology, the size of electronic parts is reduced and the precision is increasingly required. Therefore, in order to ensure the improvement of product yield, in the modern process, it is necessary to carry out inspections at various important sites to facilitate early detection. Defective products with defects or defects can reduce the waste of process resources, improve the output and yield of the production line, but also increase the time consumed by the process with multi-station inspection. Therefore, in order to speed up the inspection time, automatic vision Automated inspection technologies such as inspection and automatic optical inspection have become an indispensable role in the process.

The general automatic inspection equipment is generally used in the appearance inspection of circuit board assembly production lines, and replaces the previous manual visual inspection operations. For example, it can be used to detect the soldering of parts on the circuit board and the quality status after assembly. The technology is used to compare whether there is too much difference between the object to be tested and the standard image to judge whether the object to be tested meets the standard. However, the traditional automatic detection equipment often causes a high false alarm rate due to factors such as color difference and height drop, which increases the manpower after the event. Waste of re-inspection and processing costs for abnormal detection.

However, with the rise of modern Industry 4.0 and smart manufacturing, and in smart factories that pay attention to production efficiency and quality, quality inspection is the most important thing to ensure production line yield and product quality. One of the necessary procedures, in order to improve the accuracy of inspection operations, in the past, automated inspection equipment was often used to assist on-site production line personnel. Inspection equipment is integrated, and through machine learning or deep learning algorithm, the defect detection of automatic inspection equipment is more accurate, so as to reduce the false alarm rate of automatic inspection equipment and save a lot of labor costs, but the machine learning or deep learning adopted by the manufacturer All need to go through the training mode first, and then download the training results to the inference mode applied by the field-side automatic detection equipment or terminal equipment. In the establishment of the training mode, the biggest problem is that the ratio of good samples and defective samples is very different, and defective samples are obtained. It will also make it difficult to effectively train a model that can identify flaws.

Therefore, in the current era of automated production in smart factories, with the application of artificial intelligence technology becoming the mainstream, how to introduce artificial intelligence technology into automatic detection equipment to further reduce the missed detection rate or false alarm rate, and shorten the overall detection process. It saves a lot of labor costs and improves the output and yield of the production line. At the same time, it can collect the results of manual re-judgment on the field side as the data for subsequent model training, so as to improve the accuracy of classification judgment or identification of the object to be tested, and improve the quality of testing. If it is perfect, it needs to be redesigned by those engaged in this industry to solve it.

Therefore, in view of the above-mentioned deficiencies, the inventor collected relevant information, through multiple evaluations and considerations, and with years of experience in this industry, continuous trial work and revision, before designing this kind of collection site-end manual review. The result of the invention was born with the invention patent of the automatic detection system and its operation method as the data for the subsequent classification model retraining.

The main purpose of the present invention is that the automatic inspection system adopts a non-invasive design and can be installed without stopping the production line, and introduces the existing machine vision inspection device in the production line, When the machine vision detection device detects the image of the object to be tested with defects, the artificial intelligence module can automatically perform the classification and inference of the defects with the deep learning algorithm, and the processing platform classifies the image of the object to be tested by the artificial intelligence module The inferred results are displayed on the operation screen of the repair station in the form of automatic insertion or filling, so as to reduce the items that need to be checked manually, so that the operator can focus on the re-judgment and improve the quality of the operation. When re-judging, the processing platform will intercept the result data of the manual re-judgment input by the operator through the input unit, and then send it to the artificial intelligence module as the data required for subsequent automatic classification model re-training and learning, so as to further reduce the automatic detection system. The false alarm rate is higher, so that the quality of the detection is more perfect.

The secondary purpose of the present invention is that when the operator judges that the defects marked by AI in the image are different from those marked by AI, it means that the result of AI classification inference is different from the result of the actual manual recheck marking, and the processing platform will collect the results of manual rechecking. , as the data for retraining and learning of the subsequent artificial intelligence module, so that the subsequent detection process will no longer misjudgment the same or similar misjudged samples, thereby improving the accuracy of classification judgment or identification, while retaining all the operator's past The operation behavior allows operators to familiarize themselves with the new functions added to the existing system, and can also shorten the operator's learning curve through the interactive operation screen. In this way, not only the inspection process can be shortened, the unit capacity can be improved, but more importantly, the software and hardware can be improved. All of them are plugged into the original automatic optical inspection system, which can effectively and easily upgrade the original system and equipment, and at the same time reduce the manual re-inspection of complicated inspection items, effectively reduce labor costs, and improve the output and yield of the production line.

100: Automatic detection system

101: Machine Vision Inspection Device

102: Repair Station

103: Artificial Intelligence Modules

104: Processing Platform

200: Object to be tested

300: Operation screen

301: Image screen of the object to be tested

302: The enlarged image screen of the object to be tested

303: Windows

[Fig. 1] is a schematic diagram of the structure of the automatic detection system of the present invention.

[Fig. 2] is a flow chart of the steps of the present invention.

[Fig. 3] is a schematic diagram of the present invention, which intercepts the operator's input re-judgment result for artificial intelligence training and transmits the same input back to the repair station.

[Fig. 4] is a schematic diagram of superimposing the result of artificial intelligence classification and inference on the operation screen according to the present invention.

In order to achieve the above-mentioned purpose and effect, the technical means and structure adopted by the present invention are described in detail as follows with reference to the preferred embodiment of the present invention, so as to be fully understood.

Please refer to Figures 1 to 4, which are the schematic diagram of the structure of the automatic detection system of the present invention, the flow chart of the steps, the interception of the operator's input and the result of the re-judgment to perform artificial intelligence training and return the same input to the repair station. The schematic diagram and the schematic diagram of superimposing the result of artificial intelligence classification and inference on the operation screen. It can be clearly seen from the figure that the automatic detection system 100 of the present invention includes a machine vision detection device 101 for conveying on the machine or conveyor belt. The object to be tested 200 is imaged and the object to be tested 200 with flaws or defects is detected; a repair station 102 connected to the machine vision inspection device 101 is used to receive the image of the object to be tested 200 and pass an operation screen. 300 to display to assist an operator to perform manual re-inspection; an artificial intelligence module 103 connected to the machine vision inspection device 101 is used to capture the image of the object to be tested 200 to automate the inspection of the object to be tested 200 classification inference of defects or defects; and a processing platform 104 connecting the repair station 102 and the artificial intelligence module 103 for receiving the result data of the classification inference of the artificial intelligence module 103, and integrating and superimposing it on the It is displayed on the operation screen 300 of the repair station 102 to assist the operator in performing the manual review operation.

The present invention further provides an operation method of an automatic detection system, wherein the automatic detection system The motion detection system 100 includes the above-mentioned machine vision detection device 101 , a repair station 102 , an artificial intelligence module 103 and a processing platform 104 , and the method includes the following steps:

(S101) The machine vision inspection device 101 acquires an image of the object to be tested 200, and detects the object to be tested 200 having defects.

( S102 ) The artificial intelligence module 103 captures the image of the object to be tested 200 for classification inference, and automatically superimposes the result data of the classification inference on the operation screen 300 of the repair station 102 through the processing platform 104 .

(S103) When the operator uses the operation screen 300 to perform manual re-inspection, the result data input by the operator to the repair station 102 through the user interface will be intercepted by the processing platform 104 and sent to the artificial intelligence module 103 as the artificial intelligence module 103 Subsequent automation of the data required for retraining and learning of the classification model.

(S104) The processing platform 104 returns the result data input by the operator to the repair station 102 with the same input.

It can be clearly seen from the figure and the above implementation steps that the machine vision inspection device 101 used in the present invention is an automatic optical inspection (Automated Optical Inspection, AOI) or an automatic visual inspection (Automated Visual Inspection, AVI) equipment, and uses high precision The camera takes images of the object to be tested 200 transported on the machine or conveyor belt, and detects the object to be tested 200 with defects or defects by machine vision technology, and the object to be tested 200 is preferably assembled with a printed circuit board. (PCBA) as an example, the machine vision inspection device 101 can be used to detect the soldering of components on the circuit board and the quality status after assembly, but it is not limited to this, and can also be applied to printed circuit boards (PCBs), chip carriers Missing parts, holes, circuits, visual inspection and measurement in boards, flat panel displays (FPD), semiconductors, electronic components or other processes.

In this embodiment, at least one repair station 102 can obtain images and defects or defect data (AOI Image and Defects) of the object under test 200 detected by a plurality of machine vision inspection devices 101 through a repair server (Repair Server). And after the artificial intelligence module 103 automatically scans the product serial number (S/N) or part number (P/N) of the label on the object to be tested 200 with an inline barcode scanner (Inline Barcode Scanner), it can be displayed through a display interface (such as VGA). /DVI) communication protocol to obtain the image of the test object 200, and then use a non-intrusive image extractor (N on-intrusive Image Extractor) to perform Cropped Image (Cropped Image) and Optical Character Recognition (OCR) Relevant image processing, after extracting the information of the object to be tested 200 and the characteristic data of the image, can use the computing platform of the AI Server (AI Server) to automatically carry out the classification and inference of defects or defects with deep learning technology, etc. , and use the database (Database) to collect and store the information and image characteristic data of the object to be tested 200, or use the robotic process automation software (RPA) to automatically collect the information from the manufacturing execution system (MES) and the enterprise management platform (SAP) Data and preprocessing to provide the data needed for deep learning.

In detail, the deep learning operation of the above-mentioned AI server can be mainly divided into model training (Training) and inference (Inference) mode, which extracts relevant and available data from the database through algorithms, such as CSV (comma separated) value, whose files store table data in plain text), images or texts, etc., and preprocess the extracted feature data to establish a model for identification or classification, which is used to cluster unlabeled data. ), and for labeled data, you can provide samples for model training and learning to find the optimized deep learning model, which can be used as a classifier after the training and learning process. Recognition and classification (Classification), and then can autonomously perform classification judgment or prediction to complete the inference mode, but this part is related to the artificial intelligence module 103 adopted. Deep learning includes many different algorithms, and the processes of data collection, modeling, training, evaluation, parameter adjustment, prediction, etc. are not the same, but whether it is deep learning or machine learning, both belong to the category of artificial intelligence. To Chen Ming.

The processing platform 103 can automatically scan the product serial number (S/N) or part number (P/N) label attached or printed on the DUT 200 through a handheld scanner, and use the artificial intelligence The module 103 overlays (overlays) the data of the result (AI Inferenced result) of the image classification inference of the object to be tested 200 on the UI Simulator corresponding to the repair station 102 by means of automatic insertion or filling. The operation screen 300 can display the results of different classification inferences, so as to assist the operator to perform the re-judgment operation by viewing the operation screen 300, which can reduce the items that need to be checked manually, so that the operator can focus on the re-judgment to improve the work quality , and the method of the operator's re-judgment is to use the input unit of the user interface including but not limited to a mouse, a keyboard or a touch screen to mark whether the image defects or defects of the object under test 200 are true and whether the classification result is correct and so on, and after the processing platform 103 intercepts the verification operation input by the operator or the result of manual inspection (Manual Inspection result), and then sends the input result or the result data of manual review to the AI server of the artificial intelligence module 103 The device is used to automate the retraining and verification process of the classification model. At the same time, the processing platform 103 can return the result data of the intercepted manual review to the repair station 102 for processing with the same input.

Please refer to FIGS. 3 to 4. The automatic inspection system 100 to which the present invention is applied adopts a non-intrusive design, which can be installed without stopping the production line, and can import the existing machine vision inspection in the original production line. The device 101 (such as automatic optical inspection or automatic visual inspection equipment) can not only expand a plurality of machine vision inspection devices 101 and the repair station 102 for continuous processing, and the artificial intelligence module 103 automatically scans the object to be tested 200 with a barcode scanner. Labels, can be automatically cropped The image of the object to be tested 200 is obtained, the information of the object to be tested 200 and the feature data detected by the machine vision detection device 101 are extracted for AI training and inference, and the result data of the classification inference can be automatically superimposed on the processing platform 103. In the operation screen 300 of the repair station 102 .

For example, the above-mentioned operation screen 300 of the repair station 102 includes various settings of the host, basic operations, setting of detection conditions, menus, data lists, the original image screen 301 of the test object, and the enlarged image screen of the test object 302, and a window 303 with specific text, different background colors or prompt colors and used to represent different AI classification inference results, etc., wherein the window 303 is for the processing platform 103 to infer the image classification of the object to be measured 200 by the artificial intelligence module 103 The result data is automatically superimposed on the prompt window in the operation screen 300. When the operator visually checks the operation screen 300 for re-judgment, because the artificial intelligence module 103 will detect the defective or defective object under test detected by the machine vision inspection device 101 200 for re-inspection, without a lot of labeled image data or samples for training, the false alarm rate can be reduced by more than 90%, and the results of classification inference are superimposed on the operation screen 300, so that the operator can display it according to the window 303 The result of AI classification inference determines whether to conduct manual re-examination, including No Result, Above Confidence, and Below Conference. It is not only easier to view the results, but also through interactive operations. The screen 300 guides the operator to accept the results of AI classification and inference and reduce the items that need to be checked manually. When the number of checked items is greatly reduced, the operator can focus on re-judgment and improve the quality of work.

If the window 303 shows that the result of the AI classification inference is no result or lacks confidence, the operator can use the input unit (such as a mouse, keyboard, etc.) to mark the defect or whether the defect is a defect on the enlarged image screen 302 of the object to be tested. True, and in the operation screen 300, correct options and incorrect options can be provided for the operator to click, when the operator judges the defects of the AI mark on the image Or when the defect position is different from the marked position, it means that the result of AI classification inference is different from the result marked by the actual manual recheck, which may be caused by the misjudgment of the AI model, resulting in a classification error or manual marking error, and you need to manually confirm the correctness of the marking. The processing platform 103 will automatically intercept the verification operation input by the operator or the result of the manual inspection, and then send the result data and images of the manual review to the AI server of the artificial intelligence module 103 for automatic processing. The retraining and verification process of the classification model, and continue to train the deep learning model of the AI server, until the artificial intelligence module 103 can autonomously perform classification judgment or prediction, or stop the training when the accuracy rate no longer improves, so as to further reduce The missed detection rate or false alarm rate of the automatic detection system 100 makes the detection quality more perfect.

The processing platform 103 of the automatic detection system 100 can simultaneously collect the results of the manual re-judgment as the data required for the subsequent retraining and learning of the artificial intelligence module 103, wherein the re-training process is to add the result data of the manual re-judgment to the AI server. The learning sample database of the device, so that the classification model can automatically adjust the parameters or weights of each node, so that the subsequent detection process will no longer cause misjudgment for the same or similar misjudged samples, thereby improving the classification judgment or identification of the object to be tested 200. This allows operators to quickly familiarize themselves with new functions added to the existing system, while shortening the operator’s learning curve through the interactive operation screen 300. In this way, not only can The overall inspection process is shortened and the unit production capacity is improved. More importantly, the software and hardware are externally attached to the original automatic optical inspection system, which can effectively and easily upgrade the original system and equipment, while reducing the complexity of manual re-inspection. Testing items can effectively reduce labor costs and improve the output and yield of the production line.

The above detailed description is only for describing a preferred feasible embodiment of the present invention, but this embodiment is not intended to limit the scope of the patent application of the present invention, and all others do not depart from the present invention Equivalent changes and modifications made under the spirit of the art disclosed in the disclosure shall be included in the scope of the patent covered by the present invention.

To sum up, the automatic detection system and its operation method of the present invention can indeed achieve its effect and purpose when used, so the present invention is an invention with excellent practicability. , I hope that the review committee will approve the case as soon as possible to protect the inventor's hard work. If the review committee has any doubts, please do not hesitate to send a letter for instructions, and the inventor will do his best to cooperate.

100: Automatic detection system

101: Machine Vision Inspection Device

102: Repair Station

103: Artificial Intelligence Modules

104: Processing Platform

200: Object to be tested

Claims (8)

An automatic inspection system, which includes a machine vision inspection device for acquiring an image of an object to be tested and detecting the object to be tested with defects; a repair station connected to the machine vision inspection device for receiving the object to be tested The image of the object is displayed through an operation screen to assist an operator to perform manual re-inspection; an artificial intelligence module connected to the machine vision inspection device is used to capture the image of the object to be tested for classification and inference ; And a processing platform connecting the repair station and the artificial intelligence module, for receiving the result data of the classification inference of the artificial intelligence module and integrating it on the operation screen of the repair station, assisting the operator to perform manual restoration judgment, wherein the processing platform is used to perform the following steps: The result data of the image classification inference of the object to be tested by the artificial intelligence module is automatically superimposed on the operation screen of the repair station; intercepting the result data of the manual review entered by the operator to the repair station through a user interface; Send the result data of the manual re-judgment to the artificial intelligence module to be used as the data for the artificial intelligence module to automatically retrain and learn the classification model; and The processing platform returns the result data entered by the operator to the repair station with the same input. The automatic inspection system of claim 1, wherein the machine vision inspection device is an automatic optical inspection or automatic vision inspection equipment. The automatic detection system according to claim 1, wherein the artificial intelligence module obtains the image of the object to be tested through a communication protocol of a display interface, and uses a non-invasive image extractor to trim the image and image processing for optical character recognition to extract the The information of the measured object and the characteristic data of the image, and then use deep learning technology to automate the classification and inference of defects or defects. The automatic detection system according to claim 3, wherein the artificial intelligence module collects and stores the information of the object to be tested and the characteristic data of the image through a database. The automatic detection system as claimed in claim 1, wherein the processing platform superimposes the result data of the classification inference on a simulator of the user interface by automatically inserting or filling in, so that the operation screen displays different classifications inferred result. The automatic detection system according to claim 5, wherein the operation screen includes an original image of the object to be tested, an enlarged image of the object to be tested, and a window for displaying the results of the different classification inferences. The automatic inspection system as claimed in claim 5, wherein the operator marks the image of the object to be tested, inputs the operator's verification operation or the result of manual inspection through an input unit of the user interface. An operation method of an automatic inspection system, the automatic inspection system includes a machine vision inspection device, a repair station, an artificial intelligence module and a processing platform, and the method includes: The machine vision inspection device obtains an image of an object to be tested, and detects the object to be tested with flaws; The artificial intelligence module obtains the image of the object to be tested for classification inference, and automatically superimposes the result data of the classification inference on an operation screen of the repair station through the processing platform; When an operator uses the operation screen to perform manual re-inspection, the result data input by the operator to the repair station through a user interface will be intercepted by the processing platform and sent to the manual operator an intelligence module, which is used as data for automatic classification model retraining and learning by the artificial intelligence module; and The processing platform returns the result data entered by the operator to the repair station with the same input.

Images