TWI758134B - System for using image features corresponding to component identification for secondary inspection and method thereof - Google Patents

Abstract

A system for using image features corresponding to component identification for a secondary inspection and a method thereof are provided. By obtaining component identification of components welded on a printed circuit board (PCB), obtaining standard image features of a target component has not passed an automated optical inspection (AOI) based on the component identification of the target component, generating detection image features of a detection image of the target component, and using a detection model provided to the standard image features and the detection features to determine whether the target component can pass secondary inspection, the system and the method can determine soldering status of alternative component on PCB effectively, and can achieve the effect of improving accuracy of AOI and reducing labor cost.

Description

System and method for secondary detection of graphic features based on component identification data

An optical inspection system and method thereof, particularly a system and method for secondary inspection using pattern features based on component identification data.

Industry 4.0 (Industry 4.0), also known as the fourth industrial revolution, does not simply create new industrial technologies, but focuses on integrating existing industrial technologies, sales processes and product experience, and building adaptive capabilities through artificial intelligence technology. It integrates customers and business partners in the business process and value process to provide perfect after-sales service, and then builds a new intelligent industrial world with awareness.

With the wave of Industry 4.0 sweeping the world, manufacturers are all using intelligent manufacturing to optimize production transformation and enhance competitiveness. Smart manufacturing is based on sensing technology, network technology, automation technology, and artificial intelligence. Through the process of perception, human-computer interaction, decision-making, execution, and feedback, it realizes product design and manufacturing, enterprise management and service. of intelligence.

The electronic assembly industry is characterized by small profits but quick turnover and fierce product price competition, which makes the industry pursue more effective control and optimization of raw materials and production tools, so as to maximize the efficiency of factory production resources. For example, one of the various technologies that are often used in the electronic assembly industry is Surface-Mount Technology (SMT), which is produced by a surface-mount device (also commonly referred to as a mounter or a bonder). ) Electronic components such as resistors, capacitors, transistors, and integrated circuits are electrically connected to the printed circuit board (PCB) by soldering, so that the electronic components are mounted on the printed circuit board. The overall speed of assembling printed circuit boards can be increased by using surface mount technology.

On the other hand, due to the miniaturization and increase in density of electronic components, the possibility of poor soldering of electronic components on printed circuit boards increases accordingly. Detection has become a necessary part. Among them, Automated Optical Inspection (AOI) is a representative method for detecting welding conditions. It uses machine vision devices to obtain the surface state of the object to be inspected, and then uses computer image processing technology to detect defects commonly encountered in welding production. , as an improvement to the shortcomings of the traditional manual detection using optical instruments.

In more detail, in the production line as shown in "Fig. 1", when the printed circuit board 110 reaches the placement machine (surface mount device 130), the placement machine solders electronic components on the surface of the printed circuit board 110, and then passes through the placement machine 110. When the printed circuit board 110 of the placement machine reaches the optical inspection device 150, the optical inspection device 150 scans the printed circuit board 110 through the camera lens, captures the test image of the printed circuit board 110, and compares the test images with the image processing technology. The electronic components on the printed circuit board 110 and the qualified parameters of the corresponding electronic components in the database are used to detect whether there are defects such as foreign objects or poor soldering on the printed circuit board 110. After that, the optical detection device 150 can output the test of the printed circuit board 110. In this way, the test image is displayed on the display device (not shown in the figure) or the test image is automatically marked, so that the defects on the printed circuit board 110 can be displayed or marked for repairing by maintenance personnel.

But in fact, when the placement machine solders electronic components on the printed circuit board, if there is a shortage of electronic components, the placement machine may solder other electronic components that can be replaced on the printed circuit board. The circuit board has not been replaced. When the optical detection device detects the printed circuit board, it will continue to use the parameters of the electronic components lacking material to detect the replaced electronic components. In this way, the replaced electronic components on the printed circuit board often fail to pass. Optical detection.

To sum up, it can be seen that there has been a long-standing problem in the prior art that the optical inspection device cannot effectively judge the soldering condition of the substitute electronic component when the surface mount device uses the substitute electronic component. Therefore, it is necessary to propose an improved technical means to solve this problem. question.

In view of the problem in the prior art that the optical inspection device cannot effectively judge the soldering condition of the substitute electronic component when the surface mount device uses the substitute electronic component, the present invention discloses a system and method for secondary detection using graphic features according to the component identification data, wherein :

The system for using pattern feature secondary detection based on component identification data disclosed in the present invention at least includes: a model building module for generating a detection model using a pattern feature identification algorithm; a data loading module for obtaining a target circuit board The component identification data of the above electronic components; the image loading module is used to obtain the detection image of the target component, and the target component is any electronic component that has not passed the optical detection; the feature acquisition module is used to The component identification data obtains the standard image features of the target component and the detection image features used to generate the detection image; the component detection module is used to determine whether the target component passes the detection by using the detection model according to the detection image features and the standard image features.

The method of using pattern feature secondary detection based on component identification data disclosed in the present invention at least includes the following steps: generating a detection model by using pattern feature identification algorithm; obtaining component identification data of electronic components on a target circuit board; Detecting images, the target component is an electronic component that has not passed the optical detection; obtains the standard image features of the target component according to the component identification data of the target component; generates the detected image features of the detected image; The model determines whether the target component passes the test.

The system and method disclosed in the present invention are as described above. The difference between the present invention and the prior art is that the present invention obtains the component identification data of the electronic components soldered on the target circuit board by the surface mount device, and then obtains the information that fails the optical inspection according to the component identification data. The standard image features of the target device, and the detected image features of the detected image of the target device are generated, and the detection model is used to determine whether the target device has passed the inspection according to the detected image features and the standard image features, so as to solve the problems existing in the prior art, and can achieve The technical efficacy of improving the accuracy of optical inspection and reducing labor costs.

The features and implementations of the present invention will be described in detail below in conjunction with the drawings and examples, and the content is sufficient to enable any person skilled in the relevant art to easily and fully understand the technical means applied to solve the technical problems of the present invention and implement them accordingly, thereby achieving The effect that the present invention can achieve.

The invention can obtain the component identification data of the electronic components soldered on the circuit board, so as to obtain the standard image features of the electronic components that have not passed the optical inspection according to the component identification data of the electronic components that have not passed the optical inspection, and according to the standard image features and Inspection image features of electronic components that fail the optical inspection A second inspection is performed on the electronic components that fail the optical inspection.

The standard image feature and the detection image feature proposed in the present invention are data generated by using a convolutional neural network (Convolutional Neural Network, CNN) to calculate the standard image and the detection image of the electronic components that have not passed the optical detection respectively. way of expressing. Wherein, the inspection image is an image including electronic components that have not passed the optical detection, and the standard image is an image that includes the same electronic components and can pass the optical detection. In addition, the convolutional neural network mentioned in the present invention includes, but is not limited to, Alexnet, VGG, and the like.

First, the system operation of the present invention will be described with reference to “FIG. 2” of the system architecture diagram of the present invention using the second detection of graphic features based on the component identification data. As shown in "Fig. 2", the system of the present invention is applied in the secondary inspection device 200, including a model building module 210, a data loading module 220, an image loading module 230, a feature obtaining module 250, and a component Detection module 260 .

The model building module 210 is responsible for generating the detection model using the pattern feature recognition algorithm. The graphic feature identification algorithm proposed in the present invention includes but is not limited to loss functions (or loss formulas) such as Triplet loss/Contrastive loss/Margin loss, Pairwise Ranking loss, etc. That is to say, the model building module 210 can call the above loss function by calling (or loss formula) to produce a detection model.

For example, the model building module 210 can obtain a certain number of standard images of a certain electronic component and sample images of the same electronic component through optical inspection (referred to as "positive sample images" in the present invention), and can obtain the same The sample images of the electronic components that have not passed the optical inspection (referred to as "negative sample images" in the present invention), then, the model building module 210 can use the convolutional neural network to calculate the standard image, the positive sample image and the negative sample image. Image features (for example, calculated directly by using the convolutional neural network or by calling the feature acquisition module 250), and the calculated image features of the standard image, the image features of the positive sample images, and the image features of the negative sample images (in the Also referred to in the present invention as "standard image features", "positive sample image features", and "negative sample image features") to train the graphics feature recognition algorithm (for example, the standard image features, positive sample image features, negative sample image features, The sample image features are input to the Triplet loss loss function, so that the model building module 210 continuously adjusts the margin constant of the Triplet loss loss function) until the graphic feature recognition algorithm determines the Euclidean distance (Euclidean distance) between the standard image features and the positive sample image features. When the Euclidean distance between the standard image feature and the negative sample image feature is greater than the same threshold value, the establishment of the detection model can be completed. Wherein, the above-mentioned threshold value is usually a value that can be determined in the above-mentioned training process.

The data loading module 220 is responsible for reading component identification data of a plurality of electronic components on the target circuit board. The target circuit board mentioned in the present invention is the printed circuit board 110 detected by the placement machine (surface mount device 130 ) and the optical detection device 150 ; the component identification data and the electronic components have a one-to-one correspondence, which can indicate the corresponding The electronic components can be arbitrarily arranged by any number of characters, numbers, letters and symbols.

The data loading module 220 can be connected to the placement machine (surface mount device 130 ) that solders electronic components on the target circuit board, and the placement machine downloads the component information of all electronic components soldered on the target circuit board; data The loading module 220 can also be connected to a specific relay device (not shown in the figure), such as a server, and the relay device downloads the component information of all electronic components soldered by the placement machine on the target circuit board. The component information includes component identification data and installation position data, and the installation position data mentioned in the present invention can represent the position of the electronic component on the target circuit board.

The image loading module 230 is responsible for reading the component identification data of the target component and detecting the image. In the present invention, the target component is any electronic component that fails the optical inspection among all the electronic components on the target circuit board.

The image loading module 230 can receive the test record output by the optical inspection device 150 that performs automatic optical inspection on the target circuit board, and can read the component identification data and the inspection image of the target element from the received test record; the image The loading module 230 can also be connected to a relay device (not shown in the figure) that receives and stores the test records output by the optical detection device 150, and the relay device downloads the component identification data of the target components contained in the test records and detection images. The test record output by the optical inspection device 150 may include the test image of the target circuit board and the component identification data of the target components that fail the optical inspection, wherein the test image is an image output by the optical inspection device covering the entire target circuit board . In addition, the test record can also include the test position information or test image of the target element. The above test position information can indicate the position and size of the target element in the test image, for example, the coordinates of the diagonal corner of the target element in the test image, and For example, the coordinates and length and width of a specific vertex of the target element in the test image.

In some embodiments, if the test record does not include the test image, the image loading module 230 may also read the component identification data of the target component and the test record from the test record after receiving the test record output by the optical detection device 150 . The test position information is tested, and the test image of the target device can be extracted from the test image included in the test record according to the test position information.

The image loading module 230 can also create a data table corresponding to the acquired target component identification data and the detected image, so as to provide the feature acquisition module 250 for use, but the invention is not limited to this. Wherein, each piece of data in the data table established by the image loading module 230 includes a component identification data and a detection image of the corresponding electronic component.

The feature obtaining module 250 is responsible for obtaining standard image features of the target device according to the device identification data of the target device obtained by the image loading module 230 . For example, the feature acquisition module 250 can read the component identification data of the target component from the data table created by the image loading module 230, and can read and read the component identification from the pre-established image data The standard image feature of the target device corresponding to the data.

In some embodiments, if the pre-created image data does not include the standard image features of the target device, but only includes the standard image of the target device, the feature acquisition module 250 can also read the target device according to the device identification data of the target device The standard image of the target device can be calculated using a convolutional neural network to generate standard image features of the target element.

The feature obtaining module 250 is also responsible for generating the detected image features of the detected image obtained by the image loading module 230 . For example, the feature acquisition module 250 can read out the detection image of the target element from the data table established by the image loading module 230, and can use the convolutional neural network to calculate the readout detection image to generate the target element detection image features.

The component detection module 260 is responsible for determining whether the target component passes the detection using the detection model established by the model establishment module 210 according to the detection image features obtained by the feature obtaining module 250 and the standard image features.

Next, an embodiment is used to explain the operation system and method of the present invention, and please refer to “FIG. 3A” for the flowchart of the method for secondary detection of graphic features based on component identification data according to the present invention. In this embodiment, it is assumed that the present invention is applied to the secondary detection device 200 . The secondary detection device 200 is arranged in the production line, and the operation sequence of the secondary detection device 200 is arranged after the optical detection device 150 .

First, before the secondary inspection apparatus 200 starts to inspect the electronic components on the printed circuit board, the model building module 210 may first generate an inspection model by using a graphic feature recognition algorithm (step 310 ). In this embodiment, it is assumed that the developer can pre-select a certain number of standard images, positive sample images and negative sample images of each electronic component, and can select the selected standard images, positive sample images and negative samples of each electronic component The image is provided to the model building module 210, so that the model building module 210 can obtain the standard image feature of the standard image, the positive sample image feature of the positive sample image, and the sample image feature of the negative sample image through the feature obtaining module 250, and then, The model building module 210 can use the Triplet loss loss function to compare the Euclidean distance between the standard image feature and the positive sample image feature and the Euclidean distance between the standard image feature and the negative sample image feature in the feature space, and continuously adjust the margin constant, and so on. , until the Euclidean distance between the standard image feature and the positive sample image feature calculated by the Triplet loss loss function is less than the threshold value (assuming 1.1), and the Euclidean distance between the standard image feature and the negative sample image feature is greater than the threshold value, and the detection can be completed Model establishment.

After the model building module 210 generates the inspection model, the secondary inspection device 200 can be used in the production line. In the production line, the printed circuit board 110 will first arrive at the placement machine (surface mount device 130 ), and the placement machine will solder a plurality of electronic components on the printed circuit board 110 according to the preset installation information. After that, the data is loaded into The module 220 can obtain the component identification data of the electronic components soldered by the placement machine on the target circuit board (step 320). In this embodiment, it is assumed that the data loading module 220 can be connected to the placement machine, and can receive the component identification data of the electronic components soldered by the placement machine on the target circuit board.

After the printed circuit board 110 passes through the placement machine, the printed circuit board 110 will reach the optical inspection device 150, and the optical inspection device 150 can test the electronic components soldered on the printed circuit board 110 through automatic optical inspection and generate corresponding Test Record.

After the printed circuit board 110 passes through the optical inspection device 150, if any electronic components on the printed circuit board 110 do not pass the optical inspection by the optical inspection device 150, the test record generated by the optical inspection device 150 may include the printed circuit board The test image of 110 and the component identification data of the target components on the printed circuit board 110 that failed the optical inspection.

After the printed circuit board 110 passes through the optical inspection device 150 , the image loading module 230 can obtain the inspection image of the target element that fails the optical inspection from the test record output by the optical inspection device 150 (step 330 ). In this embodiment, assuming that the test record output by the optical inspection device 150 already includes the test image of the target device, the image loading module 230 can read the device identification data and the test image of the target device from the test record; If the test record output by the optical detection device 150 does not include the detection image of the target device, the image loading module 230 can first read the test record output by the optical detection device 150 ( Step 331 ), and read the component identification data and test position information of the target device from the test record, and extract the test image from the test image included in the test record according to the read test position data (step 335 ). In addition, the image loading module 230 can also write the read component identification data and the obtained inspection image into the data table as a piece of data.

After the image loading module 230 obtains the detection image of the target device, the feature obtaining module 250 can obtain the standard image features of the target device according to the device identification data of the target device (step 350 ). In this embodiment, if the developer has created a correspondence table between the component identification data of the target component and the standard image features in advance, the feature acquisition module 250 can directly read the standard image features corresponding to the component identification data from the correspondence table; If the developer only pre-stores the standard image of the target device, the feature acquisition module 250 can, as shown in the flow of "Fig. 3C", load the meta-identification data corresponding to the target device created by the image loading module 230 with the target device. After obtaining the component identification data of the target component from the data table of the standard image (steps 340 and 351 ), first read the standard image of the target component according to the obtained component identification data (step 353 ), and use the convolutional neural network to The read standard image is calculated to generate standard image features (step 355).

Likewise, after the image loading module 230 obtains the detection image of the target device, the feature obtaining module 250 can also generate the detection image feature of the detection image obtained by the image loading module 230 (step 360 ). Similar to the above, the feature obtaining module 250 can use the convolutional neural network to calculate the detected image to generate the detected image features.

After the feature acquisition module 250 acquires the standard image features and generates the detection image features, the component detection module 260 can use the standard image features acquired by the feature acquisition module 250 and the generated detection image features using the model creation module 210 The detection model determines whether the target component passes the detection (step 370 ). In this embodiment, the detection model can calculate the Euclidean distance between the standard image feature and the detected image feature, and determine whether the target element passes the test according to whether the Euclidean distance is smaller than a threshold value.

In this way, when the placement machine (surface mount device 130 ) solders the substitute electronic component on the printed circuit board 110 and the optical inspection device 150 determines that the substitute electronic component fails the optical inspection, the present invention can more accurately determine that the substitute electronic component fails the optical inspection. Electronic components instead of optical detection pass the test.

From the above, it can be seen that the difference between the present invention and the prior art is that after obtaining the component identification data of the electronic components soldered on the target circuit board by the surface mount device, the standard of obtaining the target components that have not passed the optical inspection according to the component identification data Image features, and generate the detection image features of the detection image of the target element, and use the detection model to determine whether the target element has passed the detection technical means according to the detection image characteristics and the standard image characteristics. When the surface mount device uses an alternative electronic component, the optical inspection device will not be able to effectively determine the problem of the soldering condition of the alternative electronic component, thereby achieving the technical effect of improving the accuracy of optical inspection and reducing labor costs.

Furthermore, the method for secondary detection of graphic features based on component identification data of the present invention can be implemented in hardware, software, or a combination of hardware and software, and can also be implemented in a centralized manner in a computer system or distributed with different components. Implemented in a decentralized manner across several interconnected computer systems.

Although the embodiments disclosed in the present invention are as above, the above-mentioned contents are not intended to directly limit the scope of the patent protection of the present invention. Any person with ordinary knowledge in the technical field to which the present invention pertains, without departing from the spirit and scope disclosed by the present invention, makes slight modifications to the form and details of the implementation of the present invention, all belong to the patent protection of the present invention scope. The scope of patent protection of the present invention shall still be defined by the appended patent application scope.

110: Printed circuit boards 130: Surface Mount Device 150: Optical detection device 200: Secondary detection device 210: Model building module 220:Data loading module 230: Image loading module 250: Feature Acquisition Module 260: Component inspection module Step 310: Generate a detection model using a pattern feature identification algorithm Step 320: Obtain the component identification data of the electronic components on the target circuit board Step 330: Obtain the inspection image of the target element that fails the optical inspection Step 331: Read the test record output by the optical detection device, the test record includes the test image and the test position information of the target element Step 335: Extract the test image of the target device from the test image according to the test position information of the target device Step 340: Create a data table of the component identification data corresponding to the target component and the detection image Step 350: Obtain the standard image features of the target device according to the device identification data of the target device Step 351: Read the component identification data of the target component from the data table Step 353: Read the standard image of the target device according to the device identification data of the target device Step 355: Compute the standard image using a convolutional neural network to generate standard image features Step 360: Generate the detected image features of the detected image Step 370: Use the detection model to determine whether the target element passes the detection according to the detection image feature and the standard image feature

Figure 1 is a schematic diagram of a conventional production line. FIG. 2 is a system architecture diagram of the second detection system based on the component identification data using the graphic feature according to the present invention. FIG. 3A is a flowchart of the method for secondary detection of graphic features based on component identification data according to the present invention. FIG. 3B is a flowchart of the method for obtaining a detection image according to the present invention. FIG. 3C is a flowchart of the method for generating standard image features proposed by the present invention.

Step 310: Generate a detection model using a pattern feature identification algorithm

Step 320: Obtain the component identification data of the electronic components on the target circuit board

Step 330: Obtain the inspection image of the target element that fails the optical inspection

Step 350: Obtain the standard image features of the target device according to the device identification data of the target device

Step 360: Generate the detected image features of the detected image

Step 370: Use the detection model to determine whether the target element passes the detection according to the detection image feature and the standard image feature

Claims (10)

A method for secondary detection using pattern features based on component identification data is applied to a secondary detection device, the method at least comprising the following steps: obtaining component identification data of a plurality of electronic components on a target circuit board; obtaining a target component A detection image, the target component is any electronic component that has not passed the optical detection; obtain a standard image feature of the target component according to the component identification data of the target component; obtain a positive sample that has passed the optical detection Image and a negative sample image that failed the optical detection, and extract the image features of the positive sample image and the negative sample image; use the standard image feature, the positive sample image feature and the negative sample image feature to perform a graphic feature recognition algorithm method for training until the Euclidean distance between the standard image feature and the positive sample image feature is less than a threshold value and the Euclidean distance between the standard image feature and the negative sample image feature is greater than the threshold value, thereby generating a detection model; using The vector data generated by the convolutional neural network (Convolutional Neural Network, CNN) calculation of the detection image is used as a detection image feature of the detection image; and the detection model is used to calculate the Euclidean of the detection image feature and the standard image feature distance, and according to whether the calculated Euclidean distance is smaller than the threshold value, it is judged whether the target element passes the detection. The method for secondary detection using graphic features based on component identification data as claimed in claim 1, wherein the step of obtaining the detection image of the target component further comprises the target included in a test record output by an optical detection device The test position information of a component, the step of retrieving the test image from a test image included in the test record. The method for secondary detection using graphic features based on component identification data as claimed in claim 1, wherein the method further comprises establishing component identification data corresponding to the target component and the detection image after the step of obtaining the detection image of the target component Steps of a data sheet. The method for secondary detection of graphic features based on component identification data according to claim 1, wherein the step of obtaining the standard image feature of the target component according to the component identification data of the target component is to read the target from a data table and read a pre-established standard image according to the component identification data of the target component, and use the convolutional neural network to calculate the standard image to generate the standard image feature. The method for using pattern feature secondary detection based on component identification data as claimed in claim 1, wherein the step of obtaining the component identification data of a plurality of electronic components on the target circuit board is to connect to a surface mount device or a relay The device is used to obtain the component identification data of the electronic components soldered by the surface mount device on the target circuit board. A system for secondary detection using graphic features according to component identification data is applied to a secondary detection device. The system at least comprises: a data loading module for obtaining components of a plurality of electronic components on a target circuit board identification data; an image loading module for obtaining a detection image of a target element, the target element being any electronic element that has not passed the optical detection among the electronic elements; a feature obtaining module for obtaining a detection image according to the The element identification data of the target element obtains a standard image feature of the target element, and uses the vector data generated by the convolutional neural network to calculate the detected image as one of the detected image features of the detected image; a model building module for obtaining a positive sample image that passes the optical detection and a negative sample image that fails the optical detection, extracting image features of the positive sample image and the negative sample image, and using the standard image The feature, the positive sample image feature and the negative sample image feature train a graphic feature recognition algorithm until the Euclidean distance between the standard image feature and the positive sample image feature is less than a threshold and the standard image feature and the negative image feature are The Euclidean distance between the sample image features is greater than the threshold value, so as to generate a detection model; and a component detection module is used to calculate the Euclidean distance between the detection image feature and the standard image feature by using the detection model, and according to the calculated Euclidean distance Whether the Euclidean distance is smaller than the threshold value determines whether the target element passes the detection. The system for secondary detection of graphic features based on component identification data as claimed in claim 6, wherein the image loading module is further used for one of the target components included in a test record output by an optical detection device The test position information is extracted from a test image included in the test record. As claimed in claim 6, the system for secondary detection of graphic features based on component identification data, wherein the image loading module is further configured to create a data table of component identification data corresponding to the target component and the detected image. The system for secondary detection of graphic features based on component identification data as described in claim 6, wherein the feature acquisition module is further configured to read the component identification data of the target component from the data table, and based on the target component's identification data The component identification data reads a pre-established standard image, and uses a convolutional neural network to calculate the standard image to generate the standard image feature. The system for secondary detection of graphic features based on component identification data as claimed in claim 6, wherein the data loading module is connected to a surface mount device or a relay device to obtain the surface mount device soldered on the target Component identification data of the electronic components on the circuit board.

Images