TW202238447A - System for using image features corresponding to component identification for secondary inspection and method thereof - Google Patents

Abstract

A system for using image features corresponding to component identification for a secondary inspection and a method thereof are provided. By obtaining component identification of components welded on a printed circuit board (PCB), obtaining standard image features of a target component has not passed an automated optical inspection (AOI) based on the component identification of the target component, generating detection image features of a detection image of the target component, and using a detection model provided to the standard image features and the detection features to determine whether the target component can pass secondary inspection, the system and the method can determine soldering status of alternative component on PCB effectively, and can achieve the effect of improving accuracy of AOI and reducing labor cost.

Description

System and method for secondary inspection using graphic features based on component identification data

An optical inspection system and method thereof, especially a system and method for secondary inspection using graphic features based on component identification data.

Industry 4.0 (Industry 4.0), also known as the fourth industrial revolution, does not just create new industrial technologies, but focuses on integrating existing industrial technologies, sales processes and product experience, and establishes adaptable products through artificial intelligence technology. Smart factory based on nature, resource efficiency and human factors engineering, and integrate customers and business partners in business process and value process to provide perfect after-sales service, and then construct a new intelligent industrial world with awareness.

With the wave of Industry 4.0 sweeping the world, manufacturers are all using smart manufacturing to optimize production transformation and enhance competitiveness. Smart manufacturing is based on sensing technology, network technology, automation technology, and artificial intelligence, through the process of perception, human-computer interaction, decision-making, execution, and feedback to realize product design and manufacturing, enterprise management and services of wisdom.

The characteristics of small profits but quick turnover and fierce product price competition in the electronic assembly industry make the industry pursue more effective control and optimization of raw materials and production tools, so as to maximize the efficiency of factory production resources. For example, one of the various technologies commonly used in the electronics assembly industry is Surface-Mount Technology (SMT). ) Electrically connect electronic components such as resistors, capacitors, transistors, and integrated circuits to a printed circuit board (PCB) through soldering, so that the electronic components are mounted on the printed circuit board. The overall speed of assembling printed circuit boards can be increased by using surface mount technology.

On the other hand, due to the miniaturization and increase in density of electronic components, the possibility of poor soldering of electronic components on the printed circuit board increases accordingly. Therefore, in the process of manufacturing printed circuit boards using surface mount technology, the soldering status Detection has become a necessary part. Among them, Automated Optical Inspection (AOI) is a representative method for detecting welding conditions. It uses machine vision devices to obtain the surface state of the items to be inspected, and then uses computer image processing technology to detect defects often encountered in welding production. , as an improvement to the shortcomings of traditionally using optical instruments to detect manually.

In more detail, on the production line shown in "Fig. 1", when the printed circuit board 110 arrives at the placement machine (surface mount device 130), the placement machine solders electronic components on the surface of the printed circuit board 110, and passes through When the printed circuit board 110 of the mounter arrives at the optical detection device 150, the optical detection device 150 scans the printed circuit board 110 through the photographic lens, captures the test image of the printed circuit board 110, and compares the test image with the image processing technology. The qualified parameters of the electronic components on the printed circuit board 110 and the corresponding electronic components in the database are used to detect whether there are foreign objects or defects such as poor soldering on the printed circuit board 110. After that, the optical detection device 150 can output the test results of the printed circuit board 110 In this way, the test image is displayed or automatically marked on the test image through a display device (not shown in the figure), so that defects on the printed circuit board 110 can be displayed or marked for maintenance personnel to repair.

But in fact, when the placement machine solders electronic components on the printed circuit board, if there is a shortage of electronic components, the placement machine may solder other electronic components that can be replaced on the printed circuit board. The circuit board has not been replaced. When the optical detection device detects the printed circuit board, it will continue to use the parameters of the electronic components that are missing to detect the replaced electronic components. In this way, the replaced electronic components on the printed circuit board often cannot pass through. Optical detection.

In summary, it can be known that the prior art has long had the problem that the optical detection device cannot effectively judge the soldering status of the replacement electronic component when the surface mount device uses the replacement electronic component. Therefore, it is necessary to propose an improved technical means to solve this problem. question.

In view of the problem in the prior art that the optical detection device cannot effectively judge the soldering status of the replacement electronic component when the surface mount device uses the replacement electronic component, the present invention discloses a system and method for secondary detection using graphic features based on component identification data, wherein :

The system disclosed in the present invention for secondary detection using graphic features based on component identification data at least includes: a model building module for generating a detection model using a graphic feature recognition algorithm; a data loading module for obtaining a target circuit board The component identification data of the above electronic components; the image loading module is used to obtain the detection image of the target component, and the target component is any electronic component that has not passed the optical inspection; the feature acquisition module is used to obtain the detection image of the target component according to the The component identification data obtains the standard image features of the target component, and the detection image features used to generate the detection image; the component detection module is used to judge whether the target component passes the detection based on the detection image features and the standard image features using the detection model.

The method disclosed by the present invention using graphic features for secondary detection based on component identification data, the steps at least include: using graphic feature recognition algorithm to generate a detection model; obtaining component identification data of electronic components on the target circuit board; obtaining the target component Detection image, the target component is an electronic component that has not passed the optical inspection; obtain the standard image feature of the target component based on the component identification data of the target component; generate the detection image feature of the detection image; use the detection method based on the detection image feature and the standard image feature The model judges whether the target component passes the inspection.

The system and method disclosed in the present invention are as above, and the difference between the present invention and the prior art is that the present invention obtains the components that have not passed the optical inspection based on the component identification data after obtaining the component identification data of the electronic components soldered on the target circuit board by the surface mount device. The standard image features of the target component, and generate the detection image features of the detection image of the target component, and use the detection model to judge whether the target component has passed the detection based on the detection image features and standard image features, so as to solve the problems existing in the prior art, and can achieve The technical effect of improving the accuracy of optical inspection and reducing labor costs.

The features and implementation methods of the present invention will be described in detail below in conjunction with the drawings and embodiments, the content is enough to enable anyone familiar with the relevant art to easily and fully understand the technical means used to solve the technical problems of the present invention and implement them accordingly, thereby realizing The effect that the present invention can achieve.

The present invention can obtain the component identification data of the electronic components soldered on the circuit board, so as to obtain the standard image features of the electronic components that have not passed the optical inspection based on the component identification data of the electronic components that have not passed the optical inspection, and based on the standard image features and Inspection image features of electronic components that fail optical inspection Perform a second inspection on electronic components that fail optical inspection.

The standard image features and detection image features mentioned in the present invention are the data generated by using the convolutional neural network (CNN) to calculate the standard images and detection images of electronic components that have not passed the optical detection, usually in the form of vector expressed in a way. Wherein, the inspection image is an image containing electronic components that have not passed optical inspection, and the standard image is an image containing the same electronic components that can pass optical inspection. In addition, the convolutional neural network mentioned in the present invention includes but not limited to Alexnet, VGG, etc.

In the following, the system structure of the present invention will be described by using the "Fig. 2" system structure diagram of the second detection system using graphic features based on the component identification data proposed by the present invention. As shown in "Fig. 2", the system of the present invention is applied in the secondary detection device 200, which includes a model building module 210, a data loading module 220, an image loading module 230, a feature acquisition module 250, components Detection module 260.

The model building module 210 is responsible for generating detection models using image feature recognition algorithms. The graphic feature recognition algorithms mentioned in the present invention include but are not limited to loss functions (or loss formulas) such as Triplet loss/Contrastive loss/Margin loss, Pairwise Ranking loss, etc. That is to say, the model building module 210 can call the above loss functions (or loss formula) to produce a detection model.

For example, the model building module 210 can obtain a certain number of standard images of an electronic component and sample images of the same electronic component that have passed optical inspection (referred to as “positive sample images” in the present invention), and can obtain the same Sample images of electronic components that have not passed optical inspection (referred to as "negative sample images" in the present invention), and then, the model building module 210 can use convolutional neural networks to calculate the standard image, positive sample images, and negative sample images Image features (such as directly using the convolutional neural network calculation or calling the feature acquisition module 250 to calculate), and can use the calculated image features of the standard image, the image features of the positive sample image and the image features of the negative sample image (in In the present invention, they are also referred to as "standard image features", "positive sample image features", and "negative sample image features". The sample image features are input to the Triplet loss function, so that the model building module 210 continuously adjusts the margin constant of the Triplet loss loss function) until the graphic feature recognition algorithm judges the Euclidean distance between the standard image feature and the positive sample image feature (Euclidean When the Euclidean distance between standard image features and negative sample image features is greater than the same threshold, the establishment of the detection model can be completed. Wherein, the above-mentioned threshold value is usually a value that can be determined during the above-mentioned training process.

The data loading module 220 is responsible for reading the component identification data of multiple electronic components on the target circuit board. The target circuit board mentioned in the present invention is the printed circuit board 110 detected by the mounter (surface mount device 130) and the optical detection device 150; component identification data and electronic components have a one-to-one correspondence, which can indicate the corresponding Electronic components can be composed of any number of characters, numbers, letters, and symbols arranged arbitrarily.

The data loading module 220 can be connected to the placement machine (surface mount device 130) that solders electronic components on the target circuit board, and the component information of all electronic components soldered on the target circuit board is downloaded by the placement machine; data The loading module 220 can also be connected to a specific relay device (not shown in the figure), such as a server, and the relay device downloads component information of all electronic components soldered on the target circuit board by the placement machine. Wherein, the component information includes component identification data and installation position data, and the installation position data mentioned in the present invention can indicate the position of the electronic component on the target circuit board.

The image loading module 230 is responsible for reading the component identification data of the target component and inspecting the image. In the present invention, the target component is any electronic component that fails the optical inspection among all the electronic components on the target circuit board.

The image loading module 230 can receive the test record output by the optical detection device 150 for automatic optical detection of the target circuit board, and can read the component identification data and detection image of the target component from the received test record; image The loading module 230 can also be connected to a relay device (not shown in the figure) that receives and stores the test records output by the optical detection device 150, and the relay device downloads the component identification data of the target components contained in the test records and detection images. Wherein, the test record output by the optical detection device 150 may include the test image of the target circuit board and the component identification data of the target components that have not passed the optical detection, wherein the test image is an image that covers the entire target circuit board output by the optical detection device . In addition, the test record can also include the test position information or the test image of the target component. The above test position information can indicate the position and size of the target component in the test image, for example, the coordinates of the diagonal corners of the target component in the test image, and For example, the coordinates and length and width of a specific vertex of the target component in the test image.

In some embodiments, if the test record does not include the inspection image, the image loading module 230 can also read out the component identification data and The test position information, and the test image of the target component can be extracted from the test image contained in the test record according to the test position information.

The image loading module 230 can also create a data table corresponding to the obtained component identification data and inspection images of the target components, so as to provide the feature acquisition module 250 for use, but the invention is not limited thereto. Wherein, each item of data in the data table created by the image loading module 230 includes a component identification data and a detection image of the corresponding electronic component.

The feature obtaining module 250 is responsible for obtaining standard image features of the target component according to the component identification data of the target component obtained by the image loading module 230 . For example, the feature acquisition module 250 can read the component identification data of the target component from the data table created by the image loading module 230, and can read and identify the component from the pre-established image data The standard image feature of the target component corresponding to the data.

In some embodiments, if the pre-established image data does not contain the standard image features of the target component, but only contains the standard image of the target component, the feature acquisition module 250 can also read the target component according to the component identification data of the target component standard image, and the convolutional neural network can be used to calculate the read standard image to generate standard image features of the target component.

The feature obtaining module 250 is also responsible for generating detection image features of the detection images obtained by the image loading module 230 . For example, the feature acquisition module 250 can read the detection image of the target component from the data table created by the image loading module 230, and can use the convolutional neural network to perform calculations on the read detection image to generate the target component detected image features.

The component detection module 260 is responsible for judging whether the target component passes the detection according to the detection image features acquired by the feature acquisition module 250 and the standard image features using the detection model established by the model building module 210 .

Next, an embodiment is used to explain the operating system and method of the present invention, and please refer to "Fig. 3A" for the flow chart of the method for secondary detection using graphic features based on component identification data proposed by the present invention. In this embodiment, it is assumed that the present invention is applied to the secondary detection device 200 . Wherein, the secondary detection device 200 is arranged in the production line, and the operation sequence of the secondary detection device 200 is arranged after the optical detection device 150 .

Firstly, before the secondary inspection device 200 starts to inspect the electronic components on the printed circuit board, the model building module 210 may first generate an inspection model by using a graphic feature recognition algorithm (step 310 ). In this embodiment, it is assumed that the developer can pre-select a certain number of standard images, positive sample images, and negative sample images of each electronic component, and can use the selected standard images, positive sample images, and negative sample images of each electronic component The image is provided to the model building module 210, so that the model building module 210 can obtain the standard image features of the standard image, the positive sample image features of the positive sample image, and the sample image features of the negative sample image through the feature acquisition module 250, and then, The model building module 210 can use the Triplet loss function to compare the Euclidean distance between the standard image feature and the positive sample image feature and the Euclidean distance between the standard image feature and the negative sample image feature in the feature space, and continuously adjust the margin constant, and so on. , until the Euclidean distance between the standard image feature and the positive sample image feature calculated by the Triplet loss loss function is less than the threshold value (assumed to be 1.1), and the Euclidean distance between the standard image feature and the negative sample image feature is greater than the threshold value, and the detection can be completed Model building.

After the model building module 210 generates the inspection model, the secondary inspection device 200 can be used in the production line. On the production line, the printed circuit board 110 will first arrive at the placement machine (surface mount device 130), and the placement machine will weld multiple electronic components on the printed circuit board 110 according to the preset installation information. After that, the data will be loaded The module 220 can obtain the component identification data of the electronic component soldered on the target circuit board by the placement machine (step 320 ). In this embodiment, it is assumed that the data loading module 220 can be connected to the placement machine, and can receive the component identification data of the electronic components soldered on the target circuit board by the placement machine.

After the printed circuit board 110 passes through the placement machine, the printed circuit board 110 will reach the optical detection device 150, and the optical detection device 150 can test the electronic components soldered on the printed circuit board 110 through automatic optical detection and generate corresponding Test Record.

After the printed circuit board 110 passes through the optical inspection device 150, if any electronic component on the printed circuit board 110 fails to pass the optical inspection performed by the optical inspection device 150, the test record generated by the optical inspection device 150 may include the printed circuit board The test image of 110 and the component identification data of the target components on the printed circuit board 110 that have not passed the optical inspection.

After the printed circuit board 110 passes through the optical inspection device 150 , the image loading module 230 can obtain inspection images of target components that fail the optical inspection from the test records output by the optical inspection device 150 (step 330 ). In this embodiment, assuming that the test record output by the optical detection device 150 already contains the test image of the target component, the image loading module 230 can read the component identification data and test image of the target component from the test record; and if The test record output by the optical detection device 150 does not contain the detection image of the target component, then the image loading module 230 can first read the test record output by the optical detection device 150 ( Step 331 ), and read out the component identification data and test position information of the target component from the test record, and extract the test image from the test image contained in the test record according to the read test position data (step 335 ). In addition, the image loading module 230 can also write the read component identification data and the obtained inspection image into a data table as a piece of data.

After the image loading module 230 acquires the inspection image of the target component, the feature acquisition module 250 can acquire the standard image feature of the target component according to the component identification data of the target component (step 350 ). In this embodiment, if the developer has pre-established a correspondence table between the component identification data and the standard image features of the target component, the feature acquisition module 250 can directly read the standard image features corresponding to the component identification data from the correspondence table; And if the developer only pre-stores the standard image of the target component, then the feature acquisition module 250 can, as shown in the flow of "Fig. 3C", combine the meta-identification data and After the component identification data of the target component is obtained from the data table of the standard image (steps 340 and 351), the standard image of the target component is first read according to the obtained component identification data (step 353), and the convolutional neural network is used to The read standard images are calculated to generate standard image features (step 355 ).

Similarly, after the image loading module 230 obtains the inspection image of the target component, the feature acquisition module 250 may also generate inspection image features of the inspection image obtained by the image loading module 230 (step 360 ). Similar to the above, the feature acquisition module 250 can use a convolutional neural network to perform calculations on the detection image to generate detection image features.

After the feature acquisition module 250 acquires standard image features and generates inspection image features, the component inspection module 260 can use the model building module 210 to generate according to the standard image features acquired by the feature acquisition module 250 and the generated inspection image features The inspection model judges whether the target component passes the inspection (step 370). In this embodiment, the detection model can calculate the Euclidean distance between the standard image feature and the detected image feature, and judge whether the target component passes the detection according to whether the Euclidean distance is smaller than a threshold value.

In this way, when the placement machine (surface mount device 130) welds the replacement electronic component on the printed circuit board 110 and the optical detection device 150 judges that the replacement electronic component has not passed the optical detection, the present invention can more accurately determine that it has failed. Whether the electronic components replaced by optical inspection pass the inspection.

In summary, it can be seen that the difference between the present invention and the prior art lies in the standard of obtaining the target components that have not passed the optical inspection based on the component identification data after obtaining the component identification data of the electronic components soldered on the target circuit board by the surface mount device. Image features, and generate the detection image features of the detection image of the target component, and use the detection model to judge whether the target component has passed the detection based on the detection image features and standard image features. The technical means can solve the problems of the previous technology. When the surface mount device uses alternative electronic components, the optical inspection device will not be able to effectively judge the soldering status of the alternative electronic components, thereby achieving the technical effect of improving the accuracy of optical inspection and reducing labor costs.

Moreover, the method of the present invention using graphic features for secondary detection based on component identification data can be implemented in hardware, software, or a combination of hardware and software, and can also be implemented in a centralized manner in a computer system or distributed with different components Implemented in a decentralized manner over several interconnected computer systems.

Although the embodiments disclosed in the present invention are as above, the content described is not intended to directly limit the scope of protection of the present invention. Anyone with ordinary knowledge in the technical field of the present invention, without departing from the spirit and scope disclosed in the present invention, makes some changes and modifications to the form and details of the implementation of the present invention, all of which belong to the patent protection of the present invention scope. The scope of patent protection of the present invention shall still be defined by the scope of the attached patent application.

110: Printed circuit board 130: Surface mount device 150: Optical detection device 200: Secondary detection device 210:Model building module 220: Data loading module 230: Image loading module 250: Feature acquisition module 260:Component detection module Step 310: Generate a detection model using a graphical feature recognition algorithm Step 320: Obtain the component identification data of the electronic components on the target circuit board Step 330: Obtain inspection images of target components that fail optical inspection Step 331: Read the test record output by the optical detection device, the test record includes the test image and the test position information of the target component Step 335: Extract the detection image of the target component from the test image according to the test position information of the target component Step 340: Create a data table corresponding to the component identification data of the target component and the detection image Step 350: Obtain the standard image feature of the target component according to the component identification data of the target component Step 351: Read out the component identification data of the target component from the data table Step 353: Read the standard image of the target component according to the component identification data of the target component Step 355: Computing the standard image using convolutional neural network to generate standard image features Step 360: Generate the detection image features of the detection image Step 370: Use the detection model to judge whether the target component passes the detection according to the detection image features and the standard image features

Figure 1 is a schematic diagram of a conventional production line. Figure 2 is a system architecture diagram of the present invention using graphic features for secondary detection based on component identification data. FIG. 3A is a flow chart of the method for secondary detection using graphic features based on component identification data proposed by the present invention. FIG. 3B is a flow chart of the method for obtaining detection images proposed by the present invention. FIG. 3C is a flow chart of the method for generating standard image features proposed by the present invention.

Step 310: Generate a detection model using a graphical feature recognition algorithm

Step 320: Obtain the component identification data of the electronic components on the target circuit board

Step 330: Obtain inspection images of target components that fail optical inspection

Step 350: Obtain the standard image feature of the target component according to the component identification data of the target component

Step 360: Generate the detection image features of the detection image

Step 370: Use the detection model to judge whether the target component passes the detection according to the detection image features and the standard image features

Claims (10)

A method for secondary inspection using graphic features based on component identification data is applied to a secondary inspection device, and the method at least includes the following steps: generating a detection model using a pattern feature recognition algorithm; Obtain component identification data of multiple electronic components on a target circuit board; Obtaining an inspection image of a target component, where the target component is any electronic component that has not passed optical inspection among the electronic components; Obtain a standard image feature of the target component based on the component identification data of the target component; generating a detected image feature of the detected image; and The detection model is used to judge whether the target component passes the detection according to the detection image feature and the standard image feature. The method for secondary inspection using graphic features based on component identification data as described in claim 1, wherein the step of obtaining the inspection image of the target component further includes the target contained in a test record output by an optical inspection device The test location information of the component is a step of extracting the test image from a test image included in the test record. The method for secondary detection using graphic features based on component identification data as described in claim 1, wherein the method further includes establishing component identification data corresponding to the target component and the detection image after the step of obtaining the detection image of the target component The step of a data table, and the step of obtaining the standard image feature of the target component according to the component identification data of the target component is to read the component identification data of the target component from the data table, and according to the components of the target component A pre-established standard image is read for the identification data, and convolution neural network is used to calculate the standard image to generate the standard image feature. The method for secondary detection using graphic features based on component identification data as described in claim 1, wherein the step of generating the detection image features of the detection image is to use a convolutional neural network to calculate the detection image to generate the detection image features . As described in claim 1, the method for secondary inspection using graphic features based on component identification data, wherein the step of using the graphic feature recognition algorithm to generate the inspection model is to obtain a positive sample image that passes optical inspection and one that fails optical inspection Negative sample image, and extract the image features of the positive sample image and the negative sample image, and use the standard image feature, the positive sample image feature and the negative sample image feature to train the graphic feature recognition algorithm until the standard The Euclidean distance between the image feature and the positive image feature is less than a threshold and the Euclidean distance between the standard image feature and the negative image feature is greater than the threshold, so as to generate the detection model. A system for secondary inspection using graphic features based on component identification data is applied to a secondary inspection device, and the system includes at least: a model building module for generating a detection model using a pattern feature recognition algorithm; A data loading module for obtaining component identification data of a plurality of electronic components on a target circuit board; An image loading module, used to obtain a detection image of a target component, the target component is any electronic component that has not passed optical inspection among the electronic components; A feature obtaining module, used to obtain a standard image feature of the target component according to the component identification data of the target component, and a detection image feature used to generate the detection image; and A component detection module is used for judging whether the target component passes the detection according to the detection image feature and the standard image feature using the detection model. The system for secondary inspection using graphic features based on component identification data as described in Claim 6, wherein the image loading module is further used for one of the target components contained in a test record output by an optical detection device The test location information is the test image extracted from a test image included in the test record. The system for secondary inspection using graphic features based on component identification data as described in claim 6, wherein the image loading module is further used to create a data table corresponding to the component identification data of the target component and the inspection image, and the The feature acquisition module is further used to read the component identification data of the target component from the data table, read a pre-established standard image according to the component identification data of the target component, and use the convolutional neural network to identify the standard Image calculations are performed to generate the standard image features. The system for secondary detection using graphic features based on component identification data as described in claim 6, wherein the feature acquisition module uses a convolutional neural network to calculate the detected image to generate the detected image feature. As described in claim 6, the system using graphic features for secondary inspection based on component identification data, wherein the model building module obtains a positive sample image that passes optical inspection and a negative sample image that fails optical inspection, and extracts the The image features of the positive sample image and the negative sample image, and use the standard image feature, the positive sample image feature and the negative sample image feature to train the graphic feature recognition algorithm until the standard image feature and the positive sample image The Euclidean distance between the features is less than a threshold and the Euclidean distance between the standard image feature and the negative sample image feature is greater than the threshold, so as to generate the detection model.

Images