WO2023115409A1

WO2023115409A1 - Pad detection method and apparatus, and computer device and storage medium

Info

Publication number: WO2023115409A1
Application number: PCT/CN2021/140593
Authority: WO
Inventors: 逯金辉; 沈小乐; 毛抒艺; 郭学胤
Original assignee: 深圳技术大学
Priority date: 2021-12-22
Filing date: 2021-12-22
Publication date: 2023-06-29

Abstract

A pad detection method and apparatus (200) applied to a flexible printed circuit (FPC), and a computer device (300) and a storage medium, which belong to the technical field of FPC detection in artificial intelligence. By means of accurately labelling pad position coordinates of each product, using a program to analyze pad position coordinates of each pad, then performing local calibration on the analyzed coordinates, and finally detecting an area, which is labeled on an FPC image, using a ResNet neural network model, the detection precision and speed can be greatly improved.

Description

A pad detection method, device, computer equipment and storage medium

technical field

The present application relates to the technical field of FPC detection in artificial intelligence, and in particular to a pad detection method, device, computer equipment and storage medium applied to flexible circuit boards.

Background technique

Flexible printed circuit (Flexible Printed Circuit, FPC) has been widely used for its light weight, thin thickness, free bending and folding and other excellent properties. However, the domestic quality inspection of FPC mainly relies on manual visual inspection, which is costly and inefficient. With the rapid development of electronic technology, the design of circuit boards tends to be more and more high-precision and high-density. The traditional manual detection method can no longer meet the production needs. The automatic detection of FPC defects has become an inevitable trend of development.

There is an existing pad detection method, which is based on computer vision and detection according to edge detection method, zero-mean method, background subtraction method, machine learning method and other methods, wherein:

1. Edge detection method: The edge detection method is a spatial domain detection method, which extracts the edge contour of the part through a specific algorithm, detects the edge of the part, and judges whether the part is defective by comparison;

2. Zero-mean method: The zero-mean method is to construct a zero-mean map after processing the image of the part, and to identify the damaged area by comparing and analyzing different thresholds during image segmentation;

3. Background subtraction method: Subtract the background image (excluding defects) estimated or calculated in advance from the preprocessed image, leaving a residual image containing defects and random noise;

4. Machine learning method: Based on manually defined features and feature extraction methods, these features are used to train machine learning classifiers to achieve final defect detection.

However, the applicant found that the traditional pad detection method is generally not intelligent, and the traditional method can only detect defects under certain conditions, such as a certain size, or an obvious defect outline with strong contrast and low noise under specific lighting conditions. Algorithms based on machine learning have certain robustness, but the natural disadvantages of artificial features are weak representation ability and poor adaptability. Therefore, based on traditional machine learning algorithms, the features that need to be detected cannot be well learned, resulting in the accuracy of the final detection results. lower. It can be seen that the traditional pad detection method has the problem of low accuracy.

technical problem

The purpose of the embodiments of the present application is to provide a pad detection method, device, computer equipment and storage medium applied to flexible circuit boards, so as to solve the problem of low accuracy of traditional pad detection methods.

technical solution

In order to solve the above technical problems, the embodiment of the present application provides a pad detection method applied to flexible circuit boards, which adopts the following technical solutions:

Accept model training requests that carry circuit board sample images and pad location information;

performing a segmentation operation on the circuit board sample image according to the pad position information to obtain a pad sample image;

Sending the pad sample image to a terminal device for labeling the pad sample image;

receiving the pad annotation image sent by the terminal device, the pad annotation image is a pad sample image carrying annotation information, and the annotation information is good product information or defective product information;

Call the original ResNet neural network model, and use the pad sample image as training data and the label information as result data to perform model training operations on the original ResNet neural network model to obtain the target ResNet for identifying the quality of the pad image. neural network model;

Accept the pad detection request carrying the image of the circuit board to be tested;

Inputting the image of the circuit board to be tested into the target ResNet neural network model to perform a pad detection operation to obtain a pad detection result;

Outputting the detection result of the pad.

Further, after the step of segmenting the circuit board sample image according to the pad position information to obtain the pad sample image, the following steps are further included:

The pad sample image is converted from the RGB color space to the HSV color space according to the conversion formula, and the conversion formula is expressed as:

Among them, max=max(r,g,b), min=min(r,g,b).

Further, when calling the original ResNet neural network model, and using the pad sample image as training data and the label information as result data, the original ResNet neural network model is trained to obtain a Before the step of the target ResNet neural network model of disk image quality, the following steps are also included:

A pre-classification operation is performed on the pad sample images according to the feature method to obtain multiple groups of pad sample images with similar pad shapes.

Further, the step of inputting the image of the circuit board to be tested into the target ResNet neural network model to perform a pad detection operation to obtain a pad detection result specifically includes the following steps:

Counting the change area of the B channel value in the RGB color space of each pad position to be tested in the image of the circuit board to be tested;

The image of the circuit board to be tested is detected according to the preset good product ratio threshold S1, the defective product ratio threshold S2 and a judgment function, and the judgment function is expressed as:

Wherein, the _PI represents the ratio of the change area to the area of the pad position to be tested; f(ROI _i ) represents that the pad detection operation is performed on the circuit board image to be tested by the target ResNet neural network model The results obtained.

In order to solve the above technical problems, the embodiment of the present application also provides a pad detection device applied to flexible circuit boards, which adopts the following technical solutions:

A training request receiving module, configured to receive a model training request carrying circuit board sample images and pad position information;

A segmentation operation module, configured to perform a segmentation operation on the circuit board sample image according to the pad position information to obtain a pad sample image;

An annotation sending module, configured to send the pad sample image to a terminal device, so as to perform an annotation operation on the pad sample image;

An annotation receiving module, configured to receive the pad annotation image sent by the terminal device, the pad annotation image is a pad sample image carrying annotation information, and the annotation information is good product information or defective product information;

The model training module is used to call the original ResNet neural network model, and use the pad sample image as training data and the label information as result data to perform model training operations on the original ResNet neural network model to obtain a Targeted ResNet neural network model for disk image quality;

A detection request receiving module, configured to accept a pad detection request carrying an image of a circuit board to be tested;

The pad detection module is used to input the image of the circuit board to be tested into the target ResNet neural network model to perform a pad detection operation to obtain a pad detection result;

A result output module, configured to output the detection result of the pad.

Further, the device also includes:

The color space conversion module is used to convert the pad sample image from the RGB color space to the HSV color space according to the conversion formula, and the conversion formula is expressed as:

Among them, max=max(r,g,b), min=min(r,g,b).

Further, the device also includes:

The pre-classification module is configured to perform a pre-classification operation on the pad sample images according to the feature method to obtain multiple groups of pad sample images with similar pad shapes.

Further, the pad detection module includes:

The change area acquisition submodule is used to count the change area of the B channel value in the RGB color space of each pad position to be tested in the image of the circuit board to be tested;

The detection sub-module is used to detect the image of the circuit board to be tested according to the preset good product ratio threshold S1, the defective product ratio threshold S2 and a judgment function, and the judgment function is expressed as:

In order to solve the above technical problems, the embodiment of the present application also provides a computer device, which adopts the following technical solutions:

It includes a memory and a processor, wherein computer-readable instructions are stored in the memory, and the processor implements the steps of the above-mentioned method for detecting pads applied to flexible circuit boards when executing the computer-readable instructions.

In order to solve the above technical problems, the embodiment of the present application also provides a computer-readable storage medium, which adopts the following technical solution:

Computer-readable instructions are stored on the computer-readable storage medium, and when the computer-readable instructions are executed by a processor, the steps of the above-mentioned method for detecting a pad applied to a flexible circuit board are implemented.

Beneficial effect

Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects:

The present application provides a pad detection method applied to a flexible circuit board, including: accepting a model training request carrying a circuit board sample image and pad position information; performing a segmentation operation to obtain a pad sample image; sending the pad sample image to a terminal device for labeling the pad sample image; receiving the pad label image sent by the terminal device, and the pad label image It is a pad sample image carrying annotation information, the annotation information is good product information or defective product information; the original ResNet neural network model is called, and the pad sample image is used as training data, and the annotation information is the result data pair The original ResNet neural network model performs a model training operation to obtain a target ResNet neural network model for identifying the quality of the pad image; accepts a pad detection request carrying an image of the circuit board to be tested; inputs the image of the circuit board to be tested Perform pad detection operation to the target ResNet neural network model to obtain a pad detection result; output the pad detection result. This application accurately marks the pad position coordinates of each product, uses the program to analyze the pad position coordinates of each pad, and then performs local calibration on the parsed coordinates, and finally uses the ResNet neural network model to detect the flexible circuit board (FPC ) area marked on the image can greatly improve the detection accuracy and speed.

Description of drawings

In order to illustrate the solution in this application more clearly, a brief introduction will be given below to the accompanying drawings that need to be used in the description of the embodiments of the application. Obviously, the accompanying drawings in the following description are some embodiments of the application. Ordinary technicians can also obtain other drawings based on these drawings on the premise of not paying creative work.

FIG. 1 is an exemplary system architecture diagram to which the present application can be applied;

FIG. 2 is a flow chart of the implementation of the pad detection method applied to flexible circuit boards provided in Embodiment 1 of the present application;

Fig. 3 is a schematic diagram of Mark points provided by Embodiment 1 of the present application;

Fig. 4 is a schematic diagram of dividing the ring into four parts provided by Embodiment 1 of the present application;

Fig. 5 is a schematic structural diagram of a pad detection device applied to a flexible circuit board provided in Embodiment 2 of the present application:

Fig. 6 is a schematic structural diagram of an embodiment of a computer device according to the present application.

Embodiments of the present invention

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field of the application; the terms used herein in the description of the application are only to describe specific embodiments The purpose is not to limit the present application; the terms "comprising" and "having" and any variations thereof in the specification and claims of the present application and the description of the above drawings are intended to cover non-exclusive inclusion. The terms "first", "second" and the like in the description and claims of the present application or the above drawings are used to distinguish different objects, rather than to describe a specific order.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The occurrences of this phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is understood explicitly and implicitly by those skilled in the art that the embodiments described herein can be combined with other embodiments.

In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

As shown in FIG. 1 , a system architecture 100 may include

terminal devices

101 , 102 , 103 , a network 104 and a server 105 . The network 104 is used as a medium for providing communication links between the

terminal devices

101 , 102 , 103 and the server 105 . Network 104 may include various connection types, such as wires, wireless communication links, or fiber optic cables, among others.

Users can use

terminal devices

101 , 102 , 103 to interact with server 105 via network 104 to receive or send messages and the like. Various communication client applications can be installed on the

terminal devices

101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, and the like.

Terminal devices

101, 102, 103 can be various electronic devices with display screens and support web browsing, including but not limited to smartphones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, dynamic Video experts compress standard audio layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, moving picture experts compress standard audio layer 4) player, laptop portable computer and desktop computer, etc.

The server 105 may be a server that provides various services, such as a background server that provides support for pages displayed on the

terminal devices

101 , 102 , 103 .

It should be noted that the pad detection method applied to flexible circuit boards provided in the embodiments of the present application is generally executed by the server/terminal device, and correspondingly, the pad detection device applied to flexible circuit boards is generally set on the server/terminal device middle.

It should be understood that the numbers of terminal devices, networks and servers in Fig. 1 are only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.

Continuing to refer to FIG. 2 , it shows the implementation flow chart of the pad detection method applied to the flexible circuit board provided by Embodiment 1 of the present application. For the convenience of description, only the parts relevant to the present application are shown.

The above-mentioned pad detection method applied to a flexible circuit board includes the following steps:

In step S201, a model training request carrying a circuit board sample image and pad position information is accepted.

In this embodiment of the application, before receiving the model training request, it is necessary to manually mark the template image with a polygon frame to obtain the position of the pad, that is, the region of interest ROI to be detected, and store the coordinate position in the Json file.

In step S202, the circuit board sample image is segmented according to the pad position information to obtain a pad sample image.

In the embodiment of the present application, when making the pad sample image, the minimum bounding box algorithm is used to frame the ROI area, and the ROI images of 1000 images are cut out.

In the embodiment of this application, the foreground layouts of different products are different. When replacing a batch of products, first use manual labeling to mark all the foreground area positions to be detected on the sample image with the smallest rectangular bounding box to obtain C _i (x, y, w, h)∈C, where x, y are the horizontal and vertical coordinates of the upper left corner of the smallest bounding box, and w, h are the width and height of the smallest bounding box.

In some optional implementations of this embodiment, the pad area to be detected is usually irregular in shape, and there is still a part of the background area in the minimum bounding box. According to the brightness difference between the pad area and the background, the image is converted from RGB color to The space is converted to the HSV color space, and the conversion formula is

Where max=max(r,g,b), min=min(r,g,b), after converting to HSV space, use the Otsu method to automatically threshold the H channel, and set the value of the background area to zero.

In step S203, the pad sample image is sent to the terminal device for marking the pad sample image.

In the embodiment of the present application, the cut ROI image is manually marked as a good product or a defective product.

In the embodiment of the present application, in order to provide the data sets required for subsequent neural network model training, the existing data sets need to be divided into defective data sets and non-defective data sets. Since the amount of data is not very large, the data can be manually screened, or the data set can be preliminarily screened by extracting obvious defect features and then manually screened, which can reduce the workload. This embodiment detects the discoloration defect of the pad, and judging whether a pad contains a defect needs to judge whether the discoloration area exceeds the allowable ratio, whether the degree of discoloration exceeds the allowable degree, and if a rainbow-like discoloration occurs, that is, a sudden color change, no matter the size of the discoloration area. judged to be defective. Because the color of the non-defective pads collected by the image acquisition system is white and bright, and the color of the defective area is reddish or yellowish, it can be seen from the RGB color space that the change from bright white to red or yellow is mainly related to the decrease in the pixel value of the blue channel. According to this color feature, the data set is initially screened, and then manually screened to confirm that it is divided into a defective data set and a non-defective data set.

Considering that the light intensity may change slightly when the actual image acquisition system acquires the circuit board image, this embodiment strengthens the data set. According to the conversion formula between RGB color space and HSV color space, it can be known that only the value of max(R,G,B) can be modified to simulate the change of light intensity. According to this method, max(R,G,B )=max(R,G,B)±d operation to get the enhanced data set

In step S204 , receiving the pad annotation image sent by the terminal device, the pad annotation image is a pad sample image carrying annotation information, and the annotation information is good product information or defective product information.

In step S205, the original ResNet neural network model is called, and the model training operation is performed on the original ResNet neural network model with the pad sample image as the training data and the label information as the result data, and the target ResNet neural network model for identifying the quality of the pad image is obtained. network model.

In the embodiment of this application, ResNet is a network structure proposed by He et al. in the 2015 ImageNet Challenge, which alleviates the problem that the performance of the network degrades with the deepening of the network depth. The experiment in the paper Deep Residual Learning for Image Recognition found that the deep network has a degradation problem (Degradation problem): when the network depth increases, the network accuracy is saturated or even declines. This phenomenon can be seen directly in the figure: the 56-layer network is even worse than the 20-layer network. This would not be an overfitting problem, since the training error is equally high for a 56-layer network. Because the deep network has the problem of gradient disappearance or explosion, which makes the deep learning model difficult to train.

In the embodiment of this application, for a stacked layer structure (several layers stacked) when the input is x, the learned features are denoted as H(x), and now we hope that it can learn the residual F(x)= H(x)-x, so the original learning feature is F(x)+x. Residual learning is easier than direct learning of original features, because when the residual is 0, the accumulation layer only does the identity mapping at this time, at least the network performance will not decline, in fact the residual will not be 0, which will also This enables the stacking layer to learn new features based on the input features, resulting in better performance.

In the embodiment of this application, the ResNet50 network is used in the existing deep learning package of "Halcon", which shows that it has certain reliability in industrial inspection. Therefore, this paper uses the ResNet network to detect pad defects combined with inter-channel information. The network structure consists of convolution, pooling, improved residual block and linear classifier. The input features undergo convolution, batch regularization, pooling, and then repeat the above process, and finally the output and input features are superimposed pixel by pixel as the next input; after convolution, the importance of each channel is extracted, so that It is multiplied by the feature layer after convolution, and finally added to the input feature, so that the convolution can obtain the fusion information of the space and channel of the local area.

In the embodiment of the present application, images of good products and defective products are divided into a training set and a verification set according to a ratio of 3:1. Input the training set samples to the network, use the BP algorithm to update the model weights, input the verification set to the network, verify the effect of the model, and repeat this operation until the model converges to a satisfactory accuracy rate.

In the embodiment of this application, for the FPC image to be detected, the B channel value distribution histogram of the RGB color space of each ROI is counted. For a single ROI, if the ratio P of the discoloration area to the pad area is greater than the threshold S2, the FPC is not Good product; if S1<P≤S2, use the trained model f(x) to detect the ROI, if the detection result is a defective product, then the FPC is a defective product, and if the detection result is a good product, then detect the next ROI; If P≤S1, detect the next ROI; if all ROIs are good, the FPC is good, otherwise it is bad. The judgment result of the current FPC is obtained through the judgment function R(X), when R(X)=0. The current FPC is a good product, otherwise it is a defective product. The formula is shown in formula (1):

Among them, X is the current FPC image, and n is the number of ROIs of the FPC

In step S206, the pad detection request carrying the image of the circuit board to be tested is accepted.

In the embodiment of the present application, for the image acquisition of the FPC, an area array industrial camera is used to shoot above the FPC board, and the image file is transmitted to the image acquisition card of the computer to save the image file.

In step S207, the image of the circuit board to be tested is input to the target ResNet neural network model to perform a pad detection operation, and a pad detection result is obtained.

In step S208, the detection result of the pad is output.

In the embodiment of the present application, a pad detection method applied to a flexible circuit board is provided, including: accepting a model training request carrying a circuit board sample image and pad position information; The image is segmented to obtain the pad sample image; the pad sample image is sent to the terminal device for labeling of the pad sample image; the pad label image sent by the terminal device is received, and the pad label image carries label information pad sample image, the label information is good product information or defective product information; the original ResNet neural network model is called, and the model training operation is performed on the original ResNet neural network model with the pad sample image as the training data and the label information as the result data. The target ResNet neural network model used to identify the quality of the pad image; accept the pad detection request carrying the image of the circuit board to be tested; input the image of the circuit board to be tested into the target ResNet neural network model for the pad detection operation, and obtain the pad Detection result; output pad detection result. This application accurately marks the pad position coordinates of each product, uses the program to analyze the pad position coordinates of each pad, and then performs local calibration on the parsed coordinates, and finally uses the ResNet neural network model to detect the flexible circuit board (FPC ) area marked on the image can greatly improve the detection accuracy and speed.

In some optional implementation manners of this embodiment, before step S205, further include:

The pad sample images are pre-classified according to the feature method, and multiple groups of pad sample images with similar pad shapes are obtained.

In the embodiment of this application, for the sake of generality, that is, we only detect the pads, regardless of the shape of the product picture, a JSON file should be prepared for each picture, which marks the outline coordinates of the pads to be tested. Because each picture is taken by the camera moving across the entire board, there will be a certain deviation from the coordinates marked in the JSON file, so registration is required.

After registration, cut out the pad picture of the current picture as a test set.

According to the project background, we know that although the shape of each FPC pad is different, what remains unchanged is that each FPC board has a prototype pad, so it can be registered through the prototype pad.

Registration steps:

In order to save costs and other considerations, multiple circuit boards will be made up in the production of circuit boards, and when the spliced boards are passed to the image acquisition system described in step 2, the image of each circuit board will be collected according to the method of moving the camera or moving the spliced boards. Therefore, the position coordinates contained in the corresponding JSON file will have a relative offset relative to the position of the pad on the circuit board image. In the circuit board design, Mark points are set to help the optical positioning of the placement machine. This method also uses this point to register the JSON file and the circuit board image.

Mark points are composed of mark points and open areas, as shown in Figure 3.

This method proposes a ring-based solid circle matching algorithm. The algorithm first generates a circular template based on the coordinates of the Mark point marking area in the JSON file. The inner diameter of the ring is 1 pixel smaller than the edge diameter of the Mark marking area, and the outer diameter of the ring is larger than the Mark edge. 1 pixel in diameter. Divide the template into 4 arcs at 45 degrees and 135 degrees, as shown in Figure 4.

Note that the sets of pixel values on the upper, lower, left, and right sides of the inner and outer arcs are Ui, Uo, Di, Do, Li, Lo, Ri, Ro, respectively. The color characteristics of the marked area and the open area of the Mark point are different, and the recognition threshold δ is set. According to the formula ( 2) (3) Calculate the offset distance between the X-axis and the Y-axis, update the template coordinates to completely move the inner ring of the template to the inner side of the Mark point marking area, and the outer ring of the template is outside the Mark point marking area.

Where x _offset represents the horizontal offset distance of the template relative to the current circuit board image, and y _offset represents the vertical offset distance of the template relative to the current circuit board image.

After registration, first set the pixel value of the non-to-be-detected area of the circuit board image to 0 according to the pad coordinates in the JSON file, and then calculate the minimum rectangular bounding box of each pad according to formula 4,

Wherein (x ₀ , y ₀ ) is the upper left corner point of the bounding box, (x ₁ , y ₁ ) is the lower right corner point of the bounding box, X, Y are the horizontal axis coordinate set and the vertical axis coordinate set of the pad. On the circuit board image, the minimum bounding box is determined according to the coordinates of the above two points to cut out the pad image and save it to the dataset, thereby generating a single pad image dataset with a black background, that is, a pixel value of 0 in the non-pad area.

In the embodiment of this application, after generating a single pad image data set with a black background, that is, a pixel value of 0 in the non-pad area, in order to provide the data set required for subsequent neural network model training, the existing data set needs to be divided into Defective and non-defective datasets. Since the amount of data is not very large, the data can be manually screened, or the data set can be preliminarily screened by extracting obvious defect features and then manually screened, which can reduce the workload. This embodiment detects the discoloration defect of the pad, and judging whether a pad contains a defect needs to judge whether the discoloration area exceeds the allowable ratio, whether the degree of discoloration exceeds the allowable degree, and if a rainbow-like discoloration occurs, that is, a sudden color change, no matter the size of the discoloration area. judged to be defective. Because the color of the non-defective pads collected by the image acquisition system is white and bright, and the color of the defective area is reddish or yellowish, it can be seen from the RGB color space that the change from bright white to red or yellow is mainly related to the decrease in the pixel value of the blue channel. According to this color feature, the data set is initially screened, and then manually screened to confirm that it is divided into a defective data set and a non-defective data set.

Considering that the light intensity may change slightly when the actual image acquisition system acquires the circuit board image, this embodiment strengthens the data set. According to the conversion formula between RGB color space and HSV color space, it can be known that only the value of max(R,G,B) can be modified to simulate the change of light intensity. According to this method, max(R,G,B )=max(R,G,B)±d operation to get the enhanced data set.

In some optional implementation manners of this embodiment, considering that the light intensity may slightly change when the actual image acquisition system acquires the circuit board image, this embodiment strengthens the data set. According to the conversion formula between RGB color space and HSV color space, it can be known that only the value of max(R,G,B) can be modified to simulate the change of light intensity. According to this method, max(R,G,B )=max(R,G,B)±d operation to get the enhanced data set.

To sum up, this application provides a pad detection method applied to flexible circuit boards, including: accepting a model training request carrying a circuit board sample image and pad position information; The circuit board sample image is segmented to obtain a pad sample image; the pad sample image is sent to the terminal device for labeling the pad sample image; the pad label image sent by the terminal device is received, and the The pad annotation image is a pad sample image carrying annotation information, and the annotation information is good product information or defective product information; the original ResNet neural network model is called, and the pad sample image is used as training data, and the annotation The information is the result data, and the model training operation is performed on the original ResNet neural network model to obtain the target ResNet neural network model used to identify the quality of the pad image; accept the pad detection request carrying the image of the circuit board to be tested; The test circuit board image is input to the target ResNet neural network model to perform a pad detection operation to obtain a pad detection result; and the pad detection result is output. This application aims at the FPC boards of different sizes provided by manufacturers that contain multiple irregularly shaped pads, and studies the design and system implementation of FPC pad defect detection algorithms based on neural networks, while meeting the basic detection requirements and improving Efficiency of defect detection. The detection method of visual inspection is easily affected by subjective and objective factors, the correctness of the results cannot be guaranteed, and the inspection process is very time-consuming. This system can solve these problems well, meets our expectations for defect detection, and has both It has the advantages of versatility and stability.

The content and results of the experiment are as follows:

(1) Register the sample image and the template image based on the coordinates of the pads marked in the JSON file. In view of the large size of the entire FPC image, the traditional template matching algorithm cannot match the rotated and translated image well. First, use the JSON file to mark the pads, and perform image registration for the circular pads, which improves the accuracy and speed of matching, and then proposes to do local template matching to find the feature points, and then do affine according to the feature points The transformation method realizes the registration of the sample image and the template image.

(2) Study the relevant image processing algorithm, and design the FPC defect extraction algorithm flow and classification method based on neural network for pad defects. Due to the difference in light and size between the sample image and the template image, as well as the noise generated in the process of image acquisition, transmission, imaging, etc., the error result will include the difference between the defect and the non-defect part. To solve this problem, this paper uses the modified Image brightness enhancement data set, first use the data set to train the network model, and then segment out the pads after registering each picture, and finally use the trained resnet network model to detect and separate the pads to realize the defect classification algorithm , and finally realize defect detection and classification.

(3) Based on the existing mechanical structure and the overall requirements of the FPC defect detection system, the process flow and overall scheme are designed, and the joint debugging and communication of the hardware part and the software part are completed, and the image acquisition is completed.

(4) Finally, according to the requirements analysis and the research objectives of the thesis, it focuses on the composition of the entire defect detection system, the systematic process flow, and the defect detection process. It integrates the visual system of FPC pad detection and classification, and traceable detection data. After experiments, the accuracy and efficiency of the software are guaranteed.

The application can be used in numerous general purpose or special purpose computer system environments or configurations. Examples: personal computers, server computers, handheld or portable devices, tablet-type devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, including A distributed computing environment for any of the above systems or devices, etc. This application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including storage devices.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. , when the computer-readable instructions are executed, they may include the processes of the embodiments of the above-mentioned methods. Wherein, the aforementioned storage medium may be a nonvolatile storage medium such as a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM).

It should be understood that although the various steps in the flow chart of the accompanying drawings are displayed sequentially according to the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages may not necessarily be executed at the same time, but may be executed at different times, and the order of execution is also It is not necessarily performed sequentially, but may be performed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

Embodiment two

Further referring to FIG. 5 , as an implementation of the method shown in FIG. 2 above, the present application provides an embodiment of a pad detection device applied to a flexible circuit board. This device embodiment is the same as the method embodiment shown in FIG. 2 Correspondingly, the device can be specifically applied to various electronic devices.

As shown in FIG. 5 , the pad detection device 200 applied to a flexible circuit board described in this embodiment includes: a training request receiving module 210, a segmentation operation module 220, a label sending module 230, a label receiving module 240, and a model training module 250 , a detection request receiving module 260 , a pad detection module 270 and a result output module 280 . in:

The training request receiving module 210 is configured to receive a model training request carrying circuit board sample images and pad position information;

A segmentation operation module 220, configured to perform a segmentation operation on the circuit board sample image according to the pad position information to obtain a pad sample image;

An annotation sending module 230, configured to send the pad sample image to a terminal device, so as to perform an annotation operation on the pad sample image;

Annotation receiving module 240, configured to receive the pad annotation image sent by the terminal device, the pad annotation image is a pad sample image carrying annotation information, and the annotation information is good product information or defective product information;

The model training module 250 is used to call the original ResNet neural network model, and use the pad sample image as training data and the labeling information as result data to perform model training operations on the original ResNet neural network model to obtain Target ResNet neural network model for pad image quality;

The detection request receiving module 260 is configured to accept a pad detection request carrying an image of the circuit board to be tested;

The pad detection module 270 is configured to input the image of the circuit board to be tested into the target ResNet neural network model to perform a pad detection operation to obtain a pad detection result;

The result output module 280 is configured to output the detection result of the pad.

V=max

Among them, X is the current FPC image, and n is the number of ROIs of the FPC

In the embodiment of the present application, for the image acquisition of the FPC, an area array industrial camera is used to shoot above the FPC board, and the image acquisition card transmitted to the computer saves the image file.

In the embodiment of the present application, a pad detection device 200 applied to a flexible circuit board is provided, including: a training request receiving module 210, configured to receive a model training request carrying a circuit board sample image and pad position information; The segmentation operation module 220 is configured to segment the circuit board sample image according to the pad position information to obtain a pad sample image; the label sending module 230 is configured to send the pad sample image to a terminal device, to perform labeling operations on pad sample images; the label receiving module 240 is configured to receive the pad label image sent by the terminal device, the pad label image is a bond sample image carrying label information, and the label information It is good product information or defective product information; the model training module 250 is used to call the original ResNet neural network model, and use the pad sample image as training data and the label information as result data to perform the original ResNet neural network model The model training operation obtains the target ResNet neural network model for identifying the image quality of the pad; the detection request receiving module 260 is used to accept the pad detection request carrying the image of the circuit board to be tested; the pad detection module 270 is used to The image of the circuit board to be tested is input to the target ResNet neural network model to perform a pad detection operation to obtain a pad detection result; the result output module 280 is configured to output the pad detection result. This application accurately marks the pad position coordinates of each product, uses the program to analyze the pad position coordinates of each pad, and then performs local calibration on the parsed coordinates, and finally uses the ResNet neural network model to detect the flexible circuit board (FPC ) area marked on the image can greatly improve the detection accuracy and speed.

In some optional implementations of this embodiment, the above-mentioned pad detection device 200 applied to a flexible circuit board further includes:

Registration steps:

Mark points are composed of mark points and open areas, as shown in Figure 3.

Where (x ₀ , y ₀ ) is the upper left corner point of the bounding box, (x ₁ , y ₁ ) is the lower right corner point of the bounding box, and X, Y are the horizontal axis coordinate set and the vertical axis coordinate set of the pad. On the circuit board image, the minimum bounding box is determined according to the coordinates of the above two points to cut out the pad image and save it to the dataset, thereby generating a single pad image dataset with a black background, that is, a pixel value of 0 in the non-pad area.

In summary, the present application provides a pad detection device 200 applied to flexible circuit boards, including: a training request receiving module 210, configured to receive a model training request carrying circuit board sample images and pad position information; The segmentation operation module 220 is configured to segment the circuit board sample image according to the pad position information to obtain a pad sample image; the label sending module 230 is configured to send the pad sample image to a terminal device, to perform labeling operations on pad sample images; the label receiving module 240 is configured to receive the pad label image sent by the terminal device, the pad label image is a bond sample image carrying label information, and the label information It is good product information or defective product information; the model training module 250 is used to call the original ResNet neural network model, and use the pad sample image as training data and the label information as result data to perform the original ResNet neural network model The model training operation obtains the target ResNet neural network model for identifying the image quality of the pad; the detection request receiving module 260 is used to accept the pad detection request carrying the image of the circuit board to be tested; the pad detection module 270 is used to The image of the circuit board to be tested is input to the target ResNet neural network model to perform a pad detection operation to obtain a pad detection result; the result output module 280 is configured to output the pad detection result. This application aims at the FPC boards of different sizes provided by manufacturers that contain multiple irregularly shaped pads, and studies the design and system implementation of FPC pad defect detection algorithms based on neural networks, while meeting the basic detection requirements and improving Efficiency of defect detection. The detection method of visual inspection is easily affected by subjective and objective factors, the correctness of the results cannot be guaranteed, and the inspection process is very time-consuming. This system can solve these problems well, meets our expectations for defect detection, and has both It has the advantages of versatility and stability.

The content and results of the experiment are as follows:

In order to solve the above technical problems, the embodiment of the present application further provides computer equipment. Please refer to FIG. 6 for details. FIG. 6 is a block diagram of the basic structure of the computer device in this embodiment.

The computer device 300 includes a memory 310 , a processor 320 , and a network interface 330 connected to each other through a system bus for communication. It should be noted that only computer device 300 is shown with components 310-330, but it should be understood that implementation of all illustrated components is not required and that more or fewer components may instead be implemented. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculation and/or information processing according to preset or stored instructions, and its hardware includes but is not limited to microprocessors, dedicated Integrated circuit (Application Specific Integrated Circuit, ASIC), programmable gate array (Field-Programmable Gate Array, FPGA), digital processor (Digital Signal Processor, DSP), embedded devices, etc.

The computer equipment may be computing equipment such as a desktop computer, a notebook, a palmtop computer, and a cloud server. The computer device can perform human-computer interaction with the user through keyboard, mouse, remote controller, touch panel or voice control device.

The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static Random Access Memory (SRAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Programmable Read Only Memory (PROM), Magnetic Memory, Magnetic Disk, Optical Disk, etc. In some embodiments, the storage 310 may be an internal storage unit of the computer device 300 , such as a hard disk or memory of the computer device 300 . In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, flash memory card (Flash Card), etc. Certainly, the memory 310 may also include both an internal storage unit of the computer device 300 and an external storage device thereof. In this embodiment, the memory 310 is generally used to store the operating system and various application software installed in the computer device 300 , such as computer-readable instructions applied to a pad detection method of a flexible circuit board. In addition, the memory 310 can also be used to temporarily store various types of data that have been output or will be output.

The processor 320 may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chips in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300 . In this embodiment, the processor 320 is configured to execute computer-readable instructions stored in the memory 310 or process data, for example, execute computer-readable instructions of the method for detecting pads applied to flexible printed circuit boards.

The network interface 330 may include a wireless network interface or a wired network interface, and the network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

The computer equipment provided by this application accurately marks the position coordinates of the pads of each product, uses the program to analyze the coordinates of the pad positions of each type of pads, and then performs local calibration on the resolved coordinates, and finally uses the ResNet neural network model to detect The area marked on the flexible circuit board (FPC) image can greatly improve the detection accuracy and speed.

The present application also provides another implementation manner, which is to provide a computer-readable storage medium, the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to The at least one processor is made to execute the steps of the method for detecting pads applied to flexible printed circuit boards as described above.

The computer-readable storage medium provided by this application accurately marks the pad position coordinates of each product, uses the program to analyze the pad position coordinates of each type of pad, and then performs local calibration on the parsed coordinates, and finally uses the ResNet neural network The network model detects the marked area on the flexible circuit board (FPC) image, which can greatly improve the detection accuracy and speed.

Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product in essence or the part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM, disk, CD) contains several instructions to make a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in the various embodiments of the present application.

Apparently, the embodiments described above are only some of the embodiments of the present application, but not all of them. The drawings show preferred embodiments of the present application, but do not limit the patent scope of the present application. The present application can be implemented in many different forms, on the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure content of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or perform equivalent replacements for some of the technical features . All equivalent structures made using the contents of the description and drawings of this application, directly or indirectly used in other related technical fields, are also within the scope of protection of this application.

Claims

A method for detecting pads applied to flexible circuit boards, comprising the steps of:

Accept model training requests that carry circuit board sample images and pad location information;

performing a segmentation operation on the circuit board sample image according to the pad position information to obtain a pad sample image;

Sending the pad sample image to a terminal device for labeling the pad sample image;

receiving the pad annotation image sent by the terminal device, the pad annotation image is a pad sample image carrying annotation information, and the annotation information is good product information or defective product information;

Call the original ResNet neural network model, and use the pad sample image as training data and the label information as result data to perform model training operations on the original ResNet neural network model to obtain the target ResNet for identifying the quality of the pad image. neural network model;

Accept the pad detection request carrying the image of the circuit board to be tested;

Inputting the image of the circuit board to be tested into the target ResNet neural network model to perform a pad detection operation to obtain a pad detection result;

Outputting the detection result of the pad.
The pad detection method applied to flexible printed circuit boards according to claim 1, characterized in that, in the step of segmenting the sample image of the circuit board according to the position information of the pads to obtain the sample image of the pads Afterwards, the following steps are also included:

The pad sample image is converted from the RGB color space to the HSV color space according to the conversion formula, and the conversion formula is expressed as:

V=max

Among them, max=max(r,g,b), min=min(r,g,b).
The pad detection method applied to flexible printed circuit boards according to claim 1, wherein the original ResNet neural network model is called, and the pad sample image is used as training data and the label information is the result Carrying out the model training operation on the data to the original ResNet neural network model, before obtaining the step of the target ResNet neural network model for identifying the quality of the pad image, the following steps are also included:

A pre-classification operation is performed on the pad sample images according to the feature method to obtain multiple groups of pad sample images with similar pad shapes.
The method for detecting pads applied to flexible printed circuit boards according to claim 1, wherein the input of the image of the circuit board to be tested to the target ResNet neural network model is performed for pad detection operations to obtain pads The steps for detecting the results specifically include the following steps:

Counting the change area of the B channel value in the RGB color space of each pad position to be tested in the image of the circuit board to be tested;

The image of the circuit board to be tested is detected according to the preset good product ratio threshold S1, the defective product ratio threshold S2 and a judgment function, and the judgment function is expressed as:

Wherein, the PI represents the ratio of the change area to the area of the pad position to be tested; f(ROI i ) represents that the pad detection operation is performed on the circuit board image to be tested by the target ResNet neural network model The results obtained.
A pad detection device applied to a flexible circuit board, characterized in that it comprises:

A training request receiving module, configured to receive a model training request carrying circuit board sample images and pad position information;

A segmentation operation module, configured to perform a segmentation operation on the circuit board sample image according to the pad position information to obtain a pad sample image;

An annotation sending module, configured to send the pad sample image to a terminal device, so as to perform an annotation operation on the pad sample image;

An annotation receiving module, configured to receive the pad annotation image sent by the terminal device, the pad annotation image is a pad sample image carrying annotation information, and the annotation information is good product information or defective product information;

The model training module is used to call the original ResNet neural network model, and use the pad sample image as training data and the label information as result data to perform model training operations on the original ResNet neural network model to obtain a Targeted ResNet neural network model for disk image quality;

A detection request receiving module, configured to accept a pad detection request carrying an image of a circuit board to be tested;

The pad detection module is used to input the image of the circuit board to be tested into the target ResNet neural network model to perform a pad detection operation to obtain a pad detection result;

A result output module, configured to output the detection result of the pad.
The pad detection device applied to flexible circuit boards according to claim 5, wherein the device further comprises:

The color space conversion module is used to convert the pad sample image from the RGB color space to the HSV color space according to the conversion formula, and the conversion formula is expressed as:

V=max

Among them, max=max(r,g,b), min=min(r,g,b).
The pad detection device applied to flexible circuit boards according to claim 5, wherein the device further comprises:

The pre-classification module is configured to perform a pre-classification operation on the pad sample images according to the feature method to obtain multiple groups of pad sample images with similar pad shapes.
The pad detection device applied to flexible circuit boards according to claim 5, wherein the pad detection module comprises:

The change area acquisition submodule is used to count the change area of the B channel value in the RGB color space of each pad position to be tested in the image of the circuit board to be tested;

The detection sub-module is used to detect the image of the circuit board to be tested according to the preset good product ratio threshold S1, the defective product ratio threshold S2 and a judgment function, and the judgment function is expressed as:

Wherein, the PI represents the ratio of the change area to the area of the pad position to be tested; f(ROI i ) represents that the pad detection operation is performed on the circuit board image to be tested by the target ResNet neural network model The results obtained.
A computer device, characterized by comprising a memory and a processor, wherein computer-readable instructions are stored in the memory, and when the processor executes the computer-readable instructions, the computer-readable instructions described in any one of claims 1 to 4 are realized. The steps of the above-mentioned pad detection method applied to flexible circuit boards.
A computer-readable storage medium, wherein computer-readable instructions are stored on the computer-readable storage medium, and when the computer-readable instructions are executed by a processor, the computer-readable instructions described in any one of claims 1 to 4 are implemented. The steps of the above-mentioned pad detection method applied to flexible circuit boards.