KR20240055609A - Method and apparatus for bubble inspection of conformal coating using rule-based patching method followed by the CNN verification - Google Patents

Abstract

The present invention discloses a method for inspecting bubbles in conformally coated boards (PCBs). The bubble inspection method according to the present invention includes the steps of acquiring a low-illuminance image taken of a substrate at a first illuminance and a high-illuminance image taken at a second illuminance higher than the first illuminance; Binarizing the low-light image and the high-light image, respectively, to generate a low-light binarized image and a high-light binarized image; A bubble rear area extraction step of extracting an area where the boundary area of the bubble and the shaded area of the bubble appear together as a bubble candidate area using the low-level binarized image and the high-level binarized image; It includes the step of verifying whether the bubble candidate area is a bubble using a deep learning model.
According to the present invention, bubble candidate areas are automatically extracted using low-light and high-light images of the conformal coating area, and the extracted candidate areas are verified by applying a convolutional neural network (CNN) to detect bubbles. Accuracy can be greatly improved and bubble detection speed can also be greatly improved. Additionally, according to the present invention, air bubbles generated in the MCU lead can be detected with high accuracy.

Description

Method and apparatus for bubble inspection of conformal coating using rule-based patching method followed by the CNN verification}

The present invention relates to a method of detecting bubbles in the conformal coating area of a printed circuit board (PCB). Specifically, the bubble area is estimated using low-light images and high-light images, and finally, a convolutional neural network (Convolutional Neural Network) is used to estimate the bubble area. This is about a bubble inspection method and device that verifies the presence of bubbles by applying Networks.

In general, conformal coating is a technology that protects MCU (Micro Control Unit) by thinly applying a coating solution to the surface of components or circuits mounted on a printed circuit board (PCB), and is a process for producing MCUs for automobiles. It is known as an essential technology in

However, as the electric vehicle market grows recently, conformal coating inspection standards are being strengthened to ensure the safety of MCUs. especially. Since air bubbles generated on the MCU leads can cause failure, the bubbles must be inspected more precisely for successful coating.

However, because bubbles vary in size and occur irregularly due to the viscosity of the solution, it is very difficult to automatically detect bubbles.

In addition, in the past, the method of detecting air bubbles was mainly used by the inspector visually observing the image of the coating area taken with a camera. This not only resulted in a large difference in inspection accuracy depending on the inspector's skill level, but also caused the bubble inspection to take too much time. Because there is a problem, there is a need to develop automated inspection equipment.

In addition, conventionally, bubbles with a diameter of 250㎛ or more were considered defective, but recently, in the case of electric vehicles, bubble tests are also required for bubbles with a diameter of 250㎛ or less to increase safety, so there is a need to further improve the accuracy of bubble inspection.

Korean Patent No. 10-2384771 (announced on 2022.04.08)

The present invention was conceived against this background, and its purpose is to more accurately and quickly detect bubbles in the conformal coating area by analyzing images captured by a camera.

One aspect of the present invention is a method of inspecting bubbles in a conformally coated substrate, comprising the steps of acquiring a low-illuminance image of the substrate at a first illuminance and a high-illuminance image of the substrate at a second illuminance higher than the first illuminance. ; Binarizing the low-light image and the high-light image, respectively, to generate a low-light binarized image and a high-light binarized image; A bubble rear area extraction step of extracting the area where the boundary area of the bubble and the shaded area of the bubble appear together as a bubble candidate area using the low-level binarized image and the high-level binarized image; Provides a bubble inspection method including the step of verifying whether a bubble candidate area is a bubble using a deep learning model.

In the bubble inspection method according to an aspect of the present invention, the low-illuminance binarized image may be an image binarized by converting the low-illuminance image into a grayscale image and then applying an adaptive threshold to the grayscale image.

Additionally, in the bubble inspection method according to an aspect of the present invention, the high-intensity binarized image may be an image binarized by converting the high-intensity image into an inverse image and then applying an adaptive threshold to the inverse image.

In addition, in the bubble inspection method according to one aspect of the present invention, the bubble candidate area extraction step includes setting a preliminary candidate area surrounding the boundary of the bubble in the low-light binarized image; Applying the set preliminary candidate area to the high-intensity binarized image to check whether the shadow of the bubble exists inside the preliminary candidate area; If the shadow of a bubble exists inside the preliminary candidate area applied to the high-intensity binarization image, a step of determining the preliminary candidate area as the bubble candidate area may be included.

In addition, in the bubble inspection method according to an aspect of the present invention, the step of determining the preliminary candidate area as the bubble candidate area includes readjusting the size and position of the area so that the bubble candidate area can surround both the boundary and the shadow of the bubble. Process may be included.

In addition, in the bubble inspection method according to one aspect of the present invention, the bubble candidate area extraction step includes setting a preliminary candidate area surrounding the shadow of the bubble in the high-intensity binarized image; Applying the set preliminary candidate area to the low-light binarized image to check whether a bubble boundary exists inside the preliminary candidate area; If a bubble boundary exists inside the preliminary candidate area applied to the low-light binarization image, a step of determining the preliminary candidate area as the bubble candidate area may be included.

Additionally, in the bubble inspection method according to one aspect of the present invention, in the step of setting a preliminary candidate area surrounding the shadow of the bubble, a shaded area surrounding the edge of the portion estimated to be shaded is set, and then the shaded area is increased by a set multiple. The step of enlarging and determining the preliminary candidate area and determining the preliminary candidate area as the bubble candidate area may include the process of reducing the area determined as the bubble candidate area to the minimum area surrounding the boundary and shadow of the bubble.

Another aspect of the present invention is an apparatus for inspecting bubbles in a conformally coated substrate, wherein a low-illuminance image is obtained by photographing the substrate at a first illuminance and a high-illuminance image is obtained by taking a second illuminance higher than the first illuminance. acquisition department; A low-light image binarization unit that binarizes a low-light image; A high-intensity image binarization unit that binarizes a high-intensity image; A bubble rear area extraction unit that extracts an area where the boundary area of the bubble and the shaded area of the bubble appear together as a bubble candidate area using the low-level binarized image and the high-level binarized image; A bubble inspection device including a verification unit that verifies whether a bubble candidate area is a bubble using a deep learning model is provided.

In the bubble inspection device according to another aspect of the present invention, the low-light image binarization unit converts the low-light image into a grayscale image and then applies an adaptive threshold to the grayscale image to generate a low-light binarization image. .

Additionally, in the bubble inspection device according to another aspect of the present invention, the high-intensity image binarization unit can convert the high-intensity image into an inverse image and then apply an adaptive threshold to the inverse image to generate a high-intensity binarization image. there is.

In addition, in the bubble inspection device according to another aspect of the present invention, the bubble candidate area extractor sets a preliminary candidate area surrounding the boundary of the bubble in the low-illuminance binarized image, and applies the set preliminary candidate area to the high-intensity binarized image to select the preliminary candidate area. It is checked whether the shadow of a bubble exists inside the area, and if the shadow of a bubble exists inside the preliminary candidate area applied to the high-intensity binarization image, the preliminary candidate area can be determined as the bubble candidate area.

In addition, in the bubble inspection device according to another aspect of the present invention, the bubble candidate area extractor sets a preliminary candidate area surrounding the shadow of the bubble in the high-intensity binarized image, and applies the set preliminary candidate area to the low-illuminance binarized image to select the preliminary candidate area. It is checked whether a bubble boundary exists inside the area, and if a bubble boundary exists inside the preliminary candidate area applied to the low-light binarization image, the preliminary candidate area can be determined as a bubble candidate area.

According to the present invention, bubble candidate areas are automatically extracted using low-light and high-light images of the conformal coating area, and the extracted candidate areas are verified by applying a convolutional neural network (CNN) to detect bubbles. Accuracy can be greatly improved and bubble detection speed can also be greatly improved. Additionally, according to the present invention, air bubbles generated in the MCU lead can be detected with high accuracy.

1 is an algorithm flowchart showing a bubble inspection method according to an embodiment of the present invention.
Figure 2 (a) is a bubble image of the conformal coating image, and (b) is a graph showing the brightness characteristics of the boundary and center of the bubble.
In Figure 3 (a) is an image of a bubble at low illumination, (b) is an image of a bubble at high illumination, (c) is an image obtained by binarizing the boundary of the bubble, and (d) is an image obtained by binarizing the shade of the bubble. , (e) is a diagram showing the area where the boundary and shadow of the bubble appear simultaneously, and (f) shows verification of the bubble area using CNN.
Figure 4 is a diagram showing bubbles of various brightness generated from the lead of the MCU.
In Figure 5, (a) is a bubble image of a low-light image, (b) is an image of the bubble boundary when an adaptive threshold is applied, and (c) is an image of the bubble boundary when a fixed threshold is applied.
Figure 6 is a diagram showing a method of estimating the bubble area using the boundary of the bubble. In Figure 6, (a) estimates the area containing the boundary of the bubble, (b) applies the area containing the boundary of the bubble to the shaded image of the bubble to check whether shading exists, (c) estimates the boundary and shading of the bubble. (d) Shows the bubble area detected through the boundary of the bubble.
Figure 7 is a diagram showing a method of estimating the bubble area using bubble shading. In Figure 7 (a), the shaded area of the bubble is expanded, (b) the area expanded through shading is applied to the bubble boundary binary image, and (c) shows the area where the bubble boundary and the shade appear simultaneously is estimated as the bubble area. , (d) is reduced to the minimum area including the boundary and shading of the bubble, and (e) shows the bubble area estimated through shading.
Figure 8 is a diagram illustrating the bubble verification process using CNN
Figure 9 (a) shows the image acquisition equipment, (b) shows the conformal coating image, and (c) shows the learning data.
Figure 10 is a diagram showing the ground truth image
Figure 11 is a diagram showing the results of bubble detection
Figure 12 is a diagram comparing bubble detection results when applying the method according to the present invention and a fixed threshold value
13 is a schematic configuration diagram of a bubble inspection device according to an embodiment of the present invention.
14 is a functional block diagram of a bubble inspection device according to an embodiment of the present invention.
Figure 15 is a functional block diagram of the image analysis unit
Figure 16 is a flow chart illustrating a method for generating a low-light binarization image.
Figure 17 is a flow chart illustrating a method for generating a high-intensity binarized image.
18 and 19 are flow charts illustrating several methods for extracting bubble candidate regions.
Figure 20 is a flow chart showing the operation of the bubble inspection device according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

For reference, in this specification, the fact that a part includes or is provided with a certain component does not mean that other components are excluded, but that it can further include or include other components, unless specifically stated to the contrary. In addition, when one element is connected, combined, or communicated with another element, it is not only directly connected, combined, or communicated with another element, but also indirectly connected with another element in between. This also includes cases of combining, or communicating. Additionally, when one component is directly connected or directly combined with another component, it means that no other elements are interposed. In addition, it is stated in advance that the drawings attached to this specification are merely illustrative to facilitate understanding of the gist of the invention, and therefore, the scope of rights of the present invention should not be limited.

First, the present invention relates to a method for detecting air bubbles generated in the MCU lead. Additionally, the bubble inspection method according to the present invention performs binarization on low-light images and high-light images to detect relatively small-sized bubbles.

The boundary of the bubble appears brighter in the low-light image, and the shading at the center of the bubble appears darker in the high-light image.

In the present invention, an adaptive threshold method is applied to efficiently binarize these bubbles. Additionally, in order to minimize non-bubble areas during the binarization process, the threshold is limited using the average and standard deviation of the image.

Then, the bubble area is estimated using the binary image of the bubble boundary and bubble center, and finally, the bubble area is verified by applying convolutional neural networks to exclude non-bubble areas.

1. Bubble detection algorithm

Below, a specific method for detecting bubbles using the boundaries and shadows of bubbles according to the present invention will be described. 1 is a flowchart showing a bubble inspection method according to the present invention.

In the present invention, basically, the boundary of the bubble is obtained from the low-illuminance image, the shade of the bubble is obtained from the high-illuminance image, the bubble area is estimated based on the combined structure, and finally whether it is a bubble is determined through the CNN algorithm. do.

Conformal coating inspection is performed under UV (Ultra Violet) lighting. Figure 2(a) is an image of a bubble obtained under UV illumination. Referring to the brightness graph in Figure 2(b), under UV illumination, the boundaries of the bubbles appear very bright due to the agglomeration of the solution, and the center of the bubble appears very dark due to the dispersion of the solution.

In particular, the shading of the boundary and center of the bubble appears clearly under different UV illumination brightness. In other words, the boundaries of bubbles appear more clearly in low-light images, and the shadows of bubbles appear more clearly in high-light images.

For example, in a low-illuminance image as shown in Figure 3 (a), the boundary of the bubble shows higher luminance than the MCU Lead, so it is easy to distinguish, but in a high-illuminance image as shown in Figure 3 (b), the boundary of the bubble and the MCU Lead are different. Because the luminance is similar, they cannot be distinguished.

Meanwhile, in high-intensity images, only the shadows of bubbles appear with low luminance, making them easy to distinguish. Since the brightness of the bubble in each image is significantly (distinctly) distinguished from the surroundings, the bubble can be easily binarized using the adaptive threshold method.

Figures 3 (c) and (d) are the results of binarizing the boundaries and shades of bubbles, respectively. Next, to estimate the bubble area, find the area where the bubble boundary and shadow appear simultaneously, as shown in Figure 3(e).

If the boundary and shadow of a bubble appear at the same time, there is a high possibility that it is a bubble, but since the area that is not a bubble can be estimated, it is verified using a convolutional neural network. Ultimately, by applying the method according to the present invention, bubbles can be detected as shown in Figure 3 (f).

(1) Binarization of bubbles using Adaptive Threshold

Bubbles appear in various brightnesses depending on their location and size. Figure 4 is an image of bubbles generated in the MCU Lead. For small bubbles, the border of the bubble appears only partially or the border appears darker than for large bubbles.

For large bubbles, the border of the bubble appears brightly. Conventionally, a fixed threshold was applied to detect bubbles with a diameter of 250㎛ or more, with bright bubble boundaries.

The present invention proposes a method for efficiently binarizing bubbles. The binarization process of bubbles is as shown in Equation 1 below.

<Equation 1>

The boundary of the bubble in Equation 1 uses a Gray Image, and the shading of the bubble uses an Inverse Image as in Equation 2.

<Equation 2>

The threshold T(x,y) in Equation 1 is defined as Equation 3.

<Equation 3>

G(i,j) in Equation 3 is the Gaussian weight, and C is a constant that minimizes the non-bubble area. Limit the threshold through C. In the embodiment of the present invention, C was applied as shown in Equation 4. μ and σ are the mean and standard deviation of the image.

<Equation 4>

In the bubble binarization process, there is a trade-off relationship between the bubble area and the non-bubble area. Therefore, the optimal C that shows the bubble area well while minimizing the non-bubble area should be applied.

Figure 5 is a comparison result when applying the method according to the present invention and a fixed threshold. The boundaries of the bubbles are shown in magenta, and the bubbles are shown in green and red.

The method according to the present invention detected boundaries in all bubbles, as shown in (b) of FIG. 5, but the method using a fixed threshold did not detect boundaries in some bubbles, as shown in (c) of FIG. 5.

(2) Bubble area estimation

Below, two methods will be described: a method of estimating the bubble area using the boundary of the bubble and a method of estimating the bubble area using the shading of the bubble.

To estimate the bubble area, the property that a shadow exists at the center of the boundary of the bubble is used. First, the process of estimating the bubble area using the bubble boundary is shown in Figure 6. Obtain a rectangular area surrounding the boundary of the bubble as shown in (a) of Figure 6.

Next, apply the same area to the shaded binary image as shown in (b) of Figure 6 and check whether the shade exists. If the border and shadow exist at the same time, it is likely to be a bubble.

Finally, as shown in (c) of Figure 6, the area containing both the boundary and shadow of the bubble is determined as the bubble area. (d) in Figure 6 is the bubble area estimated through the boundary of the bubble.

Next, a method of estimating the bubble area using the bubble's shading is explained. Excluding the boundary and shading of the bubble used for boundary-based bubble area estimation above, the remaining boundary and shading are used to estimate the bubble area.

First, for the shading, an area expanded horizontally and vertically by 2.5 times is obtained, as shown in (a) of Figure 7.

Next, the extended area of the shading is applied to the binary image of the border as shown in (b) of Figure 7, and the area where the shading and the border appear simultaneously is obtained as shown in (c) of Figure 7.

Then, it is reduced to the minimum area including the shadow and border, as shown in (d) of Figure 7.

Figure 7(e) is the bubble area estimated using shading. The bubble area estimated through the boundary and shading of the bubble is likely to be a bubble, but there may be areas that are not bubbles. Therefore, a verification process for the bubble area is necessary.

(3) Verification using convolutional neural network

The initial convolutional neural network models were classification models such as AlexNet and ResNet, and later detection models such as Yolo were announced. The performance of convolutional neural network models improves as the amount of training data increases.

However, because conformal coating images have limitations in collecting data, it is difficult to learn a detection model. In a previous study, transfer learning was applied to detect air bubbles to solve the problem of insufficient data.

In an example of the present invention, bubbles were verified by applying the VGG19 model. Figure 8 shows the verification process. The bubble area estimated through the boundary and shading of the bubble is adjusted to a size of 32*32 and verified by applying it to the VGG19 model.

2. Experiment and results

(1) Learning

To create learning data, conformal coating images are acquired using an integrated UV (Ultra Violet) lighting camera in an environment as shown in (a) of Figure 9. Figure 9(b) is a conformal coating image of the PCB lead area.

And as shown in Figure 9 (c), bubbles generated on the lead were obtained in a size of 32*32, and non-bubble data was randomly generated at locations without bubbles.

From 36 PCB samples, 1,630 bubble data and 1,630 non-bubble data were obtained on the lead.

Of the total 3,260 learning data, 2,770 were used for learning and 490 were used for model verification. As learning parameters, Adam was used as an optimizer and Mean Square Error was applied as a loss function.

(2) Bubble detection results

For performance evaluation, bubbles with a diameter of 150㎛ or more that occurred on the MCU lead were evaluated in 12 images with 960*960 resolution. According to the IPC-HDBK-830 standard, bubbles less than 50% of the spacing between conductors are generally allowed, so the minimum bubble size was set as the spacing between leads. Figure 9 shows GT (Ground Truth).

Green is a bubble with a diameter of 150㎛ or more, and orange is a bubble with a diameter of less than 150㎛. In the quantitative evaluation, bubbles smaller than 150㎛ were excluded, and the detection results for 12 experimental images are summarized in Table 1.

[Table 1]

　

TP (True Positive) means when the IoU between the detection result and GT is calculated and is greater than 0.5, FN (False Negative) means unrecognition, and FP (False Positive) means misrecognition.

And to analyze performance, precision, recall, and harmonic mean (F1-Score) were defined and used as shown in Equation 5 below. Figure 10 shows the bubble detection results. Green is TP, red is FP, and yellow is FN.

<Equation 5>

(3) Comparative evaluation according to binarization methods

The bubble inspection method according to the present invention applies an adaptive thresholding method to efficiently binarize the boundaries and shadows of bubbles. In the experiment, it is confirmed whether the method according to the present invention efficiently binarizes the brightness characteristics of the bubbles.

To compare detection performance according to binarization methods, results were compared by applying a fixed threshold instead of the Adaptive Thresholding method.

The fixed threshold binarizes bubbles by applying Multi Otsu to low-light images. Table 2 shows the comparison results according to binarization methods.

[Table 2]

The method according to the present invention had better F1-Score performance than the fixed threshold. And the method according to the present invention detected boundaries and shadows for all bubbles, as shown in Figure 12 (a).

However, the fixed threshold did not detect shadows for small bubbles as shown in (b) of Figure 12. These results show that the method according to the present invention efficiently binarizes the brightness characteristics for all bubbles.

(4) Comparative evaluation according to bubble area estimation method

The method according to the present invention constructs two binary maps including the boundary and shadow information of bubbles, extracts a bubble candidate area from the map, and then performs CNN verification on the candidate area to provide a high-speed algorithm for bubble detection. Implementation was pursued.

To compare whether the method according to the present invention effectively estimates the bubble area, we compared the bubble detection performance and time when the Sliding Window method was applied to the entire image. When applying the Sliding Window method, the parameters in Table 3 below were applied to estimate the area of all bubbles present in the image.

[Table 3]

Table 4 summarizes the comparison results between the method according to the present invention and the Sliding Window method.

[Table 4]

Here, Time is the average time it takes to estimate the candidate area of the bubble for each test image. The method according to the present invention has a recall decrease of about 0.0285 compared to the Sliding Window method, but because it estimates the bubble area using the brightness characteristics of the bubble, it takes less time and efficiently obtains the bubble area. Additionally, among the 159 false positive bubble detection results, 141 were results of detecting bubbles smaller than 150μm. There were 17 detection results that were not actual bubbles.

As described above, the present invention proposes a method for detecting bubbles using the brightness characteristics of bubbles appearing in conformal coating images. The method according to the present invention detects air bubbles in three steps.

The first step uses adaptive threshold to binarize the high-light and low-light images, respectively. Through this, the boundaries of the bubble and the shading at the center of the bubble are binarized, and the outline of the bubble is formed.

The second step uses the outline in the binary image to select a candidate area for the bubble using the area and location of the outline. In other words, if the boundary of a bubble and the shadow appear at the same time, there is a high possibility that it is a bubble, so the boundary area including the shadow is estimated to be the bubble area.

The final step is to verify the estimated bubble area using a convolutional neural network and remove areas that show brightness characteristics similar to bubbles, such as the metal surface of the MCU lead.

In an example of the present invention, the candidate region of the bubble was verified using the VGG19 model. Additionally, in an embodiment of the present invention, the images obtained from 12 PCBs showed performance of Precision 0.5923, Recall 0.9429, and F1-Score 0.7276 for bubbles of 150㎛ or more in diameter that occurred in the MCU lead.

3. Air bubble inspection device

Below, a bubble inspection device that performs a bubble inspection by applying the inspection method described above will be described.

The bubble inspection device 100 according to an embodiment of the present invention includes a printed circuit board 50 (hereinafter referred to as a 'board' for convenience), as illustrated in the schematic configuration diagram of FIG. 13 and the functional block diagram of FIG. 14. This placed substrate stand 60, a camera 150 for photographing the substrate 50 from the upper part of the substrate stand 60, a lighting 160 installed around the camera 150, and a photographing position for the camera 150. It includes a camera transport means 152, a control unit 101, an input unit 170, a display 180, a communication unit 190, etc.

Although not shown in the drawing, it may include substrate transfer means such as a linear guide or picker that transfers the substrate 50 to the inspection position.

This bubble inspection device 100 may be installed inside the conformal coating equipment that applies the coating solution to the substrate, may be installed adjacent to the exit side of the conformal coating equipment, or may be installed independently of the conformal coating equipment. It may be installed.

The camera 150 acquires image data by photographing the board to be inspected. If there are multiple inspection areas on one board, the camera 150 can move to the top of each inspection area along the camera transfer means 152 to capture images. The camera 150 is preferably a high-resolution, high-magnification camera, but the specific type or magnification is not particularly limited.

It is desirable to install the lighting 160 so that it is coupled to one side of the camera 150 and moves with the camera 150, but it is not necessarily limited to this, so it can be installed separately from the camera 150 or using a separate moving means. Lighting 160 can also be moved. Meanwhile, if the conformal coating solution contains a fluorescent material that reacts to ultraviolet rays, it is preferable to use ultraviolet light 160.

In an embodiment of the present invention, an image captured of the substrate 50 at a first illuminance (hereinafter referred to as a 'low-illuminance image') and an image captured of the substrate 50 at a second illuminance higher than the first illuminance (hereinafter referred to as a 'high-illuminance image') Since the bubble inspection is performed using (referred to as 'image'), lighting 160 whose illuminance can be adjusted must be used.

Since the illuminance of the lighting 160 may be applied differently depending on the color and physical properties of the substrate and/or coating liquid, it is desirable to experimentally determine the intensity and/or illuminance difference between high and low illuminance using the actual substrate 50 in advance. do.

At this time, the high illuminance and low illuminance can be determined using an image directly taken of the substrate 50 with the camera 150. Additionally, when determining the illuminance, the illuminance of the lighting 160 can be directly determined using a illuminance meter, or the illuminance can be determined using image data captured by the camera 150.

When using image data, for example, based on the HSV color space where the H value ranges from 0 to 360, when shooting low-light images, lighting is performed so that the H average value falls within the first range (e.g., 225 to 230). When the illuminance is set to 160 and a high-intensity image is taken, the illuminance of the light 160 can be set so that the H average value falls within a second range (e.g., 200 to 225) that is smaller than the first range.

However, the method of controlling low and high light levels is not necessarily limited to this, so low and high light levels can be determined in any number of different ways depending on the work object or environment.

The input unit 170 is an input means for the user of the bubble inspection device 100 and may include at least one of various input means such as a keyboard, button, touch pad, touch screen, mouse, joystick, and trackball.

The display 180 can display the image of the substrate captured by the camera 150, the image processing results performed by the image analysis unit 130, the results of extracting the bubble candidate area, the operating status of the bubble inspection device 100, etc. .

The communication unit 190 includes a wired or wireless communication means for communicating with an external electronic device or server. Wireless communication means may include short-range wireless communication means such as Wi-Fi, Bluetooth, and ZigBee, and mobile communication means.

The bus 109 is a transmission path for data or electrical signals between each component of the bubble inspection device 100, and may be provided in the form of a circuit pattern or cable.

The control unit 101 controls the overall operation of the bubble inspection device 100 and executes the bubble inspection method according to an embodiment of the present invention.

The control unit 101 includes a processor 110, a memory 112, an image acquisition unit 120, an image analysis unit 130, a verification unit 140, etc.

The processor 110 executes a computer program stored in the memory 112 and performs a predetermined operation.

The memory 112 may store computer programs, various parameters, data, etc. for the operation of the bubble inspection device 100. The memory 112 may include non-volatile memory such as flash memory and volatile memory such as RAM. The memory 112 may include a mass storage device such as HDD, ODD, or SDD.

The image acquisition unit 120 controls the camera 150 and the lighting 160 to obtain a low-light image and a high-light image of the substrate 50, respectively.

When an operation command is input through the input unit 170 or a set condition is met, the image acquisition unit 120 controls the camera 150 and the lighting 160 to control the entire area and/or a partial area of the substrate 50. For each, low-light images and high-light images can be automatically acquired.

The image analysis unit 130 extracts a candidate bubble area using the low-intensity image and the high-intensity image acquired by the image acquisition unit 120.

Referring to the functional block diagram of FIG. 15, the image analysis unit 130 may include a low-illuminance image binarization unit 132, a high-illuminance image binarization unit 134, a bubble candidate area extraction unit 136, and the like.

The low-light image binarization unit 132 binarizes the low-light image to generate a low-light binarized image.

According to an embodiment of the present invention, the low-illuminance image binarization unit 132 first converts the low-illuminance image into a grayscale image (ST11), as illustrated in the flowchart of FIG. 16, and blurs the grayscale image. Ring processing is performed (ST12), and then an adaptive threshold is applied to generate a low-light binarized image from the grayscale image. (ST23)

The high-intensity image binarization unit 134 binarizes the high-intensity image to generate a high-intensity binarized image.

According to an embodiment of the present invention, the high-intensity image binarization unit 134 converts the high-intensity image into an inverse image (ST21), as illustrated in the flowchart of FIG. 17, and performs blurring on the inverse image. (ST222), and then an adaptive threshold is applied to generate a high-intensity binarized image from the inverted image. (ST23)

However, since the binarization method for low-light images and high-light images is not necessarily limited to this, each binarized image may be generated using other methods.

The bubble candidate area extraction unit 136 extracts the bubble candidate area using the low-light binarization image and the high-light binarization image of the same substrate generated by the low-light image binarization unit 132 and the high-light image binarization unit 134, respectively. The bubble candidate area can be extracted by applying the following two methods.

The first method is based on bubble boundaries, as shown in the flow chart of Figure 18.

First, as previously described in relation to FIG. 6, one or more preliminary candidate areas (areas indicated by a square in FIG. 6(a)) surrounding the portion presumed to be the boundary of the bubble in the low-light binarized image are set. (ST31)

Next, the location information (coordinates, etc.) of the set preliminary candidate area is extracted and the same area is applied to the high-intensity binarized image. (ST32)

Next, it is determined whether a part presumed to be the shadow of a bubble exists inside the preliminary candidate area applied to the high-intensity binarized image. (ST33)

If, in step ST33, it is confirmed that a part presumed to be the shadow of a bubble exists inside the preliminary candidate area applied to the high-intensity binarization image, the area is determined as a bubble candidate area because both the boundary and the shadow of the bubble exist. do.

At this time, the preliminary candidate area is an area set based on the boundary of the bubble, so after it is determined as the bubble candidate area, the bubble candidate area surrounds both the boundary and the shade of the bubble, like the bubble candidate area shown at the top in Figure 6 (c). It is advisable to readjust the size and location of the area so that it can be accommodated. (ST34)

If, in step ST33, it is confirmed that the part estimated to be the shadow of a bubble does not exist inside the preliminary candidate area, the corresponding area is excluded from the bubble candidate area. (ST35)

The second method is based on bubble shading, as shown in the flow chart of Figure 19.

First, as previously described in relation to FIG. 7, one or more preliminary candidate areas (areas indicated by square dotted lines in FIG. 7(a)) surrounding the portion presumed to be the shadow of the bubble in the high-intensity binarized image are set.

At this time, after first setting the shaded area surrounding the edge of the part estimated to be shaded, the area enlarged horizontally and vertically by a set multiple (e.g., 2.5 times) based on the center point of the set shaded area is set as a preliminary candidate area. It is desirable. This is because the boundaries of bubbles are mostly spaced outside the shaded area, so the preliminary candidate area must be much wider than the shaded area to include the boundary of the bubble. (ST41)

Next, the location information (coordinates, etc.) of the set preliminary candidate area is extracted and the same area is applied to the low-light binarized image. (ST42)

Next, it is determined whether a portion presumed to be the boundary of the bubble exists inside the preliminary candidate area applied to the low-light binarization image. (ST43)

If, in step ST43, it is confirmed that a part presumed to be a bubble boundary exists inside the preliminary candidate area applied to the low-light binarization image, the area is determined as a bubble candidate area because both the bubble border and shadow exist. .

On the other hand, since the preliminary candidate area is an area arbitrarily set too wide to include the boundary of the bubble, after it is determined as the bubble candidate area, the bubble candidate area is shaded with the boundary of the bubble, as shown in Figure 7 (d). It is desirable to reduce it to the minimum surrounding area. (ST44)

If, in step ST43, it is confirmed that the part estimated to be the boundary of a bubble does not exist inside the preliminary candidate area, the corresponding area is excluded from the bubble candidate area. (ST45)

In the above, a method of extracting a bubble candidate area based on the boundary of a bubble and a method of extracting a bubble candidate area based on the bubble's shading were explained. These extraction methods may be carried out independently of each other or may be carried out complementary to each other. It may be possible.

As an example, both the first method and the second method may be applied to the entire substrate and the area commonly extracted as the bubble candidate area may be determined as the final candidate area, and the area extracted as the bubble candidate area in either method may be selected as the final candidate area. It can also be decided as a candidate area.

As another example, after extracting a substrate candidate region by applying one method to the entire substrate, the substrate candidate region may be extracted by reapplying another method to the portion for which the substrate candidate region was not extracted.

Meanwhile, the verification unit 140 finally determines whether the plurality of bubble candidate areas extracted from the image analysis unit 130 are bubbles using an artificial intelligence-based image analysis model. In an embodiment of the present invention, the bubble candidate area is adjusted to a size of 32 pixels*32 pixels, and then the presence of bubbles is finally verified using a synthetic neural network model (e.g., VGG19 model).

The artificial intelligence model used in the verification unit 140 is preferably a deep learning model such as CNN, but is not necessarily limited thereto.

The verification unit 140 may display a color effect or output a visual effect according to a set method so that the inspector can easily recognize the area determined to be a bubble among the areas marked as bubble candidate areas.

Additionally, the verification unit 140 may output the verification results to the display 180 or transmit them to an external electronic device or server through the communication unit 190.

Some or all of the above-described functions of the image acquisition unit 120, image analysis unit 130, and verification unit 140 are stored in the memory 112 and provided in the form of a computer program executed by the processor 110. It can be. In addition, some or all of the functions of the image acquisition unit 120, image analysis unit 130, and verification unit 140 may be provided in the form of hardware such as ASIC (Application Specific Integrated Circuit), and the hardware and software may be integrated. It may also be provided in a format.

Hereinafter, the operation flow of the bubble inspection device 100 according to an embodiment of the present invention will be described with reference to the flowchart of FIG. 20.

First, when the inspector places the board to be inspected 50 on the board holder 60 and inputs an image acquisition command, the image acquisition unit 120 controls the camera 150 and lighting 160 to display the board 50. Take pictures. At this time, the inspection area can be designated and photographed, or the entire substrate 50 can be photographed.

In particular, in an embodiment of the present invention, the image acquisition unit 120 adjusts the lighting 160 to a preset first illuminance and a second illuminance higher than the first illuminance, thereby creating a low-illuminance image and a high-illuminance image for the substrate 50. Acquire each image. (ST51, ST52)

Next, the low-light/high-light image binarization units 132 and 134 apply adaptive thresholds to the low-light and high-light images, respectively, to generate low-light and high-light binarized images, respectively.

Specifically, as shown in Figures 16 and 17, respectively, in the case of a low-light image, a low-light binarized image is generated after conversion to a grayscale image, and in the case of a high-light image, a high-level image is generated after conversion to an inverse image. Also generates a binarized image. (ST53)

Next, the bubble candidate area extractor 136 uses the low-level binarized image and the high-level binarized image to extract the area where the border and shadow appear together as the bubble candidate area. The specific method is as described in relation to FIGS. 18 and 19. (ST54)

Next, the verification unit 140 analyzes each image of the extracted bubble candidate area using a deep learning-based image analysis model to verify whether it is a bubble. The verification results of the verification unit 140 may be displayed through the display 180 of the bubble inspection device 100. (ST55)

When the verification unit 140 outputs the results for all input bubble candidate images, the processor 110 outputs the completion of the bubble inspection for the corresponding substrate 50 through the display 180, etc. to indicate the completion of the bubble inspection for the substrate 50. You can request removal.

Meanwhile, the bubble inspection method according to the embodiment of the present invention described above can be implemented in the form of program instructions that can be performed through various computer means and recorded on a computer-readable recording medium.

At this time, the computer-readable recording medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the recording medium may be specially designed and configured for the present invention or may be known and available to those skilled in the art of computer software.

Computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. It may include at least one of optical media, ROM, RAM, flash memory, SSD, etc.

Additionally, program instructions may include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc.

In addition, although the preferred embodiments of the present invention have been described above, the present invention is not limited to the previously described embodiments and can be implemented with various modifications or modifications in the specific application process, and the modified or modified embodiments are also included in the patent claims described later. If it includes the technical idea of the present invention disclosed in, it will naturally fall within the scope of the rights of the present invention.

50: substrate 60: substrate mounting table
100: Bubble inspection device 101: Control unit 109: Bus
110: Processor 112: Memory 120: Image acquisition unit
130: Image analysis unit 132: Low-light image binarization unit
134: High intensity image binarization unit 136: Bubble candidate area extraction unit
140: Verification unit 150: Camera 152: Camera transport means
160: Lighting 170: Input unit 180: Display
190: Department of Communications

Claims (12)

In a method of inspecting bubbles in a conformally coated substrate,
Obtaining a low-illuminance image taken of the substrate at a first illuminance level and a high-illuminance image taken at a second illuminance level higher than the first illuminance level;
Binarizing the low-light image and the high-light image, respectively, to generate a low-light binarized image and a high-light binarized image;
A bubble rear area extraction step of extracting the area where the boundary area of the bubble and the shaded area of the bubble appear together as a bubble candidate area using the low-level binarized image and the high-level binarized image;
Step of verifying whether the bubble candidate area is a bubble using a deep learning model
Bubble inspection method including According to paragraph 1,
The low-light binarization image is a bubble inspection method characterized in that the low-light image is converted to a grayscale image and then binarized by applying an adaptive threshold to the grayscale image. According to paragraph 1,
The high-intensity binarized image is a bubble inspection method characterized in that the high-intensity image is converted to an inverse image and then binarized by applying an adaptive threshold to the inverse image. According to paragraph 1,
The bubble candidate area extraction step is,
Setting a preliminary candidate area surrounding the boundary of the bubble in the low-light binarized image;
Applying the set preliminary candidate area to the high-intensity binarized image to check whether the shadow of the bubble exists inside the preliminary candidate area;
If the shadow of a bubble exists inside the preliminary candidate area applied to the high-intensity binarization image, determining the preliminary candidate area as the bubble candidate area.
Bubble inspection method comprising: According to paragraph 4,
The step of determining the preliminary candidate area as a bubble candidate area includes a process of readjusting the size and position of the bubble candidate area so that it surrounds both the boundary and the shadow of the bubble. According to paragraph 1,
The bubble candidate area extraction step is,
Setting a preliminary candidate area surrounding the shadow of the bubble in the high-intensity binarized image;
Applying the set preliminary candidate area to the low-light binarized image to check whether a bubble boundary exists inside the preliminary candidate area;
If a bubble boundary exists inside the preliminary candidate area applied to the low-light binarization image, determining the preliminary candidate area as the bubble candidate area.
Bubble inspection method comprising: According to clause 6,
In the step of setting a preliminary candidate area surrounding the shadow of the bubble, a shaded area surrounding the edge of the part estimated to be shaded is set, and then the shaded area is enlarged by a set multiple to determine it as a preliminary candidate area,
The step of determining the preliminary candidate area as the bubble candidate area includes reducing the area determined as the bubble candidate area to the minimum area surrounding the boundary and shadow of the bubble. In a device for inspecting bubbles in a conformally coated substrate,
an image acquisition unit that acquires a low-illuminance image taken of the substrate at a first illuminance level and a high-illuminance image taken at a second illuminance level higher than the first illuminance level;
A low-light image binarization unit that binarizes a low-light image;
A high-intensity image binarization unit that binarizes a high-intensity image;
A bubble rear area extraction unit that extracts an area where the boundary area of the bubble and the shaded area of the bubble appear together as a bubble candidate area using the low-level binarized image and the high-level binarized image;
Verification unit that verifies whether the bubble candidate area is a bubble using a deep learning model
Bubble inspection device comprising: According to clause 8,
The low-light image binarization unit is a bubble inspection device characterized in that it converts the low-light image into a grayscale image and then applies an adaptive threshold to the grayscale image to generate a low-light binarization image. According to clause 8,
The high-intensity image binarization unit is a bubble inspection device characterized in that it converts the high-intensity image into an inverse image and then applies an adaptive threshold to the inverse image to generate a high-intensity binarization image. According to clause 8,
The bubble candidate area extraction unit sets a preliminary candidate area surrounding the boundary of the bubble in the low-illuminance binarized image, applies the set preliminary candidate area to the high-intensity binarized image, and checks whether the shadow of the bubble exists inside the preliminary candidate area, A bubble inspection device that determines the preliminary candidate area as a bubble candidate area if the shadow of a bubble exists inside the preliminary candidate area applied to the high-intensity binarization image. According to clause 8,
The bubble candidate area extraction unit sets a preliminary candidate area surrounding the shadow of the bubble in the high-intensity binarized image, applies the set preliminary candidate area to the low-illuminance binarized image, and checks whether the boundary of the bubble exists inside the preliminary candidate area. A bubble inspection device that determines the preliminary candidate area as a bubble candidate area if a bubble boundary exists inside the preliminary candidate area applied to the low-light binarized image.