KR102172768B1 - System for inspecting substrate defects by bubbles and Method for inspecting substrate defects by bubbles - Google Patents

Abstract

In an embodiment of the present invention, a method for inspecting substrate defects due to bubbles for inspecting defects of a substrate due to bubbles generated during coating of a substrate in a photographed image of a substrate, (A) in the deep learning defect determination unit, Step of learning the bubble-existing image and the bubble-free image as learning data; (B) In the image processing unit, a point where a plurality of photographed images are predicted as bubbles in the substrate image generated by the image combination is searched as a candidate position, and the candidate position Extracting a candidate region for a from the substrate image and outputting the ROI image; (C) In the deep learning defect determination unit, the input ROI image is deep-learned based on pre-stored learning data, and the presence or absence of bubbles in the ROI image is determined, and the result image displaying the bubble defect area in the ROI image is output. step; And (D) in the feature extraction-based inspection unit, a feature value for the bubble defect region is calculated according to a preset base inspection algorithm in the result image, and if the feature value falls within a preset defect determination range, it is finally judged as “defective”, If the feature value is not included in the defective determination range, it is preferable to include the step of final determination as "normal".

Description

System for inspecting substrate defects by bubbles and Method for inspecting substrate defects by bubbles}

The present invention relates to a substrate defect inspection system due to bubbles and a method for inspecting substrate defects due to bubbles, and in detail, only a portion of the substrate image predicted as a bubble is extracted as an ROI image, and the presence or absence of bubbles from the ROI image by a deep learning method. It is an object of the present invention to provide a substrate defect inspection system by bubbles and a method of inspecting substrate defects due to bubbles, by which the determination and derivation of the bubble defect area are performed.

In general, a coating solution is applied to a printed circuit board (PCB) on which electronic components are mounted. This is to protect from moisture, dust, high temperature environment, electromagnetic waves, etc. through coating because the printed circuit board on which the electronic component is mounted is sensitive to the surrounding environment. However, when coating, air bubbles may occur depending on the temperature of the PCB surface, foreign matter, ambient humidity, and the mounting condition of the component.

Bubbles present in the coating solution coated on the printed circuit board and solidified may swell when the temperature increases, causing the coating solution to fall off. In addition, peeling of the coating solution may cause damage to electronic components due to moisture and may cause malfunction of a device on which a printed circuit board is mounted. In addition, a coating thickness of less than an appropriate level may cause the electronic component mounted on the printed circuit board to be removed due to vibration.

Conventionally, the coating state of a printed circuit board has been inspected using machine vision, but there is a limitation in technically precise inspection.

For example, in the case of a pattern of similar shape with a similar change in brightness on the solder surface, especially the hole, it is difficult to distinguish it from air bubbles with machine vision technology, so all detected items are displayed on the NG viewer and the operator determines it later. There was an inconvenience of becoming a system (Semi-Auto system) or requiring an operator to visually inspect it once more.

If the inspection is conducted based on deep learning, which has been recently studied, there is an advantage in that it is possible to accurately detect defects of a printed circuit board on which electronic components are mounted. However, the cost required to construct the deep learning inspection system is high, and the inspection speed is very slow in the case of inspecting the defects of the substrate using only the deep learning method, and it is difficult to apply the deep learning method to processes that require high speed. . In particular, the defect inspection of the substrate according to the deep learning method has a disadvantage in that it cannot cope with the case where the inspection conditions need to be finely changed according to the process situation.

Korean Patent Registration No. 10-1848173 discloses a printed circuit board coating inspection apparatus and method.

The present invention extracts only the portion of the substrate image predicted as a bubble as an ROI image, and determines the presence or absence of a bubble from the ROI image and derivation of the bubble defect area from the ROI image by a deep learning method, so that the defect of the board due to the bubble can be quickly inspected. It is an object of the present invention to provide a substrate defect inspection system due to possible air bubbles and a method for inspecting substrate defects due to air bubbles.

In addition, the present invention performs a feature extraction-based inspection reflecting the process conditions using the result image in which the bubble defect area is displayed on the ROI image, thereby intensively inspecting the problem area and at the same time, the substrate caused by bubbles in accordance with the process conditions. It is an object of the present invention to provide a substrate defect inspection system and a method for inspecting defects due to bubbles, which can inspect the defects of

In an embodiment of the present invention, a substrate defect inspection system due to bubbles for inspecting defects of a substrate due to bubbles generated during coating of a substrate in a photographed image in which a substrate is photographed, a plurality of photographed images is generated by image combination. An image processing unit in which a point predicted as a bubble in the resulting substrate image is searched as a candidate position, and a candidate region for the candidate position is extracted from the substrate image and output as an ROI image; A deep learning defect determination unit that determines the presence or absence of bubbles in the ROI image while the input ROI image is deep-learned based on the previously stored learning data, and outputs a result image in which the bubble defect region is displayed on the ROI image; And the input result image is calculated as a feature value for the bubble defect region according to a preset base inspection algorithm, and if the feature value falls within the preset defect determination range, it is finally judged as "defective", and the feature value falls within the defect determination range. If not included, it is preferable to include a feature extraction-based inspection unit that is finally judged as "normal".

In one embodiment of the present invention, the deep learning defect determination unit includes: a common network in which the ROI image is normalized in the form of neural network input data, down-sampled, and converted into a feature map; A decision network in which the input feature map is deep-learned based on pre-stored learning data, and the presence or absence of bubbles is determined; And the feature map is up-sampled to the size of the ROI image and converted into an ROI image for reading, and when the bubble defect region is derived while the ROI image for reading is deep-learned based on the learning data, the bubble defect region is found in the ROI image for reading. It is preferable that a region division network in which the displayed result image is output is included, and the ROI image for reading is an ROI image size, and each pixel is binarized into 0 and 1 according to a preset condition.

In one embodiment of the present invention, in the common network and the judgment network, the bubble-existing image and the bubble-free-existing image are learned as learning data, and the domain division network learns only the bubble-existing image as learning data, and the bubble-existing image is learned from the judgment network. It is preferable that an image determined to have air bubbles is included.

In an embodiment of the present invention, the image processing unit includes: a substrate image generator configured to generate a substrate image by combining a plurality of photographed images photographed under illumination of different wavelengths according to a preset image processing algorithm; A candidate location search unit for searching for a candidate location in the substrate image and calculating a list listed in the searched candidate location according to the image processing algorithm; And an ROI image generator that calculates a candidate region based on the candidate location as a center point, extracts the candidate region from the substrate image, and outputs the ROI image according to the list.

On the other hand, in an embodiment of the present invention, a method for inspecting substrate defects due to bubbles for inspecting defects of a substrate due to bubbles generated during coating of a substrate in a photographed image in which the substrate is photographed includes: (A) a deep learning defect determination unit In the step of learning the bubble-existing image and the bubble-free image as learning data; (B) In the image processing unit, a point where a plurality of photographed images are predicted as bubbles in the substrate image generated by the image combination is searched as a candidate position, Extracting the candidate region for the candidate position from the substrate image and outputting the ROI image; (C) In the deep learning defect determination unit, the input ROI image is deep-learned based on pre-stored learning data, and the presence or absence of bubbles in the ROI image is determined, and the result image displaying the bubble defect area in the ROI image is output. step; And (D) in the feature extraction-based inspection unit, a feature value for the bubble defect region is calculated according to a preset base inspection algorithm in the result image, and if the feature value falls within a preset defect determination range, it is finally judged as “defective”, If the feature value is not included in the defective determination range, it is preferable to include the step of final determination as "normal".

In an embodiment of the present invention, step (B) includes: (B1) generating a substrate image by a combination of a plurality of photographed images photographed under illumination of different wavelengths according to a preset image processing algorithm; (B2) searching for candidate positions in the substrate image according to the image processing algorithm, and calculating a list listed in each candidate position; And (B3) according to the list, a candidate region having a preset size is calculated using the candidate position as the center point, and if a candidate position is randomly selected from the list according to the preset overlapping region removal algorithm, the selection candidate for the selected candidate position Searching for an unselected candidate position within the overlapping range of the region, and removing the searched candidate position from the list; And (B4) extracting a candidate region from the substrate image and outputting an ROI image according to the list after step B3.

In an embodiment of the present invention, it is preferable that the overlapping range is preset in the image processing unit within a range of 1/2 of the area of the selected candidate region, with the selected candidate position as the center point.

In an embodiment of the present invention, in step C, (C1) an ROI image is input to a common network, the ROI image is normalized in the form of neural network input data, down-sampled, and converted into a feature map. step; (C2) determining the presence or absence of bubbles while a feature map is input from the common network to the decision network, and the feature map is deep-learned based on the learning data; And (C3) a feature map is input from the common network to the area division network, the feature map is converted into an up-sampled ROI image for reading to the image size input to the common network, and the ROI image for reading is based on the learning data When the bubble defect area is derived during deep learning, the step of outputting the bubble defect area as a result image displayed on the reading ROI image is included, and in the reading ROI image, each pixel of the ROI image is set to 0 and 1 according to a preset condition. The binarization treatment is preferred.

In an embodiment of the present invention, it is preferable that steps C2 and C3 are processed in parallel after step C1, so that the determination of the presence or absence of bubbles and derivation of the bubble defect regions are performed separately.

In an embodiment of the present invention, in step C2, it is preferable that the determination network is deep-learning the inputted plurality of feature maps in a parallel structure, so that the determination of the presence or absence of bubbles in the plurality of feature maps is performed collectively. Do.

In one embodiment of the present invention, in step C3, the region division network deep-learns a plurality of input ROI images for reading in a parallel structure, so that the derivation of bubble defect regions in a plurality of reading ROI images is collectively performed. It is preferably carried out.

In an embodiment of the present invention, in step C, in the common network and the determination network, the bubble-existing image and the bubble-existing image are learned as learning data, and the domain division network learns only the bubble-existing image as learning data, and the bubble-existing image. It is preferable that the phase which is judged to have air bubbles in the judgment network is included.

The present invention extracts only the portion of the substrate image predicted as a bubble as an ROI image, and determines the presence or absence of a bubble from the ROI image and derivation of the bubble defect area from the ROI image by a deep learning method, so that the defect of the board due to the bubble can be quickly inspected. I can.

In addition, the present invention performs a feature extraction-based inspection reflecting the process conditions using the result image in which the bubble defect area is displayed on the ROI image, thereby intensively inspecting the problem area and at the same time, the substrate caused by bubbles in accordance with the process conditions. You can check the defects.

That is, the present invention is a process in which defect inspection of a substrate due to bubbles is performed, a process of generating only a portion predicted as a bubble in a substrate image as an ROI image, a process of determining the presence or absence of bubbles from the ROI image, and a process of deriving a bubble defect area, and finally , As the result image with the bubble defect area displayed on the ROI image is subdivided into the process of final judgment of the defect of the substrate due to the bubble in accordance with the process conditions, the result image is carried out by feature extraction-based inspection. Instead of inspecting, substrate defect inspection is performed on a portion where defects are predicted, so that defects of the substrate due to air bubbles can be quickly and accurately inspected.

In the present invention, as an ROI image having a small data capacity compared to a substrate image is deep-learned, it is possible to improve deep learning work efficiency compared to a deep-learning of the substrate image, and at the same time, the system cost of implementing the deep learning defect reading unit. Can be implemented at low cost.

In addition, the present invention is reflected in the process conditions during the feature extraction-based inspection of the resultant image in which the bubble defect area is displayed, and as it is determined whether the substrate is defective due to the bubble, the substrate judged as a substrate defect by deep learning , It is not necessary for the operator to additionally determine the defect of the substrate with the naked eye by reflecting the process conditions, so that the defect of the substrate due to air bubbles can be automatically implemented.

1 and 2 are views for explaining a system for inspecting substrate defects due to air bubbles according to an exemplary embodiment of the present invention.
3 is a flowchart of a method for inspecting substrate defects due to air bubbles according to an embodiment of the present invention.
4 is an image processing algorithm according to an embodiment of the present invention.
5 is an algorithm for removing an overlapped area according to an embodiment of the present invention.
6 is a view for explaining a process of extracting an ROI image from a substrate image in an embodiment of the present invention.
7 is a diagram illustrating a process of determining a defect of a substrate due to bubbles from an ROI image in an embodiment of the present invention.
8 is a diagram for explaining a process in which an ROI image is converted into input data of a neural network and applied to a decision network and a region division network in an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, a description will be made of a substrate defect inspection system and a substrate defect inspection method due to bubbles according to a preferred embodiment of the present invention.

The substrate defect inspection system 100 due to bubbles according to an embodiment of the present invention is to implement the purpose of quickly and accurately inspecting the defects of the substrate due to bubbles. To this end, the present invention extracts only the portion of the substrate image that is predicted to be where bubbles exist as an ROI image, and determines the presence or absence of bubbles from the ROI image by a deep learning method, derivation of the bubble defect area, and the bubble defect area. Subdivided into feature extraction-based inspection, substrate defect inspection is performed by air bubbles, so that substrate defect inspection is performed on parts where defects are predicted, rather than inspection of substrate defects due to bubbles for the entire substrate image. Is composed.

1 and 2, the substrate defect inspection system 100 due to bubbles according to an embodiment of the present invention includes an image processing unit 110, a deep learning defect determination unit 120, and a feature extraction-based inspection unit 130. And a data storage unit 140.

The image processing unit 110 searches for a point predicted as a bubble in the substrate image as a candidate position, and each candidate region for each candidate position is extracted from the substrate image to generate an ROI image. Referring to FIG. 2, the image processing unit 110 includes a substrate image generation unit 111, a candidate location search unit 113, and an ROI image generation unit 115.

Before the image processing unit 110 generates the substrate image, several photographed images in which the substrate is photographed by the machine vision unit 160 are acquired. The machine vision unit 160 includes a lighting 161, a camera 165 and a lighting control unit 163. The machine vision unit 160 photographs a substrate moving along the substrate moving unit 150.

The plurality of photographed images are images photographed by the camera 165 when different wavelengths are irradiated. The illumination 161 is controlled by the illumination control unit 163, and irradiates different wavelengths to the substrate. The camera 165 photographs the substrate when the illumination 161 is irradiated onto the substrate according to the wavelength preset in the illumination control unit 163. The plurality of photographed images are provided from the machine vision unit 160 to the substrate image generation unit 111.

The substrate image generation unit 111 generates a substrate image by image processing a plurality of photographed images. In the present embodiment, the substrate image is formed by logically or arithmetically combining a plurality of photographed images photographed by the substrate having different wavelengths of illumination 161 at the same location, thereby expressing bubbles. For example, any one of the plurality of photographed images is a photographed image taken when a UV wavelength is irradiated onto the substrate. In addition, the other photographed image is photographed at at least one wavelength of R beam, G beam, and B beam.

The candidate location search unit 113 searches for candidate locations in the substrate image according to a preset image processing algorithm, and calculates a list listed in each candidate location.

According to the list, the ROI image generator 115 calculates a candidate region having a predetermined size based on the candidate position as a center point, and extracts each candidate region from the substrate image to generate an ROI image. Here, the preset size is the size of the ROI image. ROI stands for Region Of Interest.

The deep learning defect determination unit 120 determines the presence or absence of bubbles while deep learning based on the learning data previously stored in the ROI image, and when the bubble defect area is derived, the resulting image displaying the bubble defect area on the ROI image is output.

The deep learning defect determination unit 120 is composed of a common network 121, a determination network 123, and an area division network 125. In the common network 121 and the determination network 123, the bubble presence image and the bubble absence image are learned as learning data. The region division network 125 learns only the bubble-existing image as the learning data, and the bubble-existing image includes the one determined by the determination network 123 to have a bubble.

In the common network 121, the ROI image input from the image processing unit 110 is normalized in the form of neural network input data. Thereafter, the ROI image is down-sampled and converted into a feature map. The feature map is output from the common network 121 and input to the decision network 123 and the area division network 125, respectively.

In this embodiment, the normalization process of the ROI image is to pre-process the image before the ROI image is input to the decision network 123 and/or the region division network 125 which is a neural network. In the normalization process of the ROI image, the image mean is 0 or close to 0, and the data distribution is properly distributed in positive and negative numbers, so that bubbles can be well expressed. For example, it can be normalized by subtracting or dividing the image mean or by calculating the Z-Score.

The decision network 123 determines only the presence or absence of bubbles from the feature map in a deep learning method. The feature map input from the common network 121 to the determination network 123 is deep-learned based on the previously stored learning data to determine the presence or absence of bubbles.

For example, when the input feature map matches the feature map of the bubble presence image, the determination network 123 determines that a bubble exists. Alternatively, when the input feature map matches the feature map of the bubble-free image, the determination network 123 determines that there is no bubble.

That is, the determination network 123 determines only the presence or absence of bubbles in the input feature map by the deep learning method. What is determined as the presence of bubbles in the decision network 123 is stored in the data storage unit 140 to be used as learning data. Then, the determination network 123 treats the determination as the absence of air bubbles as garbage data.

The area division network 125 derives a substrate defect area from a feature map in a deep learning method. The feature map input from the common network 121 to the area division network 125 is converted into an up-sampled ROI image for reading to the size of the ROI image input to the common network 121, and the ROI image for reading is the learning data. Based on the deep learning, when the bubble defect area is derived, the result image displaying the bubble defect area on the reading ROI image is output. In the ROI image for reading, each pixel of the ROI image is binarized into 0 and 1 according to a preset condition.

The feature extraction-based inspection unit 130 extracts the features of the bubble defect region as the result image is image-processed according to a preset base inspection algorithm, and determines whether there is a final defect on the substrate. BLOB algorithm can be used as the base inspection algorithm.

Hereinafter, a method for inspecting substrate defects due to bubbles will be described with reference to FIGS. 2 to 8.

The method for inspecting substrate defects due to bubbles according to an exemplary embodiment of the present invention is a method of inspecting defects in a substrate due to bubbles generated during coating of a substrate in a substrate image in which a substrate is photographed.

First, in the deep learning defect determination unit 120, the bubble-existing image and the bubble-free image are learned as learning data (step A). Here, the bubble presence image and the bubble absence image are photographs of the coated substrate.

Next, the substrate is photographed using a machine vision method. A plurality of photographed images photographed with illumination 161 of different wavelengths on the substrate at the same location are provided to the image processing unit 110, and the ROI image is generated by the image processing unit 110 (step B).

According to the image processing algorithm according to FIG. 4, the image combination and noise are removed from the substrate image generator 111, and a plurality of photographed images are converted into one substrate image. As described above, in the substrate image, a plurality of photographed images photographed by the substrate having different wavelengths of illumination 161 at the same position are logically or arithmetically combined, thereby expressing bubbles.

Subsequently, a point predicted as a bubble in the substrate image is searched as a candidate position by the candidate position search unit 113. In the candidate location search unit 113, a list listing each candidate location is calculated.

In the ROI image generator 115, a candidate region having a predetermined size is calculated based on the candidate position as a center point according to the list.

Thereafter, when a candidate position is randomly selected from the list according to the overlapping region removal algorithm preset in the ROI image generator 115 (see Fig. 5), an unselected candidate position within the overlapping range of the selected candidate region for the selected candidate position Is searched, and the searched candidate position is removed from the list. The overlapping range is preset in the ROI image generation unit 115 within a range of 1/2 of the area of the selected candidate region, using the selected candidate position as the center point.

As shown in FIG. 5, the overlapping region removal algorithm is an algorithm that repeatedly removes and checks whether there is another candidate position at a position close to 1/2 of the area of the candidate region to be inspected. Subsequently, the selected candidate position, which has become a reference, is set as a flag as a previously calculated position so as not to be affected when calculating other candidate positions. The overlapping area elimination algorithm is repeated until all candidate positions listed in the list have been calculated.

Thereafter, the ROI image generation unit 115 cuts a candidate region of a predetermined size from the substrate image using the candidate position as the center point according to the list to generate the ROI image. For example, the ROI image size is selected so that it does not have an odd number during an intermediate process in which the image size is downsampled. For example, assuming that the ROI image size is 176 and downsampling is performed four times, the ROI image size in each step is 176->88->44->22->11, and all intermediate steps from the start are even-numbered. Able to know. Alternatively, assuming that the ROI image size is 112 and downsampling is performed four times, the image size is reduced to 112->56->28->14->7 (see FIG. 8). Referring to FIG. 8, an ROI image having an input ROI image size of 112×112 is image-processed in the common network 121 and converted into a feature map having an image size of 7×7.

A process of extracting an ROI image from a substrate image will be described with reference to FIG. 6 as follows.

As shown in FIG. 6(a), the substrate image is a point where a bubble is predicted by a combination of a plurality of photographed images as described above. Numbers 1 to 6 shown in Fig. 6(a) indicate candidate positions. The candidate position can be calculated by the BLOB algorithm. The candidate position may be a bubble or a hole. As explained in the background art, a hole has a pattern of a similar shape with a similar change in brightness on the soldering surface compared to the pattern of air bubbles, so it is difficult to distinguish it with the image processing method by machine vision, so it is searched as a candidate position. Is likely to be.

Thereafter, as shown in Fig. 6(b), a candidate region is derived with a predetermined size using each candidate position as a center point. In the present invention, for convenience of description, a candidate region is referred to as follows. Candidate 1 (11A) is an area having a predetermined size based on candidate position 1 as a center point. Candidate regions 2 (12A) to 6 (16A) are regions having a predetermined size based on each of the candidate positions 2 to 6 as a center point.

Referring to FIG. 6B, when a candidate region is derived for each candidate position, a case of overlapping between the candidate regions may occur. In this example, candidate area 2 (12A) and candidate area 3 (13A) are overlapped, and candidate area 4 (14A) and candidate area 5 (15A) are overlapped.

Thereafter, as shown in FIG. 6(c), it is performed in the process of removing the overlapped candidate regions according to the overlapping region removal algorithm. Referring to FIG. 6(c), the process of performing the overlapping region removal algorithm will be described. When a candidate position 1 listed in the list is selected, another candidate within the area 11B of 1/2 of the candidate region 1 11A Whether a location exists is searched, and if no other candidate location is searched, candidate location 1 is set to the list as a searched candidate location.

When the candidate position 2 listed in the list is selected, it is searched for whether another candidate position exists within the range of 1/2 of the candidate area 2 12A. In this process, if another candidate location, which is a candidate location 3, is searched within the 1/2 area 12B of the candidate area 2 (12A), the candidate location 3 is removed from the list. This process is repeated until no other candidate positions are searched within 1/2 of the candidate region.

Thereafter, the above-described process is repeatedly performed for the candidate positions 4 to 6 listed in the list, and a candidate region as shown in FIG. 6(d) is determined. Even if candidate regions overlap like candidate regions 4 (14A) and candidate region 5 (15A), for example, if candidate position 4 does not fall within the area (15A) of 1/2 of candidate region 5 (15A), candidate region 4 ( Even if both candidate positions 4 and 5 are present in 14A) and both candidate positions 4 and 5 are present in candidate region 5 (15A), they are calculated as separate candidate regions. In this embodiment, reference numerals 11A to 16A designate candidate regions, and reference numerals 11B to 16B designate a half area range of each candidate region.

6(e) shows an ROI image from which each candidate region is extracted from the substrate image. In this embodiment, for convenience of explanation, an image in which candidate position 1 is the center point is referred to as ROI image 1 (21), an image in which candidate position 2 is the central point is referred to as ROI image 2 (22), and candidate position 4 is The image as the central point is referred to as ROI image 3(23), the image in which candidate position 5 is the central point is referred to as ROI image 4(24), and the image in which candidate position 6 is the central point is referred to as ROI image 5(25).

Thereafter, ROI image 1 (21) to ROI image 5 (25) are input to the deep learning defect determination unit 120, and the presence or absence of bubbles or bubble defect regions are displayed as a result image (step C).

First, referring to FIG. 7, ROI image 1 (21) to ROI image 5 (25) are input to the common network 121. In the common network 121, ROI images 1 (21) to ROI images 5 (25) are normalized in the form of neural network input data, down-sampled, and transformed into a feature map. In this embodiment, for convenience of explanation, the feature map for the ROI image 1 (21) is referred to as a feature map 1 (31), and the feature map for the ROI image 2 (22) is referred to as the feature map 2 (32). do. In addition, the feature maps for the ROI image 3 (23) to the ROI image 5 (25) are referred to as feature maps 3 to 5 (33, 34, 35), respectively.

Referring to FIG. 7, feature maps 1 to 5 (31, 32, 33, 34, 35) are input to the decision network 123 and the area division network 125, respectively.

The decision network 123 is deep-learning based on the learning data previously stored for the feature maps 1 (31) to the feature maps 5 (35) input from the common network 121 to determine the presence or absence of bubbles. The determination network 123 learns the bubble-existing image and the bubble-free image as learning data.

The decision network 123 deep-learns the inputted plurality of feature maps in a parallel structure, and collectively determines the presence or absence of bubbles in the plurality of feature maps. That is, the decision network outputs result values for each feature map while the feature maps 1 (31) to 5 (35) are simultaneously deep-learned based on the learning data. In the judgment network, one of two values of "normal" or "defective" is output as a result value according to the presence or absence of bubbles in the input feature map. In addition, it is preferable that the decision network 123 is designed so that even if two candidate positions exist in one ROI image, if even one bubble exists as a result of the deep learning, it is determined to be defective.

For example, the decision network 123 determines whether the input feature map 1 (31) to feature map 5 (35) has a feature that is closer to any of the bubble-existing image or the bubble-free image. The presence or absence of air bubbles for (35) is judged. In the present embodiment, for convenience of explanation, it is assumed that holes exist in candidate positions 1, 3 and 5, and bubbles exist in candidate positions 2, 4 and 6.

When the feature map 1 (31) is deep-learned based on the learning data and matches the feature map of the bubble-free image, the decision network 123 outputs a result value as "normal" to the feature map 1 (31). In addition, when the decision network 123 and DPT feature map 2 (32) are matched with the feature map of the bubble-existing image while deep learning based on the learning data, a result value is calculated as "defective" for the feature map 2 (32). . Such a process is performed collectively for the feature map 3 to the feature map 5 (33, 34, 35) in the decision network 123. As a result, feature map 3 (33) in which candidate location 4 exists is "bad", feature map 4 (34) in which candidate location 5 exists is "normal", and feature map 5 (35) in which candidate location 6 exists ) Is calculated as "bad".

On the other hand, the region division network 125 converts the feature map input from the common network 121 into an ROI image for reading, and deep-learns the ROI image for reading based on the previously stored learning data to derive a bubble defect region. The region division network 125 learns only the bubble-existing image as the learning data, and the bubble-existing image includes the image determined by the determination network 123 that the bubble exists.

Specifically, feature maps 1 (31) to feature maps 5 (35) are input from the common network 121 to the area division network 125. The feature map is converted into an ROI image for reading that is up-sampled to the size of the ROI image input to the common network 121. Referring to FIG. 8, in the area division network 125, a feature map of a 7×7 image size is upsampled to a read ROI image of a 112×112 ROI image size. In the ROI image for reading, each pixel of the ROI image is binarized into 0 and 1 according to a preset condition.

In this embodiment, for convenience of explanation, the image in which the feature map 1 (31) is up-sampled is referred to as a reading ROI image 1 (41), and the image in which the feature map 2 (32) is up-sampled is referred to as a reading ROI image. Referred to as 2(42). In the same manner, an image in which the feature maps 3 to 5 (33, 34, 35) are up-sampled is referred to as a reading ROI image 3 to ROI image 5 (43, 44, 45).

And, in the region division network 125, the reading ROI image 1 to the reading ROI image 5 (41, 42, 43, 44, 45) are deep-learned based on the learning data. In this embodiment, the reading ROI image 1 to the reading ROI image 5 (41, 42, 43, 44, 45) are deep-learned in a parallel structure in the area division network 125, and the reading ROI image 1 to the reading The bubble defect regions in each of the ROI images 5 (41, 42, 43, 44, 45) are collectively derived. Accordingly, the present invention can improve the inspection speed for deriving the bubble defect area.

Subsequently, when the bubble defect area is derived from the read ROI image 1 to the read ROI image 5 (41, 42, 43, 44, 45), the area division network 125 outputs a result image displaying the bubble defect area. The resulting image is image-processed so that the bubble defect area is distinguished from the surrounding area in the reading ROI image. Exemplarily, the resulting image is a bubble defect area displayed in a different color from the surroundings on the reading ROI image.

The region division network 125 is deep-learned based on the bubble-existing image, which is the learning data previously stored for the feature map 1 (31), and if the candidate location 1 (hole) does not match the bubble-existing image, it is treated as garbage data. This is because candidate position 1 alone does not correspond to the defect of the substrate.

The area division network 125 is deep-learning based on the learning data on the feature map 2 (32), and when the candidate location 2 (bubble) matches the bubble-existing image, the part for the candidate location 2 is displayed as a bubble defect area. do. For example, in the ROI image for reading on the feature map 2 (32), a portion for the candidate position 2 and the remaining portion are displayed as a substrate defect area for the candidate position 2 so that the portion for the candidate position 2 and the rest are distinguished.

This process is also performed for feature maps 3 to 5 (33, 34, 35). The feature map 3 (33) and the feature map 5 (35) are data in which bubbles are present, and the ROI image for reading on the feature map 3 (33) is displayed as a substrate defect region in a corresponding portion at the candidate position 4. In addition, in the ROI image for reading on the feature map 5 (35), a portion corresponding to the candidate position 6 is displayed as a substrate defect area. Feature map 4 (34) is data with holes, and feature map 3 to feature map 5 (33, 34, 35) for feature map 4 (34) feature map 3 to feature map 5 (33, 34, 35) Prizes are treated as garbage data.

The domain division network 125 outputs a result image for each feature map while deep learning at the same time based on the learning data in a parallel manner for feature maps 1 (31) to feature maps 5 (35). That is, with respect to the feature map 2 (32), feature map 3 (33), and feature map 5 (35) determined to have bubbles, each of the result images in which the bubble defect regions are displayed is output. On the other hand, the feature map 1 (31) and the feature map 4 (34) determined to be free of air bubbles are garbage data, and the result image is not output.

As the present invention is designed to subdivide a configuration for determining input data by a deep learning method into a judgment network and an area division network 125, and the judgment network and the area division network 125 share a common network 121 at the front end, , It is possible to improve the speed of determining the input data. At the same time, the present invention simplifies the configuration of a network and enables deep learning to be performed at low cost.

Meanwhile, the result image is input to the feature extraction-based inspection unit 130. In the feature extraction-based inspection unit 130, the resulting image is image-processed according to a preset base inspection algorithm, and a feature value for the bubble defect region is calculated.

The feature extraction-based inspection unit 130 calculates a feature value for the bubble defect region according to a preset base inspection algorithm in the result image, and if the feature value falls within the preset defect determination range, it is finally judged as "defective", and the feature value If it is not included in this defect judgment range, it is finally judged as "normal". Here, the defect determination range reflects the process conditions according to the substrate and is adjusted according to the work process.

The base inspection algorithm is an algorithm that quantifies the result image, and the BLOB algorithm may be applied, but is not limited thereto, and various algorithms may be applied as long as it is an algorithm capable of quantifying an image.

By the process as described above, the present invention extracts only the part predicted to be bubbles in the substrate image as an ROI image, and determines the presence or absence of bubbles from the ROI image and derivation of the bubble defect area from the ROI image by a deep learning method. Can be quickly inspected for defects.

In addition, the present invention performs a feature extraction-based inspection reflecting the process conditions using the result image in which the bubble defect area is displayed on the ROI image, thereby intensively inspecting the problem area and at the same time, the substrate caused by bubbles in accordance with the process conditions. You can check the defects.

That is, the present invention is a process in which defect inspection of a substrate due to bubbles is performed, a process of generating only a portion predicted as a bubble in a substrate image as an ROI image, a process of determining the presence or absence of bubbles from the ROI image, and a process of deriving a bubble defect area , As the result image with the bubble defect area displayed on the ROI image is subdivided into the process of final judgment of the defect of the substrate due to the bubble in accordance with the process conditions, the result image is carried out by feature extraction-based inspection. Instead of inspecting, substrate defect inspection is performed on a portion where defects are predicted, so that defects of the substrate due to bubbles can be quickly and accurately inspected.

In the present invention, as an ROI image having a small data capacity compared to a substrate image is deep-learned, it is possible to improve deep learning work efficiency compared to a deep-learning of the substrate image, and at the same time, the system cost of implementing the deep learning defect reading unit. Can be implemented at low cost.

In addition, according to the present invention, after the bubble defect area is derived by the deep learning method, when the feature extraction-based inspection of the result image in which the bubble defect area is displayed is reflected in the process condition, it is determined whether or not the substrate is defective due to bubbles, For a substrate determined as a substrate defect by deep learning, the operator does not need to additionally determine the defect of the substrate by visually reflecting the process conditions, so that the defect of the substrate due to bubbles can be automatically implemented.

That is, in the present invention, it is possible to determine in detail whether the air bubbles are a problem to the overall defect of the substrate or whether the air bubbles are present but do not cause a problem of the defect of the substrate. Accordingly, the present invention can quickly and accurately inspect the defect of the substrate due to air bubbles.

The present invention described above is not limited to the above-described embodiments and the accompanying drawings, and various substitutions, modifications and changes are possible within the scope of the technical spirit of the present invention. It will be obvious to those who have the knowledge of.

100: board defect inspection system due to air bubbles
110: image processing unit 111: substrate image generation unit
113: candidate location search unit 115: ROI image generation unit
120: deep learning defect determination unit 121: common network
123: decision network 125: area division network
130: feature extraction-based inspection unit 140: data storage unit
150: substrate moving unit 160: machine vision unit
161: lighting 163: lighting control unit
165: camera

Claims (12)

In the substrate defect inspection system due to bubbles for inspecting the defect of the substrate due to bubbles generated during coating of the substrate in a photographed image of the substrate,
An image processing unit in which a point predicted as the bubble in the substrate image generated by the image combination of the plurality of photographed images is searched as a candidate position, and a candidate region for the candidate position is extracted from the substrate image and output as an ROI image;
A deep learning defect determination unit configured to determine the presence or absence of the bubbles in the ROI image while the input ROI image is deep-learning based on pre-stored learning data, and output a result image in which the bubble defect region is displayed on the ROI image; And
In the input result image, a feature value for the bubble defect region is calculated according to a preset base inspection algorithm, and if the feature value falls within a preset defect determination range, it is finally judged as "defective", and the feature value is the It includes a feature extraction-based inspection unit that is finally judged as "normal" if it is not included in the defective determination range,
The image processing unit,
A substrate image generator configured to generate the substrate image by combining the plurality of photographed images photographed under illumination of different wavelengths according to a preset image processing algorithm;
A candidate location search unit for searching for the candidate location in the substrate image and calculating a list listed in the searched candidate location according to the image processing algorithm; And
According to the list, the candidate region is calculated based on the candidate position as a center point, the candidate region is extracted from the substrate image, and an ROI image generator for outputting the ROI image. Inspection system. The method of claim 1, wherein the deep learning defect determination unit,
A common network in which the ROI image is normalized into neural network input data, down-sampled, and converted into a feature map;
A determination network in which the input feature map is deep-learned based on pre-stored learning data, and the presence or absence of the bubble is determined; And
When the feature map is up-sampled to the size of the ROI image and converted into an ROI image for reading, and the ROI image for reading is deep-learned based on the learning data and a bubble defect region is derived, the ROI image for reading And an area division network in which a result image in which the bubble defect area is displayed is output,
The ROI image for reading is the size of the ROI image, and each pixel is binarized into 0 and 1 according to a preset condition. The method of claim 2,
In the common network and the determination network, a bubble-existing image and a bubble-free image are learned as the learning data,
In the region division network, only the bubble-existing image is learned with the learning data, and the bubble-existing image includes an image determined by the determination network as the existence of the bubble. delete In the substrate defect inspection method due to bubbles for inspecting the defect of the substrate due to bubbles generated during coating of the substrate in a photographed image of the substrate,
(A) in the deep learning defect determination unit, the step of learning the bubble-existing image and the bubble-free image as learning data;
(B) In the image processing unit, a point predicted as the bubble in the substrate image generated by the image combination of a plurality of the photographed images is searched as a candidate position, and a candidate region for the candidate position is extracted from the substrate image to be ROI Outputting as an image;
(C) In the deep learning defect determination unit, while deep learning the input ROI image based on pre-stored learning data, the presence or absence of the bubble in the ROI image is determined, and the bubble defect region is displayed on the ROI image. Outputting a result image; And
(D) In the feature extraction-based inspection unit, the result image is calculated as a feature value for the bubble defect region according to a preset base inspection algorithm, and if the feature value falls within a preset defect determination range, it is determined as "defective". And, if the feature value is not included in the defective determination range, including the step of finally determining as "normal",
The (B) step,
(B1) generating the substrate image by a combination of a plurality of photographed images photographed under illumination of different wavelengths according to a preset image processing algorithm;
(B2) searching for the candidate positions in the substrate image according to the image processing algorithm, and calculating a list listed in each of the candidate positions;
(B3) According to the list, the candidate region having a preset size is calculated using the candidate position as a center point, and if a candidate position is arbitrarily selected from the list according to a preset overlapping region removal algorithm, Searching for an unselected candidate position within the overlapping range of the selection candidate region for the selected candidate region, and removing the searched candidate position from the list; And
(B4) after the step B3, according to the list, the candidate region is extracted from the substrate image and the ROI image is output. delete The method of claim 5,
And the overlapping range is preset to the image processing unit within a range of 1/2 of the area of the selected candidate region, with the selected candidate position as a center point. The method of claim 5, wherein the step C,
(C1) the ROI image is input to a common network, the ROI image is normalized in the form of neural network input data, down-sampled, and converted into a feature map;
(C2) determining the presence or absence of the bubble while the feature map is input from the common network to the decision network, and the feature map is deep-learned based on the learning data; And
(C3) The feature map is input from the common network to an area division network, the feature map is converted into a read ROI image up-sampled to the image size input to the common network, and the read ROI image is When the bubble defect area is derived while deep learning based on the learning data, the step of outputting the bubble defect area as a result image displayed on the reading ROI image,
The ROI image for reading is characterized in that each pixel of the ROI image is binarized into 0 and 1 according to a preset condition. The method of claim 8,
The step C2 and step C3 are processed in parallel after the step C1 to determine the presence or absence of the bubble and the derivation of the bubble defect area separately. The method of claim 8, wherein in step C2,
The determination network is characterized in that the plurality of input feature maps are deep-learned in a parallel structure, and the determination of the presence or absence of the bubbles in the plurality of feature maps is collectively performed. . The method of claim 8, wherein in step C3,
In the region division network, the plurality of input ROI images for reading are deep-learned in a parallel structure, and the derivation of the bubble defect regions in the plurality of reading ROI images is performed collectively. Board defect inspection method. The method of claim 8, wherein in step C,
In the common network and the determination network, the bubble-existing image and the bubble-free image are learned as the learning data,
Wherein the region division network learns only the bubble-existing image as learning data, and the bubble-existing image includes an image determined by the determination network as the existence of the bubble.

Images