JP2022134614A - Substrate inspection method - Google Patents

Abstract

To provide a substrate inspection method that increases the number of teacher data of defective products to improve reliability of an inspection model.SOLUTION: A substrate inspection method includes: an image processing step for processing each of image data of a plurality of inspection object sections on a substrate to generate defective image data showing that each of the plurality of inspection object sections is defective; a model generation step for learning normal image data showing that the inspection object section is normal and defective image data as teacher data to generate a substrate inspection model; and a determination step for determining whether the substrate is normal or defective by applying image data of the plurality of inspection object sections on the substrate to be inspected to the substrate inspection model.SELECTED DRAWING: Figure 1

Description

The present invention relates to a board inspection method, a board inspection apparatus, and a model generation apparatus.

In the above technical field, Patent Literature 1 discloses a technique of using captured images of past components with similar color distribution tendencies as teacher data when generating inspection logic for new components in circuit board inspection. .

JP-A-2006-292725

However, in the technique described in the above document, there is a limit to the number of teaching data, especially the number of teaching data of defective products, and the reliability of the inspection model cannot be increased.

An object of the present invention is to provide a technique for solving the above problems.

In order to achieve the above object, a board inspection method according to the present invention comprises:
an image processing step of processing image data of each of a plurality of parts to be inspected on a substrate to generate defective image data indicating that each of the parts to be inspected is defective;
a model generation step of learning the normal image data indicating that the inspection target portion is normal and the defective image data as teaching data to generate a substrate inspection model;
a determination step of determining whether the substrate is normal or defective by applying image data of a plurality of inspection target portions on the substrate to be inspected to the substrate inspection model;
including.

In order to achieve the above object, a board inspection apparatus according to the present invention includes:
an image processing unit that processes image data of each of a plurality of inspection target portions on a substrate to generate defective image data indicating that each of the plurality of inspection target portions is defective;
a model generation unit that learns normal image data indicating that the inspection target portion is normal and the defective image data as teacher data to generate a board inspection model;
a determination unit that determines whether the substrate is normal or defective by applying image data of a plurality of inspection target portions on the substrate to be inspected to the substrate inspection model;
with
The model generation unit has a model verification unit that verifies the accuracy of the generated board inspection model, and if the accuracy of the board inspection model is insufficient based on the verification result of the model verification unit, the image processing is performed. The training data generated by the unit is added and learned to generate a new board inspection model.

In order to achieve the above object, the model generation device according to the present invention includes:
an image processing unit that processes image data of each of a plurality of inspection target portions on a substrate to generate defective image data indicating that each of the plurality of inspection target portions is defective;
a model generation unit that learns normal image data indicating that the inspection target portion is normal and the defective image data as teacher data to generate a board inspection model;
with
The model generation unit has a model verification unit that verifies the accuracy of the generated board inspection model, and if the accuracy of the board inspection model is insufficient based on the verification result of the model verification unit, the image processing is performed. The training data generated by the unit is added and learned to generate a new board inspection model.

According to the present invention, the reliability of the inspection model can be improved by increasing the number of teaching data of defective products.

4 is a flow chart showing the procedure of a board inspection method according to the first embodiment; It is a figure explaining the outline|summary of the board|substrate test|inspection which concerns on 2nd Embodiment. FIG. 10 is a diagram illustrating an outline of image data acquisition according to the second embodiment; FIG. 8 is a flow chart showing the flow of the substrate inspection system according to the second embodiment; It is a block diagram which shows the structure of the board|substrate inspection system which concerns on 2nd Embodiment. FIG. 12 is a block diagram showing the configuration of a generation method table according to the second embodiment; FIG. FIG. 11 is a block diagram showing the configuration of an inspection model table according to the second embodiment; FIG. 9 is a flow chart showing a processing procedure of a model generating device according to the second embodiment; 9 is a flow chart showing a processing procedure of the substrate inspection apparatus according to the second embodiment; It is a figure which shows the example of the image data of the board|substrate which concerns on 2nd Embodiment. It is a figure which shows the imaging part which concerns on 2nd Embodiment. 9 is a flowchart showing a processing procedure of an imaging control unit according to the second embodiment; It is a figure which shows the imaging example of the board|substrate by the imaging part which concerns on 2nd Embodiment. It is a figure explaining normal image data and defective image data of a solder image concerning a 2nd embodiment. FIG. 10 is a diagram showing generation of pseudo-failure image data from solder image data according to the second embodiment; FIG. 10 is a diagram showing another generation of pseudo defect image data from solder image data according to the second embodiment; FIG. 9C is a diagram showing another specific example of generating the pseudo-failure image data of FIG. 8C according to the second embodiment; FIG. 10 is a diagram showing input of teacher data used for generating a board inspection model according to the second embodiment; FIG. 11 is a diagram showing an output of a verification result of the board inspection model according to the second embodiment; It is a figure which shows evaluation of the verification result of the model for a board|substrate inspection which concerns on 2nd Embodiment. FIG. 10 is a diagram showing another evaluation of the verification result of the board inspection model according to the second embodiment; It is a block diagram which shows the structure of the board|substrate inspection system which concerns on 3rd Embodiment. It is a figure which shows the structure of the board|substrate inspection system containing the board|substrate inspection apparatus of 4th Embodiment.

BEST MODE FOR CARRYING OUT THE INVENTION Exemplary embodiments of the present invention will be described in detail below with reference to the drawings. However, the components described in the following embodiments are merely examples, and the technical scope of the present invention is not limited to them.

[First embodiment]
A board inspection method 100 as a first embodiment of the present invention will be described with reference to FIG. A board inspection method 100 is a method for inspecting a board based on image data.

As shown in FIG. 1, the board inspection method 100 includes an image processing step S101, a model generation step S102, and a determination step S103. In, each of image data of a plurality of inspection target portions on a substrate is processed to generate defect image data indicating that each of the plurality of inspection target portions is defective. In the model generation step S102, normal image data indicating that the inspection target portion is normal and defective image data are learned as teaching data to generate a substrate inspection model. In determination step S103, it is determined whether the substrate is normal or defective by applying image data of a plurality of inspection target portions on the substrate to be inspected to the substrate inspection model.

According to the present embodiment, the reliability of the board inspection model can be improved by generating the image data representing the defective board and increasing the number of the teacher data of the defective product.

[Second embodiment]
Next, a board inspection apparatus according to a second embodiment of the present invention will be described. In the board inspection apparatus according to the present embodiment, an example of judging whether the "solder" on the board is good or bad will be described. However, the board inspection apparatus of the present embodiment can accurately inspect the pass/fail of other components on the board and the pass/fail of the arrangement on the board in the same way as pass/fail of "solder". can.

<Prerequisite technology>
As disclosed in Japanese Patent Application Laid-Open No. 2002-200003, there is a technique for performing non-defective product determination by taking an image of a solder fillet using a three-color light source color highlight method. However, in the conventional substrate inspection apparatus, since the judgment performance is low, in order to reliably judge defective parts as defective, it is necessary to shift the judgment criteria to the side of normal parts, so it is necessary to judge normal parts as defective. rice field. Therefore, when a normal product is determined to be defective, it is necessary for the user to check and make a determination again. In addition, creating inspection logic requires a great deal of experience and effort on the part of engineers. The problem is that the Many normal products can be produced, but defective products are produced for various reasons, so it is impossible to produce all kinds of defective products. Therefore, there is a possibility that it may not be possible to determine a defective sample, which should be considered as defective, although it is closer to a normal product, using only past images. In particular, for a new component, since there is no previous image, it is not possible to generate an inspection logic with sufficiently high accuracy.

<Overview of board inspection according to the present embodiment>
An outline of board inspection according to the present embodiment will be described with reference to FIG. 2A. In the figure, ◯ is an image of a normal inspection target portion (for example, a portion where an IC chip is mounted) on the substrate, and x is an image of a defective inspection target portion. A picked-up image 201 in FIG. 2 is an image cut out (extracted) from an imaged board. As shown in the captured image 201, most of the captured images are images of normal parts to be inspected, and the number of acquired images of defective parts to be inspected is small. For example, as the yield of substrate products is improved, the number of defective inspection target portions acquired decreases. In this state, the number of defective product images as training data is insufficient, and the accuracy of the board inspection model generated by the learning 205 is lowered.

Therefore, in the present embodiment, as shown in the generated image 202, by changing the image of the inspection target region of the captured image 201, the image of the inspection target region of the pseudo defect is generated. The image of the pseudo-defective inspection target generated from the generated image 202 is added to the image 204 of the defective inspection target portion as teacher data. It should be noted that since the number of acquired images of defective inspection target sites is small, most of the generated images are images of pseudo-defective inspection target sites. Therefore, in order to improve the accuracy of the board inspection model, for example, a board image of a boundary area between a normal inspection target part and a defective inspection target part is useful. Also generate an image of the inspection target part of

In model generation (learning) 205 by deep learning or the like for generating a board inspection model, a large number of images of normal parts to be inspected from the captured image 201 and a small number of images of pseudo-normal parts to be inspected from the generated image 202 are generated. are used as the training data 203 of the image of the normal inspection target site. On the other hand, as the teacher data of the images of the defective inspection target portions, the image of a small number of defective inspection target portions from the captured image 201 and the images of a large number of pseudo-defective inspection target portions from the generated image 202 are used. A model verification (diagnostic logic) 206 verifies the board inspection accuracy of the board inspection model generated using the generated image of the part to be inspected for the pseudo defect. It should be noted that an image of a pseudo-defective inspection target portion or an image of a pseudo-normal inspection target portion may also be used for verification of the board inspection model by the model verification 206 . In particular, it is difficult to obtain an image of a boundary range where it is difficult to distinguish between a normal inspection area and a defective inspection area from an image of an actually manufactured substrate. The image of the inspection target site is valid. In the verification of the board inspection model by the model verification 206, if the board inspection accuracy is not sufficient, the teacher data is further added to generate the board inspection model, and the generation and verification are repeated. Even in that case, it is unclear whether the captured image data contains image data for improving the accuracy of the board inspection model. Therefore, when the cause of insufficient accuracy is known, it is effective to generate an image of a pseudo-defective inspection target region and an image of a pseudo-normal inspection target region. In addition, a large amount of images of pseudo-defective inspection target parts whose image features are known and pseudo-normal inspection target part images are input, and it is determined whether the accuracy of the board inspection model is improved or lowered. may also provide information that improves the accuracy of the board inspection model.

If it is determined that the accuracy of the board inspection model is sufficient in the model verification 206 using the diagnostic logic, the board inspection model is mounted or incorporated in the board inspection apparatus (actual machine) 210, and the actual manufactured board is produced. A pass/fail judgment 211 is executed. If at least one inspected portion on the board is determined to be defective, the board is determined to be defective.

It should be noted that not all images obtained by imaging or generated images may be used, but only a part of them may be used. For example, images of parts to be inspected that are extremely defective and pseudo images that cannot actually exist may be deleted from the training data. Alternatively, instead of using the captured image or generated image as it is, an image that has been moved up, down, left, or right, or an image that has been reversed up, down, left, or right, that is, processed by so-called augmentation may be used. Augmentation can be expected to have the effect of learning similar defects even if imaging is shifted or reversed to the left and right, and that it is possible to flexibly judge actually generated defective samples.

FIG. 2B is a diagram for explaining an overview of obtaining a captured image 201 of a site to be inspected on a substrate in this embodiment. First, imaging is repeated to acquire a plurality of captured images covering the entire substrate 222 to generate a composite image 221 . An image of one substrate 222 is acquired from the synthesized image 221 . Further, from the image of one substrate 222, the captured images 201 of a plurality of parts to be inspected on the substrate are cut out.

<Flow chart showing the flow of circuit board inspection>
In the flow of the circuit board inspection system in FIG. 3, first, in step S301, image data of a portion to be inspected is obtained from the image data of the circuit board imaged by the imaging unit. In step S303, it is determined whether the acquired image data is image data of a normal inspection target portion or image data of a defective inspection target portion. In the case of image data of a normal inspection target portion, in step S305, it is determined whether or not the image data is used as an original image for generating image data of a pseudo-defective inspection target portion. If the image is to be used as the original image, image data of the inspection target portion of the pseudo defect is generated in step S391. On the other hand, in the case of the image data of the defective inspection target portion, it is determined in step S307 whether or not the image data is used as the original image for generating the image data of the pseudo-defective inspection target portion. If the image is to be used as the original image, image data of the inspection target portion of the pseudo defect is generated in step S392. Here, steps S391 and S392 function as step S309 for generating image data of the inspection target portion of the pseudo defect. In step S309, various measures are taken to generate the image data of the portion to be inspected for the pseudo defect. For example, in a board inspection model, the accuracy of sorting a normal inspection target portion from a defective inspection target portion depends on the number of image data in the boundary range. For this purpose, it is desirable that the image data of the pseudo-defective inspection target portion generated from the image data of the normal inspection target portion and the defective inspection target portion be included in the boundary range.

Next, in step S311, the image data of the imaged normal inspection target region, and the pseudo-defective inspection target region generated from the image data of the normal inspection target region and the image data of the defective inspection target region in step S309. and the image data of the imaged defective inspection target portion. In step S313, it is determined whether the number of acquired image data is sufficient to generate a board inspection model. If it is insufficient, the process returns to step S309 to further generate image data of the inspection target portion of the pseudo defect. Alternatively, the process may return to step S301 and obtain image data of a new inspection target portion from the captured image of the substrate. By this processing, image data necessary for generating a substrate inspection model is prepared by adding image data of generated pseudo-defective inspection target portions to image data of a relatively small number of defective inspection target portions. can be done.

When sufficient image data of the inspection target portion including the image data of the pseudo-defective inspection target portion is collected, in step S315, a substrate inspection model is generated using the collected image data as teacher data. Then, in step S317, it is verified whether the generated board inspection model has sufficient accuracy in determining whether the inspection target portion is normal or defective. In such verification, whether there is no error in the cause of the inspection result of the defective inspection target part, or whether the inspection result is clearly separated (determination of the normal inspection target part and the defective inspection target part) whether the degree of anomaly is sufficiently separated), and so on. Then, in step S319, it is determined from the verification result whether or not the board inspection model has accuracy that can be used. If the accuracy is insufficient, the process returns to step S309 to further generate image data of the inspection object portion of the pseudo defect to improve the accuracy. Alternatively, the process may return to step S301 to acquire image data of a new inspection target portion of the imaged board. In this case as well, as image data supplementation for improving accuracy, the image data of the pseudo-defective inspection target portion generated from the image data of the normal inspection target portion and the defective inspection target portion is used as the image data of the normal inspection target portion. It is desirable that the area is included in the boundary range between the site and the defective inspection target site.

If the verification result of the board inspection model is sufficient, in step S321, the generated board inspection model is mounted and used for inspection of the board by inspection of the inspection target portion.

<Configuration of circuit board inspection system>
In FIG. 4, board inspection system 400 includes model generation device 410 and board inspection device 420 . Note that the model generation device 410 and the board inspection device 420 may be installed at different locations via a network, or may be installed at the same location. The model generation device includes an image processing unit 406 and a model generation unit 407 .

The image processing unit 406 has a generation method table 461, and generates processed image data of a pseudo-defective inspection target portion from the captured image data of the inspection target portion. The model generation unit 407 has an inspection model table 471, and deep learning is performed using image data of a normal inspection target portion, image data of a defective inspection target portion, and image data of a pseudo-defective inspection target portion as teacher data. A model for board inspection is generated by technology or the like. The model generation unit 407 also has a model verification unit 408 . The model verification unit 408 verifies the accuracy of the board inspection model generated by the model generation unit 407 by inputting the image data of the inspection target portion. Then, the model generation unit 407 sets or transmits the board inspection model that has passed the verification by the model verification unit 408 to the board inspection device 420 .

Board inspection apparatus 420 includes determination unit 409 , imaging unit 421 , image generation unit 423 , and display unit 430 . The determination unit 409 uses the board inspection model that has passed the verification by the model verification unit 408 to inspect the parts to be inspected based on the image data of the board. The board is determined to be defective. The imaging unit 421 sequentially images the substrates (A→B→C) 422 brought for inspection. The image generation unit 423 generates image data of the substrate from a plurality of images captured by the imaging unit 421 , cuts out image data of the inspection target site from the image data of the substrate, and sends the image data to the determination unit 409 . The display unit 430 displays, for example, the image data of the board, the image data of the part to be inspected, the verification result of the board verification model, the verification result of the board using the board verification model, and the like.

Note that the imaging unit 421 and the display unit 430 may be provided outside the substrate inspection apparatus 420 .

(Generation method table)
The generation method table 461 of FIG. 5A is used by the image processing unit 406 of FIG. 4 to generate image data of a pseudo-defective inspection target portion based on the image data of the inspection target portion generated by imaging the board. . The generation method table 461 stores whether the image processing target data 561 is based on image data or three-dimensional CAD data (3D CAD data). The image data processing method 562 for the defect inspection target portion based on the image data includes changing the shape, changing the area area, changing the color, and the like. On the other hand, as an image data processing method 562 for a defective inspection target portion based on 3D CAD data, there is a change in object shape data. Then, the processing procedure 563 is stored in association with the image data processing method 562 of each defective inspection target portion.

(Model table for inspection)
The inspection model table 471 in FIG. 5B is used by the model generation unit 407 in FIG. 4 to generate image data of a normal inspection target portion and image data of a defective inspection target portion generated by imaging the board. . Furthermore, the inspection model table 471 in FIG. 5B is used to generate a board inspection model from the image data of the pseudo-defect inspection target portion generated by the image processing unit 406 in FIG. The inspection model table 471 stores teacher data 571 input to an AI learning program such as deep learning technology, and board inspection models 572 that are learning results of the AI learning program. As the teaching data 571, a large amount of data necessary for AI learning, including image data of normal inspection target portions and image data of defective inspection target portions and pseudo-defective inspection target portions, is input. The board inspection model 572 includes parameters P1 to Pn for generating the board inspection model.

<Processing Procedure of Model Generating Device>
In FIG. 6A, the model generating device 410 determines in step S611 whether or not it is collection of teacher data. In the case of collection of teacher data, the model generation device 410 collects image data of the imaged normal inspection target region in step S613. Then, in step S615, the model generating device 410 generates image data of a pseudo-defective inspection target portion based on the image data of the normal inspection target portion and the image data of the defective inspection target portion. In step S617, the model generating device 410 collects the image data of the defective inspection target portion and the image data of the pseudo-defective inspection target portion, and stores them in a board image database (not shown).

If the collection of teacher data is not to be performed, the model generating device 410 determines in step S621 whether or not to generate a board inspection model. In the case of generation of an inspection model, the model generation device 410, in step S623, collects the collected image data of the inspection target portion, the image data of the defective inspection target portion, and the image data of the pseudo-defective inspection target portion. is obtained from the board image database. Then, in step S621, the model generation device 410 generates a substrate inspection model using the acquired image data of the normal inspection target portion and the defect and pseudo-defect inspection target portions as teacher data. The model generating device 410 saves the generated board inspection model in step S627.

If neither the collection of teacher data nor the generation of an inspection model is performed, the model generating device 410 determines in step S631 whether or not the generated board inspection model is to be verified. In the case of verification of the board inspection model, the model generation device 410 acquires the saved board inspection model in step S633. In step S635, the model generation device 410 acquires image data for which it is known whether the inspection target portion is normal or defective. Then, in step S637, the model generating device 410 verifies the board inspection model depending on whether or not the known image data has been correctly inspected. In this verification, in addition to the inspection result of whether the inspection target portion is normal or defective, the inspection result of the defective inspection target portion is based on whether there is any error in the inspection result. Alternatively, whether or not the separation of inspection results is clear (whether the degree of abnormality between the determination of a normal inspection target portion and the determination of a defective inspection target portion is sufficiently separated) is used as a criterion. In step S639, the model generation device 410 determines whether or not the accuracy of the verification result of the board inspection model is equal to or greater than a predetermined threshold value α. If the accuracy of the verification result is equal to or higher than the predetermined threshold value α, the model generation device 410 determines and notifies that the board inspection model to be verified can be used, and sends the board inspection model to the board inspection device 420 in step S641. Set or send. On the other hand, if the accuracy of the verification result is not equal to or greater than the predetermined threshold value α, the model generation device 410 determines that the accuracy of the board inspection model to be verified is insufficient and further collects the image data of the inspection target portion in step S643. to notify that the board inspection model needs to be updated.

<Processing procedure of substrate inspection device>
In FIG. 6B, board inspection apparatus 420 determines whether or not the inspection is for a new board in step S651. If the inspection is for a new board, the board inspection apparatus 420 reads out the board inspection model acquired from the model generation apparatus 410 and installs it in the board inspection apparatus 420 in step S653. Board inspection apparatus 420 acquires image data of the unknown board in step S655. Then, in step S657, board inspection apparatus 420 inspects the board by inspecting the image data of the inspection target portion of the unknown board. Board inspection apparatus 420 determines whether the inspection result is normal in step S659. In this determination, if there is at least one defective inspection target portion, the substrate is determined to be defective. If the inspection result is normal, board inspection apparatus 420 notifies in step S661 that the board to be inspected can be used. On the other hand, if the inspection result is not normal (if defective), board inspection apparatus 420 notifies that the board to be inspected is defective in step S663.

<Image data of imaged board>
Image data of a substrate imaged by the imaging unit 421 according to the present embodiment will be described below with reference to FIGS. 7A to 7D.

FIG. 7A shows an example of image data of a portion to be inspected (for example, a mounted component) of a board that can be used in this embodiment. The image data is obtained by irradiating light of a plurality of colors from the light emitting unit at different incident angles to the mounted component on the board, and capturing the reflected light. obtained by acquiring such imaging data. Possible images include a TSL image 701, a coaxial illumination image (RGB image) 702, a height image 703, and a 3D image 704, as shown in FIG. 7A. In this embodiment, an example using a height image 703 generated from a coaxial illumination image (RGB image) 702 and an example using a 3D image 704 generated from 3D CAD data will be described.

FIG. 7B is a diagram showing the configuration of the imaging unit 421 that captures the coaxial illumination image (RGB image) 702 and generates the height image 703 . The imaging section 421 includes an imaging unit 710 and an imaging control section 730 . The imaging unit 710 has a camera 711 and a coaxial epi-illumination 712 as camera units, and an illumination unit 720 .

The illumination unit 720 is configured to project illumination light for imaging by the camera 711 onto the surface of the substrate 422, which is the object to be inspected. Illumination unit 720 comprises one or more light sources that emit light at a wavelength or wavelength band selected from the wavelength band detectable by the imaging element of camera 711 . The illumination light is not limited to visible light, and ultraviolet light, X-rays, or the like may be used. If multiple light sources are provided, each light source is configured to project a different wavelength of light (eg, red, blue, and green) onto the surface of substrate 422 at a different projection angle. The illumination unit 720 is a side illumination source that projects illumination light obliquely onto the inspection surface of the inspection object 740, and is composed of an upper light source (TOP) 722, a middle light source (SIDE) 723, and a lower light source (LOW). 724 is provided. Upper light source 722 , middle light source 723 , and lower light source 724 , which are side illumination sources, are ring illumination sources, respectively, which surround the optical axis of camera 711 and project illumination light obliquely onto the inspection surface of substrate 422 . is configured to Each of the upper light source 722, the intermediate light source 723, and the lower light source 724, which are the side illumination sources, may be configured by annularly arranging a plurality of light sources. In addition, the upper light source 722, middle light source 723, and lower light source 724, which are side illumination sources, are configured to project illumination light at different angles of 17°, 45°, and 60°, respectively, with respect to the inspection surface. ing. TOP is red (R) light, LOW is blue (B) light, and SIDE is switching between three colors (R, G, B) light.

The projection unit 721 projects a pattern onto the inspection surface of the substrate 422 . The board 422 on which the pattern is projected is imaged by the camera 711 . A height image generation unit 731 of the imaging control unit 730 creates a height map of the inspection surface of the substrate based on the imaged pattern image of the substrate 422 . The height image generator 731 detects a local mismatch of the pattern image with respect to the projected pattern, and obtains the height of the part based on the local mismatch. In other words, the change in the captured pattern with respect to the projected pattern corresponds to the height change on the inspection surface. The projected pattern is preferably a one-dimensional striped pattern in which bright lines and dark lines are alternately and periodically repeated. The projection unit 721 is arranged to project the fringe pattern obliquely onto the inspection surface of the substrate 422 . A height discontinuity on the inspection surface of the substrate 422 appears as a pattern shift in the stripe pattern image. Therefore, the height difference can be obtained from the amount of pattern deviation. For example, the height image generator 731 creates a height map by a phase shift method (PMP: Phase Measurement Profilometry) using a fringe pattern whose brightness changes according to a sine curve. In the phase shift method, the shift amount of the fringe pattern corresponds to the phase difference of the sine curve. Further, by making the striped pattern into an arc or polygon centered on the projection unit 721 or by changing the thickness, it is possible to generate a more appropriate height image.

Note that the height measurement using the captured image for generating the height image may be measurement using a variable pitch projector. In height measurement using a variable-pitch projector, a fringe pattern having a trigonometric density distribution is projected, and the height is calculated in units of pixels from four fringe images captured with a shift of 1/4 wavelength. According to this height measurement, it is possible to measure parts up to 20 mm in height by combining thick and thin stripes, and by projecting stripes from the four directions of north, south, east and west, it is possible to minimize the shadow area of high parts. .

Also, although the height image generation unit 731 is provided in the imaging control unit 730 , it may be provided in the board inspection device 420 .

A processing procedure of the imaging control unit 730 will be described according to the flowchart of FIG. 7C. As a preliminary preparation, the imaging control unit 730 images the reference plane with a short-cycle fringe pattern in step S711. In step S713, the image capturing control unit 730 captures an image of the reference plane with a long-period fringe pattern. At the time of measurement, the imaging control unit 730 images the substrate 422 in a short-cycle fringe pattern in step S721. In step S723, the imaging control unit 730 captures an image of the substrate with a long-period fringe pattern. Then, in step S725, the imaging control unit 730 performs phase calculation from the short-period image acquired in step S721. In step S727, the imaging control unit 730 performs phase calculation from the long-period image acquired in step S723. Then, in step S729, the imaging control unit 730 calculates the height from the combination of the phase calculated from the short period and the phase calculated from the long period.

FIG. 7D shows an imaging example 760 of the substrate 422 by the imaging unit 421 . Each of TOP, SIDE, and LOW corresponds to the imaging result by the irradiation light of FIG. 7B. Synthesis (RGB) is synthesis of three-color images by SIDE, and synthesis (TOP, SIDE, LOW) is synthesis of all acquired images. And the height (Depth) image is the height (Depth) image of the heat map generated from the synthesis (TOP, SIDE, LOW).

<Generation of pseudo defect image data>
8A to 8C, generation of image data of a pseudo-defect inspection target portion in the image processing unit 406 will be described below.

FIG. 8A is a diagram for explaining discrimination between normal image data and defective image data in a solder image of a portion to be inspected. In FIG. 8A, image 811 is an image of a normal solder joint, and image 812 is an image of a defective solder joint (non-leaking). In the heat map of the height image, in the image 811 of a normal soldered portion in which the copper foil and the lead are firmly connected with solder, the order of red, green, and blue is from the tip of the solder toward the lead. On the other hand, in the image 812 of the defective soldered portion with few contact points between the lead and the solder, the order is blue→green→red.

Based on the knowledge of FIG. 8A, it is possible to generate pseudo-defective solder image data in the solder portion by changing the order of red, green, and blue, their areas, and the like.

A procedure for generating pseudo-defective solder image data from solder image data based on the knowledge of FIG. 8A will be described according to FIG. 8B. FIG. 8A shows a procedure for generating pseudo-defective solder image data from solder image data using an image editing program. Image editing programs have menus such as [Eyedropper], [Brush], and [Blur/Sharp]. If you select [Eyedropper], select the color you want to overlay. It is also possible to select a color from another image with a dropper. If you select [Draw with brush], the color selected with the eyedropper will be drawn with a brush. In addition, the application range can be selected from the right tab. Select [Blur/Sharpen] to use a brush to blur or sharpen. Note that the range can be selected from the right tab. Using the functions of these image editing software (programs), image data of pseudo-defective solder is generated from the solder image data.

In the upper part of FIG. 8B, image data of pseudo-defective solder having non-leakage features is generated from normal solder image data. The solder portion of a normal image is often dark blue to light blue. Layer light blue → green → red on the solder part in order. Then, by smoothing each color boundary, a pseudo-defective image with non-leakage features is generated. The color and area of the heat map of the height image are changed with the limit of an impossible state or a very rare state.

In the lower part of FIG. 8B, a sample with further abnormalities is created based on the image of defective solder. In other words, this is an example in which the amount of solder is small. In the case of a smaller amount of solder, creating an image that serves as a boundary for pass/fail judgment is effective in judging defective samples that are more difficult to judge. It should be noted that if the direction is changed in an excessively abnormal direction, the teaching data becomes useless or unnecessary teaching data. On the other hand, if an image of defective solder is changed to an image of normal solder, it is possible to create an image that serves as a boundary for judging quality.

FIG. 8C is a diagram for generating pseudo-defective solder image data from solder image data using 3D CAD data. That is, from a 3D model 831 of normal soldering, a model 832 of a defective solder sample is created using 3D CAD. Then, based on the model 832 of the defective sample, a technique such as ray tracing can be used to generate a height image and generate a TSL image.

FIG. 8D is a diagram showing another specific example of generating the image data of the pseudo-defective solder in FIG. 8C. FIG. 8D shows a 3D model 840 on the substrate 422 created by reproducing the defective solder. The 3D model 840 has parameters such as shape information of IC leads, copper foil, and solder, color information of each material, reflection characteristics, and the like. A 3D model 840 of such soldering is simulated as if it were captured by the imaging unit 710 . That is, based on the design data of the device, a camera and lighting are arranged on the upper part. Here, ring lighting of three colors, red, green, and blue, and the camera are arranged so as to look into the solder from the center of each ring. As a result, techniques such as ray tracing can be used to simulate how the defective solder looks with the arranged camera.

The image data of the area to be inspected for a pseudo-defect and the image data of the area to be inspected for a pseudo-normal are automatically generated by recognizing the image change tendency of the area to be inspected for a defect using deep learning technology. be able to. In deep learning, a large amount of defective image data, normal image data, and generated pseudo-defective image data and pseudo-normal image data are used as training data. By doing so, it is possible to prevent generation of pseudo-defective image data and pseudo-normal image data which cannot occur.

<Generation of board inspection model>
FIG. 9 shows an input 900 of teacher data used by the model generator 407 to generate a board inspection model. In FIG. 9, heat map images of 20 solder portions are displayed. The user or the board inspection apparatus 420 selects an image indicating normality and an image indicating a defect from among these, and inputs them to an AI program such as deep learning of the model generation unit 407 as teacher data. In FIG. 9, an image 901 with a check is training data indicating that the image is normal, and an image 902 without a check is training data indicating that the image is defective. The selection of images may be based on the knowledge of a skilled engineer, or may be based on criteria recognized by deep learning or the like. At this time, in addition to sorting (labeling) two types of good products and defective products, multiple sorting (labeling) according to the types of defects such as good products, non-wetting, small solder, no solder, floating parts, no parts labeling). In this way, by performing a plurality of selections according to the types of defects, it is possible to generate a board inspection model that enables more delicate inspection.

A description of processing by an AI program such as deep learning is omitted.

<Verification of board inspection model>
Verification of the board inspection model in the model verification unit 408 will be described below with reference to FIGS. 10A to 10C.

FIG. 10A shows the output of the verification result of the board inspection model by the model verification unit 408 . In FIG. 10A, a heat map image (Original in the figure) of the solder portion input to the board inspection model for verification, and a heat map image representing the degree of abnormality in the inspection result by the board inspection model are superimposed on the original image. A pair with (Heatmap in the figure) is shown. This visualizes the nodes that are activated when the trained deep learning extracts features, and superimposes the areas judged to be abnormal on the original image to display which parts were judged to be abnormal. This is for clarity. If the heat map and the original image are superimposed on each other, it may be difficult to see them. In that case, only the heat map image may be displayed.

In FIG. 10A, there is no change between the original image and the heat map image for a sample of a normal inspection site with few abnormal portions. On the other hand, in a sample of a defective inspection target portion having many abnormal portions or a high degree of abnormality, the heat map image indicates which portion is the cause of the abnormality determination. Here, the degree of abnormality represents the difference from the normal one, that is, the degree of difference between the verified inspection target region and the normal inspection target region.

FIG. 10B shows the evaluation results of the original image 1011 and the heat map image 1012 of the sample of the defective inspection target portion in FIG. 10A, which are the verification results of the board inspection model. This is an example of a defect when the chip part has little solder, and in the heat map image 1012, a portion 1021 with insufficient solder is displayed with a high degree of abnormality.

The model verification unit 408 determines that a sample of a normal inspection target portion is normal and a sample of a defective inspection target portion is determined to be defective. Further verify whether the display of is correct as the reason for the inspection result. For example, in the example of FIG. 10B, there are many failure cases where there is insufficient solder on the chip component, and this is a common reason. However, in FIG. 10B, when the portion with a high degree of abnormality exists in the area 1022 or the area 1023, it is determined that the precision of the board inspection model is insufficient. This is because region 1022 is a lead portion, not a solder portion, and is therefore an erroneous cause of solder failure inspection results. Also, since the region 1023 is not a solder portion but a substrate portion, it is an erroneous cause of the solder defect inspection result. Furthermore, it is also verified whether or not the NG type "non-leakage" of the determination result 1030 corresponds to the portion with a high degree of abnormality.

FIG. 10C shows another evaluation of the board inspection model verification result by the model verification unit 408 . A mixture matrix 1041 of determination evaluation indicates the correspondence between the normal/defective true value of the sample and the normal/defective determination by the board inspection model. Verification of the board inspection model may be determined by setting a threshold for each number of correspondences. For example, it is permissible to determine that a sample of a normal inspection target site is defective, but it is not permissible to determine that a sample of a defective inspection target site is normal. The upper limit of the number of normal samples determined to be defective is one threshold for determining the reliability of the board inspection model.

The table 1042 of the degree of abnormality is a table in which the degree of abnormality of the inspection result of the board inspection model is plotted for each verification sample. From this list 1042 of the degrees of abnormality, the over-determination rate and the lowest degree of abnormality can be found, and the difference between them is calculated as the model reliability. Then, as the verification score 1043, model reliability, oversight, and over-determination rate are presented. Oversight and over-determination rate are essential as verification items, but another method may be used for model reliability. The absolute number of judgments may also be an indicator of the reliability of the model.

If the verification results show that the reliability and accuracy of the board inspection model are not sufficient, further collection or generation of training data for generating the board inspection model will improve the reliability and accuracy of the board inspection model. Generation and verification of a model for inspection are repeated. In the case of collecting or generating additional teacher data, if the cause of insufficient reliability and accuracy of the board inspection model is found, it is possible to generate teacher data that strengthens that part. In addition, it has become possible to generate teacher data that is not random but has a certain tendency in order to find the cause of insufficient reliability and accuracy of the board inspection model.

According to the present embodiment, teacher data for generating a board inspection model can be easily supplemented, samples for verification of the board inspection model can be easily generated, and efficient and highly reliable board A model for inspection can be provided. In other words, it is possible to generate a highly accurate board inspection model from a small amount of teacher data obtained from imaging the actual board. Furthermore, even an inexperienced user can generate a highly accurate board inspection model.

[Third embodiment]
Next, a board inspection system including a board inspection apparatus according to a third embodiment of the present invention will be described. The board inspection system including the board inspection apparatus according to the present embodiment differs from the second embodiment in that the board inspection apparatus also has the model generation function of the model generation apparatus. Since other configurations and operations are similar to those of the second and third embodiments, the same configurations and operations are denoted by the same reference numerals, and detailed description thereof will be omitted.

<Configuration of circuit board inspection system>
A substrate inspection system 1100 in FIG. 11 is a substrate inspection apparatus 1120 that also has a model generation function. 4 except that the model generation unit 407 and the determination unit 409 are connected inside the board inspection apparatus 1120 so that the board inspection model can be transferred. Therefore, redundant description is omitted.

According to the present embodiment, in addition to the effects of the above-described embodiments, it is possible to automatically realize high-precision inspection of substrates at high speed without reducing human power.

[Fourth Embodiment]
Next, a board inspection system including a board inspection apparatus according to a fourth embodiment of the present invention will be described. Compared to the second and third embodiments, the board inspection system including the board inspection apparatus according to the present embodiment collects board inspection result data from a plurality of user sites to further improve accuracy. The difference is that a board inspection model is created and a user can use board inspection results of other users. Since other configurations and operations are similar to those of the second and third embodiments, the same configurations and operations are denoted by the same reference numerals, and detailed description thereof will be omitted.

<Configuration of circuit board inspection system>
The board inspection system 1200 of FIG. 12 includes a board image bank 1210 , a board inspection model supplier 1220 , and a plurality of board related users 1230 interconnected by a network 1240 . The board image bank 1210 has a board image database 1212 that stores board images as training data from a board inspection model supplier 1220, a plurality of board related users 1230, or other providers. The board inspection model supplier 1220 uses the teacher data provided from the board image database 1212 to generate a board inspection model in the board inspection model generation unit 1221 and stores it in the board inspection model database 1222 in a searchable manner. . Each board-related user 1230 has a local board image database 1233 and board inspection equipment 1231 including a board inspection model generator 1232 . Note that there may be a board-related user 1230 who does not have the board inspection model generation unit 1232 . Board-related users 1230 are required to register as members of the board image bank 1210, and use of teacher data by other users is restricted.

According to this embodiment, in addition to the effects of the above embodiment, a larger number and variety of training data and samples can be used, so that a more reliable and versatile board inspection model can be generated. can. In other words, for individual users, training data, particularly training data for defective substrates, may become uneven due to factors such as manufacturing materials, individual manufacturers, manufacturing machines, and manufacturing environments. According to this embodiment, it is possible to generate a substrate inspection model that is not affected by manufacturing materials, individual manufacturers, manufacturing machines, manufacturing environments, and the like.

[Other embodiments]
In the present embodiment, the inspection of the board, particularly the inspection of the "solder" is explained as an example. It can be applied as long as it can be inspected for good/bad, and the same effect can be obtained.

Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the technical scope of the present invention. Also, any system or apparatus that combines separate features included in each embodiment is included in the technical scope of the present invention.

Further, the present invention may be applied to a system composed of a plurality of devices, or may be applied to a single device. Furthermore, the present invention can also be applied when an information processing program that implements the functions of the embodiments is supplied to a system or apparatus and executed by a built-in processor. In order to realize the functions of the present invention on a computer, the technical scope of the present invention includes a program installed in a computer, a medium storing the program, a server for downloading the program, and a processor executing the program. . In particular, non-transitory computer readable media storing programs that cause a computer to perform at least the processing steps included in the above-described embodiments fall within the scope of the present invention.

In order to achieve the above object, a board inspection method according to the present invention comprises:
A substrate inspection method using a plurality of substrate image data captured by an imaging unit in a substrate inspection apparatus including an imaging unit for imaging a substrate including a portion to be inspected multiple times,
an image processing step of processing board image data to generate defective image data including the inspected portion determined to be defective;
a model generation step of learning normal image data including the inspection target portion determined to be normal and the defective image data as teacher data to generate a board inspection model;
a determination step of determining whether the board to be inspected is normal or defective by applying the combined image data generated by synthesizing the plurality of board image data to the board inspection model; ,
A substrate inspection method comprising:

In order to achieve the above object, a board inspection method according to the present invention comprises:
A substrate inspection method using a plurality of substrate image data captured by an imaging unit in a substrate inspection apparatus having an imaging unit for imaging different partial areas on a substrate including a portion to be inspected, the method comprising:
an image processing step of generating defect image data including the inspected portion judged to be defective by cutting out an image of the inspected portion from the combined image data generated by synthesizing the plurality of board image data ;
a model generation step of learning normal image data including the inspection target portion determined to be normal and the defective image data as teacher data to generate a board inspection model;
By applying inspection target combined image data generated by synthesizing the plurality of board image data of the inspection target board to the board inspection model, it is possible to determine whether the inspection target board is normal or defective. a judgment step for judging;
including.

Claims (5)

an image processing step of processing image data of each of a plurality of parts to be inspected on a substrate to generate defective image data indicating that each of the parts to be inspected is defective;
a model generation step of learning the normal image data indicating that the inspection target portion is normal and the defective image data as teaching data to generate a substrate inspection model;
a determination step of determining whether the substrate is normal or defective by applying image data of a plurality of inspection target portions on the substrate to be inspected to the substrate inspection model;
A board inspection method comprising: an image processing unit that processes image data of each of a plurality of inspection target portions on a substrate to generate defective image data indicating that each of the plurality of inspection target portions is defective;
a model generation unit that learns normal image data indicating that the inspection target portion is normal and the defective image data as teacher data to generate a board inspection model;
a determination unit that determines whether the substrate is normal or defective by applying image data of a plurality of inspection target portions on the substrate to be inspected to the substrate inspection model;
with
The model generation unit has a model verification unit that verifies the accuracy of the generated board inspection model, and if the accuracy of the board inspection model is insufficient based on the verification result of the model verification unit, the image processing is performed. A circuit board inspection device that adds and learns the teacher data generated by the unit to generate a new circuit board inspection model. The model verification unit has a display unit that displays a portion determined to be abnormal by the board inspection model in the image data of the inspection target portion so that it can be compared with the image data of the inspection target portion,
3. The board inspection apparatus according to claim 2, wherein the display unit further displays a list of degrees of abnormality indicating the degree of difference between the inspected portion verified by the model verification unit and the normal inspected portion. The image data of the inspection target portion is image data generated based on the image data of the substrate that has been imaged, or image data generated based on the three-dimensional CAD data of the substrate,
The image processing unit
when the image data of the inspection target site is image data obtained by imaging, using image editing software to change the image data of the inspection target site to the defective image data;
When the image data of the inspection target portion is image data generated based on three-dimensional CAD data, the three-dimensional CAD data is changed to three-dimensional CAD data representing the defective inspection target portion, and ray tracing 4. The substrate inspection apparatus according to claim 2, wherein said defective image data is generated. an image processing unit that processes image data of each of a plurality of inspection target portions on a substrate to generate defective image data indicating that each of the plurality of inspection target portions is defective;
a model generation unit that learns normal image data indicating that the inspection target portion is normal and the defective image data as teacher data to generate a board inspection model;
with
The model generation unit has a model verification unit that verifies the accuracy of the generated board inspection model, and if the accuracy of the board inspection model is insufficient based on the verification result of the model verification unit, the image processing is performed. A model generation device that adds and learns the teacher data generated by the unit to generate a new board inspection model.

Images