JP2023122140A - Teacher data generation device, teacher data generation method, and program - Google Patents

Abstract

To easily generate teacher data including an image where an unnecessary area other than a defective area is reduced.SOLUTION: A teacher data generation device 4 comprises: an image reception section 41 for receiving a predetermined size of defective image including a defect detection area and also receiving defective information indicating the range of the detection area in the defective image from an inspection device 2 for imaging an object so as to detect a defect; a segmented image generation section 42 for segmenting an area including the detection area from the defective image as a segmented image based on the defect information; a display control section 43 for displaying at least a part of the defective image on a display 35; a determination result reception section 44 for receiving an input of a defect type determination result by an operator with respect to the defective image displayed on the display 35; and a teacher data generation section 45 for generating teacher data by labelling the determination result to the segmented image. Thus, the teacher data including an image where an unnecessary area other than the defective area is reduced can be easily generated.SELECTED DRAWING: Figure 3

Description

The present invention relates to technology for generating training data.

2. Description of the Related Art An inspection apparatus that detects defects by imaging an object outputs a defect image of a predetermined size including a defect area when a defect is detected. It is conceivable to classify the types of defects indicated by such defect images using a trained model (classifier). In this case, the operator determines the defect type for the defect image prepared in advance (that is, by annotation), thereby generating teacher data in which the defect image is labeled with one defect type. The learned model is generated by performing learning using the data.

In Patent Document 1, a region containing an image defect is acquired by an operator's input, and correction is performed to widen the outer edge of the region so that the number of pixels contained inside the region increases by a predetermined amount. A technique is disclosed for generating learning data by associating corrected regions with the image.

JP 2019-87078 A

By the way, the defect image output from the inspection apparatus has a fixed size and includes many unnecessary areas other than the defect area. Therefore, it is difficult to obtain a highly accurate trained model even if learning is performed using teacher data including such defect images. As in the technique of Patent Document 1, it is conceivable to obtain an image in which unnecessary areas are reduced by an operator inputting an area including a defect, but this increases the work load. Therefore, there is a need for a method of easily generating teacher data including an image in which unnecessary regions other than defective regions are reduced.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and it is an object of the present invention to easily generate teacher data including an image in which unnecessary areas other than defective areas are reduced.

According to a first aspect of the present invention, there is provided a training data generation apparatus for generating training data, wherein a defect image having a predetermined size including a defect detection area and the defect detection area are generated from an inspection apparatus for detecting defects by imaging an object. an image reception unit that receives defect information indicating the range of the detection area in a defect image; a cutout image generation unit that cuts out an area including the detection area from the defect image as a cutout image based on the defect information; a display control unit for displaying at least part of the defect image on a display; a determination result receiving unit for receiving an input of a defect type determination result input by a worker for the defect image displayed on the display; and the clipped image. and a teacher data generation unit that labels the determination result and generates teacher data.

The invention according to claim 2 is the training data generation device according to claim 1, wherein each position of the object belongs to one region type out of a plurality of region types, and the defect information contains area type information indicating the area type to which the detection area belongs, and the clipped image generation unit stores an expansion amount set for each area type, and is specified using the area type information An area obtained by extending the detection area by the amount of extension is included in the clipped image.

The invention according to claim 3 is the training data generating apparatus according to claim 2, wherein the object is a printed circuit board, and the plurality of area types include at least a plating area and a solder resist area.

The invention according to claim 4 is the teacher data generating apparatus according to any one of claims 1 to 3, wherein one of a plurality of inspection sensitivities is selected for each position of the object. is set, the defect information includes inspection sensitivity information indicating the inspection sensitivity used when detecting the detection area, and the clipped image generation unit determines the expansion amount set for each inspection sensitivity is stored, and an area obtained by extending the detection area by an amount of extension specified using the inspection sensitivity information is included in the clipped image.

The invention according to claim 5 is the training data generation device according to any one of claims 1 to 4, wherein each position of the object corresponds to one region type out of a plurality of region types. the defect information includes area type information indicating the area type to which the detection area belongs; to label.

According to a sixth aspect of the present invention, there is provided a teaching data generation method for generating teaching data, comprising: a) a defect image of a predetermined size including a defect detection area and a defect image of a predetermined size from an inspection apparatus that images an object to detect defects; , receiving defect information indicating the range of the detection area in the defect image; b) extracting an area including the detection area from the defect image as a clipped image based on the defect information; and c) a step of displaying at least a portion of the defect image on a display; d) receiving an input of a defect type determination result by an operator for the defect image displayed on the display; and labeling the determination result to generate teacher data.

The invention according to claim 7 is the training data generation method according to claim 6, wherein each position of the object belongs to one region type out of a plurality of region types, and the defect information contains area type information indicating the area type to which the detection area belongs, and in the step b), an extension amount set for each area type is prepared and specified using the area type information A region obtained by expanding the detection region by the expansion amount is included in the cutout image.

The invention according to claim 8 is the training data generation method according to claim 7, wherein the object is a printed circuit board, and the plurality of area types include at least a plating area and a solder resist area.

The invention according to claim 9 is the teacher data generation method according to any one of claims 6 to 8, wherein one of a plurality of inspection sensitivities is selected for each position of the object. and the defect information includes inspection sensitivity information indicating the inspection sensitivity used in detecting the detection area, and in the step b), an extension amount set for each inspection sensitivity is prepared. and an area obtained by extending the detection area by an amount of extension specified using the inspection sensitivity information is included in the clipped image.

The invention according to claim 10 is the training data generation method according to any one of claims 6 to 9, wherein each position of the object corresponds to one region type out of a plurality of region types. the defect information includes area type information indicating the area type to which the detection area belongs, and in step e), in addition to the determination result, the area type to which the detection area belongs is included in the clipped image A trained model for defect classification of the one region type is generated using a plurality of labeled teacher data labeled with the one region type.

The invention according to claim 11 is a program that causes a computer to generate training data, and execution of the program by the computer causes the computer to: a) inspect an object to detect defects; a step of receiving a defect image of a predetermined size including a detection area of and defect information indicating the range of the detection area in the defect image; b) an area including the detection area from the defect image based on the defect information as a clipped image; c) displaying at least a portion of the defect image on a display; and d) inputting the defect type determination result by the operator for the defect image displayed on the display. and e) labeling the cutout image with the determination result to generate training data.

According to the present invention, it is possible to easily generate teacher data including an image in which unnecessary areas other than defective areas are reduced.

It is a figure which shows the structure of an inspection system. It is a figure which shows the structure of a computer. It is a figure which shows the structure of a teacher data generation apparatus. FIG. 10 is a diagram showing the flow of processing for generating teacher data; It is a figure which shows a captured image. It is a figure which shows a captured image. It is a figure which shows a captured image. It is a figure which shows a defect image. It is a figure which shows a defect image. It is a figure which shows a defect image. It is a figure which shows the vicinity of a defect area. It is a figure which shows a defect image. It is a figure which shows the vicinity of a defect area. It is a figure which shows the vicinity of a defect area. It is a figure which shows the vicinity of a defect area. It is a figure which shows a printed circuit board. It is a figure which expands and shows a part of printed circuit board. FIG. 10 is a diagram showing another example of a classifier;

(First embodiment)
FIG. 1 is a diagram showing the configuration of an inspection system 1 according to the first embodiment of the present invention. The inspection system 1 inspects a printed circuit board, which is an object. An inspection system 1 includes an inspection device 2 and a computer 3 . In FIG. 1, the functional configuration realized by the computer 3 is surrounded by a dashed rectangle. The inspection device 2 includes an imaging unit (not shown), a moving mechanism, and a defect detection unit. The imaging unit images the printed circuit board. The moving mechanism moves the printed circuit board relative to the imaging section. The defect detection section detects defects from the image output from the imaging section. When a defect is detected by the defect detector, a defect image of a predetermined size (also called a defect block size) including the defect area is output to the computer 3 .

FIG. 2 is a diagram showing the configuration of the computer 3. As shown in FIG. The computer 3 is a general computer system including a CPU 31, a ROM 32, a RAM 33, a fixed disk 34, a display 35, an input unit 36, a reader 37, a communication unit 38, a GPU 39, and a bus 30. has a configuration of The CPU 31 performs various arithmetic processing. The GPU 39 performs various types of arithmetic processing related to image processing. The ROM 32 stores basic programs. The RAM 33 stores various information. The fixed disk 34 stores information. The display 35 displays various information such as images. The input unit 36 includes a keyboard 36a and a mouse 36b that receive inputs from the operator. The reader 37 reads information from a computer-readable recording medium 81 such as an optical disk, magnetic disk, magneto-optical disk, or memory card. The communication unit 38 transmits and receives signals to and from other components of the inspection system 1 and external devices. Bus 30 is a signal circuit that connects CPU 31 , GPU 39 , ROM 32 , RAM 33 , fixed disk 34 , display 35 , input section 36 , reading device 37 and communication section 38 .

In the computer 3 , the program 811 is read from the recording medium 81 through the reading device 37 and stored in the fixed disk 34 in advance. Program 811 may be stored on fixed disk 34 via a network. The CPU 31 and GPU 39 execute arithmetic processing while using the RAM 33 and fixed disk 34 according to the program 811 . The CPU 31 and GPU 39 function as arithmetic units in the computer 3 . A configuration other than the CPU 31 and the GPU 39 may be employed that functions as an arithmetic unit.

In the inspection system 1, the computer 3 executes arithmetic processing and the like according to the program 811, thereby realizing the functional configuration enclosed by the dashed line in FIG. That is, the CPU 31 , GPU 39 , ROM 32 , RAM 33 , fixed disk 34 and their peripheral components of the computer 3 realize the teacher data generation device 4 , the learning section 51 and the classifier 52 . All or part of these functions may be realized by a dedicated electric circuit. Also, these functions may be realized by a plurality of computers.

The classifier 52 is a trained model that classifies a defect indicated by a defect image input from the inspection apparatus 2 into a true defect or a false defect. The learning unit 51 generates a trained model (classifier 52) by performing learning using a plurality of teacher data, which will be described later. The teacher data generation device 4 generates teacher data used in the learning section 51 .

FIG. 3 is a diagram showing the configuration of the training data generation device 4. As shown in FIG. The training data generation device 4 includes an image reception unit 41 , a clipped image generation unit 42 , a display control unit 43 , a determination result reception unit 44 , and a training data generation unit 45 . The image reception unit 41 is connected to the inspection device 2 and receives inputs such as defect images from the inspection device 2 . The clipped image generator 42 clips a clipped image, which will be described later, from the defect image. The display control unit 43 is connected to the display 35 and displays a defect image or the like on the display 35 . The determination result reception unit 44 is connected to the input unit 36 and receives input from the operator via the input unit 36 . The teacher data generation unit 45 labels the clipped image to generate teacher data.

FIG. 4 is a diagram showing the flow of processing in which the teacher data generating device 4 generates teacher data. First, the image reception unit 41 receives a defect image and defect information described later from the inspection device 2 (step S11).

Here, an example of processing for detecting defects by the inspection device 2 will be described. FIG. 5 is a diagram showing a multi-gradation captured image of part of a printed circuit board. For example, the captured image is a color image. The captured image may be a grayscale image. A plurality of types of regions are provided on the main surface of the printed circuit board. Specifically, a plated area plated with a metal such as copper, a solder resist area having a solder resist provided on the surface (hereinafter also referred to as "SR area"), letters and symbols printed on the solder resist, etc. A silk region, which is an opening of a through hole, and the like are provided. In addition, the SR area can be distinguished into a first SR area in which the lower layer of the solder resist is a copper foil and a second SR area in which the lower layer of the solder resist is the base material of the printed circuit board, and the two have different colors. As described above, each position on the main surface of the printed circuit board belongs to one of a plurality of area types including the plating area, first SR area, second SR area, silk area, and the like.

The example of FIG. 5 includes a region 61 indicating the plating region and a region 62 indicating the SR region. The region 62 includes a region 621 indicating the first SR region and a region 622 indicating the second SR region. In the following description, the regions 61, 62, 621, 622 are similarly referred to as "plating region 61", "SR region 62", "first SR region 621" and "second SR region 622". For other types of areas on the printed circuit board, the corresponding areas in the captured image are called by the same name.

In the defect detection unit of the inspection apparatus 2, for example, by referring to design data (such as CAM data), the region type to which each position in the captured image belongs is specified. A normal range of gradation values for each color component is set for each area type. In the captured image, the gradation value at each position is compared with the normal range for each color component, and a set of pixels outside the normal range is detected as a defect area. In the example of FIG. 5, a region 71 that is darker than the surroundings exists on the first SR region 621, and the region 71 is a defect region 71 recognized by the operator observing the captured image. In FIG. 6, the outer edge of an area 72 detected as a defect by the inspection apparatus 2 (hereinafter referred to as "detection area 72") is indicated by a dashed line. In the example of FIG. 6, the detection area 72 approximately coincides with the defect area 71 .

In the inspection apparatus 2, when a defect is detected, an image of a predetermined size including the detection area 72 is acquired as a defect image. Also, defect information indicating the position and shape (including size) of the detection area 72 in the defect image is acquired. Various well-known methods (inspection logic, etc.) may be used for defect detection, and different methods may be used for each region type.

When starting to generate teacher data, the inspection apparatus 2 acquires in advance a plurality of defect images from a large number of captured images of a plurality of printed circuit boards. The plurality of defect images have the same size (defect block size) and indicate the same size area on the printed circuit board. Each defect image is associated with defect information indicating the position and shape of the detection area 72 . In step S11 of FIG. 4, the image reception unit 41 receives a plurality of defect images and defect information of the plurality of defect images. For example, defect information of a plurality of defect images are included in one list while being associated with the plurality of defect images.

Subsequently, the clipped image generator 42 clips a region including the detection region 72 from each defect image as a clipped image (step S12). In the example of FIG. 6, as shown in FIG. 7, a region of a circumscribing rectangle 73 (indicated by a dashed line in FIG. 7) of the detection region 72 is cut out as a cut-out image. Each side of the circumscribing rectangle 73 is parallel to the vertical direction (column direction) or horizontal direction (row direction) of the defect image. Depending on the design of the clipped image generator 42, the minimum circumscribing rectangle (each side may be inclined with respect to the vertical direction and the horizontal direction) that can be set for the detection region 72 is the region of the clipped image. may be cut out as

Further, the defect image is displayed on the display 35 by the display control unit 43 (step S13). The image displayed on display 35 may be either the entire defect image or a portion thereof. For example, a clipped image of the defect image may be displayed, or an image obtained by expanding the clipped image by a predetermined number of pixels, that is, an image including the detection area 72 and its surroundings may be displayed. In this manner, the display control unit 43 displays at least part of the defect image on the display 35. FIG. In one example, thumbnails of a plurality of defect images are arranged and displayed in a window on the display 35 , and when the operator selects a thumbnail of one defect image via the input unit 36 , the corresponding defect is displayed on the display 35 . At least part of the image (hereinafter referred to as "selected defect image") is displayed. The selection of defect images for display on display 35 may be performed by various well-known techniques.

The determination result reception unit 44 receives the operator's input of the determination result as to whether the selected defect image displayed on the display 35 is a true defect or a false defect (step S14). In one example, a window on display 35 is provided with a button indicating "true defect" and a button indicating "false defect" along with the selected defect image. The operator confirms the selected defect image and selects one of the buttons via the input unit 36, thereby inputting the determination result indicating whether the defect indicated by the selected defect image is a true defect or a false defect. done. Input of the determination result is received by the determination result receiving unit 44 . Input of the determination result by the operator may be performed by various well-known methods.

The teacher data generation unit 45 generates teacher data by labeling the clipped image with the determination result (step S15). The training data is data that includes a clipped image obtained from the defect image and a judgment result of the defect image by the operator. The training data may include defect images. In practice, the operator inputs determination results for a plurality of defect images to generate a plurality of teacher data. As described above, the teacher data generation process is completed, and a plurality of teacher data (learning data sets) are obtained.

When a plurality of teacher data are generated, in the learning unit 51 of FIG. 1, the output of the classifier for the input of the clipped image in the plurality of teacher data and the determination result (true defect or false defect) indicated by the plurality of teacher data Machine learning is performed to generate a classifier so that A classifier is a trained model that classifies a defect indicated by an image into a true defect or a false defect. In generating a classifier, parameter values included in the classifier and the structure of the classifier are determined. Machine learning is performed, for example, by deep learning using a neural network. The machine learning may be performed by well-known methods other than deep learning. The classifier (actually, parameter values and information indicating the structure of the classifier) is transferred to and introduced into the classifier 52 .

When the inspection system 1 inspects the printed circuit board, the inspection device 2 acquires a plurality of captured images indicating a plurality of positions of the printed circuit board, and inspects the presence or absence of defects in the plurality of captured images. When a defect is detected, an image of a predetermined size including the detection area 72 is output to the classifier 52 as a defect image. The classifier 52 classifies the defect indicated by the defect image into a true defect or a false defect, and stores the classification result or outputs it to the outside. In the preferred inspection system 1, in the clipped image generation unit 42 of the computer 3, the region of the circumscribed rectangle 73 of the detection region 72 in the defect image is clipped as a clipped image in the same manner as when the teacher data is generated. is input to classifier 52 . As a result, the classifier 52 can more accurately classify whether the defect indicated by the defect image is a true defect or a false defect.

Here, a process of a comparative example for generating teacher data will be described. 8A and 8B are diagrams showing defect images, including defect area 71. FIG. In FIGS. 8A and 8B , the defect area 71 is hatched with narrower parallel lines than the SR area 62 , and the defect area 71 substantially matches the detection area acquired by the inspection apparatus 2 . Note that in the example of FIG. 8B , a set of multiple defective partial areas 711 is detected as one defective area 71 .

In the processing of the first comparative example, the entire defect image is used as the teacher data image. As shown in FIGS. 8A and 8B, the defect image usually shows an area considerably larger than the defect area 71. Therefore, in the first comparative example, the feature of the unnecessary area other than the defect area 71 is also obtained by the learning unit 51. is used for learning by In other words, since the image of the teacher data does not efficiently show the features of the defect area 71 (detection area), the classification accuracy of the classifier is low.

In the process of the second comparative example, an area of a certain size including the defect area 71 is cut out from the defect image and used as the image of the teacher data. In FIGS. 8A and 8B, the clipping region A1 clipped from the defect image in the second comparative example is indicated by a two-dot chain line. The size of the cutout region A1 is empirically determined, for example. In the second comparative example, unnecessary regions other than the defect region 71 are reduced in the image of the teacher data (the image of the cutout region A1) compared to the first comparative example, but are still included to some extent. Also, as in the example of FIG. 8B, when the defect area 71 is relatively large, it protrudes from the cutout area A1, so the image of the teacher data does not show all the features of the defect area 71 (detection area). will not be.

Furthermore, in the first and second comparative examples, a large amount of teacher data is required in order to increase the classification accuracy of the classifier, and the number of inputs of judgment results (annotations) for the defect image by the operator increases. put away. Even with a large amount of training data, it may not be possible to generate a highly accurate classifier.

On the other hand, in the training data generation device 4 of FIG. It is input and accepted by the image accepting unit 41 . Based on the defect information, the clipped image generator 42 clips a region including the detection region 72 from the defect image as a clipped image. Further, at least a part of the defect image is displayed on the display 35 by the display control unit 43, and the determination result receiving unit 44 receives the input of the determination result of the true defect or the false defect by the operator for the displayed defect image. . Then, the teacher data generation unit 45 labels the extracted image with the determination result to generate teacher data.

This makes it possible to easily generate teacher data including an image (cutout image) in which unnecessary areas other than the defect area 71 are reduced. The image also shows almost all features of the defect area 71 . In this way, by using teacher data that efficiently shows the features of the defect area 71, it is possible to generate a highly accurate trained model (classifier 52) with less teacher data, and reduce the number of annotations by the operator. be able to. In FIGS. 8A and 8B, the circumscribed rectangle 73 of the detection area cut out as the cut-out image is indicated by a dashed line.

(Second embodiment)
Next, teaching data generation processing according to the second embodiment of the present invention will be described. FIG. 9 is a diagram showing a defect image, showing an example in which a defect area 71 exists on a plated area 61. FIG. In FIG. 9, the defect region 71 is marked with parallel oblique lines with narrower intervals than the SR region 62 (the same applies to FIGS. 11 to 14 described later). FIG. 10 is an enlarged view showing the vicinity of the defect area 71, in which the detection area 72 acquired by the inspection apparatus 2 is blacked out (the same applies to FIGS. 12 to 14 described later). When the defect area 71 exists on the plating area 61, the outer edge of the detection area 72 tends to substantially match the outer edge of the defect area 71 recognized by the operator who observes the defect image. substantially overlaps the entire defect area 71 .

FIG. 11 is a diagram showing a defect image, showing an example in which a defect area 71 exists on the SR area 62. As shown in FIG. FIG. 12 is an enlarged view showing the vicinity of the defect area 71, and a set of a plurality of detection partial areas 721 is detected as one detection area 72. FIG. In FIGS. 11 and 12, the outer edge of the defective area 71 is indicated by broken lines, indicating that the outer edge of the defective area 71 (that is, the boundary with the surroundings) is unclear (the same applies to FIG. 14, which will be described later). ). When the defect area 71 exists on the SR area 62, the outer edge of the detection area 72 tends to be smaller than the outer edge of the defect area 71 recognized by the operator who observes the defect image. overlaps only part of the defect area 71 . Incidentally, the plating region 61 and the SR region 62 may use different defect detection methods.

As described above, each position on the main surface of the printed circuit board belongs to one of a plurality of region types, and the inspection apparatus 2 also identifies the region type to which each position in the captured image belongs. be done. In the inspection apparatus 2 in this processing example, when a defect is detected, area type information indicating the area type to which the detection area 72 belongs is generated and included in the defect information.

In the generation of teacher data by the teacher data generator 4, the image reception unit 41 receives the defect image and the defect information from the inspection device 2 (FIG. 4: step S11). As described above, the defect information includes area type information in addition to the position and shape of the detection area 72 in the defect image. In the clipped image generation unit 42, a region obtained by extending the circumscribing rectangle of the detection region 72 vertically and horizontally is clipped as a clipped image according to the region type to which the detection region 72 belongs (step S12).

Specifically, the number of pixels (a natural number; the same shall apply hereinafter) by which the circumscribing rectangle is expanded vertically and horizontally is set in advance for each of a plurality of region types, and the expansion amount is set in advance for each of the plurality of region types. 42 is stored and prepared. As described above, the outer edge of the detection area 72 on the plated area 61 tends to substantially coincide with the outer edge of the defect area 71, so the expansion amount for the plated area 61 is a relatively small number of pixels (for example, 0 to 5 pixels). pixels). Therefore, in the example of FIG. 10 where the detection region 72 belongs to the plating region 61, as shown in FIG. A very slightly expanded region is cut out as a cut image. The clipped image includes substantially the entire defect area 71 .

Further, since the outer edge of the detection area 72 on the SR area 62 tends to be smaller than the outer edge of the defect area 71, the expansion amount for the SR area 62 is set to a relatively large number of pixels (eg, 10 to 20 pixels). be. Therefore, in the example of FIG. 12 where the detection area 72 belongs to the SR area 62, as shown in FIG. 14, an area 74 obtained by extending the circumscribing rectangle 73 of the detection area 72 by the extension amount is cut out as a cutout image. In FIG. 14, the circumscribing rectangle 73 and the area 74 are indicated by dashed lines. The cropped image (ie, region 74) includes substantially the entire defect region 71. FIG. A region 74 obtained by expanding the circumscribing rectangle 73 of the detection region 72 by the expansion amount is the same as the circumscribing rectangle of the region obtained by expanding the detection region 72 by the expansion amount.

In the training data generating device 4, after the selected defect image is displayed on the display 35 (step S13), the operator inputs the determination result of the true defect or the false defect for the selected defect image, and the input is accepted ( step S14). Then, teacher data is generated by labeling the clipped image with the determination result (step S15). After that, the classifier 52 is generated using a plurality of teacher data in the same manner as in the processing example described above.

In the inspection of the printed circuit board by the inspection system 1, when a defect is detected by the inspection device 2, an image of a predetermined size including the detection area 72 is output to the computer 3 as a defect image, and the classification result by the classifier 52 is obtained. . In the preferred inspection system 1, in the same manner as when generating the teacher data, an area obtained by extending the circumscribing rectangle 73 of the detection area 72 vertically and horizontally according to the area type to which the detection area 72 belongs is cut out as a cutout image. The clipped image is input to the classifier 52 . As a result, the classifier 52 can more accurately classify whether the defect indicated by the defect image is a true defect or a false defect.

As described above, in this processing example, the defect information includes the area type information indicating the area type to which the detection area 72 belongs. In the clipped image generation unit 42, the extension amount set for each area type is stored, and the area obtained by extending the detection area 72 by the extension amount specified using the area type information is included in the clipped image. be done. As a result, it is possible to obtain a preferable cropped image showing approximately the entire defect area 71, and to generate a highly accurate trained model (classifier 52). Since the printed circuit board is mostly occupied by the plated region and the SR region, the plurality of region types preferably include at least the plated region and the solder resist region from the viewpoint of obtaining a preferable cutout image.

(Third embodiment)
Next, teaching data generation processing according to the third embodiment of the present invention will be described. FIG. 15 is a diagram showing the entire printed circuit board 9. As shown in FIG. The printed circuit board 9 that is being manufactured includes a waste board area 92 that is a portion that will be removed in the final product. In FIG. 15, the waste substrate area 92 is hatched. FIG. 16 is an enlarged view of a portion B1 surrounded by a dashed line in the printed circuit board 9 of FIG. As shown in FIG. 16, in the printed circuit board 9, small plated areas are densely arranged, and there are areas 91 (areas surrounded by thin broken lines in FIG. 16) where fine wiring patterns are provided.

A defect existing in the area 91 greatly affects the operation of the printed circuit board 9, so in the inspection apparatus 2 in this processing example, a stricter inspection sensitivity is set for the area 91 than for other areas. Hereinafter, the area 91 will be referred to as a "first sensitivity setting area 91". On the other hand, since defects existing in the above-described discarded board area 92 hardly affect the operation of the printed circuit board 9, a looser inspection sensitivity is set for the discarded board area 92 than for other areas. The discarded substrate area 92 is hereinafter referred to as a "second sensitivity setting area 92". An intermediate inspection sensitivity is set in a region 93 other than the first sensitivity setting region 91 and the second sensitivity setting region 92 . Hereinafter, the area 93 will be referred to as a "third sensitivity setting area 93".

In this manner, one of a plurality of inspection sensitivities is set at each position on the printed circuit board 9 . In the above-described example in which the gradation value at each position of the captured image is compared with the normal range in the inspection apparatus 2, the inspection sensitivity is the width of the normal range. A normal range narrower than the other areas is set in the first sensitivity setting area 91 , and a normal range wider than the other areas is set in the second sensitivity setting area 92 . As described above, various methods may be used for defect detection, and the method of setting the inspection sensitivity is appropriately changed according to the defect detection method.

In the inspection apparatus 2, for example, by referring to design data (such as CAM data), each position in the captured image is assigned to any one of the first sensitivity setting area 91, the second sensitivity setting area 92, and the third sensitivity setting area 93. The belonging is identified and the normal range to compare is obtained. Then, the gradation value at that position is compared with the normal range, and a set of pixels outside the normal range is acquired as the detection area 72 . In the inspection apparatus 2, the inspection sensitivity information is included in the already-described defect information. The inspection sensitivity information is information that can specify the inspection sensitivity used when detecting the detection area 72, and the inspection sensitivity information in this processing example includes the first sensitivity setting area 91, the second sensitivity setting area 92, and the second sensitivity setting area 92. This is information indicating any one of the three sensitivity setting areas 93 .

In the generation of teacher data by the teacher data generator 4, the image reception unit 41 receives the defect image and the defect information from the inspection device 2 (FIG. 4: step S11). As described above, the defect information includes inspection sensitivity information in addition to the position and shape of the detection area 72 in the defect image. In the clipped image generation unit 42, an area obtained by extending the circumscribing rectangle 73 of the detection region 72 vertically and horizontally is clipped as a clipped image according to the inspection sensitivity used when detecting the detection region 72 (step S12).

Specifically, the number of pixels by which the circumscribing rectangle 73 is expanded vertically and horizontally is used as the expansion amount, and the expansion amount is set in advance for each of the plurality of inspection sensitivities, and is stored and prepared in the clipped image generation unit 42. there is At the loosest inspection sensitivity (that is, when the detection area 72 is located in the second sensitivity setting area 92), the outer edge of the detection area 72 tends to be smaller than the outer edge of the defect area 71, so the expansion amount is relatively A large number of pixels α (eg, 8 to 12 pixels) is used. At the strictest inspection sensitivity (that is, when the detection area 72 is located in the first sensitivity setting area 91), the outer edge of the detection area 72 tends to substantially match the outer edge of the defect area 71, so the expansion amount is relatively A small number of pixels β (for example, 0 to 3 pixels) is used. With an intermediate inspection sensitivity (that is, when the detection area 72 is located in the third sensitivity setting area 93), the outer edge of the detection area 72 tends to be slightly smaller than the outer edge of the defect area 71, so the expansion amount is the number of pixels γ (for example, 4 to 7 pixels) between the number of pixels for the weakest inspection sensitivity and the number of pixels for the strictest inspection sensitivity.

As described above, the expansion amount is the largest when the inspection sensitivity is the loosest, and the expansion amount is the smallest when the inspection sensitivity is the strictest. In other words, α>γ>β is satisfied. As a result, the area obtained by extending the circumscribing rectangle 73 of the detection area 72 by the amount of extension, that is, the clipped image includes almost the entire defect area 71 .

In the training data generating device 4, after the selected defect image is displayed on the display 35 (step S13), the operator inputs the determination result of the true defect or the false defect for the selected defect image, and the input is accepted ( step S14). Then, teacher data is generated by labeling the clipped image with the determination result (step S15). After that, the classifier 52 is generated using a plurality of teacher data in the same manner as in the processing example described above.

In the inspection of the printed circuit board by the inspection system 1, when a defect is detected by the inspection device 2, an image of a predetermined size including the detection area 72 is output to the computer 3 as a defect image, and the classification result by the classifier 52 is obtained. . In the preferred inspection system 1, similarly to when the teacher data is generated, an area obtained by extending the circumscribing rectangle 73 of the detection area 72 vertically and horizontally according to the inspection sensitivity used when detecting the detection area 72 is cut. The cutout image is cut out as an output image, and the cutout image is input to the classifier 52 . As a result, the classifier 52 can more accurately classify whether the defect indicated by the defect image is a true defect or a false defect.

As described above, in this processing example, one inspection sensitivity out of a plurality of inspection sensitivities is set for each position on the printed circuit board, and an inspection indicating the inspection sensitivity used when detecting the detection area 72 is performed. Sensitivity information is included in the defect information. In the clipped image generation unit 42, the expansion amount set for each inspection sensitivity is stored, and the region obtained by expanding the detection region 72 by the expansion amount specified using the inspection sensitivity information is included in the clipped image. be done. As a result, it is possible to obtain a preferable cropped image showing approximately the entire defect area 71, and to generate a highly accurate trained model (classifier 52).

(Fourth embodiment)
Next, teaching data generation processing according to the fourth embodiment of the present invention will be described. As described above, each position on the main surface of the printed circuit board belongs to one of the plurality of area types. In the inspection apparatus 2, when a defect is detected, area type information indicating the area type to which the detection area 72 belongs is generated and included in the defect information.

Steps S11 to S14 in FIG. 4 in this processing example are the same as those in the first embodiment. In step S12, as in the second embodiment, a region obtained by extending the circumscribing rectangle 73 of the detection region 72 vertically and horizontally may be cut out as a cutout image according to the region type to which the detection region 72 belongs. Further, as in the third embodiment, a region obtained by extending the circumscribing rectangle 73 of the detection region 72 vertically and horizontally is cut out as a cutout image according to the inspection sensitivity used when detecting the detection region 72. may

The teacher data generator 45 generates teacher data by labeling the cutout image with the area type to which the detection area 72 belongs, in addition to the operator's judgment result of the selected defect image as to whether it is a true defect or a false defect. (step S15). In the training data generation process, a plurality of training data for each area type is generated from a plurality of defect images. Here, it is assumed that a plurality of teacher data for the plating area and a plurality of teacher data for the SR area are generated.

The learning unit 51 performs machine learning using a plurality of teaching data for the plating area, thereby generating a trained model 521 for the plating area shown in FIG. Also, by performing machine learning using a plurality of teacher data for the SR region, a learned model 522 for the SR region is generated.

When the inspection system 1 inspects the printed circuit board, the inspection device 2 acquires a plurality of captured images indicating a plurality of positions of the printed circuit board, and inspects the presence or absence of defects in the plurality of captured images. When a defect is detected, a defect image of a predetermined size including the detection area 72 is output to the classifier 52 along with defect information including area type information. In the classifier 52, when the detection area 72 of the defect image belongs to the plating area, the learned model 521 for the plating area is used to classify the defect indicated by the defect image into a true defect or a false defect. When the detection area 72 of the defect image belongs to the SR area, the trained model 522 for the SR area is used to classify the defect indicated by the defect image into a true defect or a false defect.

As described above, in this processing example, the defect information includes the area type information indicating the area type to which the detection area 72 belongs. In the teacher data generation unit 45, the cutout image is labeled with the region type to which the detection region 72 belongs, in addition to the result of the operator's determination of a true defect or a false defect. As a result, the learning unit 51 can use a plurality of teacher data labeled with one region type to generate a learned model for defect classification of the region type. By generating a trained model for each area type in this way, the classification accuracy can be further improved.

Various modifications are possible for the training data generation device 4 and the training data generation method.

The defect information that is input from the inspection device 2 to the training data generation device 4 only needs to indicate the range of the detection area 72 in the defect image, and is not limited to information indicating the position and shape of the detection area 72 . For example, the defect information may indicate the range of the circumscribed rectangle of the detection area 72 in the defect image (that is, the range in each of the vertical direction and the horizontal direction).

The region of the defect image that is cut out as the cut-out image may be determined based on the defect information and includes the detection region 72 . The area substantially circumscribing the detection area 72 includes not only the area circumscribing the detection area 72 but also the area circumscribing the area obtained by expanding the detection area 72 by the expansion amount described above.

In the above embodiment, in step S14 of FIG. 4, the operator inputs the determination result of the true defect or the false defect for the defect image. etc.) may be input. That is, the determination result receiving unit 44 receives the input of the defect type determination result (including the determination result of true defect or false defect) by the operator for the defect image displayed on the display 35 .

In the second embodiment, when the detection region 72 includes portions belonging to two or more different region types, the extension amount for any one of the two or more region types is used in expanding the detection region 72. may From the viewpoint of obtaining a preferable cropped image showing substantially the entire defect area 71, it is preferable to use the maximum expansion amount among the expansion amounts for the two or more area types.

In the third embodiment, when the detection region 72 includes parts detected with two or more different test sensitivities, the extension of the detection region 72 is the amount of expansion for any one of the two or more test sensitivities. may be used. From the standpoint of obtaining a preferable clipped image showing substantially the entire defect region 71, it is preferable to use the maximum extension amount among the extension amounts for the two or more inspection sensitivities.

The object to be inspected by the inspection apparatus 2 may be a substrate such as a semiconductor substrate or a glass substrate in addition to the printed circuit board. Also, the inspection device 2 may detect defects in objects other than the substrate, such as machine parts. The teacher data generation device 4 can easily generate preferable teacher data used to generate trained models for defect classification of various objects.

The configurations in the above embodiment and each modified example may be combined as appropriate as long as they do not contradict each other.

2 inspection device 3 computer 4 teacher data generation device 9 printed circuit board 35 display 41 image reception unit 42 clipped image generation unit 43 display control unit 44 determination result reception unit 45 teacher data generation unit 72 detection area 521, 522 trained model 811 program S11-S15 steps

Claims (11)

A training data generation device that generates training data,
an image receiving unit that receives a defect image of a predetermined size including a defect detection area and defect information indicating the range of the detection area in the defect image from an inspection apparatus that detects defects by imaging an object;
a clipped image generating unit that clips a region including the detection region from the defect image as a clipped image based on the defect information;
a display control unit that displays at least part of the defect image on a display;
a determination result receiving unit that receives an input of a defect type determination result by a worker for the defect image displayed on the display;
a training data generation unit that labels the cutout image with the determination result to generate training data;
A teacher data generation device characterized by comprising: The training data generation device according to claim 1,
each position of the object belongs to one region type among a plurality of region types;
The defect information includes area type information indicating an area type to which the detection area belongs,
The cut-out image generation unit stores the expansion amount set for each area type, and includes the area obtained by expanding the detection area by the expansion amount specified using the area type information in the cut-out image. A teacher data generation device characterized by: The training data generation device according to claim 2,
The teacher data generating apparatus, wherein the object is a printed circuit board, and the plurality of area types include at least a plating area and a solder resist area. The training data generation device according to any one of claims 1 to 3,
One inspection sensitivity among a plurality of inspection sensitivities is set for each position of the object,
The defect information includes inspection sensitivity information indicating the inspection sensitivity used when detecting the detection area,
The clipped image generation unit stores an expansion amount set for each inspection sensitivity, and includes an area obtained by expanding the detection area by the expansion amount specified using the inspection sensitivity information in the clipped image. A teacher data generation device characterized by: The training data generation device according to any one of claims 1 to 4,
each position of the object belongs to one region type among a plurality of region types;
The defect information includes area type information indicating an area type to which the detection area belongs,
The teacher data generation device, wherein the teacher data generation unit labels the clipped image with an area type to which the detection area belongs in addition to the determination result. A teacher data generation method for generating teacher data,
a) a step of receiving a defect image of a predetermined size including a defect detection area and defect information indicating the range of the detection area in the defect image from an inspection apparatus that detects defects by imaging an object;
b) cutting out a region including the detection region from the defect image as a cutout image based on the defect information;
c) displaying at least a portion of said defect image on a display;
d) a step of receiving an input of a defect type determination result by an operator for the defect image displayed on the display;
e) labeling the cutout image with the determination result to generate training data;
A teacher data generation method characterized by comprising: The teacher data generation method according to claim 6,
each position of the object belongs to one region type among a plurality of region types;
The defect information includes area type information indicating an area type to which the detection area belongs,
In the step b), an extension amount set for each area type is prepared, and an area obtained by extending the detection area by an extension amount specified using the area type information is included in the clipped image. A training data generation method characterized by: The teacher data generation method according to claim 7,
The teacher data generation method, wherein the object is a printed circuit board, and the plurality of area types include at least a plating area and a solder resist area. The teacher data generation method according to any one of claims 6 to 8,
One inspection sensitivity among a plurality of inspection sensitivities is set for each position of the object,
The defect information includes inspection sensitivity information indicating the inspection sensitivity used when detecting the detection area,
In the step b), an expansion amount set for each inspection sensitivity is prepared, and an area obtained by expanding the detection area by an expansion amount specified using the inspection sensitivity information is included in the clipped image. A training data generation method characterized by: The teacher data generation method according to any one of claims 6 to 9,
each position of the object belongs to one region type among a plurality of region types;
The defect information includes area type information indicating an area type to which the detection area belongs,
In step e), in addition to the determination result, the cutout image is labeled with a region type to which the detection region belongs,
A teacher data generation method, wherein a trained model for defect classification of one region type is generated using a plurality of teacher data labeled with one region type. A program that causes a computer to generate training data, wherein execution of the program by the computer causes the computer to:
a) receiving a defect image of a predetermined size including a defect detection area and defect information indicating a range of the detection area in the defect image from an inspection apparatus that detects defects by imaging an object;
b) cutting out a region including the detection region from the defect image as a cutout image based on the defect information;
c) displaying at least a portion of said defect image on a display;
d) a step of receiving an input of a defect type determination result by an operator for the defect image displayed on the display;
e) labeling the cutout image with the determination result to generate training data;
A program characterized by causing the execution of

Images