TW202334900A - Training data generation apparatus, training data generation method, and program - Google Patents

Abstract

A training data generation apparatus 4 includes an image receiver 41 for receiving a defect image of a predetermined size and defect information from an inspection apparatus 2 that detects a defect by capturing images of objects, the defect image including a detection area of a defect, the defect information indicating a range of the detection area in the defect image, a cut-out image generator 42 for cutting out an area including the detection area from the defect image as a cut-out image based on the defect information, a display controller 43 for displaying at least a portion of the defect image on a display 35, a determination result receiver 44 for receiving an input of a determination result of a defect type by an operator for the defect image displayed on the display 35, and a training data generator 45 for generating training data by labeling the cut-out image with the determination result. This makes it possible to easily generate training data including an image in which unnecessary areas other than a defect area are reduced.

Description

Teacher data generation device, teacher data generation method and program

The present invention relates to a technology for generating teacher materials.

In an inspection device for detecting defects by photographing an object, when a defect is detected, a defect image of a predetermined size including a defect area is output. Consider using a learned model (classifier) to classify the types of defects represented by the defect image. In this case, the operator determines the defect type (that is, performs annotation) on the defect image prepared in advance, thereby generating teacher information that marks a defect type on the defect image, and uses a plurality of teachers Learn the data to generate the above-mentioned learned model.

Furthermore, Japanese Patent Application Laid-Open No. 2019-87078 (Document 1) discloses a method of obtaining a region containing a defect in an image through an operator's input, and performing correction to widen the outer edge of the region so that the region is A predetermined number of pixels are included internally, and the corrected area is associated with the image, thereby generating learning completion data.

However, the defect image output from the inspection device has a fixed size and contains a large amount of redundant areas other than the defective area. Therefore, even if teacher materials containing such defective images are used for learning, it is difficult to obtain a highly accurate learned model. As shown in the method in Document 1, it is also considered that the operator inputs the area containing the defect to obtain an image with reduced redundant areas, but this will increase the operational burden. Therefore, there is a need for teacher materials that can easily generate images containing redundant areas other than defective areas.

The present invention focuses on a teacher material generating device for generating teacher materials, and aims to easily generate teacher materials including images in which redundant areas other than defective areas are reduced.

The teacher data generating device of the present invention is provided with: an image receiving unit that receives: a defect image of a predetermined size that includes a detection area of the defect from an inspection device for photographing an object to detect defects; and defect information that represents The range of the aforementioned detection area in the aforementioned defect image; the cutout image generating unit is configured to cut out the area including the aforementioned detection area from the aforementioned defective image as a cutout image based on the aforementioned defect information; and the display control unit is configured to use At least a part of the aforementioned defect image is displayed on the display; the determination result accepting unit is to accept the determination result of the defect type input by the operator for the aforementioned defect image displayed on the aforementioned display; and the teacher data generation unit is to perform the aforementioned cutting The image is marked with the aforementioned determination results to generate teacher information.

According to the present invention, it is easy to generate teacher materials including images in which redundant areas other than defective areas are reduced.

Preferably, each position of the object belongs to one of a plurality of area types; the defect information includes area type information, and the area type information represents the area type to which the detection area belongs; and the shearing The image generating unit memorizes the expansion amount set for each region type, and includes the region obtained by expanding the detection region according to the expansion amount specified using the region type information in the cutout image.

Preferably, the aforementioned object is a printed circuit board; and the plurality of aforementioned region types include at least a plating region and a solder resist region.

Preferably, one of a plurality of inspection sensitivities is set for each position of the object; the defect information includes inspection sensitivity information, and the inspection sensitivity information represents the inspection sensitivity used in the detection of the inspection area; The clipped image generating unit stores the expansion amount set for each inspection sensitivity, and includes in the clipped image a region obtained by expanding the detection area according to the expansion amount specified using the inspection sensitivity information.

Preferably, each position of the aforementioned object belongs to one area type among a plurality of area types; the aforementioned defect information includes area type information, and the aforementioned area type information represents the area type to which the aforementioned detection area belongs; and the aforementioned teacher information The generation unit marks not only the determination result but also the area type to which the detection area belongs to the cutout image.

The present invention also focuses on a teacher data generation method for generating teacher data. The teacher data generation method of the present invention includes: step a, receiving from an inspection device used to photograph objects to detect defects: a defect image of a predetermined size, which includes a detection area of defects; and defect information, which represents the aforementioned defects. The range of the aforementioned detection area in the image; step b is to cut out the area containing the aforementioned detection area from the aforementioned defect image based on the aforementioned defect information as a cut image; step c is to display the aforementioned defect image on the display At least a part of; step d, is to accept the determination result of the defect type input by the operator for the aforementioned defect image displayed on the aforementioned display; and step e, is to mark the aforementioned determination result on the aforementioned cut image to generate teacher materials.

Preferably, each position of the object belongs to one area type among a plurality of area types; the defect information is area type information, and the area type information includes the area type indicating the area type to which the detection area belongs; in the above step In step b, the expansion amount set for each area type is prepared, and the area obtained by expanding the detection area according to the expansion amount specified using the area type information is included in the cut image.

Preferably, the aforementioned object is a printed circuit board; and the plurality of aforementioned area types include at least a plating area and a solder resist area.

Preferably, one of a plurality of inspection sensitivities is set for each position of the object; the defect information includes inspection sensitivity information, and the inspection sensitivity information represents the inspection sensitivity used in the detection of the inspection area; In the aforementioned step b, an expansion amount set for each inspection sensitivity is prepared, and an area obtained by expanding the detection area according to the expansion amount specified using the inspection sensitivity information is included in the cutout image.

Preferably, each position of the aforementioned object belongs to one area type among a plurality of area types; the aforementioned defect information includes area type information, and the aforementioned area type information represents the area type to which the aforementioned detection area belongs; in the aforementioned step In e, the cut image is marked with not only the judgment result but also the area type to which the detection area belongs, and a learned model for defect classification of the area type is generated using a plurality of teacher materials marked with one area type.

The present invention also focuses on a program for causing a computer to generate teacher information. By causing the computer to execute the program of the present invention, the aforementioned computer executes: step a, receiving from an inspection device used to photograph an object to detect defects: a defect image of a predetermined size, which contains the detection area of the defect; and defect information , represents the range of the aforementioned detection area in the aforementioned defect image; step b, based on the aforementioned defect information, cuts out the area containing the aforementioned detection area from the aforementioned defect image as a cut image; step c, involves making the display Display at least a part of the aforementioned defect image; step d is to accept the determination result of the defect type input by the operator for the aforementioned defect image displayed on the aforementioned display; and step e is to mark the aforementioned determination result on the aforementioned cutout image And generate teacher information.

The above objects, other objects, features, forms, and advantages will become clear from the following detailed description of the present invention with reference to the accompanying drawings.

(First Embodiment) FIG. 1 is a diagram showing the structure of the inspection system 1 in the first embodiment of the present invention. The inspection system 1 inspects a printed circuit board as a target object. The inspection system 1 includes an inspection device 2 and a computer 3 . In Figure 1, the functional configuration implemented by the computer 3 is surrounded by a dotted rectangle. The inspection device 2 includes an imaging unit (not shown), a moving mechanism, and a defect detection unit. The photography department takes photos of printed circuit boards. The moving mechanism moves the printed circuit board relative to the imaging unit. The defect detection unit detects defects from the image output by the imaging unit. When a defect is detected, the defect detection unit outputs a defect image of a predetermined size (also called defect block size) including the defect area to the computer 3 .

FIG. 2 is a diagram showing the structure of the computer 3 . Computer 3 series has the composition of a general computer system, including CPU (Central Processing Unit; central processing unit) 31, ROM (Read-Only Memory; read-only memory) 32, RAM (Random Access Memory; random access memory) 33 , hard disk 34, display 35, input unit 36, reading device 37, communication unit 38, GPU (Graphics Processing Unit; graphics processing unit) 39 and bus 30. The CPU31 performs various calculation processes. The GPU39 performs various operations related to image processing. ROM32 is the basic memory program. RAM33 stores various information. The hard disk 34 performs information storage. The display 35 displays various information such as images. The input unit 36 includes a keyboard 36a and a mouse 36b that accept input from the operator. The reading device 37 reads information from computer-readable recording media 81 such as optical disks, magnetic disks, optical disks, and memory cards. The communication unit 38 sends and receives signals to and from other components of the inspection system 1 and external devices. The bus 30 is a signal circuit for connecting the CPU 31 , the GPU 39 , the ROM 32 , the RAM 33 , the hard disk 34 , the display 35 , the input unit 36 , the reading device 37 and the communication unit 38 .

In the computer 3 , the program 811 is previously read from the recording medium 81 as a program product via the reading device 37 and the program 811 is stored in the hard disk 34 . The program 811 can also be stored in the hard disk 34 via the network. The CPU 31 and the GPU 39 use the RAM 33 and the hard disk 34 to perform calculation processing according to the program 811. The CPU 31 and the GPU 39 function as a computing unit in the computer 3 . In addition to the CPU 31 and the GPU 39, other structures that function as a computing unit may be adopted.

In the inspection system 1, the computer 3 executes calculation processing according to the program 811, thereby realizing the functional configuration surrounded by the dotted line in Fig. 1 . That is, the CPU 31 , GPU 39 , ROM 32 , RAM 33 , hard disk 34 and peripheral components of these components of the computer 3 realize the teacher material generating device 4 , the learning unit 51 and the classifier 52 . All or part of these functions can also be realized by dedicated electronic circuits. Moreover, these functions can also be implemented by multiple computers.

The classifier 52 is a learned model and is used to classify defects represented by the defect image input from the inspection device 2 into real defects or pseudo defects. The learning unit 51 performs learning using a plurality of teacher data described below to generate a learned model (classifier 52). The teacher material generating device 4 generates teacher materials used in the learning unit 51 .

FIG. 3 is a diagram showing the structure of the teacher material generating device 4. The teacher material generating device 4 includes an image receiving unit 41 , a cutout image generating unit 42 , a display control unit 43 , a determination result accepting unit 44 , and a teacher material generating unit 45 . The image receiving unit 41 is connected to the inspection device 2 and receives defect images and the like input from the inspection device 2 . The cutout image generating unit 42 cuts out a cutout image described below from the defect image. The display control unit 43 is connected to the display 35 and displays a defect image or the like on the display 35 . The determination result receiving unit 44 is connected to the input unit 36 and accepts input from the operator via the input unit 36 . The teacher material generating unit 45 adds a mark to the cutout image to generate teacher material.

FIG. 4 is a diagram showing the processing flow of the teacher material generating device 4 for generating teacher materials. First, the image receiving unit 41 receives a defect image and defect information described below from the inspection device 2 (step S11).

Here, an example of the process of detecting defects by the inspection device 2 will be described. FIG. 5 is a diagram showing a multi-tone photographic image of a part of the printed circuit board. For example, the captured image is a color image. The captured image can also be a grayscale image. A plurality of areas are provided on the main surface of the printed circuit board. Specifically, there are a plating area plated with metal such as copper, a solder resist area (hereinafter also referred to as "SR area") with solder resist on the surface, and screens such as characters or symbols printed on the solder resist. (silk) area, a through hole area which is an opening of a through hole, etc. Moreover, the SR region can be divided into a first SR region in which the lower layer of the solder resist is a copper foil and a second SR region in which the lower layer of the solder resist is the base material of the printed circuit board, and the colors of the two are different. As described above, each position on the main surface of the printed circuit board belongs to any one of a plurality of area types including the plating area, the first SR area, the second SR area, the screen area, and the like.

The example of FIG. 5 includes an area 61 representing a plating area and an area 62 representing an SR area. The area 62 includes an area 621 representing a first SR area and an area 622 representing a second SR area. In the following description, the regions 61, 62, 621, and 622 are also referred to as the "plating region 61", the "SR region 62", the "first SR region 621", and the "second SR region 622". The same names are also used for other types of areas on the printed circuit board to refer to the corresponding areas of the captured image.

In the defect detection unit of the inspection device 2, the area type to which each position of the captured image belongs is specified by referring to design data (CAM (computer-aided manufacturing) data, etc.), for example. Furthermore, the normal range of the hue value of each color component is set for each area type. In the captured image, the hue value at each position is compared with the normal range for each color component, and a set of pixels outside the normal range is detected as a defective area. In the example of FIG. 5 , there is an area 71 darker than the surrounding area in the first SR area 621 , and this area 71 is the defective area 71 recognized by the operator who observes the captured image. In FIG. 6 , the outer edge of the area 72 (hereinafter referred to as the “detection area 72”) detected as a defect by the inspection device 2 is indicated by a dotted line. In the example of FIG. 6 , the detection area 72 substantially coincides with the defective area 71 .

In the inspection device 2, when a defect is detected, an image of a predetermined size including the detection area 72 is obtained as a defect image. Furthermore, defect information indicating the position and shape (including size) of the detection area 72 in the defect image is obtained. Furthermore, in the detection of defects, various well-known methods (inspection logic, etc.) can be used, and different methods can also be used according to various area types.

When starting to generate the teaching data, the inspection device 2 obtains a plurality of defect images from a plurality of photographed images corresponding to a plurality of printed circuit boards in advance. The plurality of defect images have the same size (defect block size) and represent areas of the same size in the printed circuit board. Furthermore, each defect image is associated with defect information indicating the position and shape of the detection area 72 . In step S11 of FIG. 4 , the image receiving unit 41 receives a plurality of defect images and defect information of the plurality of defect images. For example, the defect information of a plurality of defect images is included in a list in a state of being respectively associated with the plurality of defect images.

Next, in the cutout image generating unit 42, a region including the detection region 72 is cut out from each defect image as a cutout image (step S12). In the example of FIG. 6 , as shown in FIG. 7 , the area surrounding the rectangle 73 (indicated by a dotted line in FIG. 7 ) of the detection area 72 is cut out as a cutout image. Each side of the circumscribed rectangle 73 is parallel to the up-down direction (column direction) or the left-right direction (row direction) of the defect image. Through the design of the cutout image generating unit 42, an area of the minimum circumscribed rectangle that can be set for the detection area 72 (each side may be inclined in the up-down direction and the left-right direction) can be cut out as a cutout image.

Then, the display control unit 43 displays the defect image on the display 35 (step S13). The image displayed on the display 35 may be the entire defect image or a part thereof. For example, a cropped image of the defect image may be displayed, an image in which the cropped image is enlarged by a predetermined number of pixels may be displayed, that is, an image including the detection area 72 and the surroundings of the detection area 72 may be displayed. . In this way, the display control unit 43 causes the display 35 to display at least a part of the defect image. In one example, a plurality of thumbnails of defect images are arranged and displayed in a window on the display 35. When the operator selects a thumbnail of a defect image through the input unit 36, the defect is displayed on the display 35. At least a portion of the image (hereinafter referred to as the "selected defect image"). The defect image displayed on the display 35 can also be selected using various well-known methods.

The determination result accepting unit 44 accepts the determination result of a real defect or a pseudo defect input by the operator with respect to the selected defect image displayed on the display 35 (step S14). In one example, in addition to selecting the defect image, the window on the display 35 is also provided with a button indicating "real defect" and a button indicating "pseudo defect". The operator confirms the selected defect image and selects any button through the input unit 36 to input a determination result indicating whether the defect represented by the selected defect image is a real defect or a pseudo defect. The input of the determination result is accepted by the determination result accepting unit 44 . When the operator inputs the judgment result, various well-known methods can be used.

In the teacher material generation unit 45, the cut image is marked with the determination result, thereby generating teacher materials (step S15). The teacher data is data including a cutout image obtained from the defect image and the operator's judgment result of the defect image. Teacher materials may also contain images of defects. In fact, the operator inputs the judgment results for a plurality of defect images and generates a plurality of teacher materials. Through the above operations, the teacher data generation process is completed, and a plurality of teacher data (learning data sets) are obtained.

After the plurality of teacher materials are generated, mechanical learning is performed in the learning unit 51 in FIG. defects or pseudo-defects) and generate a classifier. The classifier is a learned model that is used to classify the defects represented by the image into real defects or pseudo defects; in the generation process of the classifier, the parameter values contained in the classifier and the structure of the classifier are determined. Machine learning is performed, for example, by deep learning using a neural network. This machine learning can also be performed by well-known methods other than deep learning. The classifier (actually parameter values or information representing the construction of the classifier) is transferred and imported into the classifier 52 .

When the inspection system 1 inspects a printed circuit board, the inspection device 2 obtains a plurality of photographed images representing a plurality of positions of the printed circuit board, and inspects the plurality of photographed images for defects. When a defect is detected, an image of a predetermined size including the detection area 72 is output to the classifier 52 as a defect image. In the classifier 52, the defects represented by the defect images are classified as real defects or pseudo defects, and the classification results are memorized or output to the outside. In the preferred inspection system 1, in the cutout image generating unit 42 of the computer 3, the area of the circumscribed rectangle 73 of the detection area 72 in the defect image is cut into a cutout image in the same manner as when the teacher material is generated. image, and the cropped image is input to the classifier 52. Thereby, the classifier 52 can classify whether the defect represented by the defect image is a real defect or a pseudo defect with higher accuracy.

Here, a comparative example of processing for generating teacher data will be described. 8A and 8B are diagrams showing defective images, including the defective area 71. In FIGS. 8A and 8B , the defective region 71 is marked with parallel diagonal lines having a narrower interval than the intervals in the SR region 62 , and the defective region 71 substantially coincides with the detection region obtained by the inspection device 2 . Furthermore, in the example of FIG. 8B , a set of a plurality of defective partial areas 711 is detected as one defective area 71 .

In the processing of the first comparative example, the entire defect image is used as the image of the teacher's material. As shown in FIGS. 8A and 8B , since the defect image usually represents an area significantly larger than the defective area 71 , in the first comparative example, the features of the extra area other than the defective area 71 are also used in the learning of the learning unit 51 . In other words, since the image of the teacher's material does not effectively represent the characteristics of the defective area 71 (detection area), the classification accuracy of the classifier is reduced.

In the process of the second comparative example, a fixed-size area including the defect area 71 in the defect image is cut out and used as an image of the teacher material. In FIGS. 8A and 8B , the clipping area A1 clipped from the defect image in the second comparative example is represented by a two-dot chain line. The size of the shearing area A1 is determined empirically, for example. In the second comparative example, although the redundant area other than the defective area 71 in the image of the teacher material (image of the cropping area A1) is smaller than in the first comparative example, it still exists to a certain extent. Furthermore, as shown in the example of FIG. 8B , since the defective area 71 protrudes from the cropping area A1 if it is large, the image of the teacher's material cannot express all the features of the defective area 71 (detection area).

Furthermore, in the first comparative example and the second comparative example, a large amount of teacher materials are required to improve the classification accuracy of the classifier, so the number of times the operator inputs the judgment results of the defect image (the number of annotations) increases. Sometimes it is not possible to generate a highly accurate classifier even with large amounts of teacher data.

In contrast, in the teaching material generating device 4 of FIG. 3 , the inspection device 2 inputs a defect image of a predetermined size including the detection area 72 of the defect and defect information indicating the position and shape of the detection area 72 in the defect image. , and the image receiving unit 41 receives the defect image and the defect information. In the cutout image generating unit 42, a region including the detection area 72 is cut out from the defect image as a cutout image based on the defect information. Then, the display control unit 43 displays at least part of the defect image on the display 35 , and the determination result accepting unit 44 accepts a determination result of a real defect or a pseudo defect input by the operator with respect to the displayed defect image. Then, the teacher material generating unit 45 marks the judgment result on the cutout image and generates teacher material.

This makes it possible to easily generate teacher materials including an image (cropped image) in which redundant areas other than the defective area 71 are reduced. Furthermore, substantially all the characteristics of the defective area 71 are expressed in this image. In this way, by using teacher data that effectively expresses the characteristics of the defect area 71, a highly accurate learned model (classifier 52) can be generated using less teacher data, and the number of annotations by the operator can also be reduced. In addition, in FIGS. 8A and 8B , the circumscribed rectangle 73 of the detection area cut out as the cutout image is indicated by a dotted line.

(Second embodiment) Next, the teacher material generation process in the second embodiment of the present invention will be described. FIG. 9 is a diagram showing a defect image, and shows an example in which the defective area 71 exists on the plating area 61 . In FIG. 9 , the defective region 71 is marked with parallel diagonal lines whose intervals are narrower than the intervals of the SR regions 62 (the same applies to FIGS. 11 to 14 to be described later). FIG. 10 is an enlarged view showing the vicinity of the defective area 71, and the detection area 72 obtained by the inspection device 2 is completely blackened (the same applies to FIGS. 12 to 14 to be described later). When there is a defective area 71 on the plating area 61, the outer edge of the detection area 72 is often roughly consistent with the outer edge of the defective area 71 identified by the operator observing the defect image. In Figure 10, the entire detection area 72 system It substantially overlaps with the entire defective area 71 .

FIG. 11 is a diagram showing a defect image, showing an example in which a defective area 71 exists on the SR area 62 . FIG. 12 is an enlarged view showing the vicinity of the defective area 71. In the figure, a set of a plurality of detection partial areas 721 is detected as one detection area 72. In FIGS. 11 and 12 , the outer edge of the defective region 71 is indicated by a dotted line, and the outer edge of the defective region 71 (that is, the boundary with the surroundings) is not clear (the same applies to FIG. 14 to be described later). When there is a defective area 71 on the SR area 62, the outer edge of the detection area 72 is often smaller than the outer edge of the defective area 71 recognized by the operator observing the defect image. In FIG. 12, the detection area 72 is different from the defective area 71. parts overlap. Furthermore, the defect detection methods in the plating area 61 and the SR area 62 may also be different.

As described above, each position on the main surface of the printed circuit board belongs to one of a plurality of area types, and the inspection device 2 also specifies the area type to which each position in the captured image belongs. In the inspection device 2 in this processing example, when a defect is detected, area type information indicating the area type to which the detection area 72 belongs is generated, and the area type information is included in the defect information.

While the teacher material generating device 4 is generating the teacher material, the image receiving unit 41 receives the defect image and defect information from the inspection device 2 (Fig. 4: step S11). As mentioned above, the defect information not only includes the position and shape of the detection area 72 in the defect image, but also includes area type information. In the cutout image generating unit 42, based on the area type to which the detection area 72 belongs, a region obtained by extending the circumscribed rectangle of the detection area 72 up, down, left, and right is cut out as a cutout image (step S12).

Specifically, the number of pixels (which is a natural number, the same applies below) that will expand the circumscribed rectangle up, down, left, and right is used as the expansion amount, and the expansion amount is set in advance for each of the plurality of area types, and the expansion amount is stored in the cutout image. Prepared in the generation unit 42. As mentioned above, since the outer edge of the detection area 72 on the plating area 61 is often substantially consistent with the outer edge of the defect area 71, the expansion amount of the plating area 61 is set to a smaller number of pixels (for example, 0 pixels to 5 pixels). Therefore, in the example of FIG. 10 in which the detection area 72 belongs to the plating area 61 , as shown in FIG. 13 , the area of the circumscribing rectangle 73 (indicated by a dotted line in FIG. 13 ) of the detection area 72 is cut out or slightly expanded. The resulting area is used as a cropped image. This cropped image includes substantially the entire defective area 71 .

Furthermore, since the outer edge of the detection area 72 on the SR area 62 is often smaller than the outer edge of the defect area 71, the expansion amount of the SR area 62 is set to a smaller number of pixels (for example, 10 to 20 pixels). Therefore, in the example of FIG. 12 in which the detection area 72 belongs to the SR area 62, the area 74 obtained by expanding the circumscribing rectangle 73 of the detection area 72 by the expansion amount is cut out as a cutout image as shown in FIG. 14 . In FIG. 14 , the enclosing rectangle 73 and the area 74 are represented by dotted lines. The cropped image (ie, area 74 ) includes substantially the entire defective area 71 . Furthermore, the area 74 obtained by expanding the circumscribed rectangle 73 of the detection area 72 by the expansion amount is the same as the area 74 obtained by expanding the detection area 72 by the expansion amount.

In the teaching material generating device 4, after the selected defect image is displayed on the display 35 (step S13), the operator inputs a determination result of a true defect or a pseudo defect with respect to the selected defect image and accepts the input (step S14). Then, the cut image is marked with the judgment result, thereby generating teacher materials (step S15). Thereafter, a classifier 52 is generated using a plurality of teacher materials, in the same manner as in the above-described processing example.

When the inspection device 2 detects a defect during the inspection of the printed circuit board by the inspection system 1, an image of a predetermined size including the detection area 72 is output to the computer 3 as a defect image, and the classification result of the classifier 52 is obtained. In the preferred inspection system 1, similarly to the generation of teacher materials, an area obtained by extending the circumscribed rectangle 73 of the detection area 72 up, down, left, and right is cut out according to the area type to which the detection area 72 belongs as a cutout image. , and input the cut image to the classifier 52. Thereby, the classifier 52 can classify whether the defect represented by the defect image is a real defect or a pseudo defect with higher accuracy.

As described above, in this processing example, the area type information indicating the area type to which the detection area 72 belongs is included in the defect information. The cutout image generating unit 42 stores the expansion amount set for each area type, and includes in the cutout image a region obtained by expanding the detection area 72 according to the expansion amount specified using the area type information. . Thereby, a preferable cropped image representing substantially the entire defect area 71 can be obtained, and a highly accurate learned model (classifier 52) can be generated. In the printed circuit board, since the plated area and the SR area occupy the majority, from the viewpoint of obtaining a better shear image, it is preferable that the plurality of area types include at least the plated area and the solder resist area.

(Third embodiment) Next, the teacher material generation process in the third embodiment of the present invention will be described. FIG. 15 is a diagram showing the entire printed circuit board 9 . The printed circuit board 9 during the manufacturing process includes a portion that will be removed from the final product, that is, a waste substrate area 92 . In FIG. 15 , the discarded substrate area 92 is marked with parallel diagonal lines. FIG. 16 is an enlarged view of the portion B1 surrounded by a dotted line in the printed circuit board 9 of FIG. 15 , and the discarded substrate area 92 is surrounded by a thick dotted line. As shown in FIG. 16 , the printed circuit board 9 has a region 91 (a region surrounded by a thin dotted line in FIG. 16 ) in which small plating regions or fine wiring patterns are closely arranged.

Since defects present in the area 91 will greatly affect the operation of the printed circuit board 9 , the inspection sensitivity set for the area 91 in the inspection device 2 in this processing example is stricter than that for other areas. Hereinafter, the area 91 is referred to as the "first sensitivity setting area 91". On the other hand, since the defects existing in the waste substrate area 92 described above hardly affect the operation of the printed circuit board 9 , the inspection sensitivity set for the waste substrate area 92 is looser than that of other areas. Hereinafter, the discarded substrate area 92 is referred to as the "second sensitivity setting area 92". Furthermore, an intermediate inspection sensitivity is set in an area 93 other than the first sensitivity setting area 91 and the second sensitivity setting area 92 . Hereinafter, the area 93 is referred to as the "third sensitivity setting area 93".

In this way, any one of the plurality of inspection sensitivities is set at each position of the printed circuit board 9 . In the inspection device 2 , in the above-described example in which the hue value at each position of the captured image is compared with the normal range, the inspection sensitivity is the width of the normal range. The normal range set in the first sensitivity setting area 91 is narrower than other areas, and the normal range set in the second sensitivity setting area 92 is wider than other areas. As mentioned above, various methods can be used for defect detection, and the setting method of the inspection sensitivity is appropriately changed according to the defect detection method.

In the inspection device 2, for example, by referring to design data (CAM data, etc.), it is specified that each position in the captured image belongs to the first sensitivity setting area 91, the second sensitivity setting area 92, and the third sensitivity setting area 93. Which area, to obtain the normal range should be compared. Then, the hue value at the position is compared with the normal range, and a set of pixels outside the normal range is obtained as the detection area 72 . In the inspection device 2, the inspection sensitivity information is included in the defect information described above. The inspection sensitivity information is information that can specify the inspection sensitivity used in the detection of the detection area 72. In this processing example, the inspection sensitivity information is used to indicate the first sensitivity setting area 91, the second sensitivity setting area 92, and the third sensitivity. Set the information of any area in area 93.

While the teacher material generating device 4 is generating the teacher material, the image receiving unit 41 receives the defect image and defect information from the inspection device 2 (Fig. 4: step S11). As mentioned above, the defect information not only includes the position and shape of the detection area 72 in the defect image, but also includes inspection sensitivity information. In the cutout image generating unit 42 , based on the inspection sensitivity used for detecting the detection area 72 , an area in which the circumscribed rectangle 73 of the detection area 72 is expanded vertically, downwardly, leftward, and right is cut out as a cutout image (step S12).

Specifically, the number of pixels by which the circumscribed rectangle 73 is expanded up, down, left, and right is used as the expansion amount. The expansion amount is set in advance for each of the plurality of inspection sensitivities, and the expansion amount is stored in the cutout image generating unit 42 for preparation. Under the loosest inspection sensitivity (that is, when the detection area 72 is located in the second sensitivity setting area 92), since the outer edge of the detection area 72 is often smaller than the outer edge of the defect area 71, the expansion amount is set to a larger pixel. number α (e.g. 8 pixels to 12 pixels). Under the most stringent inspection sensitivity (that is, when the detection area 72 is located in the first sensitivity setting area 91), since the outer edge of the detection area 72 is often roughly consistent with the outer edge of the defect area 71, the expansion amount is set to be small. Number of pixels β (e.g. 0 pixels to 3 pixels). At the intermediate inspection sensitivity (that is, when the detection area 72 is located in the third sensitivity setting area 93), since the outer edge of the detection area 72 is often slightly smaller than the outer edge of the defect area 71, the expansion amount is set to be at the maximum inspection sensitivity. The number of pixels γ between the number of pixels when the inspection sensitivity is relaxed and the number of pixels when the inspection sensitivity is the most stringent (for example, 4 pixels to 7 pixels).

As mentioned above, the expansion amount is the largest when the inspection sensitivity is the loosest, and the expansion amount is the smallest when the inspection sensitivity is the strictest. In other words, α>γ>β is satisfied. As a result, the area obtained by expanding the circumscribed rectangle 73 of the detection area 72 by the expansion amount, that is, the cropped image includes substantially the entire defect area 71 .

In the teaching material generating device 4, after the selected defect image is displayed on the display 35 (step S13), the operator inputs a determination result of a true defect or a pseudo defect with respect to the selected defect image and accepts the input (step S14). Furthermore, teacher data is generated by marking the judgment result on the cut image (step S15). Thereafter, a classifier 52 is generated using a plurality of teacher materials, in the same manner as in the above-described processing example.

When the inspection device 2 detects a defect while the inspection system 1 inspects the printed circuit board, an image of a predetermined size including the detection area 72 is output to the computer 3 as a defect image and a classification result of the classifier 52 is obtained. In the preferred inspection system 1, similarly to the generation of teacher materials, an area obtained by extending the circumscribed rectangle 73 of the detection area 72 up, down, left, and right is cut out based on the inspection sensitivity used for detection. The image is cropped, and the cropped image is input to the classifier 52 . Thereby, the classifier 52 can classify whether the defect represented by the defect image is a real defect or a pseudo defect with higher accuracy.

As described above, in this processing example, one of a plurality of inspection sensitivities is set for each position of the printed circuit board, and the inspection sensitivity information indicating the inspection sensitivity used for detection of the detection area 72 is included in the defect information. . The cutout image generating unit 42 memorizes the expansion amount set for each inspection sensitivity, and includes in the cutout image a region obtained by expanding the detection area 72 according to the expansion amount specified using the inspection sensitivity information. Thereby, a preferable cropped image representing substantially the entire defect area 71 can be obtained, and a highly accurate learned model (classifier 52) can be generated.

(Fourth Embodiment) Next, the teacher material generation process in the fourth embodiment of the present invention will be described. As mentioned above, each position on the main surface of the printed substrate belongs to one of a plurality of area types. In the inspection device 2, when a defect is detected, area type information indicating the area type to which the detection area 72 belongs is generated and the area type information is included in the defect information.

Steps S11 to S14 in FIG. 4 in this processing example are the same as those in the above-described first embodiment. In step S12 , similarly to the second embodiment, a region obtained by extending the circumscribed rectangle 73 of the detection region 72 up, down, left, and right may be cut out according to the region type to which the detection region 72 belongs as a cropped image. Furthermore, like the third embodiment, an area obtained by extending the circumscribed rectangle 73 of the detection area 72 vertically, downwardly, leftwardly, and vertically may be cut out as a cutout image based on the inspection sensitivity used for detection of the detection area 72 . .

In the teacher material generation unit 45, in addition to marking the operator's judgment result of the selected defect image as a real defect or a pseudo defect, the cutout image is also marked with the area type to which the detection area 72 belongs, thereby generating teacher materials. (Step S15). In the teacher material generation process, a plurality of teacher materials corresponding to each area type are generated based on a plurality of defect images. Here, a plurality of teacher materials for the plating area and a plurality of teacher materials for the SR area are generated.

The learning unit 51 performs mechanical learning using a plurality of teaching materials for the plating area, thereby generating a learned model 521 for the plating area shown in FIG. 17 . Furthermore, machine learning is performed using a plurality of teacher materials for the SR area, thereby generating a learned model 522 for the SR area.

When the inspection system 1 inspects the printed circuit board, the inspection device 2 obtains a plurality of photographed images showing a plurality of positions of the printed circuit board, and inspects the plurality of photographed images for defects. When a defect is detected, a defect image of a predetermined size including the detection area 72 is output to the classifier 52 together with defect information including area type information. In the classifier 52, when the detection area 72 of the defect image belongs to the plating area, the learned model 521 for the plating area is used to classify the defects represented by the defect image into real defects or pseudo defects. When the detection area 72 of the defect image belongs to the SR area, the learned model 522 for the SR area is used to classify the defects represented by the defect image into real defects or pseudo defects.

As described above, in this processing example, the area type information indicating the area type to which the detection area 72 belongs is included in the defect information. In the teacher's material generating unit 45, not only the result of the operator's determination of a real defect or a pseudo defect is marked on the cutout image, but also the area type to which the detection area 72 belongs is marked. Thereby, the learning unit 51 can generate a learned model for defect classification of a region type using a plurality of teacher materials labeled with the region type. In this way, by generating a learned model for each region type, the classification accuracy can be further improved.

The above-described teacher material generating device 4 and teacher material generating method may be modified in various ways.

The defect information input from the inspection device 2 to the teaching material generating device 4 only needs to represent the range of the detection area 72 in the defect image, and is not limited to the position and shape of the detection area 72 . For example, the defect information may also represent the range of the circumscribed rectangle of the detection area 72 in the defect image (that is, the ranges in the up and down directions and the left and right directions).

The area of the defect image to be cut out as the cutout image is determined based on the defect information and includes the detection area 72 , and is preferably an area substantially circumscribing the detection area 72 . The area substantially circumscribing the detection area 72 includes not only the area circumscribing the detection area 72 but also the area circumscribing the area obtained by expanding the detection area 72 by the expansion amount described above.

In the above-described embodiment, in step S14 of FIG. 4 , the operator inputs the determination result of a true defect or a pseudo defect with respect to the defect image. However, defect types other than true defects and pseudo defects (such as foreign matter attachment, etc.) may also be input. film peeling, etc.). That is, the determination result accepting unit 44 accepts the determination result of the defect type (the determination result including a true defect or a pseudo defect) input by the operator with respect to the defect image displayed on the display 35 .

In the second embodiment, when the detection area 72 includes parts belonging to two or more different area types, any one of the two or more area types may be used in the expansion of the detection area 72 The corresponding expansion amount. From the viewpoint of obtaining a suitable cropped image representing substantially the entire defect area 71, it is preferable to use the largest expansion amount among the expansion amounts corresponding to the two or more area types.

In the third embodiment, when the detection area 72 includes parts detected with two or more different inspection sensitivities, any one of the two or more inspection sensitivities can be used for the expansion of the detection area 72 . The expansion corresponding to the gain. From the viewpoint of obtaining a preferable cropped image representing substantially the entire defect area 71 , it is preferable to use the largest expansion amount among the expansion amounts corresponding to the two or more inspection sensitivities.

The object inspected by the inspection device 2 may be a substrate such as a semiconductor substrate or a glass substrate in addition to a printed circuit board. Furthermore, the inspection device 2 can also detect defects in objects other than substrates such as mechanical parts. The teacher data generating device 4 can easily generate better teacher data for generating a learned model for classifying defects of various objects.

The configurations in the above-described embodiments and modifications may be appropriately combined within the scope of not contradicting each other.

The present invention has been described and explained in detail above, but the description is only illustrative and not restrictive. Therefore, a plurality of modifications or forms may be adopted without departing from the scope of the present invention.

1: Check the system 2: Check the device 3:Computer 4:Teacher data generation device 9:Printed substrate 30:Bus 31:CPU 32:ROM 33: RAM 34:Hard disk 35:Display 36:Input part 36a:Keyboard 37: Reading device 38: Ministry of Communications 39:GPU 41:Image receiving department 42:Cut image generation part 43: Display control part 44: Judgment Result Acceptance Department 45:Teacher Information Generation Department 51:Learning Department 52:Classifier 61: Plating area 62:SR area 71: Defect area 72:Detection area 73: Encircled rectangle 74:Area 81:Recording media 91: First gain setting area 92: Second gain setting area 93: The third gain setting area 521,522: Model learned 621: First SR area 622:Second SR area 711: Defect partial area 721: Detect some areas 811:Program S11 to S15: Steps

[Fig. 1] is a diagram showing the structure of the inspection system. [Fig. 2] is a diagram showing the structure of a computer. [Fig. 3] is a diagram showing the structure of a teacher material generating device. [Fig. 4] is a diagram showing the processing flow of generating teacher data. [Fig. 5] is a diagram showing a captured image. [Fig. 6] is a diagram showing a captured image. [Fig. 7] is a diagram showing a captured image. [Fig. 8A] is a diagram showing a defect image. [Fig. 8B] is a diagram showing a defect image. [Fig. 9] is a diagram showing a defect image. [Fig. 10] is a diagram showing the vicinity of a defective area. [Fig. 11] is a diagram showing a defect image. [Fig. 12] is a diagram showing the vicinity of a defective area. [Fig. 13] is a diagram showing the vicinity of a defective area. [Fig. 14] is a diagram showing the vicinity of a defective area. [Fig. 15] is a diagram showing a printed circuit board. [Fig. 16] is an enlarged view of a part of the printed circuit board. [Fig. 17] is a diagram showing another example of the classifier.

2: Check the device

4:Teacher data generation device

35:Display

36:Input part

41:Image receiving department

42:Cut image generation part

43: Display control part

44: Judgment Result Acceptance Department

45:Teacher Information Generation Department

Claims (11)

A teacher information generating device generates teacher information and has: The image receiving unit receives from an inspection device for photographing an object to detect defects: a defect image of a predetermined size, which is a detection area including the defect; and defect information, which indicates the range of the detection area in the defect image. ; The cutout image generation unit cuts out the area including the detection area from the aforementioned defect image as a cutout image based on the aforementioned defect information; The display control unit causes at least part of the aforementioned defective image to be displayed on the display; The determination result accepting unit accepts the determination result of the defect type input by the operator with respect to the defect image displayed on the display; and The teacher data generation unit marks the judgment result on the cut image to generate teacher data. The teacher material generating device as described in claim 1, wherein each position of the aforementioned object belongs to one of a plurality of area types; The aforementioned defect information includes area type information, and the aforementioned area type information indicates the area type to which the aforementioned detection area belongs; The cropped image generating unit stores the expansion amount set for each region type, and includes in the cropped image a region obtained by expanding the detection region according to the expansion amount specified using the region type information. The teacher material generating device according to claim 2, wherein the object is a printed substrate; The plurality of aforementioned area types include at least a plating area and a solder resist area. The teacher data generating device according to any one of claims 1 to 3, wherein one of a plurality of inspection sensitivities is set for each position of the aforementioned object; The aforementioned defect information includes inspection sensitivity information, and the aforementioned inspection sensitivity information represents the inspection sensitivity used in the inspection of the aforementioned inspection area; The clipped image generating unit stores the expansion amount set for each inspection sensitivity, and includes in the clipped image a region obtained by expanding the detection area according to the expansion amount specified using the inspection sensitivity information. The teacher material generating device as described in any one of claims 1 to 3, wherein each position of the aforementioned object belongs to one of a plurality of area types; The aforementioned defect information includes area type information, and the aforementioned area type information indicates the area type to which the aforementioned detection area belongs; The teacher information generating unit marks not only the judgment result but also the area type to which the detection area belongs to the cutout image. A method for generating teacher information, which generates teacher information and has the following features: Step a is to receive from an inspection device used to photograph objects to detect defects: a defect image of a predetermined size, which contains the detection area of the defect; and defect information, which represents the range of the detection area in the aforementioned defect image; Step b is to cut out the area including the detection area from the aforementioned defect image as a cut image based on the aforementioned defect information; Step c is to display at least part of the aforementioned defect image on the display; Step d is to accept the determination result of the defect type input by the operator regarding the defect image displayed on the display; and Step e is to mark the aforementioned judgment result on the aforementioned cut image to generate teacher information. The method for generating teacher data as described in claim 6, wherein each location of the aforementioned object belongs to one of a plurality of area types; The aforementioned defect information includes area type information, and the aforementioned area type information indicates the area type to which the aforementioned detection area belongs; In the aforementioned step b, the expansion amount set for each area type is prepared, and the area obtained by expanding the detection area according to the expansion amount specified using the area type information is included in the cutout image. The method for generating teacher materials as described in claim 7, wherein the aforementioned object is a printed substrate; The plurality of aforementioned area types include at least a plating area and a solder resist area. The method for generating teacher data as described in any one of claims 6 to 8, wherein one of a plurality of inspection sensitivities is set for each position of the aforementioned object; The aforementioned defect information includes inspection sensitivity information, and the aforementioned inspection sensitivity information represents the inspection sensitivity used in the inspection of the aforementioned inspection area; In the aforementioned step b, an expansion amount set for each inspection sensitivity is prepared, and an area obtained by expanding the detection area according to the expansion amount specified using the inspection sensitivity information is included in the cutout image. The method for generating teacher materials as described in any one of claims 6 to 8, wherein each location of the aforementioned object belongs to one of a plurality of area types; The aforementioned defect information includes area type information, and the aforementioned area type information indicates the area type to which the aforementioned detection area belongs; In the aforementioned step e, the aforementioned cut image is marked with not only the aforementioned determination result but also the area type to which the aforementioned detection area belongs; Using a plurality of teacher data labeled with a region type, a learned model for classifying defects of the aforementioned region type is generated. A program for causing a computer to generate teacher information, and causing the computer to execute the program by causing the computer to execute: Step a is to receive from an inspection device used to photograph objects to detect defects: a defect image of a predetermined size, which contains the detection area of the defect; and defect information, which represents the range of the detection area in the aforementioned defect image; Step b is to cut out the area including the detection area from the aforementioned defect image as a cut image based on the aforementioned defect information; Step c is to display at least part of the aforementioned defect image on the display; Step d is to accept the determination result of the defect type input by the operator regarding the defect image displayed on the display; and Step e is to mark the aforementioned judgment result on the aforementioned cut image to generate teacher information.

Images