JP2020135051A - Fault inspection device, fault inspection method, fault inspection program, learning device and learned model - Google Patents

Abstract

To provide a fault inspection device, fault inspection method, fault inspection program, learning device and learned model that can highly accurately detect a fault even if a fault size is quite small in an inspection image acquired from a product targeted for an inspection.SOLUTION: A fault inspection device comprises: an image processing unit that inputs data on an image data to a learned model created by machine learning using a first image group made of an image in which an inspection-purpose image includes a fault of a pixel and a second image group made of an image not including the fault, and for repairing only the fault included in the image, performs a computation based on the learned model, and thereby outputs data on an output image; a calculation unit that compares regions corresponding to the input image and output image to thereby calculate difference information quantifying a difference between both image; and a determination unit that determines whether the input image includes the fault or not on the basis of the difference information calculated by the calculation unit.SELECTED DRAWING: Figure 2

Description

The present invention relates to a defect inspection device, a defect inspection method, a defect inspection program, a learning device, and a trained model.

The methods for inspecting the appearance of a product are (1) a visual method, (2) a method using an image processing algorithm that emphasizes defects in appearance and extracts defects from features such as size and shape. It can be roughly divided into (3) a method by comparison with a preset normal sample and (4) a method using machine learning.

Of these, the above (1) has a problem that although it has high sensitivity, it takes a lot of manpower to perform the inspection, and a problem that the inspection sensitivity differs depending on the person performing the inspection and his / her physical condition.

The above (2) has a problem that it is difficult to design an algorithm when the feature of the detection target cannot be clearly defined. Further, the above (2) has a problem that even if the characteristics of the defects can be clearly defined, the skill of a skilled engineer is required for the design of the algorithm, and the development takes time. Further, the above (2) also has a problem that it is difficult to extract only the defective portion when the shape or structure of the product is complicated.

Although the above (3) can be dealt with even when the shape and structure of the product are complicated, when there are multiple inspection targets, it is necessary to carry out comparative calculation for all the normal images of the corresponding products. Therefore, there is a problem that the calculation cost increases. Further, the above (3) has a problem that the difference between the normal image and the abnormal image is caused by the difference in color, the difference in dimensions, and the like, and the abnormal image does not necessarily show a defective image.

Under such circumstances, the method using machine learning in (4) above is drawing attention. According to this method, even if the feature to be detected cannot be clearly defined, the feature can be acquired by learning. Therefore, there is an advantage that it is only necessary to prepare an image to be inspected and it is not necessary to design an algorithm from scratch for each inspection target.

As an abnormality detection method using machine learning, local features that are invariant with respect to lighting changes, rotation, and scaling on the image are acquired, and abnormality detection is performed by using machine learning for these local features. A method of detecting defects in an image having a local feature amount determined to be abnormal has been proposed (see, for example, Patent Document 1).

In addition, as an image inspection method using machine learning, by using the Z score calculated as a feature quantity and the joint probability, the burden rate of the model of a good image generated by machine learning (expected value classified into the model). ), And a method of detecting defects by using the weighted pixel average value and the weighted pixel dispersion value with this burden rate as a weight has been proposed (see, for example, Patent Document 2).

JP-A-2016-110290 Japanese Unexamined Patent Publication No. 2017-167624

However, in the technique of Patent Document 1 described above, among the images for inspection for detecting defects, when the image to be inspected has a plurality of types of fine patterns and the defect size generated is extremely small, or the product When the shape of the fine pattern is the inspection target, the difference in the amount of local features in the acquired fine pattern in the region where the defect occurs is small, and the detection accuracy of the defect may be lowered.

Further, in the technique of Patent Document 2 described above, since the influence of minute defects on the weighted pixel average value and the weighted pixel dispersion value weighted by the burden rate of the model of a good product is small, the defect detection accuracy is sufficiently high. It was difficult to say.

The present invention has been made in view of the above, and is a defect inspection device capable of detecting defects with high accuracy even when the defect size is extremely small in the inspection image acquired from the product to be inspected. , Defect inspection methods, inspection programs, learning devices, and trained models.

In order to achieve the above object, the defect inspection device of the present invention is a defect inspection device for determining whether or not an image for inspection contains defects of pixels, and is a first image group composed of images including the defects. The data of the input image is input to the trained model for repairing only the defects included in the image, which is generated by machine learning using the second image group consisting of the image not including the defects. Difference information that quantifies the difference between the two images by comparing the image processing unit that outputs the data of the output image by performing the calculation based on the trained model and the corresponding area between the input image and the output image. A calculation unit for calculation and a determination unit for determining whether or not the input image includes the defect based on the difference information calculated by the calculation unit are provided.

Further, in the above invention, the defect inspection apparatus of the present invention includes a first image included in the first image group and an image included in the second image group in a region corresponding to the first image. Further, a learning unit that acquires a second image, calculates the difference information for the first image and the second image, and sequentially optimizes the parameter group of the trained model according to the calculation result. Be prepared.

Further, in the above invention, the defect inspection apparatus of the present invention can switch between an inspection mode in which the image processing unit, the calculation unit and the determination unit are operated, and a learning mode in which the learning unit is operated.

Further, in the defect inspection apparatus of the present invention, in the above invention, the calculation unit calculates the difference information for each pixel, and the determination unit determines the position of the defect based on the calculation result of the calculation unit.

In addition, the defect inspection device of the present invention further includes a classification unit that classifies the types of defects possessed by the input image based on the difference information in the above invention.

Further, in the defect inspection apparatus of the present invention, in the above invention, the data of the output image output by the trained model is the same as the data of the input image except for the defects.

Further, the defect inspection apparatus of the present invention further includes a defect extraction image generation unit that generates data of a defect extraction image in which only defects are extracted based on the difference information in the above invention.

Further, in the above invention, the defect inspection apparatus of the present invention displays on the display unit one or more selected from the group consisting of the determination result of the determination unit, the input image, the output image, and the defect extraction image. It also has a part.

Further, in the above-mentioned invention, the defect inspection apparatus of the present invention is an intermediate layer having a multi-layer structure in which an input layer into which data of the input image is input and a signal output by the input layer are input in the trained model. It is composed of a neural network having an output layer to which a signal output by the intermediate layer is input and outputting data of the output image, and each layer has one or a plurality of nodes.

Further, in the defect inspection apparatus of the present invention, in the above invention, the image data of the first image group and the second image group are quantized, and the image processing unit uses the input image data. After quantization, it is input to the trained model.

Further, in the defect inspection apparatus of the present invention, in the above invention, the quantization is binarization.

Further, in the defect inspection apparatus of the present invention, in the above invention, the trained model is a first quantized image group and a second quantized image group obtained by quantizing the image data of the first image group and the second image group, respectively. It is generated by machine learning using a quantized image group, the input layer further inputs data obtained by quantizing the input image, and the output layer receives data obtained by quantizing the output image. Further output.

Further, the defect inspection device of the present invention is a convolutional neural network of the type in which the mesosphere does not have a fully connected layer and a pooling layer in the above invention.

Further, in the defect inspection device of the present invention, in the above invention, the neural network includes a bypass path.

The defect inspection method of the present invention is a defect inspection method performed by a defect inspection device for determining whether or not an image for inspection contains defects of pixels, and is a first image group composed of an image including the defects and the defects. The trained model is generated by machine learning using a second image group consisting of images that do not contain the above, and the data of the input image is input to the trained model for repairing only the defects contained in the image. Calculation that quantifies the difference between the two images by comparing the image processing step that outputs the data of the output image by performing the calculation based on the above and the corresponding area between the input image and the output image. It includes a step and a determination step of reading the difference information calculated in the calculation step from the storage unit and determining whether or not the input image includes the defect.

The defect inspection program of the present invention comprises a defect inspection device for determining whether or not an image to be inspected contains pixel defects, and includes a first image group consisting of images including the defects and an image not including the defects. By inputting the data of the input image to the trained model for repairing only the defect contained in the image, which is generated by machine learning using the two image groups, and performing the calculation based on the trained model. In the image processing step of outputting the data of the output image, the calculation step of calculating the difference information quantifying the difference between the two images by comparing the corresponding areas of the input image and the output image, and the calculation step. Based on the calculated difference information, a determination step of determining whether or not the input image includes the defect is executed.

The learning device of the present invention is a learning device that generates a trained model for operating a computer so as to output image data obtained by repairing only pixel defects from an image, and comprises an image including the defects. The first image included in the first image group and the second image included in the second image group not including the defect and which is an image of the region corresponding to the first image are acquired, and the first image is obtained. It is provided with a learning unit that repeats a process of calculating difference information quantifying the difference between an image and the second image and sequentially optimizing a parameter group of the trained model according to the calculation result.

In the trained model of the present invention, an input layer into which input image data is input and a signal output by the input layer are input, and an intermediate layer having a multi-layer structure and a signal output by the intermediate layer are input. It has an output layer that outputs data of an output image in which only the defects of the input image are repaired, and each layer is a neural network composed of one or a plurality of nodes, and the intermediate layer has a fully connected layer and a pooling layer. It is composed of a convolutional neural network of a non-type, inputs the data of the input image to the input layer, performs an operation based on the learned network parameters which are network parameters of the neural network, and outputs the image from the output layer. Make the computer function to output the data of.

In the trained model of the present invention, in the above invention, the input layer further inputs the quantized data of the input image, and the output layer further outputs the quantized data of the output image.

The term "repair" in the present invention means to create a normal image by removing a defect portion in an image of a normal product, that is, an image in which defects such as adhesion or chipping exist in the normal image, that is, a defect image. , Means.

According to the present invention, defects can be detected with high accuracy even when the defect size is extremely small in the inspection image obtained from the product to be inspected.

FIG. 1 is a diagram showing an overall configuration of a defect inspection system according to the first embodiment. FIG. 2 is a block diagram showing a functional configuration of the defect inspection system according to the first embodiment. FIG. 3 is a diagram showing an outline of a convolutional neural network used by the defect inspection apparatus according to the first embodiment. FIG. 4 is a flowchart showing an outline of the processing performed by the defect inspection apparatus according to the first embodiment. FIG. 5 is a diagram schematically showing a specific processing flow of the defect inspection apparatus according to the first embodiment. FIG. 6 is a diagram schematically showing a specific processing flow of the defect inspection apparatus according to the second embodiment. FIG. 7 is a diagram schematically showing a specific processing flow performed by the defect inspection apparatus according to the third embodiment. FIG. 8 is a block diagram showing a functional configuration of the defect inspection system according to the fourth embodiment. FIG. 9 is a flowchart showing an outline of the processing performed by the defect inspection apparatus according to the fourth embodiment. FIG. 10 is a flowchart showing an outline of the processing performed by the defect inspection apparatus according to the modified example of the fourth embodiment.

Hereinafter, embodiments for carrying out the present invention (hereinafter, referred to as “embodiments”) will be described in detail with reference to the drawings. The present invention is not limited to the embodiments described below.

(Embodiment 1)
FIG. 1 is a diagram showing an overall configuration of a defect inspection system according to the first embodiment. FIG. 2 is a block diagram showing a functional configuration of the defect inspection system according to the first embodiment. The defect inspection system 1 shown in these figures has a defect of performing defect inspection based on a data acquisition device 2 that acquires image data of a product P conveyed on a transport path L and image data acquired by the data acquisition device 2. It includes an inspection device 3 and a display device 4 for displaying the defect inspection result of the defect inspection device 3. Note that FIG. 1 schematically shows a situation in which the data acquisition device 2 acquires image data of the product P from vertically above the transport path L.

The data acquisition device 2 has an image pickup device and is electrically connected to the defect inspection device 3. The image pickup device is configured by using a solid-state image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

The defect inspection device 3 includes a data acquisition unit 31, an image processing unit 32, a calculation unit 33, a determination unit 34, a control unit 35, and a storage unit 36. The defect inspection device 3 is electrically connected to the data acquisition device 2 and the display device 4.

The data acquisition unit 31 acquires data such as image data sent from the data acquisition device 2, outputs the data to the storage unit 36, and stores the data.

The image processing unit 32 inputs the data of the input image into the trained model for repairing only the defects included in the image, and outputs the data of the output image by performing the calculation based on the trained model. The trained model is pre-generated by machine learning using a first image group consisting of images including defects and a second image group consisting of images not including defects. As the machine learning used here, for example, deep learning using a convolutional neural network can be mentioned. Information such as network parameters in the trained model is stored in the storage unit 36.

FIG. 3 is a diagram showing an outline of a convolutional neural network. The convolutional neural network (hereinafter referred to as CNN) 100 shown in the figure has an input layer 101, an intermediate layer 102, and an output layer 103. The intermediate layer 102 has a plurality of sets including the convolution layer 121, the batch normalization layer 122, and the activation function 123, and the batch normalization layer 122 is provided on the output side of each convolution layer 121. The intermediate layer 102 does not have a fully bonded layer and a pooling layer. Further, the CNN 100 has a bypass route that does not pass from the input layer 101 to the intermediate layer 102. The sum of the output of the intermediate layer 102 and the output of the input layer 101 via the bypass path is input to the output layer 103.

In the CNN 100, the dimensions of the input layer 101 and the output layer 103 need to be the same. That is, the input image data input to the CNN 100 and the output image data output by the CNN 100 need to have the same width, height, and number of channels. Here, the number of channels is the number of dimensions in the depth direction of the image, for example, 1 in the case of a grayscale image and 3 in the case of an RGB color image. The reason why the dimensions of the input layer 101 and the output layer 103 need to be the same is that when the calculation unit 33 compares the input image and the output image, the comparison process is performed because there are no corresponding pixels between the image data. This is to avoid the situation where the above cannot be implemented.

The CNN 100 does not have to reduce the dimension of the feature amount map in the intermediate layer 102 by performing external zero filling or the like in the folding layer 121. Further, as the activation function 123, for example, Relu can be mentioned.

By performing the calculation using the CNN 100, the image processing unit 32 can create the data of the normal image corresponding to the input image without suppressing the reduction of the dimension and deteriorating the fine features. That is, when the image processing unit 32 performs an operation using the CNN 100, when the input image contains defects, it is possible to obtain an output image in which only the defects are repaired. If the input image does not contain any defects, the image processing unit 32 performs an operation using the CNN 100, and the input image is output as it is.

The calculation unit 33 calculates the difference information that quantifies the difference between the two images by comparing the corresponding regions of the input image and the output image. As the comparison operation performed by the calculation unit 33, for example, SSD (Sum of Squared Difference) or SAD (Sum of Absolute Difference), which is a general method for comparing the difference between two image data, can be used. .. The calculation unit 33 creates the data of the difference image by the difference of the brightness value for each pixel of the input image and the output image. As a result, it is possible to detect minute defects that occur locally in the inspection image.

The calculation unit 33 calculates the difference information according to the type of the defect to be detected. For example, when the detection target is a bright defect (a defect whose brightness value is higher than that of the surroundings), the calculation unit 33 generates the difference image data by subtracting the brightness value of the input image from the brightness value of the output image. , The number of pixels whose luminance value is equal to or higher than a predetermined threshold in the data is measured. In this case, the calculation unit 33 uses the sum of the number of adjacent pixels among the pixels whose luminance value is equal to or greater than the threshold value, that is, the information corresponding to the size of the defect portion as the difference information.

Further, when the detection target is a dark defect (a defect that the brightness value is lower than that of the surroundings), the calculation unit 33 subtracts the brightness value of the output image from the brightness value of the input image for each pixel to obtain the data of the difference image. The pixels that are generated and whose luminance value is positive and whose luminance value exceeds a predetermined threshold value (different from the above threshold value) in the data are extracted. This makes it possible to extract only dark defects with high accuracy.

The determination unit 34 determines whether or not the input image contains defects by comparing the evaluation value calculated by the calculation unit 33 with a predetermined reference value. As a method of setting the reference value, for example, a method of artificially setting or a method of automatically setting can be considered.

In the method of artificially setting the reference value, when the defect detection standard is clear, for example, by setting the threshold value of the total number of defects to 1, it is possible to detect minute defects of one pixel or more. Further, for example, by setting the reference value of the area of the connected components on the image to 10, it is possible to detect defects robustly to the influence of noise.

On the other hand, in the method of automatically setting the reference value, the statistic of the evaluation value group by comparison with the output image group acquired in advance from a plurality of input image groups is used. As the statistic, for example, the standard deviation or the maximum value can be used.

The control unit 35 controls the operation of the defect inspection system 1 in an integrated manner. The control unit 35 has a display control unit 37 that displays the difference image generated by the calculation unit 33 and the determination result of the determination unit 34 on the display device 4. The display control unit 37 displays a plurality of difference images for each defect type on the display device 4, superimposes the position information of the defect on the original image and displays it on the display device 4, or enlarges only the peripheral pixels of the defect. It is also possible to display an image on the display device 4.

The storage unit 36 stores data including various programs for operating the defect inspection device 3 and various parameters necessary for the operation of the defect inspection device 3. The various programs also include a trained model in which the image processing unit 32 executes an operation. The storage unit 36 includes a ROM (Read Only Memory) in which various programs and the like are pre-installed, a RAM (Random Access Memory) for storing calculation parameters and data of each process, an HDD (Hard Disk Drive), and an SSD (Solid State). Drive) etc. are used.

Various programs can be recorded on a computer-readable recording medium such as an HDD, a flash memory, a CD-ROM, a DVD-ROM, or a Blu-ray (registered trademark) and widely distributed. In addition, the data acquisition unit 31 can acquire various programs via the communication network. The communication network referred to here is realized by, for example, an existing public line network, LAN (Local Area Network), WAN (Wide Area Network), etc., and may be wired or wireless.

The defect inspection device 3 having the above functional configuration includes one or more hardware such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), and an FPGA (Field Programmable Gate Array). Constructed using.

The display device 4 is a display made of a liquid crystal or an organic EL (Electro Luminescence), and is electrically connected to the defect inspection device 3. The display device 4 acquires and displays the display data output from the defect inspection device 3 under the control of the display control unit 37.

FIG. 4 is a flowchart showing an outline of the processing performed by the defect inspection device 3. First, the data acquisition unit 31 acquires image data from the data acquisition device 2 (step S1).

Subsequently, the image processing unit 32 performs image processing by performing an operation using the trained model (step S2). Specifically, the image processing unit 32 outputs the data of the output image by inputting the data of the input image into the CNN 100 and performing the calculation.

After that, the calculation unit 33 calculates the difference information that quantifies the difference between the input image and the output image by comparing the corresponding regions (step S3).

Subsequently, the determination unit 34 determines whether or not the input image contains defects based on the difference information (step S4).

After that, the display control unit 37 causes the display device 4 to display the determination result (step S5). The display control unit 37 causes the display device 4 to display the difference image which is the calculation result of the calculation unit 33 and the determination result of the determination unit 34.

FIG. 5 is a diagram schematically showing a specific processing flow of the defect inspection device 3, and is a diagram schematically showing a detection flow when detecting a dark defect. The elliptical pattern at the lower right of the input image 201 includes defects due to dark defects. The image processing unit 32 inputs the data of the input image 201 into the CNN 100 and performs an operation to create the data of the output image 202 with the defective part repaired. The output image 202 is a normal image in which only the defective portion is repaired.

Subsequently, the calculation unit 33 creates the data of the difference image 203 by subtracting the input image from the output image for each pixel, and calculates the evaluation value of the difference image 203 as the difference information. The difference image 203 is an image in which only the defective portion of the input image 201 is emphasized. The evaluation value at this time is, for example, the brightness value of the pixel.

The determination unit 34 determines that the product of the input image 201 includes defects when there are pixels in which the brightness value of the difference image 203 calculated by the calculation unit 33 is equal to or greater than a predetermined threshold value.

According to the first embodiment described above, only the defects included in the image are generated by machine learning using the first image group consisting of the images including the defects and the second image group consisting of the images not including the defects. To output the data of the output image by inputting the data of the input image to the trained model to be repaired and performing the calculation based on the trained model, and to compare the corresponding areas of the input image and the output image. To calculate the difference information that quantifies the difference between the two images, and to determine whether the input image contains defects based on the difference information calculated by the calculation unit, for inspection obtained from the product to be inspected. Defects can be detected with high accuracy even when the defect size is extremely small in the image.

Further, according to the first embodiment, when the inspection image (input image) acquired in the manufacturing process of the product contains defects, the normal image in which the defects are repaired using the trained model generated by machine learning. (Output image) is created, the inspection image and the normal image are compared, and the difference information is calculated. Therefore, even if the shape of the product is complicated and the defects are minute, the defects are highly accurate. Can be detected.

(Embodiment 2)
The defect inspection device according to the second embodiment detects defects caused by a molding error of the product. Since the configuration of the defect inspection system according to the second embodiment is the same as that of the first embodiment, the same components as those of the first embodiment will be described using the same reference numerals as those of the first embodiment. However, the specific configuration of each layer of the CNN 100 is generally different from that of the first embodiment.

In the second embodiment, the image processing unit 32 performs preprocessing on the data of the input image. The preprocessing performed by the image processing unit 32 is a quantization process for converting pixel values into discrete values. Hereinafter, as an example of the quantization process, the case of the binarization process in which the pixel values are discretized by two values will be described, but the preprocessing of the second embodiment is not limited to the binarization process.

The calculation unit 33 creates data for a difference image by subtracting a brightness value from the output image for each pixel of the input image, and measures the number of pixels whose brightness value is equal to or greater than a certain value in the difference image. When a molding error occurs in the product, the value of the difference information calculated by the calculation unit 33, that is, the evaluation value of the difference image is equal to or higher than a predetermined threshold value.

FIG. 6 is a diagram schematically showing a specific processing flow of the defect inspection device 3 according to the second embodiment. First, the image processing unit 32 performs binarization processing on the data of the input image 301 (original image).

After that, the image processing unit 32 inputs the binarized input image 302 data to the CNN 100 and performs an operation to output the data of the output image 303 with the defective portion repaired.

Subsequently, the calculation unit 33 creates the data of the difference image 304 by subtracting the luminance value of the binarized input image 302 from the output image 303 for each pixel, and calculates the difference information of the difference image 304, that is, the evaluation value. To do.

The determination unit 34 determines whether or not the input image 301 includes defects based on the evaluation value calculated by the calculation unit 33. When the evaluation value is equal to or greater than the threshold value, the determination unit 34 determines that the input image 301 contains defects.

According to the second embodiment described above, as in the first embodiment, defects can be detected with high accuracy even when the defect size is extremely small in the inspection image acquired from the product to be inspected. ..

Further, according to the second embodiment, the luminance information of the product is lost by performing the quantization process such as the binarization process on the data of the input image, and the difference focusing only on the shape information is detected. It becomes possible. Therefore, defects caused by molding errors of the product can be detected with high accuracy.

(Embodiment 3)
The defect inspection device according to the third embodiment detects bright defects and defects caused by molding errors of the product. Since the configuration of the defect inspection system according to the third embodiment is the same as that of the first embodiment, the same components as those of the first embodiment will be described using the same reference numerals as those of the first embodiment.

In the third embodiment, the image processing unit 32 performs quantization processing such as binarization processing on the input image data as preprocessing, and separates the binarized input image data and the original input image data. Input to CNN100 through the channel of, and output the data of two output images corresponding to each input image.

The calculation unit 33 creates the data of the difference image from the set of the original input image and the output image, and also creates the data of the difference image from the set of the input image and the output image subjected to the quantization process, and creates the data of the difference image from the set of the input image and the output image. Each evaluation value is calculated based on the data.

The determination unit 34 determines the presence or absence of bright defects from the evaluation value based on the data of the difference image created from the set of the original input image and the output image, while creating from the set of the input image and the output image subjected to the quantization process. From the evaluation value based on the data of the difference image, the presence or absence of defects due to the molding error of the product is determined.

FIG. 7 is a diagram schematically showing a specific processing flow performed by the defect inspection apparatus according to the third embodiment. First, the image processing unit 32 generates the data of the input image 402 obtained by binarizing the input image 401. Subsequently, the image processing unit 32 inputs the data of the two input images 401 and 402 to the CNN 100, and outputs the output image 403 in which the original input image 401 is restored and the output in which the input image 402 after the binarization process is restored. The data of the image 404 is output. FIG. 7 illustrates a case where a bright defect occurs in the rectangular pattern in the upper left part of the image and a defect due to a molding error of the product occurs in the elliptical pattern in the lower right part.

Subsequently, the calculation unit 33 generates data of two difference images 405 and 406 using the input image 401 and the output image 403, and the input image 402 and the output image 404, respectively. In the case shown in FIG. 7, in the difference image 405, only the defect portion 451 due to the bright defect is extracted, and in the difference image 406, only the defect portion 461 due to the molding error is extracted. The methods for generating the difference images 405 and 406 are the same as the methods for generating the difference images described in the first and second embodiments, respectively.

After that, the calculation unit 33 calculates the evaluation values for the difference images 405 and 406, respectively. The calculation of the evaluation value for the difference image 405 is the same as that described in the first embodiment, and the calculation of the evaluation value for the difference image 406 is the same as that described in the second embodiment.

The determination unit 34 determines the presence or absence of defects based on the two evaluation values. In the case shown in FIG. 7, the determination unit 34 includes two types of defects, that is, a defect (defect portion 451) appearing in the difference image 405 and a defect (defect portion 461) appearing in the difference image 406, in the input image 401. judge.

After that, when the display control unit 37 causes the display device 4 to display the determination result, the information about the type of the defect determined by the determination unit 34 may be further displayed.

According to the third embodiment described above, as in the first embodiment, defects can be detected with high accuracy even when the defect size is extremely small in the inspection image acquired from the product to be inspected. ..

Further, according to the third embodiment, since the determination is performed using the original input image and the input image subjected to the quantization process, two different types of defects can be inspected at the same time. As a result, two types of defect inspections can be performed at the same time while reducing the consumption of calculation resources and the calculation time. Although two types of defects have been described here, it is also possible to simultaneously inspect a plurality of three or more types of defects, such as detecting dark defects.

(Embodiment 4)
FIG. 8 is a block diagram showing a functional configuration of the defect inspection system according to the fourth embodiment. The defect inspection system 1A shown in the figure includes a data acquisition device 2, a defect inspection device 3A, and a display device 4.

The defect inspection device 3A has a learning unit 38 that performs learning to optimize the network parameters of the CNN 100 in addition to the functional configuration of the defect inspection device 3. The learning unit 38 includes data of the first image included in the first image group consisting of images including defects, and images included in the second image group not including defects and in a region corresponding to the first image. The data of a certain second image is acquired, the difference information that quantifies the difference between the first image and the second image is calculated, and the parameter group (network parameter) of the trained model is sequentially optimized according to the calculation result. The process of converting is repeated. In this sense, the defect inspection device 3A has the function of the learning device according to the fourth embodiment.

The learning unit 38 prepares a sufficient number of sets of teacher images corresponding to the input images to make a learning image pair, and performs deep learning using, for example, a miscalculation back propagation method and a stochastic gradient descent method. Optimize the network parameters of CNN100. When the input image is a defect image, the teacher image is a normal image corresponding to the input image, and when the input image is a normal image, the teacher image is the same as the input image. In this way, by performing deep learning based on the learning image pair, the learning unit 38 can learn a method of repairing only the defective portion of the defective image and creating a normal image.

FIG. 9 is a flowchart showing an outline of the processing performed by the defect inspection device 3A. Steps S1 to S5 are the same as the processing of the defect inspection device 3 described in the first embodiment. The specific processing content of steps S1 to S5 is the same as the processing content of any of the first to third embodiments.

In step S6, the learning unit 38 updates the trained model by optimizing the network parameters of the CNN 100 using the data of the input image and the data of the corresponding teacher image.

According to the fourth embodiment described above, as in the first embodiment, defects can be detected with high accuracy even when the defect size is extremely small in the inspection image acquired from the product to be inspected. ..

Further, according to the fourth embodiment, an image processing algorithm capable of detecting an inspection image acquired in a manufacturing process of an arbitrary product with high accuracy even if it has a complicated background and has minute defects. Can be obtained by learning.

(Modified Example of Embodiment 4)
FIG. 10 is a flowchart showing an outline of the processing performed by the defect inspection device 3A according to the modified example of the fourth embodiment. In this modification, the defect inspection device 3A collectively performs the learning process at the end of the inspection.

When the defect inspection device 3A starts the inspection (step S10), the data acquisition unit 31 acquires data from the data acquisition device 2 (step S11). Subsequent steps S12 to S15 are the same as steps S2 to S5 in FIG. 9, respectively.

When the inspection is completed after step S15 (step S16: Yes), the process proceeds to step S17 and the learning unit 38 performs the learning process (step S17). After that, the defect inspection device 3A ends a series of processes.

If the inspection is not completed in step S16 (step S16: No), the defect inspection device 3A returns to step S11.

The user may be able to set whether to perform the learning process of the learning unit 38 sequentially or at the end of the inspection, or the control unit 35 automatically processes according to the inspection conditions such as the inspection content and the inspection amount. It may be configured to determine.

(Other embodiments)
Although the embodiments for carrying out the present invention have been described so far, the present invention should not be limited to the above-described embodiments 1 to 4. For example, the data acquisition device 2 is not essential, and the data acquisition unit 31 may acquire the data to be inspected via the communication network, or may acquire the data stored in the storage unit 36.

Further, machine learning is not limited to the deep learning described above, and for example, a support vector machine, a decision tree, a random forest, or AdaBoost may be used.

The present invention can be used for defect inspection of products in a manufacturing process.

1, 1A Defect inspection system 2 Data acquisition device 3, 3A Defect inspection device 4 Display device 31 Data acquisition unit 32 Image processing unit 33 Calculation unit 34 Judgment unit 35 Control unit 36 Storage unit 37 Display control unit 38 Learning unit 100 Convolutional neural network (CNN)
101 Input layer 102 Intermediate layer 103 Output layer 201, 301, 302, 401, 402 Input image 202, 303, 403, 404 Output image 204, 304, 405, 406 Difference image 231, 341, 451, 461 Defects

Claims (19)

A defect inspection device that determines whether an image for inspection contains pixel defects.
A trained model generated by machine learning using a first image group consisting of images including the defects and a second image group consisting of images not including the defects, and for repairing only the defects contained in the image. On the other hand, an image processing unit that outputs the data of the output image by inputting the data of the input image and performing the calculation based on the trained model.
A calculation unit that calculates difference information that quantifies the difference between the two images by comparing the corresponding areas of the input image and the output image.
A determination unit that determines whether or not the input image includes the defect based on the difference information calculated by the calculation unit.
Defect inspection device equipped with. The first image included in the first image group and the second image included in the second image group and which is an image of a region corresponding to the first image are acquired, and the first image and the first image and the second image are acquired. The defect inspection apparatus according to claim 1, further comprising a learning unit that calculates the difference information with respect to the second image and repeats a process of sequentially optimizing the parameter group of the trained model according to the calculation result. The defect inspection device according to claim 2, wherein the inspection mode for operating the image processing unit, the calculation unit, and the determination unit can be switched between the learning mode for operating the learning unit. The calculation unit calculates the difference information for each pixel.
The defect inspection device according to any one of claims 1 to 3, wherein the determination unit determines the position of a defect based on the calculation result of the calculation unit. The defect inspection apparatus according to any one of claims 1 to 4, further comprising a classification unit for classifying the types of defects possessed by the input image based on the difference information. The defect inspection device according to any one of claims 1 to 5, wherein the data of the output image output by the trained model is the same as the data of the input image except for the defects. The defect inspection apparatus according to any one of claims 1 to 6, further comprising a defect extraction image generation unit that generates data of a defect extraction image in which only defects are extracted based on the difference information. The defect inspection device according to claim 7, further comprising a display control unit that displays on the display unit one or more selected from the group consisting of the determination result of the determination unit, the input image, the output image, and the defect extraction image. The trained model is
An input layer into which the data of the input image is input and
A signal output from the input layer is input, and an intermediate layer having a multi-layer structure and an intermediate layer
It has an output layer to which a signal output by the intermediate layer is input and outputs data of the output image.
The defect inspection apparatus according to any one of claims 1 to 8, wherein each layer comprises a neural network having one or more nodes. The image data of the first image group and the second image group are quantized.
The image processing unit
The defect inspection apparatus according to any one of claims 1 to 9, wherein the data of the input image is quantized and then input to the trained model. The defect inspection apparatus according to claim 10, wherein the quantization is binarization. The trained model was generated by machine learning using the first quantized image group and the second quantized image group obtained by quantizing the image data of the first image group and the second image group, respectively. It is a thing
The input layer further inputs data obtained by quantizing the input image.
The defect inspection device according to claim 9, wherein the output layer further outputs data obtained by quantizing the output image. The neural network
The defect inspection apparatus according to claim 9 or 12, wherein the intermediate layer is a type of convolutional neural network having no fully connected layer and a pooling layer. The defect inspection device according to claim 13, wherein the neural network includes a bypass path. This is a defect inspection method performed by a defect inspection device that determines whether or not an image for inspection contains pixel defects.
A trained model generated by machine learning using a first image group consisting of images including the defects and a second image group consisting of images not including the defects, and for repairing only the defects contained in the image. On the other hand, an image processing step of inputting the data of the input image and outputting the data of the output image by performing the calculation based on the trained model, and
A calculation step of calculating difference information that quantifies the difference between the two images by comparing the corresponding regions of the input image and the output image, and
A determination step of reading the difference information calculated in the calculation step from the storage unit and determining whether or not the input image includes the defect.
Defect inspection method including. For defect inspection equipment that determines whether an image for inspection contains pixel defects
A trained model generated by machine learning using a first image group consisting of images including the defects and a second image group consisting of images not including the defects, and for repairing only the defects contained in the image. On the other hand, an image processing step of inputting the data of the input image and outputting the data of the output image by performing the calculation based on the trained model, and
A calculation step of calculating difference information that quantifies the difference between the two images by comparing the corresponding regions of the input image and the output image, and
A determination step for determining whether or not the input image includes the defect based on the difference information calculated in the calculation step, and
Defect inspection program to execute. It is a learning device that generates a trained model for operating a computer so as to output image data obtained by repairing only pixel defects from an image.
A first image included in the first image group consisting of images including the defects, and a second image which is an image included in the second image group not including the defects and is an image of a region corresponding to the first image. A learning unit that repeats the process of acquiring the above, calculating the difference information quantifying the difference between the first image and the second image, and sequentially optimizing the parameter group of the trained model according to the calculation result. A learning device to be equipped. The input layer where the input image data is input and
A signal output from the input layer is input, and an intermediate layer having a multi-layer structure and an intermediate layer
An output layer in which a signal output by the intermediate layer is input and output image data in which only defects of the input image are repaired is output.
Have,
Each layer is a neural network consisting of one or more nodes, and the intermediate layer is composed of a type of convolutional neural network having no fully connected layer and a pooling layer.
The computer functions so as to input the data of the input image into the input layer, perform an operation based on the network parameters of the neural network and learned network parameters, and output the data of the output image from the output layer. Trained model to let. Data obtained by quantizing the input image is further input to the input layer.
The trained model according to claim 18, wherein the output layer further outputs data obtained by quantizing the output image.

Images