TW202242390A - Defect inspection device, defect inspection method, and manufacturing method - Google Patents

Abstract

This defect inspection device for inspecting for a defect in an object under inspection on the basis of an image of the object under inspection comprises a plurality of defect identifiers that use prescribed machine learning models to identify different types of defects on the basis of an image. The types of defects identified by the individual defect identifiers are some of a prescribed number of defect types subject to identification by the defect inspection device. This configuration may be embodied as a defect inspection device, defect inspection method, or manufacturing method. This configuration makes it possible to facilitate defect inspection system management.

Description

Defect inspection device, defect inspection method, and manufacturing method

The present invention relates to a defect inspection device, a defect inspection method and a manufacturing method. The present invention relates, for example, to a technique for determining the state of a defect generated in an object to be inspected using image data showing an image of the object to be inspected. This case claims the priority of Japanese Patent Application No. 2020-207561 filed in Japan on December 15, 2020, and its contents are incorporated herein.

When managing the quality of products, it is more important to determine the status of products. For example, in a glass substrate whose surface is covered with a thin film such as a metal or an oxide, fine electronic components or the like may be formed on the surface. This glass substrate can be used for various displays, such as a liquid crystal display, a photomask, an electronic device support, an information recording medium, a planar antenna, etc., for example. Defects generated on the surface of the glass substrate cause defects mainly including disconnection. Defects such as damage, dirt, etc. Therefore, high cleanliness, flatness, etc. are required on the surface of the glass substrate.

In order to reduce defects such as disconnection, it is considered to analyze the state of the surface of the glass substrate, determine the state of the defect, and specify the cause of the defect as needed, and implement countermeasures against the manufacturing process. Therefore, it is possible to try to classify the types of defects using a machine learning model on the images of the inspected object. For example, Patent Document 1 describes a defect inspection method for classifying defects generated on a wafer as a substrate using a deep learning (Deep Learning) model.

Generally, deep learning models need to use a large amount of training data to learn model parameters that show the relationship between input and output in advance. Depending on the amount of training data used for learning, there is a risk of frequent misclassification. For example, when classifying the type of defect shown in an image, the probability of being misclassified as a specific type may increase. In this case, some users may want to modify the model parameters in order to avoid misclassification. [Prior Art Literature] [Patent Document]

[Patent Document 1] Japanese Patent Laid-Open No. 2019-124591

[Problem to be solved by the invention]

However, when modifying model parameters in a machine learning model mainly based on a deep learning model, it is difficult to only correct the misclassification of a specific category, and it is easy to affect the classification results of other categories. That is, even if the model parameters are corrected so that the classification of a specific type is performed correctly, the classification of other types may become incorrect. Since it is necessary to correct the model parameters in consideration of the classification of all types, it may take a lot of labor or time for system management.

The present invention was made in view of the above points, and one of the problems of the present invention is to provide a defect inspection device, a defect inspection method, and a manufacturing method that facilitate system management in defect inspection. [Technical means to solve the problem]

(1) The present invention was made to solve the above-mentioned problems, and one aspect of the present invention is a defect inspection device for inspecting defects generated in the object to be inspected based on an image of the object to be inspected, Equipped with a plurality of defect discriminators, the defect discriminator uses a specific machine learning model to discriminate the types of the above-mentioned defects based on the above-mentioned images; It is part of a specific number of defect types that are subject to discrimination.

(2) Another aspect of the present invention is a defect inspection method that inspects defects generated in the object to be inspected based on an image of the object to be inspected, and has a plurality of defect discrimination steps based on the above-mentioned Image, using a specific machine learning model to identify different types of the above-mentioned defects; the types of the above-mentioned defects identified in each defect identification step are the specific number of defect types set as the object of identification in the above-mentioned defect inspection method one part.

(3) Another aspect of the present invention may be a glass manufacturing method comprising an inspection step using the defect inspection device of (1) or the defect inspection method of (2), and the object to be inspected is glass. [Effect of Invention]

According to the present invention, system management in defect inspection can be made easier. For example, it can reduce or eliminate the frequency of misjudgment of a specific type without affecting the judgment results of other types of defects.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, the configuration of this embodiment will be described. FIG. 1 is a schematic block diagram showing a configuration example of a defect inspection device according to this embodiment. The defect inspection device 100 of this embodiment is an inspection device for inspecting defects that may occur in the object to be inspected using image data showing an image of the object to be inspected. The defect inspection device 100 executes a plurality of defect identification steps. The defect identification step is to obtain image data showing an image of the object to be inspected, and to use a specific machine learning model to identify defects generated in the object to be inspected based on the obtained image data. Kind of steps. The candidates for one or more types of defects to be identified in each defect identification step are the types of defects of a specific number (M types, M being a specific integer greater than or equal to 2) that are to be identified as a whole of the defect inspection apparatus 100 ( Hereinafter, it is part of the distinguishable type). In addition, the candidates for one or more types of defects to be discriminated differ depending on the defect discriminating step.

The defect inspection device 100 includes a control unit 110 , an imaging unit 130 , an input and output unit 140 , an operation unit 150 , a display unit 160 , and a memory unit 170 . The control part 110 executes the process for realizing the function of the defect inspection apparatus 100, or the process for controlling the function. The control unit 110 includes general-purpose components such as a processor, and can be configured as a computer. The processor reads the program stored in the memory unit 170 in advance, executes the processing instructed by the command described in the read program, and realizes its function. In this case, performing a process instructed by an instruction described in a program is sometimes referred to as executing a program, executing a program, or the like. Part or all of the control unit 110 is not limited to general-purpose hardware such as processors, and may also include dedicated hardware such as LSI (Large Scale Integration: large integrated circuit), ASIC (Application Specific Integrated Circuit: special application integrated circuit) And constitute. The functional parts realizing the functions of the control part 110 will be described later.

The imaging unit 130 captures images showing various objects existing in the field of view around itself, and outputs image data showing the captured images to the control unit 110 . The imaging unit 130 may have a mechanism for changing imaging conditions for one subject, or may have a mechanism for imaging under a plurality of imaging conditions at a time. Shooting conditions are conditions that intentionally affect captured images. Depending on the type of defect occurring in the object to be inspected, the brightness or shape of the image of the defect may vary significantly depending on imaging conditions. The imaging conditions are specified by, for example, any one of brightness, imaging direction, lighting method, etc., or a combination thereof.

FIG. 3 exemplifies a case where an image of a common subject is captured under four imaging conditions at a time. The four types of imaging conditions are classified into two types as combinations of brightness and imaging direction, and are classified into two types as reflection and transmission as lighting methods. In the combination of brightness and imaging direction, there are bright field and dark field. Bright field is an imaging condition in which the subject Sb is irradiated with light from the illumination, and the imaging is set at a position and direction within the field of view in which the incident direction of the reflected light reflected from the subject Sb or the transmitted light transmitted is included. part 130 (camera). In the bright field, since reflected light or transmitted light from the subject Sb directly enters the camera, a bright image is captured. The dark field is an imaging condition in which the subject Sb is irradiated with light from the illumination, and the incident direction of the reflected light from the subject Sb or the transmitted light that passes through is not included in the field of view, but the subject Sb The imaging unit 130 is provided at a position and direction included in the field of view. In the dark field, since the reflected light or transmitted light from the object Sb is not directly incident on the camera, but the scattered light generated on the surface of the object Sb is incident, an image darker than that in the bright field is captured. The imaging unit 130 may include any one of a digital still camera for capturing still images and a digital video camera for capturing moving images. A video is composed of still images that are repeatedly captured at constant time intervals (for example, 1/120 to 1/12 second).

Returning to FIG. 1 , the input/output unit 140 is connected to other devices wirelessly or wired so that various data can be input and output. The input/output unit 140 includes, for example, an input/output interface or a communication interface. The input/output unit 140 is connected to, for example, various control devices, measuring devices, and other devices used in manufacturing steps.

The operation unit 150 accepts the user's operation, and generates an operation signal corresponding to the accepted operation. The operation unit 150 may include, for example, dedicated members such as buttons, knobs, and dials, or may include general-purpose members such as a mouse and a keyboard. The operation unit 150 may be an input interface for receiving operation signals from other devices wirelessly or through wires. Other devices can be portable devices such as remote controllers and multi-functional mobile phones, for example. The operation unit 150 outputs the obtained operation signal to the control unit 110 .

The display unit 160 displays display information such as images, characters, and symbols based on display data input from the control unit 110 . The display unit 160 may include, for example, any one of a liquid crystal display, an organic electroluminescent display, and the like.

The storage unit 170 stores various data used for processing executed by the control unit 110 and various data obtained by the control unit 110 in addition to the above-mentioned programs. The storage unit 170 includes, for example, a nonvolatile (non-transitory) storage medium such as ROM (Read Only Memory), flash memory, and HDD (Hard Disk Drive). In addition, the storage unit 170 includes RAM (Random Access Memory: random access memory), and volatile storage media such as temporary registers.

The control unit 110 includes a defect detection unit 112, a defect determination unit 114, a comprehensive determination unit 116, a model learning unit 118, a manufacturing step management unit 120, a new type determination unit 122, and a determination input unit 124 as a functional unit for realizing its functions. constitute. These functional parts may be constituted by respectively including dedicated components, and the processor may execute a specific program to perform its functions. In the following description, the object to be inspected is mainly a glass substrate for a flat panel display (FPD: Flat Panel Display), but the object to be inspected may be other objects.

The control unit 110 includes a defect detection unit 112, a defect determination unit 114, a comprehensive determination unit 116, a model learning unit 118, a manufacturing step management unit 120, a new type determination unit 122, and a determination input unit 124 as a functional unit for realizing its functions. constitute. These functional parts may be constituted by respectively including dedicated components, and the processor may execute a specific program to perform its functions. In the following description, the object to be inspected is mainly a glass substrate for a flat panel display (FPD: Flat Panel Display), but the object to be inspected may be other objects.

Returning to FIG. 1 , the defect detection unit 112 outputs, to the defect determination unit 114 , an image showing a portion including each one defect region as image data of a defect image. When the defect detection unit 112 does not detect a defect area from the image of the test object, it judges the image of the test object as a good product image, and outputs the image data showing the good product image to the comprehensive judgment unit 116. Moreover, in this case, the defect detection part 112 may output the instruction|instruction information which instructs the display of an input screen to the judgment input part 124. In addition, when the size of the area identified as a defect area does not reach the detection threshold of a specific size for each of the horizontal direction and the vertical direction, the defect detection unit 112 may ignore the determination result of the defect area and determine it as a normal area. . As a detection threshold, a size sufficiently smaller than the size of a typical defect can be set in advance in the defect detection unit 112 .

The defect discriminating unit 114 includes a plurality of defect discriminators. In the following description, the number of defect discriminators is set to N (N is a predetermined integer greater than 2), and sub-numbers such as defect discriminators 114-1 and 114-2 are sometimes marked to distinguish each defect discriminator . Similarly, sometimes the components of each defect discriminator are marked with sub-numbers to distinguish them. Each defect discriminator may not only be constituted by hardware, but also realize its function by executing a specific program.

The N defect discriminators 114-1 to 114-N discriminate the type of the defect shown in the image displayed on the image data input from the defect detection unit 112 as one type for each defect generated in the object to be inspected. Or the handling of any one of multiple defect candidates. Candidates for one or a plurality of types of defects that can be identified may be different among the N defect classifiers 114-1 to 114-N. These candidates form a part of M kinds of discriminable types that can be discriminated as the defect detection unit 112 as a whole. Therefore, M is equal to N, or more than N. When M and N are equal, N defect discriminators 114-1 to 114-N are used to determine whether the types of defects generated in the inspected object are different types of defects, or are the same types of defects. The probability of a level 1 discriminator functioning. When M is greater than N, at least one defect discriminator functions as a multi-level discriminator for determining whether the type of defect generated in the object to be inspected is any of a plurality of types, or as a probability of each type. In the following description, the case where M and N are equal is mainly taken as an example.

Next, the functional configuration of the defect discriminator 114-1 will be described. The defect discriminators 114-2 to 114-N have the same functional configuration as the defect discriminator 114-1 unless otherwise specified, and the description of the defect discriminator 114-1 is referred to. The defect discriminator 114-1 executes preprocessing steps, reasoning steps, defect judgment steps, and good/failure judgment steps (corresponding to steps S112-1, S114-1, S116-1, and S118-1 in FIG. 2 ). The preprocessing step includes processing for matching the format of the image data input to the own device with the format required as input to the inference step (for example, the number of elements of the input value). One or both of dimensionality reduction, resizing, or both are included in the preprocessing step.

Resizing is the process of changing the number of pixels in the horizontal direction and vertical direction of the partial area set as the processing target. Resizing can be zoomed in or zoomed out. The defect discriminator 114-1 interpolates the signal value of each pixel of the input image to determine the signal value of each pixel of the enlarged or reduced image. The defect discriminator 114-1 can use, for example, well-known interpolation methods such as bilinear interpolation and bicubic interpolation for interpolation. The defect discriminator 114-1 does not simply change the size of the image at a specific magnification, but can determine that for each defect region, including the entire defect region, the maximum value of its horizontal or vertical diameter becomes a partial region The constant r (for example, a real number greater than 0.5 and less than 1) times the magnitude of the horizontal or vertical direction. The diameter is equivalent to the length of a line segment in all directions transverse to the area of the defect. For example, when the shape of the detected defect is an ellipse, the maximum value as the diameter of the defect is represented by the length of the major axis. When the shape of the detected defect is a rectangle, the maximum value of the diameter of the defect is represented by the length of the long side.

Dimensionality reduction is a process of assembling a plurality of images of a partial area captured under different imaging conditions into a smaller number of images. Dimensionality reduction is achieved by image synthesis. Image synthesis includes a process of calculating a weighted sum of luminance values between a plurality of images for each pixel as a new signal value. The weighted sum is equivalent to the product of the luminance value of each image and the weight coefficient corresponding to the image, that is, the sum of the images of the multiplication value. Although the minimum number of images generated by image synthesis is one, it may be plural (for example, three). In image synthesis, for example, the method of alpha blending can be used. Alpha blending is a method of normalizing the sum of weight coefficients (alpha values) for each image to be combined to 1. Between the defect discriminators 114-1 to 114-N, the ratio of the weight coefficients between images may be different. Thereby, according to the type of defect of each defect discriminator, a larger value can be set for the weight coefficient of the image under the imaging conditions under which the defect can be easily detected than the weight coefficient of other images. For example, for damage, it is set in advance so that the weighting coefficient for an image captured under a bright field is relatively small, and the weighting coefficient for an image captured under a dark field is set relatively large. Lesions were more reliably detected from images taken under dark field.

When the number of images generated by dimensionality reduction is set to three, the defect discriminator 114-1 can use the signal value of each generated image as a color signal value of a different hue in For each pixel, image data displaying one color image obtained by integrating images of each of the three color tones is generated. As a colorimetric system for expressing a color image, an RGB colorimetric system, a YCrCb colorimetric system, or the like can be used. An example is shown in which the images Im01 to Im04 of four partial regions with different imaging conditions are synthesized into one color image Im05 by dimensionality reduction. First, the defect discriminator 114-1 acquires the images Im01 and Im02 captured under the bright field and the images Im03 and Im04 captured under the dark field. At this time, the images Im01 and Im03 are images obtained by capturing transmitted light, and the images Im02 and Im04 are images obtained by capturing reflected light. The shape or brightness of the defect varies depending on the imaging conditions. The image Im05 is represented by a synthetic value obtained by synthesizing signal values of four images in the defect discriminator 114-1 by using different weight coefficients for red, green, and blue. For example, the signal value of the red channel of the image Im05 is synthesized by combining the luminance value of the image Im01 and the luminance value of the image Im02 with a ratio of 0.7:0.3 as the respective weight coefficients. The signal value of the green channel of the image Im05 is synthesized by combining the luminance value of the image Im03 and the luminance value of the image Im04 with a ratio of 0.4:0.6 as each weight coefficient. The signal value of the blue channel of the image Im05 is synthesized by combining the luminance value of the image Im02 and the luminance value of the image Im03 with a ratio of 0.5:0.5 as each weight coefficient.

The inference step is a step of determining the probability that the type of defect occurring in the object to be inspected is a specific type with respect to the image data showing the image to be processed. The defect discriminator 114-1 inputs the signal value of each pixel forming the image data as an input value, uses a specific machine learning model for the input value, and calculates its probability as an output value. The output value is a real value between 0 and 1. In the defect discriminator 114-1, a parameter group (model parameter) for calculating an output value from an input value is set in advance. As the machine learning model, for example, a neural network such as a convolutional neural network (CNN: Convolutional Neural Network) or a recurrent neural network (RNN: Recurrent Neural Network) can be used. The machine learning model that can be used is not limited to the neural network, and random forest (RF: Random Forest), support vector machine (SVM: Support Vector Machine) and other methods can also be used.

The defect judging step is a step of judging whether or not it corresponds to the defect of the type to be judged based on the probability calculated in the reasoning step. For example, the defect classifier 114 - 1 judges that the category is met when the calculated probability is greater than the specific defect judgment threshold, and judges that the class does not match when the calculated probability is below the defect judgment threshold. The defect judgment threshold can be set independently for each type of defect to be judged. For example, in each defect discriminator, a value as small as a defect judgment threshold corresponding to a type of defect with a high occurrence risk is set in advance, and a value corresponding to a defect judgment threshold corresponding to a type of defect with a low occurrence risk is set in advance Such a large value. A higher risk of occurrence means that the possibility of material or economic losses due to occurrence is higher, or that such losses are relatively large. That is, there is a tendency that the higher the occurrence of a type of defect with a higher risk, that is, a high-risk defect, the less it is allowed to be rejected due to missed detection, and the lower the occurrence of a type of defect with a lower risk, that is, a low-risk defect, the more acceptable overdetection. For example, the risk for damage, air bubbles, foreign objects, dirt each decreases in that order, while the risk for damage is higher. Therefore, by setting the defect judgment threshold smaller for higher-risk defects, the reproducibility (recall) of detecting defects relatively seamlessly can be improved. By setting the defect judgment threshold to be larger as the type of defect with a lower risk occurs, that is, a low-risk defect, the precision that can be relatively reliably detected as a defect can be improved. The defect discriminator 114 - 1 generates a defect mark indicating a conforming or non-conforming defect for the target area set as the determination target, and stores the defect mark generated in association with the target area in the memory unit 170 . In addition, in this case, like the above-mentioned "foreign matter" and "dirt", the name of the object may be used to refer to the state where a specific object is attached or mixed as a type of defect.

The good/failure judgment step is a step of judging the good/failure of the defective area based on a specific judgment reference value for the detected defect when it is judged to conform to a specific defect type in the defect judgment step. When it is judged to be non-compliant in the defect judgment step, the good/failure judgment step is not performed. For example, when the size (dimension) of the detected defect is larger than the judgment reference value shown in the judgment data preset in its own device, the defect discriminator 114-1 judges the area as a defective product. When the size of the area is below the judgment reference value, the area is judged as a good product. Each defect is generally represented by a graph on a two-dimensional plane, but the defect discriminator 114-1 determines the maximum value of the diameter of the detected defect as the size of the defect. The judgment standard value can be set independently for each type of defect of the judgment object. For example, the judgment standard value for damage is 100-300 μm, and the judgment standard value for foreign matter is 50-120 μm. The defect discriminator 114 - 1 generates a defective product mark indicating a good product or a defective product for the defect and its type to be judged, associates it with the target area, and stores the defective product mark in the memory unit 170 .

The comprehensive judgment unit 116 refers to the good or bad state of each detected defect, and judges the overall state of the object to be inspected. For example, the comprehensive determination unit 116 counts the number of defects determined as defective products by referring to all the good/failure judgments indicated by the good/failure marks related to each defect of the object to be inspected, regardless of the type of the defect. The comprehensive judgment unit 116 judges the object to be inspected as a defective product when the number of defects of all types is greater than the number of specific judgment standards, and when the number of defects of all types is less than the number of specific judgment standards In this case, the inspected object is judged to be a good product. Since the defect determination part 114 parallelizes the quality determination steps between the types of defects, one defect is associated with a plurality of types of defects and a defective product flag is displayed. In the comprehensive determination unit 116, the defects associated with the plurality of defective product marks can be processed as conforming to the plurality of types. For example, when three defects are detected from the object to be inspected, the comprehensive determination unit 116 determines that a certain defect 001 corresponds to both damage and foreign matter, determines that defect 002 corresponds to dirt, and determines that defect 003 corresponds to foreign matter and dirt. The defect identification unit 114 may specify three as the number of defects that actually occur, or five as the total number of each type of defects.

In addition, in the comprehensive judging unit 116, for example, judging data representing the judging reference number of defects for each type of defect can be set in advance, and the state of the object to be inspected can be judged using the judging data (rule base). The comprehensive judgment unit 116 refers to the good/failure judgment indicated by the good/failure mark for each defect for each type of defect, and counts the number of defects. If there is a type of defect whose counted number of defects is greater than the number of judgment criteria shown in the judgment data, the comprehensive judgment unit 116 judges the object to be inspected as a defective product, and if there is no such type of defect, it will be The inspection body was judged to be a good product. The number of judgment criteria for defect types with higher risk of occurrence can be set less, and the number of judgment criteria for defect types with lower risk of occurrence can be set larger. For example, if the types of defects are damage and dust, the number of determination reference objects is set to 1 and 5, respectively. FIG. 5 exemplifies a case where one damage and two dusts are detected from the object to be inspected. If it is assumed that only a specific type of defect, that is, dust, is paid attention to, and other types of defects are not concerned, since the number of detected dust is 2, which is less than the number of judging criteria, the inspected object can be judged as a good product. However, since the number of damages is one, which is equal to the number of judgment references, the comprehensive judgment unit 116 judges the object to be inspected as a defective product. Therefore, by comparing the type of each defect with the judgment reference number, it is possible to avoid the risk of judging an inspected object having an unacceptable defect as a good product.

The weight coefficient of each type of defect can be preset in the comprehensive judgment unit 116, and the sum of the multiplication value of the weight coefficient of each type of defect and the number of defects, that is, the weighted sum among the defect types, can be calculated as the effective number. The comprehensive judgment unit 116 can judge the object to be inspected as a defective product when the effective number is more than a specific judgment reference number, and judge the object to be inspected as a good product when the effective number is less than the judgment reference number. When image data representing a good product image is input from the defect detection unit 112 to the comprehensive determination unit 116, the object to be inspected can be determined to be a good product. The comprehensive judging unit 116 may output the instruction information indicating the display input screen to the judging input when all the defects and types indicated by the good/failure mark are judged as good or bad for each defect of the inspected object regardless of the type of the defect. Section 124.

The inspected objects judged to be good products are shipped, and the inspected objects judged to be defective products are discarded or returned to the manufacturing process. The comprehensive judgment unit 116 outputs judgment result information indicating the judgment result to the manufacturing step management unit 120 . The manufacturing process management part 120 refers to the judgment result information input from the comprehensive judgment part 116, and discards or returns the inspected object judged to be a defective product to the manufacturing step. The manufacturing step management unit 120 outputs, for example, a control signal indicating return to the manufacturing step or discard to the manufacturing facility. In addition, the comprehensive judgment unit 116 may include information on the number of defects of each type of defects or the number of all defects in the judgment result information.

The model learning unit 118 calculates, as model parameters, a group of parameters of the machine learning model for discriminating the type of a defect by the defect discriminating unit 114 . The model learning unit 118 uses a specific machine-learning method for each type of defect to be discriminated against for training data (training data, also referred to as learning data, supervised data, etc.). The model performs a learning process to calculate model parameters. In the model learning unit 118, prior to the learning process, the training data including a set of known input values (typically, 1,000 to 10,000 or more) and an output value corresponding to the input values, that is, a data set, is set in advance. . As an input value, image data including a signal value of each pixel is used. The output value of each group included in the training data is assigned 1 when the image shown in the image data used as the input value shows an image of a defect of the type to be discriminated, and assigned 0 in other cases. . In the learning process, the model learning unit 118 can use, for example, training data configured by adding (annotation) the type of defect and its output value to each image data set as an input value.

In the learning process, the model learning unit 118 updates the model parameters until the magnitude of the difference between the calculated value calculated using a specific machine learning model and the output value becomes close to zero with respect to the input value as a whole complex array. When the change amount of the model parameters before and after the update, or the change amount of the difference between before and after the update does not reach a specific convergence judgment threshold, it can be determined that the model parameters have converged. For updating model parameters, methods such as steepest descent, stochastic gradient descent, conjugate gradient method, and back propagation can be used, for example. As the index value of the magnitude of the difference, for example, error functions such as sum of squared differences (SSD: Sum of Squared Differences) and cross entropy error (cross entropy error) can be used. The model learning unit 118 sets the model parameters calculated for each type of defect in the defect discriminator related to the discrimination of the type of defect. In addition, the model learning part 118 can set the output value related to the defect type to 1 for the type of defect input from the judgment input part 124, and add the image data set as the judgment object as a data set of the input value to the Training materials related to the type of defect. In addition, the model learning unit 118 may set the output values related to other types to 0, and add the image data set as the judgment target as a data group of input values to the training data related to the defect type. In addition, the model learning unit 118 can update the model parameters related to each defect type using the newly added training data (transfer learning).

The manufacturing process management part 120 controls the manufacturing process of the product based on the state of the defect which arises in the product which is the inspection object judged by the defect judging part 114 . In the manufacturing process management part 120, for example, the state of a defect and the information of a correction condition are contained in advance, and the control data which correlates these and displays are set in advance. The correction condition is a condition for correcting the production condition used in the production step at that point in time to give the corrected production condition. The manufacturing process management unit 120 can use, as an example of information on the status of defects, the number of various types of defects generated in the object to be inspected. The number of each type of defect is transmitted by the judgment result information input from the comprehensive judgment unit 116 . The manufacturing conditions may include operation parameters for executing the manufacturing steps of the product, and environmental parameters indicating its environment. The operating parameters may include the rotation speed and power consumption that form the power of the manufacturing equipment. The environmental parameters may include temperature, pressure, etc. Correction conditions can be expressed as the amount of change in any type of parameter given to the modified manufacturing conditions. The manufacturing process management unit 120 uses the control data to specify information on correction conditions corresponding to the state of the reference defect. The manufacturing step management section 120 generates control information indicating changes in manufacturing conditions under the determined modified conditions, and outputs the generated control information to the manufacturing equipment. The manufacturing equipment corrects the manufacturing conditions using the corrected conditions shown in the manufacturing information input from the manufacturing step management unit 120, and executes the manufacturing steps under the corrected manufacturing conditions.

The new type determination unit 122 determines whether the type of defect detected from the object is a new type different from any of the known types of defects. For example, when the new type determination unit 122 determines that the type of defect detected by the defect detection unit 112 is not any of the types of defects determined by the defect determination unit 114 , it determines that the type of the defect is a new type. For example, the new type determination unit 122 can determine that the type of the defect is a new type by displaying a defect that does not match any of the defect marks for the type of defect corresponding to each of the defect classifiers 114 - 1 to 114 -N. When the new type determination unit 122 determines that the type of the defect is a new type, it may display a notification screen indicating that the type of the defect is a new type on the display unit 160 . Also, in this case, the new type determination unit 122 may output instruction information for displaying an input screen displayed on the determination input unit 124 . Thereby, the operator as the user can be noticed that the type of defect is a new type, and the input of the type of defect or good or bad judgment can be facilitated.

The judgment input unit 124 inputs an operation signal from the operation unit 150 , and the operation signal indicates the type of defect occurring in the object to be inspected represented by the image shown in the image data, or the quality of the object to be inspected. The judgment input unit 124 can generate an input screen, for example, and display the generated input screen on the display unit 160, and the above input screen includes: the image of the subject shown in the image data; Press to indicate the type of defect, the quality of the object to be inspected, or both. The so-called pressing means that besides actually pressing, an operation signal indicating a position included in the display area is input from the operation unit 150 or other devices according to the operation. As screen components, for example, buttons, check boxes, menu bars, etc. can be used. In addition, the input screen may be displayed on the display unit 160 when instruction information for displaying the display of the input screen is input from the defect detection unit 112 , the comprehensive determination unit 116 , or the new type determination unit 122 . Thereby, the judgment input of category or good/failure is facilitated for the user. The judgment input unit 124 outputs the type of the input defect and the information of the quality judgment of the inspected object to the comprehensive judgment unit 116 . In addition, the judgment input unit 124 may output the type of the input defect to the model learning unit 118 .

(check processing) Next, an example of inspection processing in this embodiment will be described. FIG. 2 is a flow chart showing an example of inspection processing in this embodiment. (Step S102 ) The imaging unit 130 captures an image of the glass substrate for FPD as an example of the object to be inspected. Thereafter, the process proceeds to step S104. (Step S104 ) The defect detection unit 112 detects a defective region, that is, a defect region, from the image of the object captured by the imaging unit 130 . Thereafter, the process proceeds to step S110.

The processing of step S110 includes the processing of steps S110-1 to S110-N. The processes of steps S110-1 to S110-N are executed in parallel in each of the common partial regions including the defect regions detected from the captured images by the defect discriminators 114-1 to 114-N. Since the processing of steps S110-2 to S110-N is the same as the processing of step S110-1, the description thereof is referred to. (Step S112-1) The defect discriminator 114-1 performs a pre-processing step on the image in a partial area. The defect discriminator 114-1 includes dimension reduction for images captured under different imaging conditions, and size adjustment of the number of elements of the dimensionally reduced image to the value input to the image reasoning step . In one example, when the number of elements of an image to be processed is the number of pixels in the horizontal direction, the number of pixels in the vertical direction, and the number of dimensions (number of frames) are 200, 200, and 4, respectively, the pre-processed image The number of elements in the horizontal direction, the number of elements in the vertical direction, and the number of dimensions can be 224, 224, and 3, respectively. The defect discriminator 114-1 can be reconstructed into a two-dimensional color image whose number of pixels in the horizontal direction and the number of pixels in the vertical direction are 224 and 224, respectively. Thereafter, it proceeds to the processing of step S114-1.

(Step S114-1) The defect discriminator 114-1 performs inference processing on the signal value of each pixel of the image processed before displaying. The defect discriminator 114-1 takes the signal value of each pixel as an input value, uses a specific machine learning model, and calculates the probability that the type of defect represented by a part of the region matches the specific type of defect in its own device, as an output value. Thereafter, it proceeds to the processing of step S116-1. (Step S116-1) The defect judging unit 114-1 executes a defect judging step. The defect discriminator 114-1 determines whether the type of the defect matches the type of the specific defect in its own device by checking whether the calculated probability is greater than the specific defect determination threshold set in its own device. Thereafter, it proceeds to the processing of step S118-1. (Step S118-1) The defect discriminator 114-1 executes a good/failure judging step. The defect classifier 114-1 judges good or bad based on, for example, whether the size of the detected defect is larger than a judgment reference value set in its own device. Thereafter, the process proceeds to step S122.

(Step S122 ) The comprehensive determination unit 116 refers to the good or bad status of each detected defect type, and determines whether it is a good product or a defective product as the overall state of the object to be inspected. The comprehensive judgment unit 116 judges whether it is a good product or a defective product by, for example, determining whether the number of defects that are defective products is larger than a specific judgment reference number. Thereafter, the manufacturing process management unit 120 adopts the inspected objects judged to be good products as products to be shipped, and discards or returns the inspected objects judged to be defective products to the manufacturing process. Thereafter, the processing of FIG. 2 ends.

As mentioned above, the defect inspection apparatus 100 of this embodiment is equipped with the defect discriminator 114-1-114-N for each type of defect, and judges whether the detected defect corresponds to the type. Therefore, the correction of the model parameters used for the determination of one type will not affect the determination of other types of defects. In addition, imaging conditions, the imaging unit 130, or pre-processing suitable for the type of each defect can be used. And, N defect discriminators 114-1 to 114-N are arranged in parallel. That is, each defect discriminator determines whether the type of the detected defect corresponds to the specific defect type regardless of the respective determination results of other defect discriminators. Therefore, even if the types of defects to be judged increase, the processing time does not change, and each defect classifier selects information corresponding to a defect of that type, such as the The characteristic quantity inherent in the category. However, the more types of defects to be judged, the more computing resources are required.

In the defect inspection device 100, the defect discriminators 114-1˜114-N may also be arranged in series. More specifically, when the defect discriminator 114-n (n is an integer greater than 1 and less than N-1) determines that the type of defect detected in the image in the partial area does not match the type of specific defect in its own device, The defect discriminator 114-n+1 starts the process of judging whether the type of the detected defect matches the type of a specific defect in its own device. When the defect discriminator 114-n determines that the type of defect detected from the image in the partial area matches the specific type in its own device, it determines the type of the detected defect as the type determined to be consistent, and does not perform defect discrimination. The processing after the device 114-n+1. In the example shown in FIG. 6, in step S110-1, the defect discriminator 114-1 determines whether the type of the detected defect is damage, and ends the process of FIG. 6 when it is determined that it matches damage. When it is determined that it is non-compliant damage, the defect discriminator 114-1 proceeds to the processing of step S110-2. In step S110-2, the defect discriminator 114-2 determines whether the type of the detected defect is a bubble, and when it is determined that it matches a bubble, the process of FIG. 6 ends. When it is determined that the bubble does not match, the defect discriminator 114-2 enters the subsequent processing.

Therefore, by connecting the defect discriminators 114-1 to 114-N in series, the number of defect discriminators performing the defect discrimination step is limited to one, so limited computing resources can be effectively utilized. Also, since the learning of the model parameters used in the subsequent defect discriminator can be performed independently of the learning of the model parameters used in the preceding defect discriminator, it can be completed without the problem of increasing the time required for learning. However, the timing at which the subsequent defect discriminator starts to execute the defect discriminating step is after the defect discriminating step of the preceding defect discriminator is completed. Therefore, as illustrated in FIG. 7 , as the defect inspection apparatus 100 as a whole has more types of defects to be discriminated, the processing time tends to be slower. In FIG. 7 , the vertical axis and the horizontal axis respectively show the processing time and the number of models. The number of models corresponds to the type of defect to be detected, that is, the number of defect discriminators. The processing time illustrated in FIG. 7 includes a component proportional to the number of models and a constant component independent of the number of models (approximately 0.043 seconds, see dashed line). The former is equivalent to the time required for judging compliance with each defect type. The latter is equivalent to the time spent on photography, pre-processing, etc. regardless of the type of defect. However, when the number of defect discriminators is 10 or less, the processing time is almost the same as the case of using the conventionally used multi-stage model. With regard to accuracy, when comparing the case where the machine learning model adopted by the defect discriminator is a two-level model, and the case of a multi-level model, the equivalent or two-level model has higher accuracy. The level 2 model is a machine learning model used to determine whether the type of defect conforms to a specific type 1. The multi-level model is a machine learning model used to determine whether the type of defect is any one of a plurality of specific types. In the example shown in FIG. 8 , the false answer rates were 4.0% and 2.0% for each of the multi-level model and the two-level model, and the two-level model had higher accuracy.

In addition, for the defect classifiers 114-1 to 114-N connected in series, the first to N-th types of defects that are the object of discrimination can be determined in advance in descending order of their frequency of occurrence. Since the more frequently occurring types of defects are detected earlier, the processing time does not become excessive to complete. In addition, the defect classifiers 114-1 to 114-N may predetermine the first to Nth types of defects that are respectively subject to discrimination in descending order of their risk levels. Since the defects of the higher risk type are detected earlier, the damage caused by the detection delay of the defects can be reduced.

Next, another configuration example of the new type determination unit 122 will be described using FIG. 9 . The new type determination unit 122 executes a new type determination step (step S122-n). The new category determination step includes a preprocessing step (step S122-a), an inference step (step S122-b), a new category degree calculation step (step S122-c), and a new category degree determination step (step S122-d). However, in FIG. 9, although the illustration of step S110-3 is omitted, it does not mean that the processing of step S110-3 is omitted.

In step S122-a, the new type determination unit 122 performs the same previous processing steps as step S112-1 or step S112-2. In step S122-b, the new type determination unit 122 performs inference using an N-level machine learning model on a partial image input to its own device, and extracts the feature value calculated in the inference. As an N-level machine learning model, for example, an N-level neural network (for example, CNN) that calculates the probability of each of the N types of defects as an output value can be used. In the new type determination part 122, a model parameter is set in advance, and the model parameter is for each group, for the type of the defect shown in the image shown in the image data set as the input value, use 1 as the dimension corresponding to the type. output value, and append 0 as the training data for the output value of the dimension corresponding to other types to learn. In addition, the new type determination unit 122 may obtain the calculated value output from a specific intermediate layer as a feature quantity showing the characteristics of the image in addition to the input layer and output layer of the neural network constituting its own device. In addition, using the set model parameters, calculate the feature quantity of each image of the known defect type by the method described later to display the characteristics of the image (defect), and for each defect type, the representative value of the feature quantity ( For example, center of gravity) is set in the new type determination unit 122 in advance.

In step S122-c, the new type determination unit 122 calculates the distance between the representative value of the feature value determined for each type of defect and the obtained feature value, and determines the minimum value among the distances calculated for each type as the new value. Kind of degree. FIG. 10 shows two-dimensional vectors showing feature quantities corresponding to detected defects (detection defects) with star marks in the feature quantity space extended by feature quantities X and Y as requirements. And, as the type of defect, the distance between the representative value of air bubbles and the feature amount among the distances between the representative values (X-shaped marks) of each of dirt, damage, air bubbles, and dust and the calculated feature amount is calculated as a new type degree.

Returning to FIG. 9, in step S122-d, the new type determination unit 122 determines that the type of the defect is a new type when the calculated new type degree is greater than the threshold value of the specific distance, and is below the threshold value of the specific distance. , it is determined that the type of defect is not a new type. The new category determination unit 122 determines a value sufficiently larger than the variance of the feature value of any category as the threshold value of the distance. As an index value showing the distance, for example, Mahalanobis' distance, Minkowski's distance, Manhattan distance, cosine similarity, etc. can be used. In addition, when the new type determination unit 122 determines that the type of the defect is a new type, it may output new type detection information indicating that the type of the defect set as the determination target is a new type to the comprehensive determination unit 116 . Furthermore, when the new type detection information is input from the new type determination portion 122, the comprehensive judgment unit 116 may discard the good/failure flag for the defect indicated by the new type detection information. In this way, it is avoided to mistakenly adopt the stored judgment result.

The new type determination unit 122 may specify the feature amount of an image showing a defect determined as a new type as a representative value of the new type. The new type determination unit 122 can determine that the type of the defect shown in the image data is a new type when the distance between the feature quantity calculated from other image data and the representative value of the new type is within a certain distance. Furthermore, the model learning unit 118 may perform learning processing using training data to determine model parameters for discriminating new types of defects from image data, and the training data includes image data assigned to images showing new types of defects as As an input value, assign 1 as a plurality of sets of output values, and assign image data showing images of other types of defects as input values, and assign 0 as a plurality of sets of output values. The defect identifying unit 114 may constitute a new defect determiner for identifying a new type of defect using the determined model parameters.

Next, an example of the control of the manufacturing steps by the manufacturing step management unit 120 will be described using FIGS. 11 and 12 . The processing illustrated in FIG. 11 and FIG. 12 includes the processing of steps S102-S126. Regarding the processing of steps S102-S122, the above description is used. However, in step S122 , the comprehensive judgment unit 116 outputs to the manufacturing process management unit 120 judgment result information showing the number of defects for each type as an example of the state of defects occurring in the product to be inspected. The processing of steps S102-S122 is included in a series of inspection steps, and the state of defects of each product of the object to be inspected, or a time series thereof is obtained as a quality trend. In step S124, the manufacturing process management unit 120 refers to the control data preset in its own device, and determines the correction condition (described later) corresponding to the state of the defect indicated by the judgment result information. In step S126, the manufacturing step management section 120 generates control information indicating a change in the manufacturing conditions under the determined modified conditions, and outputs the generated control information to the manufacturing equipment (feedback). The manufacturing facility corrects the manufacturing conditions using the correction conditions shown in the manufacturing information input from the manufacturing step management unit 120 . A time series of the corrected manufacturing conditions for each product, or the like, is obtained as a control trend. In the manufacturing step, the manufacturing equipment executes the manufacturing step using the corrected manufacturing conditions (step S200 ). This embodiment can be realized by a manufacturing method including the inspection step of steps S102-S122.

In addition, the processing of steps S102-S126 may be incorporated in the middle of the manufacturing step S200 such as between the cutting step S203 and the grinding step S204, rather than after the manufacturing step S200. In this case, in step S126, the manufacturing step management part 120 may output the generated control information to the upstream manufacturing equipment (feedback) that executes the steps earlier than steps S102-S126, or instead of this, the manufacturing step management part 120 can further output the generated control information to the downstream manufacturing equipment that executes the steps subsequent to steps S102-S126 (feed-forward). Thereby, the manufacturing conditions in the steps downstream of the steps S102-S126 are further streamlined. For example, when the steps S102-S126 are combined before the grinding step S204, the grinding amount is adjusted in the grinding step S204 based on the control information generated in the step S126.

Next, an application example to a manufacturing process of a glass substrate will be described. The overall manufacturing process is described in more detail, for example, in International Publication No. 2012/090766 and Patent No. 5983406, and the grinding step is described in more detail, for example, in Patent No. 4862404 and Patent No. 4207153. A glass substrate is manufactured using manufacturing methods, such as a float method and a fusion method. The manufacturing step S200 of the glass substrate illustrated in FIG. 13 includes, for example, a melting step S201, a forming step S202, an incision and cutting step S203, a grinding step S204, a product drop step S205, and a crushing step S208. Melting mechanism, forming mechanism, cutting and cutting mechanism, grinding mechanism, product falling step S205 and crushing step S208 for performing melting step S201, forming step S202, cutting and cutting step S203, grinding step S204, product falling step S205 and crushing step S208 The lower mechanism and the crushing mechanism constitute the manufacturing equipment (not shown). The defect inspection device 100 of this embodiment may also be included in manufacturing equipment.

As the melting means, for example, a melting furnace is used. In the melting furnace, in the melting step S201, the glass raw material is heated to melt to form a molten glass. As the forming mechanism, for example, a forming device is used. The forming device has a molten tin tank, and in the forming step S202, the molten glass transferred from the melting furnace is spread on the tin in the molten tin tank, and the ribbon-shaped glass having a specific width is formed into a glass ribbon. The glass ribbon is placed on the main conveying path of the conveying rollers, and can be conveyed to a packaging mechanism (not shown) as a glass substrate to be a product. During the period until reaching the packaging mechanism, the cutting and cutting step S203, the grinding step S204, and the product dropping step S205 are performed on the glass substrate. As the cutting and cutting mechanism for performing the cutting and cutting step S203, for example, a cutting and breaking device is used. The incision and breaking device is formed by using the glass ribbon flowing in the main conveyance path as a glass substrate of a specific size. It is equipped with a puncture line processing device upstream in the conveying direction, and a breaking device is provided downstream of the incision line processing device. In the cutting and cutting step S203, the cutting line processing device is provided with a cutter, and the cutting line is formed on the glass ribbon by pressing the front end of the cutter against the surface of the glass ribbon with a specific pressure. The breaking device divides the glass ribbon along the cutting line and divides it into glass substrates of a specific size.

As a polishing mechanism for performing the polishing step S204, a polishing device that polishes the surface of the divided glass substrate is used. The polishing apparatus includes, for example, a plurality of circular grinding tools that rotate and revolve, and continuously polishes the glass substrate while moving in the conveying direction of the glass substrate (continuous type). The grinding device is based on the moving center line of the glass substrate, forms a pair of circular grinding tools with a diameter smaller than the width of the glass substrate, and arranges 2 pairs in a zigzag shape along the moving direction, so that the circular grinding tools exceed the moving center line. Grinding the surface of the glass substrate.

The polishing device may have a configuration (discontinuous type) for polishing the surface of the glass substrate in a stationary state. The polishing apparatus includes, for example, a substrate attaching table, a film frame mounting table, a polishing table, a film frame removing table, and a substrate removing table. The substrate attaching station attaches the glass substrate to the film frame. The film frame installation table installs the film frame on the lower part of the carrier. After the film frame is installed on the carrier, the grinding table makes the carrier and the grinding table relatively close, and presses the grinding surface of the substrate attached to the film body to the grinding table for grinding. The film frame removal station removes the film frame from the carrier. The substrate removal table removes the ground glass substrate from the film frame.

As the product dropping mechanism for performing the product dropping step S205, a dropping device is used. The dropping device has a control unit and a rotary mechanism, and controls whether to drop the manufactured glass substrate from the main conveying path in the product dropping step S205. When the control unit receives the control signal indicating return to the manufacturing step from the defect inspection apparatus 100, it rotates the connection member forming the main conveyance path in the rotary mechanism. Thereby, the glass substrate judged to be a defective product falls, passes through a packaging mechanism, and is returned to a manufacturing process without shipment. The dropped glass substrate is broken by hitting an obstacle and becomes a broken object. On the other hand, when the control unit does not receive the control signal, the main conveyance path is maintained. Therefore, the glass substrate judged to be a good product is conveyed to the packaging mechanism and becomes a shipping object. As a pulverizing mechanism for performing the pulverizing step S208, a pulverizer is used. The pulverizer is provided with a rotary blade for pulverizing the broken object, and forms glass shavings which are glass raw materials in the pulverization step S208. The glass shavings are transported by the belt conveyor installed in the manufacturing equipment and supplied to the melting furnace.

In the inspection step of the present embodiment, typically, the polished glass substrate produced in the polishing step S204 is used as an object to be inspected. Depending on the type of defect to be inspected, the molten glass produced in the melting step S201, the glass ribbon produced in the forming step S202, or the glass substrate before grinding produced in the cutting and cutting step S203 can be used.

The defect inspection device 100 of this embodiment can discriminate the types of defects that may occur in each step, and realize control corresponding to the discriminated defects. For example, bubbles generated in the glass substrate tend to be more likely to be generated as the temperature of the molten glass in the melting step S201 is lower than a specific reference temperature. Therefore, in the control data for controlling the manufacturing process, it is preliminarily set in association that the larger the amount of air bubbles as the detected defect state, the larger the temperature rise of the melting furnace as the correction condition. Thereby, the manufacturing process management part 120 can raise the temperature of a melting furnace as the detected bubble increases.

The foreign matter adhering to the surface of the glass substrate tends to remain as the polishing time in the polishing step S204 is shorter than the reference time. Therefore, in the control data, it is preliminarily set in association so that the larger the amount of foreign matter as the detected defect state, the larger the increase in the polishing time as the correction condition. In this way, the manufacturing step management unit 120 can extend the grinding time as the detected foreign objects increase. Damage to the surface of the glass substrate tends to occur due to deterioration of the abrasive used in the polishing step S204. Therefore, in the control data, the reference amount of the damage amount as the detected defect state and the replacement of the grinding tool as the correction condition are set in advance in association with each other. Thereby, the manufacturing process management unit 120 can replace the grinding tool when the detected damage amount exceeds the reference value. Damage also tends to occur due to insufficient pressing force on the surface of the glass substrate of the abrasive tool used in the polishing step S204. Therefore, in the control data, it can be set in advance in association with the amount of damage that is the state of the detected defect, the larger the amount of increase in the pressing force of the polishing tool as the correction condition becomes. Thereby, the manufacturing process management part 120 can increase the pressing force on the glass substrate surface of the grinding tool as the detected damage increases.

In addition, the manufacturing process management unit 120 can receive an input from the operation unit 150 of an operation signal indicating a correction condition relative to the manufacturing condition, and include information associating the change condition with the state of the defect shown in the judgment result information. Control data and save. Thereby, the control data is updated with the information which correlates the manufacturing conditions instructed to the manufacturing equipment and the states of the detected defects by the operator operating as the user. Thus, updated control data can be used to control manufacturing conditions based on the status of detected defects.

(machine learning model) Next, an example of the machine learning model of this embodiment will be described. Fig. 14 shows CNN as an example of the machine learning model of this embodiment. In the example shown in FIG. 14, it is a two-level model, that is, the input value to the CNN is two-dimensional image data, and the one-dimensional probability (scalar) is calculated as the output value from the CNN. CNN is a type of artificial neural network, which has an input layer, a plurality of intermediate layers, and an output layer. The CNN illustrated in FIG. 14 has an input layer In02, 6 intermediate layers, and an output layer Out16. The 6 intermediate layers have 3 convolutional layers Cv04, Cv08, and Cv12, 2 pooling layers P106, P110, and a fully coupled layer Fc14. However, after alternately repeating the 1-layer convolutional layer and 1-layer pooling layer twice, a subsequent convolutional layer Cv12, and then a subsequent fully-coupled layer Fc14. Each layer has one or more nodes (also called nodes, neurons, etc.). Each node outputs a function value of a specific function corresponding to an input value as an output value.

The input layer In02 outputs the signal value of each sample point indicated by the measurement signal input as the input value to the next layer respectively. Each sample point corresponds to one pixel. At each node of the input layer In02, the signal value of the sample point corresponding to the node is input, and the input signal value is output to the corresponding node of the next layer. Preset the number of cores in the convolutional layer. The number of cores corresponds to the number of cores used for processing (for example, operations) on various input values. The number of cores is usually less than the number of input values. A core means a processing unit for calculating one output value at a time. The output value calculated in a certain layer is used as the input value of the next layer. Cores are also known as filters. The kernel size shows the number of input values used for one processing in the kernel. The core size is usually an integer greater than 2.

The pooling layer and the convolutional layer calculate the feature quantities showing their characteristics based on multiple input values. As feature quantities, the output values from any specific layer in the convolutional layers Cv04, Cv08, Cv12 and pooling layers P106, P110 can be used to determine new types of defects. The convolutional layer is a layer as follows, that is, at each of the plurality of nodes, the convolution operation is performed on each core for the input value input from the previous layer to calculate the convolution value, and the calculated convolution value is calculated and calculated. The function value of the specific activation function relative to the correction value obtained by adding the bias value is used as an output value, and the calculated output value is output to the next layer. In addition, in the convolution operation, one or more input values are input to each node from the upper layer, and an independent convolution coefficient is used for each input value. The parameters of convolution coefficient, bias value and activation function are part of a set of model parameters.

As the activation function, for example, a rectified linear unit (Rectified Linear Unit), a double sigmoid function, or the like can be used. A normalized linear cell is defined as a function whose output value relative to an input value below a certain threshold (eg, 0) is determined as the threshold, and input values exceeding a certain threshold are output directly. Thus, the threshold may be part of a set of model parameters. Also, regarding the convolutional layer, whether to refer to the input value from the node of the previous layer, and whether to output the output value to the node of the next layer can also be part of a set of model parameters. Therefore, unlike the fully coupled layer described later, each node of the convolutional layer is not necessarily coupled to all nodes of the previous layer by inputting an input value, and is not necessarily coupled by outputting an output value to all nodes of the next layer.

The pooling layer is a layer with nodes that determine a representative value based on the input values input from multiple nodes in the previous layer, and output the determined representative value as an output value to the next layer. As the representative value, for example, a value that statistically represents a plurality of input values such as a maximum value, an average value, and a mode value is used. The stride is preset in the pooling layer. The stride shows the range of adjacent nodes one layer above the reference input value for 1 node. Therefore, a pooling layer can also be regarded as a layer that reduces (downsamples) the input value from the previous layer to a lower dimension and provides the output value to the next layer.

A fully-coupled layer is a layer that, at each of a plurality of nodes, performs a convolution operation on the input value input from the previous layer to calculate a convolution value, and calculates and adds the calculated convolution value to a bias value. The obtained calculated value is used as an output value, and the calculated output value is output to the next layer. That is, the fully coupled layer is a layer that outputs an operation value obtained by performing convolution processing on all of the plurality of input values input from the previous layer using parameter sets (cores) that are smaller in number than the number of input values. Therefore, in the fully coupled layer, the parameters of the convolution coefficient, bias value and activation function become part of a set of model parameters. In this way, by arranging the fully coupled layer immediately before the output layer, the final output value can be derived while reducing degrees of freedom while fully considering the components that significantly affect the characteristic values given by the upper layer.

In addition, the number of CNN layers, the type of each layer, the number of nodes in each layer, etc. are not limited to those shown in FIG. 14 . The CNN of this embodiment is only required to have a configuration capable of calculating the probability of each type of defect as an output value for a measurement signal having a signal value of each of a plurality of sample points as an input value. However, the CNN of this embodiment preferably has an intermediate layer as shown in FIG. 14 , and the intermediate layer is formed by sequentially stacking one or more convolutional layers and pooling layers alternately for one cycle or more. This is to lock components that have a significant impact on feature values through the repetition of convolutional layers. In addition, in the repetition of the convolutional layer, the pooling layer can be omitted.

Also, as a multi-level model of three or more levels, a machine learning model in which the number of calculation elements is three or more vector values as output values may be used. Taking the case of applying to the determination of the type of defect as an example, the element of the output value is obtained as the probability of the type of defect corresponding to the element. In the example shown in FIG. 14 , it is necessary to preset parameter groups corresponding to the requirements of each output value in the fully coupled layer Fc14 . Thus, in a multilevel model, optimization based on specific criteria is performed on all sets of parameters during model learning. Therefore, as described above, even if it is intended to modify the parameter set only for the determination of one type of defect, it may affect the determination results of other types of defects. Therefore, the defect identification unit 114 of the present embodiment preferably uses a plurality of two-level models for determining conformity/nonconformity of each type of defect.

In addition, in the above example, although the signal value of each pixel is mainly used as the input value to the machine learning model, it is not limited to this. The control unit 110 of the defect inspection device 100 may include a feature analysis unit (not shown) that calculates feature quantities of the features of the pattern (including defects) shown in the display image based on the image data. Furthermore, all or a part of the defect discriminators 114-1 to 114-N may be used instead of the signal value for each pixel, or the feature quantity calculated by the feature analysis unit may be used as an input value to the machine learning model together with the signal value. The model learning unit 118 replaces the signal value of each pixel as an input value for forming the training data for calculating the model parameters of the machine learning model, or uses the feature quantity calculated by the feature analysis unit together with the signal value as an input value for machine learning. The input value of the model is used. As the feature quantity, for example, shape feature parameters such as circularity, Euler number, and Feret diameter, HOG (Histograms of Oriented Gradients: Histogram of Oriented Gradients) feature quantity, SIFT (Scaled Invariance Feature Transform: scale-invariant Feature Transformation) Any one of the feature quantities used for image recognition, such as feature quantities, or a combination thereof. In other words, the calculated feature quantity can be used as a feature quantity of a defect indicating the state of the object to be inspected. The new type determination unit 122 can use the feature quantity calculated by the feature analysis unit to determine the new type. In addition, in the judgment data, as information showing the state of the object to be inspected, instead of the presence or absence or the number of defects for each type of defect, image feature values may be included. Furthermore, the defect identification unit 114 or the comprehensive determination unit 116 may further include: a defect identification unit that refers to the determination data and determines the type of the defect based on the image feature value analyzed by the feature analysis unit.

As described above, the defect inspection apparatus 100 of this embodiment is a defect inspection apparatus that inspects defects generated in an object to be inspected based on an image of the object to be inspected, and includes a plurality of defect discriminators based on the image For example, using a specific machine learning model to identify different types of defects. The type of defect discriminated by each defect discriminator is a part of the types of a specific number of defects which are set as discrimination objects by the defect inspection device. In addition, the plurality of defect discriminators can respectively determine whether the type of defect generated in the object to be inspected corresponds to a specific type of defect. In addition, the object to be inspected is glass, and can be realized by a manufacturing method having an inspection step using the above-mentioned defect inspection device. With this configuration, each defect discriminator determines whether or not the types of defects detected from the image are part of the specific number of types of defects. When a specific defect discriminator changes the parameter set used to determine the type of defect, unlike the case where all the types of a specific number of defects are set as the target of judgment, the parameter sets used by other defect discriminators, Or the determination of the type of the defect is affected. In particular, when each defect discriminator determines whether or not a specific defect is met, deterioration of determination accuracy can be avoided. Thus, system administration becomes easier.

Also, each of the plurality of defect discriminators can discriminate in parallel the type of defect generated in the object to be inspected regardless of the judgment results of other defect discriminators. With this configuration, since the determination of conformity/nonconformity for each type of defect is performed in parallel, even if the types of defects to be determined increase, the processing time does not increase, and thus rapid processing can be realized.

Also, the number of defect discriminators is N, and the nth defect discriminator judges whether the type of defect generated in the inspected object conforms to the type of the nth defect, and the nth defect discriminator judges whether the type of defect generated in the inspected object When the type of the defect does not match the type of the nth defect, the n+1th defect discriminator can start the process of judging whether the type of the defect generated in the above-mentioned inspected object matches the type of the n+1th defect. With this configuration, each defect classifier executes in series the process of judging whether or not it matches the type of a specific defect. Since the amount of processing is prevented from becoming excessively large, it contributes to realization of economy.

Also, n is defined as the descending order of occurrence frequency or risk level of the nth defect type. With this configuration, since the types of defects with higher frequency of occurrence or types of defects with higher risk of occurrence are discriminated with higher priority, damage due to occurrence of defects can be suppressed as a whole system.

Moreover, a plurality of defect discriminators can respectively use machine learning models to extract feature quantities of defects. With this configuration, even if a specific characteristic amount is not defined in advance, a characteristic corresponding to each defect type is expressed.

In addition, a model learning unit may be provided that determines model parameters for discriminating the specific type of defect using a machine learning model for image data showing images including a specific type of defect. With this configuration, the model parameters for discriminating the type of the defect can be determined using the training data showing the relationship between the image data used as an input value and the type of defect used as an output value. Therefore, determination accuracy can be improved by using model parameters corresponding to the use environment to determine the type of defect.

In addition, the defect classifier can also classify the type of defect based on image data showing a plurality of images under different imaging conditions of the object to be inspected. With this configuration, it is possible to more accurately determine the type of defect based on differences in image characteristics for each imaging condition.

In addition, it is also possible to include: a new type determination unit that displays the distance in space between the characteristic quantities of the defects, that is, the representative feature quantities predetermined for each type of defect, and the feature quantities extracted from the image, i.e. The distance of the extracted feature is calculated, and when the calculated distance is greater than the threshold value of the specific distance for any of the types of defects, the type of defect detected from the image is determined as a new type. With this configuration, a defect whose characteristics are different from those of a known defect can be identified as a new type of defect. Therefore, step management corresponding to defects with different characteristics can be facilitated.

Moreover, you may be equipped with the new type determination part which determines the type of the defect detected from the said image as a new type, when the defect classifier which successfully discriminated the type of a defect does not exist. With this configuration, it is possible to discriminate a defect whose type cannot be discriminated as a new type of defect. Thus, step management that is not dependent on the type of known defect can be facilitated.

In addition, it may further include a manufacturing process management section that determines correction conditions for correcting the manufacturing conditions of the object to be inspected based on the states of defects discriminated by the plurality of defect detectors. With this configuration, it is possible to efficiently control the manufacturing steps of the object to be inspected in accordance with the state of the discriminated defect without relying on manpower.

In addition, it may also include: a feature analysis unit that analyzes the feature quantity of a defect based on an image; and a second defect discriminator that uses judgment data showing the relationship between the feature quantity of a defect and the type of the defect, based on the above-mentioned feature analysis unit. The characteristic quantity of the analyzed defect determines the type of the defect. With this configuration, the type of the defect corresponding to the detected characteristic amount of the defect is specified based on the known relationship between the characteristic amount of the defect and the type of defect. Since the types of defects can be determined without relying on the machine learning model at all, the amount of processing can be reduced.

In addition, a judgment input unit for specifying the type of defect occurring in the object to be inspected as the type of defect indicated by the obtained operation input may be provided. With this configuration, the type of defect judged by the user can be known. Since the types of defects can be determined without relying on the machine learning model, the risk of determination errors can be reduced.

In addition, the second defect classifier or the judgment input unit may specify the defect occurring in the object to be inspected before the plurality of defect classifiers specify the type of the defect occurring in the object to be inspected. With this configuration, since the judgment using the known relationship between the feature quantity of the defect and the type of the defect or the user's judgment is performed, even if the type of the defect cannot be determined by the machine learning model, the type of the defect can be identified .

Moreover, in the manufacturing method of the glass of this embodiment, the object to be inspected is glass, and may have the inspection process using the said defect inspection apparatus.

As mentioned above, although the embodiment of this invention was described in detail with reference to drawing, the specific structure is not limited to the above-mentioned, and various design changes etc. are possible in the range which does not deviate from the summary of this invention.

For example, the defect inspection device 100 may be implemented as a part of the manufacturing equipment of the object to be inspected, or may be a single machine independent of the object to be inspected. The defect inspection device 100 is not limited to manufacturing equipment, and may acquire image data from other devices such as a data accumulation device and a PC (Personal Computer). Moreover, the defect inspection apparatus 100 may be equipped with the imaging part 130, the operation part 150, and the display part 160, and some or all of them may be omitted. The imaging unit 130 , the operation unit 150 , and the display unit 160 are respectively connectable via the input and output unit 140 .

In the defect inspection device 100, part or all of the model learning unit 118, the manufacturing process management unit 120, the new type determination unit 122, and the determination input unit 124 may be omitted. One or both of the defect detection unit 112 and the comprehensive judgment unit 116 may be omitted depending on the type of the object to be inspected or the type or number of defects to be detected. The width, length, and thickness of the glass to be inspected are arbitrary. In addition, the defect inspection device 100 can be applied to determine the presence or absence of defects or the type of defects in objects other than glass as the object to be inspected, for example, circuit boards, wafers, and the like.

In addition, part or all of the defect inspection apparatus 100 in the above-mentioned embodiment may be realized as an integrated circuit such as LSI (Large Scale Integration). Each functional block of the defect inspection device 100 may be individually processed, or a part or all of them may be integrated and processed. In addition, the method of forming an integrated circuit is not limited to LSI, and it may be realized by a dedicated circuit or a general-purpose processor. In addition, when a technology of integrating circuits instead of LSI appears due to the advancement of semiconductor technology, it is also possible to use an integrated circuit utilizing this technology. [Industrial availability]

According to the defect inspection device, defect inspection method, and manufacturing method of the above-mentioned various types, each defect discriminator determines whether the types of defects detected from the image are part of the types of the specific number of defects. When a specific defect discriminator changes the parameter set used to determine the type of defect, unlike the case where all the types of a specific number of defects are set as the target of judgment, the parameter sets used by other defect discriminators, Or the determination of the type of the defect is affected. Thus, system administration becomes easier.

100: Defect inspection device 110: control department 112:Defect detection department 114:Defect identification department 116:Comprehensive Judgment Department 118: Model Learning Department 120:Manufacturing step management department 122: New Type Judgment Department 124: Judgment input unit 130: Camera department 140: Input and output section 150: Operation Department 160: display part 170: memory department Cv04, Cv08, Cv12: convolutional layer Fc14: fully coupled layer In02: Input layer Out16: output layer Pl06, Pl10: pooling layer S102, S104, S122, S124, S126, S200, S208: steps S102-S122: Steps S110-1～S110-3: Steps S112-1～S112-3: Steps S114-1～S114-3: Steps S116-1～S116-3: Steps S118-1～S118-3: Steps S122-a, S122-b, S122-c, S122-d, S122-n: steps S201～S205: Steps Sb: object to be inspected

FIG. 1 is a schematic block diagram showing a configuration example of a defect inspection device according to this embodiment. Fig. 2 is a flow chart showing the first example of inspection processing in this embodiment. Fig. 3 is an explanatory diagram for explaining imaging conditions in the imaging section of the present embodiment. Fig. 4 is a diagram showing an example of an image of a subject to be inspected. Fig. 5 is a diagram showing another example of an image of a subject to be inspected. Fig. 6 is a diagram showing a connection example of the defect discriminator of this embodiment. Fig. 7 is a diagram showing an example of the processing time of the defect discrimination processing in this embodiment. Fig. 8 is a diagram showing an example of the error rate of the type of defect caused by the machine learning model of the present embodiment. Fig. 9 is a flow chart showing a second example of inspection processing in this embodiment. Fig. 10 is an explanatory diagram for explaining the distance in the feature quantity space of the present embodiment. Fig. 11 is a flowchart showing a third example of inspection processing in this embodiment. FIG. 12 is an explanatory diagram of an example of feedback processing from the inspection step to the manufacturing step in this embodiment. Fig. 13 is a flow chart showing an example of the manufacturing steps of the glass of this embodiment. Fig. 14 is a diagram showing an example of the machine learning model of this embodiment.

S102, S104, S122: steps

S110-1~S110-3: Steps

S112-1~S112-3: Steps

S114-1~S114-3: Steps

S116-1~S116-3: Steps

S118-1~S118-3: Steps

Claims (15)

A defect inspection device for inspecting defects generated in the object to be inspected based on an image of the object to be inspected, comprising: A plurality of defect discriminators, based on the above-mentioned images, the defect discriminator uses a specific machine learning model to distinguish different types of the above-mentioned defects; The types of the above-mentioned defects discriminated by each defect discriminator are part of the types of a specific number of defects that the above-mentioned defect inspection device sets as discrimination objects. The defect inspection device as claimed in item 1, wherein Each of the plurality of defect discriminators discriminates in parallel the types of defects generated in the object under inspection regardless of the judgment results of other defect discriminators. The defect inspection device as claimed in item 1, wherein The number of the above defect discriminators is N; The nth (n is an integer greater than 1 and less than N-1) defect discriminator determines whether the type of defect generated in the above-mentioned inspected object conforms to the type of the nth defect; When the nth defect discriminator judges that the type of defect generated in the above-mentioned inspected object does not meet the type of the n-th defect, the n+1th defect discriminator starts to judge whether the type of defect generated in the above-mentioned inspected object Handling according to the type of defect n+1. The defect inspection device as claimed in claim 3, wherein The above n is determined in descending order of occurrence frequency or risk level of the nth defect type. The defect inspection device according to any one of claims 1 to 4, wherein A plurality of the above-mentioned defect discriminators respectively judge whether the type of the defect generated in the above-mentioned inspected object corresponds to a specific type of defect. The defect inspection device according to any one of Claims 1 to 5, which has: The model learning unit determines model parameters for discriminating the specific type of defect using the above-mentioned machine learning model on the image data representing the image containing the specific type of defect. The defect inspection device according to any one of claims 1 to 6, wherein The defect discriminator discriminates the type of the defect based on image data representing a plurality of images under different imaging conditions of the object to be inspected. The defect inspection device according to any one of Claims 1 to 7, which has: The new type determination unit calculates the distance in space of the feature quantity representing the feature of the defect, that is, the distance between the representative feature quantity predetermined for each type of the above-mentioned defect and the feature quantity extracted from the above-mentioned image, that is, the extracted feature quantity distance; and When the calculated distance is greater than a threshold value of a specific distance with respect to any of the above-mentioned defect types, the type of defect detected from the above-mentioned image is determined to be a new type. The defect inspection device according to any one of Claims 1 to 7, which has: The new type determination part determines the type of the defect detected from the said image as a new type, when there is no defect discriminator which successfully discriminated the type of defect. The defect inspection device according to any one of Claims 1 to 9, further comprising: A manufacturing process management unit that determines correction conditions for correcting manufacturing conditions of the inspected object based on the states of the defects discriminated by the plurality of defect classifiers. The defect inspection device according to any one of Claims 1 to 10, which has: Feature analysis department, which analyzes the feature quantity of defects based on image data; A second defect discriminator, which uses determination data indicating the relationship between the feature quantity of the defect and the type of the defect, and determines the type of the defect based on the feature quantity of the defect analyzed by the above-mentioned feature analysis section; and A determination input unit that determines the type of defect occurring in the object to be inspected as the type of defect indicated by the acquired operation input. The defect inspection device according to claim 11, wherein Before determining the type of the defect generated in the object under inspection by the plurality of defect classifiers, the defect generated in the object under inspection is identified by the second defect classifier or the judgment input unit. A defect inspection method, which inspects defects generated in the object to be inspected based on an image of the object to be inspected, and has: a plurality of defect identification steps, which use a specific machine learning model to identify different types of the above-mentioned defects based on the above-mentioned images; and The above-mentioned defect types discriminated in each defect discriminating step are part of a specific number of defect types set as discriminating objects in the above-mentioned defect inspection method. A glass manufacturing method using the defect inspection method of claim 13, and The object to be inspected is glass. A method of manufacturing glass having an inspection step using the defect inspection device according to any one of claims 1 to 12, and The object to be inspected is glass.

Images