JP2018005640A - Classifying unit generation device, image inspection device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a classifying unit generation device capable of easily generating a classifying unit with high classifying accuracy.SOLUTION: A control part 30 of a classifying unit generation device generates a CNN (Convolutional Neutral Network) 313 by deep layer learning based on first teacher data (for example, each teacher data in a first teacher data base or the like) including a plurality of sample images capturing a defect. Moreover, each of a plurality of sample images capturing the defect is inputted into the intermediate layer of the CNN, and a plurality of kinds of feature amounts are obtained from the intermediate layer with respect to a plural kinds of second images. Further, a decision tree group 327 is generated by machine learning based on the plurality of kinds of feature amounts.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a classifier generation device, an image inspection device, and a program.

  2. Description of the Related Art Conventionally, an image inspection apparatus that inspects the presence or absence of defects (scratches, foreign matter, etc.) of an inspection target based on a captured image (particularly, a feature amount) obtained by imaging the inspection target is known (for example, Patent Documents). 1).

JP 2006-266872 A

  It is determined by a person (a person who manages generation of a classifier that classifies images, a user of a classifier, etc.) what kind of feature quantity is used as the feature quantity of Patent Document 1. However, it is usually difficult to find a useful feature amount when classifying an image, and it is difficult to obtain an accurate classifier that uses a useful feature amount.

  It is an object of the present invention to provide a classifier generation device, an image inspection device, and a program that can easily generate a classifier with high classification accuracy.

In order to achieve the above object, a classifier generation device according to a first aspect of the present invention includes:
CNN generation means for generating a CNN (Convolutional Neural Network) by deep learning based on first teacher data including a plurality of first images in which defects are shown;
A feature amount for inputting each of a plurality of second images showing a defect to the intermediate layer constituting the CNN generated by the CNN generation means, and acquiring a plurality of types of feature amounts for each of the plurality of second images. Acquisition means;
A classifier generating means for generating a classifier by machine learning based on second teacher data including at least some types of feature quantities of the plurality of types of feature quantities acquired by the feature quantity acquisition means;
Is provided.

In order to achieve the above object, an image inspection apparatus according to the second aspect of the present invention provides:
The classifier generator;
The classifier generated by the classifier generator, and
An image in which an inspection object is captured and an image of a defect is classified by the classifier.

In order to achieve the above object, a program according to the third aspect of the present invention provides:
On the computer,
A CNN generating step of generating a CNN (Convolutional Neural Network) by deep learning based on first teacher data including a plurality of first images in which defects are reflected;
A feature amount for inputting each of a plurality of second images showing defects to the intermediate layer forming the CNN generated by the CNN generation step, and acquiring a plurality of types of feature amounts for each of the plurality of second images. An acquisition step;
A classifier generation step of generating a classifier by machine learning based on second teacher data including at least some types of feature amounts of the plurality of types of feature amounts acquired by the feature amount acquisition step;
Is executed.

  According to the present invention, a classifier having high classification accuracy can be easily generated.

It is a figure which shows the structural example of the external appearance inspection apparatus which concerns on one Embodiment of this invention. It is an example of a defect. It is a figure which shows the structural example of the control part etc. for implement | achieving a CNN production | generation function. It is a figure which shows the structural example of CNN. It is a figure which shows the structural example of the control part etc. for implement | achieving a decision tree production | generation function. It is a figure which shows the structural example of the control part etc. for implement | achieving an image inspection function. It is a flowchart which shows an example of a feature-value reduction process. It is a figure which shows the structural example of the control part etc. for implement | achieving the construction function of a teacher database (teacher data). It is a flowchart which shows an example of the teacher data generation process which a teacher data generation part performs. It is an example of the screen displayed on a display part under control of a teacher data generation part.

(Outline of the appearance inspection apparatus 100)
The image inspection apparatus 100 according to an embodiment of the present invention has the following functions.
(1) CNN generation function for generating CNN (Convolutional Neural Network) by deep learning (2) From a plurality of decision trees by machine learning using feature values output from the intermediate layer of the CNN as teacher data A decision tree generation function for generating a decision tree group by a random forest method (3) a function of performing an image inspection based on an image obtained by imaging an inspection target (for example, a product to be inspected for defects in an actual production line); Specifically, an image inspection function that detects a defect candidate based on an image obtained by imaging an inspection object and classifies the type of the detected defect candidate using the intermediate layer of the CNN and the decision tree group.

  In this embodiment, the inspection object is a transparent film. Defects include foreign matter (black spots), scratches (black scratches), etc. on the transparent film. In the generation of the CNN and the generation of the decision tree group, a sample image (assumed to be prepared in advance) in which a transparent film (sample) with defects actually attached is used as teacher data. The CNN is particularly preferably DCNN (Deep Convolutional Neural Network).

(Configuration of the image inspection apparatus 100)
As shown in FIG. 1, the image inspection apparatus 100 includes an imaging unit 10, a storage unit 20, a control unit 30, an operation unit 40, and a display unit 50. These are connected to the bus 60.

  The imaging unit 10 includes a camera using a CCD (Charge Coupled Device) image sensor or a camera using a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The imaging unit 10 is controlled by the control unit 30 to image the inspection target and obtain a captured image in which the inspection target is captured. The imaging unit 10 supplies the image data of the captured image to the control unit 30.

  The storage unit 20 includes a hard disk or a flash memory. The storage unit 20 stores a program, a first teacher database (detailed later), a second teacher database (detailed later), and various data including data constituting each.

  The control unit 30 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and the like. Programs and data stored in the storage unit 20 are read out to the RAM. The CPU performs predetermined processing in accordance with the program read into the RAM and using the data read into the RAM, so that each part (teacher data supply unit 311, CNN 313, feature quantity) of the control unit 30 is performed. The acquisition unit 321, the dimension reduction unit 323, the random forest processing unit 325, the decision tree group 327, the image processing unit 331, the analysis unit 333, and the like), and the above (1) to (3) of the image inspection apparatus 100. Realize the function.

  The operation unit 40 includes a keyboard, a mouse, and the like, and receives an operation from the user.

  The display unit 50 includes a liquid crystal display, an EL (Electro Luminescence) display, or the like, and displays various images. The display unit 50 may be a touch panel. In this case, the touch panel is also the operation unit 40.

(Configuration and operation of control unit 30)
Next, the configuration and operation of the control unit 30 will be described with reference to FIGS. The types of defects or defect candidates in the image inspection here are black spots and black scratches as shown in FIG. Black dots are images of foreign matters (dust etc.) adhering to the inspection target (transparent film). The black scratch is an image of a scratch attached to the inspection target. As illustrated in FIGS. 2A-1 to 2A-4, there are various modes even with the same black spot. The control unit 30 classifies defects that have different aspects but can be classified as the same black spot as black spots. Such a thing is the same also in a black crack (refer to Drawing 2 (B-1))-Drawing 2 (B-3)).

(Control unit 30 that generates CNN by deep learning)
As illustrated in FIG. 3, the control unit 30 includes a teacher data supply unit 311 and a CNN 313. The teacher data supply unit 311 reads out teacher data of the first teacher database (Data Base) stored in the storage unit 20 and supplies it to the CNN 313. The CNN 313 performs deep learning using the supplied teacher data.

  The first teacher database is created in advance and includes a plurality of teacher data (teacher data A and the like). One teacher data includes a sample image (image A or the like) in which a transparent film with a defect (sample to be inspected) is shown, and a type of defect (black scratch or the like in the sample image; hereinafter referred to as a correct answer type). It consists of. The sample image may be an image obtained by performing noise removal or contrast enhancement on the captured image in addition to the captured image obtained by capturing the sample to be inspected. The sample image is, for example, a gray scale image. In the first teacher database, a sample image and a correct answer type are stored in association with each other for each teacher data. The correct answer type is a type in which the person visually confirms and classifies the image, and is correct data in each teacher data.

  The teacher data supply unit 311 acquires teacher data one by one from the first teacher database and supplies the teacher data to the CNN 313. The CNN 313 learns whenever one piece of teacher data is supplied.

  The configuration of the CNN 313 is, for example, as shown in FIG. Layer 313I is provided. The CNN 313 classifies the type of image (including classification of the type of defect candidate shown in the image). Note that classifying the type of image is synonymous with classifying the type of the subject (including defects and defect candidates) in the image.

  An image (luminance data or the like) whose classification is to be classified is input to the convolution layer 313B. Hereinafter, this image is referred to as an input image. The convolution layer 313B also has a role as an input layer of the CNN 313.

  The convolution layer 313B performs convolution on the input image, and the pooling layer 313C performs pooling on the input image after convolution. The convolution layer 313D performs convolution again on the input image after pooling, and the pooling layer 313E performs pooling again on the input image after convolution. Convolution is to perform a convolution operation on a pixel in a certain area centered on a certain pixel. Such convolution can express image blur processing, contour extraction processing, and the like, and can extract local features on the image. Pooling is a process of converting the luminance value of each pixel of an image into one value for each fixed region. One value is a maximum value (Max Pooling) or an average value (such as a luminance value in the region). Average Pooling) can be used. By reducing the positional sensitivity within the region by pooling, it is possible to obtain the translation invariance (robustness against translation) of the image.

  The normalization layer 313F normalizes the information output from the pooling layer 313E so that the subsequent identification layer can be easily processed. Normalization may be performed by an appropriate method. For example, the normalization layer 313F absorbs light and shade differences such as image brightness differences.

  The first perceptron layer 313G to the third perceptron layer 313I process the information output from the normalization layer 313F to identify (classify) the type of the input image. The information output from the third perceptron layer 313I is information indicating the type of the input image specified by the classification. This information is output in a format in which, for example, the type of the input image specified by the classification and the likelihood of the type (possibility of the type) are associated with each other. For example, the type of defect of the input image specified by the classification is output in a format such as black scratch: 90%, black point: 10%. The CNN 313 may output the most likely type as a classification result.

  The convolution layer 313B, the pooling layer 313C, the convolution layer 313D, the pooling layer 313E, and the normalization layer 313F are collectively referred to as an intermediate layer. Note that an input layer as the first layer of the intermediate layer may be provided before the convolution layer 313B. A plurality of normalization layers 313F may be provided at appropriate positions. The first perceptron layer 313G to the third perceptron layer 313I are collectively referred to as an identification layer (corresponding to a classifier that classifies images). An output layer as the final layer of the identification layer may be provided after the third perceptron layer 313I. The intermediate layer is in the preceding stage of the identification layer and can be regarded as a layer for extracting the feature amount of the input image. Here, information supplied from the normalization layer 313F to the first perceptron layer 313G is a feature amount of the input image. Here, a plurality of types (for example, 4096 types) of feature quantities (represented by numerical values) are extracted and output to the identification layer.

  Three or more convolution layers and pooling layers may be provided, and the processing of convolution and pooling as one set may be performed three or more times. Moreover, in one convolution layer, the convolution may be performed twice or more. In the convolution layer, other arithmetic processing may be performed in addition to the convolution. As in the intermediate layer, an appropriate feature amount can be extracted by performing a process of performing pooling a plurality of times after performing convolution on the input image. In particular, the feature quantity extracted here is a quantity related to a very abstract feature (not a specific feature such as a roundness found by a person) in the sense that it is difficult for a person to grasp. In many cases, the feature amount is more appropriate for classifying the image than the feature amount that can be found. Further, an appropriate number of perceptron layers constituting the identification layer can be adopted.

  As the CNN 313 before the supply of the teacher data is started, an existing one can be adopted (a well-known learning method can also be adopted). The CNN 313 performs deep learning by updating variables used in convolution and pooling (such as filter variables used for convolution). The CNN 313 before the start of supplying the teacher data may be learned to some extent (including those learned by using an image other than a defect) (so-called fine tuning) or unlearned. There may be. The learning of the CNN 313 ends when all the teacher data included in the first teacher database is used. Alternatively, the CNN 313 may be classified using the test data prepared in advance, and the learning of the CNN 313 may be terminated when a desired accuracy rate is obtained. Note that the CNN 313 used thereafter does not perform any further learning, but may perform further learning at an appropriate timing not mentioned. In addition, since only the intermediate layer of CNN 313 is used thereafter, the identification layer may be deleted.

(Control unit 30 for generating a decision tree group by the random forest method)
In addition to the CNN 313 generated above, the control unit 30 includes a feature amount acquisition unit 321, a dimension reduction unit 323, and a random forest processing unit 325 as shown in FIG.

  The feature amount acquisition unit 321 accesses the second teacher database stored in advance in the storage unit 20. The second teacher database is created in advance and includes a plurality of teacher data (teacher data a and the like). One teacher data includes a sample image (image a or the like) showing a transparent film with a defect (sample to be inspected), a feature amount of the sample image (hereinafter also referred to as a sample feature amount), and the sample image. And the type of defect (black scratch etc., hereinafter referred to as correct answer type). Note that the sample feature amount is not initially stored in the second teacher database, but is added at a later-described timing. In the second teacher database, a sample image, a sample feature amount, and a correct answer type are stored in association with each other for each teacher data. The explanation about the correct answer type and the sample image is based on the first teacher database.

  At least a part of the sample images and the correct answer types in the second teacher database and the first teacher database may be common. The sample image and the correct answer type may all be common, and in that case, only the second teacher database may be prepared. By sharing the sample image and the correct answer type in both databases, the required data amount can be reduced.

  The feature amount acquisition unit 321 reads one teacher data from the second teacher database, and inputs a sample image included in the read teacher data (processing target teacher data) to the CNN 313. The CNN 313 extracts and outputs a plurality of types of feature amounts representing the supplied sample image in the intermediate layer. The feature quantity acquisition unit 321 adds a plurality of types of feature quantities extracted in the intermediate layer to the second teacher database as sample feature quantities included in the teacher data to be processed this time. The feature quantity acquisition unit 321 performs these processes for all teacher data in the second teacher database. Thereby, the second teacher database (after the addition of the feature amount) is generated from the second teacher database (before the addition of the feature amount) in FIG. A database different from the second teacher database may be generated for the sample feature amount and the correct answer type, and the following processing may be performed based on the database.

  Thereafter, the dimension reduction unit 323 reduces the dimension of the feature amount (type of feature amount) extracted by the feature amount acquisition unit 321. As an example, the dimension reduction unit 323 performs statistical processing such as principal component analysis and manifold learning (including those using an ISO map) based on all sample feature amounts stored in the second teacher database. And one or more feature amounts that can be reduced are specified from among a plurality of types of feature amounts output from the CNN 313. The feature amount specified here is referred to as a reduced feature amount. The dimension reduction unit 323 deletes one or more types of reduction feature amounts from a plurality of types of sample feature amounts for each teacher data stored in the second teacher database, and updates the second teacher database (FIG. 5). Refer to the second teacher database (after feature amount reduction), where xi (i = 1, 2, 3, 4,... Is reduced). Plural types of feature quantities remain after deletion. Hereinafter, each remaining feature amount after reducing the reduced feature amount from a plurality of types of feature amounts is referred to as a remaining feature amount. Note that the dimension reduction unit 323 records, in the storage unit 20, information that identifies which type of feature quantity the reduction feature quantity is.

  The random forest processing unit 325 generates a decision tree group 327 including a plurality of decision trees by machine learning based on the random forest method using each teacher data in the third teacher database as teacher data including a correct answer. Each teacher data here includes a plurality of types of remaining feature amounts and correct answer types. The decision tree group 327 generated here is a component of the control unit, and a component of a classification device that classifies image types together with the intermediate layer of the CNN 313. Each of the plurality of decision trees constituting the decision tree group 327 can be said to be a weak classifier with low correlation and low classification accuracy, but depending on the randomness given when the weak classifier is generated (for example, which feature amount The decision tree (decision tree group) as a whole is a classifier having high classification performance by, for example, randomly selecting whether to use each decision tree).

  Note that the dimension reduction unit 323 does not delete the reduction feature quantity from the second teacher database, but the dimension reduction unit 323 determines what type of feature quantity the reduction feature quantity recorded in the storage unit 20 is. With reference to the information to be identified, when the random forest processing unit 325 generates the decision tree group 327, the reduced feature amount may be deleted from the sample feature amount.

(Control unit 30 that performs image inspection based on an image obtained by imaging an inspection object)
The control unit 30 includes an image processing unit 331 and an analysis unit 333 as shown in FIG. 6 in addition to the CNN 313 (particularly the intermediate layer), the dimension reduction unit 323, and the decision tree group 327 generated as described above. The image processing unit 331 is connected to the imaging unit 10. The imaging unit 10 images the inspection target at an appropriate timing. For example, the imaging control unit included in the control unit 30 controls the imaging unit 10 to perform imaging. The image processing unit 331 acquires an image obtained by imaging the inspection target from the imaging unit 10. The image processing unit 331 performs processing for determining whether a defect is included in the acquired image. For example, the image processing unit 331 binarizes the image from the imaging unit 10 (which may have been subjected to noise removal, contrast enhancement, or the like) with a predetermined threshold, and a black region in the binarized image Is identified by a labeling process or the like. If the number of pixels in the specified area is equal to or greater than a predetermined number, the image processing unit 331 determines that the area is a defect, and cuts out an image in a predetermined range centered on the defect from the captured image.

  Then, the image processing unit 331 inputs the clipped image to the CNN 313 as an image with a defect candidate. If the sample image is a grayscale image, the image with a defect candidate is also grayscale. In this case, the image output from the imaging unit 10 may be grayscale, or the image processing unit 331 may convert the color image to grayscale. As described above, the format of the sample image and the format of the image with defect candidates are the same.

  The intermediate layer of the CNN 313 extracts a plurality of types of feature amounts of the input defect candidate image. The extracted plural types of feature amounts are reduced by the dimension reduction unit 323. Specifically, the dimension reduction unit 323 reads information specifying which type of feature quantity the reduction feature quantity is from the storage unit 20. Then, based on the read information, the dimension reduction unit 323 deletes the reduced feature amount from the plurality of types of feature values extracted by the intermediate layer of the CNN 313, and deletes the types of feature values (a plurality of types of remaining feature values). Feature amount) is supplied to the decision tree group 327. The decision tree group 327 classifies the types of defect candidates shown in the defect candidate image based on the supplied plural types of remaining feature amounts, and classifies each decision tree classification result (classified type). The data is supplied to the analysis unit 333. Note that each decision tree is generated by the random forest method, and the classification is classified using some or all of the remaining feature amounts of all types from the dimension reduction unit 323. To do. Therefore, depending on the type of feature quantity randomly selected when generating the decision tree group 327, the decision tree group 327 may not use all types of remaining feature quantities.

  The analysis unit 333 aggregates the classification results of each decision tree of the decision tree group 327, and outputs the aggregation result to the display unit 50 as the classification result of the entire decision tree group 327. For example, the analysis unit 333 associates the type of the defect candidate identified in the image with the defect candidate identified in the current classification with the probability of the type (possibility of the type) in association with the display unit 50. To display. For example, the analysis unit 333 displays the types of defect candidates in a format such as black flaws: 90%, black dots: 10%, and the like. Note that the analysis unit 333 may take the majority of the classification results of each decision tree of the decision tree group 327 and output the largest number of classification results to the display unit 50 as the classification result of the entire decision tree group 327. The analysis unit 333 may take the probability distribution of the classification result of each decision tree of the decision tree group 327 and output the classification result having the maximum average value to the display unit 50 as the classification result of the entire decision tree group 327.

(Effects of this embodiment)
According to the present embodiment, the feature quantity input to the decision tree group 327 is extracted by the learned intermediate layer of the CNN 313 (particularly, the process of performing one set of convolution and pooling is performed a plurality of times). A feature amount useful for classification of types (as described above, an abstract feature amount but a useful feature amount in the sense that it is difficult for a person to grasp) can be obtained by a machine. Further, the classification accuracy is improved by using the decision tree group 327 generated by the random forest method instead of the identification layer of the CNN 313. As described above, the image inspection in the present embodiment has good classification accuracy (it is known that the classification accuracy is improved by experiments). In addition, since the person does not determine what kind of feature quantity is input to the decision tree group 327, it is not necessary to rely on human experience to determine the feature quantity to be used. For this reason, according to the present embodiment, the classification of defect candidate types can be maintained with high accuracy.

  Further, since the feature quantity used for generating the classifier (in this case, the decision tree group 327) used for classifying the defect type is extracted by the intermediate layer of the learned CNN, the classifier for generating the classifier is generated. Useful features can be obtained by machine, and as a result, the classification of defect types can be made highly accurate (experiment shows that the above configuration is also suitable for defect classification. ).

  In addition, since the dimension reduction unit 323 reduces some types of feature amounts of the plurality of types of feature amounts output from the intermediate layer of the CNN 313, processing related to generation of the decision tree group 327, and image inspection The burden of processing can be reduced.

(About feature reduction)
The dimension reduction unit 323 may reduce the feature amount by executing the feature amount reduction process illustrated in FIG.

  The dimension reduction unit 323 measures the degree of contribution to the correct answer rate for each one of each type of feature quantity stored in the second teacher database (database before dimension reduction) (step S301). The dimension reduction unit 323 generates, for example, all types of feature quantities, and measures the correct answer rate of the generated classifier. Next, the dimension reduction unit 323 deletes the feature amount as a contribution measurement target, generates it using the remaining feature amount after the deletion, and measures the correct answer rate of the generated classifier. The classifier here may be a classifier such as a support vector machine, a neural network, or a random forest classifier. The measurement of the correct answer rate is performed using test data prepared separately, for example. The dimension reduction unit 323 specifies the difference between the correct answer rates of the two classifiers generated as a contribution. The dimension reduction unit 323 performs the above measurement for all feature amounts, and specifies contributions for all feature amounts. The greater the deterioration of the correct answer rate (the difference in correct answer rate), the higher the contribution.

  Next, the dimension reduction unit 323 creates a table in which the feature amounts are sorted in ascending order of contribution (step S302). Thereafter, the dimension reduction unit 323 sets a counter K indicating the number of feature values to be deleted to “1” (step S303). Next, the dimension reduction unit 323 refers to the table, and reduces the top K feature values with a low contribution from all types of feature values before deletion. Further, the process of K = K + 1 is performed (step S304). Then, the dimension reduction unit 323 generates a classifier using the remaining feature amount after deletion, and measures the correct answer rate of the generated classifier (step S305).

  Next, the dimension reduction unit 323 determines whether or not the measured correct answer rate is equal to or greater than a preset threshold value (step S306). The dimension reduction part 323 performs the process after step S304 again, when it is more than a threshold value (step S306: Yes). As a result, the number of feature values to be deleted increases sequentially (with this, the correct answer rate of the classifier also decreases).

  When the measured correct answer rate is less than the preset threshold value (step S306: No), the dimension reduction unit 323 performs the process of K = K-1. This is because the value of K that satisfies the threshold value of the correct answer rate is acquired. Then, the dimension reduction unit 323 reduces the feature amounts up to the Kth in ascending order of contribution (step S307).

  By such processing, the feature amount can be reduced.

(About generation of teacher data)
Deep learning requires a huge amount of teacher data. Here, as described above, the teacher data constituting the first teacher database and the second teacher database are the sample image of the transparent film with defect (sample to be inspected) and the correct answer type of the defect reflected in the sample image. And including. When the first teacher database and the second teacher database are generated (when generating the teacher data), the type of defect is given by a person visually observing the image. However, if this is performed for a large number of teacher data, it takes time and effort. It will take very much. For this reason, a teacher database may be constructed as follows. In the following, the construction of the first teacher database will be described as an example, but the second teacher database can be constructed in the same manner. In the following description, it is assumed that a first teacher database in which the correct answer type is not stored (see the first teacher database (before adding the correct answer type in FIG. 8 in which only sample images are stored)) is prepared in advance. . In addition, it is assumed that information (program or the like) of an algorithm for extracting the feature amount of the sample image is designed in advance by a person and prepared in the storage unit 20. Here, the number of sample images is 10,000. The sample image is an image in which the defect and its surroundings are cut out from a captured image obtained by imaging the entire transparent film with a defect (may be subjected to noise removal, contrast enhancement, or the like).

  As illustrated in FIG. 8, the control unit 30 that performs such processing includes a teacher data generation unit 341. The teacher data generation unit 341 performs teacher data generation processing illustrated in FIG.

  First, the teacher data generation unit 341 randomly selects and acquires 500 sample images from among the 10,000 sample images stored in the storage unit 20 (step S401). The teacher data generation unit 341 displays the selected sample image on the display unit 50 and accepts an input of a correct answer (step S402). The user operates the operation unit 40 to input the type shown in the image displayed on the display unit 50 as the correct answer type. The teacher data generation unit 341 adds the input correct answer type to the first teacher database in association with the sample image that is the target of the correct answer type. Thereafter, the teacher data generation unit 341 generates the decision tree group 343 by the random forest method using all the teacher data stored in the first teacher database and to which the correct answer type is added (step S403). Specifically, the teacher data generation unit 341 calculates a feature amount for each sample image using the algorithm information prepared in the storage unit 20, and determines based on the calculated feature amount and the correct answer type. A tree group 343 is generated. In the second and subsequent steps S403, the old decision tree group 343 is replaced with the new decision tree group 343 generated this time.

  Next, the teacher data generation unit 341 randomly selects another predetermined number (for example, 1000) of sample images from the 10,000 sample images (step S404). Note that step S404 is repeated until the correct answer type is associated with all 10,000 sample images. For this reason, in the last step S404, all remaining sample images (for example, 500 sample images) not associated with the correct answer type are selected. The teacher data generation unit 341 classifies the sample image selected in step S404 using the decision tree group 343 (step S405). Specifically, the teacher data generation unit 341 calculates a feature amount for each of the selected sample images (refer to the above for the calculation method), inputs the calculated feature amount to the decision tree group 343, and inputs each of the sample images. The type (type of defect) is classified.

  After that, the teacher data generation unit 341 displays a list of the sample images to be classified this time and the classification results using the decision tree group 343 on the display unit 50 for each sample image. A correct answer is received (step S406). The classification result may be the largest classification result as a result of the majority of the classification results by each decision tree of the decision tree group 343. In addition, you may make it easy to see a screen by sorting and displaying those with the same classification result. The correct answer and incorrect answer of the classification are determined by visual observation of the user, and the user corrects the type only for the incorrect answer (performed via the operation unit 40). For example, as shown in FIG. 10, sample images 51 and classification results 52 are displayed as a set in a matrix, and the user selects an incorrect classification result and inputs a correct answer. In FIG. 10, only part of the sample images and the classification results are denoted by reference numerals. In addition, the teacher data generation unit 341 sets the classification result obtained by the decision tree group 343 or the input type when there is an input (correction) by the user as the correct answer type as a sample image that is the target of the correct answer type. Correspondingly, it is additionally stored in the first teacher database (step S406). Thereafter, the teacher data generation unit 341 determines whether there is an unprocessed sample image (an image that is not associated with the correct answer type) (step S407). If there is an unprocessed sample image (step S407; No), the process of step S403 is performed again. If there is no unprocessed sample image (step S407; Yes), this process is terminated.

  By such a series of processes, the first teacher database (or the second teacher database) can be constructed relatively easily. The generation of the decision tree group by the process of step S403 may be omitted when the decision tree group 343 having a predetermined correct answer rate is obtained. The correct answer rate is obtained from the number of correct answers input (corrected) by the user in step S406. When the number of correct answers input (corrected) by the user is equal to or less than a predetermined number, the subsequent generation of the decision tree group 343 may be omitted. Note that the decision tree group 343 may be changed to a classifier such as another support vector machine (the number of necessary teacher data is smaller than the deep learning of the CNN). Such a method for generating teacher data can be generally used for teacher data including correct answer data.

(Modification)
The present invention is not limited to the above embodiment. Various changes (including deletion of configuration and processing) may be added to the above embodiment. Although a modification is illustrated below, you may combine at least one part of each modification.

(Modification 1)
The dimension amount reduction unit 323 may be omitted. In this case, the decision tree group 327 is generated from all types of feature amounts output from the intermediate layer of the CNN 313, and all types of feature amounts are input to the decision tree group 327 during image inspection.

(Modification 2)
The part for extracting the feature amount is the intermediate layer and the first layer (that is, the first perceptron layer 313G) or the first or second layer (that is, the first to second perceptron layers 313G to 313H) of the CNN 313. (The intermediate layer in the above description may be replaced with these layers). The number of layers used may be determined according to the number of perceptron layers of the identification layer. Outputs from these layers (N (an integer greater than or equal to 1) perceptron layer counting from the intermediate layer side) can also be regarded as feature amounts, and with such a configuration, classification accuracy can be improved.

(Modification 3)
When generating the decision tree group 327 and the like, the teacher data may be increased. For example, the numerical value of a certain feature amount of certain teacher data may be changed within a predetermined range, and the teacher data having the changed numerical value may be used as other teacher data different from the certain teacher data. Thereby, even when only a small amount of teacher data can be prepared, the decision tree group 327 and the like can be generated.

(Modification 4)
Teacher data supply unit 311, CNN 313, feature quantity acquisition unit 321, dimension reduction unit 323, random forest processing unit 325, decision tree group 327, image processing unit 331, analysis unit 333, teacher data generation unit 341, decision tree group 343 Each or a combination of two or more includes one or more processors (including GPU (Graphics Processing Unit), DSP (Digital Signal Processor), etc.), one or more processing circuits that do not depend on the program, and the processor. You may comprise from the combination with the said processing circuit. The storage unit 20, the control unit 30, the operation unit 40, and the display unit 50 may be configured by various computers such as a personal computer. The program may be supplied by various storage media such as a CD-ROM (Compact Disc Read Only Memory), download from the Internet, or the like.

(Modification 5)
The classification in the image inspection may be performed by another classifier such as a support vector machine or a CNN identification layer. Further, in the above embodiment, the defect to be classified is the above-described black point or black scratch for the sake of understanding of the description, but is not limited to these, and there are various types (in fact, not limited to two types). , Many are set). For example, white scratches, white spots, and the like may be included (in this case, the image processing unit 331 specifies a white region in an image obtained by binarizing the image from the imaging unit 10 by labeling processing, etc. Detect defects). Further, as the type of defect to be classified, there may be “others” that do not belong to any of the types prepared in advance. The inspection object is not limited to a transparent film. The inspection object may be a plain object to be inspected having a fine pattern such as a glass substrate or a semiconductor wafer in addition to a film, or a plain pattern. Further, the object to be classified by the image inspection may not be a defect.

(Configuration disclosed in this specification)
Configurations taking the above embodiment and modifications as an example will be described below. Each of the following configurations can be combined at least partially.

(1) A feature amount extraction means (for example, an intermediate layer of CNN 313) that performs a process of performing pooling after performing convolution on an image a plurality of times and extracts a plurality of types of feature amounts from the image;
Feature quantity output from the intermediate layer of the CNN 313 when there is no feature quantity of at least a part of the plurality of types of feature quantities extracted by the feature quantity extraction unit (for example, the remaining feature quantity and the dimension reduction unit 323) Etc.) to a plurality of decision trees, and classifying means (for example, a combination of a dimension reduction unit 323, a decision tree group 327, an analysis unit 333, etc.) for classifying the image based on the outputs of the plurality of decision trees ,
(For example, a computer provided with the control part 30, the control part 30, etc.).

  Note that the feature amount extraction unit may be, for example, one obtained by extracting the intermediate layer from the learned CNN 313, or may be an intermediate layer that is included in the learned CNN 313. The feature amount extraction means includes a unit that further performs a process other than performing the pooling process a plurality of times after performing the convolution. For example, the feature amount extraction unit includes the intermediate layer and the first layer (that is, the first perceptron layer 313G) or the first or second layer (that is, the first to second perceptron layers 313G to 313H) of the CNN 313. ).

  The classification means may include a plurality of decision trees. When inputting feature quantities to a plurality of decision trees, the feature quantities inputted to each decision tree may naturally be different for each decision tree (since they are generated by a random forest). Since the plurality of decision trees include those generated by a random forest, as a whole, inputable feature values (remaining feature values or feature values output from the intermediate layer of CNN 313 when there is no dimension reduction unit 323) Etc.) may not be used. Therefore, “at least some types of feature amounts” may be only feature amounts that are actually used in a plurality of decision trees. For example, when generating the decision tree group 327, the random forest processing unit 325 of the control unit 30 records the feature amount information used in each decision tree in the storage unit 20, and the control unit 30 uses the decision tree. The part of the group 327 that inputs the feature quantity to each decision tree refers to the information and inputs the feature quantity to each decision tree (note that the dimension reduction unit 323 reduces the feature quantity by referring to the information. May be)

  A plurality of decision trees, CNN intermediate layers, and the like are obtained by machine learning, and in particular, their properties and structure can be specified by a machine learning method.

(2) It further comprises a reduction means (for example, a dimension reduction unit 323) that reduces some types of feature quantities from the plurality of types of feature quantities extracted by the feature quantity extraction means
The classification unit includes at least some types of feature amounts (for example, actually used in a plurality of decision trees after the feature amount reduced by the reduction unit among the plurality of types of feature amounts. Only the feature amount may be input to the plurality of decision trees.
You may do it.

(3) The feature quantity extraction means includes first to Nth layers (N is an integer equal to or greater than 1) which is a partial layer of a CNN (Convolutional Neural Network) intermediate layer and the CNN identification layer. And the same function (for example, the intermediate layer and the first to Nth layers (N is an integer of 1 or more)) (including those extracted from CNN).
The plurality of types of feature amounts extracted by the feature amount extraction means are output from the Nth layer,
You may do it.

(4) An image inspection apparatus in which the imaging unit 10 is added to the image classification apparatus, and a program that causes a computer to execute processing executed by the image classification apparatus.

(5) A plurality of first images showing a defect (for example, a sample image or the like, a captured image obtained by capturing a sample having a defect, an image obtained by performing noise removal, contrast enhancement, or the like on the captured image) CNN generating means (for example, control unit 30 or the like) that generates a CNN (Convolutional Neural Network) (for example, CNN 313 or the like) by deep learning based on first teacher data (for example, each teacher data of the first teacher database) including )When,
A plurality of second images (for example, sample images or the like, which are images of defects and images of the samples with defects), or the captured images that are captured in the intermediate layer that constitutes the CNN generated by the CNN generation unit Feature quantity acquisition means (for example, a feature quantity from an intermediate layer of the CNN) that inputs a plurality of types of feature quantities for each of the plurality of second images. The acquisition unit 321 etc. Note that the feature amount acquisition means is the first to Nth (N is an integer of 1 or more) th from the intermediate layer side which is a part of the CNN intermediate layer and the identification layer of the CNN. The feature amount may be obtained from an output from the Nth layer, which is the last layer of the intermediate layer, or when the intermediate layer or the intermediate layer and the first to Nth layers are used here, CNN More than that The layers may be deleted. A),
In the second teacher data (for example, the feature amount and correct answer type of each teacher data in the second teacher database) including at least some types of feature amounts of the plurality of types of feature amounts acquired by the feature amount acquisition unit. A classifier generating means (for example, a random forest processing unit 325) that generates a classifier by machine learning based on it,
(For example, a computer including the control unit 30, the control unit 30, etc.).

  At least some images of the plurality of first images and at least some images of the plurality of second images may be a common image. The classifier may be a support vector machine or the like.

(6) Reduction means for reducing some types of feature quantities from the plurality of types of feature quantities acquired by the feature quantity acquisition means (for example, a dimension reduction unit 323 that reduces the types of feature quantities by statistical processing). In addition,
The classifier generating unit generates the classifier based on second teacher data including the remaining feature amount after being reduced by the reduction unit among the plurality of types of feature amounts.
You may do it.

(7) The first teacher data includes the plurality of first images, and the correct answer types of defects (for example, correct answer types of defects that are correct data) reflected in each of the plurality of first images. Including
First presenting means for presenting a part of the first images to the user (for example, a teacher data generating unit 341 for displaying a sample image on the display unit 50);
Correct answer accepting means (for example, teacher data for accepting an operation from the user via the operation unit 40) that accepts a correct answer of a defect type reflected in each of the first images presented to the user by the first presenting means. Generating unit 341),
Second classifier generating means (for example, random forest) that generates a second classifier (for example, decision tree group 343) based on the correct answer received by the correct answer receiving means and the part of the first image. Teacher data generation unit 341 for generating a decision tree group by the
Classifying means (for example, a teacher for classifying an image using the decision tree group 343) that classifies the type of defect reflected in another first image of the plurality of first images by using the second classifier. Data generation unit 341),
A second presentation unit (for example, a teacher data generation unit 341 that displays the screen of FIG. 10 on the display unit 50) that presents the classification result classified by the classification unit to the user;
Correction accepting means (for example, a teacher data generation unit 341 that accepts a correction operation from the user via the operation unit 40) that accepts a correction operation by the user to correct the classification result error;
Setting means for setting the classification result corrected by the correction operation received by the correction receiving means and the classification result of the correct answer classified by the classification means as the correct answer type included in the first teacher data ( For example, a teacher data generation unit 341) that performs processing such as step S406,
May be further provided.

(8) A program for causing a computer to execute processing executed by the classifier generation device. In addition, the classifier generating device and the classifier generated by the classifier generating device, an image obtained by imaging an inspection object and an image of a defect (a captured image obtained by imaging the inspection object, An image inspection apparatus that classifies an image obtained by performing processing such as noise removal and contrast enhancement on the captured image) using the classifier.

(9) In addition to the above-described configuration, each of the plurality of first images (for example, sample images) and the correct type of each of the plurality of first images (for example, the correct type of defects reflected in the sample image) , Are apparatuses for constructing teacher databases associated with each other (for example, a computer including the control unit 30, the control unit 30, etc.),
First presenting means for presenting a part of the first images to the user (for example, a teacher data generating unit 341 for displaying a sample image on the display unit 50);
Correct answer accepting means (for example, a teacher data generating part 341 that accepts an operation from the user via the operation unit 40) that accepts the correct answer of the type of the first image presented to the user by the first presenting means. When,
Second classifier generating means (for example, random forest) that generates a second classifier (for example, decision tree group 343) based on the correct answer received by the correct answer receiving means and the part of the first image. Teacher data generation unit 341 for generating a decision tree group by the
Classifying means for classifying the type of another first image among the plurality of first images using the second classifier (for example, a teacher data generation unit 341 for classifying an image using the decision tree group 343) etc,
A second presentation unit (for example, a teacher data generation unit 341 that displays the screen of FIG. 10 on the display unit 50) that presents the classification result classified by the classification unit to the user;
Correction accepting means (for example, a teacher data generation unit 341 that accepts a correction operation from the user via the operation unit 40) that accepts a correction operation by the user to correct the classification result error;
Setting means for setting the classification result corrected by the correction operation received by the correction receiving means and the classification result of the correct answer classified by the classification means as the correct answer type included in the first teacher data ( For example, a teacher data generation unit 341) that performs processing such as step S406,
A teacher database construction device comprising:
Alternatively, a program for causing a computer to execute the above various means, or a method for generating a teacher database by a method similar to the process executed by the above means.

  With the configuration described above, the teacher database becomes easier than when all correct answer types are entered by a person.

DESCRIPTION OF SYMBOLS 10 Imaging part 20 Storage part 30 Control part 40 Operation part 50 Display part 100 Image inspection apparatus 311 Teacher data supply part 313 CNN
321 feature quantity acquisition unit 323 dimension reduction unit 325 random forest processing unit 327 decision tree group 331 image processing unit 333 analysis unit 341 teacher data generation unit 343 decision tree group

Claims (5)

CNN generation means for generating a CNN (Convolutional Neural Network) by deep learning based on first teacher data including a plurality of first images in which defects are shown;
A feature amount for inputting each of a plurality of second images showing a defect to the intermediate layer constituting the CNN generated by the CNN generation means, and acquiring a plurality of types of feature amounts for each of the plurality of second images. Acquisition means;
A classifier generating means for generating a classifier by machine learning based on second teacher data including at least some types of feature quantities of the plurality of types of feature quantities acquired by the feature quantity acquisition means;
A classifier generating apparatus comprising: A reduction means for reducing some types of feature quantities from the plurality of types of feature quantities acquired by the feature quantity acquisition means;
The classifier generating unit generates the classifier based on second teacher data including the remaining feature amount after being reduced by the reduction unit among the plurality of types of feature amounts.
The classifier generation device according to claim 1. The first teacher data includes the plurality of first images and a correct answer type of a defect reflected in each of the plurality of first images.
First presentation means for presenting a part of the first images to the user among the plurality of first images;
Correct answer accepting means for accepting from the user a correct answer of the type of defect reflected in each of the partial first images presented to the user by the first presenting means;
Second classifier generating means for generating a second classifier based on the correct answer received by the correct answer receiving means and the partial first image;
Using the second classifier, classifying means for classifying the type of defect reflected in another first image of the plurality of first images;
Second presentation means for presenting the classification results classified by the classification means to the user;
Correction accepting means for accepting a correction operation by the user to correct an error in the classification result;
Setting means for setting the classification result corrected by the correction operation received by the correction receiving means and the classification result of the correct answer classified by the classification means as the correct answer type included in the first teacher data; ,
The classifier generation device according to claim 1, further comprising: The classifier generator according to any one of claims 1 to 3,
The classifier generated by the classifier generator, and
Classifying an image obtained by imaging an inspection object and showing a defect by the classifier,
Image inspection device. On the computer,
A CNN generating step of generating a CNN (Convolutional Neural Network) by deep learning based on first teacher data including a plurality of first images in which defects are reflected;
A feature amount for inputting each of a plurality of second images showing defects to the intermediate layer forming the CNN generated by the CNN generation step, and acquiring a plurality of types of feature amounts for each of the plurality of second images. An acquisition step;
A classifier generation step of generating a classifier by machine learning based on second teacher data including at least some types of feature amounts of the plurality of types of feature amounts acquired by the feature amount acquisition step;
A program that executes

Images