JP2017040522A - Teaching support method and image classification method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a support for enabling a user to perform efficient teaching operation that the user performs with a reduced user burden.SOLUTION: A two-dimensional map space is set as a square on an XY plane. At two diagonal vertices of the square, a reference vector Rmin as a zero vector and a reference vector Rmax such that values of normalized elements are all 1 are arranged, and at the other two vertices, reference vectors R1, R2 representing two typical examples corresponding to two classification categories that a user specifies are arranged. A poisson equation is solved within the map space based upon them as boundary conditions so as to determine initial values of reference vectors at respective nodes indicated by grating points. The respective nodes which are thus initialized are learnt with a tutor image so as to generate a self-organization map, and similarities between the typical examples and an unlearnt tutor image are visualized.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an image classification technique to which an image processing technique is applied, and more particularly to a technique for supporting a user's teaching work of assigning a classification category to a plurality of images.

  As a technique for automatically classifying a plurality of images into a plurality of categories according to the degree of similarity between images, there is a technique using supervised learning. This technology estimates the feature value distribution of each classification category from the feature value distribution (sample distribution) of the teacher image collected in advance, and determines the classifier (discriminator) by finding the boundary between categories that minimizes misclassification. Function). As typical examples, a linear discrimination method, a neural network, a support vector machine (SVM), and the like are known.

  In this technique, it is necessary to improve the classification accuracy that a classification category in which the image is to be classified is appropriately given to the teacher image. However, the assignment of the classification category to the teacher image is performed by a teaching work based on the visual judgment of the user (expert), the work burden is extremely large, and a certain amount of erroneous teaching is unavoidable.

  For example, Patent Documents 1 and 2 disclose techniques for suppressing a decrease in classification accuracy caused by erroneous teaching. In these techniques, the variation in the feature amount of the image assigned to each classification category by the teaching work is evaluated for each classification category, and the validity of the teaching result is determined.

JP2013-167545A Japanese Patent Application Laid-Open No. 2014-070944

  In the above prior art, even though an image that may have been inappropriately taught can be extracted, there is no clue as to how to correct it. For example, information such as how far the image is different from other images with the same classification category or how similar the image is with other classification categories is not presented to the user. . In addition, there is no difference from normal supervised learning in that teaching is necessary for all images first. For this reason, it does not necessarily reduce the burden on the user who performs the teaching work.

  As a classification method that does not depend on the user's subjectivity, a method using unsupervised learning is also conceivable. However, since the results vary depending on how the initial conditions are given, the purpose of classifying images based on their appearance is that the user must eventually give appropriate initial conditions based on experience, and the learning algorithm It does not meet such demands, nor does it provide necessary information.

  For these reasons, when a user performs a teaching operation for assigning a classification category to a plurality of images, for example, by computer processing, a support for reducing the user burden and enabling an efficient teaching operation is provided. Although desirable, no such technology has been established.

  This invention is made in view of the said subject, and it aims at providing the support for reducing a user burden with respect to the teaching work which a user performs, and enabling efficient teaching work.

  The teaching support method according to the present invention provides a classification category for M (M is an integer greater than N) images including N typical images corresponding to each of N (N is an integer of 2 or more) types of classification categories. An initialization process in which the computer initializes a reference vector of each node in a map space having a dimension corresponding to the number of classification categories; A learning step of classifying each of the images into nodes having a reference vector closest to the feature vector of the image in the map space, and learning each node by updating the reference vector of each node based on the classification result; An output step of visualizing and presenting the distribution of the image in the map space obtained in the learning step;

  Each of these steps is described in T.W. It is similar to the self-organizing map (SOM) processing algorithm proposed by Kohonen. In SOM, a multi-dimensional input vector can be mapped and visualized in an arbitrary dimensional map space, and a similar input vector can be obtained by unsupervised learning in which a reference vector indicating each node in the map space is learned by the input vector. Are arranged close to each other on the map space. When the input vector is a feature vector representing an image, images having similar features are arranged close to each other in the map space to form a cluster. By utilizing this, it is possible to visualize and present a plurality of images according to the degree of similarity between them.

  In the conventional SOM that does not use teacher data in principle, the initial value of the reference vector of each node in the map space is randomly generated. For this reason, even in the same image set composed of a plurality of images, the position of the cluster formed on the map space depends on the initial value. Such a property is not suitable for the purpose of supporting the teaching work.

  Therefore, in the initialization process of the present invention, in order to achieve the above object, a hypercube having (N + 2) or more vertices is used as the map space, and among the vertices, a plurality of feature vectors are formed at the first vertex. A feature vector whose feature value is equal to or smaller than the minimum value among the corresponding feature values in the feature vectors representing the M images, the second vertex farthest from the first vertex In addition, a feature vector in which each of a plurality of feature value values constituting the feature vector is equal to or greater than a maximum value among the corresponding feature value values in the feature vector representing each of the M images. A feature vector corresponding to each of the typical images is arranged as a reference vector of a node corresponding to the vertex at a vertex other than the second vertex, and a reference vector of each node other than the vertex is set. The initial value of Le is given as a solution of Poisson's equation for a reference vector boundary conditions of the respective vertices.

  In the invention thus configured, the vector field formed by the initial value of each reference vector uniquely forms a smooth hypersurface in the map space as if it were a potential field in a space to which a charge distribution is given. Reference vectors corresponding to N typical images given in advance are arranged at the vertices of a hypercube forming a map space, and a sufficient distance is secured. In addition, a reference vector in which all the feature values constituting the feature vector have a smaller value than each image and a reference vector in which all the feature values have a larger value than each image are the farthest vertices in the map space. By being arranged at (first vertex and second vertex), the feature vectors of each image are distributed over a wide range in the map space in the direction along the axis connecting the first vertex and the second vertex. It will be. That is, the entire map space can be used effectively to show the image distribution with a wide dynamic range.

  In the map space formed in this way, images having similar characteristics are gathered to form a cluster. Therefore, images having characteristics similar to the typical image are collected around the N typical images arranged at each vertex. Even if the images are not similar to any of the typical images, images similar to each other may gather together in the map space to form a cluster.

  SOM is a technique suitable for visualizing and expressing a large number of data represented as vectors in a multidimensional feature space in a lower dimensional map space, and various specific expression methods have been proposed. Also in the present invention, the teaching work by the user can be supported by visualizing and presenting to the user the distribution of the images classified as described above using an appropriate one of such expression methods.

  Specifically, in the present invention, if at least one typical image is specified for each of the N classification categories from M (> N) images, which of the remaining images is compared with the typical image. The degree of similarity is automatically determined and the result is visualized. Therefore, the user simply designates (teaches) typical images corresponding to several classification categories from a plurality of images, and how many images are similar to the typical images or clusters at positions different from the typical images. It is possible to obtain useful information for continuing the teaching work, such as whether there is an image forming the image. That is, the user is shown the suggestion of the classification category to be assigned to the untaught image and the suitability of the classification category assigned to the taught image.

  For example, when there are many other images in the vicinity of a typical image, the user can teach them to the classification category corresponding to the typical image at once. If there are few images distributed in the vicinity of a certain typical image, the classification category can be abolished or the typical image can be selected again. Further, if a cluster is formed at a position different from the vicinity of any typical image, a classification category corresponding to the cluster can be newly created.

  As described above, according to the teaching support method of the present invention, when at least one typical image is specified for several classification categories, the similarity of other images to the typical image is visualized and presented. . Therefore, it is possible to provide effective support for the user to efficiently perform the teaching work with a small work load.

  Further, the image classification method according to the present invention presents the distribution in the map space of the plurality of images by the acquisition step of acquiring a plurality of images and the teaching support method described above, and classifies each of the plurality of images with a classification category. And a classification step of classifying images using a classifier that has learned the plurality of images and the taught classification category as teacher data.

  In the invention configured as described above, since the burden on the user in the teaching work as a preliminary stage for classifying the images is reduced, it is possible to reduce the processing cost of the image classification. Moreover, since the reliability of the teaching result can be increased by providing appropriate support for the teaching work, it is possible to improve the accuracy of image classification.

  According to the present invention, information related to a classification category to be assigned to an untaught image, information related to the suitability of a classification category for a taught image, and the like are presented to the user, and the user performs a teaching operation performed by the user. It is possible to provide support for reducing the burden and enabling efficient teaching work.

1 is a diagram showing a schematic configuration of an inspection system to which the present invention can be preferably applied. It is a flowchart which shows operation | movement of a defect detection apparatus. It is a flowchart which shows operation | movement of a defect classification device. It is a flowchart which shows the prior learning process performed by the defect classification device. It is a figure explaining the initial value of a reference vector. It is a figure which shows the example of a display output of a classification result. It is a figure which shows the example of a display output of a classification result. It is a figure which shows the example of a display output in a three-dimensional map space.

  FIG. 1 is a diagram showing a schematic configuration of an inspection system to which the present invention can be preferably applied. This inspection system 1 is an inspection system that performs inspection of defects such as pinholes and foreign matters appearing on the appearance of a semiconductor substrate S to be inspected, and automatically classifies detected defects. The inspection system 1 includes a defect detection device 2 that detects a defect from an inspection target region on the substrate S, a defect classification device 3 that analyzes the defect detected by the defect detection device 2 in more detail and identifies the type of defect, and It has. The defect detection device 2 is incorporated in the production line of the substrate S, and the inspection system 1 is a so-called in-line system.

  The defect detection apparatus 2 includes an image acquisition unit 20 that acquires an image of an inspection target area on the substrate S, and a detection unit 21 that analyzes the image data from the image acquisition unit 20 and detects a defect. The image acquisition unit 20 has an imaging element such as a CCD sensor or a CMOS sensor, for example, and acquires an image data by acquiring an image of an inspection target area on the substrate S, a stage 202 that holds the substrate S, and A stage driving unit 203 that moves the stage 202 relative to the imaging unit 201 is provided. The stage driving unit 203 includes a ball screw, a guide rail, and a motor. The imaging unit 201 and the stage driving unit 203 are controlled by a mechanism control unit 211 provided in the detection unit 21, and operate according to a control command from the mechanism control unit 211, so that an inspection target region on the substrate S is changed. Take an image.

  The defect detection unit 212 performs appropriate data processing on the image data obtained by imaging by the imaging unit 201 to detect defects. The defect detection unit 212 performs appropriate data processing such as noise removal processing or smoothing processing on the image data given from the imaging unit 201, and performs defect detection based on the image data thus processed. More specifically, when an image captured on the actual substrate S is compared with an image of a defect-free substrate stored in advance in the storage unit 213, and the difference exceeds a predetermined reference It is determined that the image includes a defect. The defect detection unit 212 transmits the detected defect image data to the defect classification device 3 via the interface (I / F) 214.

  The defect classification device 3 determines the type of the detected defect based on the defect data given from the defect detection device 2. That is, the defect classification device 3 has a function of automatically classifying defects (ADC: Automatic Defect Classification) into classification categories to which the defect belongs based on the defect image data given from the defect detection device 2. The defect classification apparatus 3 is an apparatus that implements an embodiment of the present invention. As described below, the function as the defect classification device 3 can be realized by executing a control program describing each process according to the present invention using hardware resources of a general personal computer. Is possible.

  The defect classification device 3 implements each functional block shown in FIG. 1 by dedicated hardware and software by executing a control program read in advance. The defect classification device 3 includes an interface (I / F) 31 for receiving defect image data from the defect detection device 2, and a defect classification unit (ADC) 32 that determines the type of the defect by automatic classification based on the defect image data. It has. The interface 31 can communicate with other external devices via, for example, a LAN (Local Area Network).

  The image data given from the defect detection device 2 and various processing data generated in the process in the defect classification unit 32 are stored in the storage unit 33 as needed. The defect classifying unit 32 classifies the defect image into a plurality of classifications set in advance corresponding to the defect type based on the values of the plurality of types of feature amounts calculated by the feature amount calculating unit 34 based on the given defect image data. Sort into one of the categories.

  Furthermore, the defect classification device 3 includes an input receiving unit 35 such as a keyboard and a mouse that receives an operation input from a user, and a display unit 36 that displays visual information for the user such as an operation procedure and a processing result.

  The defect classification unit 32 performs, in software, a process of classifying the detected defects using an appropriate learning algorithm such as SVM (Support Vector Machine), neural network, decision tree, discriminant analysis, and the like. Specifically, appropriate learning is performed in advance based on defect image data given from the defect inspection apparatus 2 and stored and stored in the storage unit 33, and teaching information from a user regarding a classification category in which the defect image should be classified. Perform machine learning using an algorithm. When new image data corresponding to an unclassified defect image is given, automatic classification based on the learning result is performed to classify the new defect image into any one of the classification categories.

  FIG. 2 is a flowchart showing the operation of the defect detection apparatus. The defect detection apparatus 2 images the inspection target substrate S, and executes the following defect detection processing as an operation for detecting defects from the imaging result. When the inspection target substrate S is first loaded on the stage 202 of the image acquisition unit 20 (step S101), the mechanism control unit 211 controls the imaging unit 201 and the stage driving unit 203 to perform a predetermined inspection target on the substrate S. The position is imaged and an image is acquired (step S202). For the image data obtained in this way, the defect detection unit 212 performs processes such as brightness adjustment and noise removal as necessary. Further, the image data corresponding to the inspection target position is read out from the image data of the defect-free substrate stored in the storage unit 213 in advance, and the difference in pixel units from the image data obtained by imaging is calculated. (Step S103).

  If the captured image does not include a defect, the difference from the previously stored defect-free image is small, but if a defect is included, a large difference appears. By utilizing this fact, it is possible to determine the presence or absence of a defect by obtaining the difference between the two. Specifically, appropriate smoothing processing and binarization processing are performed on the obtained difference image (step S104), and if there is a region where a significant difference is recognized, that portion is extracted as a defective portion ( Step S105). For an image in which a defect is detected (step S106), the image is transmitted as a defect image to the defect classification device 3 via the interface 214 (step S107). By performing the above processing on the entire substrate S while sequentially changing the inspection target position (step S108), the inspection of the substrate S is completed.

  The defect detection device 2 inspects each position of the inspection target substrate S to determine whether there is a defect. On the other hand, the defect classification device 3 determines the type of detected defect, analyzes the tendency of occurrence of the defect, and feeds back the result to the production line, or adjusts the processing content of the defect detection processing by the defect detection device 2 Used to do By analyzing the detected defect by the defect classification device 3, the defect detection device 2 can execute a process specialized for defect detection, so that a defect can be detected at high speed.

  FIG. 3 is a flowchart showing the operation of the defect classification apparatus. As described above, the defect classification device 3 classifies the defect image output from the defect detection device 2 into a classification category corresponding to the defect type. More specifically, when the defect classification unit 32 performs pre-learning (described later) based on the defect image data collected in advance (step S201), an unclassified defect image is newly given from the defect detection device 2. (Step S202), the feature quantity calculation unit 34 calculates a plurality of feature quantity values that characterize the defect image (Step S203). Then, based on the calculated feature value, the defect classification unit 32 classifies the defect image into one of a plurality of classification categories using a classifier configured by prior learning (step S204).

  A plurality of types of classification categories are set in advance corresponding to a plurality of types of defects that may occur in the substrate S. For example, foreign materials, scratches, bubbles, pinholes, pattern disconnections, short circuits, and the like are representative. However, it is not limited to these.

  FIG. 4 is a flowchart showing a pre-learning process executed by the defect classification apparatus. The pre-learning process is performed based on a teacher image obtained by imaging various defects and teaching information indicating which classification category the teacher image should be classified, which is taught by the user. The more teacher images that show the typical appearance of the defect, the better the learning effect and the higher the classification accuracy. On the other hand, teaching work for assigning a classification category to each of a large number of teacher images cannot be processed automatically as long as the device is unlearned, and the user must rely on visual judgment of an experienced user. . When there are many teacher images, not only does the amount of work for the user increase, but it is also required to teach each teacher image consistently in order to suppress misclassification caused by variations in teaching. Therefore, the work burden on the user is significantly increased.

  Therefore, the pre-learning process of this embodiment is configured such that various support functions are provided to the user in order to reduce the burden on the user and enable efficient teaching work by the user. Specifically, for a user, one type of typical example (typical image) in which the characteristics of the classification category appear strongly for each of the classification categories is identified from a large number of teacher images. The work of selecting one by one is required. When these are designated by the user, the defect classification device 3 determines the degree of similarity between the images based on the given information for the other teacher images, visualizes the result, and presents the result to the user. The user can refer to the presented information and review the teaching input for other images and the setting of the classification category, which is far less than teaching work without any clues. Teaching work can be performed efficiently with work load. The processing content for this will be described below.

  First, a plurality of defect images used as teacher images are collected (step S301). Here, the defect images output from the defect detection device 2 are collected by sequentially storing them as teacher images in the storage unit 33. For example, the defect images are collected in an image library prepared in advance or in an external storage. It may be an aspect in which teacher image data is acquired. Of course, more defect images are required than the number of categories to be classified.

  Next, the user is required to designate the type of classification category for classifying these teacher images and one typical example for each classification category by displaying an appropriate message on the display unit 36. The designation input from the user is received by the input receiving unit 35 (step S302). The designation of the classification category and typical example can be changed after the fact, and a strict judgment is not required from the user at this point. Of course, the subsequent processing becomes more efficient when accurate designation is made early.

  Subsequently, a map dimension for visualizing and displaying the degree of similarity of each teacher image is set (step S303). As will be described later, in this embodiment, how similar another teacher image is to the teacher image designated as a typical example is displayed in various display modes. For example, the distribution of the teacher image can be expressed as a scatter diagram in a map space of an arbitrary dimension, and the dimension can be selected. As for the setting of this dimension, user designation may be accepted, or it may be automatically set according to the number of types of the designated classification category.

  In principle, for example, a hypercube having an arbitrary dimension can be used as the map space. Among these, the map space that can be visualized in such a manner that the user can intuitively understand the image distribution is a square in a two-dimensional space, a cube in a three-dimensional space, or the like. In addition, it is possible to express a map in a higher dimension by using a plurality of squares or cubes.

  Next, a plurality of types of feature amounts representing the features of each teacher image are calculated by the feature amount calculation unit 34 (step S304). Each teacher image has its features represented numerically by a feature vector having a plurality of feature amounts as elements. For the purpose of easy visualization in the map space, the feature amount of each image may be expressed in a form normalized in a predetermined numerical range, for example, 0 to 1. Hereinafter, it is assumed that the feature vector of each teacher image is represented by the feature amount normalized in this way.

  Note that the execution order of step S302 to step S304 is not limited to the above, and the same result can be obtained by executing in any order.

  As a technique for mapping and visualizing an image distribution in a multidimensional feature space to a lower dimensional map space, T.K. The self-organizing map (SOM) proposed by Kohonen (Kohonen) is famous. In SOM, a node performs learning by adjusting a reference vector of each node constituting a map space according to a feature vector of a teacher image that is an input vector. Then, an unknown image is mapped based on the learned reference vector of the node. Specifically, the position of the image in the map space is determined by arranging the image at a node having a reference vector close to the feature vector of the unknown image.

  Also in this embodiment, by performing mapping using the basic principle of SOM, the similarity between images is expressed by arranging images similar to each other at close positions in the map space. However, the method of setting the initial value of the reference vector of each node is different from the known SOM, and typical information specified by the user is reflected in the mapping. For this purpose, an initial value of a reference vector determined based on typical information specified by the user is arranged for a node corresponding to each vertex of the map space set as a hypercube (step S305).

  FIG. 5 is a diagram for explaining the initial value of the reference vector. More specifically, FIG. 5 (a) shows the arrangement of reference vectors to nodes corresponding to the vertices of the map space, and FIG. 5 (b) shows the arrangement of reference vectors to each node after completion of initialization. . Here, there are two types of classification categories, and one typical example is designated for each category. Each teacher image is mapped to a two-dimensional map space on the XY plane.

  5 (a) and 5 (b), in order to facilitate understanding of the principle, one of a plurality of elements constituting the reference vector R = (r1, r2,..., Ri,. Only one element ri corresponding to the feature quantity is shown. That is, the reference vector W is expressed as if it were a scalar quantity. However, in accordance with the SOM principle, the reference vector R as a vector quantity is actually arranged in the map space (in this case, the XY plane).

  In this embodiment, as shown in FIG. 5A, among the vertices of the map space, corresponding to the two vertices that are located farthest in the map space (in this example, located at the diagonal of the square). The node includes a reference vector Rmin in which the value of each element ri (i = 1, 2,...) Is less than or equal to the minimum value of the element ri of the feature vectors of all teacher images, and each element ri (i = 1, 1). 2,...) Are respectively arranged with reference vectors Rmax whose value is equal to or greater than the maximum value of the element ri of the feature vectors of all the teacher images.

  For example, a zero vector (0, 0,..., 0) can be used as the reference vector Rmin. Further, as the reference vector Rmax, for example, a vector (1, 1,..., 1) having the normalized maximum feature value 1 as the value of each element can be used.

  The nodes corresponding to the other two vertices of the square map space include reference vectors R1 equal to the two typical feature vectors corresponding to the two classification categories (eg, “bubble” and “foreign matter”) specified by the user. , R2 are arranged respectively. In order to enable such an arrangement, the map space needs to be a hypercube having (N + 2) or more vertices with respect to the number N of classification categories (N is an integer of 2 or more). In other words, if the number N of types of classification categories is determined, the dimension of the map space can be determined.

  As described above, when the classification category number N is 2, the map space can be two or more dimensions, for example, an appropriate shape on a two-dimensional plane, for example, a square. When the classification category number N is 3 to 6, the map space can be three-dimensional or more, for example, a cube. For example, when the classification category number N is 7 to 14, the map space can be a four-dimensional or higher hypercube.

  For N typical examples, when the number of vertices in the map space is greater than (N + 2), there will be vertices where no typical examples are placed. An appropriate reference vector different from the reference vectors Rmin, Rmax and the feature vector of the typical example, for example, a reference vector equal to an average vector obtained from the feature vectors of all the teacher images can be arranged at such vertices. .

  In this way, the reference vectors of other nodes in the map space are initialized from the reference vectors R1, R2, Rmin, Rmax arranged at the nodes corresponding to the vertices (step S306). That is, the initial value of the reference vector of each node represented as a grid point in FIG. Specifically, the reference vectors arranged as shown in FIG. 5 (a) are treated as if they were electric charges arranged in the space, and a potential field is formed in the map space by placing these reference vectors at each position. Assume that

  It is known that the potential distribution in the space can be obtained as a solution of the Poisson equation having the charge arrangement in the space (or the potential at a specific position) as a boundary condition. In particular, the electrostatic field is described by the Laplace equation, and the potential of each part in the space is represented by the average value of the potential in the vicinity. As a method for calculating the potential distribution numerically, for example, there is a method (Gauss = Seidel method) in which the potential of each part is described by an average value of potentials in the vicinity and this is repeated until there is no change in potential at all positions. Are known. Since such knowledge is well known, derivation of the calculation formula is omitted.

Applying this idea to a reference vector in map space,
(1) A reference vector corresponding to a typical feature vector is arranged at a node corresponding to a vertex of the map space.
(2) Place a zero vector at each node other than the vertex.
(3) Replace the reference vector of each node other than the vertex with the average of the reference vectors of neighboring nodes.
(4) Repeat (3) until there is no change in the reference vectors of all nodes.
The processing procedure is obtained. By obtaining the reference vector of each node in the above procedure, the reference vector is initialized.

  By initializing the reference vector of each node by the above processes (1) to (4), the initial value of the reference vector of each node is determined. At this time, the scalar field (potential field) formed by the element ri of the reference vector of each node on the XY plane is a smooth curved surface with relatively little unevenness in the map space, as shown in FIG. That is, the vector field formed by each reference vector is a smooth hypersurface.

  Learning using other teacher images is performed using the reference vector thus obtained as an initial value in the same way as in SOM. That is, each teacher image other than the typical example is classified into a node having a reference vector closest to the feature vector of the teacher image (step S307). The distance between the feature vector and the reference vector can be evaluated by, for example, the size of the inner product of both vectors, and among these nodes, the node having a reference vector that maximizes the inner product with the feature vector of the teacher image However, it can be said that the teacher image is a node to be classified.

  Then, the reference vector of the node into which the teacher image is classified and the neighboring nodes are updated so as to approach the feature vector of the teacher image classified into the node (step S308). In order to prevent the learning result from being influenced by the selection order of the teacher images, the reference vectors are updated using the feature vectors classified into the respective nodes after the classification of all the teacher images is completed. When a plurality of teacher images are classified into one node, a vector obtained by averaging the feature vectors is used. This is the same as the concept of so-called batch learning type SOM.

  Steps S307 and S308 are repeated until there is no change in a predetermined convergence condition, for example, all reference vectors (step S309). When the convergence condition is satisfied (YES in step S309), learning of each node is finished, and the classification result of the teacher image finally obtained is displayed and output on the display unit 36 as an improved self-organizing map, Presented to the user (step S310).

  6 and 7 are diagrams showing examples of display output of classification results. In the example shown in FIG. 6A, the map space nodes and the number of teacher images classified into the nodes are displayed as a matrix table of 5 rows × 5 columns. That is, in each frame of FIG. 6A, a node number for identifying a node is shown in the upper left, and the number of teacher images classified into the node is shown in the lower right. The frame corresponding to each node is displayed in an arrangement according to the position in the map space. The typical example F1 corresponding to the reference vector R1 is included in the frame of the node number 00 at the upper left, and the typical example F2 corresponding to the reference vector R2 is included in the frame of the node number 44 at the lower right. In the figure, a portion surrounded by a dotted line indicates a portion where many teacher images are gathered to form a cluster.

  A cluster including many teacher images is formed in the nodes including these typical examples F1 and F2 and the nodes in the vicinity thereof, indicating that the specified typical examples are generally appropriate. In particular, the upper left node 00 including the typical example F1 is the center of a cluster in which a large number of teacher images are gathered, indicating that the typical example F1 is suitable as a typical example of this classification category. Further, it can be seen from this result that it is appropriate to assign the same classification category (for example, bubble) to the teacher image classified into the node 00 and a node very close to the node 00.

  The display screen of the table shown in FIG. 6A is a screen that displays a list of images Im classified into the node 00 as shown in FIG. 6B when the user selects the node 00, for example. The window W can be switched. When the selected node is a node including a typical example designated by the user, the typical example Ic and another image Im are displayed side by side on the same screen so that the user can compare them on the screen. It is more preferable.

  If there is no problem in checking the images displayed in the list, the user operates the cursor C to select them all at once and move them to one of the classification category folders (for example, “bubble”) on the left side of the screen. It is possible to teach a plurality of images at once. On the other hand, if an image that is not appropriate to be classified into the same category is included, it can be excluded from selection or taught to another category. By doing in this way, the teaching work by the user becomes extremely efficient as compared with the conventional technique in which each teacher image is visually determined, and the work burden on the user is reduced.

  On the other hand, in FIG. 6A, in the cluster formed in the lower right including the node 44 in which the typical example F2 is classified, the node 44 is slightly off the center of the cluster. Can be read as the center of the cluster. Therefore, it is suggested that a typical example of the classification category (for example, “foreign matter”) is preferably selected from the teacher image F2a classified in the node 33. Also in this case, the teacher image classified into the node 33 can be displayed on the list screen, and a more preferable typical example can be searched again.

  Further, it can be seen from FIG. 6A that another cluster is formed near the upper right corner of the table apart from the clusters including the typical examples F1 and F2. This suggests that there are a considerable number of teacher images including defect types different from the “bubbles” indicated by the upper left cluster and the “foreign matter” indicated by the lower right cluster. Therefore, if it is possible to newly designate a classification category (for example, “scratch”) corresponding to this cluster and a typical example F3 selected from the cluster, such a defect type can be dealt with later. Thus, classification can be performed.

  In addition, all the teacher images are arranged on the map space in various modes such that a small number of teacher images form a smaller cluster or are isolated without belonging to any cluster. In this way, it is clearly shown as a distribution on the map how much the other teacher images are similar to or dissociated from the typical image.

  In the display example shown in FIG. 7A, the classification result of the teacher image is displayed as a scatter diagram on the XY plane. In the display example shown in FIG. 7B, the number of teacher images for each node is shown as a graph. In FIG. 7A, each dot corresponds to one teacher image, and the number of dots represents the number of images, but the image is represented by a circle having a size corresponding to the number of teacher images classified into each node. A distribution may be represented. Also in these display modes, the user can easily read how other teacher images are distributed in the vicinity of the designated typical example or in a position separated from the designated typical example. It is more preferable that these display modes can be appropriately switched according to the purpose of the user.

  FIG. 8 is a diagram showing a display output example in the three-dimensional map space. Even when the dimension of the map space is three-dimensional, the reference vectors Rmin and Rmax are arranged at two vertices corresponding to the diagonal line of the cube in the XYZ space. At the other six vertices, typical examples corresponding to a maximum of six classification categories are respectively arranged.

  A group of teacher images similar to the typical example arranged at the vertex gathers in the vicinity of the vertex in the map space to form a cluster. A large cluster is formed if there are many similar images, and a small cluster is formed if there are few similar images. In addition, an image group similar in features different from the typical example forms a cluster at a position different from the vertex. By displaying such a map, as in the case of a two-dimensional map, the user forms how many images similar to the specified typical example exist, or forms a large cluster that was not specified as a typical example. It is possible to intuitively grasp whether or not there is an image group that does.

  In addition to the vertices where the typical examples are arranged, the reference vectors arranged at the vertices are artificially created and correspond to feature vectors of virtual images that do not necessarily exist. For this reason, the image gathering in the vicinity of the said vertex may decrease.

  In addition, clusters formed respectively so as to include a plurality of typical examples arranged at different vertices may be integrated to make their boundaries unclear. In such a case, for example, the boundary may be made clearer by performing relearning so that the typical examples are arranged at more distant vertices.

  Returning to FIG. 4, in the pre-learning process of this embodiment, it is possible to re-specify a typical example and newly establish a category. That is, when it is necessary to re-specify the classification category or typical example (YES in step S311), the process returns to step S302, and the designation of the classification category and typical example is newly accepted. On the other hand, if re-designation is not necessary, the user can perform a teaching operation of designating and inputting a classification category to be classified for an unteached teacher image while using the classification result displayed as described above as reference information.

  When the teaching input is received for all the teacher images (step S312), the defect classification unit 32 selects an appropriate machine learning algorithm based on the feature amount of each teacher image and the teaching information about the classification category assigned to the teacher image. To configure a classifier (step S313). Thus, the pre-learning process is completed, and the classifier thus configured is applied to the classification of the unknown defect image.

  As described above, in this embodiment, by classifying an unknown defect image using a classifier configured by supervised learning, it is possible to perform defect classification based on the degree of similarity with a typical defect type. . Although teaching work for a large number of teacher images can be a heavy burden on the user, in this embodiment, if the user designates one typical example for each classification category, the degree of similarity with typical examples of other teacher images is increased. It is automatically determined and the result is visualized and presented. Therefore, the burden of the teaching work by the user is greatly reduced, and the teaching work can be performed efficiently. That is, this embodiment can provide extremely effective support for the teaching work performed by the user.

  Although the SOM concept is used as a method for visualizing the determination result, the present embodiment is different from the conventional SOM in setting the initial value of the reference vector in the map space. The conventional SOM algorithm is for enabling unsupervised learning. In the basic SOM method, the reference vector of each node in the map space is initialized with a random number. For this reason, the vector field formed in the map space by the initialized reference vector includes many irregularities in principle.

  When a feature vector of a teacher image is given as an input vector, the teacher image is classified into a node having a reference vector closest to the feature vector. When the reference vectors are initialized at random as described above, teacher images with relatively small feature vector differences may be arranged at positions that are largely separated on the map space. This is particularly noticeable in the early stages of learning. For this reason, although two images having similar characteristics may form different clusters, they are not suitable for classification based on typical examples designated by the user.

  In SOM, a method of orderly placing reference vectors in an orderly manner based on the principal component analysis results of all input data is also used. However, the results obtained thereby are classified based on typical examples specified by human subjectivity. The probability of matching is very low. Thus, in unsupervised learning that does not receive any teaching from the user, it is difficult to provide a classification method that matches the user's sense.

  On the other hand, in this embodiment, at the stage of determining the initial value of the reference vector, a minimum user input, that is, identification of several classification categories and designation of one typical example for each classification category are accepted. . Based on these pieces of information, a reference vector in the map space is initialized.

  Specifically, a node corresponding to the two farthest vertices in the hypercube constituting the map space has a reference vector Rmin in which the value of each element is equal to or less than the feature vector of all the teacher images, and the value of each element is all And a reference vector Rmax that is equal to or greater than the feature vector of the teacher image. By arranging these reference vectors at the two vertices corresponding to the diagonal of the hypercube, the feature vectors of each teacher image are distributed over a wide range in the map space, and the dynamic range of the map space is effectively used. be able to.

  Further, a reference vector corresponding to a typical feature vector specified by the user is arranged at the other vertex of the hypercube. As a result, the typical examples are arranged apart from each other in the map space, so that a sufficient distance is secured between one typical example and the other typical examples, and the boundaries of the clusters to which each belongs are ambiguous. Can be suppressed. In the conventional SOM which does not receive the instruction from the user about the typical example, it is impossible in principle to clearly separate the typical examples on the map and classify them according to the user's sense.

  Then, the initial value of the reference vector of each other node is given as a solution at each node position of the Poisson equation using the reference vector arranged at the node corresponding to each vertex of the map space as a boundary condition. For this reason, the vector field formed by the reference vectors in the initialized map space is a smooth hypersurface with relatively few irregularities. Therefore, the probability that two teacher images having similar feature vectors are divided by the unevenness of the field is small, and they are expected to be arranged at close positions in the map space to form a single cluster.

  When teacher images with similar characteristics form multiple clusters in the map space, the user needs to determine whether they should really be separate clusters or have been accidentally fragmented as a result of initialization . This increases the burden on the user and goes against the purpose of supporting teaching. Such problems can be avoided by placing the reference examples specified by the user on the map and setting a reference vector that forms a smooth connection field between them as the initial value of each node. The

  Then, by learning the node with the reference vector initialized in this way from the teacher image other than the typical example, the teacher images having characteristics similar to the typical example are gathered in the vicinity of the typical example on the map space, but similar to each other. No teacher image appears at a farther position. Thus, if one typical example is designated for each classification category, the relationship with the typical example is automatically visualized on the map space for the other teacher images. The user can perform teaching work with reference to the presented result, and the burden on the user in teaching work for supervised learning is greatly reduced.

  In classification based on unsupervised learning, the classification result does not necessarily conform to human senses, and the user may feel uncomfortable with the classification result. On the other hand, in classification based on supervised learning, the number of classification categories must be determined in advance, or the user must visually determine all the teacher images. In the classification method of this embodiment, these drawbacks are eliminated.

  As described above, in this embodiment, the “teaching support method” according to the present invention is applied to the pre-learning process shown in FIG. 4, and steps S302 to S306 are included in the “initialization process” of the present invention. Steps S307 to S309 correspond to the “learning step” of the present invention. Step S310 corresponds to the “output step” of the present invention. Further, the “image classification method” of the present invention is applied to the defect classification processing shown in FIG. 3, and step S201 of these corresponds to the “acquisition process” and “teaching process” of the present invention. Step S204 corresponds to the “classification step” of the present invention.

  In the above-described embodiment, the computer on which the function as the defect classification device 3 is mounted corresponds to the “computer” of the present invention, and the display unit 36 functions as the “display unit” of the present invention.

  The present invention is not limited to the above-described embodiment, and various modifications other than those described above can be made without departing from the spirit of the present invention. For example, in the above embodiment, a two-dimensional map space is visualized by one square, and a three-dimensional map space is visualized by one cube. However, a multi-dimensional map space may be displayed using a plurality of these.

  Further, for example, the above embodiment is the inline type inspection system 1 installed on the production line of the substrate S. However, the teaching support method and the image classification method according to the present invention are provided independently from the production line. It can also be realized by a system. In addition, as described above, the teaching support method and the image classification method according to the present invention can be realized by causing a personal computer or workstation having a general hardware configuration to execute a control program describing the above processing contents. is there. Therefore, the present invention can be provided as a control program that is additionally incorporated in an existing inspection apparatus.

  For example, in the above-described embodiment, a zero vector is used as the reference vector Rmin, and a vector whose normalized element values are all 1 is used as the reference vector Rmax. However, the present invention is not limited to this. For example, the feature vectors of all the teacher images are compared for each element (feature amount), and a vector having the minimum value (or maximum value) of each element as the value of the element is referred to as the reference vector Rmin (or reference vector Rmax). It may be used as In the above embodiment, each feature amount is normalized, but this is for the convenience of processing and is not an essential requirement.

  Moreover, although the said embodiment is an inspection apparatus which test | inspects the defect of a semiconductor substrate, the application object of this invention test | inspects not only the apparatus which test | inspects a semiconductor substrate but other objects, for example, a printed circuit board, a glass substrate, etc. Or a surface inspection device for inspecting the surface condition of various materials. Furthermore, the image to be classified is not limited to the defect image of the substrate as described above, and may be, for example, an image obtained by imaging a biological sample such as a pathological tissue or a cell, an image for human face recognition, and the like. It is also possible to apply the present invention for the purpose of classifying such images.

  As described above, the learning process of the teaching support method according to the present invention has been described with reference to the specific convergence of the classification into the nodes of the image and the update of the reference vector based on the classification result as described in the specific embodiment. It may be configured to repeat alternately until the condition is satisfied. It is rare that an initial value according to the purpose is obtained by one update of the reference vector, and the classification accuracy can be further improved by repeating the processing so as to converge the learning result.

  Further, for example, the learning step may be configured not to update the reference vector arranged at each vertex of the map space. By doing this, since the reference vector corresponding to the typical image specified by the user is held at the vertex, a cluster of images similar to the typical image can be formed near the vertex of the map space. Can be clearly separated.

  Further, in the initialization step, when the vertex of the map space is larger than (N + 2), feature vectors corresponding to each of N typical images are used as reference vectors for N vertices other than the first and second vertices. The reference vectors corresponding to the average of the feature vectors of the M images may be arranged at the remaining vertices. In this way, by placing an average reference vector at vertices other than the vertices where the typical example is arranged, it is possible to suppress a teacher image having characteristics similar to the typical example from being drawn to other vertices. Images similar to the example can be effectively collected around the typical example.

  Further, for example, a square or a cube may be used as a map space, and in the output step, a scatter diagram representing an image distribution in the map space may be displayed and output on the display means. In such a display mode, it becomes obvious at a glance how much each teacher image is similar to the typical example or how many other teacher images are similar to each typical example.

  Further, for example, in the output step, one typical image and at least one other image in the vicinity of the typical image may be displayed and output on the same screen of the display means. According to such a display mode, the user can evaluate another image determined to be similar to the typical image by comparing with the typical image.

  Further, in the image classification method according to the present invention, prior to the teaching step, a typical image designation input may be received from the images acquired in the acquisition step. With such a configuration, it is possible to collect various images regardless of whether or not they are typical images in the acquisition process, and by using many collected images as teacher images, classification with excellent classification performance is possible. Can be configured.

  The present invention can be suitably applied to a technical field in which images having ambiguity and variation are classified based on their characteristics, such as substrate defect inspection, biological sample analysis, and face recognition.

DESCRIPTION OF SYMBOLS 1 Inspection system 2 Defect detection apparatus 3 Defect classification apparatus (computer)
32 Defect classification unit 34 Feature amount calculation unit 35 Input reception unit 36 Display unit (display means)
201 Imaging unit S302 to S306 Initialization process S307 to S309 Learning process S310 Output process S201 Acquisition process, teaching process S204 Classification process

Claims (8)

The computer supports teaching work to assign classification categories to M (M is an integer larger than N) images including N typical images corresponding to each of N (N is an integer of 2 or more) types of classification categories. A teaching support method, wherein the computer includes:
An initialization step of initializing a reference vector of each node in a map space having a dimension according to the number of classification categories;
A learning step of classifying each of the images into nodes having a reference vector closest to the feature vector of the image in the map space, and learning each node by updating the reference vector of each node based on the classification result;
Performing an output step of visualizing and presenting the distribution of the image in the map space obtained in the learning step, and in the initialization step,
A hypercube having (N + 2) or more vertices is defined as the map space, and among the vertices,
At the first vertex, a feature vector in which each of the plurality of feature value values constituting the feature vector is equal to or less than a minimum value among the corresponding feature value values in the feature vectors representing the M images,
At the second vertex farthest from the first vertex, each of a plurality of feature amount values constituting the feature vector is the maximum of the corresponding feature amount values in the feature vector representing each of the M images. Feature vectors that are greater than or equal to the value
A feature vector corresponding to each of the typical images at vertices other than the first and second vertices,
Place each as a reference vector of the node corresponding to the vertex,
A teaching support method in which an initial value of a reference vector of each node other than the vertex is given as a solution of a Poisson equation using the reference vector of each vertex as a boundary condition.   The teaching support method according to claim 1, wherein in the learning step, classification of the image into the node and update of a reference vector based on a classification result are alternately repeated until a predetermined convergence condition is satisfied.   The teaching support method according to claim 1, wherein, in the learning step, a reference vector arranged at each vertex of the map space is not updated.   In the initialization step, when the vertex of the map space is larger than (N + 2), refer to feature vectors corresponding to each of the N typical images to N vertices other than the first and second vertices. 4. The teaching support method according to claim 1, wherein the teaching support method is arranged as a vector, and a reference vector corresponding to an average of feature vectors of the M images is arranged at the remaining vertices.   The teaching support method according to claim 1, wherein a square or a cube is used as the map space, and in the output step, a scatter diagram representing the distribution of the image in the map space is displayed and output on a display unit.   6. The teaching support according to claim 1, wherein in the output step, the one typical image and at least one other image in the vicinity of the typical image are displayed and output on the same screen of the display means. Method. An acquisition step of acquiring a plurality of images;
A teaching step of presenting a distribution of the plurality of images in the map space by the teaching support method according to any one of claims 1 to 6, and receiving a classification category teaching input for each of the plurality of images;
A classification step of classifying images using a classifier that has learned the plurality of images and the taught classification category as teacher data.   The image classification method according to claim 7, wherein a designation input of the typical image is received from images acquired in the acquisition step prior to the teaching step.

Images