JP2022134742A - Information processing device, information processing method, and program - Google Patents

Abstract

To make it possible to efficiently confirm whether or not learning data to be a base of a sorter is appropriate.SOLUTION: An information processing device can access a storage part which stores a plurality of feature values in association with learning images and storing the plurality of feature values in association with inspection images. The information processing device includes: a display part; a reference feature determination part for determining one or more reference features; a reference feature value range determination part for determining a reference feature value range in a space defined by a reference feature value; an image extraction part for extracting learning images and/or inspection images in which the reference feature values are included in the reference feature value range, and a display control part for displaying extraction learning images and/or extraction inspection images extracted by the image extraction part in the display part.SELECTED DRAWING: Figure 10

Description

The present invention relates to an information processing device, an information processing method, and a program.

2. Description of the Related Art In the manufacturing industry and the like, products are sometimes inspected by analyzing images obtained by imaging the products. In such an inspection, first, a classifier composed of a neural network or the like is caused to perform machine learning using learning images prepared by imaging in advance, thereby optimizing the weighting for each feature amount of the learning images. . By inputting a desired inspection image to the machine-learned classifier, an attribute indicating the quality of the inspection image is output.

A classifier generated by machine learning may make an erroneous decision due to various causes. Misjudgment may be caused by, for example, inappropriate learning data, or inappropriate algorithm although learning data is appropriate. Regarding this point, for example, in Patent Document 1, in order to enable the operator to check the validity of the teacher data, adjust the teacher database and re-learn the classifier, plot the feature amount of the teacher image and , and a plot of feature values of an analysis image are displayed on an input/output device.

JP 2020-160543 A

The operator may check the output result of the classifier for each inspection image, for example, in order to confirm whether the accuracy of the classifier is adequate. Furthermore, when the accuracy of the classifier is found to be inappropriate, it is necessary to check whether the learning data is appropriate. It is necessary to confirm whether the label is appropriate. The greater the number of learning images and inspection images, the greater the burden of these checking operations.

Therefore, in one aspect, the present invention has been made in view of such circumstances, and an object of the present invention is to efficiently confirm whether or not learning data that is the basis of a classifier is appropriate. It is to provide the technology that makes it possible.

The present invention adopts the following configurations in order to solve the above-described problems.

In other words, an information processing apparatus according to one aspect of the present disclosure stores a plurality of feature amounts indicating each of the plurality of features of a learning image in association with a learning image used for learning a classifier that determines attributes of an image. and an information processing device capable of accessing a storage unit that stores a plurality of feature quantities representing respective features of the inspection image in association with an inspection image to be classified by the classifier, the information processing device comprising: a display unit; , a range in a space defined by a reference feature determination unit that determines at least one reference feature that is a reference feature from a plurality of features, and at least one reference feature that is a feature that indicates the at least one reference feature A reference feature value range determination unit that determines a certain reference feature value range, and extracts a learning image and/or an inspection image in which at least one reference feature value is included in the reference feature value range from among the learning images and/or inspection images. An image extraction unit, and a display control unit that causes a display unit to display the extracted learning image and/or the extracted inspection image, which are the learning image and/or the inspection image extracted by the image extraction unit.

In the above configuration, at least one reference feature is determined from a plurality of features of a learning image used for learning a classifier, and at least one reference feature is a feature indicating the at least one reference feature. A reference feature range is determined. Then, out of the learning images and/or inspection images to be classified by the classifier, the learning images and/or inspection images whose reference feature amount is included in the reference feature amount range are extracted, and then the extracted learning image and/or the inspection image is displayed on the display. Therefore, the operator can efficiently confirm whether or not the learning data on which the classifier is based is appropriate.

In the information processing device according to the above aspect, the reference feature determination unit may determine at least one reference feature according to selection by the operator. According to this aspect, since the operator can select the reference feature, convenience for the operator is improved.

In the information processing device according to the above aspect, the reference feature determination unit may determine at least one reference feature using a predetermined feature selection algorithm. According to this configuration, since the reference feature is determined by a predetermined feature selection algorithm, convenience for the operator is improved.

In the information processing device according to the above aspect, the reference feature amount range determination unit determines a reference image that is a reference learning image or an inspection image from the learning image and/or the inspection image, and selects at least one reference from the reference image. A threshold for the distance defined by the feature amount may be determined, and a range in which the distance from the reference image is equal to or less than the threshold may be determined as the reference feature amount range. According to this configuration, the reference image is determined from the learning image and/or the inspection image, and the range in which the distance defined by at least one reference feature amount from the reference image is equal to or less than the threshold is determined as the reference feature amount range. Therefore, it is possible to efficiently check the learning image and/or the inspection image that are relatively close to the reference image.

In the information processing apparatus according to the above aspect, the reference feature amount range determination unit may receive an operator's operation of specifying a reference feature amount area, and determine the specified area as the reference feature amount range. According to this configuration, the operator can determine the reference feature amount range through the operation of designating the area of the reference feature amount, thereby improving convenience for the operator.

In the information processing device according to the aspect described above, the display control unit further displays information indicating the attribute assigned to the extracted learning image and/or information indicating the attribute determined by the classifier for the extracted inspection image to the display unit. may be displayed in According to this configuration, the operator can determine whether the information indicating the attribute given to the extracted learning image is appropriate, and whether the attribute determined by the classifier for the extracted test image is appropriate. can be confirmed.

In the information processing apparatus according to the above aspect, the display control unit further includes at least one reference feature value associated with the extracted learning image in the storage unit and/or at least one reference feature value associated with the extracted inspection image in the storage unit. The two reference feature amounts may be displayed on the display unit. According to this configuration, the operator can confirm at least one reference feature associated with the extracted learning image and/or at least one reference feature associated with the extracted inspection image. .

In the information processing device according to the above one aspect, the display control unit further sets the representative point of the learning image and/or the representative point of the inspection image in a reference feature amount space, which is a space defined by at least one reference feature amount, to It may be displayed on the display unit. According to this configuration, it is possible for the operator to grasp the relationship of the distance defined by the reference feature amount of the learning image and/or the inspection image in the reference feature amount space.

ADVANTAGE OF THE INVENTION According to this invention, the technique which makes it possible to confirm efficiently whether the learning data used as the basis of a classifier is appropriate can be provided.

It is a schematic diagram showing an example of composition of inspection system 1 concerning this embodiment. It is a schematic diagram showing an example of composition of information processor 20 concerning this embodiment. It is a figure which shows an example of the data structure of the learning data 23a based on this embodiment. It is a figure which shows an example of the data structure of test|inspection data 23b which concern on this embodiment. It is a figure which shows an example of the operation|movement flow of the learning process which the inspection system 1 which concerns on this embodiment performs. It is a figure which shows an example of the operation|movement flow of the test|inspection process which the test|inspection system 1 which concerns on this embodiment performs. It is a figure which shows an example of the operation|movement flow of the screen display process which the inspection system 1 which concerns on this embodiment performs. FIG. 10 is a diagram showing an example of a screen 1000 displayed by the information processing device 20 according to the embodiment; 2 is a diagram showing an example of a screen 2000 displayed by an information processing apparatus 20 according to the embodiment; FIG. 3 is a diagram showing an example of a screen 3000 displayed by the information processing device 20 according to the embodiment; FIG.

Preferred embodiments of the present invention will be described with reference to the accompanying drawings. In each figure, the same reference numerals have the same or similar configurations. In the present disclosure, "A and/or B" may include any of the cases where only A is included, only B is included, and both A and B are included.

FIG. 1 is a schematic diagram showing an example of the configuration of an inspection system 1 according to this embodiment. The inspection system 1 according to the present embodiment is a system for inspecting an object W by determining attributes of an inspection image generated by imaging the object W using a predetermined classifier. . The inspection system 1 includes, for example, an imaging device 10 and an information processing device 20 .

The imaging device 10 includes an image sensor such as a CCD (Charge-Coupled Device) or a CMOS (Complementary MOS). The image is output to the information processing device 20 . Specifically, in a learning process to be described later, the imaging device 10 outputs a learning image obtained by imaging the object W to the information processing device 20 . Further, in an inspection process to be described later, the imaging device 10 outputs an inspection image obtained by imaging the object W to the information processing device 20 .

The target object W is not particularly limited, regardless of whether it is a final product or a part, and whether it is a finished product or an intermediate product. The object W may be transported by a transport device 100 as shown in FIG. 1, for example. The conveying device 100 is configured, for example, as a belt conveyor, and conveys the object W in the direction indicated by the arrow T in accordance with control signals supplied from the information processing device 20 or another controller (not shown).

The information processing device 20 is configured by, for example, one or more computers. The information processing device 20 includes a classifier generated by performing machine learning on learning data based on learning images obtained by imaging the object W. FIG. The information processing apparatus 20 inspects the object W by determining attributes of an inspection image obtained by imaging the object W using the classifier.

FIG. 2 is a schematic diagram showing an example of the configuration of the information processing device 20 according to this embodiment. The information processing device 20 includes, for example, a display unit 21, an operation unit 22, a storage unit 23, and a processing unit 24.

The display unit 21 may be any device as long as it can display video, images, and the like, such as a liquid crystal display and an organic EL (Electro-Luminescence) display. The display unit 21 displays an image corresponding to the image data supplied from the processing unit 24, an image corresponding to the image data, and the like.

The operation unit 22 may be any device as long as it can operate the information processing apparatus 20, such as a touch panel or key buttons. The operator can use the operation unit 22 to input characters, numbers, symbols, and the like. The operation unit 22 generates a signal corresponding to the operation when operated by the operator. The generated signal is supplied to the processing unit 24 as an operator's instruction.

The storage unit 23 is, for example, an auxiliary storage device such as a hard disk drive or solid state drive, and stores various programs executed by the processing unit 24, learning data 23a, inspection data 23b, classifier 23c, and the like. Various programs, for example, computer-readable portable recording media such as CD-ROMs and DVD-ROMs (computers, other devices, machines, etc. can read information such as programs recorded) , electrical, magnetic, optical, mechanical, or chemical storage medium) may be installed in the storage unit 23 using a known setup program or the like. In addition, the storage unit 23 may temporarily store temporary data related to predetermined processing.

FIG. 3 is a diagram showing an example of the data structure of the learning data 23a according to this embodiment. The training data includes, for example, image data of training images, one or more feature amounts of the training images, and labels of the training images.

The image ID is identification information (ID) for identifying a learning image. The image data of the learning image is, for example, image data generated by imaging the target object W for generating the learning image by the imaging device 10 . The image data of the learning image may include, for example, parameters such as hue, saturation, and brightness associated with each pixel forming the learning image.

A feature amount is information that quantitatively and qualitatively indicates a predetermined feature of an image. The feature amount may be, for example, a feature amount extracted from the learning image by the feature amount extraction unit 243, which will be described later. In the example shown in FIG. 3, four feature amounts from the first feature amount to the fourth feature amount are shown as an example, but the types of feature amounts (features) possessed by the learning data are not limited to four. There may be one, two, three, five or more. The feature amount may be information included in the image data (parameters such as hue, saturation, and brightness associated with each pixel).

A label is information indicating an attribute of a learning image. The number and content of labels can be arbitrarily set by the operator. A "defective product" indicating that the contained object W is defective may be included. The label may, for example, have been entered by an operator. Note that when machine learning for generating the classifier 23c is performed by unsupervised learning, the learning data 23a may not include labels.

FIG. 4 is a diagram showing an example of the data structure of inspection data 23b according to this embodiment. The inspection data includes, for example, image data of an inspection image, one or more feature amounts of the inspection image, and labels of the inspection image.

The image ID is identification information (ID) for identifying an inspection image. The image data of the inspection image is, for example, image data generated by imaging the object W to be inspected by the imaging device 10 . The image data of the inspection image may include, for example, parameters such as hue, saturation, and brightness associated with each pixel forming the inspection image.

A feature amount is information that quantitatively and qualitatively indicates a predetermined feature of an image. The feature amount may be, for example, a feature amount extracted from the inspection image by the feature amount extraction unit 243 . In the example shown in FIG. 4, four feature amounts from the first feature amount to the fourth feature amount are shown as an example, but the types of feature amounts (features) possessed by inspection data are not limited to four. It may be 1, 2, 3, or 5 or more depending on the learning data. The feature amount may be information included in the image data (parameters such as hue, saturation, and brightness associated with each pixel).

A label is information indicating an attribute of an inspection image. The number and content of labels can be arbitrarily set by the operator according to the labels of the learning images. A “non-defective product” and a “defective product” indicating that the object W included in the image is a defective product may be included. The label may be output by inputting the inspection image into the classifier 23c, for example. Note that when machine learning for generating the classifier 23c is performed by unsupervised learning, the label of the inspection data 23b may be the result of the correct/wrong judgment.

As described above, the storage unit 23 stores, for example, the classifier 23c for determining image attributes. The classifier 23c is generated, for example, by machine learning using the learning data 23a, and outputs the attribute of the image according to the input of the feature amount of the image. For example, if the classifier 23c is configured as a neural network consisting of an input layer, an output layer, and at least one intermediate layer, the storage unit 23 stores weighting coefficients between these layers. The classifier 23c outputs, for example, the attribute of the inspection image related to the inspection data 23b in accordance with the input of the inspection data 23b. The attributes of the image output by the classifier 23c can be arbitrarily set, but in this embodiment, as an example, the attributes of the inspection image include "non-defective product" and "defective product". Here, the attribute "non-defective item" may be an attribute indicating that the object W included in the image is a non-defective item. Also, the attribute “defective item” may be an attribute indicating that the object W included in the image is defective.

The processing unit 24 includes a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), etc., and controls each component in accordance with information processing. The processing unit 24 controls the operation of the display unit 21 and the like so that various processes of the information processing device 20 are executed in appropriate procedures based on the programs stored in the storage unit 23, the operation of the operation unit 22, and the like. do. The processing unit 24 executes processing based on programs (operating system programs, driver programs, application programs, etc.) stored in the storage unit 23 . Also, the processing unit 24 can execute a plurality of programs (application programs, etc.) in parallel.

The processing unit 24 includes an imaging control unit 241, a preprocessing unit 242, a feature amount extraction unit 243, a machine learning execution unit 244, a classification execution unit 245, a reference feature determination unit 246, and a reference feature amount range determination unit. 247 , an image extraction unit 248 , and a display control unit 249 . Each of these units is a functional module implemented by a program executed by a processor included in the processing unit 24 . Alternatively, each of these units may be implemented in the information processing device 20 as an independent integrated circuit, microprocessor, or firmware.

The imaging control unit 241 transmits a control signal to the imaging device 10 to control imaging by the imaging device 10 . The imaging control unit 241 also acquires an image generated by imaging from the imaging device 10 and stores it in the storage unit 23 . The image may be, for example, a learning image generated by imaging the object W for generating a learning image, an inspection image generated by imaging the object W to be inspected, or the like.

The preprocessing unit 242 performs predetermined preprocessing in a learning process for generating the classifier 23c by machine learning using the learning data and/or an inspection process for performing determination on the inspection data using the generated classifier 23c. to run. For example, the preprocessing unit 242 may assign a label to each learning image when supervised learning is performed as the learning process. Labeling may be performed, for example, in response to an operator's operation via the operation unit 22 . Further, for example, when unsupervised learning is performed as the learning process, the preprocessing unit 242 may remove images other than desired images, such as non-defective images, as noise. Further, for example, in the learning process, the preprocessing unit 242 may perform a process of removing unnecessary noise from the learning data, a process of filling missing items in the learning data, and the like. In the inspection process, the preprocessing unit 242 may perform a process of removing unnecessary noise from the inspection data, a process of filling missing items in the inspection data, and the like.

The feature amount extraction unit 243 extracts a predetermined feature amount from each image in learning processing and/or inspection processing. For example, the feature amount extraction unit 243 extracts a predetermined feature amount from the learning image in the learning process. The feature amount extracted from the learning image is stored in the storage unit 23 as part of the learning data, for example. For example, the feature amount extraction unit 243 extracts a predetermined feature amount from the inspection image in the inspection process. The feature amount extracted from the inspection image is stored in the storage unit 23 as part of the inspection data, for example. The feature amount extracted from each image by the feature amount extraction unit 243 may be one, or may be two or more. In addition, the type of the feature amount is not particularly limited, but for example, parameters such as hue, saturation, and brightness associated with each pixel constituting the learning image, and the average frequency, entropy, and average frequency obtained from the parameters It may be luminance or the like. Alternatively, the feature amount can be, for example, the color of the object, the edge information of the object, the position of the object, and the like. The edge information may include the position, shape, length, etc. of the edge. The position of the object includes not only the position of the object itself but also the position of part of the object.

The feature quantity extraction unit 243 generates not only a feature quantity directly generated from an image, but also a feature quantity (secondary feature quantity) generated based on at least one other feature quantity (primary feature quantity). By doing so, the feature amount (secondary feature amount) may be extracted. In other words, the feature amount extraction unit 243 can perform "feature amount compression" by generating one secondary feature amount from a plurality of primary feature amounts.

The machine learning execution unit 244 generates the classifier 23 c by machine learning using the learning data stored in the storage unit 23 . The machine learning method is not particularly limited, but for example, the initial values of the parameters of the classifier 23c are randomly determined, and the parameters are adjusted while feeding back the image recognition errors output from the classifier 23c. , so-called deep learning, or a technique such as a well-known genetic algorithm (GA) may be employed.

The classification executing unit 245 inputs inspection data related to an inspection image to the classifier 23c and outputs the attributes of the inspection image (“non-defective product”, “defective product”, etc.), thereby executing an inspection of the object W. do. The classification execution unit 245 may store the attribute of the output inspection image in the storage unit 23 in association with the inspection data.

A reference feature determination unit 246 determines at least one reference feature from a plurality of features associated with the learning image and/or the inspection image. For example, the reference feature determining unit 246 stores a plurality of feature values stored as learning data in the storage unit 23 in association with the learning image, or a plurality of feature values stored in the storage unit 23 as inspection data in association with the inspection image. At least one reference feature is determined from the quantity. Here, the reference feature is, for example, a feature for defining a reference feature amount space or a reference feature amount range, which will be described later. That is, it can be said that the reference feature is a feature that serves as a reference when examining the accuracy of the classifier 23c. The number of reference features is not particularly limited, and may be one, two, or three or more. Specifically, the reference feature determination unit 246 determines at least one feature as a reference feature from a plurality of features included in each of the learning data and/or inspection data stored in the storage unit 23 . The reference feature determiner 246 may select features, for example, based on the operator's selection of desired features. Alternatively, the reference feature determiner 246 may select features using, for example, a predetermined feature selection algorithm. The predetermined feature selection algorithm is not particularly limited as long as it is a known algorithm that selects a feature suitable for a model from a plurality of features, but may include algorithms such as filter method, wrapper method, and embedding method, for example. The filtering method, also called univariate feature quantity selection, is a method of statistically verifying the relationship between individual feature quantities and objectives, and selecting the feature quantities considered to be the most dominant. The wrapper method, which is also called iterative feature quantity selection, is a method of combining a plurality of feature quantities, verifying prediction accuracy, and searching for a combination with the highest accuracy. The built-in method, also called model-based feature selection, is a method of performing machine learning and feature selection at the same time, and a representative algorithm is a decision tree.

The reference feature amount range determining unit 247 determines a reference feature amount range, which is a predetermined range in the space defined by the reference feature amount. Here, the reference feature amount range may be, for example, a range for extracting a learning image and/or an inspection image.

For example, the reference feature amount range determination unit 247 determines a reference image that serves as a reference from the learning image and the inspection image, calculates the distance between each other image and the reference image defined by the reference feature amount, and The reference feature amount range may be determined based on the distance. Specifically, the reference feature amount range determining unit 247 determines a threshold value for a distance defined by the reference feature amount from the reference image (reference feature amount distance), and determines that the reference feature amount distance from the reference image is A range equal to or less than the threshold may be determined as the reference feature value range. Here, the method for calculating the distance can be arbitrarily set, but may include, for example, Euclidean distance, Mahalanobis distance, Manhattan distance, and the like. Further, the threshold value may be the reference feature amount distance for an image having a predetermined order of the reference feature amount distance from the reference image among the learning images and the inspection images. In this case, the reference feature amount range includes images up to a predetermined order of the reference feature amount distance from the reference image among the learning images and the inspection images. Alternatively, the threshold may be a predetermined value. In this case, the reference feature amount range includes images whose reference feature amount distance from the reference image is up to a predetermined value among the learning images and the inspection images. Note that when the reference feature amount distance is defined as the Euclidean distance, the reference feature amount range can be represented as a circle in the reference feature amount space. Also, when the reference feature amount distance is defined as the Mahalanobis distance, the reference feature amount range can be represented as an ellipse in the reference feature amount space. Also, when the reference feature amount distance is defined as the Manhattan distance, the reference feature amount range can be represented as a rectangle in the reference feature amount space.

Alternatively, in determining the reference feature amount range, it is not always necessary to determine the reference image. For example, the reference feature amount range determination unit 247 may receive an operation to specify an arbitrary area in the reference feature amount space, which is a space defined by the reference feature amount, and determine the reference feature amount range based on the area. good. Specifically, the reference feature amount range determination unit 247 accepts designation of an arbitrary area within the reference feature amount space drawn by the operator's operation such as dragging, and then designates the area as the reference feature amount range. You may decide. The shape of the area is not limited to a circle, and may include rectangles, other polygons, approximate shapes thereof, and arbitrary irregular shapes drawn freehand or the like.

The image extractor 248 extracts desired images from the learning images and/or inspection images. For example, the image extraction unit 248 extracts an image whose reference feature amount is included in the reference feature amount range from the learning images and/or inspection images. Specifically, for each image (learning image and/or inspection image) to be extracted, a reference feature value stored in association with the image in the storage unit 23 or the like is specified, and the specified reference feature value is included in the reference feature value range. When it is determined that the specified reference feature amount is included in the reference feature amount range, the image extraction unit 248 extracts the image. When it is determined that the specified reference feature amount is not included in the reference feature amount range, the image extraction unit 248 does not extract the image.

The display control unit 249 causes the display unit 21 to display various screens and information based on various display data. For example, the display control unit 249 generates display data based on various information included in the learning data and/or inspection data stored in the storage unit 23, and generates learning data and/or inspection data based on the display data. , is displayed on the display unit 21 . Specifically, the display control unit 249 causes the display unit 21 to display the learning image and/or the inspection image. The learning image and/or inspection image to be displayed may be, for example, the learning image and/or inspection image extracted by the image extracting unit 248, but may be a learning image and/or inspection image arbitrarily selected by the operator. The inspection image may be displayed on the display unit 21 . In addition, the display control unit 249 may cause the display unit 21 to display items (feature amounts, labels, etc.) of each learning data and/or each inspection data stored in the storage unit 23 or the like. Further, the display control unit 249 may cause the display unit 21 to display a feature amount list, which is a list of each feature amount associated with each of the learning images and/or inspection images in the storage unit 23 or the like. Further, the display control unit 249 may cause the display unit 21 to display representative points of the learning image and/or the inspection image in a space defined by a predetermined feature amount (feature amount space).

FIG. 5 is a diagram showing an example of an operation flow of learning processing executed by the inspection system 1 according to this embodiment.

(S101) First, a learning image is generated by capturing an image of the object W for learning using the imaging device 10 . Specifically, for example, the imaging control unit 241 controls the imaging device 10 to capture an image of the learning object W in response to the operator's operation of the operation unit 22 . As a result, a learning image in which the object W is captured is generated. The number of learning images to be generated is not particularly limited, and may be one or two or more.

In step S201, the operator divides the target object W into non-defective products and defective products in advance, and then images the non-defective target object W and the defective target object W separately to generate learning images. good too. In this case, for example, in step S202, which will be described later, it is possible to collectively assign a label of "non-defective product" or "defective product" to the collectively captured learning images.

(S102) Next, each learning image is labeled. For example, the operator causes the display unit 21 to display each learning image by operating the operation unit 22, and then performs an operation of assigning a “non-defective product” or “defective product” label to each learning image. This is done for the part 22 . Alternatively, the operator visually checks the object W to determine whether it is a non-defective product or a defective product, and then assigns a “non-defective product” or “defective product” label to each learning image corresponding to the target object W. You may perform operation to the operation part 22 to carry out. In response to the operation, the preprocessing unit 242 causes the storage unit 23 to store the assigned label in association with the learning image. As a result, the assigned label is stored in the storage unit 23 as part of the learning data.

(S103) Next, after extracting a feature amount from each learning image, the feature amount extraction unit 243 stores the feature amount in the storage unit 23 in association with the learning image. Thereby, the feature amount is stored in the storage unit 23 as part of the learning data. The number of feature amounts extracted by the feature amount extraction unit 243 may be one, or may be two or more. In addition, the type of the feature amount is not particularly limited, for example, parameters such as hue, saturation, and brightness associated with each pixel constituting the learning image, and average frequency, entropy, and average frequency obtained from the parameters It may be luminance or the like. Alternatively, the feature amount may be, for example, the color of the object, the edge information of the object, the position of the object, or the like. The edge information may include the position, shape, length, etc. of the edge. The position of an object includes the position of a part of the object as well as the position of the object.

(S104) Next, the machine learning execution unit 244 generates the classifier 23c by machine learning using the learning data stored in the storage unit 23. FIG. The generated classifier 23c is stored in the storage unit 23, for example. The machine learning method is not particularly limited, and may be, for example, deep learning, genetic algorithm (GA), or the like. With this, the learning process ends.

FIG. 6 is a diagram showing an example of an operation flow of inspection processing executed by the inspection system 1 according to this embodiment.

(S201) First, an inspection image is generated by capturing an image of the object W to be inspected using the imaging device 10 . Specifically, for example, the imaging control unit 241 controls the imaging device 10 and images the object W to be inspected in response to the operator's operation of the operation unit 22 . As a result, an inspection image in which the object W is imaged is generated. The number of inspection images to be generated is not particularly limited, and may be one or two or more.

(S202) Next, the feature quantity extraction unit 243 extracts a feature quantity from each inspection image, and stores the feature quantity in the storage unit 23 in association with the inspection image. Thereby, the feature amount is stored in the storage unit 23 as part of the inspection data. The feature amount extracted by the feature amount extraction unit 243 may include at least the feature amount extracted from the learning image in the learning process described above. That is, the number of feature amounts extracted by the feature amount extraction unit 243 may be one, or may be two or more. In addition, the type of the feature amount is not particularly limited. It may be luminance or the like. Alternatively, the feature amount may be, for example, the color of the object, the edge information of the object, the position of the object, or the like. The edge information may include the position, shape, length, etc. of the edge. The position of an object includes the position of a part of the object as well as the position of the object.

(S203) The classification executing unit 245 inputs the inspection data related to the inspection image to the classifier 23c, and outputs the attributes of the inspection image (“non-defective product”, “defective product”, etc.). Run an inspection. The classification executing unit 245 stores the attribute of the output inspection image in the storage unit 23 in association with the inspection data. With this, the inspection process is completed.

FIG. 7 is a diagram showing an example of an operation flow of screen display processing executed by the inspection system 1 according to this embodiment.

(S301) First, the reference feature determination unit 246 determines at least one feature from among the first to n-th features as a reference feature.

The reference feature determining unit 246 may, for example, accept selection of a reference feature by the operator and determine the reference feature according to the selection. The selection of reference features can be made, for example, in screen 1000 shown in FIG. The screen 1000 is displayed on the display unit 21 by the display control unit 249 based on the learning data and inspection data stored in the storage unit 23, for example. The screen 1000 may include, for example, a feature quantity list 1001 that is a list of each feature quantity associated with each of the learning images and/or inspection images. The feature amount list 1001 shows image IDs, types, labels, and feature amounts for learning images and inspection images. There may be one or more types of feature quantity. In FIG. 8, the first to fourth feature amounts are shown as an example. The order in which the images are arranged in the feature quantity list is not particularly limited, but in FIG. 8, as an example, the learning images and the inspection images are arranged in order of image ID. The feature quantity list may be sortable for each item such as image ID, type, label, and feature quantity. (ascending or descending order, etc.).

The operator can select a desired reference feature, for example, by selecting a cell labeled "first feature" in the first row included in the feature list. According to the selection, the reference feature determining unit 246 determines the feature corresponding to the selected cell as the reference feature. When the reference feature is determined in the feature amount list, the display control unit 249 may sort and display the feature amount list according to the reference feature amount, which is the feature amount corresponding to the reference feature.

Note that in step S301, the reference feature determination unit 246 may determine features by selecting features using a predetermined feature selection algorithm such as a filter method, a wrapper method, and an embedding method. For example, the reference feature determination unit 246 selects features using the feature selection algorithm and determines the reference features in response to the operator performing an operation to instruct the determination of the reference features by the feature selection algorithm. good.

(S302) Next, the display control unit 249 causes the display unit 21 to display the representative points of the learning image and/or the inspection image in the reference feature amount space according to the operator's operation via the operation unit 22 . Here, the reference feature amount space is a space defined by at least one reference feature amount determined in step S301 described above. The display control unit 249 causes the display unit 21 to display, for example, a screen 2000 shown in FIG. A screen 2000 includes a graphical representation 2001 of the reference feature space.

In FIG. 9, as an example, when the reference features determined by the reference feature determination unit 246 are the p-th feature and the q-th feature, that is, when the reference feature amounts are the p-th feature amount and the q-th feature amount, the reference A graphical representation 2001 of feature space is shown. In the graph representation 2001 of the reference feature amount space shown in FIG. 9, the horizontal axis indicates the p-th feature amount, and the vertical axis indicates the q-th feature amount.

As shown in FIG. 9, a graphical representation 2001 of the reference feature space may include plots (representative points) arranged at positions corresponding to reference features of each image (learning image and/or test image). In FIG. 9, white circles indicate plots of learning images labeled as good products (good product learning images), and white triangles indicate learning images labeled as defective products (defective product learning images). , black circles indicate plots of inspection images labeled as good (good inspection images), and black triangles indicate inspection images labeled as defective (defective learning images). plots are shown. The display control unit 249 may be able to arbitrarily switch the presence or absence of display of these plots and the manner thereof, for example, according to the operation of the operator via the operation unit 22 or the like. For example, in the graph display 2001 of the reference feature space, the display control unit 249 may display only the plot of (all or part of) the learning image in accordance with the operation of the operation unit 22, or the inspection image ( Only plots of (all or part of) may be displayed, or both plots of (all or part of) training images and plots of (all or part of) test images may be displayed. In addition, the display control unit 249 displays the label associated with each image (learning image or inspection image) and each feature amount (including feature amounts other than the reference feature amount) in a manner associated with each plot. may

(S303) Next, the reference feature amount range determination unit 247 determines a reference feature amount range, which is a predetermined range in the space defined by the reference feature amount.

The reference feature amount range may be determined according to an operation on the feature amount list 1001 shown in FIG. For example, when the operator selects a desired learning image or inspection image in the feature amount list 1001, the reference feature amount range determination unit 247 determines the selected learning image or inspection image as the reference image. For example, by selecting any row included in the feature list, the operator can select the image corresponding to that row as the reference image. According to the selection, the reference feature amount range determination unit 247 determines the image corresponding to the selected row as the reference image. In the feature amount list 1001, even if it is possible to select the reference feature and the reference image at the same time by selecting the cell in the row corresponding to the specific image and the column corresponding to the specific feature amount. good. That is, for example, when the cell with the first feature amount “91” for the learning image with the image ID “L-0001” shown in FIG. amount) as the reference feature (reference feature amount), and the image extraction unit 248 may decide the learning image with the image ID “L-0001” as the reference image.

Then, the reference feature amount range determining unit 247 determines a threshold for the distance defined by the reference feature amount from the reference image (reference feature amount distance), and then determines that the reference feature amount distance from the reference image is equal to or less than the threshold. is determined as the reference feature amount range. Here, the threshold value may be the reference feature amount distance for an image having a predetermined order of the reference feature amount distance from the reference image among the learning images and the inspection images. In this case, the reference feature amount range includes images up to a predetermined order of the reference feature amount distance from the reference image among the learning images and the inspection images. Alternatively, the threshold may be a predetermined value. In this case, the reference feature amount range includes images whose reference feature amount distance from the reference image is up to a predetermined value among the learning images and the inspection images.

The reference feature amount range may be determined according to an operation on the graph representation 2001 of the reference feature amount space shown in FIG. For example, when the operator selects a desired learning image or inspection image in the graph display 2001 of the reference feature space, the reference feature amount range determination unit 247 determines the selected learning image or inspection image as the reference image. The operator can select any plot shown in the graphical representation 2001 of the reference feature amount space as the reference image. FIG. 9 shows, as an example, a case where an image (non-defective product inspection image A) corresponding to a plot of black circles surrounded by approximately pentagonal objects labeled A is selected as a reference image. there is

Then, the reference feature amount range determining unit 247 determines a threshold for the distance defined by the reference feature amount from the reference image (reference feature amount distance), and then determines that the reference feature amount distance from the reference image is equal to or less than the threshold. is determined as the reference feature amount range. The threshold may be a predetermined value. In other words, the reference feature amount range includes images whose reference feature amount distance from the reference image is up to a predetermined value among the learning images and the inspection images. For example, in FIG. 9, the threshold is a predetermined value d. That is, the circle M with a radius d centered on the plot of the non-defective product inspection image A, which is the reference image, is the reference feature amount range. In this way, the range in which the distance from the reference image in the feature amount space (reference feature amount distance) is within a predetermined threshold is the reference feature amount range. Note that the threshold value may be the reference feature amount distance for an image having a predetermined order of the reference feature amount distance from the reference image among the learning images and the inspection images. Note that the reference feature amount range shown in FIG. 9 is an example when the reference feature amount distance is defined as the Euclidean distance. If the reference feature distance is defined as the Mahalanobis distance, the reference feature range can be represented as an ellipse in the reference feature space. Also, when the reference feature amount distance is defined as the Manhattan distance, the reference feature amount range can be represented as a rectangle in the reference feature amount space.

In determining the reference feature amount range, it is not always necessary to determine the reference image in the graphical display 2001 of the reference feature amount space. For example, the reference feature amount range determination unit 247 may receive an operation to specify an arbitrary area in the reference feature amount space, and determine the reference feature amount range based on the area. Specifically, the reference feature amount range determination unit 247 accepts designation of an arbitrary area within the reference feature amount space drawn by an operation such as dragging by the operator. Then, the reference feature amount range determination unit 247 may determine the region as the reference feature amount range. The shape of the area is not limited to a circle, and may include rectangles, other polygons, approximate shapes thereof, and arbitrary irregular shapes drawn freehand or the like.

(S304) Next, the image extraction unit 248 extracts images whose reference feature amounts are included in the reference feature amount range determined in step S303 described above, from among the learning images and/or inspection images. Specifically, for each image (learning image and/or inspection image) to be extracted, a reference feature amount stored in association with the image in the storage unit 23 is specified, and the specified reference feature amount is It is determined whether or not it is included in the reference feature amount range determined in step S303 described above. When it is determined that the specified reference feature amount is included in the reference feature amount range, the image extraction unit 248 extracts the image. When it is determined that the specified reference feature amount is not included in the reference feature amount range, the image extraction unit 248 does not extract the image.

As an example, the graph display 2001 of the reference feature amount space in FIG. A plot of black circles surrounded by substantially rectangular objects marked with D and a plot of black triangles surrounded by substantially rectangular objects labeled D are shown. This indicates that the non-defective product learning image B, the non-defective product inspection image C, and the defective product inspection image D are extracted from the reference feature amount range represented by the circle M. In other words, the position coordinates represented by the p-th feature amount and the q-th feature amount of the non-defective product learning image B, the non-defective product inspection image C, and the defective product inspection image D, respectively, and the p-th feature value of the non-defective product inspection image A, which is the reference image. The distance to the position coordinates represented by the feature amount and the q-th feature amount is within d.

(S305) Next, the display control unit 249 obtains image data from the storage unit 23 for the images (learning images and/or inspection images) extracted by the image extracting unit 248, and based on the image data, The extracted image is displayed on the display unit 21 . FIG. 10 shows an example of a screen 3000 on which an image extracted by the image extractor 248 is displayed. As shown in FIG. 10, a screen 3000 includes a graphical representation 3001 of the reference feature space. The graphical representation 3001 may be substantially identical to the graphical representation 2001 shown in FIG. Further, as shown in FIG. 10, the screen 3000 includes a reference image display area 3002 and an extracted image display area 3003 other than the reference image.

In the display area 3002 of the reference image, for example, in addition to the image of the non-defective product inspection image A that is the reference image, the type, image ID, label attached to the image, reference feature amount (p-th feature amount and q-th feature amount) is included. Note that the display area 3002 may include feature amounts other than the reference feature amount of the reference image (non-defective product inspection image A). As described above, when the reference feature amount range is determined without determining the reference image, the screen 3000 may not include the display area 3002 of the reference image.

In the display area 3003 of the reference image, for example, in addition to each image of the good product learning image B, the good product inspection image C, and the defective product inspection image D, which are the extracted images, the type, the image ID, and the label attached to the image are displayed. Alternatively, the label determined by the classifier 23c and the reference feature amount (p-th feature amount and q-th feature amount) are included. Note that the display area 3003 may include feature amounts other than the reference feature amount of the reference image (non-defective product inspection image A). As described above, when the reference feature amount range is determined without determining the reference image, the screen 3000 may not include the display area 3002 of the reference image.

The reference feature amount range determination unit 247 may update the reference image, the reference feature amount range, and the like in accordance with an operation on the graph display 3001 of the reference feature amount space. For example, the reference feature amount range determination unit 247 determines an image (learning image or inspection image) corresponding to the selected plot as a new reference image in response to selection of a plot in the graph display 3001, thereby obtaining a reference image. may be updated. Further, for example, the reference feature amount range determination unit 247 determines the designated area as a new reference feature amount range in response to an operation of designating the area of the reference feature amount range in the graph display 3001, thereby determining the reference feature amount range. Amount ranges may be updated. When the reference image, the reference feature amount range, and the like are updated in accordance with an operation on the graph display 3001 and the like, the display control unit 249 displays the graph display 3001 of the reference feature amount space included in the screen 3000 in accordance with the update. Alternatively, the display of the image display areas 3002 and 3003 may be updated.

With the configuration of the inspection system 1 according to the present embodiment as described above, the operator can easily confirm information such as the image, feature amount, and label of a desired image (learning image or inspection image). Become. For example, in the example shown in FIG. 10, the defective part G1 is generated in the product included in the non-defective product inspection image A, which is the reference image. Therefore, the non-defective product inspection image A is an image that should be determined as a "defective product" by the classifier 23c. Therefore, when confirming the reference feature amount range represented by the circle M in the graph display 3001 of the reference feature amount space, the distance defined by the reference feature amount from the non-defective product inspection image A (originally, the “defective product” inspection image) is It can be confirmed that the non-defective product learning image B and the non-defective product learning image D are extracted as similar learning images. Here, when the images of the non-defective product learning image B and the non-defective product learning image D are checked, although there is no particular problem in the product included in the image of the non-defective product learning image B, the product included in the image of the non-defective product learning image D It can be seen that there is a defective portion G2 in . Therefore, it is found that the non-defective product learning image D should have been given the label of "defective product", but was mistakenly labeled as "non-defective product". Therefore, the operator can update the learning data by changing the label of the non-defective product learning image D to "defective product" and generate the classifier 23c based on the updated new learning data. Thus, in the inspection system 1 according to this embodiment, it is possible to easily confirm whether or not the learning data that is the basis of the classifier is appropriate.

The embodiments described above are merely examples of the present invention in every respect. It goes without saying that various modifications and variations can be made without departing from the scope of the invention. That is, in implementing the present invention, a specific configuration according to the embodiment may be appropriately employed. Although the data appearing in this embodiment are explained in terms of natural language, more specifically, they are specified in computer-recognizable pseudo-language, commands, parameters, machine language, and the like.

DESCRIPTION OF SYMBOLS 1... Inspection system 10... Imaging device 10... Information processing apparatus 21... Display part 22... Operation part 23... Storage part 23a... Learning data 23b... Inspection data 23c... Classifier 24... Processing part 241... Imaging control unit 242... Preprocessing unit 243... Feature amount extraction unit 244... Machine learning execution unit 245... Classification execution unit 246... Reference feature determination unit 247... Reference feature amount range determination unit 248... Image extraction unit 249 Display control unit W Target object 100 Conveying device

Claims (11)

storing a plurality of feature quantities respectively representing a plurality of features of the learning image in association with a learning image used for learning a classifier for determining attributes of an image; An information processing device capable of accessing a storage unit that stores a plurality of feature amounts respectively representing the plurality of features of the inspection image in association with each other,
a display unit;
a reference feature determination unit that determines at least one reference feature that is a reference feature from the plurality of features;
a reference feature amount range determination unit that determines a reference feature amount range that is a range in a space defined by the at least one reference feature amount that is a feature amount that indicates the at least one reference feature;
an image extraction unit that extracts, from the learning images and/or the inspection images, a learning image and/or an inspection image in which the at least one reference feature amount is included in the reference feature amount range;
a display control unit that causes the display unit to display an extracted learning image and/or an extracted inspection image, which are learning images and/or inspection images extracted by the image extraction unit;
Information processing device. 2. The information processing apparatus according to claim 1, wherein said reference feature determination unit determines said at least one reference feature according to selection by an operator. 3. The information processing apparatus according to claim 1, wherein said reference feature determination unit determines said at least one reference feature by a predetermined feature selection algorithm. The reference feature value range determination unit,
determining a reference image that is a reference learning image or an inspection image from the learning image and/or the inspection image;
determining a threshold for the distance defined by the at least one reference feature from the reference image;
The information processing apparatus according to any one of claims 1 to 3, wherein the range in which the distance from the reference image is equal to or less than the threshold is determined as the reference feature amount range. 4. The reference feature value range determination unit according to any one of claims 1 to 3, wherein the reference feature value range determination unit receives an operation by an operator to specify the region of the reference feature value, and determines the specified region as the reference feature value range. The information processing device according to . The display control unit further causes the display unit to display information indicating the attribute assigned to the extracted learning image and/or information indicating the attribute determined by the classifier for the extracted test image. Item 6. The information processing device according to any one of Items 1 to 5. The display control unit further includes the at least one reference feature associated with the extracted learning image in the storage unit and/or the at least one reference feature associated with the extracted inspection image in the storage unit. The information processing apparatus according to any one of claims 1 to 6, wherein the amount is displayed on the display unit. The display control unit further causes the display unit to display a representative point of the learning image and/or a representative point of the inspection image in a reference feature amount space defined by the at least one reference feature amount. The information processing apparatus according to any one of claims 1 to 7. The display control unit further controls at least one of the plurality of feature amounts associated with the learning image and/or at least one of the plurality of feature amounts associated with the inspection image. 9. The information processing apparatus according to any one of claims 1 to 8, wherein a list of feature amounts of is displayed on said display unit. storing a plurality of feature quantities respectively representing a plurality of features of the learning image in association with a learning image used for learning a classifier for determining attributes of an image; A computer capable of accessing a storage unit that stores a plurality of feature amounts respectively representing the plurality of features of the inspection image in association with each other,
Determining at least one reference feature that is a reference feature from the plurality of features;
Determining a reference feature range that is a range in a space defined by the at least one reference feature that is a feature indicating the at least one reference feature;
extracting, from the learning images and/or inspection images, a learning image and/or inspection image in which the at least one reference feature amount is included in the reference feature amount range;
displaying an extracted learning image and/or an extracted inspection image, which are the extracted learning image and/or inspection image, on a display unit;
Information processing method that performs storing a plurality of feature quantities respectively representing a plurality of features of the learning image in association with a learning image used for learning a classifier for determining attributes of an image; a computer capable of accessing a storage unit that stores a plurality of feature amounts respectively representing the plurality of features of the inspection image in association with each other;
Determining at least one reference feature that is a reference feature from the plurality of features;
Determining a reference feature range that is a range in a space defined by the at least one reference feature that is a feature indicating the at least one reference feature;
extracting, from the learning images and/or inspection images, a learning image and/or inspection image in which the at least one reference feature amount is included in the reference feature amount range;
displaying an extracted learning image and/or an extracted inspection image, which are the extracted learning image and/or inspection image, on a display unit;
program to run.

Images