JP7236292B2 - Evaluation device, evaluation method, evaluation program, and inspection device - Google Patents

Description

The present invention relates to an evaluation device, an evaluation method, and an evaluation program for evaluating a classifier that classifies the states of articles. The present invention also relates to an inspection apparatus for inspecting articles.

A technique for classifying the state of an article by referring to an image containing the article as a subject is widely used. When implementing such a technique using a classifier constructed by machine learning, it is necessary to evaluate the learned classifier in the process of completing the classifier.

As a technology that may contribute to solving such problems, for example, an evaluation device described in Patent Document 1 can be cited. This evaluation device uses a learning goal reception unit that receives a learning goal, an evaluation unit that evaluates the classifier for at least the evaluation items included in the learning goal and generates evaluation data, and the learning goal and the evaluation data, and a determining unit for determining whether the classifier has achieved its learning goal.

Japanese Patent Application Laid-Open No. 2018-181184 (published on November 15, 2018)

The inventors compared the area in the image that the classifier pays attention to when classifying the state of the article and the area in the image that the person pays attention to when classifying the state of the article. As a result, even with a classifier having a certain generalization ability (for example, an evaluation device that is determined to have achieved a learning goal by the evaluation device described in Patent Document 1), when classifying the state of an article, human It was found that there are classifiers that focus on a completely different region from the one focused on by . Such classifiers are considered to be nothing more than accidental reproductions of human classification results on limited test data. Therefore, such a classifier cannot be expected to improve generalization ability through additional learning, and is a so-called "bad" classifier, and can be a classifier that replaces the classification of articles by humans. do not have. Techniques for evaluating classifiers from this point of view have not existed so far. Of course, the evaluation device described in Patent Literature 1 does not evaluate the classifier from such a point of view.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and an object of the present invention is to provide an evaluation apparatus capable of evaluating a classifier for classifying the state of an article from the viewpoint of similarity with a person in a region of interest. , an evaluation method, and an evaluation program. Another object of the present invention is to realize an inspection apparatus using such an evaluation apparatus.

In order to solve the above problems, an evaluation apparatus according to an aspect of the present invention refers to an image including an article as a subject, and performs classification processing for classifying the state of the article according to an algorithm determined by learning. An evaluation device for evaluating each of a plurality of classifiers. The evaluation device uses an image including, as a subject, an article whose state has been classified by a person as a sample image, and a heat map corresponding to each classifier, which is a region in the sample image to which the classifier pays attention in the classification process. Based on the comparison between the area in the sample image indicated by the corresponding heat map and the area in the sample image that the person focuses on in order to classify the state of the article, each an evaluation unit for evaluating the classifier.

In order to solve the above problems, an evaluation method according to an aspect of the present invention refers to an image including an article as a subject, and executes classification processing for classifying the state of the article according to an algorithm determined by learning. It is an evaluation method for evaluating each of a plurality of classifiers. The evaluation method uses an image including, as a subject, an article whose state has been classified by a person as a sample image, a heat map corresponding to each classifier, and a region in the sample image to which the classifier focuses in the classification process. and comparing the area in the sample image indicated by the corresponding heat map with the area in the sample image that the person focuses on to classify the state of the article, each and an evaluation step of evaluating the classifier.

In order to solve the above problems, an evaluation program according to one aspect of the present invention is an evaluation program that causes a computer to operate as the evaluation device described above, and causes the computer to function as each part of the evaluation device.

To solve the above problems, an inspection apparatus according to one aspect of the present invention is an inspection apparatus that inspects articles. The inspection device includes the plurality of classifiers, the evaluation device, and a selection unit that selects one of the plurality of classifiers based on the evaluation result of the evaluation device, The article is inspected using the classifier selected by the selection unit.

According to the above configuration, it is possible to evaluate a classifier that classifies the state of an article from the viewpoint of similarity with a person regarding the region of interest.

In the evaluation device according to an aspect of the present invention, the evaluation unit includes an area in the sample image that the person pays attention to for classifying the state of the article, in the area in the sample image indicated by the corresponding heat map. It is preferable to rate classifiers that include overlapping regions higher than classifiers that do not.

According to the above configuration, in order to classify the state of an article, it is possible to highly evaluate a classifier that focuses on a region similar to a region that a person focuses on.

In the evaluation device according to an aspect of the present invention, the evaluation unit includes an area in the sample image that the person pays attention to for classifying the state of the article, in the area in the sample image indicated by the corresponding heat map. It is preferable to rate classifiers that contain non-overlapping regions higher than classifiers that do not.

According to the above configuration, in order to classify the state of an article, in addition to focusing on areas similar to the area of human attention, a classifier that focuses on a unique area that is not similar to the area of human attention is provided. , can be appreciated.

In the evaluation apparatus according to one aspect of the present invention, each of a plurality of images containing articles classified by a person as subjects is used as a sample image, and a heat map corresponding to each classifier is created for each sample image, and the above evaluation is performed. Preferably, the unit evaluates each classifier based on evaluation results of evaluating each classifier for each sample image.

According to the above configuration, the classifier that classifies the state of the article can be evaluated with higher accuracy.

In the evaluation device according to the aspect of the present invention, the evaluation unit classifies the state of the article included in each sample image as a subject in each of the plurality of classifiers so that the person classifies the state of the article. Preferably, classifiers with a higher percentage of matching results are rated higher.

According to the above configuration, in order to classify the state of an article, it is possible to focus on a region similar to a region that a person pays attention to, and to highly evaluate a classifier with high classification accuracy.

According to one aspect of the present invention, a classifier that classifies the state of an article can be evaluated in terms of similarity with a person regarding a region of interest.

1 is a block diagram showing the physical configuration of an evaluation device according to a first embodiment of the present invention; FIG. 2 is a block diagram showing the functional configuration of the evaluation device shown in FIG. 1; FIG. 2 is a flowchart showing the flow of an evaluation method S1 executed by the evaluation device shown in FIG. 1; FIG. 4 is a diagram showing an example of a heat map generated in the creation step shown in FIG. 3; FIG. (a) to (c) are schematic diagrams showing an example of a region of interest by an inspector in each sample image. (d) to (e) are schematic diagrams showing regions of interest by a certain classifier indicated by heat maps of sample images corresponding to the classifier. (f) to (h) are schematic diagrams showing the region of interest by the other classifier indicated by the heat map of each sample image corresponding to the classifier. It is a figure which shows an example of evaluation of each classifier in the evaluation step shown in FIG. It is a block diagram showing a functional configuration of an inspection apparatus according to a second embodiment of the present invention. 8A is a diagram showing a region of interest by an inspector in the embodiment of the inspection apparatus shown in FIG. 7; FIG. (b) and (c) are diagrams showing regions of interest by each classifier. FIG. 5 is a diagram illustrating a classifier with a low evaluation in the embodiment of the inspection apparatus shown in FIG. 4;

[Embodiment 1]
(Physical configuration of evaluation device)
A physical configuration of the evaluation device 1 according to the first embodiment of the present invention will be described with reference to FIG. FIG. 1 is a block diagram showing the physical configuration of the evaluation device 1. As shown in FIG.

The evaluation device 1 is a computer having a bus 10, a main memory 11, a processor 12, an auxiliary memory 13, and an input/output interface 14, as shown in FIG. Main memory 11 , processor 12 , auxiliary memory 13 and input/output interface 14 are interconnected via bus 10 . As the main memory 11, for example, single or multiple semiconductor RAMs (random access memories) are used. Processor 12 may be, for example, a single or multiple microprocessors, a single or multiple digital signal processors, a single or multiple microcontrollers, or a combination thereof. As the auxiliary memory 13, for example, a single or multiple HDDs (Hard Disk Drives), single or multiple SSDs (Solid State Drives), or a combination thereof is used. Also, part or all of the auxiliary memory 13 may be storage on a network connected via a communication interface (not shown). As the input/output interface 14, for example, a USB (Universal Serial Bus) interface, a short-range communication interface such as infrared rays or Bluetooth (registered trademark), or a combination thereof is used.

For example, an input device 20 and an output device 30 are connected to the input/output interface 14 . As the input device 20, for example, a keyboard, mouse, touch pad, microphone, or a combination thereof is used. A display, a printer, a speaker, or a combination thereof is used as the output device 30, for example. Note that the evaluation device 1 may incorporate a keyboard and touch pad functioning as the input device 20 and a display functioning as the output device 30, like a notebook computer. Moreover, the evaluation device 1 may incorporate a touch panel functioning as the input device 20 and the output device 30 like a smart phone or a tablet computer.

The auxiliary memory 13 stores a program P for causing the processor 12 to execute an evaluation process S1, which will be described later. The processor 12 expands the program P stored in the auxiliary memory 13 onto the main memory 11, and executes each instruction included in the program P expanded onto the main memory 11, thereby executing the instructions included in the evaluation process S1, which will be described later. perform each step that is In addition, the auxiliary memory 13 stores various data referred to by the processor 12 in order to execute the evaluation process S1, which will be described later.

Here, a form in which the processor 12 executes the evaluation process S1, which will be described later, according to the program P stored in the auxiliary memory 13, which is an internal storage medium, has been described, but the present invention is not limited to this. That is, it is also possible to employ a form in which the processor 12 executes the evaluation process S1, which will be described later, according to the program P stored in the external recording medium. In this case, as the external recording medium, a computer-readable "non-transitory tangible medium" such as a tape, disk, card, semiconductor memory, or programmable logic circuit can be used. Alternatively, an embodiment may be adopted in which the processor 12 performs the evaluation process S1 described later according to a program P obtained from a network connected via a communication interface (not shown). In this case, the network may be, for example, the Internet, a wired LAN (Local Area Network), a wireless LAN, or a combination of at least some of them.

Moreover, although the form which implement|achieves the evaluation apparatus 1 using a single computer was demonstrated here, it is not limited to this. That is, a form in which the evaluation apparatus 1 is realized using a plurality of computers configured to be able to communicate with each other may be adopted. In this case, each step constituting the evaluation process S1, which will be described later, can be executed in parallel by these computers.

(Functional configuration of evaluation device)
A functional configuration of the evaluation device 1 will be described with reference to FIG. FIG. 2 is a block diagram showing the functional configuration of the evaluation device 1. As shown in FIG.

The evaluation device 1 includes, as shown in FIG. 2, a plurality of classifiers C1, C2, . Here, n is an arbitrary natural number representing the number of classifiers C1, C2, . . . , Cn. These blocks are functional blocks implemented by executing the instructions of the program P described above by the processor 12 described above.

Each classifier Ci (i=1, 2, . . . , n) refers to an image containing an article as a subject and performs classification processing for classifying the state of the article using an algorithm determined by machine learning (hereinafter referred to as "model ”). A model that can be used for the classification process is, for example, a neural network that receives an image including an article as an object and outputs the state of the article.

The model used by each classifier Ci is constructed by prior machine learning before the evaluation by the evaluation device 1 starts. In prior machine learning, (1) model selection, (2) model tuning (network structure and hyperparameter tuning), and (3) model machine learning are performed.

The creation unit 101 uses images I1, I2, . , 2, . . . , n) corresponding to the heat map Mij (j=1, 2, . Here, the heat map Mi refers to an image showing an area in the sample image Ij focused on by the classifier Ci in the classification process.

For example, the heat map Mij is an image that expresses the certainty of a specific class classification or the depth of association with color intensity. In this case, for example, in the heat map Mij, relatively dark areas represent areas in the sample image Ij focused on by the classifier Ci to classify the state of the article. Alternatively, the heat map Mij is an image in which the pixel value of each pixel is set according to the magnitude of the effect on the output of the classifier Ci. In this case, for example, areas with relatively high pixel values in the heat map Mij represent areas in the sample image Ij that the classifier Ci focused on to classify the state of the article. Hereinafter, in each sample image Ij, the area that the classifier Ci pays attention to for classifying the state of the article is also referred to as the "area of interest" by the classifier Ci. There may be one region of interest by the classifier Ci, or there may be multiple regions. For creating the heat map Mij, for example, Grad-CAM (Gradient-weighted Class Activation Mapping), which is a well-known algorithm, can be used.

Note that the creating unit 101 generates the heat map Mij corresponding to each classifier Ci only for the sample images Ij whose classification result by the classifier Ci matches correct information (indicating the result of classification by a person) ( The sample image Il for which the result of classification by the classifier Ci does not match the correct information may not be classified).

The evaluation unit 102 divides the region of interest by the classifier Ci indicated by the heat map Mij (that is, the region in the sample image Ij to which the classifier Ci focuses in order to classify the state of the article) and the A block that evaluates each classifier Ci based on comparison with regions in the sample image Ij of interest to the inspector. Hereinafter, the region in the sample image Ij that the inspector pays attention to is also referred to as the region of interest of the inspector. For example, the region of interest by the inspector is input via the input device 20 . Specifically, the evaluation unit 102 may display the sample image Ij on the display, and acquire a region specified by operating a touch pad, mouse, or the like as a region of interest for the inspector. The region of interest by the inspector may be at one point or at a plurality of points.

When the region of interest by the inspector and the region of interest by the classifier Ci are highly similar, the classifier Ci is a classifier with high generalization ability (or an improvement in generalization ability can be expected through additional learning in the future). It is considered to be Conversely, when the similarity between the region of interest by the inspector and the region of interest by the classifier Ci is low, the classifier Ci has low generalization ability (or it is expected that the generalization ability will be improved by additional learning in the future). difficult) classifier. The evaluation unit 102 evaluates each classifier Ci from this point of view. The evaluation result for each classifier Ci may be expressed in two stages such as good or bad, may be expressed in three stages or more, or may be expressed as a numerical value within a predetermined range.

For example, the evaluation unit 102 ranks a classifier Ci that includes a region of interest that overlaps with the region of interest of the inspector higher than the classifier Ck (k≠i) that does not, among the regions of interest of the classifier Ci. evaluate. Here, two regions overlap means that there is a common portion of these two regions. Hereinafter, among the regions of interest by the classifier Ci, a region of interest that overlaps the region of interest by the inspector is also referred to as an overlapping region.

In addition, the evaluation unit 102 selects a classifier Ci that includes both a region that overlaps with the region of interest of the inspector and a region that does not overlap with the region of interest of the inspector, in the region of interest by the classifier Ci. Evaluate it higher than the classifier Ck that is not. Hereinafter, among the regions of interest by the classifier Ci, a region of interest that does not overlap with the region of interest by the inspector is also referred to as a unique region. The unique region is a region of interest that the inspector does not focus on among the regions of interest by the classifier Ci, that is, a region of interest that the classifier Ci uniquely focuses on. Note that the evaluation unit 102 may determine, based on input information from the inspector, whether or not to consider a region of interest that does not overlap with the region of interest of the inspector, among the regions of interest by the classifier Ci, as a unique region. As a result, the evaluation unit 102 operates by regarding a region of interest, which the examiner believes to be a possible region of interest, among the regions of interest unique to such a classifier Ci as a unique region.

In addition, the evaluation unit 102 determines the rate at which the result of classifying the state of the article included as the subject in each sample image Ij matches the result of classifying the state of the article by the inspector (hereinafter, also referred to as the “accuracy rate”). ) higher than classifiers Ck that do not. In this case, the evaluation unit 102 evaluates each classifier Ci based on the result of comparison between the region of interest by the classifier Ci and the region of interest by the inspector, and the accuracy rate of the classifier Ci.

Specifically, the result of classifying the state of the article by the inspector is input to the evaluation unit 102 as correct information through the input device 20 . The evaluation unit 102 also receives the classification result output from each classifier Ci. Then, for each classifier Ci, the evaluation unit 102 calculates, as the correct answer rate, the rate at which the classification result of each sample image Ij output from the classifier Ci matches the correct information. For example, for each classifier Ci, the evaluation unit 102 sets the evaluation result based on the comparison between the region of interest by the classifier Ci and the region of interest by the inspector such that the higher the accuracy rate, the higher the evaluation result. weighting.

For each classifier Ci, the evaluation unit 102 compares the region of interest by the classifier Ci with the region of interest by the inspector, and compares the sample image Ij in which the classification result output from the classifier Ci matches the correct information. (no comparison is performed for non-matching sample images Il) may be employed.

(Evaluation method)
The evaluation method S1 executed by the evaluation device 1 will be described with reference to FIGS. 3 to 6. FIG.

FIG. 3 is a flow chart showing the flow of the evaluation method S1. The evaluation method S1 includes a creation step S101 and an evaluation step S102, as shown in FIG.

The creation step S101 is a step of creating a heat map Mij corresponding to each classifier Ci by using an image including, as a subject, an article whose state has been classified by an inspector as a sample image Ij. As described above, in the present embodiment, a plurality of sample images I1, I2, . . . , Im are input to each classifier Ci, and a plurality of heat maps Mi1, Mi2, . Here, the creating unit 101 creates a heat map for the sample image Ij whose classification result by the classifier Ci matches the correct information, and does not create a heat map for the sample image Il (l≠j) that does not match. .

The evaluation step S102 evaluates each classifier Ci is the step of evaluating As described above, in the present embodiment, the region of interest by the classifier Ci and the region of interest by the inspector are compared for each of the plurality of sample images I1, I2, . . . , Im. Also, for each classifier Ci, a correct answer rate is calculated, which is a rate at which the classification results of the plurality of sample images I1, I2, . Then, each classifier Ci is evaluated based on the comparison result and the accuracy rate, and the evaluation result is output.

FIG. 4 is a diagram showing an example of the heat map generated in the creation step S101. Here, sample images I1, I2, and I3 are input to each classifier Ci. Sample images I1, I2, and I3 include articles O1, O2, and O3 as subjects, respectively. Here, the correct information of the sample images I1, I2, and I3 are class A, class B, and class C, respectively.

The classification results of the sample image I1 by the classifiers C1 and C2 are correct in class A, respectively, and a heat map M11 corresponding to the classifier C1 and a heat map M21 corresponding to the classifier C2 are created.

The classification results of the sample image I2 by the classifiers C1 and C2 are correct in class B, respectively, and a heat map M12 corresponding to the classifier C1 and a heat map M22 corresponding to the classifier C2 are created.

The classification result of the sample image I3 by the classifier C1 is an incorrect answer in class B, and a heat map corresponding to the classifier C1 is not created. The classification result of the sample image I3 by the classifier C2 is correct in class C, and a heat map M23 corresponding to the classifier C2 is created.

An example of comparison processing in the evaluation step S102 will be described with reference to FIGS. 5 and 6. FIG.

FIGS. 5A, 5B, and 5C are schematic diagrams showing regions of interest R1, R2, and R3 of the inspector in the sample images I1, I2, and I3.

FIG. 5(d) shows the region of interest indicated by the heat map M11 described above. The heat map M11 includes two regions of interest P11 and P12. Among the regions of interest P11 and P12 by the classifier C1, the region of interest P11 is an overlapping region that overlaps the region of interest R1 by the inspector. Among the regions of interest P11 and P12 by the classifier C1, the region of interest P12 is a unique region that does not overlap with the region of interest R1 by the inspector.

FIG. 5(e) shows the region of interest indicated by the heat map M12 described above. The heat map M12 includes two regions of interest P21 and P22. Among the regions of interest P21 and P22 by the classifier C1, the region of interest P21 is an overlapping region that overlaps the region of interest R2 by the inspector. Among the regions of interest P21 and P22 by the classifier C1, the region of interest P22 is a unique region that does not overlap with the region of interest R2 by the inspector.

FIG. 5(f) shows the region of interest indicated by the heat map M21 described above. The heat map M21 includes one region of interest Q1. The region of interest Q1 by the classifier C2 does not overlap with the region of interest R1 by the inspector.

FIG. 5(g) shows the region of interest indicated by the heat map M22 described above. The heat map M22 includes one region of interest Q2. The region of interest Q2 by the classifier C2 does not overlap with the region of interest R2 by the inspector.

FIG. 5(h) shows the region of interest indicated by the heat map M23 described above. The heat map M23 includes one region of interest Q3. The region of interest Q3 by the classifier C2 is an overlapping region that overlaps the region of interest R3 by the inspector.

FIG. 6 is a table in which correct/incorrect answers, presence/absence of overlapping regions, and presence/absence of unique regions for each classifier Ci are arranged for each sample image Ij, and shows evaluation results of each classifier Ci. Note that if there is no overlapping area in the focused area indicated by the heat map Mij, the presence or absence of the unique area is not described.

In the example of FIG. 6, the evaluation unit 102 generates the evaluation result of each classifier Ci based on the result of comparing the n classifiers Ci based on the accuracy rate, overlapping area rate, and unique area rate. there is Here, the overlapping area rate is the ratio of the number of sample images Ij for which the classifier Ci has detected overlapping areas to the total number m of sample images Ij. The unique region rate is the ratio of the number of sample images Ij in which the classifier Ci has detected unique regions to the total number of sample images Ij. Also, in this example, a two-level evaluation result of "good" or "bad" is generated.

In addition, the evaluation unit 102 performs evaluation processing that emphasizes the overlapping area rate more than the correct answer rate and the unique area rate. For example, the evaluation unit 102 selects a classifier Ci that satisfies a condition of a high accuracy rate (e.g., a threshold value or more) among the classifiers Ci that satisfy a condition of a high overlapping area rate (e.g., the highest rate) among n classifiers. Assume that the evaluation result of the device Ci is "good". Then, the evaluation unit 102 evaluates the other classifiers Ck by comparing with the classifier Ci for which the evaluation result is “good”.

For example, the overlapping area rate of 60% for the classifier C1 is the highest among n. Also, the accuracy rate of 70% of the classifier C1 is equal to or higher than the threshold (for example, 70%). Therefore, the evaluation unit 102 sets the evaluation result of the classifier C1 as "good".

The classifier C2 has the same overlapping area rate of 60% as the classifier C1, but has a higher accuracy rate of 80% and a lower unique area rate of 10%. In this case, classifier C2 is neither superior nor inferior to classifier C1, and is determined to be equivalent. Therefore, the evaluation unit 102 sets the evaluation result of the classifier C2 as "good".

Also, the classifier C3 has the same overlapping area rate of 60% as the classifier C1, but both the accuracy rate of 60% and the unique area rate of 0% are lower. In this case, classifier C3 is determined to be inferior compared to classifier C1. Therefore, the evaluation unit 102 sets the evaluation result of the classifier C3 as "no".

Also, the classifier C4 has the same overlapping area rate of 60% and correct answer rate of 70% as the classifier C1, but has a lower unique area rate of 20%. In this case, classifier C4 is determined to be inferior compared to classifier C1. Therefore, the evaluation unit 102 sets the evaluation result of the classifier C4 as "no".

Also, the classifier C5 has the same overlapping area rate of 60% and unique area rate of 30% as the classifier C1, but has a lower accuracy rate of 60%. In this case, classifier C5 is determined to be inferior compared to classifier C1. Therefore, the evaluation unit 102 sets the evaluation result of the classifier C5 as "no".

Further, the classifier C6 has a lower overlapping area rate of 50%, the same correct answer rate of 70%, and a higher unique area rate of 40% than the classifier C1. In this case, since a low overlapping area rate is more important than a high unique area rate, the classifier C6 is determined to be inferior to the classifier C1. Therefore, the evaluation unit 102 sets the evaluation result of the classifier C6 as "no".

Also, the classifier Cn satisfies the condition that the overlapping area rate of 30% is extremely low (for example, less than or equal to the threshold). In this case, the evaluation unit 102 evaluates the classifier Cn as "no" without comparing the classifier Cn with other classifiers Ci.

(Effect of evaluation device 1)
The evaluation apparatus 1 according to the present embodiment evaluates a classifier Ci that includes an overlap region that overlaps the region focused by the inspector higher than the classifier Ck that does not, among the regions targeted by the classifier Ci. . In addition, the evaluation apparatus 1 according to the present embodiment classifies the classifier Ci, which further includes a unique region that does not overlap with the region focused by the inspector, from the classifier Ck that does not, in the region focused by the classifier Ci. appreciate it even more. Therefore, the evaluation apparatus 1 according to the present embodiment can evaluate each classifier Ci from the viewpoint of the similarity between the region targeted by the classifier Ci and the region targeted by the inspector.

(Modification 1)
In the evaluation method S1 according to the present embodiment, the evaluation unit 102 receives the region of interest by the inspector as input, and compares the region of interest by the classifier and the region of interest by the inspector. Not limited to this, in the evaluation method S1, the inspector may compare the region of interest by the classifier and the region of interest by the inspector. In this case, the evaluation unit 102 displays, on a display or the like, the region of interest by the classifier indicated by the heat map generated by the generation unit 101 . In addition, the evaluation unit 102 acquires, via the input device 30, information representing the above-described comparison result by the inspector who visually recognized the region of interest displayed on the display.

(Modification 2)
Further, in this embodiment, a configuration is adopted in which a plurality of classifiers C1, C2, . However, the invention is not so limited. For example, a configuration in which some or all of the plurality of classifiers C1, C2, . That is, the plurality of classifiers C1, C2, . . . , Cn are not essential components of the evaluation device 1.

[Embodiment 2]
In this embodiment, the evaluation apparatus 1 according to the first embodiment is used to configure an inspection apparatus 2 that inspects articles.

FIG. 7 is a block diagram showing the functional configuration of the inspection device 2. As shown in FIG. Since the physical configuration of the inspection device 2 is the same as that of the evaluation device 1 described with reference to FIG. 1, detailed description thereof will be omitted.

As shown in FIG. 7 , the inspection device 2 includes an evaluation device 1 and a selection section 203 . The inspection device 2 uses the classifier Ci selected by the selection unit 203 to inspect the article.

The selection unit 203 is a block that selects one of the classifiers Ci from the plurality of classifiers C1, C2, . For example, the selection unit 203 may select one or more classifiers Ci whose evaluation results satisfy a predetermined condition (for example, the evaluation results are "good").

Specifically, the selection unit 203 is configured so that the image input to the inspection apparatus 2 is input to the selected classifier Ci, and the output from the selected classifier Ci is output to the outside as the inspection result. , switching processing is performed. The switching process by the selection unit 203 is performed after the evaluation by the evaluation device 1 is completed and before the operation of inspecting the article by the inspection device 2 is started. Further, a process of additionally learning the selected one or more classifiers Ci may be further executed before the operation of the inspection apparatus 2 is started.

[Example 1]
Example 1 using the inspection device 2 will be described with reference to FIG. In Example 1, evaluation was performed by the evaluation device 1 using a sample image I1 including a crimp terminal as an article as a subject.

Each classifier Ci is constructed by prior machine learning so as to classify the state of the crimp terminal into one of the following eight states based on the image including the crimp terminal as an object.

Condition 1: Good condition Condition 2: Shallow stroke Condition 3: Deep stroke Condition 4: Barrel curled Condition 5: Protrusion to the transition part Condition 6: Protrusion to the inset side Condition 7: Wire breakage Condition 8: Bad bell mouth Fig. 8 (a) ) is a diagram showing a region of interest by an inspector in the sample image I1. In this example, the inspector focuses on the area containing the barrel of the crimp terminal.

In the present embodiment, the classifiers C1 and C2 both classify the crimp terminal included as a subject in the sample image I1 described above as "state 4: barrel turn-up", and the classification result of the state classified by the inspector is Outputs matching classification results.

FIG. 8(b) is a diagram showing two regions of interest by the classifier C1. Here, one region of interest by the classifier C1 is an overlapping region that overlaps the region of interest by the inspector. The other region of interest is a unique region that does not overlap with the region of interest of the inspector.

FIG. 8(c) is a diagram showing one region of interest by the classifier C2. Here, the region of interest by the classifier C2 does not overlap with the region of interest by the inspector.

FIG. 9 is a diagram explaining the evaluation of the classifier C2. As shown in FIG. 9, although the classification result by the classifier C2 matches the classification result by the inspector, the evaluation apparatus 1 does not include an overlapping region in the region of interest by the classifier C2. Lower the evaluation than C1. Such a classifier C2 may have obtained a correct classification result by chance based on an image portion having a weak relationship with the examiner's criteria, and there is a possibility that the accuracy rate will not improve even if additional learning is performed. be. The inspection device 2 can inspect the article using the classifier C1 suitable for additional learning without selecting such a classifier C2.

[Additional notes]
The present invention is not limited to the above-described embodiments, but can be modified in various ways within the scope of the claims, and can be obtained by appropriately combining technical means disclosed in different embodiments. is also included in the technical scope of the present invention.

1 evaluation device 2 inspection device 10 bus 11 main memory 12 processor 13 auxiliary memory 14 input/output interface 20 input device 30 output device C1, C2, .

Claims (8)

An evaluation device for evaluating each of a plurality of classifiers that refer to an image containing an article as a subject and perform classification processing for classifying the state of the article according to an algorithm determined by learning,
a creation unit that creates a heat map corresponding to each classifier, which indicates an area that the classifier pays attention to in the classification process in a sample image including, as a subject, an article whose state has been classified by a person;
an evaluation unit that evaluates each classifier based on a comparison of the area in the sample image indicated by the corresponding heat map and the area in the sample image that the person focuses on to classify the state of the article. ing,
An evaluation device characterized by: The evaluation unit selects a classifier that includes, among the regions in the sample image indicated by the corresponding heat map, a region that overlaps with the region in the sample image that the person focuses on in order to classify the state of the article. , rate higher than the classifier not included,
The evaluation device according to claim 1, characterized by: The evaluation unit selects a classifier that includes an area that does not overlap with the area in the sample image that the person focuses on in order to classify the state of the article, in the area in the sample image indicated by the corresponding heat map. , evaluates more highly than classifiers that do not contain
3. The evaluation device according to claim 2, characterized in that: The creation unit creates a heat map corresponding to each classifier for each sample image, using each of a plurality of images including articles classified by a person as a subject as a sample image,
The evaluation unit evaluates each classifier based on the evaluation result of evaluating each classifier for each sample image,
4. The evaluation device according to any one of claims 1 to 3, characterized in that: The evaluation unit selects, from among the plurality of classifiers, a classifier having a higher rate that the result of classifying the state of the article included as the subject in each sample image matches the result of the classification of the state of the article by the person. further appreciate the
5. The evaluation device according to claim 4, characterized in that: An evaluation method for evaluating each of a plurality of classifiers that refer to an image containing an article as a subject and perform classification processing for classifying the state of the article according to an algorithm determined by learning,
a creation step of creating a heat map corresponding to each classifier, the heat map indicating an area of interest to the classifier in the classification process in a sample image including, as a subject, an article whose state has been classified by a person;
an evaluation step of evaluating each classifier based on a comparison of the area in the sample image indicated by the corresponding heat map to the area in the sample image that the person focuses on to classify the condition of the article. ing,
An evaluation method characterized by 6. An evaluation program that causes a computer to operate as the evaluation apparatus according to any one of claims 1 to 5, wherein the computer functions as each part of the evaluation apparatus. An inspection device for inspecting an article,
the plurality of classifiers;
an evaluation device according to any one of claims 1 to 5;
a selection unit that selects one of the plurality of classifiers based on the evaluation result of the evaluation device;
An inspection device that inspects the article using the classifier selected by the selection unit.

Images