JP6995031B2 - Information processing equipment, information processing methods, and information processing programs - Google Patents

Description

The present invention relates to a determination system that makes a determination regarding an object based on a classification result that automatically classifies an object in an image, and more particularly to an information processing device that manages the determination result in the determination system.

A technique for automatically classifying an object in an image using a machine-learned algorithm is known. However, the accuracy of automatic classification is not 100%. For this reason, classification accuracy is ensured by using visual judgment by humans together. For example, in the following Patent Document 1, a label indicating the content and the category of the content is displayed, and when the user modifies the label, the label of the content is updated with the modified label and then the teacher data is used. The configuration to be used is described.

Japanese Patent No. 6291844

In the above-mentioned conventional technique, the result of the visual determination is only used for updating the label, and it is not supposed to be used for other purposes. One aspect of the present invention is to provide an information processing apparatus or the like capable of expanding the range of utilization of the result of visual determination.

In order to solve the above problems, the information processing apparatus according to one aspect of the present invention classifies an object in an image by a machine-learned classification unit, and whether or not the object belongs to a predetermined classification. An information processing device that classifies the classification results of the classification unit in the determination system for determining, and in a visual determination performed on an object classified by the classification unit, (1) the predetermined classification of the object. The visual judgment result acquisition unit that acquires the visual judgment result, which is the result of determining whether or not the classification result belongs to, and (2) the correctness of the classification result by the classification unit, and the above (1) and (2) in the visual determination result. ) Is provided with a classification unit for classifying the classification results of the above classification unit.

In order to solve the above problem, the information processing method according to one aspect of the present invention classifies an object in an image by a machine-learned classification unit, and whether or not the object belongs to a predetermined classification. It is an information processing method by an information processing apparatus that classifies the classification result of the above-mentioned classification unit in the determination system for determining, and in the visual determination performed on the object classified by the above-mentioned classification unit, (1) the above-mentioned object. The visual judgment result acquisition step for acquiring the visual judgment result which is the result of determining whether or not the classification belongs to the predetermined classification and (2) the correctness of the classification result, and the above (1) and (1) in the visual judgment result. 2) Includes a classification step for classifying the classification results of the classification unit according to the combination of 2).

According to one aspect of the present invention, the range of utilization of the visual determination result can be expanded.

It is a block diagram which shows an example of the main part structure of the information processing apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the structural example of the information processing system including the said information processing apparatus. It is a figure which shows the example which photographed the food produced in the food production factory as the determination object of the said information processing system. It is a figure which shows the pattern classification of the determination result of the said information processing apparatus. It is a figure which shows the UI screen example in the visual determination. It is a figure which shows the pattern classification of the determination result when the classification by the classification part could not be performed. It is a flowchart which shows an example of the process which the information processing apparatus determines the correctness of a product, and classifies the pattern of the determination result. It is a flowchart which shows an example of the division processing performed in S13 of FIG. It is a flowchart which shows an example of the process which the said information processing apparatus generates training data based on the classification of the said pattern. It is a flowchart which shows an example of the process which the information processing apparatus which concerns on Embodiment 2 of this invention calculates the number of erroneous determinations. It is a figure explaining the process which determines the timing of relearning by the information processing apparatus which concerns on Embodiment 3 of this invention. It is a block diagram which shows the structural example of the control part of the information processing apparatus which concerns on Embodiment 4 of this invention. It is a figure explaining the process executed by the information processing apparatus.

It is a figure explaining the process to execute.
[Embodiment 1]
An embodiment of the present invention will be described with reference to FIGS. 1 to 9.

〔System configuration〕
First, the configuration of the information processing system 500 according to the present embodiment will be described with reference to FIG. FIG. 2 is a diagram showing a configuration example of the information processing system 500. The information processing system 500 has a function of determining the correctness of a predetermined object from an image of the object. As shown in the figure, the information processing system 500 includes an information processing device 1, a visual inspection terminal 2, and a terminal device 3.

The information processing apparatus 1 automatically classifies the object into one of a plurality of predetermined classification categories by a predetermined algorithm from the captured image of the object. Then, the information processing apparatus 1 determines whether or not the object is correct or not depending on whether or not the classification result matches a predetermined classification category. That is, the information processing apparatus 1 determines that the classification result matches the predetermined classification category as positive (there is no problem with the object), and if the classification result does not match, it determines as negative (there is a problem with the object).

The visual inspection terminal 2 is a device for visually determining an object that is not judged to be good by the information processing apparatus 1. Further, the visual inspection terminal 2 also targets images that were difficult to classify by the information processing apparatus 1 for visual determination. The information processing system 500 uses both the automatic determination by the information processing apparatus 1 and the visual determination by the visual inspection terminal 2 in combination to improve the efficiency of the determination and the reliability of the determination.

The terminal device 3 acquires an image of an object and transmits it to the information processing device 1, and receives a determination result based on the image from the information processing device. The terminal device 3 may have a function of outputting a determination result by an image, voice, or the like.

FIG. 2 shows an example in which the quality determination of the product P manufactured in the manufacturing factory is performed by the information processing system 500 arranged in the monitoring room. Therefore, the terminal device 3 acquires the image 201 of the product P and transmits it to the information processing device 1. Then, the information processing apparatus 1 that has received the image 201 determines the quality of the product P, which is the object of the determination.

If it is determined to be a non-defective product, the information processing device 1 notifies the terminal device 3 of the determination result, whereby the manufacturing factory recognizes that the product P is a non-defective product. On the other hand, when it is determined that the product is defective, the image 201 whose determination result is NG (defective) is transmitted from the information processing apparatus 1 to the visual inspection terminal 2.

The visual inspection terminal 2 displays the image 201, and the visual inspector visually inspects the displayed image 201 to determine the quality of the product P, and inputs the result to the visual inspection terminal 2. Then, the visual inspection terminal 2 notifies the information processing apparatus 1 of the input visual determination result. As a result, the information processing apparatus 1 can detect when the result of the defect determination of the own machine is incorrect.

The information processing device 1 and the visual inspection terminal 2 may be arranged in a facility (in the manufacturing factory in the illustrated example) where the object is located. In this case, the terminal device 3 may be omitted from the components of the information processing system 500, or the information processing device 1 may be configured as a device integrated with the visual inspection terminal 2. Further, each device included in the information processing system 500 may be a stationary device or a portable device. Further, the number of visual inspection terminals 2, the number of terminal devices 3, and the number of visual inspectors may be plural.

〔Device configuration〕
A more detailed configuration of the information processing apparatus 1 will be described with reference to FIG. FIG. 1 is a block diagram showing an example of a main configuration of the information processing apparatus 1. As shown in the figure, the information processing apparatus 1 includes a control unit 10 that controls and controls each unit of the information processing apparatus 1, and a storage unit 20 that stores various data used by the information processing apparatus 1. Further, the information processing device 1 has an input unit 30 for receiving data input to the information processing device 1, a communication unit 40 for the information processing device 1 to communicate with another device, and the information processing device 1 for outputting data. The output unit 50 is provided.

Further, the control unit 10 includes an image acquisition unit 101, a classification unit 102, a classification success / failure determination unit 103, a correctness / rejection determination unit (judgment unit) 104, a determination result management unit (classification unit) 105, and a visual determination unit (visual determination result acquisition). Unit) 106, training data generation unit 107, erroneous determination number calculation unit 108, and re-learning time determination unit 109 are included. Further, the image 201 and the training data 202 are stored in the storage unit 20.

The image acquisition unit 101 acquires an image of an object to be determined by the information processing apparatus 1. In the present embodiment, the image obtained by photographing the object is input to the information processing apparatus 1 via the input unit 30 and stored as the image 201 in the storage unit 20. Therefore, the image acquisition unit 101 acquires the image 201 from the storage unit 20. Of course, the method by which the image acquisition unit 101 acquires the image 201 is not particularly limited. For example, the image acquisition unit 101 may control the image pickup device to image an object and acquire the image 201 from the image pickup device.

The classification unit 102 classifies the object into any of a plurality of predetermined classification categories. This classification is performed based on the result of machine learning of the correspondence between the image and the classification category of the object appearing in the image. The algorithm applicable to the classification unit 102 is not particularly limited, but when high determination accuracy is desired, it is preferable to use a machine learning algorithm such as a neural network, and in particular, a deep-learned neural network (for example, ResNet: Residual Network or the like). It is more preferable that the determination is made by. In the present embodiment, an example will be described in which the classification unit 102 is a ResNet that outputs the scores of each of a plurality of predetermined classification categories by using an image of an object as input data. This score is a numerical value indicating the certainty (probability) that the object shown in the image corresponds to each classification category, and the classification category with the highest score is the most probable as the classification category of the object. Therefore, the classification category with the highest score is the classification result of the classification unit 102.

The classification success / failure determination unit 103 determines whether or not the classification result of the classification unit 102 satisfies a predetermined evaluation criterion. Specifically, the classification success / failure determination unit 103 determines whether or not the evaluation criteria are satisfied by comparing the score output by the classification unit 102 with the threshold value. Then, the classification success / failure determination unit 103 determines that the classification of the classification unit 102 fails if none of the scores output by the classification unit 102 for each classification category satisfy the evaluation criteria, and the score satisfying the evaluation criteria is If there is, it is judged that the classification is successful.

The correctness determination unit 104 determines whether or not the object belongs to a predetermined classification category. Specifically, the correctness determination unit 104 collates the classification result of the classification unit 102 with the information indicating the classification category of the object that should be originally shown in the image 201, and determines the correctness of the object shown in the image 201. do. For example, when performing a correctness determination based on an image 201 obtained by photographing a product manufactured on the production line of the product A, the correctness determination unit 104 determines that the product shown in the image 201 is a correct product if it is classified as the product A. Judgment is made, and if it is classified into another product, it is judged to be an erroneous product.

The determination result management unit 105 divides the classification results of the classification unit 102 into predetermined patterns and manages them. Although the details will be described later, in the visual determination performed on the classified object by the classification unit 102, (1) whether or not the object belongs to a predetermined classification and (2) whether or not the classification result by the classification unit 102 is correct or not are determined. It is judged. The determination result management unit 105 classifies the classification results of the classification unit 102 into predetermined patterns according to the combination of the above (1) and (2) in the visual determination result.

The visual determination unit 106 causes the visual inspection terminal 2 to display the image 201, causes the visual inspector to make a visual determination, and acquires the visual determination result via the visual inspection terminal 2. Although the details will be described later, the visual determination unit 106 is the result of visually determining (1) whether or not the object belongs to a predetermined classification and (2) whether or not the classification result of the classification unit 102 is correct or not. Acquire the judgment result. As described above, these visual determination results are used for the above classification by the determination result management unit 105.

The training data generation unit 107 associates the classification category of the object reflected in the image 201 with the image 201 as correct answer data, and generates training data to be used for machine learning of the classification unit 102. Although the details will be described later, the training data generation unit 107 can automatically generate correct training data by using appropriate correct answer data according to the image 201 according to the above classification by the determination result management unit 105. ..

The erroneous determination number calculation unit 108 is determined by the correctness determination unit 104 to belong to a predetermined classification, but the object does not actually belong to the predetermined classification, and the classification unit 102 is erroneously classified as belonging to the predetermined classification. Calculate the number of times (number of erroneous judgments). The erroneous determination number calculation unit 108 will be described in the second embodiment. If it is not necessary to calculate the number of erroneous determinations, the erroneous determination number calculation unit 108 may be omitted.

The re-learning time determination unit 109 determines the time for re-learning of the classification unit 102. The re-learning time determination unit 109 will be described in the third embodiment. If it is not necessary to determine the re-learning time, the re-learning time determination unit 109 may be omitted.

As described above, the image 201 is an image of an object to be determined, and the training data 202 is training data used for machine learning of the classification unit 102. The training data generation unit 107 generates training data from the image 201 that is the target of the determination of the information processing apparatus 1, and the training data is stored as the training data 202 and used for re-learning of the classification unit 102. The training data 202 includes training data not generated by the training data generation unit 107 (for example, training data obtained by processing the training data generated by the training data generation unit 107, training data manually created by the user, or the like). May be included.

Among the components of the information processing apparatus 1 described above, at least the classification unit 102 and the correctness determination unit 104 classify the object in the image to determine whether or not the object belongs to a predetermined classification. A judgment system has been constructed. These components may be provided in a device different from the information processing device 1. In this case, the determination result management unit 105 acquires the classification result of the classification unit 102 and the determination result of the correctness determination unit 104, and performs the above-mentioned classification.

[Example of application of information processing system]
The use (determination target) of the information processing system 500 is not particularly limited, but it can be used, for example, for quality control of a food production factory. This will be described with reference to FIG. FIG. 3 is a diagram showing an example in which a food product produced in a food production factory is photographed as a determination target of the information processing system 500.

FIG. 3A shows a part of the food production line 600. The production line 600 includes a transport unit 602 that transports the food 601 in the direction of the white arrow in the figure. Further, the transport unit 602 is provided with an image pickup table 604 that holds the image pickup device 603 above the transport unit 602. As a result, the food 601 transported by the transport unit 602 can be photographed from directly above.

FIG. 3B shows an example of an image 201 taken by the image pickup apparatus 603 of FIG. 3A. This image 201 shows food 601 which is packed meat. Image 201 also shows the label 6011 and the price tag sticker 6012 affixed to the pack of food 601.

The information processing system 500 determines the classification category of food 601 on the production line 600 (whether it is pork or beef, for roasted meat, chopped meat, etc.) on the production line 600. It is possible to confirm that the correct food 601 is being transported. Since high accuracy is required for quality control in food production factories, it is preferable to apply the information processing system 500 that outputs extremely reliable determination results.

[Judgment result pattern]
The determination result of the information processing apparatus 1 can be classified into five patterns. Then, the determination result management unit 105 determines which pattern the determination result of the information processing apparatus 1 corresponds to. This will be described with reference to FIG. FIG. 4 is a diagram showing pattern classification of the determination result of the information processing apparatus 1. Here, an example in which the information processing apparatus 1 determines whether the product manufactured at the factory is a correct product (a product to be manufactured) or an incorrect product (a product different from the product to be manufactured). Is assumed.

In the example of FIG. 4, the product ID of the correct product is "ID_A". The factory is supposed to manufacture products with "ID_A", but different products may be manufactured by mistake. The product ID of the product manufactured by mistake is "ID_B". The classification unit 102 determines the classification category (product ID) of the manufactured product from the image of the manufactured product. Then, the correctness determination unit 104 determines that the product is correct if the classification result of the classification unit 102 is "ID_A", and that the product is incorrect if the classification result of the classification unit 102 is other than "ID_A". judge.

Therefore, as shown in the record in the first row of FIG. 4A, if the product of "ID_A" is manufactured and the classification result of the classification unit 102 is also the product of "ID_A", the correct product is manufactured. , It means that the classification unit 102 was able to classify correctly. In FIG. 4, the name of this pattern is "T_OK". In the pattern of "T_OK", since the classification result of the classification unit 102 is "ID_A", the correctness determination unit 104 determines that the product is correct.

On the other hand, as shown in the record in the second row of FIG. 4A, if the product of "ID_B" is manufactured and the classification result of the classification unit 102 is also the product of "ID_B", an erroneous product is manufactured. However, it means that the classification unit 102 was able to classify correctly. In FIG. 4, the name of this pattern is "T_NG". In the pattern of "T_NG", since the classification result of the classification unit 102 is "ID_B", the correctness / rejection determination unit 104 determines that the product is incorrect.

Here, as shown in the record on the third row of FIG. 4A, when the product of "ID_B" is manufactured, the classification result of the classification unit 102 may be the product of "ID_A". .. In this case, an erroneous product was manufactured, and the classification of the classification unit 102 was also erroneous, but the classification result matched the correct product. In FIG. 4, the name of this pattern is "F_OK". In the pattern of "F_OK", since the classification result of the classification unit 102 is "ID_A", the correctness / rejection determination unit 104 determines that the product is correct, but the correct product is not actually manufactured. That is, the determination result of the correctness determination unit 104 is an erroneous determination.

Further, as shown in the record on the fourth row of FIG. 4A, if the product of "ID_A" is manufactured and the classification result of the classification unit 102 is "ID_B", the correct product is manufactured. However, it means that the classification of the classification unit 102 was incorrect. In FIG. 4, the name of this pattern is "F_NG1". In the pattern of "F_NG1", since the classification result of the classification unit 102 is "ID_B", the correctness determination unit 104 determines that the product is incorrect. In this case, since the visual judgment is performed, it is confirmed by the visual judgment that the correct product is manufactured.

Further, as shown in the record on the fifth row of FIG. 4A, if the product of "ID_B" is manufactured and the classification result of the classification unit 102 is "ID_C" (other than "ID_A"). , The wrong product is manufactured, the classification of the classification unit 102 is also wrong, and the classification result does not match the correct product. In FIG. 4, the name of this pattern is "F_NG2". In the pattern of "F_NG2", since the classification result of the classification unit 102 is "ID_C", the correctness / rejection determination unit 104 determines that the product is incorrect. In this case, since the visual judgment is performed, it is confirmed by the visual judgment that an erroneous product is manufactured.

The determination result management unit 105 manages the classification result of the classification unit 102 or the image 201 used for the classification by using the visual determination result by classifying them into the pattern as described above. Specifically, as shown in FIG. 4B, in the visual determination, a visual inspector is made to determine whether the correct product is manufactured and whether the classification unit 102 correctly classifies the product.

Specifically, when the correct product is manufactured by the visual determination and the classification of the classification unit 102 is also determined to be correct, it corresponds to the pattern of "T_OK". Further, when it is determined by visual determination that the correct product is manufactured but the classification of the classification unit 102 is incorrect, it corresponds to the pattern of "F_NG1". Further, when it is determined by visual determination that an erroneous product is manufactured, but the classification of the classification unit 102 is correct, it corresponds to the pattern of "T_NG". Then, when it is determined by visual determination that an erroneous product is manufactured and the classification of the classification unit 102 is also erroneous, it corresponds to the pattern of "F_NG2" or "F_OK".

In this way, the determination result management unit 105 manages the classification result of the classification unit 102 or the image 201 used for the classification by using the visual determination result by classifying them into the pattern as described above. Can be done. Further, in the visual judgment, the visual inspector only needs to confirm two points, that is, whether the correct product is manufactured and whether the correct classification is made, so that the load related to the visual judgment can be minimized. As a result, quick visual determination can be realized.

[UI screen example in visual judgment]
An example of a UI (User Interface) screen in visual determination will be described with reference to FIG. FIG. 5 is a diagram showing an example of a UI screen in visual determination. In the screen example of (a) in the figure, the correct product image and the captured image are displayed. Further, in this screen example, as question 1, a question sentence asking whether the products shown in the above two images match, and the answer options (YES / NO) are displayed.

When the visual inspection terminal 2 displays such a screen, the visual inspector can compare the images on the screen and input either YES or NO to the visual inspection terminal 2. Further, the visual inspection terminal 2 notifies the information processing apparatus 1 of the input contents of the visual inspector, whereby the determination result management unit 105 of the information processing apparatus 1 can specify whether or not the correct product is manufactured. ..

Further, in the screen example of FIG. 5B, the captured image and the classification result of the classification unit 102 are displayed. Further, in this screen example, as question 2, a question sentence asking whether the classification result is correct and an answer option (YES / NO) are displayed.

When such a screen is displayed by the visual inspection terminal 2, the visual inspector can compare the image on the screen with the classification result and input either YES or NO to the visual inspection terminal 2. Then, the visual inspection terminal 2 notifies the information processing apparatus 1 of the input contents of the visual inspector, whereby the determination result management unit 105 of the information processing apparatus 1 can specify the correctness of the classification result.

Therefore, by sequentially displaying the UI screens as shown in FIGS. 5A and 5 on the visual inspection terminal 2, the determination result management unit 105 determines whether or not the correct product is manufactured and whether or not the classification result is correct or not. Can be identified. As a result, as shown in FIG. 4B, the determination result management unit 105 can determine the pattern from the combination of whether or not the correct product is manufactured and whether or not the classification is correct or not.

[About other patterns]
When the product shown in the image 201 is a product for which the classification unit 102 has not learned the classification category, the classification by the classification unit 102 cannot be performed, so that a visual determination is required. Then, the classification of the pattern based on the visual determination result is simpler than the example of FIG. 4 because the classification by the classification unit 102 has not been completed. This will be described with reference to FIG. FIG. 6 is a diagram showing a pattern classification of a determination result when the classification by the classification unit 102 could not be performed.

As shown in FIG. 6A, when the classification by the classification unit 102 cannot be performed, the pattern "OK" that the manufactured product is the correct product and the manufactured product is the wrong product. There are two patterns of the pattern "NG".

In this case, as shown in (b) of the figure, in the visual determination, it is determined whether or not the correct product is manufactured. In this visual determination, for example, a UI screen as shown in FIG. 5A may be displayed. If the correct product is manufactured, the pattern is "OK", and if the manufactured product is incorrect, the pattern is "NG".

[Processing flow (product correctness judgment and pattern classification)]
The flow of processing executed by the information processing apparatus 1 will be described with reference to FIG. 7. FIG. 7 is a flowchart showing an example of a process (information processing method) executed by the information processing apparatus 1. In the following, an example in which the information processing apparatus 1 determines whether or not the manufactured product (specifically, the food product 601 in FIG. 3) is correct will be described.

In S1, the image acquisition unit 101 acquires the image 201 stored in the storage unit 20. The image 201 may be, for example, an image of the food 601 taken by the image pickup apparatus 603 (see FIG. 3). Further, the correctness determination unit 104 acquires the product ID of the product shown in the image 201. In the production line 600 as shown in FIG. 3, since the predetermined food 601 is manufactured and transported, the product ID of the transported food 601 can be obtained. However, since it is possible that the food 601 that should not be originally produced is erroneously manufactured and enters the production line 600, the information processing apparatus 1 makes a determination and erroneously detects the food 601 that has entered the production line 600. ..

In S2, the classification unit 102 classifies the products shown in the image 201 acquired in S1. Specifically, the classification unit 102 calculates a score indicating the certainty corresponding to the classification category for each of the plurality of preset classification categories (each corresponding to a different product ID). The arithmetic processing in this determination may increase the processing load. Therefore, the classification unit 102 may cause an external server to perform this calculation process and acquire the score as the calculation result from the server.

In S3, the classification success / failure determination unit 103 determines whether or not the determination result of S2 includes a score equal to or greater than the threshold value. When the classification success / failure determination unit 103 determines that the category is included (YES in S3), the classification success / failure determination unit 103 is notified of the classification category whose score exceeds the threshold value, and the process proceeds to S4. On the other hand, when the classification success / failure determination unit 103 determines that the score equal to or higher than the threshold value is not included (NO in S3), the determination result is unknown, and the image 201 acquired in S1 is transmitted to the visual determination unit 106. The process proceeds to S12. The classification success / failure determination unit 103 may output the determination result of unknown via the output unit 50 or the like, or may notify the terminal device 3 of the determination result and output the determination result.

In S4, the correctness / rejection determination unit 104 determines whether or not the classification result and the product ID match. Specifically, the correctness / rejection determination unit 104 determines whether or not the classification category notified from the classification success / failure determination unit 103 and the product classification category indicated by the product ID acquired in S1 match. Further, the correctness / rejection determination unit 104 may output the determination result to the output unit 50, or may notify the terminal device 3 of the determination result and output the determination result. If it is determined that the correctness / rejection determination unit 104 matches (YES in S4), the process proceeds to S5. On the other hand, when it is determined that the correctness / rejection determination unit 104 does not match (NO in S4), the process proceeds to S6.

The mode of output of the determination result is arbitrary. For example, an image showing the correctness of the product may be displayed. In addition, the score of the classification category may be displayed. Further, the output may be performed in a mode other than the display. For example, when the product is erroneous, the terminal device 3 may output a sound such as a buzzer to notify that there is food 601 erroneously on the production line 600. Further, for example, when the production line 600 is provided with a mechanism for branching the transport path of the food 601 being transported, the terminal device 3 controls the mechanism to obtain the food 601 determined to be erroneous. It may be excluded from the production line 600.

In S5, the determination result management unit 105 determines that the classification is "T_OK". Then, the determination result management unit 105 stores the determination result in association with the image 201 acquired in S1. For example, the determination result management unit 105 may store the identification information (ID) of the image 201 acquired in S1 in association with the determination result, and may store the additional information of the image 201, for example, the header of the image 201 or the like. The determination result may be stored in. When the processing of S5 is completed, the processing of the information processing system 500 is completed.

In S6 (visual determination result acquisition step), the visual determination unit 106 makes a visual determination. Specifically, the visual determination unit 106 transmits the image 201 acquired in S1 to the visual inspection terminal 2, and notifies the visual inspection terminal 2 of the product ID acquired in S1 and the classification result of S2. Then, the visual determination unit 106 acquires the result of the visual determination by the visual inspector from the visual inspection terminal 2, and notifies the determination result management unit 105 of the result. Then, the determination result management unit 105 classifies the classification result of S2 by the processing (classification step) of S7 to S11 described below based on the acquired visual determination result.

In the visual determination, the visual inspection terminal 2 may display a screen as shown in FIG. 5, for example, and accept the input of the visual determination result. Further, such a screen may be configured to be generated by the visual inspection terminal 2 or may be generated by the visual determination unit 106 and transmitted to the visual inspection terminal 2.

In S7, the determination result management unit 105 determines whether or not the classification is correct but the product is incorrect. If the classification is correct but the product is determined to be incorrect (YES in S7), the process proceeds to S8. On the other hand, if it is determined that the product is correct or the classification is incorrect (NO in S7), the process proceeds to S9. The fact that the product is correct means that the product shown in the image 201 is the correct product (the product with the product ID acquired in S1). Further, the correct classification means that the classification unit 102 correctly classifies (the product shown in the image acquired in S1 belongs to the classification category of the score that is equal to or higher than the threshold value in S3). That is, in the determination result management unit 105, the answer to question 1 (see (a) in FIG. 5) in the visual determination in S6 is NO, and the answer to question 2 (see (b) in FIG. 5) is YES. If there is, it is determined as YES in S7. On the other hand, if the answer to question 1 is YES or the answer to question 2 is NO, the determination result management unit 105 determines NO in S7.

In S8, the determination result management unit 105 determines that the classification is "T_NG". Then, the determination result management unit 105 stores the determination result in association with the image 201 acquired in S1. When the processing of S8 is completed, the processing of the information processing system 500 is completed.

In S9, the determination result management unit 105 determines whether or not the product is correct but the classification is incorrect. If it is determined that the product is correct but the classification is incorrect (YES in S9), the process proceeds to S10. On the other hand, if it is determined that both the product and the classification are incorrect (NO in S9), the process proceeds to S11. Specifically, if the answer to question 1 in the visual determination of S6 is YES and the answer to question 2 is NO, the determination result management unit 105 determines YES in S9, and the answers to questions 1 and 2 are answered. If both are NO, it is determined as NO in S9.

In S10, the determination result management unit 105 determines that the classification is "F_NG1". Then, the determination result management unit 105 stores the determination result in association with the image 201 acquired in S1. When the processing of S10 is completed, the processing of the information processing system 500 is completed. In the pattern of "F_NG1", the judgment result of the correctness judgment unit 104 is wrong because the classification is wrong, but the actually manufactured product is correct (see FIG. 4), so the judgment result. The management unit 105 may correct the result of the correctness determination. For example, the determination result management unit 105 may notify the terminal device 3 of the corrected determination result and output the product so that the product is treated as a non-defective product.

In S11, the determination result management unit 105 determines that the classification is "F_NG2". Then, the determination result management unit 105 stores the determination result in association with the image 201 acquired in S1. When the processing of S11 is completed, the processing of the information processing system 500 is completed.

In S12, as in S6, the visual determination unit 106 makes a visual determination. Specifically, the visual determination unit 106 transmits the image 201 acquired in S1 to the visual inspection terminal 2, and also transmits the product ID acquired in S1 and the classification result of S2 (the classification category having the highest score). Notify the visual inspection terminal 2. Then, the visual determination unit 106 acquires the result of the visual determination by the visual inspector from the visual inspection terminal 2, and notifies the determination result management unit 105 of the result.

In S13, the determination result management unit 105 performs the classification process. The details will be described later based on FIG. 8, but the classification result of the classification unit 102 is divided into a plurality of patterns by the processing of S13, and the determination result of S12 and the classification of S13 are associated with the image 201 acquired in S1. It will be remembered. When the processing of S13 is completed, the processing of the information processing system 500 is completed. Further, the determination result management unit 105 may output the determination result of the visual determination of S12 to the output unit 50, or may notify the terminal device 3 to output the determination result.

As described above, the visual determination unit 106 determines (1) whether or not the object belongs to the above classification category and (2) classification in the visual determination (S6, S12) of the object that has been classified by the classification unit 102. The visual determination result, which is the result of determining whether the classification result of the unit 102 is correct or not, is acquired. Then, the determination result management unit 105 classifies the classification results of the classification unit 102 according to the combination of the above (1) and (2) in the visual determination result (S7 to S11, S13).

This makes it possible to expand the range of utilization of the visual determination result. For example, it is used for automatic generation of training data described below, calculation of a predicted value of the number of occurrences of the pattern "F_OK" described in the second embodiment, or determination of an appropriate re-learning time of the classification unit 102 described in the third embodiment. can do.

[Processing flow (classification processing)]
The details of the classification process performed in S13 of FIG. 7 will be described with reference to FIG. FIG. 8 is a flowchart showing an example of the determination result classification process. In the processing of S131 to S137 of FIG. 8, the classification result of the classification unit 102 is classified into a plurality of patterns based on the visual determination result, as in the case of S4, S5, S7 to S11 of FIG.

In S131, the correctness determination unit 104 determines whether or not both the classification and the product are correct based on the visual determination result in S12 of FIG. 7. If the correctness determination unit 104 determines that both the classification and the product are correct (YES in S131), the process proceeds to S132. On the other hand, when the correctness determination unit 104 determines that at least one of the classification and the product is incorrect (NO in S131), the process proceeds to S133.

In S132, the determination result management unit 105 determines that the classification is "T_OK". Then, the determination result management unit 105 stores the determination result in association with the image 201 acquired in S1 of FIG. 7, and the classification process ends. When storing the image 201 here, it is preferable to store the image 201 so that it can be distinguished from the image 201 stored as corresponding to "T_OK" in S5 of FIG. 7. This is because there is a possibility that the image 201 stored as corresponding to “T_OK” in S5 of FIG. 7 actually contains “F_OK” is not zero.

In S133, the determination result management unit 105 determines whether or not the classification is correct but the product is incorrect. If it is determined that the classification is correct but the product is incorrect (YES in S133), the process proceeds to S134. On the other hand, if it is determined that the product is correct or the classification is incorrect (NO in S134), the process proceeds to S135.

In S134, the determination result management unit 105 determines that the classification is "T_NG". Then, the determination result management unit 105 stores the determination result in association with the image 201 acquired in S1 of FIG. 7, and the classification process ends. When storing the image 201 here, it is preferable to store the image 201 so that it can be distinguished from the image 201 stored as corresponding to "T_NG" in S8 of FIG. 7. The same applies to S136 and S137 described below.

In S135, the determination result management unit 105 determines whether or not the product is correct but the classification is incorrect. If it is determined that the product is correct but the classification is incorrect (YES in S135), the process proceeds to S136. On the other hand, if it is determined that both the product and the classification are incorrect (NO in S135), the process proceeds to S137.

In S136, the determination result management unit 105 determines that the category is "F_NG1". Then, the determination result management unit 105 stores the determination result in association with the image 201 acquired in S1 of FIG. 7, and the classification process ends.

In S137, the determination result management unit 105 determines that the classification is "F_NG2". Then, the determination result management unit 105 stores the determination result in association with the image 201 acquired in S1 of FIG. 7, and the classification process ends. In addition, among those determined to be "F_NG2" in S137, those actually corresponding to "F_OK" may be included.

[Processing flow (training data generation)]
The information processing apparatus 1 of the present embodiment generates training data based on the above-mentioned pattern classification. This will be described with reference to FIG. FIG. 9 is a flowchart showing an example of a process (information processing method) in which the information processing apparatus 1 generates training data based on the pattern classification.

In S21, the training data generation unit 107 acquires the image 201 whose pattern has been determined by the determination result management unit 105. Then, in S22, the training data generation unit 107 determines whether or not the pattern of the image 201 acquired in S21 corresponds to "T_OK". If it is determined to be applicable (YES in S22), the process proceeds to S23, and if it is determined not to be applicable (NO in S22), the process proceeds to S24.

In S23, the training data generation unit 107 associates the classification result of the classification unit 102 with respect to the image 201 acquired in S21 with the image 201 as training data. Then, the training data generation unit 107 stores this training data in the storage unit 20, and ends the process. The data stored in this process and in S26 and S28 described later is the training data 202 in FIG.

In S24, the training data generation unit 107 determines whether or not the pattern of the image 201 acquired in S21 corresponds to “T_NG”. If it is determined to be applicable (YES in S24), the process proceeds to S23, and training data is generated and stored as described above. On the other hand, if it is determined in S24 that it does not correspond (NO in S24), the process proceeds to S25.

In S25, the training data generation unit 107 determines whether or not the pattern of the image 201 acquired in S21 corresponds to "F_NG1". If it is determined to be applicable (YES in S25), the process proceeds to S26, and if it is determined not to be applicable, that is, if the pattern of the image 201 corresponds to "F_NG2" or "F_OK" (NO in S25), S27. Proceed to.

In S26, the training data generation unit 107 associates the image 201 acquired in S21 with the classification category corresponding to the product ID acquired in S1 of FIG. 7 as correct answer data to obtain training data. Then, the training data generation unit 107 stores this training data in the storage unit 20, and ends the process.

In S27, the visual determination unit 106 makes a visual determination. The visual determination unit 106 transmits the image 201 acquired in S21 to the visual inspection terminal 2 to display it, and causes the visual inspector to input the classification category of the object shown in the image 201. Then, the result of the visual determination by the visual inspector is received from the visual inspection terminal 2, and the result is notified to the training data generation unit 107.

In S28, the training data generation unit 107 associates the image 201 acquired in S21 with the classification category input by the visual determination in S28 as correct answer data and uses it as training data. Then, the training data generation unit 107 stores this training data in the storage unit 20, and ends the process.

As described above, the training data generation unit 107 associates the classification result of the classification unit 102 with the image 201 used for the determination classified into any of the patterns “T_OK” and “T_NG” as correct answer data. And use it as training data (S23). The pattern "T_OK" is a pattern in which the correctness determination unit 104 determines that the object is correct (belongs to a predetermined classification). Further, the pattern "T_NG" is a pattern in which it is visually determined that the classification of the classification unit 102 is correct, as shown in FIG. 4B. Therefore, according to this configuration, appropriate training data can be automatically generated. For the image 201 used for the determination classified into the pattern "T_OK", the product classification category (predetermined classification) of the product ID acquired in S1 of FIG. 7 may be associated as correct answer data.

Further, the patterns "T_OK" and "T_NG" may be the categories determined by S5 and S8 in FIG. 7, or may be the categories determined by S132 and S134 in FIG. In particular, there is a possibility that some of the patterns classified as "T_OK" in S5 of FIG. 7 are actually "F_OK". Therefore, it is preferable to generate training data from the image 201 that is determined to be "T_OK" in S132 of FIG. 8 and stored, which is confirmed by visual inspection to be not "F_OK".

Further, the image 201 stored based on the classification determination of S132 and S134 in FIG. 8 is an image in which the accuracy of the classification result of the classification unit 102 is less than the threshold value, that is, an image that is difficult to be classified by the classification unit 102. .. Therefore, by generating training data from such an image 201 and retraining the classification unit 102 using the training data, it is possible to classify such a difficult-to-classify image by the classification unit 102. can.

Further, the training data generation unit 107 associates the image 201 used for the determination classified in the pattern “F_NG1” with the classification category (predetermined classification) corresponding to the product ID as correct answer data with the training data. (S26). As shown in FIG. 4 (b), the pattern "F_NG1" is a pattern in which it is visually determined that the correct product is manufactured (the object belongs to a predetermined classification), and therefore the pattern is based on this configuration. If this is the case, appropriate training data can be automatically generated. In particular, in the pattern "F_NG1", the classification unit 102 misclassifies the image 201. Therefore, in order to prevent such misclassification from being repeated, the image 201 used for the determination classified into the pattern "F_NG1" is used. The significance of using it as training data is great. The patterns "F_NG1" and "F_NG2" may be the categories determined by S10 and S11 in FIG. 7, or may be the categories determined by S136 and S137 in FIG. By generating training data from the image 201 stored based on the classification determination of S136 and S137 in FIG. 8 and retraining the classification unit 102 using the training data, the classification result of the classification unit 102 becomes unknown. The number of cases can be reduced.

In addition, when the training data is generated from the image 201 stored in association with the division of the pattern "F_NG2" in S11 of FIG. 7 and S137 of FIG. 8, visual determination of S27 is required, but the correctness determination unit Of the number of cases judged by 104, the ratio corresponding to the pattern "F_NG2" is small. Therefore, according to the training data generation unit 107, it is possible to automatically generate a large number of appropriate training data while minimizing the labor required for visual determination.

[Embodiment 2]
Other embodiments of the present invention will be described below. For convenience of explanation, the same reference numerals are given to the members having the same functions as the members described in the above-described embodiment, and the description thereof will not be repeated. The same applies to the third and subsequent embodiments.

The information processing apparatus 1 of the present embodiment determines when a determination result is obtained by the correctness determination unit 104 for each of a plurality of images 201 obtained by photographing a plurality of different objects, based on the above-mentioned pattern classification. Calculate the number of false positives (predicted values) included in the results. The calculated number can be used, for example, as an index showing the reliability of the determination result of the correctness determination unit 104. The number of erroneous determinations (number of erroneous determinations) is the number of cases in which the determination by the correctness determination unit 104 is incorrect, that is, the number of cases in which the determination corresponding to the pattern "F_OK" is performed.

The calculation of the predicted value is performed by the erroneous determination number calculation unit 108 shown in FIG. When calculating the predicted value, the determination result management unit 105 counts the following (A) to (D). For example, each of the following values (A) to (D) is the number of times that YES is determined in S3 of FIG. 7, the number of times that the processing of S10 is performed, the number of times that the processing of S8 is performed, and the processing of S11. It may be a count of the number of times it has been passed.

(A) Total number of images 201 successfully classified by the classification unit 102 (total number of determinations by the correctness determination unit 104): N
(B) Count number of pattern "F_NG1": n_fng1
(C) Count number of pattern "T_NG": n_tng
(D) Count number of pattern "F_NG2": n_fng2
[Principle of calculation of predicted value]
In the present embodiment, it is assumed that the probability p that the correct product is manufactured and the probability q that the classification unit 102 correctly classifies are both constant values, and the number of cases in which the determination corresponding to the pattern “F_OK” is made. Predicted value (calculated value) n_fok of is calculated. This calculation principle will be described below with reference to FIG. 4 (b).

The event that the correct product is manufactured and the event that the classification unit 102 correctly classifies are independent events. Further, if the probability that the correct product is manufactured is p, the probability that the wrong product is manufactured is expressed as (1-p), and if the probability of being correctly classified is q, the wrong classification is made. The probability is expressed as (1-q).

Therefore, when the correctness determination unit 104 makes a correctness determination, the probability that the determination corresponds to each pattern shown in FIG. 4B can be expressed as follows.

(Probability corresponding to "T_OK") = p × q
(Probability corresponding to "F_NG1") = p × (1-q)
(Probability corresponding to "T_NG") = (1-p) × q
(Probability corresponding to "F_NG2" or "F_OK") = (1-p) x (1-q)
Further, the probability corresponding to "F_NG1" is expressed as n_fng1 / N. Similarly, the probability corresponding to "T_NG" is expressed as n_tng / N, and the probability corresponding to "F_NG2" is expressed as n_fng2 / N. Therefore, the following mathematical formulas (1) and (2) hold.
p × (1-q) = n_fng1 / N Formula (1)
(1-p) × q = n_tng / N Formula (2)
Further, for the above (probability corresponding to "F_NG2" or "F_OK"), the value obtained by dividing the sum of n_fng2 and n_fok by N is equal to (1-p) × (1-q), so the following formula ( 3) holds.
n_fok = N × (1-p) × (1-q) -n_fng2 Formula (3)
Therefore, by calculating the probabilities p and q from the above mathematical formulas (1) and (2) and substituting the calculated probabilities p and q into the mathematical formula (3), the number of cases in which the determination corresponding to the pattern "F_OK" is made. Predicted value n_fok can be calculated.

[Processing flow (calculation of the number of erroneous judgments)]
The flow of the process in which the information processing apparatus 1 calculates the number of erroneous determinations will be described with reference to FIG. FIG. 10 is a flowchart showing an example of a process (information processing method) in which the information processing apparatus 1 calculates the number of erroneous determinations.

In S41, the erroneous determination number calculation unit 108 calculates the probability p that the product is correct and the probability q that the classification of the classification unit 102 is correct from the count number n_fng1 of the pattern “F_NG1” and the count number n_tng of the pattern “T_NG”. Specifically, the erroneous determination number calculation unit 108 calculates the probabilities p and q using the above-mentioned mathematical formulas (1) and (2).

In S42, the erroneous determination number calculation unit 108 calculates n_fok, which is a predicted value of the number of patterns “F_OK”, from the probabilities p and q calculated in S41 and the count number n_fng2 of the pattern “F_NG2”. Specifically, the erroneous determination number calculation unit 108 calculates n_fok by substituting the probabilities p and q into the above-mentioned mathematical formula (3).

In S43, the erroneous determination number calculation unit 108 determines that the number of patterns “F_OK” calculated in S42, that is, the correctness / rejection determination unit 104 is correct, but actually causes the output unit 50 to output the number of cases where the product is not correct. This ends the process. The erroneous determination number calculation unit 108 can also calculate the probability of occurrence of the pattern "F_OK" by dividing the number of the pattern "F_OK" calculated in S42 by N.

As described above, when the determination result management unit 105 of the present embodiment outputs a determination result by the correctness determination unit 104 for each of the plurality of images 201 obtained by photographing a plurality of different products (objects) (information processing system). When the determination by 500 is made), the classification result of the classification unit 102 in the determination is counted for each of the above-mentioned pattern categories. Then, the erroneous determination number calculation unit 108 determines the number of cases in which the correctness determination unit 104 determines that the object is correct, but the object is actually erroneous and the object is correct due to the erroneous classification of the classification unit 102. , Calculated based on the result of the above count.

Here, the above-mentioned patterns include "F_NG1" (the product is correct and the classification of the classification unit 102 is incorrect) and "T_NG" (the product is incorrect and the classification of the classification unit 102 is correct). It has been. Therefore, assuming that the probabilities p and q are constant, the probabilities p and q are calculated using the above equations (1) and (2) from the number of these patterns and the total number of judgments of the correctness determination unit 104. Can be calculated.

In addition, the pattern "F_OK" is an error in the product as in "F_NG2", and the classification of the classification unit 102 also corresponds to the error pattern. Therefore, the total number of patterns "F_OK" and "F_NG2" is N × ( It is expressed as 1-p) × (1-q). Therefore, the number of patterns "F_OK" can be calculated by subtracting the count number of the pattern "F_NG2" from this total number (Formula (3)). As described above, according to the erroneous determination number calculation unit 108, the number of cases of the pattern "F_OK" can be calculated by using the count number for each category of the pattern by the determination result management unit 105.

The number of patterns "F_OK" can also be calculated from the count number of each category determined in S132, S134, S136, and S137 in FIG. In this case, the probabilities p and q are calculated using any two of the following formulas (1)'to (3)' instead of the above formulas (1) and (2), and the following formula (4)'is used. Calculate the number of patterns "F_OK". In the following mathematical formulas (1)'to (4)', N'is the number of times NO is determined in S3 of FIG. 7 (the number of images 201 determined to be NO in S3 of FIG. 7). Further, in the following mathematical formula (3)', n_tok is the count number of the pattern “T_OK”.
p × (1-q) = n_fng1 / N'Formula (1)'
(1-p) × q = n_tng / N'Formula (2)'
p × q = n_tok / N'Formula (3)'
n_fok = N'× (1-p) × (1-q) -n_fng2 Mathematical formula (4)'
[Embodiment 3]
The information processing apparatus 1 of the present embodiment determines the timing of re-learning of the classification unit 102 based on the classification of the pattern described in the first embodiment. The timing of re-learning is determined by the re-learning time determination unit 109 shown in FIG. When determining the timing of re-learning, the determination result management unit 105 counts the following (A) to (C).

(A) Total number of images 201 successfully classified by the classification unit 102 (total number of determinations by the correctness determination unit 104): N
(B) Count number of pattern "F_NG1": n_fng1
(C) Count number of pattern "T_NG": n_tng
[Timing of re-learning]
Of the above-mentioned five pattern categories, "F_OK" is a pattern in which the correctness / rejection determination unit 104 determines that the product is correct even though the wrong product is actually manufactured, and such a pattern occurs. Should be avoided as much as possible. For example, it is assumed that a new product Y similar to the correct product X is newly added, but the classification unit 102 has not learned about the new product Y. In this case, if the image 201 of the new product Y is input to the classification unit 102, the score of the classification category of the product X may be equal to or higher than the threshold value. When the score becomes equal to or higher than the threshold value, the classification result of the classification unit 102 matches the correct product, so that the correctness / rejection determination unit 104 determines that the new product Y is actually manufactured by mistake. It will be judged as correct. As a result, the new product Y may be shipped as the product X.

Here, as one of the methods for reducing the risk of occurrence of "F_OK", there is a method of re-learning the classification unit 102 using the latest training data 202. By performing re-learning, it becomes possible to classify unlearned products as described above, and the probability of misclassification by the classification unit 102 can be reduced. As a result, the probability of occurrence of "F_OK" also increases. Can be lowered. However, considering the labor, time, and cost required for re-learning, frequent re-learning is not always a good idea.

Therefore, the re-learning time determination unit 109 determines an appropriate re-learning timing of the classification unit 102 (timing immediately before the deterioration of the classification accuracy of the classification unit 102 becomes unacceptable) based on the pattern classification described in the first embodiment. decide.

For example, the re-learning time determination unit 109 may determine the re-learning time of the classification unit 102 within the period until the number of patterns “F_OK” calculated by the erroneous determination number calculation unit 108 reaches a predetermined threshold value. .. This makes it possible to increase the possibility that the pattern "F_OK" does not occur. In particular, the threshold value is preferably less than 1. This makes it possible to reduce the possibility that the pattern "F_OK" will occur more than once. For example, the re-learning time determination unit 109 may have the erroneous determination number calculation unit 108 calculate the number of patterns “F_OK” every time the determination by the correctness determination unit 104 is performed a predetermined number of times. Then, when the calculated number exceeds the threshold value, it may be determined that it is the re-learning time of the classification unit 102.

[Forecasting the number of occurrences of "F_OK"]
The re-learning time determination unit 109 may determine the re-learning timing within a range in which the predicted value of the number of occurrences of the pattern “F_OK” is less than the threshold value. The predicted value of the number of occurrences of the pattern "F_OK" can be expressed by, for example, the following mathematical formula (4). In the formula (4), m is the number of product types and N is the total number of success / failure determinations. Note that N in the formula (4) is not a number counted in the above (A) but a variable. The definitions of p and q are the same as those in the second embodiment.

(Number of occurrences of pattern "F_OK") = N × (1-p) × (1-q) / m Formula (4)
The above mathematical formula (4) is based on the premise that the probabilities p and q are constant and the probabilities p and q are the same values for all m types of products. Under this premise, the reason why the above mathematical formula (4) holds will be described below.

As shown in FIG. 4B, the phenomenon that the product is incorrect and the classification is also incorrect corresponds to the pattern “F_NG2” or “F_OK”. The probability that this pattern will occur is (1-p) × (1-q). Therefore, when the determination is performed N times, the total number of occurrences of the patterns “F_NG2” and “F_OK” is calculated as N × (1-p) × (1-q). As described above, it is assumed here that the probabilities p and q are constant. Therefore, the probability calculated from the above values (A) to (C) counted in the period until the determination by the correctness determination unit 104 is performed a predetermined number of times (for example, the period of the broken line portion in the graph of FIG. 11B). p and q can also be applied to the subsequent periods.

In addition, since there are m types of products and the probabilities p and q are the same for all products, the pattern "F_NG2" occurs (m-1) times for each occurrence of the pattern "F_OK". do. Therefore, the ratio of n_fng2 to n_fok is (m-1): 1. Therefore, 1 / m of the total number of occurrences of the patterns "F_NG2" and "F_OK" is the number of occurrences of "F_OK". Therefore, the number of occurrences of the pattern "F_OK" when the correctness determination is performed N times can be expressed by the above mathematical formula (4).

[Processing flow]
The flow of processing executed by the information processing apparatus 1 of the present embodiment will be described with reference to FIG. FIG. 11B is a flowchart showing an example of a process (information processing method) in which the information processing apparatus 1 calculates a predicted value of the number of occurrences of the pattern “F_OK”. Further, FIG. 11B shows a graph showing the relationship between the number of occurrences of the pattern “F_OK” and N.

In S61, the re-learning time determination unit 109 calculates the probability p that the product is correct and the probability q that the classification of the classification unit 102 is correct from the count number n_fng1 of the pattern “F_NG1” and the count number n_tng of the pattern “T_NG”. .. The mathematical formulas (1) and (2) described in the second embodiment are used for calculating the probabilities p and q.

In S62, the re-learning time determination unit 109 calculates the number of occurrences of the pattern “F_OK” from the probabilities p and q calculated in S61 and the number of product types m. Specifically, the re-learning time determination unit 109 calculates the number of occurrences using the above-mentioned mathematical formula (4).

In S63, the re-learning time determination unit 109 determines the re-learning timing within a range in which the number of occurrences of the pattern “F_OK” calculated in S62 is less than the threshold value. Specifically, the re-learning time determination unit 109 calculates N in which the number of occurrences of the pattern "F_OK" is near the threshold value (however, the threshold value is not exceeded). Then, the re-learning time determination unit 109 determines the time when the number of determinations of the correctness determination unit 104 reaches the calculated determination number N as the timing (timing) of the re-learning of the classification unit 102. As a result, the process of FIG. 11 is completed.

The number of occurrences calculated in S62 is represented by a linear function with N as a variable. Therefore, the graph of the relationship between the number of occurrences of the pattern "F_OK" and N is as shown in (b) of the figure. As shown in the figure, the number of occurrences, which is the value on the vertical axis, increases in proportion to the increase in the total number N of the images 201 classified by the classification unit 102. The re-learning time determination unit 109 may calculate N near the threshold value on this graph.

As described above, the re-learning time determination unit 109 calculates the value of N in which the value of the mathematical formula (4) does not reach a predetermined threshold value, and the time when the number of determinations of the correctness determination unit 104 reaches the calculated N value. Is determined as the time for re-learning of the classification unit 102. In the mathematical formula (4), N is the number of determinations by the correctness determination unit 104, that is, the number of times the classification unit 102 is classified as YES in S3 of FIG. Further, (1-p) × (1-q) indicates the probability that an event occurs in which the product (object) is incorrect (does not belong to a predetermined classification) and the classification unit 102 misclassifies. Represents. And 1 / m represents the ratio that the classification result of the classification unit 102 matches the classification category of the correct product in the case where the above event occurs.

Therefore, according to the re-learning time determination unit 109, it is possible to determine an appropriate re-learning time so that the number of occurrences of "F_OK" does not exceed the permissible range. For example, in the example of FIG. 11B, the threshold value is set to 0.75. In this case, the re-learning time determination unit 109 may obtain, for example, the value of N at which the value of the above mathematical formula (4) is 0.75, and obtain the upper limit of the number of determinations of the correctness determination unit 104 from the value. In this case, the re-learning time is the time until the number of determinations of the correctness determination unit 104 reaches the upper limit determined above. The threshold value is not limited to 0.75, but is preferably less than 1. This makes it possible to reduce the possibility that "F_OK" will occur more than once.

It should be noted that the re-learning time can also be determined from the count number of each category determined in S132, S134, S136, and S137 in FIG. In this case, the re-learning time determination unit 109 determines N'(the number of times the classification unit 102 is classified as NO in S3 of FIG. 7) in which the value of the following mathematical formula (5)'does not reach a predetermined threshold value). Is calculated. Then, the time when the number of times NO is determined in S3 of FIG. 7 reaches the value of N'calculated above is determined as the time of re-learning of the classification unit 102.

(Number of occurrences of pattern "F_OK") = N'× (1-p) × (1-q) / m Formula (5)'
Further, in the above example, the re-learning time is determined assuming that the probabilities p and q are both constant, but the re-learning time may be determined in consideration of the fluctuations of the probabilities p and q, especially the probability q. good. This makes it possible to determine a more appropriate re-learning time. For example, q ₁ is used in the interval of N ₀ ≤ N <N ₁ , and q ₂ is used in the interval of N ₁ ≤ N <N ₂ . It may be good (N ₂ > N ₁ > N ₀ ).

Further, in the above example, in the case where the product is incorrect and the classification unit 102 misclassifies, the ratio of the classification result of the classification unit 102 to the correct product classification category is 1 / m. However, it is not limited to this example. For example, the above ratio may be calculated using probabilities p and q that differ from product to product.

Further, the re-learning time determination unit 109 determines the number of unknown determinations (cumulative number after the last learning) of the classification unit 102, or when the frequency of unknown determination reaches the upper limit, the re-learning time is set. You may decide that there is. As a result, the number of images 201 that are determined to be unknown can be suppressed to a small number, so that the number of times that a visual determination is performed can be suppressed to a small number.

[Embodiment 4]
The information processing apparatus 1 of the present embodiment determines the threshold value used for the classification success / failure determination of the classification unit 102 based on the classification of the pattern described in the first embodiment. Further, the information processing apparatus 1 of the present embodiment relearns the classification unit 102 at the relearning timing determined by the method described in the third embodiment. This will be described with reference to FIGS. 12 and 13.

FIG. 12 is a block diagram showing a configuration example of the control unit 10 of the information processing apparatus 1 of the present embodiment. The control unit 10 of the information processing apparatus 1 of the present embodiment is different from the information processing apparatus 1 of FIG. 1 in that the threshold value determination unit 121 and the learning unit 122 are included. Details of the threshold value determination unit 121 and the learning unit 122 will be described below.

FIG. 13 is a diagram illustrating a process executed by the information processing apparatus 1 of the present embodiment. In (a) of the figure, a graph showing an example of the correlation between the score Sc of the classification result of the classification unit 102 and the probability q that the classification of the classification unit 102 is correct is shown. The probability q is the ratio of the correct classification results among the classification results of the classification unit 102, and can also be expressed as the classification success rate. Further, in (b) of the figure, a flowchart showing a flow of processing executed by the information processing apparatus 1 of the present embodiment is described.

As shown in FIG. 13A, the larger the score Sc output by the classification unit 102 together with the classification result, the higher the probability q that the classification is correct. Therefore, the larger the threshold value is set, the less the misclassification of the classification unit 102 can be reduced. However, the larger the threshold value T, the more cases where an unknown determination is made. Further, in the vicinity of the maximum value of the score Sc (near 100% in the example of FIG. 13), the increase rate of the probability q with respect to the increase rate of the score Sc decreases. Therefore, even if the threshold value T is set larger than necessary, a large increase in the probability q cannot be expected, and the number of unknown determinations increases, resulting in a decrease in the processing efficiency of the information processing system 500. Therefore, the threshold value determination unit 121 determines an appropriate threshold value as described below.

As shown in FIG. 13B, the threshold value determination unit 121 first determines the initial value T ₀ of the threshold value (S81). The value to be set for T ₀ may be determined in advance, or may be specified by the user by an operation via the input unit 30 or the like. After that, the information processing system 500 repeatedly executes the process shown in FIG. 7 using the threshold value T ₀ determined in S81. As a result, the determination result management unit 105 records and updates the pattern classification result, that is, the count number of each pattern.

At the stage where the classification results of the pattern are accumulated to some extent (for example, the stage where the total number of classification results reaches a predetermined number), the threshold value determination unit 121 obtains a relational expression showing the correlation between the probability q and the score Sc (S82). .. In order to obtain the above relational expression, the score output by the classification unit 102 in the classification performed in S2 of FIG. 7 is stored.

To derive the relational expression, (1) the score is divided into a plurality of numerical ranges, (2) the probability q is calculated for each division, and (3) the approximate curve, that is, the score and the probability q are derived from the value of the probability q for each division. You may do it by the procedure of finding the relational expression of. Since the method of calculating the probability q is as described in the second and third embodiments, the description is not repeated here.

For example, if the number of cases in which the score of the classification result is 80% or more and less than 90% is 100 among the classifications performed in S2 of FIG. 7, the probability q is calculated from the classification results of those 100 cases. Thereby, when the score is 80% or more and less than 90%, the probability q that the classification unit 102 correctly classifies can be calculated. Then, the probability q is calculated in the same manner for the scores in the other numerical ranges, and the approximate curve is derived from the value of the probability q in each numerical range to obtain the relational expression as shown in the graph of FIG. 13 (a). Obtainable.

Next, the threshold value determination unit 121 determines a new threshold value T ₁ using the relational expression obtained in S82 (S83). Here, the threshold value determination unit 121 determines the threshold value T ₁ so that the probability q becomes equal to or greater than a predetermined lower limit value when the threshold value T ₁ is applied. For example, in the example of FIG. 13A, the predetermined lower limit is 0.75. _In this case, for example, the value calculated by substituting 0.75 into the relational expression (graph) of FIG. 13A may be determined as the threshold value T1. Further, a value calculated by substituting a value larger than 0.75 (preferably a value smaller than T ₀ ) into the above relational expression so that the probability q is surely 0.75 or more is used. The threshold value T ₁ may be determined.

After that, the information processing system 500 repeatedly executes the process shown in FIG. ₇ using the threshold value T1 determined in S83, whereby the pattern classification result by the determination result management unit 105 is accumulated. At the stage where the pattern classification results are accumulated to some extent (for example, the stage where the total number of classification results reaches a predetermined number), the re-learning time determination unit 109 calculates the predicted value of the number of occurrences of the pattern "F_OK" (S84). ). Further, the re-learning time determination unit 109 determines the re-learning timing within a range in which the number of occurrences of the pattern "F_OK" calculated in S84 is less than the threshold value (S85). Since the processing of S84 and S85 is the same as that of S62 and S63 of FIG. 11, the description is not repeated here.

Then, the learning unit 122 relearns the classification unit 102 at the timing determined in S85 (S86). The re-learning can be performed using, for example, the training data 202 stored in the storage unit 20. By this re-learning, the trained model used by the classification unit 102 for classification is updated.

Therefore, since the relational expression between the probability q and the score Sc also changes due to the processing of S86, the processing returns to S81 after the end of S86, and the threshold value determination unit 121 determines the initial value _T0 of the threshold value. The threshold value determined after re-learning may be set to a value different from T ₀ . After that, similarly to the above, the threshold value T1 is determined in S82 to S83, the _relearning timing is determined in S84 to S85, and relearning is performed in S86. As described above, according to the process of FIG. 13B, the determination of the threshold value T1 and the re _- learning are repeatedly performed.

As described above, in the determination system including the classification unit 102 and the correctness determination unit 104, when the score of the classification result of the classification unit 102 is equal to or higher than the threshold value, the object belongs to a predetermined classification based on the classification result. Judge whether or not. Further, the threshold value determination unit 121 determines the probability q based on the correlation between the score of the classification result by the classification unit 102 and the probability q (classification success rate) which is the ratio of the correct classification result among the classification results of the score. A threshold value T ₁ that is equal to or greater than the lower limit of is determined. Thereby, a valid threshold value T ₁ can be determined within a range satisfying the required level for the probability q. The score used for determining the threshold value T ₁ is not limited to the score above the threshold value. For example, the threshold value determination unit 121 may determine the threshold value T ₁ by using all of the scores output by the classification unit 102 whose success or failure of the classification result is confirmed by visual determination.

Further, since the threshold value determination unit ₁₂₁ determines the threshold value T1 based on the score output by the classification unit 102 and the probability q calculated from the count number of each pattern classified by the determination result management unit 105, the threshold value determination unit 102 determines the threshold value T1. The threshold value T ₁ can be determined according to the learning state. For example, as in the example of FIG. 13B, the threshold value determination unit 121 classifies the score output by the classification unit 102 after the relearning and the classification of the score each time the classification unit 102 is relearned. The threshold T ₁ may be determined based on the correlation with the probability q in the result. As a result, the threshold value T ₁ can be maintained at an appropriate value while maintaining or improving the classification accuracy of the classification unit 102 by re-learning.

[Example of implementation by system]
The function of the information processing device 1 can also be realized by a system including a plurality of devices capable of communicating with each other. In this case, each of the above-mentioned processes is performed by any of a plurality of devices constituting the system. For example, a first information processing device provided with a determination result management unit 105 and a second information processing device provided with an erroneous determination number calculation unit 108 may be provided. In this case, the second information processing apparatus calculates the number of erroneous determinations using information such as the count number of each pattern counted by the first information processing apparatus. Further, for example, a third information processing device provided with a relearning time determination unit 109 may be provided, in which case the third information processing device counts each pattern counted by the first information processing device. Use information such as numbers to determine when to relearn.

[Example of implementation by software]
The control block of the information processing apparatus 1 (particularly each part included in the control unit 10) may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software. good. This also applies to the control blocks of the visual inspection terminal 2 and the terminal device 3.

In the latter case, the information processing apparatus 1 includes a computer that executes instructions of a program that is software that realizes each function. The computer includes, for example, one or more processors and a computer-readable recording medium that stores the program. Then, in the computer, the processor reads the program from the recording medium and executes the program, thereby achieving the object of the present invention. As the processor, for example, a CPU (Central Processing Unit) can be used. As the recording medium, a "non-temporary tangible medium", for example, a ROM (Read Only Memory) or the like, a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. Further, a RAM (Random Access Memory) for expanding the above program may be further provided. Further, the program may be supplied to the computer via any transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. It should be noted that one aspect of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the above program is embodied by electronic transmission.

The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and the embodiments obtained by appropriately combining the technical means disclosed in the different embodiments. Is also included in the technical scope of the present invention.

[Additional notes]
The information processing apparatus according to one aspect of the present invention is the above-mentioned classification unit in a determination system for determining whether or not an object belongs to a predetermined classification by classifying the object in an image by a machine-learned classification unit. In the visual determination performed on the classified object by the classification unit, which is an information processing device for classifying the classification results of the above, (1) whether or not the object belongs to the above-mentioned predetermined classification, and (2). The classification unit according to the combination of the visual judgment result acquisition unit that acquires the visual judgment result, which is the result of the judgment of the correctness of the classification result by the classification unit, and the above (1) and (2) in the visual judgment result. It is equipped with a section for classifying the classification results of.

The information processing apparatus was used for the classification according to the classification according to the combination of the above (1) and (2), in which the visual judgment result of the above (2) is the result that the above classification result is correct. A training data generation unit may be provided in which the classification result of the classification unit is associated with the image as correct answer data and used as training data for re-learning of the classification unit.

The information processing apparatus relates to a classification according to the combination of the above (1) and (2), in which the visual determination result of the above (1) is the result that the object belongs to the above predetermined classification. A training data generation unit may be provided in which the predetermined classification is associated with the image used for classification as correct answer data and used as training data for re-learning of the classification unit.

The training data generation unit may generate the training data from the image in which the accuracy of the classification result of the classification unit is less than the threshold value.

The information processing apparatus has the classification result of the classification unit or the predetermined classification with respect to the image used for the judgment that the judgment result of the judgment system is that the object belongs to the predetermined classification. A training data generation unit may be provided in which the classification is associated with the correct answer data and used as the training data used for the re-learning of the classification unit.

In the information processing apparatus, when a determination is made by the determination system for each of a plurality of images obtained by photographing a plurality of different objects, the division unit counts the number of classification results corresponding to each of the above categories. Of the determinations for the plurality of different objects, the determination system determines that the object belongs to the predetermined classification, but the object does not actually belong to the predetermined classification, and the object does not belong to the predetermined classification. An erroneous determination number calculation unit may be provided, which calculates the number of cases misclassified as belonging to the predetermined classification by the classification unit based on the result of the count.

The information processing apparatus may include a relearning time determination unit that determines the relearning time of the classification unit within a period until the number of cases calculated by the erroneous determination number calculation unit reaches a predetermined threshold value. ..

The information processing apparatus has (probability that the object does not belong to the predetermined classification and an event that the classification unit misclassifies occurs) × (number of classifications of the classification unit) × (the event occurs). (Probability that the classification result of the above classification unit becomes the above-mentioned predetermined classification) <(predetermined threshold value)
It is provided with a re-learning time determination unit that calculates the number of classifications of the above-mentioned classification unit that satisfies the formula of You may be.

The information processing method according to one aspect of the present invention is the above-mentioned classification unit in a determination system for determining whether or not an object belongs to a predetermined classification by classifying the object in an image by a machine-learned classification unit. In the visual determination performed on the classified object by the classification unit, which is an information processing method by the information processing device for classifying the classification result of (1), whether or not the object belongs to the predetermined classification. And (2) the above classification according to the combination of the above (1) and (2) in the above visual judgment result acquisition step for acquiring the visual judgment result which is the result of the judgment of the correctness of the above classification result. Includes a classification step that separates the classification results of the department.

An information processing program for operating a computer as the information processing apparatus, and an information processing program for operating the computer as the visual determination result acquisition unit and the division unit is also included in the scope of the present invention.

1 Information processing device 102 Classification unit 104 Correct / fail judgment unit (judgment unit)
105 Judgment result management department (classification department)
106 Visual judgment unit (visual judgment result acquisition unit)
107 Training data generation unit 108 False judgment number calculation unit 109 Re-learning time determination unit

Claims (12)

An information processing device that classifies the classification results of the above classification unit in a determination system that determines whether or not the object belongs to a predetermined classification by classifying the object in the image by a machine-learned classification unit. ,
In the visual determination performed on the objects classified by the classification unit, (1) whether or not the object belongs to the above-mentioned predetermined classification, and (2) whether or not the classification result by the above-mentioned classification unit is correct or not are determined. A visual judgment result acquisition unit that acquires the visual judgment result, which is the result, and a visual judgment result acquisition unit.
An information processing apparatus comprising: a classification unit for classifying the classification results of the classification unit according to the combination of the above (1) and (2) in the visual determination result. Of the categories according to the combination of (1) and (2), the above image used for the classification related to the category in which the visual judgment result of the above (2) is the result that the classification result of the classification unit is correct. On the other hand, the information processing according to claim 1, wherein the information processing according to claim 1 is provided with a training data generation unit which associates the classification result of the classification unit with the correct answer data and uses the training data for re-learning of the classification unit. Device. Among the classifications according to the combination of the above (1) and (2), the above used for the classification related to the classification in which the visual judgment result of the above (1) is the result that the object belongs to the above predetermined classification. The information processing according to claim 1, wherein the image is provided with a training data generation unit that associates the predetermined classification with the correct answer data and uses the training data for re-learning of the classification unit. Device. The information processing apparatus according to claim 2 or 3, wherein the training data generation unit generates the training data from the image in which the accuracy of the classification result of the classification unit is less than the threshold value. Corresponds to the classification result of the classification unit or the predetermined classification as correct answer data for the image used for the judgment that the judgment result of the judgment system is that the object belongs to the predetermined classification. The information processing apparatus according to claim 1, further comprising a training data generation unit which is used as training data for re-learning of the classification unit. When a judgment is made by the judgment system for each of a plurality of images obtained by photographing a plurality of different objects, the classification unit counts the number of classification results of the classification unit corresponding to each of the above categories.
Of the determinations for the plurality of different objects, the determination system determines that the object belongs to the predetermined classification, but the object does not actually belong to the predetermined classification, and the object does not belong to the predetermined classification. 2. Information processing equipment. 6. The claim 6 is characterized in that it includes a re-learning time determination unit that determines the re-learning time of the classification unit within a period until the number of cases calculated by the erroneous determination number calculation unit reaches a predetermined threshold value. The information processing device described in. (Probability that the object does not belong to the predetermined classification and an event of misclassification by the classification unit occurs) × (number of classifications of the classification unit) × (classification of the case where the above event occurs) The ratio of the classification result of the part to the above-mentioned predetermined classification) <(predetermined threshold value)
It is provided with a re-learning time determination unit that calculates the number of classifications of the above-mentioned classification unit that satisfies the formula of The information processing apparatus according to any one of claims 1 to 6, wherein the information processing apparatus is characterized by the above. In the determination system, when the accuracy of the classification result of the classification unit is equal to or higher than the threshold value, it is determined whether or not the object belongs to a predetermined classification based on the classification result.
A threshold value determining unit for determining the threshold value at which the classification success rate is equal to or higher than a predetermined lower limit value is provided based on the correlation between the accuracy and the classification success rate, which is the ratio of the correct classification results among the classification results of the accuracy. The information processing apparatus according to any one of claims 1 to 8, wherein the information processing apparatus is characterized by the above. Each time the classification unit is relearned, the threshold value determination unit determines the threshold value based on the correlation between the accuracy after the relearning and the classification success rate in the classification result of the accuracy. The information processing apparatus according to claim 9. Information processing by an information processing device that classifies the classification results of the above classification unit in the determination system that determines whether or not the object belongs to a predetermined classification by classifying the object in the image by the machine-learned classification unit. It ’s a method,
In the visual determination performed on the classified object by the classification unit, (1) whether or not the object belongs to the above-mentioned predetermined classification, and (2) whether or not the classification result of the above-mentioned classification unit is correct or not are determined. The visual judgment result acquisition step for acquiring the visual judgment result, which is the result, and the visual judgment result acquisition step,
An information processing method comprising: a classification step for classifying the classification results of the classification unit according to the combination of the above (1) and (2) in the visual determination result. The information processing program for operating a computer as the information processing device according to claim 1, wherein the computer functions as the visual determination result acquisition unit and the division unit.

Images