JP7130984B2 - Image judgment system, model update method and model update program - Google Patents

Description

The present invention relates to an image determination system for determining the type of image, and a model updating method and a model updating program for updating a model for estimating the type of image.

In order to appropriately perform subsequent processing corresponding to various types of images, it is common practice to manually check the content of the images and classify the images into various types. On the other hand, methods have been proposed to improve the efficiency of manual checking work and reduce the work cost.

Japanese Patent Application Laid-Open No. 2002-200002 describes an information processing apparatus that allows an operator to easily specify the correct type from among a plurality of types of forms. In the information processing apparatus described in Patent Literature 1, a feature amount corresponding to each image data classified into each class is prepared as learning data, and a classifier for classifying each class is prepared. Then, the information processing apparatus described in Patent Document 1 utilizes OVA (One-Versus-All), which is a method of extending a two-class classifier to a multi-class classifier, and converts the output of each classifier into a target class. is the reliability of

JP 2017-107455 A

When there are few images for learning, or when images of similar types that are difficult to determine are mixed, it is difficult to perform highly accurate determination from the initial stage. In such a situation, even if the type of image data is specified based on the certainty of the type of image data using a determiner, manual reconfirmation is indispensable. Therefore, in order to reduce the time and effort of manual confirmation, it is required to generate a highly accurate learning model to be used in the decision device.

If the determination probability is not high for all types of images to be determined, setting a low threshold for determining each type can be considered. However, if the threshold is set too low, there is a high risk of inducing judgment errors, which may increase the amount of manual checking and correction work. On the other hand, if the threshold value is set high from the initial stage, there is a high possibility that it will not be judged as any type, so there is a possibility that the amount of manual setting work will increase.

Patent Document 1 describes that a classifier is trained using image data, but it is unclear how much image data is used for learning, and a classifier with sufficiently high accuracy is generated from the initial stage. It's hard to say that it's done. Therefore, it is unclear whether the image data is classified into an appropriate class, and there is a problem that a classifier with low accuracy increases manual work.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an image determination system, a model update method, and a model update program capable of improving the accuracy of image determination according to use.

An image determination system according to the present invention includes a learning unit that learns a model for estimating the likelihood of each type represented by an image using learning image data, an evaluation unit that evaluates the accuracy of the learned model, and a model that: a discriminating unit that discriminates the type of image whose estimated probability exceeds a threshold, and the learning unit assigns a correct label to an indistinguishable image that is an image that the discriminating unit cannot discriminate; Alternatively , the model is re-learned using learning image data to which a correct label is assigned to an image whose type could not be determined because the probability of model estimation does not satisfy a predetermined criterion, and the evaluation unit However, the accuracy of the model before and after relearning is compared, the model with the higher accuracy is selected , and the evaluation unit evaluates the image for which the correct answer is known using the model before relearning. When the determination probability when the image is evaluated by the re-learned model is higher than the probability, the threshold is changed to the determination probability when the image is evaluated by the re-learned model. do.

In the model updating method according to the present invention , the computer discriminates the types of images whose likelihood estimated by the model for estimating the likelihood of each type represented by the image exceeds a threshold, A correct label is assigned to an image data for learning in which a correct label is assigned to a certain indistinguishable image, or to an image whose type cannot be determined because the probability of estimation by the model does not satisfy a predetermined criterion. The model is relearned using the training image data, and the computer compares the accuracy of the model before and after relearning, and selects the model with the higher accuracy. If the decision probability when evaluating a known image with the model before retraining is higher than the decision probability when evaluating the above image with the model after retraining, when evaluating with the model after retraining is changed so that the determination probability of is set as a threshold value .

The model update program according to the present invention provides a computer with a learning process of learning a model for estimating the likelihood of each type represented by an image using learning image data, an evaluation process of evaluating the accuracy of the learned model , and , a discrimination process is executed to discriminate the type of image whose probability estimated by the model exceeds a threshold, and in the learning process, a correct label is assigned to the indistinguishable image, which is an image that could not be discriminated in the discrimination process. The model is re-learned using the image data or the learning image data in which the correct label is given to the image whose type could not be determined because the probability of estimation by the model does not satisfy a predetermined criterion. , in the evaluation process, the accuracy of the model before and after relearning is compared, and the model with the higher accuracy is selected . If the judgment probability when the image is evaluated with the model after re-learning is higher than the judgment probability when the image is evaluated with the model after re-learning, change the judgment probability when evaluating with the model after re-learning to be the threshold. characterized by

According to the present invention, it is possible to improve the accuracy of image determination according to use.

1 is a block diagram illustrating one embodiment of an image determination system according to the present invention; FIG. 4 is a flow chart showing an operation example of the image determination system; FIG. 4 is an explanatory diagram showing a specific example of the operation of the image judgment system; 1 is a block diagram showing an outline of an image judgment system according to the present invention; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing one embodiment of an image judgment system according to the present invention. The image determination system 100 of this embodiment includes an image input unit 10, an image type estimation unit 20, a discriminator 30, a relearning data storage unit 31, a test data storage unit 32, and a type setting unit 40. , a learner 50 , an evaluation unit 60 , and an output unit 70 .

The output unit 70 outputs various results obtained by the image determination system. The output unit 70 is implemented by, for example, a display device or a printer.

The image input unit 10 inputs an image that the image type estimation unit 20 uses for type estimation. The image input section 10 may input an image via a communication network, or may read an image stored in a storage section (not shown) of another system.

The image type estimating unit 20 estimates the likelihood of the type of the input image using a model for estimating the likelihood of each type represented by the image (hereinafter simply referred to as a model or a learned model). The likelihood of the type may be expressed in any manner, and may be, for example, the probability of the likelihood of the type or the reliability of each type.

Any method can be used to set the type used for image classification. The type may be set for each different type of image, or may be set for the same type of image according to the degree of object represented by the image. Examples of settings for different types of images include different types of forms (resident cards, marriage registrations, seal certificates, etc.), maps (road maps, topographic maps, etc.), and floor plans (residential, office, etc.). ). Further, as an example of setting for each image of the same type, there is a case where setting is made according to the degree of deterioration of the same building or the degree of progress of a lesion.

The mode of the model used by the image type estimation unit 20 is arbitrary, and is, for example, a model generated by deep learning. In this case, the image type estimation unit 20 may apply the input image to a model to estimate the probability for each type. For example, if there are four candidates for the type of image A, types A to D, and the type A-likeness is the highest, the image type estimation unit 20 selects "type A: 85%", "type B: 10%", and "type A: 85%" for each type. %”, “Type C: 3%”, and “Type D: 2%”. A method of learning the model used by the image type estimation unit 20 will be described later.

The discriminator 30 discriminates the image type based on the likelihood of the image type estimated by the image type estimation unit 20 . Specifically, the discriminator 30 discriminates the type of image for which the probability estimated by the model used for estimation by the image type estimation unit 20 exceeds a threshold.

A threshold value used for determination may be set for each type, or may be set to a common value for all types. For example, a type that is easy to estimate may be set with a threshold higher than the thresholds for other types.

Also, for example, even if a plurality of types are set, it may be possible to distinguish between a certain type (for example, type X) and other types, or a type that can be distinguished with a very high probability ( For example, type Y.) may exist. In such a case, if only the type (type X, type Y) should be discriminated by the discriminator 30, an appropriate threshold is set for that type, and for the other types, an emergency threshold value is set. can be set with a high threshold. By doing so, it is possible to extract images of a type that is preferably manually determined.

Note that the threshold used by the discriminator 30 is updated by the evaluation unit 60, which will be described later. In the initial stage, the threshold may be set based on the user's experience, etc., or existing estimated probabilities. Alternatively, the probability of the type of image to which the correct data is assigned may be estimated using the current model, and a threshold may be set so as to satisfy the estimated probability. A method for updating the threshold will be described later.

The discriminator 30 registers the image for which the type has been discriminated (that is, the image for which the type has been discriminated) in the test data storage unit 32 . Further, the discriminator 30 may transmit the image for which the type has been discriminated to subsequent processing. The discriminator 30 may, for example, output an image whose type has been discriminated to the output unit 70 together with the type, or may notify a processor (not shown) that performs subsequent processing corresponding to the type of the image. .

On the other hand, if the type of the image cannot be determined, the discriminator 30 registers the image whose type cannot be determined (hereinafter referred to as an indistinguishable image) in the re-learning data storage unit 31 . The registered image is used in the subsequent type setting section 40 . Note that the discriminator 30 may directly transmit the indistinguishable image to the type setting unit 40 without registering the indistinguishable image in the relearning data storage unit 31 .

The type setting unit 40 assigns a type, which is correct data (label), to an image whose type could not be determined by the discriminator 30 (that is, an unidentifiable image). For example, the type setting unit 40 may display an indistinguishable image to the user and manually provide correct answer data. Since the unidentifiable image to which the correct data is assigned is used as teacher data in the subsequent re-learning process, it is preferable that the correct data be assigned with high accuracy. Also, since the image to which the correct label is assigned can be used as test data, the type setting unit 40 stores this image in the test data storage unit 32 .

The learning device 50 learns a model used in the discriminator 30 (that is, a model for estimating the likelihood of each type represented by an image) using image data for learning. In the initial stage, the learning device 50 may learn a model using, for example, existing image data as learning data.

Further, the learning device 50 re-learns the model using the data to which the correct labels are assigned to the indistinguishable images as learning image data. An indistinguishable image can be said to be an image for which the model could not estimate the type with high accuracy, or an image for which the type could not be estimated with the accuracy required by the classifier 30 (that is, the accuracy of the set threshold). It is possible to improve the accuracy with which the re-learned model estimates the type by generating learning image data with correct labels assigned to such images and performing re-learning based on this learning image data. be possible.

The method by which the learner 50 re-learns is arbitrary. For example, when the number of types does not increase, the learning device 50 may perform additional learning using the learning image data, or may combine the images used for learning in the past and the indistinguishable images to which the correct labels are assigned as learning data. may be used for new model learning.

At that time, the data used by the learning device 50 for re-learning can also be selected according to the learning method. The learning device 50 may re-learn the model, for example, using all the indistinguishable images to which the correct labels are assigned as the learning image data. In addition, in order to shorten the learning time and enhance the estimation accuracy for a specific type, the learning device 50 uses only images of a specific type among the unidentifiable images to which correct labels are assigned as learning image data. good too.

For example, improve the accuracy of estimating classes that have a low probability of being estimated to be applicable (i.e. classes that are more likely than other classes and have a low probability even if they are estimated to be of that type). I want to let For example, assume that when types A to D exist, the estimated probability of type D is 51%. In this case, it is the type that is estimated to be the highest in comparison with the other types, but the estimated probability itself is considered to be low. In this case, the learning device 50 may perform re-learning using the learning image data whose type (for example, type D) is set as the correct label.

In addition, the learning device 50 may perform re-learning using image data of the type most frequently set (or most frequently) among the indistinguishable images. This is because such types of images are considered to have low estimation accuracy. By performing re-learning using such images, it is possible to generate a model capable of accurately estimating the type with low estimation accuracy.

On the other hand, the timing at which the learning device 50 re-learns is also arbitrary. The learning device 50 may perform re-learning after a predetermined period of time has elapsed, or may perform re-learning when a predetermined number of image data for learning is accumulated as a number that is likely to ensure accuracy. Further, the learning device 50 may perform re-learning when the ratio of images whose types cannot be determined by the classifier 30 (hereinafter referred to as the undistinguishable ratio) no longer satisfies a predetermined criterion. For example, the learning device 50 calculates the ratio of the number of images for which the type cannot be determined to the total number of images to be determined as the undistinguishable ratio, and if the undistinguishable ratio exceeds a predetermined reference, You can study. By doing so, for example, when the tendency of the input image changes, automatic relearning becomes possible, which is preferable.

The evaluation unit 60 evaluates the accuracy of the model learned by the learning device 50 . Then, the evaluation unit 60 compares the accuracy of the existing model (the model before re-learning) and the model after re-learning, and selects a model with higher accuracy. The evaluation unit 60 may automatically replace the selected model as the model used by the image type estimation unit 20, or may replace it according to an instruction from the user or the like.

The evaluation unit 60 may evaluate the accuracy of the model based on a single index, or evaluate the accuracy of the model by comprehensively judging a plurality of indexes. At that time, the evaluation unit 60 may evaluate the accuracy of the model using both the image whose type is identifiable and the image whose type is not identifiable, which are the data stored in the test data storage unit 32. data may be used to assess model accuracy. For example, the evaluation unit 60 causes the existing model and the re-learned model to distinguish an indistinguishable image in parallel, and the likelihood of each classification is higher in the re-learned model than in the existing model. , it may be determined that the accuracy has increased and the model after re-learning may be selected.

In addition, the evaluation unit 60 may calculate the number (or ratio) of images that became distinguishable as a result of distinguishing images that were judged to be indistinguishable by the existing model using the model after re-learning. Then, when the calculated number (or ratio) satisfies a predetermined criterion, the evaluation unit 60 may determine that the model after re-learning has become a more accurate model and select that model. .

In addition, the evaluation unit 60 applies a plurality of test data to both the existing model and the re-learned model, calculates the likelihood for each type, and statistically processes the evaluation results for each test data (for example, , calculate the average value of the probability, and calculate the standard deviation of the probability). For example, when an average value is calculated as statistical processing, the evaluation unit 60 may select a model with a high average probability as a model with higher accuracy. Further, for example, when the standard deviation is calculated as statistical processing, the evaluation unit 60 may select a model with a small standard deviation of probability (that is, a model with less deviation in evaluation) as a model with higher accuracy. Note that the test data used for the statistical processing may be all past data of the learned type, or may be a preselected number of data.

Furthermore, when updating the model used by the image type estimation unit 20, the evaluation unit 60 may also update the threshold used by the classifier 30. FIG. For example, the degree to which the threshold is to be changed at the time of updating may be determined in advance, and the evaluation unit 60 may update the threshold according to the degree. At this time, the degree to be determined may be for each type, or may be common to the whole.

Alternatively, the evaluation unit 60 may determine the degree to which the threshold is increased according to the improved accuracy as a result of comparing the existing model and the re-learned model. For example, by learning changes in the past undistinguishable ratio when the threshold is changed in accordance with the improved accuracy, the evaluation unit 60, from the relationship between the improved accuracy, the improved threshold, and the undistinguishable ratio, The degree to which the threshold is changed may be determined.

Specifically, if the estimation probability by the image type estimation unit 20 is improved by updating to a model with high accuracy, the undistinguishable ratio is reduced. On the other hand, if the discrimination threshold value used by the discriminator 30 is increased, the non-discrimination rate will increase. Therefore, the evaluation unit 60 may increase the threshold in accordance with a predetermined allowable unidentifiable ratio. As an example of a method of changing the threshold, there is a method of evaluating an existing model and a model after re-learning using an image for which the correct answer is already known. The evaluation unit 60 calculates the determination probability when all images are evaluated with the existing model and the determination probability when all images are evaluated with the re-learned model. The evaluation unit 60 compares these determination results, and if the determination probability in the model after re-learning is high, the determination probability may be changed so as to be the threshold.

The relearning data storage unit 31 stores the indistinguishable image. Note that the relearning data storage unit 31 may store the indistinguishable image in association with the correct label assigned by the type setting unit 40 . Further, the test data storage unit 32 stores the discriminated or added type in association with the image. The relearning data storage unit 31 and the test data storage unit 32 are realized by, for example, a magnetic disk device.

The image input unit 10, the image type estimation unit 20, the classifier 30, the type setting unit 40, the learning unit 50, and the evaluation unit 60 are implemented by a CPU of a computer that operates according to a program (model update program). be. For example, the program is stored in a storage unit (not shown) provided in the image determination system, the CPU reads the program, and according to the program, the image input unit 10, the image type estimation unit 20, the discriminator 30, the type setting unit 40 , the learner 50 and the evaluation unit 60 .

The image input unit 10, the image type estimation unit 20, the classifier 30, the type setting unit 40, the learning unit 50, and the evaluation unit 60 may each be realized by dedicated hardware.

Next, the operation of this embodiment will be described. FIG. 2 is a flowchart showing an operation example of the image determination system of this embodiment. It is assumed that the model used for estimation by the image type estimation unit 20 in the initial stage has been learned in advance.

The image input unit 10 inputs an image to be determined (step S11). The image type estimation unit 20 applies the model to the input image and estimates the probability of the type of the input image (step S12). The discriminator 30 discriminates the image type based on the estimated likelihood of the image type (step S13).

If the image type cannot be determined (No in step S13), the type setting unit 40 assigns correct data to the unidentifiable image (step S14) and stores it in the relearning data storage unit 31. FIG. After that, the process of step S15 is performed. On the other hand, if the image type can be determined (Yes in step S13), the discriminator 30 notifies the image to a processor (not shown) that performs subsequent processing corresponding to the type (step S15). Further, the discriminator 30 stores the discriminated type in the test data storage unit 32 in association with the image.

The learning device 50 re-learns the model using the data to which the correct labels are assigned to the indistinguishable images as learning image data (step S16). The learning device 50 performs relearning with timing and method according to predetermined conditions. The evaluation unit 60 evaluates the accuracy of the model learned by the learning device 50, and determines whether the accuracy is higher than the existing model (step S17). If the accuracy is high (Yes in step S17), the evaluation unit 60 replaces the existing model used for estimation by the image type estimation unit 20 with the re-learned model (step S18). Furthermore, the evaluation unit 60 updates the threshold value used for discrimination by the discriminator 30 (step S19). On the other hand, if the accuracy is not high (No in step S17), the existing model is left as is and the process ends.

Next, a specific example of the image determination system of this embodiment will be described. FIG. 3 is an explanatory diagram showing a specific example of the operation of the image determination system of this embodiment. In the example shown in FIG. 3, it is assumed that images to be determined are stored in the image storage unit 11 . An AI (Artificial Intelligence) 21 illustrated in FIG. 3 corresponds to the image type estimation unit 20 and estimates the likelihood of types A, B, C and D using models.

Next, the discriminator 30 compares the estimated probability of the type with a set threshold, and identifies a type with a probability greater than the threshold. Here, images for which a type greater than the threshold (80%) is specified are classified as discriminated 30x, and images for which a type is not specified are classified as indistinguishable 30y. The user sets types A to D via the type setting unit 40 for images classified as indistinguishable 30y (that is, indistinguishable images). This data is used as re-learning data.

The images classified as 30x classified and the unidentifiable images to which the classification is set are input to the processors Pa to Pd that perform respective processing, and each processing is performed. In addition, these images are stored in the storage device 33 for discriminated images. This storage device 33 corresponds to the test data storage section 32 .

After that, the AI 51 re-learns the model using the re-learning data to generate a re-learned model. The AI 51 corresponds to the learning device 50 and performs re-learning with timing and method according to predetermined conditions. Then, the relearning model evaluator 61 compares the accuracy of the existing model pm and the model rm after relearning, and selects a model with higher accuracy. When the retrained model rm is selected, the evaluator 61 replaces the model used by the AI 21 . Together, the evaluator 61 changes the threshold used by the discriminator 30 . Note that the evaluator 61 corresponds to the evaluator 60 .

As described above, in the present embodiment, the learning device 50 re-learns the model using the indistinguishable image as learning data, and the evaluation unit 60 compares the accuracy of the model before re-learning and the model after re-learning. Compare and select the more accurate model. Therefore, it is possible to improve the accuracy of image determination according to use. That is, since a model with higher accuracy is selected according to use, it is possible to improve the accuracy by using the model to determine the image.

Next, an outline of the present invention will be described. FIG. 4 is a block diagram showing an outline of an image judgment system according to the present invention. An image determination system 80 (for example, the image determination system 100) according to the present invention includes a learning unit 81 (for example, a learning device 50) that learns a model for estimating the likelihood of each type represented by an image using learning image data. and an evaluation unit 82 (e.g., evaluation unit 60) that evaluates the accuracy of the learned model.

The learning unit 81 performs learning in which a correct label is assigned to an image (for example, an unidentifiable image) whose type cannot be determined because the probability estimated by the model does not satisfy a predetermined standard (for example, a threshold value). The model is re-learned using the image data for the model, and the evaluation unit 82 compares the accuracy of the model before re-learning and the model after re-learning, and selects the model with the higher accuracy.

Such a configuration can improve the accuracy of judging an image as it is used.

In addition, the image determination system 80 may include a determination unit (for example, the discriminator 30) that discriminates the type of image for which the probability estimated by the model exceeds a threshold. Then, the learning unit 81 may re-learn the model using the learning image data in which the correct label is assigned to the indistinguishable image, which is the image that the discriminating unit cannot discriminate. According to such a configuration, it is possible to adjust whether or not determination is possible according to the set threshold value, so it is possible to adjust the load of subsequent processing.

At this time, the learning unit 81 may perform re-learning when the undistinguishable ratio, which is the ratio of images whose types cannot be determined by the determining unit, no longer satisfies a predetermined standard. According to such a configuration, it becomes possible to automatically update the model when an image with a different tendency becomes a discrimination target.

In addition, the evaluation unit 82 uses the existing model and the re-learned model to discriminate an indistinguishable image, and the re-learned model has a higher likelihood of the discriminated type than the existing model. If not, you may select the model after re-learning. In this way, by using an indistinguishable image for evaluation of re-learning, it is possible to judge whether the accuracy of the model is improved depending on whether the indistinguishable image can be judged.

Moreover, the evaluation unit 82 may increase the threshold used by the determination unit for determination when the model used for determination is changed to a model with higher accuracy. By changing to a model with high accuracy, it is considered that the certainty of the type increases. Therefore, it is possible to appropriately adjust the subsequent manual load while improving the discrimination accuracy.

In addition, the evaluation unit 82 applies a plurality of test data to the model to calculate the likelihood for each type, and statistically processes the calculated likelihood for each test data (for example, calculation of average, calculation of deviation, etc.). ) may be used as the accuracy of the model. In this way, by using the probability of the type calculated by the model, it becomes possible to appropriately judge the accuracy of the model.

REFERENCE SIGNS LIST 10 image input unit 11 image storage unit 20 image type estimation unit 21 AI
30 discriminator 31 re-learning data storage unit 32 test data storage unit 40 type setting unit 50 learning device 51 AI
60 evaluation unit 61 evaluator 70 output unit Pa to Pb processor 100 image judgment system

Claims (7)

a learning unit that learns a model for estimating the likelihood of each type represented by an image using learning image data;
an evaluation unit that evaluates the accuracy of the learned model ;
a determination unit that determines the type of image whose likelihood estimated by the model exceeds a threshold ;
The learning unit provides learning image data in which a correct label is assigned to an indistinguishable image, which is an image that cannot be discriminated by the discriminating unit, or the probability estimated by the model does not satisfy a predetermined criterion. Re-learning the model using learning image data in which the correct label is assigned to the image whose type could not be determined as
The evaluation unit compares the accuracy of the model before relearning and the model after relearning, and selects a model with higher accuracy,
When the evaluation unit evaluates the image with a known correct answer with the model before the relearning, the evaluation probability when the image is evaluated with the model after the relearning is higher than the judgment probability when the image is evaluated with the model before the relearning. Change the judgment probability when evaluating with the model after re-learning to be the threshold
An image judgment system characterized by: 2. The image determination system according to claim 1 , wherein the learning unit performs re-learning when an indistinguishable rate, which is a ratio of images whose types cannot be determined by the determining unit, no longer satisfies a predetermined standard. The evaluation unit discriminates indistinguishable images using the existing model and the model after re-learning, and when the probability of the discriminated type is higher in the model after re-learning than in the existing model 3. The image determination system according to claim 1 , wherein the model after re-learning is selected. 4. The image according to any one of claims 1 to 3 , wherein the evaluation unit increases a threshold value used for discrimination by the discrimination unit when a model used for discrimination is changed to a model with higher accuracy. judgment system. The evaluation unit applies a plurality of test data to the model, calculates the likelihood for each type, and uses a value obtained by statistically processing the calculated likelihood for each test data as the accuracy of the model. 5. The image determination system according to any one of 4 . The computer determines the type of image whose likelihood estimated by a model for estimating the likelihood of each type represented by the image exceeds a threshold,
The computer determines that the image data for learning in which the correct label is assigned to the indistinguishable image, which is the image that cannot be distinguished, or the classification that the probability estimated by the model does not satisfy a predetermined criterion Re-learning the model using training image data in which correct labels are assigned to images that could not be determined,
The computer compares the accuracy of the model before relearning and the model after relearning, and selects a model with higher accuracy,
If the computer has a higher determination probability when evaluating the image with the model after the re-learning than the determination probability when evaluating the image with the known correct answer with the model before the re-learning, the re-learning Change the judgment probability when evaluating with the model after learning to be the threshold
A model update method characterized by: to the computer,
A learning process that uses training image data to learn a model that estimates the likelihood of each type represented by an image ,
an evaluation process for evaluating the accuracy of the learned model ; and
executing a discrimination process for discriminating a type of image whose likelihood estimated by the model exceeds a threshold ;
In the learning process, learning image data in which a correct label is assigned to an indistinguishable image, which is an image that cannot be discriminated in the discrimination process, or the probability estimated by the model does not satisfy a predetermined criterion. Re-learning the model using learning image data in which the correct label is assigned to the image whose type could not be determined,
In the evaluation process, comparing the accuracy of the model before re-learning and the model after re-learning to select a model with higher accuracy ,
In the evaluation process, if the determination probability when evaluating the image with the model after the re-learning is higher than the determination probability when evaluating the image with the known correct answer with the model before the re-learning, Change the judgment probability when evaluating with the model after re-learning to be the threshold
model update for.

Images