JP2022040352A - Model update support system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a model update support system which can support update of a model.

SOLUTION: The model update support system according to an embodiment supports update of a first model learned by a learning data group including plural pieces of example data and plural labels provided to the pieces of data, respectively. The model update support system includes a processor. The processor can output first information showing that learning of the first model is insufficient and second information showing that one of the labels is not appropriate on the basis of the classification certainty showing the certainty of classification of first data calculated using the first model and a plurality of similarities of the first data and the pieces of example data.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2022,JPO&INPIT

Description

An embodiment of the present invention relates to a model update support system.

Models trained using deep learning are used when classifying data. It is desirable to update the model from time to time in order to continuously and accurately classify the data.

Japanese Patent Application Laid-Open No. 2003-132332

An embodiment of the present invention provides a model update support system capable of supporting model update.

The model update support system according to the embodiment supports the update of the first model trained using the learning data group including the plurality of exemplary data and the plurality of labels assigned to the plurality of exemplary data. .. The model update support system includes a processing unit. The processing unit has a plurality of classification certainty factors calculated by using the first model, which indicate the certainty of classification of the first data, and a plurality of similarities between the first data and the plurality of exemplary data. Based on the similarity, it is possible to output the first information indicating that the training to the first model is not sufficient, or the second information indicating that one of the plurality of labels is not appropriate.

It is a schematic diagram which shows the structure of the model update support system which concerns on 1st Embodiment. It is a schematic diagram which illustrates the output by the model update support system which concerns on 1st Embodiment. It is a flowchart which illustrates the process using the model update support system which concerns on 1st Embodiment. It is a schematic diagram which illustrates the output by the model update support system which concerns on 1st Embodiment. It is a schematic diagram which illustrates the output by the model update support system which concerns on 1st Embodiment. It is a schematic diagram which shows the structure of the model update support system which concerns on 2nd Embodiment.

Hereinafter, each embodiment of the present invention will be described with reference to the drawings.
In the present specification and each figure, the same elements as those already described are designated by the same reference numerals, and detailed description thereof will be omitted as appropriate.

FIG. 1 is a schematic diagram showing a configuration of a model update support system according to the first embodiment.
The model update support system 110 according to the first embodiment shown in FIG. 1 is used to support the update of the trained model.

For example, a trained model may be used to classify data. If the model is sufficiently and properly trained for each classification, the model can more accurately infer the classification of the input data.

However, for some classifications, the model may not be well trained or the model may not be trained properly. In this case, the classification and the data to be inferred may be inferred to another classification. Or, it was inferred to be the classification, but the classification certainty may be low.

The classification confidence is a value calculated when the model classifies the data. The classification conviction indicates the certainty of the inferred classification. The higher the classification confidence, the more likely it is that the classification of the data inferred by the model matches the actual classification.

Hereinafter, the data in which the correct classification could not be inferred by the model, or the data in which the correct classification could be inferred but the classification certainty was low, are referred to as "abnormal" data. Data in which the correct classification can be inferred by the model and the classification certainty is high are called "normal" data.

If abnormal data occurs when classifying data by a model, it is desirable to update (re-learn) the model. However, the anomaly is often due to the training data used to train the model or the internal structure of the model. Therefore, it is not easy for the user to determine what is the cause of the abnormality.

The model update support system 110 is used to provide the user with information on the cause of the abnormality and to support the model update. By providing the information from the model update support system 110, the user can know how to update the model based on the information.

The model update support system 110 according to the embodiment includes a processing unit 10. As shown in FIG. 1, the model update support system 110 may further include an acquisition unit 20, an output unit 30, a model storage unit 51, and a learning data storage unit 52.

The acquisition unit 20 acquires information such as video and audio as digital data and outputs it to the processing unit 10. The acquisition unit 20 includes, for example, at least one of an image pickup device and a microphone. The acquisition unit 20 may store the acquired information in a storage unit (not shown). In this case, the processing unit 10 accesses the storage unit and refers to the acquired data.

The processing unit 10 includes, for example, a CPU (Central Processing Unit), an electronic circuit, and the like. The processing unit 10 includes a reception unit 11, a classification certainty calculation unit 12, a determination unit 13, a similarity calculation unit 14, and a factor selection unit 15.

For example, the acquisition unit 20 takes a picture or records a voice and acquires the first data. The reception unit 11 receives the first data output from the acquisition unit 20. When the reception unit 11 receives the first data, the classification certainty calculation unit 12 accesses the model storage unit 51 and the learning data storage unit 52.

The model storage unit 51 stores the trained first model. The learning data storage unit 52 stores the learning data group used for learning the first model. The training data group includes a plurality of training data. Each training data includes one exemplary data and one label indicating the classification of the exemplary data.

The model storage unit 51 and the learning data storage unit 52 include a storage medium such as a hard disk drive, a flash memory, or a network hard disk. One storage medium may function as the model storage unit 51 and the learning data storage unit 52.

The classification certainty calculation unit 12 inputs the first data to the first model, and causes the first model to infer the classification of the first data. The classification conviction calculation unit 12 calculates the first classification conviction based on the output of the first model at the time of this inference. The first classification conviction indicates the certainty of the classification (first classification) of the first data inferred by the first model.

Further, the classification conviction calculation unit 12 sequentially inputs a plurality of exemplary data into the first model and calculates a plurality of classification convictions. The classification conviction calculation unit 12 outputs the first data, the first classification conviction, and a plurality of classification convictions based on a plurality of exemplary data to the determination unit 13.

The determination unit 13 determines whether the first classification conviction is sufficiently high based on the first classification conviction and the plurality of classification convictions. For example, the determination unit 13 calculates an average value and variation of a plurality of classification certainty, and sets a threshold value using the average value and variation. The determination unit 13 compares the first classification conviction with the set threshold value. When the determination unit 13 determines that the first classification certainty is equal to or higher than the threshold value, the determination unit 13 determines that the first data is normal. This means that the first classification certainty is high enough and the classification of the first data is likely to be inferred correctly.

The method of setting the threshold value is not limited to this example. Instead of using a plurality of classification convictions for setting the threshold value, a value preset by the user may be used as the threshold value. In this case, it is not necessary for the classification conviction calculation unit 12 to calculate a plurality of classification convictions, and the determination unit 13 to calculate the average value and the variation.

If the first classification certainty is less than the threshold value, it means that the first data is abnormal. In this case, the determination unit 13 outputs the first data to the similarity calculation unit 14, and outputs a plurality of classification convictions regarding the plurality of exemplary data to the factor selection unit 15.

The similarity calculation unit 14 calculates a plurality of similarity using the first data and a plurality of exemplary data. The plurality of similarities indicate the similarity between the first data and the plurality of exemplary data, respectively. The similarity calculation unit 14 outputs a plurality of calculated similarities to the factor selection unit 15.

The factor selection unit 15 can select the first information or the second information based on a plurality of similarities. In this example, a case where the factor selection unit 15 appropriately selects the first information or the second information based on a plurality of similarities and at least a part of a plurality of classification certainty will be described. The first information indicates that the training on the model is not sufficient. The second information indicates that any label contained in the training data group is not appropriate.

When the first information or the second information is selected, the factor selection unit 15 outputs the selected information to the output unit 30. The factor selection unit 15 may not select the first information or the second information based on a plurality of classification certainty degrees and a plurality of similarity degrees.

The output unit 30 outputs the first information or the second information so that the user can recognize it. The output unit 30 includes at least one of a monitor, a speaker, and a printer. For example, the output unit 30 includes a monitor or a printer and outputs the first information or the second information visually. The output unit 30 may output other information together with the first information or the second information. Other information includes the first data, the first classification, the first classification certainty, the first exemplary data similar to the first data, the classification of the first exemplary data, the classification certainty about the first exemplary data, as appropriate. A label determined not to exist, a second exemplary data to which the label is attached, and the like.

According to the model update support system 110 according to the first embodiment, when the first data is abnormal, information indicating the cause can be provided to the user. The user can update the first model based on the information provided. For example, if the learning about the first classification is insufficient for the first model, the first model is retrained for the first classification. If there is an error in the label, the first model is retrained using the training data with the corrected label. This makes it possible to classify the data more accurately using the first model after that.

Hereinafter, an example of processing related to the model update support system 110 will be described more specifically.
The first model is created, for example, by the following method. First, deep learning is applied to the pre-learning model for tasks such as classifying the input data by type. Next, pre-learning is performed by inputting example data without a label. Then, Fine Tuning is performed using the illustrated (labeled) data taught for each type of data. For the created learning model, input the data to be classified and perform classification (labeling) by deep learning.

The classification certainty calculation unit 12 inputs the first data to the trained first model and acquires an output vector from the first model. The classification certainty calculation unit 12 inputs the output vector to the softmax function, and infers the classification from which the maximum value in the output vector is obtained as the classification of the first data. The maximum value is used as the classification certainty.

The determination unit 13 sequentially inputs a plurality of exemplary data included in the learning data group into the first model, and acquires a plurality of classification convictions calculated by the classification conviction calculation unit 12. The determination unit 13 compares the average (μ) of the classification certainty, the variance (σ), and the first classification certainty (x) of the first data, and determines whether the first data is normal. Determine if it is abnormal. For example, when the following "number 1" is satisfied, the determination unit 13 determines that the first data is abnormal. α is a preset coefficient.

The similarity calculation unit 14 calculates a plurality of similarities between the first data and the plurality of exemplary data. The similarity calculation unit 14 calculates the similarity based on, for example, the Euclidean distance d represented by the following “Equation 2”. For example, the greater the value of similarity between the first data and the exemplary data, the more similar the data are. In addition to the Euclidean distance, the cosine similarity may be used to calculate the similarity.

In "Equation 2", p = (p1, p2, ..., Pi) represents the output vector of the layer immediately before the final layer at the time of inferring the first data by the first model. q = (q1, q2, ... qi) represents the output vector of the layer immediately before the final layer at the time of inferring the exemplary data by the first model. Alternatively, as the output vector, an output vector two or more layers before the final layer at the time of inference of the first data and the exemplary data or an output vector of the final layer may be used.

The factor selection unit 15 can select the first information or the second information.
For example, the first information includes the following first detailed information and second detailed information. The first detailed information indicates that there was no training data regarding the first classification of the first data at the time of training to the first model. The second detailed information indicates that there was learning data regarding the first classification at the time of learning to the first model, but learning regarding the first classification is not sufficient.

The factor selection unit 15 selects the first detailed information when the first condition is satisfied. The first condition is that the maximum value of the plurality of similarities is below the first threshold value. The maximum degree of similarity between a certain example data and the first data indicates that the example data is most similar to the first data among the training data. The fact that the maximum value of the similarity is less than the first threshold value indicates that the exemplary data most similar to the first data in the training data group is not similar to the first data. This indicates that the training data did not include data similar to the first data (data belonging to the first classification) at the time of training to the first model.

When the first condition is not satisfied, the factor selection unit 15 extracts a plurality of similar data from a plurality of exemplary data. The plurality of similar data includes the first exemplary data in which the maximum value of the plurality of similarities is obtained. The plurality of similar data are data that are relatively similar to the first data among the plurality of exemplary data. The factor selection unit 15 refers to a plurality of reference convictions, each of which indicates the certainty of classification of a plurality of similar data. The plurality of reference convictions are a part of the plurality of classification convictions calculated by the classification conviction calculation unit 12.

The factor selection unit 15 calculates the average value and the variation of the plurality of reference convictions. The factor selection unit 15 selects the second detailed information when the second condition is satisfied. The second condition is that the mean value is less than the second threshold value, or the variation is greater than or equal to the third threshold value. The second condition may be that the mean value is less than the second threshold value and the variation is greater than or equal to the third threshold value.

When the maximum value of the similarity is equal to or higher than the first threshold value, it means that the training data group contains data similar to the first data. On the other hand, the fact that the average value of the plurality of reference certainty is less than the second threshold value or the variation is equal to or more than the third threshold value indicates that the first model is not sufficiently trained with respect to the first data. That is, it is shown that the training data group did not sufficiently include the exemplary data (exemplified data belonging to the first classification) similar to the first data at the time of training the first model.

When the first condition or the second condition is not satisfied, the factor selection unit 15 determines that the inappropriate label included in the learning data group is the cause of the abnormality, and selects the second information.

Alternatively, when the first condition or the second condition is not satisfied, the factor selection unit 15 refers to a plurality of classifications of a plurality of similar data inferred by the first model. The factor selection unit 15 compares a plurality of labels assigned to the plurality of similar data and a plurality of classifications for each of the plurality of similar data. The factor selection unit 15 selects the second information for a certain similar data when one of the plurality of labels and one of the plurality of classifications do not match. When the plurality of classifications and the plurality of labels match, the factor selection unit 15 does not select any information and ends the process.

FIG. 2 is a schematic diagram illustrating the output by the model update support system according to the first embodiment.
FIG. 3 is a flowchart illustrating processing using the model update support system according to the first embodiment.
4 and 5 are schematic views illustrating the output by the model update support system according to the first embodiment.
Here, an example in which an image of a dog is input to the first model and the first model infers the breed will be described. In this example, the output unit 30 is a monitor.

For example, when the determination unit 13 determines that the first data is normal, the processing unit 10 sends the first classification of the first data inferred by the first model and the first classification certainty to the output unit 30. Display.

FIG. 2 shows an output example when the first data is normal. In the example shown in FIG. 2, the processing unit 10 causes the first data a1, the first classification a2, the first classification certainty a3, the certainty a4 of each classification related to the first data, and one or more similar to the first data. The exemplary data a5, the classification a6 of the exemplary data inferred by the first model, the classification certainty a7, the label a8 attached to the exemplary data, the certainty a9 of each classification regarding the exemplary data, and the like are displayed.

When the input data is determined to be abnormal by the determination unit 13, a plurality of similarities between the data and the plurality of exemplary data are input to the factor selection unit 15, and the processing of the flowchart shown in FIG. 3 is started. Will be done. The factor selection unit 15 causes the output unit 30 to display the first data determined to be abnormal and the classification certainty of each classification regarding the first data (step S1).

The user confirms the first data displayed on the output unit 30, and determines whether or not the data has a visual defect (step S2). As an appearance defect, for example, there is a case where the entire screen is blurred and the image itself cannot be recognized. If there is a defect in appearance, the factor for determining that the data is abnormal is determined to be a defect at the time of shooting (step S3).

If there is no problem in appearance, the factor selection unit 15 causes the output unit 30 to display exemplary data having a relatively high degree of similarity to the first data (step S4). The user determines whether the displayed exemplary data is similar to the first data (step S5).

When the displayed exemplary data is not similar to the first data, the factor selection unit 15 determines that the cause of the abnormality is insufficient learning of the first model (step S6), and selects the first detailed information. That is, it is determined that the first model has not been learned about the first data.

Instead of the user determining whether the exemplary data is similar to the first data, the factor selection unit 15 may determine using the degree of similarity. For example, the factor selection unit 15 determines whether the first condition is satisfied, as described above. The factor selection unit 15 selects the first detailed information when the first condition is satisfied.

When the displayed exemplary data is similar to the first data (the first condition is not satisfied), the factor selection unit 15 determines whether the second condition is satisfied as described above (step S7). When the second condition is satisfied, the factor selection unit 15 determines that the cause of the abnormality is insufficient learning of the first model (step S8), and selects the second detailed information. More specifically, it is judged that the learning regarding the first data is performed for the first model, but the learning is not sufficient.

When the second condition is not satisfied, the factor selection unit 15 matches the classification of the plurality of similar data inferred by the first model and the label given to the plurality of similar data for each of the plurality of similar data. It is determined whether to do it (step S9). If they do not match, the factor selection unit 15 determines that the cause of the abnormality is a label error (step S10), and selects the second information. If they match, the factor selection unit 15 determines that there is no problem regarding the abnormality (step S11), and ends the process.

In the above process, when the factor selection unit 15 selects at least one of the information, the factor selection unit 15 causes the output unit 30 to display the information. FIG. 4 illustrates a case where the factor selection unit 15 causes the output unit 30 to display the first information and the second information.

In the example of FIG. 4, for the first data, the first data b1, the first classification b2, the first classification certainty b3, and the certainty b4 of each classification regarding the first data are displayed. Regarding the first information, the exemplary data c1 similar to the first data, the classification c2 of the exemplary data c1 inferred by the first model, the classification certainty c3 indicating the certainty of the classification c2, and the label c4 given to the exemplary data c1. , The certainty degree c5 of each classification regarding the exemplary data c1, the first information c6, and the like are displayed. Regarding the second information, it is given to the exemplary data d1 in which a discrepancy between the label and the classification is found, the classification d2 of the exemplary data d1 inferred by the first model, the classification certainty d3 indicating the certainty of the classification d2, and the exemplary data d1. The label d4, the certainty degree d5 of each classification regarding the exemplary data d1, the second information d6, and the like are displayed.

The area in which the example data relating to the first information or the second information is displayed may be displayed in a distinctive manner with respect to the area in which other data is displayed. In the example shown in FIG. 4, the pattern is displayed in the area where the exemplary data c1 and the exemplary data d1 are displayed so as to be distinguishable from other data. Instead of the pattern, different colors may be added.

The processing unit 10 may output still another information to the output unit 30. For example, the processing unit 10 may display an attraction map (SaliencyMap) indicating which part of the data is reacting at the time of inference on the output unit 30. The processing unit 10 may display R, G, and B of the image of the first data separately. Further, the image of the first data and the attraction degree map may be superimposed and displayed.

Here, a case where one data is input to the processing unit 10 has been described. A plurality of data (for example, a plurality of images) may be input to the processing unit 10. At this time, a histogram of the classification as shown in FIG. 5 may be presented for the plurality of data initially input. As a result, the user can determine whether or not there is a bias for each classification. If there is a bias, it is possible to detect the possibility of an arbitrarily performed inspection.

As described above, according to the model update support system including the processing unit 10 including the factor selection unit 15, the classification certainty indicating the certainty of the classification of the first data calculated using the first model. And a plurality of similarities indicating the similarity between the first data and the plurality of exemplary data, and the first information indicating that the training to the first model is insufficient, or one of the plurality of labels. It is possible to output second information indicating that one is not appropriate. By providing the user with information indicating the cause of the abnormality, it becomes easy to update the first model.

In the example shown in FIG. 1, the processing unit 10 includes a reception unit 11, a classification certainty calculation unit 12, a determination unit 13, and a similarity calculation unit 14 in addition to the factor selection unit 15. Not limited to this example, the processing unit 10 may not include other than the factor selection unit 15. For example, the similarity degree and the classification certainty may be calculated by another processing unit, and the calculation result may be input to the processing unit 10.

FIG. 6 is a schematic diagram showing the configuration of the model update support system according to the second embodiment.
In the model update support system 210 according to the second embodiment shown in FIG. 6, the processing unit 10 further includes a labeling unit 16 and an update unit 17. The model update support system 210 further includes an input unit 40.

The processing in the reception unit 11, the classification certainty calculation unit 12, the determination unit 13, the similarity calculation unit 14, and the factor selection unit 15 of the processing unit 10 is the same as that of the model update support system 110. For example, the factor selection unit 15 selects the first information or the second information, and the information is output from the output unit 30. The user operates the input unit 40 with reference to the output first information or the second information.

The input unit 40 includes at least one of a keyboard, a mouse, a touch panel, and a microphone (voice operation).

When the first information is output, the user performs an operation of adding, for example, learning data relating to the original classification of the first data determined to be abnormal to the learning data storage unit 52. When the second information is output, the user performs an operation of inputting the correct label. When the user inputs a label, the label assigning unit 16 assigns the label input by the user to the exemplary data related to the second information. The label giving unit 16 stores the exemplary data and the label in the learning data storage unit 52.

When the learning data group of the learning data storage unit 52 is changed, the update unit 17 updates (re-learns) the first model of the model storage unit 51 using the changed learning data group. The change of the training data group is the addition of the training data or the correction of the label. The update unit 17 stores the updated first model in the model storage unit 51.

When the processing unit 10 includes the labeling unit 16 and the updating unit 17, it is possible not only to provide the user with information indicating the cause of the abnormality, but also to update the first model for improving the abnormality. As a result, the convenience of the user can be improved.

In FIG. 6, one processing unit functions as a reception unit 11, a classification certainty calculation unit 12, a determination unit 13, a similarity calculation unit 14, a factor selection unit 15, a label assignment unit 16, and an update unit 17. expressed. Not limited to this example, these functions may be realized by a plurality of processing units. For example, one processing unit functions as a reception unit 11, a classification certainty calculation unit 12, a determination unit 13, a similarity calculation unit 14, and a factor selection unit 15, and another processing unit functions as a labeling unit 16 and an update unit. It may function as a function as 17. A system including those processing units can be regarded as substantially including the processing unit 10.

According to each of the embodiments described above, it is possible to provide a model update support system capable of outputting information indicating the cause of the abnormality.

The processing of the various data described above is executed based on, for example, a program (software). For example, a computer stores this program and reads this program to process the various information described above.

The processing of the above various information can be performed by a computer as a program that can be executed by a magnetic disk (flexible disk, hard disk, etc.), an optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD ± R). , DVD ± RW, etc.), semiconductor memory, or other recording medium.

For example, the information recorded on the recording medium can be read out by a computer (or an embedded system). In the recording medium, the recording format (storage format) is arbitrary. For example, the computer reads a program from the recording medium and causes the CPU to execute the instructions described in the program based on the program. In the computer, the acquisition (or reading) of the program may be performed through the network.

At least some of the above information processing may be performed in various software running on the computer based on a program installed on the computer (or embedded system) from the recording medium. This software includes, for example, an OS (operating system). This software may include, for example, middleware running on a network.

The recording medium according to the embodiment stores a program capable of causing a computer to process the various information described above. The recording medium according to the embodiment also includes a recording medium in which the program is downloaded and stored by LAN, the Internet, or the like. The above processing may be performed based on a plurality of recording media.

The computer according to the embodiment includes one or more devices (eg, a personal computer, etc.). The computer according to the embodiment may include a plurality of devices connected by a network.

Hereinafter, embodiments of the present invention have been described with reference to specific examples. However, the embodiments of the present invention are not limited to these specific examples. For example, regarding the specific configuration of each element such as the processing unit, the acquisition unit, the output unit, the input unit, and the storage unit, the present invention is similarly carried out by appropriately selecting from a range known to those skilled in the art. As long as the effect can be obtained, it is included in the scope of the present invention.

Further, a combination of any two or more elements of each specific example to the extent technically possible is also included in the scope of the present invention as long as the gist of the present invention is included.

In addition, all model update support systems that can be appropriately designed and implemented by those skilled in the art based on the model update support system described above as an embodiment of the present invention are also included in the present invention as long as the gist of the present invention is included. It belongs to the range of.

In addition, in the scope of the idea of the present invention, those skilled in the art can come up with various modified examples and modified examples, and it is understood that these modified examples and modified examples also belong to the scope of the present invention. ..

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and variations thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

10 Processing unit, 11 Reception unit, 12 Classification certainty calculation unit, 13 Judgment unit, 14 Similarity calculation unit, 15 Factor selection unit, 16 Labeling unit, 17 Update unit, 20 Acquisition unit, 30 Output unit, 40 Input unit , 51 model storage unit, 52 learning data storage unit, 110, 210 model update support system, S1 to S11 steps, a1 first data, a2 first classification, a3 first classification certainty, a4 certainty, a5 example data, a6 classification, a7 classification certainty, a8 label, a9 certainty, b1 first data, b2 first classification, b3 first classification certainty, b4 certainty, c1 exemplary data, c2 classification, c3 classification certainty, c4 label , C5 certainty, c6 first information, d1 exemplary data, d2 classification, d3 classification certainty, d4 label, d5 certainty, d6 second information

Claims (9)

A model update support system that supports updating of a first model trained using a learning data group including a plurality of exemplary data and a plurality of labels assigned to the plurality of exemplary data, respectively.
The classification certainty that indicates the certainty of classification of the first data calculated using the first model, and the plurality of similarities that indicate the similarity between the first data and the plurality of exemplary data, respectively. Based on this, the model update support provided with a processing unit capable of outputting the first information indicating that the training to the first model is not sufficient or the second information indicating that one of the plurality of labels is not appropriate. system. When the processing unit outputs the first information, the processing unit outputs the first information, the first data, the first exemplary data included in the plurality of exemplary data, and the classification certainty. The model update support system described in Item 1. The first information is
The first detailed information indicating that the learning about the first data is not performed in the first model, and
Second detailed information indicating that learning about the first data is insufficient in the first model, and
Including
The model update support system according to claim 2, wherein the processing unit can output either the first detailed information or the second detailed information as the first information. For a plurality of similar data extracted from the plurality of exemplary data, including the first similarity data from which the maximum value of the plurality of similarities was obtained, the processing unit calculated using the first model. Refer to the plurality of reference confidence levels, each of which indicates the certainty of classification of the plurality of similar data.
The processing unit has a first condition in which the maximum value of the plurality of similarities is less than the first threshold value, and the average value of the plurality of reference confidence levels is less than the second threshold value, or the plurality of reference confidence levels. The model update support system according to claim 2 or 3, wherein when at least one of the second condition in which the variation is equal to or greater than the third threshold value is satisfied, the first information is output. When the processing unit outputs the second information, the second information, one of the plurality of labels, the second exemplary data to which the one of the plurality of labels is attached, and the first model are used. The second classification of the inferred second exemplary data and the second classification are output.
The second category is the model update support system according to claim 1, which is different from the one of the plurality of labels. The processing unit
A plurality of similar data extracted from the plurality of exemplary data including the first similar data obtained from which the maximum value of the plurality of similarities is obtained, and a plurality of the plurality of similar data inferred by the first model. Classification and, see,
For each of the plurality of similar data, a part of the plurality of labels given to the plurality of similar data and the plurality of classifications are compared.
When one of the plurality of labels does not match one of the plurality of classifications, the one of the plurality of labels is used as the second exemplary data, and the one of the plurality of classifications is output as the second classification. do,
The model update support system according to claim 5. An output unit capable of outputting the first information or the second information,
An input unit capable of inputting the operation related to the first information or the second information, and
Further prepare
The model update support system according to any one of claims 1 to 6, wherein the processing unit updates the first model based on the operation when the operation is received. Accept the input of the first data,
The first data is input to a first model trained using a learning data group including a plurality of exemplary data and a plurality of labels assigned to the plurality of exemplary data, respectively, and the first data is used. Obtain the first classification certainty, which indicates the certainty of classification,
A plurality of similarities indicating the similarity between the first data and the plurality of exemplary data were calculated.
The first exemplary data is extracted from the plurality of exemplary data based on the plurality of similarities.
In the processing unit, in addition to the output of the first data, the first classification certainty, and the first exemplary data, the first data is determined to be abnormal based on the first classification certainty. Is provided with the processing unit capable of outputting information related to the learning of the first model.
The first condition set based on the plurality of similarities or the second condition set based on a plurality of reference confidence levels indicating the certainty of classification of some of the plurality of exemplary data is satisfied. In the case, the information includes the first information indicating that the training to the first model is not sufficient.
A model update support system comprising second information indicating that one of the plurality of labels is not appropriate if neither the first condition nor the second condition is satisfied. The processing unit extracts a plurality of similar data including the first exemplary data from the plurality of exemplary data based on the plurality of similarity degrees.
The first exemplary data is data in which the maximum degree of similarity is obtained among the plurality of exemplary data.
The plurality of similar data include second exemplary data different from the first exemplary data.
When the classification of the second exemplary data by the first model and the label given to the second exemplary data do not match, the processing unit assigns the second exemplary data and the second exemplary data. The model update support system according to claim 8, further outputting the label and the label.

Images