JP7266075B2 - Data selection support device, data selection support method, and data selection support program - Google Patents

Description

The present disclosure relates to technology for assisting selection of training data used in model learning.

When training a model, a large amount of training data is used as input. The prepared training data may include training data that hinders the improvement of model accuracy through learning. Training data that hinders accuracy improvement is, for example, mislabeled training data. Mislabeled training data is, for example, training data in which image data of a dog is labeled as a cat. It is desirable to perform learning after removing training data that hinders accuracy improvement.

There is a method of applying an optimization algorithm to iteratively train and validate the model to screen the training data. This method requires thousands to tens of thousands of iterations of model training and validation.

When performing learning by deep learning, it takes a long time for one training session. Therefore, unless it is a special environment, it is unrealistic to apply an optimization algorithm when learning by deep learning. A special environment is an environment in which a GPU cluster can be used. GPU is an abbreviation for Graphics Processing Unit.

Non-Patent Document 1 describes Data Cartography (DC). DC is a method of observing the temporal transition of the degree of certainty of an image, which is training data, during deep learning training to classify the image.
DC computes the mean and standard deviation of the confidence at each epoch for the image. Then, based on the mean and standard deviation, the images are classified as easy to learn, improve generalization performance, or difficult to learn. Images classified as hard to learn are likely to hinder the model from improving accuracy.
In DC, image classification can be done by performing model training and validation only once. Therefore, it can be applied to learning by deep learning.

Swayamdipta, Swabha, et al. "Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics." Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro cessing (EMNLP). 2020. Rafael Muller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems, pp. 4694-4703, 2019.

In DC, when the model is sufficiently trained, images labeled with a small number of data are classified as those that improve generalization performance, even if they should be classified as difficult to learn. It is highly possible (see Non-Patent Document 1).

In addition, deep learning tends to increase the degree of certainty (see Non-Patent Document 2). Therefore, training data tends to be easily classified into those that are easy to learn.

From the above, DC may not properly classify images that should be classified as difficult to learn. Therefore, there is a possibility that images that hinder the improvement of model accuracy are not properly extracted.

An object of the present disclosure is to realize a configuration that can appropriately extract training data that hinders improvement in model accuracy.

The data selection support device according to the present disclosure is
a certainty average calculation unit that calculates the average value μ _e of the model's certainty for each of the plurality of training data for any epoch e equal to or less than the epoch number E;
a divergence calculation unit for calculating, for each of the plurality of training data, a degree of divergence between the confidence of the model at the epoch e and the average value μ _e calculated by the confidence average calculation unit; .

The confidence average calculation unit calculates an average value μ _e of the confidence of the model for each of the plurality of training data for each epoch e equal to or less than the epoch number E,
The divergence calculator calculates the divergence between the confidence of the model at each epoch e and the average value μ _e calculated by the average confidence calculator for each of the plurality of training data. do.

The divergence calculator calculates an average value μ _i of the divergence in each epoch e.

The data selection support device further
a confidence variation calculation unit that calculates a confidence variation of the model for each of the plurality of training data for each epoch e;
The divergence calculator calculates an average value μ _i of the divergence using the variation in the confidence calculated by the confidence variance calculator.

The certainty average calculator calculates i=1, . . . _, N with respect to the correct label y _i ^* of _the training data image xi for each integer i;
The confidence variation calculation unit calculates the confidence variation σ _e as shown in Equation 2 for each epoch e,
The divergence calculator calculates i=1, . . . , N, the average value μ _i of the deviation is calculated as shown in Equation (3).

The data selection support device further
A divergence degree variation calculation unit is provided for calculating the variation in the degree of divergence calculated by the divergence degree calculation unit.

The data selection support device further
As shown in Equation 4, a divergence degree variation calculator is provided for calculating the divergence degree variation _σi calculated by the divergence degree calculator.

The data selection support device further
A data selection unit that selects training data to be deleted from the plurality of training data based on at least one of the degree of deviation and variation in the degree of deviation.

The data selection unit selects training data to be deleted based on the feature distance between each training data and other training data.

The data selection support device further
A feature display unit is provided for displaying a feature map obtained by plotting the plurality of training data, with one of the degree of deviation and the variation of the degree of deviation set as a vertical axis and the other set as a horizontal axis.

The feature display section displays feature distances between each training data and other training data.

The data selection support method according to the present disclosure is
A computer calculates the average value μ _e of the model confidence for each of the plurality of training data for any epoch e equal to or less than the epoch number E,
A computer calculates the degree of divergence between the confidence of the model at the epoch e and the mean μ _e for each of the plurality of training data.

The data selection support program according to the present disclosure is
Confidence average calculation processing for calculating the average value μ _e of the model confidence for each of the plurality of training data for any epoch e equal to or less than the epoch number E;
performing a divergence calculation process of calculating a divergence between the confidence of the model at the epoch e and the mean value _μe calculated by the confidence average calculation process for each of the plurality of training data; The computer functions as a data selection support device.

In the present disclosure, the average value μ _e of the model confidence for each of the plurality of training data is calculated for the epoch e, and the divergence between the model confidence and the average value μ _e is calculated for each of the plurality of training data. calculate. By using this degree of divergence, it is possible to realize a configuration that can appropriately extract an image that hinders improvement in model accuracy.

1 is a configuration diagram of a data selection support device 10 according to Embodiment 1. FIG. 4 is a flowchart showing the overall flow of operations of the data selection support device 10 according to Embodiment 1; 4 is a flowchart showing details of the operation of the data selection support device 10 according to the first embodiment; FIG. 5 is an explanatory diagram of the degree of divergence d _i,e according to the first embodiment; FIG. 5 is an explanatory diagram of the degree of divergence d _i,e according to the first embodiment; FIG. 4 is an explanatory diagram of the effect of the first embodiment; FIG. 2 is a configuration diagram of a data selection support device 10 according to Modification 1; FIG. 2 is a configuration diagram of a data selection support device 10 according to Embodiment 2; 8 is a flowchart showing details of the operation of the data selection support device 10 according to the second embodiment; FIG. 8 is an explanatory diagram of the effect of the second embodiment; The figure which shows the change of the certainty degree of the image of the label with many data, and the image of the label with few data. The figure which shows the change of the confidence with the image of the label with few data. FIG. 10 is a diagram showing the degree of divergence d _i,e of an image that hinders accuracy improvement in an image of a label with a small number of data; The figure which shows the change of the certainty factor with the image of the label with many data. FIG. 10 is a diagram showing the degree of divergence d _i,e of an image that hinders accuracy improvement in an image labeled with a large amount of data; FIG. 10 is a configuration diagram of a data selection support device 10 according to Embodiment 3; 10 is a flowchart showing details of the operation of the data selection support device 10 according to the third embodiment; An explanatory diagram of an image in which it is difficult to distinguish which label is correct. FIG. 10 is a configuration diagram of a data selection support device 10 according to Embodiment 4; FIG. 11 is a flow chart showing details of the operation of the data selection support device 10 according to the fourth embodiment; FIG.

Embodiment 1.
*** Configuration description ***
The configuration of the data selection support device 10 according to the first embodiment will be described with reference to FIG.
The data selection support device 10 is a computer.
The data selection support device 10 includes hardware including a processor 11 , a memory 12 , a storage 13 and a communication interface 14 . The processor 11 is connected to other hardware via signal lines and controls these other hardware.

The processor 11 is an IC that performs processing. IC is an abbreviation for Integrated Circuit. The processor 11 is, for example, a CPU, DSP, or GPU. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor. GPU is an abbreviation for Graphics Processing Unit.

The memory 12 is a storage device that temporarily stores data. Specific examples of the memory 12 are SRAM and DRAM. SRAM is an abbreviation for Static Random Access Memory. DRAM is an abbreviation for Dynamic Random Access Memory.

The storage 13 is a storage device that stores data. A specific example of the storage 13 is an HDD. HDD is an abbreviation for Hard Disk Drive. The storage 13 may be a portable recording medium such as an SD (registered trademark) memory card, CompactFlash (registered trademark), NAND flash, flexible disk, optical disk, compact disk, Blu-ray (registered trademark) disk, or DVD. SD is an abbreviation for Secure Digital. DVD is an abbreviation for Digital Versatile Disk.

The communication interface 14 is an interface for communicating with an external device. The communication interface 14 is, for example, an Ethernet (registered trademark), USB, or HDMI (registered trademark) port. USB is an abbreviation for Universal Serial Bus. HDMI is an abbreviation for High-Definition Multimedia Interface.

The data selection support device 10 includes an input reception unit 21 and an accuracy calculation unit 22 as functional components. The accuracy calculator 22 includes a certainty average calculator 221 , a certainty variation calculator 222 , and a divergence calculator 223 . The function of each functional component of the data selection support device 10 is implemented by software.
The storage 13 stores a program that implements the function of each functional component of the data selection support device 10 . This program is read into the memory 12 by the processor 11 and executed by the processor 11 . Thereby, the function of each functional component of the data selection support device 10 is realized.

Only one processor 11 is shown in FIG. However, there may be a plurality of processors 11, and the plurality of processors 11 may cooperate to execute programs that implement each function.

***Description of operation***
The operation of the data selection support device 10 according to the first embodiment will be described with reference to FIGS. 2 to 5. FIG.
The operation procedure of the data selection support device 10 according to the first embodiment corresponds to the data selection support method according to the first embodiment. Also, a program that realizes the operation of the data selection support device 10 according to the first embodiment corresponds to the data selection support program according to the first embodiment.

An overall flow of operations of the data selection support device 10 according to the first embodiment will be described with reference to FIG.
(Step S1: learning process)
The learning device 31 receives a plurality of training data 41 and a plurality of verification data 42 and performs learning by deep learning to update the model 32 . The training data 41 is data for learning the model 32 . The verification data 42 is data for verifying the accuracy of the learned model 32 .
When performing learning by deep learning, a recognition result for each piece of training data 41 is obtained. Also, a confidence factor for the recognition result is calculated. In this embodiment, it is assumed that the certainty is calculated in the range of 0 to 1. The recognition result is, for example, whether or not the recognition target is included in the image. The degree of certainty is the degree of certainty (whether the i-th training data x _i is the correct label y _i ^* ) with which the recognition result is recognized.
In deep learning, learning is repeatedly performed multiple times using each of a plurality of training data 41 . The number of repetitions of learning using the same training data 41 is called the number of epochs E. FIG. The number of epochs E is two or more. The e-th learning is called an epoch e. That is, learning from epoch 1 to epoch E is performed using the same training data 41 .

(Step S2: sorting process)
The data selection support device 10 acquires the certainty factor obtained in step S1. The data selection support device 10 calculates the degree of divergence based on the degree of certainty. Then, the training data 41 is sorted using the degree of divergence.

The processing from step S1 to step S2 is repeatedly executed as necessary. When it is executed for the second time or later, learning is performed with the selected training data 41 as input. As a result, a model 32 with high accuracy is generated.
In addition, the process of step S1 may be learned by machine learning that can acquire the degree of certainty for each epoch without relying on deep learning.

Details of the operation of the data selection support device 10 according to the first embodiment will be described with reference to FIG.
(Step S11: Input reception processing)
The input reception unit 21 acquires the certainty factor for each training data 41 obtained in step S1 of FIG. Specifically, the input receiving unit 21 acquires the certainty factor in each epoch e equal to or less than the epoch number E for each piece of training data 41 .

Ｎは訓練データ４１の数である。ｐ_{ｉ，θ（ｅ）}（ｙ_ｉ ^＊｜ｘ_ｉ）は、エポックｅにおいてパラメータθのモデルが算出した、ｉ番目の訓練データ４１の画像ｘ_ｉの正解ラベルｙ_ｉ ^＊に対する確信度である。 (Step S12: Confidence Average Calculation Process)
The certainty average calculation unit 221 calculates the average value μ _e of the certainty of the model 32 for each of the plurality of training data 41 as shown in Equation 11 for any epoch e equal to or less than the epoch number E.

N is the number of training data 41 . p _i,θ(e) (y _i ^* |x _i ) is the confidence factor for the correct label y _i ^* of the i-th training data 41 image x _i calculated by the model with parameter θ at epoch e.

As a specific example, the confidence factor average calculation unit 221 calculates the average value μ _E of the confidence factors of the model 32 for the epoch E, which is the final epoch. Note that the confidence factor average calculation unit 221 may calculate the average value μ _e of the confidence factors of the model 32 not only for the final epoch but also for an epoch e (e<E) in the middle stage. Further, the certainty average calculation unit 221 may calculate the average value μ _e of the certainty of the model 32 for a plurality of epochs e.

In Embodiment 1, the certainty average calculation unit 221 calculates the average value μ _e of the certainty of the model 32 for each epoch e equal to or less than the epoch number E. FIG. That is, the certainty average calculation unit 221 calculates e=1, . . . , E, compute the mean μ _e of the confidence of the model 32 .

ここでは、ばらつきとして標準偏差が計算される。しかし、ばらつきとして分散が計算されてもよい。 (Step S13: Confidence Variation Calculation Process)
The certainty variation calculation unit 222 calculates the variation σ _e of the certainty of the model 32 for each of the plurality of training data 41 as shown in Equation 12 for the epoch e. The epoch e is the epoch in which the mean value μ _e of the confidence factor of the model 32 was calculated in step S12.

Here, the standard deviation is calculated as the variation. However, the variance may be calculated as a scatter.

In Embodiment 1, for each epoch e equal to or less than the number of epochs E, the average value μ _e of the confidence factor of the model 32 is calculated. Therefore, the certainty factor variation calculation unit 222 calculates the certainty factor variation σ _e of the model 32 for each epoch e equal to or less than the number of epochs E.

(Step S14: Deviation degree calculation process)
The divergence calculation unit 223 calculates the divergence d _i,e between the confidence of the model 32 at the epoch e and the average value μ _e for each of the plurality of training data 41 as shown in Equation 13. The average value μ _e is the value calculated in step S12. The epoch e is the epoch in which the mean value μ _e of the confidence factor of the model 32 was calculated in step S12.

As shown in FIG. 4, the degree of divergence d _i,e is the distance between the certainty of the training data 41 to be processed and the average value μ _e of the certainty of all the training data 41 . That is, the degree of divergence d _i,e indicates how high or low the degree of certainty for the training data 41 to be processed is compared to the average value μ _e .

これにより、乖離度ｄ_ｉ，ｅは、確信度のばらつきが小さい場合には、絶対値が大きくなり、確信度のばらつきが大きい場合には、絶対値が小さくなる。つまり、処理対象の訓練データ４１についての確信度と、全ての訓練データ４１の確信度の平均値μ_ｅとの距離が近い場合でも、確信度のばらつきが小さい場合には、乖離度ｄ_ｉ，ｅは大きくなる。一方、処理対象の訓練データ４１についての確信度と、全ての訓練データ４１の確信度の平均値μ_ｅとの距離が遠い場合でも、確信度のばらつきが大きい場合には、乖離度ｄ_ｉ，ｅは小さくなる。 The degree of divergence d _i,e calculated by Equation 13 may be used. However, in Embodiment 1, as shown in Equation 14, the divergence calculation unit 223 calculates the distance between the certainty of the training data 41 to be processed and the average value μ _e of the certainty of all the training data 41. is divided by the variation σ _e of the confidence to calculate the degree of divergence d _i,e .

As a result, the degree of divergence d _i,e has a large absolute value when the variation in the certainty is small, and a small absolute value when the variation in the certainty is large. In other words, even if the distance between the certainty of the training data 41 to be processed and the average value μ _e of the certainty of all the training data 41 is close, if the variation of the certainty is small, the divergence di _{, e} becomes larger. On the other hand, even when the distance between the certainty of the training data 41 to be processed and the average value μ _e of the certainty of all the training data 41 is long, if the variation of the certainty is large, the divergence di _{, e} becomes smaller.

In Embodiment 1, for each epoch e equal to or less than the number of epochs E, the average value μ _e of the confidence factor of the model 32 is calculated. Therefore, the divergence calculator 223 calculates the divergence d _i,e for each of the plurality of training data 41 in each epoch e equal to or less than the epoch number E. Then, in the first embodiment, the divergence degree calculation unit 223 calculates _the mean value μ Compute _i . Note that the divergence degree calculation unit 223 may calculate other statistical values such as the median value of the divergence degrees d _i,e instead of the average value μ _i .

That is, as shown in FIG. 5, the degree of divergence d _i,e at each epoch e is calculated. Then, for each of the plurality of training data 41, the average μ _i of the degree of divergence d _i,e in all epochs e is calculated.

*** Effect of Embodiment 1 ***
As described above, the data selection support apparatus 10 according to Embodiment 1 calculates the degree of divergence d _i,e between the certainty of the model and the average value μ _e for each of a plurality of pieces of training data. By using the degree of divergence d _i,e , it is possible to realize a configuration that can appropriately extract an image that hinders improvement in the accuracy of the model 32 .

Suppose that many training data 41 have low confidence in epoch e. If the training data 41 is simply evaluated using the certainty factor, many of the training data 41 may be determined to be inappropriate. For example, as shown in FIG. 6, assume that the average value _μe of the confidence factor is 0.35. Assume that 0.40 is used as a confidence threshold, and that the training data 41 having a confidence below the threshold is determined to be inappropriate. In this case, many training data 41 are determined to be inappropriate.
However, when the degree of divergence d _i,e is used, only the training data 41 with a particularly low degree of certainty can be extracted even when the degree of certainty is generally low. The reason for the overall low confidence is that the number of epochs where the model 32 has not been sufficiently learned or the number of target labels is smaller than other labels, but images that hinder the accuracy improvement of the model 32 can be appropriately extracted. there is a possibility.

In particular, in Embodiment 1, the divergence degree d _i,e is calculated by dividing the distance by the variation σ _e of the certainty. This makes it possible to appropriately extract an image that hinders improvement in the accuracy of the model 32 in consideration of the variation σ _e of the confidence.

Further, in Embodiment 1, the average value μ _i of the degrees of divergence d _i,e in all epochs e equal to or less than the number of epochs E is calculated. This makes it possible to appropriately extract an image that hinders improvement in the accuracy of the model 32 in consideration of the temporal transition of the degree of divergence d _i,e .

***Other Configurations***
<Modification 1>
In Embodiment 1, each functional component is realized by software. However, as Modification 1, each functional component may be implemented by hardware. Differences between the first modification and the first embodiment will be described.

The configuration of the data selection support device 10 according to Modification 1 will be described with reference to FIG.
When each functional component is realized by hardware, the data selection support device 10 includes an electronic circuit 15 instead of the processor 11, memory 12 and storage 13. FIG. The electronic circuit 15 is a dedicated circuit that realizes the functions of each functional component, memory 12 and storage 13 .

Electronic circuit 15 may be a single circuit, multiple circuits, programmed processors, parallel programmed processors, logic ICs, GAs, ASICs, FPGAs. GA is an abbreviation for Gate Array. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field-Programmable Gate Array.
Each functional component may be implemented by one electronic circuit 15, or each functional component may be implemented by being distributed among a plurality of electronic circuits 15. FIG.

<Modification 2>
As a modification 2, some functional components may be implemented by hardware and other functional components may be implemented by software.

The processor 11, the memory 12, the storage 13 and the electronic circuit 15 are called a processing circuit. That is, the function of each functional component is realized by the processing circuit.

Embodiment 2.
The second embodiment differs from the first embodiment in that the variation σ _i of the degree of divergence d _i,e is calculated. In the second embodiment, this different point will be explained, and the explanation of the same point will be omitted.

*** Configuration description ***
The configuration of the data selection support device 10 according to the second embodiment will be described with reference to FIG.
The data selection support device 10 differs from the data selection support device 10 shown in FIG. 1 in that it includes a divergence degree variation calculator 224 as a functional component. The functions of the divergence degree variation calculator 224 are implemented by software or hardware in the same manner as other functional components.

***Description of operation***
The operation of the data selection support device 10 according to the second embodiment will be described with reference to FIG.
The processing from step S21 to step S24 is the same as the processing from step S11 to step S14 in FIG.

ここでは、ばらつきとして標準偏差が計算される。しかし、ばらつきとして分散が計算されてもよい。 (Step S25: deviation degree variation calculation process)
The divergence degree variation calculation unit 224 calculates the variation σ _i of the divergence degree d _i,e calculated in step S 24 as shown in Equation 16 for each of the plurality of training data 41 .

Here, the standard deviation is calculated as the variation. However, the variance may be calculated as a scatter.

*** Effect of Embodiment 2 ***
As described above, the data selection support device 10 according to the second embodiment calculates the variation σ _i of the degree of divergence d _i,e . By using this variation σ _i , it is possible to realize a configuration that can appropriately extract an image that contributes to improving the accuracy of the model 32 .

FIG. 10 shows a feature map in which a plurality of training data 41 are plotted, with the vertical axis representing the average value μ _i of the degree of deviation d _i,e and the horizontal axis representing the variation σ _i of the degree of deviation d _i,e . FIG. 10 assumes that the model 32 identifies whether the training data 41 is normal or abnormal. Normal points are plotted when the recognition result in the final epoch is normal, and abnormal points are plotted when it is abnormal.
When the average value μ _i of the degree of divergence d _i,e is equal to or greater than the first threshold (−2.0 in FIG. 10), the image is classified as easy to learn. When the variation σ _i of the degree of divergence d _i,e is equal to or greater than the second threshold (3 in FIG. 10), the image is classified as one that improves generalization performance. When the average value μ _i of the degree of divergence d _i,e is less than the first threshold, the image is classified as difficult to learn.
In this manner, the images, which are the training data 41, can be classified based on the average μ _i of the degree of divergence d _i,e and the variation σ _i of the degree of divergence d _i,e . Then, for example, images classified as difficult to learn may be deleted from the training data 41 . Further, it is conceivable to add similar images to the training data 41 for images classified as those that improve the generalization performance. Images classified as easy-to-learn may be thinned out to reduce the number of images.

そして、非特許文献１では、確信度の平均値μ＾_ｉを縦軸とし、確信度のばらつきσ＾_ｉを横軸として、複数の訓練データ４１をプロットした特徴マップが示されている。非特許文献１では、確信度の平均値μ＾_ｉが高い画像が学習容易なものに分類される。確信度のばらつきσ＾_ｉが高い画像が汎用化性能を向上させるものに分類される。確信度の平均値μ＾_ｉが低い画像が学習困難なものに分類される。 The effect of the data selection support device 10 according to the second embodiment will be described by comparison with the DC described in Non-Patent Document 1. FIG.
In Non-Patent Document 1, as shown in Equation 17, for each of a plurality of training data 41, the average value _μ̂i of confidence factors in each epoch is calculated. Further, in Non-Patent Document 1, as shown in Equation 18, the confidence factor variation σ^ _i is calculated for each of the plurality of training data 41 .

Non-Patent Document 1 shows a feature map in which a plurality of training data 41 are plotted with the average value μ^ _i of the confidence factor on the vertical axis and the variation of the confidence factor σ^ _i on the horizontal axis. In Non-Patent Document 1, an image with a high average confidence factor μ^ _i is classified as an easy-to-learn image. Images with high confidence variance σ^ _i are classified as those with improved generalization performance. An image with a low average confidence factor μ^ _i is classified as difficult to learn.

In deep learning, confidence tends to be high. Therefore, training data tends to be classified as easy to learn (see Non-Patent Document 2). In addition, when the model is sufficiently trained, images labeled with a small amount of data may have a high degree of certainty. At this time, the image of the label with a small number of data may have large variations. Therefore, there is a possibility that an image with a label with a small amount of data that should be classified as difficult-to-learn may be classified as easy-to-learn or to improve generalization performance. Therefore, with the DC described in Non-Patent Document 1, it is difficult to appropriately extract an image that hinders improvement in model accuracy.

In FIG. 11, the dotted line indicates the transition of the average confidence factor for each epoch of images labeled with a large amount of data. The solid line indicates the transition of the average confidence factor for each epoch of images labeled with a small amount of data. As shown in FIG. 11, the change in the degree of certainty with respect to the epoch is different for images labeled with a large number of data and images labeled with a small number of data. Images with more data labels reach a higher confidence state in the early epochs because the training data is more and the model can learn a lot of features. On the other hand, as shown in FIG. 11, for images with labels with a small number of data, the confidence increases after a certain amount of epoch, or the confidence never increases, or is in between. However, when the degree of certainty of an image labeled with a small amount of data increases, features other than the label may be learned. A case where a feature other than a label is learned is, for example, a case where whether or not an image has a tank is learned, but whether or not there is a tree is learned. Models that detect anomalies are supposed to have learned the location of anomalies, but they often learn other than the location of the anomalies.
This is because the model sees the features other than the label as being emphasized rather than the label. Images that have learned anything other than labels are not useful as training data. Therefore, it is necessary to classify images with labels with a small amount of data, which may have been learned other than labels, as difficult-to-learn images.
However, in FIG. 11, if the number of epochs E is 50, the average value μ^ _i of the confidence in the DC described in Non-Patent Document 1 is about 0.5. In addition, since certainty factors range from high values to low values, the variation σ^ _i of certainty factors becomes a high value. Therefore, an image with a label with a small amount of data does not have a low degree of certainty, so it is not classified as difficult to learn. In addition, since the state of confidence is transitioned from a state of low to a state of high, it is classified as one that improves generalization performance. Also, in FIG. 11, if the number of epochs E is 100, then the average value μ^ _i of the confidence in the DC described in Non-Patent Document 1 is about 0.8. Moreover, since there are many high values as the confidence, the variation σ^ _i of the confidence becomes a low value. Therefore, it is classified as easy to learn.
As shown in FIG. 11, even if the certainty of the image of the label with a small amount of data does not eventually rise to near 1.0, for example, even if it is about 0.5, the number of epochs E is large and the model is sufficiently learned. , the average value μ^ _i increases, so it is not classified as difficult to learn. Therefore, DC leads to erroneous decisions.

In FIG. 12, transition of the certainty factor of an image of a label with a small number of data is represented in a format different from that in FIG. The confidence level of images with labels with a small amount of data has increased after a certain epoch. Then, when the number of epochs reaches 50, the certainty becomes a value close to 1.0. Therefore, as described above, an image with a label having a small number of data is not classified as difficult to learn.
In FIG. 13, the certainty factor and the divergence factor d _i,e of certain training data 41 are added to FIG. 12 . Some training data 41 here is training data 41 of an image that hinders accuracy improvement, among images with labels having a small number of data. As shown in FIG. 13, the confidence level increases as the epoch progresses, even for the training data 41 that hinders accuracy improvement. Therefore, in DC, some training data 41 is not classified as difficult to learn, like other images with labels having a small number of data.
However, the confidence of the training data 41, which hinders improvement in accuracy, is lower than the average value _μe of the confidence of other images. Therefore, the degree of divergence d _i,e becomes a low value (large negative value), and the average value μ _i of the degree of divergence d _i,e becomes a low value. Therefore, in the data selection support device 10 according to the second embodiment, it is possible to classify the training data 41, which hinders accuracy improvement, as difficult-to-learn data.
Note that even if the degree of certainty changes as the epoch progresses, if the degree of deviation d _i,e does not change, the variation σ _i of the degree of deviation d _i,e will not increase. Therefore, it is not categorized as improving general-purpose performance.

In FIG. 14, transition of the certainty factor of the image of the label with a large amount of data is represented in a format different from that in FIG. Confidence for images with labels with a large number of data has been high since early epochs. Therefore, as described above, images with labels having a large amount of data are not classified as difficult-to-learn images, but are classified as easy-to-learn images.
In FIG. 15, the certainty factor and the divergence factor d _i,e of certain training data 41 are added to FIG. 14 . Some training data 41 here is training data 41 of an image that hinders improvement in accuracy among images labeled with a large number of data. There is a possibility that image training data 41 of images that hinder improvement in accuracy may be included in images labeled with a large amount of data. As shown in FIG. 15, even for training data 41 that hinders improvement in accuracy, the degree of certainty may increase as the epoch progresses. This is because deep learning tends to have a high degree of certainty. Therefore, in DC, some training data 41 is not classified as difficult to learn, but is classified as easy to learn, like other images with labels having a large number of data.
However, the confidence of the training data 41, which hinders improvement in accuracy, is lower than the average value _μe of the confidence of other images. Therefore, the degree of deviation d _i,e becomes a low value, and the average value μ _i of the degree of deviation d _i,e becomes a low value. Therefore, in the data selection support device 10 according to the second embodiment, it is possible to classify the training data 41, which hinders accuracy improvement, as difficult-to-learn data.

The average μ _i of the degree of divergence d _i,e indicates the difficulty of learning. Also, the variation σ _i of the degree of divergence d _i,e indicates the change in the difficulty of learning. In the data selection support device 10 according to Embodiment 2, by using the average value μ _i of the degree of deviation d _i,e and the variation σ _i of the degree of deviation d _{i, e} , it is possible to appropriately determine whether learning is easy or not. It is possible to determine whether it is difficult.

Embodiment 3.
Embodiment 3 differs from Embodiments 1 and 2 in that the training data 41 is selected based on the results calculated by the accuracy calculator 22 . In the third embodiment, this different point will be explained, and the explanation of the same point will be omitted.
Embodiment 3 describes a case where a function is added to Embodiment 2. FIG. However, it is also possible to add functions to the first embodiment.

*** Configuration description ***
The configuration of the data selection support device 10 according to the third embodiment will be described with reference to FIG.
The data selection support device 10 differs from the data selection support device 10 shown in FIG. 8 in that it includes a data selection unit 23 as a functional component. The function of the data selector 23 is realized by software or hardware, like other functional components.

***Description of operation***
The operation of the data selection support device 10 according to the third embodiment will be described with reference to FIG.
The processing from step S31 to step S35 is the same as the processing from step S21 to step S25 in FIG.

(Step S36: data sorting process)
The data selection unit 23 selects the training data 41 based on the average μ _i of the degrees of divergence d _i,e calculated in step S34 and the variation σ _i of the degrees of divergence d _i,e calculated in step S35. do.
As a specific example, the data selection unit 23 classifies images as described with reference to FIG. That is, the data selection unit 23 classifies the images into those that are easy to learn, those that improve generalization performance, and those that are difficult to learn. Then, the data selection unit 23 may delete images classified as difficult to learn from the training data 41 . In addition, the data selection unit 23 may thin out images classified as easy-to-learn images in order to reduce the number of images. The data selection unit 23 may thin out more images classified as easy-to-learn images as the average value μ _i of the degree of divergence d _i,e is higher.
Note that for images classified as those that improve the generalization performance, notification may be made to add similar images to the training data 41 .

*** Effect of Embodiment 3 ***
As described above, the data selection support device 10 according to Embodiment 3 selects the training data 41 based on the average value μ _i of the degrees of divergence d _i,e and the variation σ _i of the degrees of divergence d _i,e. . This makes it possible to automatically delete the unnecessary training data 41 without manual intervention.
The unnecessary training data 41 are the training data 41 that is likely to hinder the accuracy improvement of the model 32 and the training data 41 unnecessary for the accuracy improvement of the model 32 .

***Other Configurations***
<Modification 4>
The incorrectly labeled training data 41 has a low average μ _i of the degree of divergence d _i,e . However, even if the labeling is correct, the average μ _i of the degree of divergence d _i,e may be low. For example, if the features are similar between different labels, the average μ _i of the degree of divergence d _i,e may be low. In addition, an image with unique features and a small number of types may have a low average μ _i of the degrees of divergence d _i,e .
The training data 41 shown in FIG. 18A is an image of a dog, labeled as dog, and is suitable training data 41 . However, some images of the dog X are difficult to distinguish from the image of the mop shown in (B). Therefore, if the training data 41 contains few images of the dog X in (A) and many images of the mop as in (B), the image of the dog X is affected by the training data 41 labeled with the mop. Confidence is hard to come by.
For images such as dog X, it is desirable to add an image of the same looking dog to the training data 41 instead of deleting it from the training data 41 . That is, dog X is unique in characteristics compared to other dogs. Therefore, it is desirable to add images of dogs having similar characteristics to the training data 41 for learning. However, as described in the third embodiment, if the threshold value of the average value μ _i of the degree of divergence d _i,e is simply used to determine whether or not to delete the image, the image like the dog X will be deleted. It may get lost.

It is desirable to remove only images that are mislabeled. As an example, if all of the following conditions (1) to (3) are satisfied, the image is considered to be labeled incorrectly. (1) Confidence does not increase even after learning. (2) images with different labels are similar in features; (3) images with the same label are not similar in features;
Regarding (1), it can be judged from the transition of the degree of divergence d _i,e when the epoch progresses. (2) and (3) can be judged by the distance between the features. The distance is a cos distance as a specific example. If the features are similar, the distance is short. If the features are not similar, the distance is long.

Therefore, the data selection unit 23 may perform the following processing to determine whether or not the image is incorrectly labeled.
Procedure 1. The data selection unit 23 prepares a learned model 32 .
Procedure 2. The data selection unit 23 uses the trained model 32 to extract about 512-dimensional features of the image.
Step 3. The data selection unit 23 calculates the distance (cos distance, etc.) between each image.
Step 4. For each image, the data selection unit 23 identifies the minimum distance between an image with a label different from that of the image and the image with the target label.
Step 5. For each image, the data selection unit 23 identifies the minimum distance between an image with the same label as the image and the image with the target label.
Step 6. The data selection unit 23 extracts images that satisfy all of the following conditions C1 to C3 as images that are difficult to learn (images with incorrect labeling). (C1) The average μ _i of the degrees of deviation d _i,e is less than or equal to the reference degree of deviation (eg, −2). (C2) The minimum distance in Procedure 4 is less than or equal to the first distance (eg, 0.1). (C3) The minimum distance in procedure 5 is greater than or equal to the second distance (eg, 0.3).
Step 7. The data selection unit 23 determines to delete the extracted image.
Note that here the minimum distance between the labeled images of interest was used. However, an average distance may be used instead of the minimum distance.

Embodiment 4.
Embodiment 4 differs from Embodiments 1 and 2 in that a feature map or the like is displayed to select training data 41 . In the fourth embodiment, this different point will be explained, and the explanation of the same point will be omitted.
In a fourth embodiment, a case in which functions are added to the second embodiment will be described. However, it is also possible to add functions to the first embodiment.

*** Configuration description ***
The configuration of the data selection support device 10 according to the fourth embodiment will be described with reference to FIG.
The data selection support device 10 differs from the data selection support device 10 shown in FIG. 8 in that it includes a characteristic display section 24 as a functional component. The function of the characteristic display section 24 is realized by software or hardware, like other functional components.

***Description of operation***
The operation of the data selection support device 10 according to the fourth embodiment will be described with reference to FIG.
The processing from step S41 to step S45 is the same as the processing from step S21 to step S25 in FIG.

(Step S46: Feature display processing)
The feature display unit 24 displays the feature map shown in FIG. 10 and allows the user to select the training data 41 . Note that the _{characteristic} display unit ₂₄ is not limited to the form of the characteristic map _shown in FIG _. should be displayed.
The sorting is performed by the user confirming the classified images. In other words, the judgment is made by looking at the image instead of uniformly processing with a threshold value.

*** Effect of Embodiment 4 ***
As described above, in the fourth embodiment, a feature map or the like is displayed to allow the user to select training data 41 . This makes it possible to prevent the training data 41 that should not be deleted from being deleted.

It is desirable to add an image of a dog with the same appearance to the training data 41 instead of deleting the image like the dog X in FIG. 18A from the training data 41 . However, as described in the third embodiment, if the threshold value of the average value μ _i of the degree of divergence d _i,e is simply used to determine whether or not to delete the image, the image like the dog X will be deleted. It may get lost. On the other hand, by displaying a feature map or the like and allowing the user to select the training data 41 as in the fourth embodiment, it is possible to prevent images such as the dog X from being deleted. Also, a decision to add an image like dog X can be prompted.

***Other Configurations***
<Modification 5>
It is difficult to determine whether or not to delete simply based on the average value μ _i of the degree of divergence d _i,e . Therefore, the feature display unit 24 may display the distance described in the fourth modification together with the average value μ _i of the degrees of divergence di _,e . That is, the feature display unit 24 may display at least one of the minimum distance in procedure 4 and the minimum distance in procedure 5 together with the average value μ _i of the degree of divergence di _,e .
As a specific example, the feature display unit 24 plots and displays the information of each image in a two-dimensional space in which the vertical axis is the average value μ _i of the degrees of divergence d _{i and e} and the horizontal axis is the minimum distance in step 4. do. Further, the feature display unit 24 may plot and display the information of each image in a two-dimensional space with the average value μ _i of the degrees of divergence d _{i and e} as the vertical axis and the minimum distance in step 5 as the horizontal axis. . In addition, the feature display unit 24 displays each image in a three-dimensional space with the minimum distance in procedure 4 as the X axis, the minimum distance in procedure 5 as the Y axis, and the average value μ _i of the degrees of divergence d _{i and e} as the Z axis. information may be plotted and displayed.
As a result, it is possible to determine whether or not to delete in consideration of the distance described in Modification 4 together with the average value μ _i of the degree of divergence d _i,e .

<Modification 6>
In the above description, selection of the training data 41 has been described. However, it is also possible to screen validation data 42 instead of training data 41 by the same technique.

Note that "unit" in the above description may be read as "circuit", "process", "procedure", "process", or "processing circuit".

The embodiments and modifications of the present disclosure have been described above. Some of these embodiments and modifications may be combined and implemented. Also, any one or some may be partially implemented. It should be noted that the present disclosure is not limited to the above embodiments and modifications, and various modifications are possible as necessary.

10 data selection support device, 11 processor, 12 memory, 13 storage, 14 communication interface, 15 electronic circuit, 21 input reception unit, 22 accuracy calculation unit, 221 confidence average calculation unit, 222 confidence variation calculation unit, 223 divergence Calculation unit 224 deviation degree variation calculation unit 23 data selection unit 24 feature display unit 31 learning device 32 model 41 training data 42 verification data.

Claims (12)

a certainty average calculation unit that calculates the average value μ _e of the model's certainty for each of the plurality of training data for each epoch e equal to or less than the epoch number E;
a divergence calculation unit for calculating, for each of the plurality of training data, the degree of divergence between the confidence of the model at each epoch e and the average value μ _e calculated by the average confidence calculation unit; A data selection support device provided. 2. The data selection support device according to claim 1 , wherein the divergence calculation unit calculates an average value _μi of the divergence in each epoch e. The data selection support device further
a confidence variation calculation unit that calculates a confidence variation of the model for each of the plurality of training data for each epoch e;
3. The data sorting support device according to claim 2 , wherein the divergence calculating unit calculates the average value _μi of the divergence using the variation in the certainty calculated by the certainty variation calculating unit. The certainty average calculator calculates i=1, . . . _, N with respect to the correct label y _i ^* of _the training data image xi for each integer i;
The confidence variation calculation unit calculates the confidence variation σ _e as shown in Equation 2 for each epoch e,
The divergence calculator calculates i=1, . . . 4. The data selection support device according to claim 3 , wherein the average value μ _i of the degree of deviation is calculated as shown in Equation 3 for the training data for each integer i of N, .

The data selection support device further
5. The data selection support device according to any one of claims 1 to 4 , further comprising a divergence degree variation calculation unit that calculates variations in the divergence degree calculated by the divergence degree calculation unit. The data selection support device further
5. The data selection support device according to claim 4, further comprising a divergence degree variation calculation unit that calculates the deviation degree variation _σi calculated by the divergence degree calculation unit as shown in Equation 4 .

The data selection support device further
7. The data selection support device according to claim 5 , further comprising a data selection unit that selects training data to be deleted from the plurality of training data based on at least one of the degree of deviation and variation in the degree of deviation. 8. The data selection support device according to claim 7 , wherein the data selection unit selects training data to be deleted based on a feature distance between each training data and other training data. The data selection support device further
9. Any one of claims 5 to 8 , further comprising a feature display unit for displaying a feature map obtained by plotting the plurality of training data, with one of the degree of deviation and the variation of the degree of deviation set as a vertical axis and the other set as a horizontal axis. 1. The data selection support device according to 1. 10. The data selection support device according to claim 9 , wherein the feature display unit displays feature distances between each piece of training data and other training data. a computer, for each epoch e equal to or less than the number of epochs E, calculates the average value μ _e of the model confidence for each of the plurality of training data;
A data sorting support method in which a computer calculates, for each of the plurality of training data, the degree of divergence between the degree of confidence of the model in each of the epochs e and the average value μ _e . Confidence average calculation processing for calculating the average value μ _e of the model confidence for each of a plurality of training data for each epoch e equal to or less than the epoch number E;
a deviation calculation process for calculating, for each of the plurality of training data, the deviation between the confidence of the model at each epoch e and the average value μ _e calculated by the confidence average calculation process; A data selection support program that causes a computer to function as a data selection support device.

Images