JP2020021301A - Training data evaluation device, training data evaluation method, and program - Google Patents

Abstract

To provide a training data evaluation device for evaluating contributions of training data against requirements for identifying a subject to be inferred.SOLUTION: A training data evaluation device 1 comprises: a batch data generation section 10 for splitting training data into a plurality of batch data; a training processing section 11 for training a model by sequentially using the plurality of batch data; a training progress storage section 22 in which, through being trained by the training processing section 11, models in steps obtained through changes of the model by sequentially applying the batch data thereto, and information of the batch data generating each model, are stored; a follow-up processing section 12 which, by applying test data to the plurality of models stored in the training progress storage section 22, evaluates each model for selecting batch data on the basis of evaluation results of each model; and an outputting section 13 for outputting a result of the selected batch data.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a training data evaluation device, a training data evaluation method, and a program for training data used in machine learning.

  In recent years, research on machine learning systems has been actively conducted. Machine learning systems have become extremely sophisticated, and applications to security and self-driving cars, for example, are being studied.

  Patent Literature 1 discloses a machine learning management invention that selects an appropriate machine learning algorithm for target data in order to improve the prediction performance of a learning model. In the present invention, a model search is repeatedly performed on the same data by repeatedly creating and evaluating a model while changing a machine learning algorithm. When this model search is repeated, data generated in the process of the model search performed in the past and stored in the cache is reused.

  In a machine learning system, a requirement for specifying an object of inference (for example, detection of a vehicle running on a road in the case of automatic driving) is determined, and a model for performing the inference using a large amount of training data in advance. Is learned (Non-Patent Document 1).

JP 2017-228086 A

Laura L. Pullum Brian J. Taylor Majorie A. Darrah `` Guidance for the Verification and Validation of Neural Networks ''

  In the machine learning system, the training data set associated with the requirements was all training data used for machine learning. That is, the model is generally evaluated from the viewpoint of whether or not the result of training using all the training data meets the requirements.

  By the way, when a new model is created, training data used in past model development may be reused. In this case, when inheriting the past requirement items even in the new model, not only reuse the training data of the past model development but also know which training data was effective for the past model generation. Model development can be performed efficiently.

  In view of the above background, an object of the present invention is to provide a training data evaluation device that evaluates the contribution of training data to a requirement for specifying an inference target.

  The present invention employs the following technical means to solve the above problems. The reference numerals in the claims and the parentheses described in this section are examples showing the correspondence with specific means described in the embodiment described below as one aspect, and limit the technical scope of the present invention. It does not do.

  A training data evaluation device (1) according to the present invention includes a batch data generation unit (10) for dividing training data into a plurality of batch data, and a training processing unit (11) for training a model by sequentially using the plurality of batch data. A training progress storage unit (22) storing a model of a process of being changed by the training in which the batch data is sequentially applied, and information of the batch data for generating each model; and a training progress storage unit (22). A tracking processing unit (12) for applying the test data to the plurality of models stored in the storage unit, evaluating each of the models, and selecting batch data based on the evaluation result of each model; An output unit (13) for outputting.

  By dividing the training data into a plurality of batch data and testing the trained model using each batch data, it is possible to evaluate which batch data has led to the generation of a model satisfying the requirements.

  A training data evaluation device (5) according to another aspect of the present invention is a training data evaluation device (5) for evaluating selected data selected from training data. A verification unit configured to generate a model based on the selected model and to apply the test data to the generated model to evaluate the model, thereby verifying the selected data; Here, the selection data is selected as a model capable of generating a model capable of performing inference that meets requirements with predetermined accuracy, and the selection method is not limited.

  Thus, the selection data can be verified by evaluating the model generated using the training data excluding the selection data. That is, when the evaluation of the model generated by the training data excluding the selection data is low, it is confirmed that the evaluation of the selection data is high. Conversely, when the evaluation of the model generated by the training data excluding the selection data is high, it is considered that the evaluation of the entire training data is high, and it can be seen that only the selection data is not particularly high.

  In the training data evaluation method of the present invention, the step of dividing the training data into a plurality of batch data (S10), the step of training a model by sequentially using the plurality of batch data (S12), and the batch data are sequentially applied. A step (S13) of storing in the training progress storage unit (22) a model in the process of being changed by the training and information of the batch data for generating each model; and storing the training progress storage unit (22) in the training progress storage unit (22). Applying the test data to the plurality of models, evaluating each of the models, selecting batch data based on the evaluation results of each model (S17), and outputting the batch data selection results (S18). ).

  The program of the present invention is a program for evaluating training data, a computer, a step of dividing the training data into a plurality of batch data, a step of training the model using a plurality of batch data sequentially, and A step of storing in the training progress storage unit the model of the process of changing by the training in which the batch data is sequentially applied, and the information of the batch data that generated each model; and a plurality of steps stored in the training progress storage unit. A step of applying the test data to the model to evaluate each of the models, selecting batch data based on an evaluation result of each model, and outputting the batch data selection result is executed.

  According to the present invention, it is possible to evaluate which training data of the training data leads to the evaluation of the model.

It is a figure showing composition of a training data evaluation device of a 1st embodiment. (A) It is a schematic diagram which shows training data. (B) It is a figure showing an example which divided training data into batch data. FIG. 4 is a diagram illustrating an example of data stored in a tracking result storage unit. It is a flowchart which shows operation | movement of the training data evaluation apparatus of 1st Embodiment. It is a figure showing the composition of the training data evaluation device of a 2nd embodiment. It is a flowchart which shows operation | movement of the training data evaluation apparatus of 2nd Embodiment. It is a figure showing the composition of the training data evaluation device of a 3rd embodiment. It is a figure for explaining selection of training data by a training data selection part. It is a flowchart which shows operation | movement of the training data evaluation apparatus of 3rd Embodiment. It is a figure showing the composition of the training data evaluation device of a 4th embodiment. It is a flowchart which shows operation | movement of the training data evaluation apparatus of 4th Embodiment. It is a figure showing the composition of the training data evaluation device concerning other examples.

  Hereinafter, a training data evaluation device according to an embodiment of the present invention will be described with reference to the drawings. In the embodiment described below, a neural network model will be described as an example, but the present invention can also be used for evaluating training data for training another model.

(First Embodiment)
FIG. 1 is a diagram illustrating a configuration of a training data evaluation device 1 according to the first embodiment. The training data evaluation device 1 has a training data storage unit 20 that stores training data to be evaluated. The requirement of the model generated by the training data is, for example, `` recognizing the car on the own lane '', and the training data for this is a large amount of images with a bounding box indicating the car on the image taken from the windshield. It is an image.

  The training data evaluation device 1 of the present embodiment evaluates a large amount of training data and selects training data suitable for generating a model that satisfies the requirement of “recognizing a car on the own lane”. It is. As a model, an N-layer convolutional neural network model is used.

  The training data evaluation device 1 has a batch data generation unit 10, a training processing unit 11, a tracking processing unit 12, and an output unit 13. The batch data generation unit 10 has a function of dividing a large amount of training data stored in the training data storage unit 20 into batch data.

  FIG. 2A is a schematic diagram illustrating a large amount of training data stored in the training data storage unit 20. Each square shown in FIG. 2A simulates an image obtained by adding a bounding box indicating an automobile to an image taken from a windshield. The batch data generator 10 divides the training data into batch data, as shown in FIG. In FIG. 2 (b), nine data are taken as one batch, but this is an example, and any number of training data may be included in one batch. Note that the batch data may be composed of one training data.

  The batch data generation unit 10 stores the generated batch data in the batch data storage unit 21. Note that the batch data storage unit 21 may store the training data itself included in the batch, or may store the ID of the training data included in the batch without storing the training data itself. In the case of the latter configuration, when training is actually performed, the training data is read from the training data storage unit 20.

  The training processing unit 11 performs training of the model using the training data, and performs a process of generating the model. The training processing unit 11 performs training of the model by using the batch data sequentially. The training processing unit 11 generates a model M100 by training the model using the training data of the batch 100 shown in FIG. 2B, for example, and then uses the training data of the batch 101 for the model M100. Training is performed to generate a model M101. Thus, the model is updated by sequentially applying the batch data. The training processing unit 11 stores in the training progress storage unit 22 the data of the model to be updated and the data specifying the batch data used to generate the model.

  The tracking processing unit 12 performs a tracking evaluation by applying test data to the model stored in the training progress storage unit 22. The test data is different from the training data, and is stored in the test data storage unit 23. When the training progress of the model “model M100 → model M101 →...” Is stored in the training progress storage unit 22, the tracking processing unit 12 performs a test on each of the models M100, M101,. Evaluate the model by applying the data. The tracking processing unit 12 stores an evaluation result evaluated by applying test data to each model in the tracking result storage unit 24.

  FIG. 3 is a diagram illustrating an example of data stored in the tracking result storage unit 24. The horizontal axis indicates the training progress, and the vertical axis indicates the degree to which the requirements are satisfied. “Batch 101” plotted on the far left indicates an evaluation result obtained by applying test data to the model M100 trained by the batch 101. “Batch 102” plotted second from the left is a diagram showing an evaluation result of the model M 102 obtained by further training in the batch 102 with respect to the model M 100 trained in the batch 101. In this manner, the plot shown in FIG. 3 uses more training data as it goes to the right, so that the evaluation result becomes stable.

  The tracking processing unit 12 selects, based on the data stored in the tracking result storage unit 24, batch data that has contributed to generation of a model suitable for the requirement. The tracking processing unit 12 stores the selected batch data in the selected data storage unit 25. There are various methods for selecting batch data based on the tracking result.

  For example, batch data corresponding to a model having the best evaluation result can be selected. Alternatively, batch data corresponding to a predetermined number of models may be selected from the one with the better evaluation result. Alternatively, batch data corresponding to a model with the highest degree of certainty (e.g., an output value of SOFTMAX) of the judgment output from the neural network or a predetermined number of models from the higher model may be selected.

  Further, the batch data may be selected by focusing on a change in the evaluation result. For example, batch data in which the evaluation result of the model is finally changed to a better direction may be selected, or batch data in which the evaluation result of the model has been changed to a better direction than a predetermined threshold may be selected. .

  The output unit 13 outputs data indicating the batch data selected by the tracking processing unit 12. At this time, the data (see FIG. 3) stored in the tracking result storage unit 24 may be output together. Thereby, the reason for selecting the batch data can be understood.

  FIG. 4 is a diagram illustrating an operation of the training data evaluation device 1 according to the first embodiment. The training data evaluation device 1 divides a large amount of training data stored in the training data storage unit 20 and generates batch data (S10). Subsequently, the training data evaluation device 1 determines the training order in which order the training is performed using the batch data (S11).

  Subsequently, the training data evaluation device 1 performs training using the batch data (S12), and stores the model generated by the training and the data specifying the batch data used for the training in the training progress storage unit 22 ( S13). The training data evaluation device 1 determines whether the processing of all batch data has been completed (S14). If the processing has not been completed for all the batch data (NO in S14), the model is further trained using the next batch data (S12).

  When the training has been completed for all the batch data (YES in S14), the training data evaluation device 1 reads the training process model from the training progress storage unit 22 (S15), and tests the training progress model using the test data. Is performed, and the model of the training process is evaluated (S16). Subsequently, the training data evaluation device 1 selects batch data that has generated a model suitable for the requirements based on the evaluation result of the training progress model (S17), and outputs the selection result (S18).

  The configuration of the training data evaluation device 1 according to the present embodiment has been described above. Examples of the hardware of the training data evaluation device 1 include a CPU, a RAM, a ROM, a hard disk, a display, a keyboard, a mouse, a communication interface, and the like. It is a computer provided with. The above-described training data evaluation device 1 is realized by storing a program having modules for realizing the above-described functions in a RAM or a ROM and executing the program by the CPU. Such a program is also included in the scope of the present invention.

  The training data evaluation device 1 according to the first embodiment divides training data into a plurality of batch data, tests a model that has been trained using each batch data, and determines which batch data satisfies the requirements. Can be evaluated whether or not it has been generated.

(Second embodiment)
FIG. 5 is a diagram illustrating a configuration of the training data evaluation device 2 according to the second embodiment. The basic configuration of the training data evaluation device 2 of the second embodiment is the same as that of the training data evaluation device 1 of the first embodiment, but the training data evaluation device 2 of the second embodiment has And a training progress storage unit 22 for storing a training process model. The training data evaluation device 2 of the second embodiment is different in that the model evaluation is performed while the model is trained.

  FIG. 6 is a diagram illustrating an operation of the training data evaluation device 2 according to the second embodiment. The training data evaluation device 2 divides a large amount of training data stored in the training data storage unit 20 and generates batch data (S20). Subsequently, the training data evaluation device 2 determines a training order in which order the training is performed using the batch data (S21).

  Subsequently, the training data evaluation device 2 performs training using the batch data (S22), tests the model generated by the training, and evaluates the model of the training process (S23). The training data evaluation device 2 stores, in the tracking result storage unit 24, data that specifies the batch data obtained for the evaluation of the training progress model and the generation of the model (S24). The training data evaluation device 2 determines whether to end the training based on whether all the batch data has been used (S25). If the processing of all batch data has not been completed (NO in S25), the model is trained and evaluated using the next batch data (S22 to S24). When the processing of all the batch data is completed (YES in S25), the training data evaluation device 2 selects the batch data that has generated the model suitable for the requirement based on the evaluation result of the model of the training progress (S26). , And outputs the selection result (S27).

  Like the training data evaluation device 1 of the first embodiment, the training data evaluation device 2 of the second embodiment can evaluate which batch data has led to the generation of a model satisfying the requirements. In addition, since the model obtained during the training process is tested while training the model using the batch data, it is not necessary to keep the model obtained during the training process.

(Third embodiment)
FIG. 7 is a diagram illustrating a configuration of the training data evaluation device 3 according to the third embodiment. The training data evaluation device 3 of the third embodiment evaluates batch data in the same manner as the training data evaluation device 1 of the above-described first embodiment, but repeatedly performs batch data evaluation by rearranging batches. . Then, training data that is commonly included in batches selected in different trials is selected. That is, in the first embodiment, training data with high evaluation is selected in batches, whereas in the third embodiment, training data with high evaluation is selected in units of training data. .

  The training data evaluation device 3 according to the third embodiment has a repetition processing unit 30. The repetition processing unit 30 includes a batch data generation unit 10, a training processing unit 11, and a tracking processing unit 12. The batch data generation unit 10 generates batch data from the training data, but generates different batch data for each repetition. The training processing unit 11 and the tracking processing unit 12 generate a model and generate a model of the model with respect to the batch data generated by the batch data generation unit 10 in the same manner as the training data evaluation device 1 according to the first embodiment. Perform an evaluation and select batch data based on the evaluation results. The tracking processing unit 12 stores the selected batch data in the selected batch data storage unit 25. Subsequently, the training data selection unit 14 selects training data commonly included in the selected batch data.

  FIG. 8 is a diagram for describing selection of training data by the training data selection unit 14. FIG. 8 shows an example of batch data that obtained a good result in the M-th trial of the repetitive processing and batch data that obtained a good result in the N-th trial. The training data selection unit 14 selects training data commonly included in batch data obtained in different trials. In the example shown in FIG. 8, the data A and the data B, which are shaded, are included in both the batch data, so the training data selecting unit 14 selects the data A and the data B.

  FIG. 8 shows an example of selecting data that is commonly included in the results of the two selections. However, the training data selection unit 14 determines the data that is commonly included in the results of the K times (for example, three times). May be selected, or data included in all the results may be selected.

  FIG. 9 is a flowchart illustrating the operation of the training data evaluation device 3 according to the third embodiment. The training data evaluation device 3 divides a large amount of training data stored in the training data storage unit 20 and generates batch data (S30). Subsequently, the training data evaluation device 3 determines the training order in which order the training is performed using the batch data (S31). Subsequently, the training data evaluation device 3 performs training using the batch data (S32), and stores the model generated by the training and the data specifying the batch data used for the training in the training progress storage unit 22 ( S33). The training data evaluation device 3 determines whether the processing of all batch data has been completed (S34). If the processing of all batch data has not been completed (NO in S34), the model is further trained using the next batch data (S32).

  When the processing of all the batch data is completed (YES in S34), the training data evaluation device 3 reads the model of the training process from the training progress storage unit 22 (S35), and tests the model of the training process using the test data. Is performed, and the model of the training process is evaluated (S36). Subsequently, the training data evaluation device 3 selects the batch data that has contributed to the generation of the model suitable for the requirement, based on the evaluation result of the training progress model (S37).

  Next, the training data evaluation device 3 determines whether to end the training (S38). If it is determined that the training is not to be ended (NO in S38), the training data evaluation device 3 regenerates the batch data (S30), and repeats the above processing using the new batch data (S31 to S37). When it is determined that the training is to be ended (YES in S38), the training data evaluation device 3 extracts training data commonly included in the selected batch data (S39), and outputs an extraction result (S40).

  The training data evaluation device 3 of the third embodiment reassembles the batch data and evaluates the batch data, and selects the training data that is commonly included in the selected batch data. Training data that contributes to the generation of a model that meets the requirements can be selected.

(Fourth embodiment)
FIG. 10 is a diagram illustrating a configuration of the training data evaluation device 4 according to the fourth embodiment. The basic configuration of the training data evaluation device 4 according to the fourth embodiment is the same as that of the training data evaluation device 1 according to the first embodiment, but the training data evaluation device 4 according to the fourth embodiment includes: It further includes a verification unit 15 for verifying the selected batch data. The verification unit 15 generates a model using the training data excluding the selected batch data, and performs an evaluation by applying test data to the generated model.

  FIG. 11 is a flowchart illustrating a verification operation in the training data evaluation device 4 according to the fourth embodiment. The training data evaluation device 4 reads the training data from the training data storage unit 20 (S50), and excludes the selected batch data from the read training data (S51). Next, the training data evaluation device 4 generates a model trained by the training data excluding the batch data (S52), and evaluates the model by applying test data to the generated model (S53). The training data evaluation device 4 outputs the evaluation result (S54).

  As described above, the batch data selected can be verified by testing and evaluating the model generated using the training data excluding the selected data. That is, when the evaluation of the model generated by the training data excluding the selected batch data is low, it is confirmed that the evaluation of the selected batch data is high. Conversely, when the model generated by the training data excluding the selected batch data has a high evaluation, the evaluation of the entire training data is considered to be high, and only the selected batch data has a particularly high evaluation. I understand that there is no.

  As described above, the training data evaluation device of the present invention has been described in detail with reference to the embodiments. However, the present invention is not limited to the above embodiments. In the above-described embodiment, several methods for selecting batch data based on the evaluation result of the model have been described. However, batch data may be selected using a plurality of these methods, and the sum thereof may be calculated.

  In the process of generating a model, the model greatly changes at the beginning of training, and its evaluation tends to fluctuate greatly. Therefore, the model generated at the beginning may not be evaluated. The model generated at the beginning may define “initial” as a percentage of the entire batch data, for example, and may define “initial” until one-fifth of the batch data is used. The “initial” may be defined by the absolute number of training data to be applied. For example, the “initial” may be used until batch data of 1000 sheets is used.

  In the above-described fourth embodiment, an example has been described in which the apparatus for verifying the batch data selected by the training data evaluation apparatus 1 of the first embodiment is used as an example. The data evaluation device 1 can verify the training data other than the training data selected.

  FIG. 12 is a diagram illustrating another example of the training data evaluation device. The training data evaluation device 5 illustrated in FIG. 12 includes a data evaluation unit 31, a verification unit 15, and an output unit 13. The data evaluation unit 31 has a function of selecting data that meets the requirements of the model from a large amount of training data stored in the training data storage unit 20. The method by which the data evaluation unit 31 selects data is not limited, and any method may be adopted.

  The verification unit 15 verifies whether the selected data meets the requirements of the model. The verification unit 15 removes the selection data from the training data, and generates a model using the training data from which the selection data has been removed. The verification unit 15 applies the test data to the generated model to evaluate the model generated by (training data-selection data). Based on whether the evaluation of the model is high or low, the selected data can be verified.

  The present invention is useful as an apparatus for evaluating training data used in machine learning.

1-5 training data evaluation device, 10 batch data generation unit, 11 training processing unit,
12 tracking processing unit, 13 output unit, 14 training data selection unit, 15 verification unit,
20 training data storage unit, 21 batch data storage unit, 22 training progress storage unit,
23 test data storage unit, 24 tracking result storage unit, 25 selected batch data storage unit,
26 selection training data storage unit, 30 repetition processing unit, 31 data evaluation unit

Claims (12)

A batch data generator (10) for dividing the training data into a plurality of batch data;
A training processing unit (11) for training a model by sequentially using a plurality of batch data;
A training progress storage unit (22) storing a model of a process of changing by the training to which the batch data is sequentially applied, and information of the batch data for generating each model;
A tracking processing unit (12) that applies test data to a plurality of models stored in the training progress storage unit (22), evaluates each model, and selects batch data based on an evaluation result of each model; ,
An output unit (13) for outputting the selection result of the batch data;
A training data evaluation device (1) comprising:   The training data evaluation device (1) according to claim 1, wherein the tracking processing unit (12) selects batch data corresponding to a model having the best accuracy rate for test data. The model is a model of a neural network,
The training data evaluation device (1) according to claim 1, wherein the tracking processing unit (12) selects the batch data based on a certainty factor of a determination output from a neural network model.   The training data evaluation device (1) according to claim 1, wherein the tracking processing unit (12) selects batch data that has finally changed the evaluation result of the model in a better direction.   The training data evaluation device (1) according to claim 1, wherein the tracking processing unit (12) selects batch data in which an evaluation result of the model is changed in a better direction than a predetermined threshold.   The training data evaluation device (1) according to claim 1, wherein the tracking processing unit (12) selects batch data by a plurality of different methods, and calculates a sum of the selected batch data. The batch data generating unit (10) generates different batch data, repeats the processing by the training processing unit (11) and the tracking processing unit (12), and selects a batch processing unit ( 30)
A training data selection unit (14) for selecting training data commonly included in the batch data selected by the repetition processing unit (30);
The training data evaluation device (3) according to claim 1, comprising:   The training data evaluation according to claim 1, wherein the tracking processing unit (12) does not evaluate an initially generated model among a plurality of models stored in the training progress storage unit (22). Apparatus (1).   A model is generated using the training data excluding the batch data selected by the tracking processing unit (12), and the model is evaluated by applying test data to the model. The training data evaluation device (4) according to claim 1, further comprising a verification unit (15) for verifying the selected batch data. A training data evaluation device (5) for evaluating selected data selected from training data,
A training unit (13) for generating a model based on the training data excluding the selection data and applying a test data to the generated model to evaluate the model, thereby verifying the selection data; Data evaluation device (5). Dividing the training data into a plurality of batch data (S10);
Training the model by sequentially using a plurality of batch data (S12);
A step (S13) of storing in the training progress storage unit (22) a model in a process of changing by the training in which the batch data is sequentially applied, and information of the batch data that generated each model;
Applying test data to a plurality of models stored in the training progress storage unit (22) to evaluate each of the models, and selecting batch data based on an evaluation result of each model (S17);
Outputting the selection result of the batch data (S18);
Training data evaluation method comprising: A program for evaluating training data, comprising:
Dividing the training data into a plurality of batch data;
Training the model using a plurality of batch data sequentially;
Storing the model of the process of changing by the training in which the batch data is sequentially applied, and information of the batch data that generated each model in a training progress storage unit;
Applying test data to a plurality of models stored in the training progress storage unit, evaluating each of the models, and selecting batch data based on an evaluation result of each model;
Outputting the selection result of the batch data;
A program that executes

Images