JP2023104464A - Machine learning data set evaluation support device and machine learning data set evaluation support method - Google Patents

Abstract

To configure a sophisticated learning data set that avoids inappropriate learning or lack of learning.SOLUTION: A machine learning data set evaluation support device 100 is configured to include: a learning data set holding unit 116 that holds a learning data set and label information used for supervised machine learning; an inference execution unit 111 that inputs a predetermined evaluation data set to a plurality of learned AI models and obtains a plurality of inference results; a difference analysis unit 113 that analyzes the difference between the inference results from a confidence degree of the inference result and a stability degree of the inference result in each of the plurality of learned AI models; a data evaluation unit 114 that evaluates the evaluation data set based on the analysis results of the difference; and a learning data set editing unit 115 that edits the learning data set and the label information based on the evaluation result of the evaluation data set.SELECTED DRAWING: Figure 1

Description

The present invention relates to a machine learning dataset evaluation support device and a machine learning dataset evaluation support method.

Machine learning technology has been introduced in a wide range of fields, such as monitoring and control of various devices, failure prediction, and object recognition during automatic driving using various big data such as IoT data. In machine learning technology in such fields, supervised machine learning, which has high learning accuracy and fast learning speed because humans give correct data, is useful.

In supervised machine learning, data is collected from the real world, and the output value expected when the data is input to the AI model, that is, a learning data set (training data and test data) with correct labels is created. . Of these, training data is used as teacher data to make an AI model learn, and test data is used to evaluate the accuracy of the trained AI model.

Conventional technologies related to machine learning include, for example, learning data generation devices ( See Patent Document 1) and the like have been proposed.

This technology includes processing means for performing cleansing processing on input data to generate learning data, and a machine learning algorithm, and generating means for generating a learning model based on the learning data using the machine learning algorithm. and, wherein the generating means includes a plurality of machine learning algorithms of different types as the machine learning algorithms, and the processing means processes input data with a plurality of the machine learning algorithms The learning data generation device is characterized by performing a first cleansing process corresponding to each learning algorithm.

JP 2020-38514 A

The desired requirements for the above training data set are as follows. The learning data set covers the assumed input data, a robust AI model (stable against small deviations in input values) can be constructed, and the labels assigned to the learning data are appropriate. , is.

However, it is difficult to control the progress of learning with a learning data set simply collected without special ingenuity. Therefore, unintended learning may occur. For example, there is a shortage of learning data, a case in which learning data with different correct labels are inadvertently approached, and a case in which a feature different from the learning intent is dominant.

On the other hand, according to the conventional technology, cleansing of learning data is performed based on the features of the machine learning algorithm. Therefore, it is not possible to refine the learning data by feeding back the inference characteristics of the generated learning model.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide a technique that enables the construction of a sophisticated learning data set that avoids inappropriate learning and insufficient learning.

A machine learning data set evaluation support device of the present invention for solving the above problems includes a learning data set holding unit that holds a learning data set and label information used for supervised machine learning, and a predetermined evaluation data set for a plurality of trained AIs. An inference execution unit that obtains a plurality of inference results by inputting data into the model, and analyzes the difference between the plurality of inference results from the certainty of the inference result and the stability of the inference result in each of the plurality of trained AI models. a difference analysis unit; a data evaluation unit that evaluates the evaluation data set based on the analysis result of the difference; and a learning data set held in the learning data set holding unit based on the evaluation result of the evaluation data set. and a learning data set editing unit that edits the label information.

Further, in the machine learning data set evaluation support method of the present invention, the information processing device holds learning data sets and label information used for supervised machine learning in a storage device, and a predetermined evaluation data set is stored in a plurality of trained AI models. , a process of obtaining a plurality of inference results, and a process of analyzing the difference between the plurality of inference results from the confidence of the inference result and the stability of the inference result in each of the plurality of trained AI models; executing a process of evaluating the evaluation data set based on the difference analysis result, and a process of editing the learning data set and label information held in the storage device based on the evaluation result of the evaluation data set. characterized by

According to the present invention, it is possible to construct a sophisticated training data set that avoids improper or under-learning.

It is a block diagram showing an example of functional composition of a learning data set evaluation support device of this embodiment. It is a figure which shows the hardware structural example of the learning data set evaluation assistance apparatus of this embodiment. It is a figure which shows the flow example of the learning data set evaluation assistance method of this embodiment. FIG. 4 is a flowchart of processing for obtaining an AI model trained by changing the learning data set in the embodiment. FIG. 10 is a flow diagram of processing for acquiring an AI model trained by changing the model structure in this embodiment. It is a flow diagram of inference execution processing in the present embodiment, and is a flow diagram of processing for sequentially executing inference of a trained AI model. FIG. 4 is a flow diagram of inference execution processing in the present embodiment, and is a flow diagram of processing for executing inferences of trained AI models in parallel. FIG. 4 is a flow diagram of inference execution processing in the present embodiment, and is a flow diagram of processing using an AI model whose model structure is changed at the time of inference. FIG. 10 is a flow chart of inference result difference analysis processing in this embodiment. It is a figure which shows an example of the calculation method of the certainty degree of an inference result in this embodiment, and a stability degree. It is a figure which shows the example of an analysis result of the difference analysis part in this embodiment. It is a flow chart of data evaluation processing in this embodiment. It is a figure which shows an example of the threshold value setting process which the data evaluation part in this embodiment performs. FIG. 5 is a flow diagram of learning data set editing processing in the present embodiment. It is a figure which shows an example of the determination method of similar data which the learning data set edit process part in this embodiment performs. It is a figure which shows an example of the generation method of similar data in this embodiment. 2 is a block diagram showing the functional configuration of the learning data set evaluation support device in Example 1. FIG. 4A and 4B are diagrams showing an example of evaluation data, an inference result, and a difference analysis result in Example 1. FIG. 8A and 8B are diagrams illustrating an example of a high confidence threshold, an example of a low confidence threshold, and an example of a low stability threshold in Embodiment 1; FIG. 5 is a diagram showing an example of evaluation results of evaluation data in Example 1. FIG. FIG. 10 is an example of learning data set editing processing in Example 1, and shows a case where the label of learning data and the inference result match. FIG. 10 is an example of learning data set editing processing in Example 1, and shows a case where the label of learning data and the inference result do not match. FIG. 10 is a diagram illustrating an example of learning data set editing processing in Example 1; FIG. 10 is a diagram illustrating an example of learning data set editing processing in Example 1; FIG. 11 is a block diagram showing the functional configuration of a learning data set evaluation support device in Example 2; FIG. 11 is a diagram showing a display example of the certainty factor of an inference result in Example 2; FIG. 11 is a diagram showing a display example of the stability of an inference result in Example 2; FIG. 10 is a diagram showing an example of a high confidence threshold, an example of a low confidence threshold, and an example of a low stability threshold in Example 2; FIG. 10 is a diagram showing an example of evaluation results of evaluation data in Example 2;

<Configuration example of machine learning dataset evaluation support device>
Embodiments of the present invention will be described in detail below with reference to the drawings. FIG. 1 is a diagram showing a functional configuration example of a machine learning dataset evaluation support device 100 of this embodiment. A machine learning data set evaluation support device 100 shown in FIG. 1 is a computer that enables the construction of sophisticated learning data sets that avoid inappropriate learning and insufficient learning.

As shown in FIG. 1, the machine learning dataset evaluation support device 100 of the present embodiment includes an evaluation dataset storage unit 110, an inference execution unit 111, a trained AI model storage unit 112, a difference analysis unit 113, and a data evaluation unit 114. , a learning data set editing unit 115 and a learning data set holding unit 116 .

Of these, the evaluation data set holding unit 110 holds the evaluation data set 125 . Also, the trained AI model holding unit 112 holds a trained AI model 118 . Also, the learning data set holding unit 116 holds a learning data set 126 and label information 127 .

Learning data set 126 is, for example, training data collected from the real world. Also, the label information 127 is information of an output expected when a learning data set, which is training data, is input to the AI model, that is, correct label information.

In addition, the evaluation data set 125 is a target to be input to each of the AI models that have been trained in this way, that is, the trained AI models 118 . In that case, a plurality of inference results will be obtained from the trained AI model 118 .

Specifically, the machine learning dataset evaluation support device 100 can be assumed to be a server device, a personal computer, or the like.

<Hardware configuration>
Further, the hardware configuration of the machine learning dataset evaluation support device 100 of the present embodiment is shown in FIG. 2 as follows.

That is, the machine learning dataset evaluation support device 100 includes a storage device 101 , a memory 103 , an arithmetic device 104 , an input device 105 , an output device 106 and a communication device 107 .

Among them, the storage device 101 is composed of an appropriate non-volatile storage element such as an SSD (Solid State Drive) or a hard disk drive.

Also, the memory 103 is composed of a volatile memory element such as a RAM.

Arithmetic device 104 is a CPU that reads program 102 stored in storage device 101 into memory 103 and executes it, performs overall control of the device itself, and performs various determinations, calculations, and control processes.

The input device 105 is a device such as a keyboard, mouse, and microphone that receives key input and voice input from the user.

Also, the output device 106 is a device such as a display or a speaker that outputs the processing result of the arithmetic device 104 .

Also, the communication device 107 is assumed to be a network interface card or the like that connects to an appropriate network and handles communication processing with a user terminal or the like. However, if the machine learning dataset evaluation support device 100 is a stand-alone machine, the communication device 107 can be omitted.

In the storage device 101, in addition to the program 102 for implementing the necessary functions as the machine learning dataset evaluation support device of this embodiment, the evaluation dataset 125 in the evaluation dataset holding unit 110, the learning dataset At least the learning data set 126 and label information 127 in the holding unit 115 and the trained AI model 118 in the trained AI model holding unit 112 are stored. However, the details of these will be described later.

Also, the program 102 implements the functions of the inference execution unit 111 , the difference analysis unit 113 , the data evaluation unit 114 , and the learning data set editing unit 115 by being executed by the arithmetic unit 104 .

Among these, the inference execution unit 111 inputs the evaluation data set held in the evaluation data set holding unit 110 to the plurality of trained AI models 118 and obtains a plurality of inference results.

Note that the inference execution unit 111 may use, as the plurality of trained AI models 118, AI models trained by changing the learning data set or model structure during learning.

Also, the inference execution unit 111 may use, as a plurality of trained AI models, trained AI models whose model structures have been changed at the time of inference.

Also, the inference execution unit 111 may sequentially execute a plurality of trained AI models to obtain a plurality of inference results.

Also, the inference execution unit 111 may execute a plurality of trained AI models in parallel to obtain a plurality of inference results at the same time.

In addition, the difference analysis unit 113 analyzes the difference between the plurality of inference results obtained by the above-described inference execution unit 111 from the certainty of the inference result and the stability of the inference result in each of the plurality of trained AI models 118. It is a thing.

The data evaluation unit 114 evaluates the evaluation data set 125 based on the difference analysis result obtained by the difference analysis unit 113 .

Also, the learning data set editing unit 115 edits the learning data set 126 and the label information 127 held in the learning data set holding unit 116 based on the evaluation result of the evaluation data set 125 .

Note that the above-described learning data set editing unit 115 learns learning data similar to the evaluation data in which the degree of certainty of the inference result by the inference execution unit 111 is greater than a predetermined threshold and which has a different correct label from the inference result of the evaluation target data. It may be deleted from dataset 126 .

Also, the learning data set editing unit 115 may delete from the learning data set 126 learning data similar to the evaluation data in which the degree of certainty of the inference result by the inference execution unit 111 is smaller than a predetermined threshold.

In addition, the learning data set editing unit 115 generates learning data similar to the evaluation data in which the degree of certainty of the inference result by the inference execution unit 111 is greater than a predetermined threshold and whose stability is less than a predetermined threshold. may be added to

Note that the machine learning dataset evaluation support device 100 has an analysis result display unit 117 in addition to the above, although it will be described later based on FIG. 25 instead of FIG. The analysis result display unit 117 displays the analysis result of the difference analysis unit 113 on the output device 106 .

<Flow example>
The actual procedure of the machine learning data set evaluation support method according to the present embodiment will be described below with reference to the drawings. Various operations corresponding to the machine-learning dataset evaluation support method described below are implemented by a program that the machine-learning dataset evaluation support apparatus 100 reads out to a memory or the like and executes. This program is composed of codes for performing various operations described below.

FIG. 3 is a diagram showing a flow example of the machine learning data set evaluation support method according to this embodiment. In this case, the machine learning dataset evaluation support device 100 acquires the learned AI model 118 (301). This learned AI model 118 is held in advance in the learned AI model holding unit 112 of the storage device 101 .

Subsequently, the inference execution unit 111 of the machine learning dataset evaluation support device 100 inputs one piece of evaluation data from the evaluation dataset 125 of the evaluation dataset holding unit 110 to the trained AI model 118 obtained in step 301. , perform inference processing (302). Details of this inference processing will be described later with reference to FIGS.

Next, the difference analysis unit 113 of the machine learning dataset evaluation support device 100 analyzes the confidence and stability of the result of the inference processing in step 302 and records them in the memory 103, for example (303).

Subsequently, the machine learning dataset evaluation support device 100 determines whether the processing target in the steps up to this point was the last evaluation data among the unprocessed evaluation data included in the evaluation dataset 125 (304).

As a result of the above determination, if the evaluation data is not the last one (304: NO), the machine learning dataset evaluation support device 100 returns the process to step 302.

On the other hand, as a result of the above determination, if it is the last evaluation data (304: YES), the data evaluation unit 114 of the machine learning dataset evaluation support device 100 evaluates the evaluation data, and stores the result in the memory 103, for example. (305). The details of the processing of the data evaluation unit 114 will be described later with reference to FIG. 12 and the like.

Also, the learning data set editing unit 115 of the machine learning data set evaluation support device 100 edits the learning data set 126 and the label information 127 in the learning data set holding unit 116 (306), and ends this flow. The details of the processing of this learning data set editing unit 115 will be described later with reference to FIG. 14 and the like. The process so far has produced a refined training dataset that avoids improper or under-training.

<Flow example 1 of the inference execution part>
Next, FIG. 4 shows an example flow of processing in which the inference execution unit 111 of the present embodiment generates a plurality of AI models through learning using a learning data set obtained by changing the learning data set.

In this case, the inference execution unit 111 of the machine learning dataset evaluation support device 100 acquires the pre-learning AI model from the input device 105, for example (401).

Subsequently, the inference execution unit 111 of the machine learning dataset evaluation support device 100 sets 1 to the argument i (402).

Next, the inference executing unit 111 of the machine learning dataset evaluation support device 100 acquires a plurality of learning data from the learning dataset 126 of the learning dataset holding unit 115 (403).

Subsequently, the inference executing unit 111 of the machine learning dataset evaluation support device 100 acquires the label of the learning data obtained in step 403 from the label information 127 of the learning dataset holding unit 115 (404).

Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 trains the pre-learning AI model obtained in step 401 using the learning data and labels obtained in steps 403 and 404 as teacher data (405 ).

In addition, the inference execution unit 111 of the machine learning dataset evaluation support device 100 stores the learned AI model 118 obtained by the training in step 405 in the learned AI model holding unit 112 (406).

Here, the inference execution unit 111 of the machine learning dataset evaluation support device 100 increments the value of the argument i (407), and transitions to step 408. FIG.

Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 determines whether the above argument i exceeds a predetermined value T (408).

As a result of the above determination, if the value of the argument i does not exceed T (408: NO), the inference executing section 111 returns the process to step 403. Here, when returning to step 403, the learning data included in the learning data set 126 that have not been processed are used as the next learning target, and addition, change, and deletion of the learning data are performed according to a predetermined rule or user operation. It can also be assumed that it is executed as a learning target.

On the other hand, if the value of the argument i exceeds T as a result of the above determination (408: YES), the inference execution unit 111 terminates this flow.

<Flow example 2 of the inference execution part>
Next, FIG. 5 shows an example flow of processing in which the inference execution unit 111 of this embodiment obtains an AI model trained by changing the model structure.

In this case, the inference execution unit 111 of the machine learning dataset evaluation support device 100 acquires the pre-learning AI model from the input device 105, for example (501).

Subsequently, the inference execution unit 111 of the machine learning dataset evaluation support device 100 sets 1 to the argument i (502).

Next, the inference executing unit 111 of the machine learning dataset evaluation support device 100 acquires a plurality of learning data from the learning dataset 126 of the learning dataset holding unit 115 (503).

Subsequently, the inference execution unit 111 of the machine learning dataset evaluation support device 100 acquires the label of the learning data obtained in step 503 from the label information 127 of the learning dataset holding unit 115 (504).

Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 modifies the structure of the pre-learning AI model obtained in step 501 (505). Modification of this structure, for example, among the neural network that constitutes the AI model, one of the nodes of a predetermined layer (for example, a certain layer of the intermediate layer) is randomly deleted, or the node and a predetermined layer of another layer One can imagine deleting one edge between a node and so on.

In addition, the inference execution unit 111 of the machine learning dataset evaluation support device 100 gives the learning data and labels obtained in steps 503 and 504 to the pre-learning AI model whose structure has been modified in step 505 as teacher data. Train (506).

In addition, the inference execution unit 111 of the machine learning dataset evaluation support device 100 stores the learned AI model 118 obtained by the training in step 506 in the learned AI model holding unit 112 (507).

Here, the inference execution unit 111 of the machine learning dataset evaluation support device 100 increments the value of the argument i (507), and transitions to step 508. FIG.

Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 uses the above argument i
exceeds a predetermined value T (509).

As a result of the above determination, if the value of the argument i does not exceed T (509: NO), the inference execution unit 111 returns the process to step 505. On the other hand, if the value of the argument i exceeds T as a result of the above determination (509: YES), the inference execution unit 111 terminates this flow.

<Flow example 3 of the inference execution part>
Next, FIG. 6 shows a flow of inference execution processing by the inference execution unit 111 of the present embodiment, and shows an example of a flow of processing for sequentially executing inference of a trained AI model.

In this case, the inference executing unit 111 of the machine learning dataset evaluation support device 100 acquires one piece of evaluation data from the evaluation dataset 125 of the evaluation dataset holding unit 110 (601).

Subsequently, the inference execution unit 111 of the machine learning dataset evaluation support device 100 acquires T trained AI models 118 from the trained AI model holding unit 112 (602).

Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 sets 1 as the value of the argument i (603).

Subsequently, the inference execution unit 111 of the machine learning dataset evaluation support device 100 inputs the evaluation data obtained in step 601 to the k-th trained AI model 118 and executes inference (604).

In addition, the inference execution unit 111 of the machine learning dataset evaluation support device 100 stores the result of the inference in step 605 described above, for example, in the memory 103 (605).

Also, the inference execution unit 111 of the machine learning dataset evaluation support device 100 increments the value of the argument i by one (606).

Subsequently, the inference execution unit 111 of the machine learning dataset evaluation support device 100 determines whether the value of the above argument i exceeds T (607).

As a result of the above determination, if the value of the argument i does not exceed T (607: NO), the inference executing section 111 returns the process to step 604.

On the other hand, if the value of the argument i exceeds T as a result of the above determination (607: YES), the inference execution unit 111 terminates this flow.

<Flow example 4 of the inference execution part>
Next, FIG. 7 shows a flow of inference execution processing in the present embodiment, which is an example of a flow of processing for executing inference of trained AI models in parallel.

In this case, the machine learning dataset evaluation support device 100 acquires one piece of evaluation data from the evaluation dataset holding unit 110 (701).

Subsequently, the machine learning dataset evaluation support device 100 acquires T learned AI models 118 from the trained AI model holding unit 112 (702).

Next, the machine learning dataset evaluation support device 100 deploys the T trained AI models 118 obtained in step 702 to the inference execution unit 111 (703).

Subsequently, the inference execution unit 111 of the machine learning dataset evaluation support device 100 inputs the evaluation data obtained in step 701 to the T trained AI models 118 deployed in step 703 to execute inference ( 704).

Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 stores the result of the inference in step 704, for example, in the memory 103 (705), and ends this flow.

<Flow example 5 of the inference execution part>
Next, FIG. 8 shows a flow of inference execution processing in this embodiment, which is an example of a flow of processing using an AI model whose model structure is changed at the time of inference.

In this case, the inference executing unit 111 of the machine learning dataset evaluation support device 100 acquires one piece of evaluation data from the evaluation dataset holding unit 110 (801).

In addition, the inference execution unit 111 of the machine learning dataset evaluation support device 100 acquires the trained AI model 118 from the trained AI model holding unit 112 (802).

Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 sets 1 as the value of the argument i (803).

Subsequently, the inference execution unit 111 of the machine learning dataset evaluation support device 100 partially inactivates the components of the trained AI model 118 obtained in step 802 (804). In this case, the constituent elements are, for example, nodes and edges that constitute the neural network, and the partial inactivation can be assumed to be a measure to set the weight value of the node to zero or to delete the edge.

Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 inputs the evaluation data obtained in step 801 to the trained AI model 118 partially deactivated in step 804. Inference is performed (805).

In addition, the inference execution unit 111 of the machine learning dataset evaluation support device 100 stores the result of inference in step 805 in the memory 103, for example (806).

Subsequently, the inference execution unit 111 of the machine learning dataset evaluation support device 100 increments the value of the argument i (807), and determines whether the value of the argument i exceeds a predetermined value of T (808). .

As a result of the above determination, if the value of the argument i does not exceed T (808: NO), the inference executing section 111 returns the process to step 804.

On the other hand, if the value of the argument i exceeds T as a result of the above determination (808: YES), the inference execution unit 111 terminates this flow.

<Flow example of difference analysis part>
Next, FIG. 9 shows an example flow of inference result difference analysis processing in the difference analysis unit 113 of this embodiment. Also, FIG. 10 shows an example of a calculation method of the certainty and stability of the inference result in this embodiment.

In this case, the difference analysis unit 113 of the machine learning dataset evaluation support device 100 acquires K inference results by the inference execution unit 111 from the memory 103 (901).

Further, the difference analysis unit 113 of the machine learning dataset evaluation support device 100 calculates the certainty factor of the inference result obtained in step 901 and stores it in the memory 103, for example (902).

Next, the machine learning dataset evaluation support device 100 calculates the stability of the inference result obtained in step 901, stores it in the memory 103, for example (903), and ends this flow.

As a method of calculating the confidence and stability by the above-described difference analysis unit 113, for example, as shown in FIG. , the probability can be calculated by the formula 1203, and the stability can be calculated by the formula 1204.

Similarly, when the inference result of the regression problem is expressed by expression 1205 and the mean value of the probability of the inference result of T trained AI models 118 is expressed by expression 1206, the confidence is expressed by expression 1207, and the stability is expressed by expression 1208. can be calculated by

It should be noted that the result of the certainty factor and the stability factor calculated as described above can be output as a configuration as shown in FIG. 11, for example. In the confidence factor distribution diagram 1401, each point represents the average value of the probabilities of being determined to be class k and the confidence factor of the inference result for a plurality of inference results for one evaluation data. In the stability distribution diagram 1402, each point represents the average value of probabilities of being determined to be class k and the stability of the inference results for a plurality of inference results for one piece of evaluation data.

<Flow example of the data evaluation section>
Next, FIG. 12 shows a flow example of the data evaluation unit 114 in this embodiment. In this case, the data evaluation unit 114 of the machine learning dataset evaluation support device 100 sets a high confidence threshold, a low confidence threshold, and a low stability threshold (1001).

As a setting method in this case, for example, a method of using a preset default value, a method of mechanically setting so that the evaluation data to be classified is a constant ratio, and a method of separating areas with high data density from areas with low data density. One is assumed to be mechanically set so as to make the difference, or the other is set by the operator referring to the difference analysis result (see FIG. 11).

As a specific conceptual example of threshold setting, as shown in the distribution diagram 1401 of FIG. , a high confidence threshold is set so as to cover the data around '1' and '2'.

In addition, as shown in the distribution diagram 1402, in the case of data with low confidence of the inference result, the inference result is not stable and it is difficult to recognize with certainty, the threshold of high confidence is greater than the threshold of low confidence. A low-confidence threshold is set from the distribution of low-confidence data.

In addition, as shown in the distribution diagram 1403, in a situation where the inference results of data with low stability of inference results may stably converge by learning, the low stability threshold set.

In addition, the data evaluation unit 114 of the machine learning dataset evaluation support device 100 extracts and records evaluation data whose certainty is equal to or higher than the high certainty threshold (set in step 1001) (
1002).

Next, the data evaluation unit 114 of the machine learning dataset evaluation support device 100 extracts and records evaluation data whose confidence is less than the low confidence threshold (set in step 1001) (1003).

Subsequently, the data evaluation unit 114 of the machine learning dataset evaluation support device 100 extracts and records evaluation data whose stability is less than the low stability threshold (set in step 1001) (1004), and this flow exit.

<Flow example of learning data set editor>
Next, FIG. 14 shows an example of the processing flow in the learning data set editing unit 115 in this embodiment. In this case, the learning data set editing unit 115 of the machine learning data set evaluation support device 100 acquires evaluation data from the evaluation data set holding unit 110 (1101).

Subsequently, the learning data set editing unit 115 of the machine learning data set evaluation support device 100 determines whether the evaluation data indicates a low confidence (1102).

As a result of the above determination, if the evaluation data indicates low confidence (1102: YES), the learning data set editing unit 115 obtains in step 1101 the learning data set 126 of the learning data set holding unit 116 All learning data similar to existing evaluation data are acquired (1109).

Also, the learning data set editing unit 115 acquires the label of the learning data obtained in step 1109 from the label information 127 of the learning data set holding unit 116 (1110), and shifts the processing to step 1111.

On the other hand, as a result of the above determination, if the evaluation data does not indicate the low confidence (1102: NO), the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 determines that the evaluation data is high confidence. It is determined whether it indicates the degree (1103).

As a result of the above determination, if the evaluation data does not indicate high confidence (1103: NO), the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 shifts the process to 1116.

On the other hand, as a result of the above determination, if the evaluation data indicates a high degree of certainty (1103: YES), the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 determines that the evaluation data has a low stability (1104).

As a result of the above determination, if the evaluation data does not indicate low stability (1104: NO), the learning data set editing unit 115 of the machine learning data set evaluation support device 100 All learning data similar to the evaluation data are obtained from the data set 126 (1105).

It should be noted that, as a method for determining "similarity" described above, for example, a method using equations 1501 and 1502 shown in FIG. 15 can be assumed. Alternatively, as shown in FIG. 16, a configuration 1503 of preparing an AI model trained using similar data and its degree of similarity as learning data and determining the degree of similarity can also be assumed.

Subsequently, the learning data set editing unit 115 of the machine learning data set evaluation support device 100 acquires the label of the learning data obtained in step 1105 from the label information 127 of the learning data set holding unit 16 (1106).

Also, the learning data set editing unit 115 of the machine learning data set evaluation support device 100 compares the label obtained in step 1106 with the inference result of the evaluation data obtained in step 1101 (1107).

As a result of the above comparison, if the above label matches the inference result (1108: YES), the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 shifts the process to step 1116.

On the other hand, as a result of the above determination, if the label and the inference result do not match (1108: NO), the learning data set editing unit 115 of the machine learning data set evaluation support device 100 stores the learning data set 126 in the learning data set holding unit 116. (1111).

Also, the learning data set editing unit 115 of the machine learning data set evaluation support device 100 deletes the label of the learning data from the label information 127 of the learning data set holding unit 116 (1112), and shifts the process to step 1116. .

In such a case where the confidence and stability of the evaluation data are high, if the labels of learning data similar to the evaluation data and the inference results of the evaluation data do not match, the training data and labels are deleted. It becomes possible to exclude learning data with inappropriate labels (see FIG. 22). Note that the learning data set editing unit 115 may display the learning data to be deleted and label information on the output device 106 .

On the other hand, in the case where the degree of certainty in the evaluation data is low, it is possible to eliminate inadvertent proximity of learning data with different correct labels, and to eliminate learning data in which feature amounts different from the learning intention are predominant (see FIG. 23). Note that the learning data set editing unit 115 may display the learning data to be deleted and label information on the output device 106 .

In addition, in the case where the evaluation data has a high degree of certainty and a low degree of stability, it is possible to supplement missing learning data (see FIG. 24). Note that the learning data set editing unit 115 may display the information of the learning data to be supplemented on the output device 106 .

Here, as a result of the determination in the process 1104 described above, if the evaluation data indicates low stability (1104: YES), the learning data set editing unit 115 of the machine learning data set evaluation support device 100 performs the evaluation Data similar to the data is generated (1113).

As shown in FIG. 16, this similar data is generated by a method 1601 of generating data by slightly changing the feature amount of the original data, data by slightly rotating, deforming, or discoloring the original data. and a method 1603 of generating data using a data generating neural network.

Also, the learning data set editing unit 115 of the machine learning data set evaluation support device 100 adds the learning data generated in step 1113 to the learning data set 126 of the learning data set holding unit 116 (1114).

Subsequently, the learning data set editing unit 115 of the machine learning data set evaluation support device 100 adds the label related to the learning data generated in step 1113 to the label information 127 of the learning data set holding unit 116 (1115), and starts processing. Transition to step 1116 .

In addition, the learning data set editing unit 115 of the machine learning data set evaluation support device 100 determines whether or not any unprocessed evaluation data remains in the evaluation data set 125 (1116). In other words, if the evaluation data to be processed in the next flow is not the last evaluation data (1116: NO), the learning data set editing unit 115 of the machine learning data set evaluation support device 100 advances the process to 1101. return.

On the other hand, as a result of the above determination, if there are no unprocessed data left, that is, if the evaluation data to be processed in the next flow is the last evaluation data (1116: YES), the machine learning dataset evaluation The learning data set editing unit 115 of the support device 100 ends this flow.

<Specific example: Example 1>
Subsequently, the machine learning data set evaluation support technology of the present embodiment will be described along more specific examples. FIG. 17 is a block diagram showing the functional configuration of the machine learning dataset evaluation support device 100 according to the first embodiment.

The basic configuration is the same as that shown in FIG. 1, except that the AI model held in the learned AI model holding unit 112 in the inference execution unit 111 is the sign identification AI model 118A. be. In other words, the trained AI model in this case distinguishes the road sign from the image of the road sign.

Therefore, FIG. 18 shows examples of a total of four evaluation data 1801 from 1 to 4, the inference result 1802, and the difference analysis result 1803 regarding the road sign indicating the numerical value "60". FIG. 19 illustrates threshold values 1901 to 1903 related to such difference analysis results.

FIG. 20 shows evaluation results 2001 for each evaluation data. The evaluation result of the evaluation data 1 is "the degree of certainty is equal to or higher than the threshold for high certainty, the degree of stability is equal to or higher than the threshold for low stability, and the certainty is high and the stability is also high." The evaluation result of the evaluation data 2 is "the certainty is less than the low certainty threshold, and the certainty is low." The evaluation result of the evaluation data 3 is "the stability is less than the low stability threshold, and the stability is low". Further, the evaluation result of the evaluation data 4 is "the confidence factor is equal to or higher than the high confidence threshold, the stability is equal to or higher than the low stability threshold, and the confidence is high and the stability is high".

Based on these evaluation results, as shown in FIG. 21, for evaluation data 1, similar data whose labels match the inference results are left. As shown in FIG. 22, for evaluation data 4, similar data whose labels do not match the inference results are deleted together with their labels. Also, as shown in FIG. 23, for the evaluation data 2, the learning data is deleted together with the label. Further, similar data is added to the evaluation data 3 as shown in FIG.

<Specific example: Example 2>
Further, it is possible to assume a mode in which the machine learning dataset evaluation support device 100 further includes an analysis result display unit 117 as shown in FIG. 25 . In this case, the analysis result display unit 117 causes the output device 106 to display distribution diagrams 2601 and 2602 indicating the certainty of the inference result, as shown in FIG. Similarly, the analysis result display unit 117 causes the output device 106 to display distribution diagrams 2701 and 2702 indicating the stability of the inference results, as shown in FIG.

In addition, as shown in FIG. 28, the analysis result display unit 117 outputs threshold values 2801 to 2803 for high confidence, low confidence, and low stability based on the inference results shown in FIGS. 106 to display.

Also, the analysis result display unit 117 causes the output device 106 to display the evaluation result 2901 of the evaluation data as shown in FIG.

Although the best mode for carrying out the present invention has been specifically described above, the present invention is not limited to this, and can be variously modified without departing from the scope of the invention.

According to this embodiment, by analyzing the inference characteristics of the trained AI model from the confidence and stability, data that causes inappropriate learning is removed and data that lacks learning is added. , can constitute the training dataset. As a result, it is possible to construct a sophisticated training data set that avoids inappropriate learning and insufficient learning.

At least the following will be clarified by the description of this specification. That is, the machine learning dataset evaluation support device of the present embodiment may further execute a process of displaying the analysis result on an output device.

According to this, the difference between a plurality of inference results can be clearly indicated to the user in terms of the certainty of the inference result and the stability of the inference result.

Further, in the machine learning dataset evaluation support device of the present embodiment, the inference execution unit uses an AI model trained by changing the learning dataset or model structure at the time of learning as the plurality of trained AI models. good too.

According to this, it is possible to efficiently obtain an inference result based on fluctuations in the learning data set and the model structure.

Further, in the machine learning dataset evaluation support device of the present embodiment, the inference execution unit may use, as the plurality of trained AI models, trained AI models whose model structure is changed during inference.

According to this, it is possible to reduce the number of models by sequentially changing one model, and to improve the accuracy of the analysis of the inference results by reflecting the history of the inference results of the models in the model changes. As a result, it is possible to construct a more sophisticated learning data set that avoids inappropriate learning and insufficient learning.

Further, in the machine learning dataset evaluation support device of the present embodiment, the inference execution unit may sequentially execute the plurality of trained AI models to obtain a plurality of inference results.

According to this, since only one model is executed at a time, it is possible to reduce computational resources required for inference execution.

Further, in the machine learning dataset evaluation support device of the present embodiment, the inference execution unit may execute the plurality of trained AI models in parallel to simultaneously obtain a plurality of inference results.

According to this, it is possible to shorten the processing time by simultaneously obtaining a plurality of inference results.

Further, in the machine learning dataset evaluation support device of the present embodiment, the learning dataset editing unit is configured such that the inference result of the inference result is similar to the evaluation data having a certainty greater than a predetermined threshold, and the inference result of the evaluation target data and the correct label may further execute a process of deleting learning data different from the learning data set from the learning data set.

According to this, it is possible to eliminate learning data with inappropriate correct labels.

Further, in the machine learning data set evaluation support device of the present embodiment, the learning data set editing unit deletes from the learning data set learning data similar to the evaluation data in which the certainty of the inference result is smaller than a predetermined threshold. It may be assumed that the processing to be performed is further executed.

According to this, it is possible to eliminate inadvertent proximity of learning data with different correct labels, and to eliminate learning data in which a feature amount different from the learning intention is dominant.

Further, in the machine learning data set evaluation support device of the present embodiment, the learning data set editing unit includes a learning data similar to the evaluation data in which the degree of certainty of the inference result is greater than a predetermined threshold and the degree of stability is less than a predetermined threshold. The process of generating data and adding it to the learning data set may be further performed.

According to this, it is possible to supplement missing learning data.

100 Machine learning data set evaluation support device 101 Storage device 102 Program 103 Memory 104 Arithmetic device 105 Input device 106 Output device 107 Communication device 110 Evaluation data set holding unit 111 Inference execution unit 112 Trained AI model holding unit 113 Difference analysis unit 114 Data Evaluation unit 115 Learning data set editing unit 116 Learning data set holding unit 117 Analysis result display unit 118 Trained AI model 125 Evaluation data set 126 Learning data set 127 Label information

Claims (10)

a learning data set holding unit that holds a learning data set and label information used for supervised machine learning;
an inference execution unit that inputs a predetermined evaluation data set to a plurality of trained AI models and obtains a plurality of inference results;
a difference analysis unit that analyzes the difference between the plurality of inference results from the degree of certainty of the inference result and the stability of the inference result in each of the plurality of trained AI models;
a data evaluation unit that evaluates the evaluation data set based on the difference analysis result;
a learning data set editing unit that edits the learning data set and label information held in the learning data set holding unit based on the evaluation result of the evaluation data set;
A machine learning dataset evaluation support device comprising: 2. The machine learning dataset evaluation support device according to claim 1, further executing a process of displaying the analysis result on an output device. 2. The machine learning dataset evaluation according to claim 1, wherein the inference execution unit uses an AI model trained by changing a learning dataset or model structure during learning as the plurality of trained AI models. support equipment. 2. The machine learning data set evaluation support device according to claim 1, wherein said inference execution unit uses, as said plurality of trained AI models, trained AI models whose model structures have been changed at the time of inference. 2. The machine learning dataset evaluation support device according to claim 1, wherein said inference execution unit sequentially executes said plurality of trained AI models to obtain a plurality of inference results. 2. The machine learning dataset evaluation support device according to claim 1, wherein the inference execution unit executes the plurality of trained AI models in parallel to simultaneously obtain a plurality of inference results. The learning data set editing unit performs a process of deleting from the learning data set learning data similar to the evaluation data in which the degree of certainty of the inference result is greater than a predetermined threshold and having a different correct label from the inference result of the evaluation target data. 2. The machine learning dataset evaluation support device according to claim 1, further comprising: The learning data set editing unit further executes a process of deleting from the learning data set learning data similar to the evaluation data whose certainty of the inference result is smaller than a predetermined threshold. The machine learning data set evaluation support device according to claim 1. The learning data set editing unit further performs a process of generating learning data similar to the evaluation data in which the confidence of the inference result is greater than a predetermined threshold and the stability is less than a predetermined threshold, and adding the learning data to the learning data set. 2. The machine learning data set evaluation support device according to claim 1, wherein the machine learning data set evaluation support device is characterized by: The information processing device
holding a learning data set and label information used for supervised machine learning in a storage device;
a process of obtaining a plurality of inference results by inputting a predetermined evaluation data set into a plurality of trained AI models; and a process of analyzing from the stability of the inference result, a process of evaluating the evaluation data set based on the analysis result of the difference, and a learning held in the storage device based on the evaluation result of the evaluation data set The process of editing the dataset and label information,
A machine learning dataset evaluation support method characterized by executing:

Images