JP2023086053A - Learning data management method, learning data generation apparatus, and machine learning method - Google Patents

Abstract

To provide a learning data management method which reflects data useful for learning of a model of a past generation in re-learning, a learning data generation apparatus, and a machine learning method.SOLUTION: In a learning data management method for recording, as inference result management information, information as to whether an inference result has been corrected or not, the inference result obtained by a first machine learning model trained with first learning data, an inference result management unit of an inference system serving as a learning data generation apparatus records the inference result management information. A dataset management unit extracts, as first correction data list, data for which inference results obtained by the first machine learning model have been corrected, based on the inference result management information. A learning data output unit outputs, as learning data, data including at least one piece of data included in the first correction data list. A machine learning method trains a machine learning model using the learning data output from the learning data output unit of the learning data generation apparatus.SELECTED DRAWING: Figure 1

Description

The present invention relates to learning data used for machine learning.

In recent years, inference using machine learning models has been put to practical use. As an example, an inspection system using a machine learning model has been proposed from the viewpoint of resolving labor shortages in the visual inspection of products in factories and the like. For example, in an inspection system that uses image data, it makes inferences by using labels such as scratches, defects, and distortions on the subject shown in the image data, and determines the degree of certainty for each of these as a value between 0 and 1. output the label with the maximum value of .

In inspection systems that utilize such machine learning, it is necessary to continuously update (re-learn) the model in order to maintain and improve the inference accuracy of the model. In re-learning, for example, a new model is generated by giving new learning data based on a certain model. As new training data, data with low confidence in inference in the model and data with high confidence given to erroneous labels are used as useful data for improving the accuracy of the model.

In other words, in order to perform re-learning efficiently in terms of time and data amount, it is necessary to efficiently extract and manage data considered useful for re-learning.

For example, in Patent Document 1, data with low accuracy is regarded as useful data for improving the accuracy of the model because the model is not good at it, and the inspection results of the inspection system are accumulated. A method of extracting data by sampling and assigning a correct label to the extracted data to extract new learning data is disclosed.

US2020/0151613 A1

Patent Document 1 does not consider data used for learning in past generations. In Patent Document 1, in order to collect data with low accuracy from the inference results of the inspection system, only data dependent on the model used in the current inspection system is extracted.

However, since the model is repeatedly relearned, there is a possibility that the data that was useful in the training of past generation models may not be reflected in the relearning.

An object of the present invention is to reflect data that was useful in learning models of past generations in re-learning.

A preferred aspect of the present invention is a first step of recording as inference result management information whether or not the inference result by the first machine learning model learned with the first learning data is corrected. management method.

Another preferred aspect of the present invention includes an inference result management unit, a data set management unit, and a learning data output unit, and the inference result management unit is configured to generate a first machine learning model trained with first learning data. records as inference result management information whether or not the inference result is corrected by The learning data generating device extracts data as a first correction data list, and the learning data output unit outputs data including at least one of the data included in the first correction data list as learning data.

Another preferred aspect of the present invention is a machine learning method of learning a machine learning model using learning data output from the learning data output unit of the learning data generation device.

Data that was useful in training previous generation models can be reflected in retraining.

FIG. 2 is a block diagram showing an example of system configuration; FIG. 4 is a table diagram showing an example of a data set management table; FIG. 4 is a diagram showing an example of a model management table; FIG. FIG. 4 is a table diagram showing an example of an inference result management table; FIG. 4 is an image diagram showing an example of a screen of a data set management unit; An example of a flowchart of processing of a data set management unit. An example of a flowchart of data extraction processing. An example of the flowchart of the process of a learning data output part. FIG. 4 is a table showing an execution example showing the operation of the embodiment; FIG. 4 is a table showing an execution example showing the operation of the embodiment; FIG. 4 is a table showing an execution example showing the operation of the embodiment; FIG. 4 is a table showing an example of data extracted by a data set management unit in an execution example; FIG. 4 is a table showing an example of data extracted by a data set management unit in an execution example; FIG. 4 is a table showing an example of data extracted by a data set management unit in an execution example; FIG. 11 is an image diagram showing an example of a screen of a data set management unit in an execution example; FIG. 2 is a block diagram showing an example of the configuration of a computer;

Embodiments of the present invention will be described below with reference to the drawings. In addition, the present invention is not limited by the following description.

In the configurations of the embodiments described below, the same reference numerals may be used in common for the same parts or parts having similar functions in different drawings, and redundant description may be omitted.

When there are a plurality of elements having the same or similar functions, they may be described with the same reference numerals and different suffixes. However, if there is no need to distinguish between multiple elements, the subscripts may be omitted.

Notations such as “first”, “second”, “third” in this specification etc. are attached to identify the constituent elements, and do not necessarily limit the number, order, or content thereof isn't it. Also, numbers for identifying components are used for each context, and numbers used in one context do not necessarily indicate the same configuration in other contexts. Also, it does not preclude a component identified by a certain number from having the function of a component identified by another number.

The position, size, shape, range, etc. of each component shown in the drawings, etc. may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the positions, sizes, shapes, ranges, etc. disclosed in the drawings and the like.

All publications, patents and patent applications cited herein are hereby incorporated by reference into this description.

Elements presented herein in the singular shall include the plural unless the context clearly dictates otherwise.

Although detailed description will be given in the following examples, according to the study of the inventors, the technique of Patent Document 1 has two problems. One is that the data used for learning in past generations is not considered. In order to collect data with low accuracy from the inference results of the inspection system, only data dependent on the model used in the current inspection system is extracted. However, because the model is retrained repeatedly, data that was useful in training previous generations may be leaked.

The second problem is that the results of labeling cannot be reflected in the extraction of datasets. The inference results contain mislabeled data even if the accuracy is high. These can be identified by manual labeling work, but the method of Patent Document 1 cannot extract these data as learning data.

In one embodiment, the learning history of the model and the correction history of the label for the inference result data are managed. Using these, the model that is the source of re-learning, the data whose labels are corrected for the inference results performed using the model of the generation that is the source of that model, and the data of the source model Extract the corrected label data included in the training data.

Such a configuration enables efficient extraction of label-corrected data, including past generation models. As a result, extraction of learning data in re-learning can be made more efficient. In addition, by extracting only useful data, it is possible to reduce the amount of data required for learning, which is expected to shorten the learning time.

This embodiment relates to a method of managing learning data necessary for re-learning in inference processing for inferring whether input data belongs to a certain class. As an example, defect inspection (labeling) using image data will be described. This embodiment is an example, and can be applied to other image analysis such as segmentation and classification, character recognition, and voice recognition. In addition, although an example of a machine learning model that handles a classification problem will be described in the embodiments, it can also be applied to a machine learning model that handles a regression problem in the same way.

FIG. 1 shows an example of a system configuration for realizing this embodiment. This system consists of an inference system 101 that executes inference, manages learning data, and performs re-learning, and a system user terminal 102 that is used by a user of the inference system 101. These are connected via a network 103. FIG.

The system can basically be configured with an information processing device such as a server, and has an input device, an output device, a storage device, and a processing device (CPU: Central Processing Unit) as a general computer configuration. . In the following description, the description of the configuration of the information processing apparatus will be omitted, and the description will focus on the functions of the present embodiment.

Inference system 101 includes inference unit 110 , data input unit 111 , inference result management unit 112 , learning data management unit 113 , relearning unit 114 , and input/output unit 115 . In this embodiment, these configurations are realized by the processing device executing software stored in the storage device of the information processing device. However, functions equivalent to those configured with software can also be realized with hardware such as FPGAs (Field Programmable Gate Arrays) and ASICs (Application Specific Integrated Circuits). Such aspects are also included in the scope of the examples.

In addition, this system can use an input data group 131, a data set management table 132, an inference result management table 133, a model group 135, and a model management table 134 as a database. The database can be stored, for example, in a storage device such as a magnetic disk device.

The system configuration described above may be composed of a single computer, or any part thereof may be composed of other computers connected via a network.

First, the user inputs to the inference system 101 through the input/output unit 115 a trained model to be used in the inference unit 110 and learning data used in learning the model. Network configurations such as DNN (Deep Neural Network) are generally known as models. One piece of learning data generally consists of a set of an explanatory variable (problem) and an objective variable (correct answer). It is also known to learn such a network by machine learning using learning data. In the present embodiment, description of known parts is omitted.

The input learning data is recorded in the input data group 131, and the data set management table 132 manages the group of the input data as a data set. Also, the models are recorded in the model group 135 and managed in the model management table 134 .

FIG. 2 is an example of the data set management table 132. As shown in FIG. A dataset ID 201 is an ID for identifying a dataset. A data list 202 has a list of IDs of input data included in the data set. For example, row 204 indicates that a data set with an ID of 1 consists of data with an ID of 1 to 50 in the input data group 131 . Hereinafter, a data set with an ID of 1 will be referred to as a data set 1, and data with an ID of 1 will be referred to as a data 1.

The dataset management table 132 can manage datasets of learning data and corrected data, which will be described later. Also, model test data and data to be inferred during actual operation may be managed. Also, a flag for identifying these data types may be added.

FIG. 3 is an example of the model management table 134. As shown in FIG. The model ID 301 is, for example, an ID for identifying the generation of the model. The model management table 134 includes, for a certain model, a learning data set ID 302 that is a data set used for learning, a correction data set ID 303 that represents a group of data that could not be correctly determined by inference using that model, and a model It is composed of a next model ID 304 indicating the relation of derivation by re-learning and a creation time 305 of the model. For example, in row 306, the model with model ID 1 indicates that the dataset with dataset ID 1 was used for training.

In the inference unit 110, the input/output unit 115 receives data to be inferred or test data from the user via the network 103, and performs inference using the trained model registered in the model management table 134. to run. Data to be inferred are generally explanatory variables (problems), and the inference unit 110 outputs the estimated objective variable. In this embodiment, inference is made using one model whose creation time is the latest.

The inference processing is, for example, outputting a value of 0 to 1 as a degree of similarity with respect to predetermined image features with respect to the input image data. In the case of defect inspection of products, labels representing features such as "flaw", "defect", and "distortion" are determined, and the similarity value of each label is output for the input image data. Then, the label having the largest value exceeding a certain number (for example, 0.8) for the label is determined as the label for the image data. The inference unit 110 newly assigns an ID to the input image data, records it in the input data group 131 , and records the inference result in the inference result management table 133 .

In addition to the labeling (classification) described above, object detection involves labeling an object in an image and detecting its position (XY coordinates). Skeleton detection involves labeling and locating specific locations of objects in an image. In segmentation, each pixel in the image is labeled with information to color-code the region. In speech recognition, characters are associated with speech data. The format of the training data set is basically a question-answer pair.

FIG. 4 is an example of the inference result management table 133. As shown in FIG. A data ID 401 indicates the ID of data registered in the input data group 131 . The model ID 402 records the ID of the model that performed the inference with respect to the data of the data ID 401, and the label 403 records the label output as a result of the inference. For example, line 406 indicates that the data with data ID 51 was inferred by model 1 and determined to be "normal". Line 407 indicates that data with a data ID of 52 was inferred in model 1 and determined to be "missing".

The inference unit 110 performs inference on the input data sequentially, but not all the inference results are correct. Therefore, an error in the inference result may be found after the inference process is executed. For example, when performing defect inspections on products flowing through a factory line, automatic diagnosis by an inference system and double checks are performed manually or by other inference systems. There are cases where it becomes clear. In such a case, the inference result management unit 112 records the newly found correct label in the inference result management table 133 .

For example, data 53 shown in row 408 of FIG. ing. In the data 54 indicated by line 409, the judgment of "normal" is later corrected to "wound".

If the number of inference results accumulated in the inference result management table 133 is small, the data input unit 111 can be used to add input data. In the data input unit 111 , the input/output unit 115 receives input data from the user via the network 103 and registers it in the input data group 131 . Also, the correction result 404 is registered via the inference result management unit 112 for the registered data. Further, the data input unit 111 may use the model registered in the model management table 134 to record the provisional inference result in the label 403 column. In this case, the user checks the label 403 created by the data input unit 111 and writes the correction result 404 if it is incorrect.

The learning data management unit 113 uses the data accumulated in the inference result management table 133 by the inference unit 110 to extract data sets that are candidates for data used for re-learning. The learning data management unit 113 performs inference when a sufficient amount of inference results by the inference unit 110 is accumulated, when the rate of label correction for the inference result in the inference unit 110 exceeds a certain level, or when the inference unit 113 periodically Executed by the user of the system 101.

In the learning data management unit 113, first, the data set management unit 121 extracts correction data for the model specified by the user interface, and displays the result. The user specifies the number and ratio of data to be included in the output data set for the displayed data set, and the learning data output unit 122 extracts data according to the specification and creates a new data set.

An example of the user interface 501 is shown in FIG. Such a GUI (Graphical User Interface) can be displayed on an image output device, for example. A pull-down menu 502 displays a list of models registered in the model management table 134, and the user selects one model.

Data set management unit 121 displays data in learning data information display unit 503, correction data information display unit 504, and other input data information area 505 according to the selected model.

A learning data information display section 503 displays information about the learning data of the model selected in the pull-down menu 502 . Corrected learning data information 506 represents learning data whose labels have been revised, and other learning data information 507 represents learning data whose labels have not been revised.

The corrected data information display section 504 displays information about the model selected in the pull-down menu 502, the model from which the model was learned, and the model group created based on the model. Among the inferred data, the newly labeled data is displayed. The corrected data information 508 extracts data for each model that is the basis of learning. Therefore, the number of models displayed by the corrected data information 508 increases according to the learning history of the models.

In the other input data information area 505 , information on data not included in the corrected data information 508 among the data inferred by the model selected in the pull-down menu 502 is displayed as new data information 509 .

In the example of FIG. 5, the estimation results of model 1 are not corrected for 50 learning data for which model 1 has been trained, two of the estimation results by model 1 of new data are corrected, and the other 300 are It shows that it has not been fixed.

FIG. 6 is an example of a flow chart of the dataset manager 121 for extracting data and generating display information for FIG. Start (S601) can be done at any timing. First, a model is selected from the pull-down menu 502 (S602).

Next, the data to be displayed in the learning data information display section 503 is extracted and displayed (S603). In this process, the data set management unit 121 extracts and records data with corrected labels from the learning data set of the selected model. In addition, data without label correction is extracted and recorded. Each number is displayed as corrected learning data information 506 and displayed as other learning data information 507 .

Specifically, data set management unit 121 searches model management table 134 using model ID 301 of the selected model, and extracts learning data set ID 302 used for learning of the selected model. Then, using the learning data set ID 302, the data set management table 132 is searched to extract the data set for the selected model.

Next, the data registered in the correction result 404 column of the inference result management table 133 are extracted as corrected learning data from the data included in the data set, and the number thereof is displayed in the corrected learning data information 506 . In addition, other data included in the data set are extracted, and the number thereof is displayed in the other learning data information 507.

The corrected learning data information 506 extracts all data that has been corrected even once in the past. If one data has been revised multiple times in the past, it will be counted as one.

Re-learning repeats the cycle of learning with a new data set, actually performing evaluation, and learning again with another data set if the evaluation is poor. In that case, it is important to deal with new data, but it is also important not to forget past learning. The corrected learning data information 506 is for creating a new learning data set by increasing the data of this portion if past learning is forgotten as a result of the evaluation. As a result, it becomes possible to more efficiently create a new learning data set by including the data acquired in the past learning in the learning data.

Next, information to be displayed in the corrected data information display section 504 is generated (S604, S605, S606). Based on the model selected in S602 and the next model ID 304 column of the model management table 134, the model that was the basis of the model selected in S602 and the model based on that model are sequentially extracted, and new Creates a list of model dependencies, ordered first. Also, the creation time 305 of the selected model is set as the reference time (S604).

For each model in the dependency list, inference is performed in that model, data whose label has been modified after the reference time is extracted, and the number thereof is displayed in the modified data information 508 (S605-S606).

The corrected data information 508 extracts data whose labels have been corrected, including related previous generations or subsequent generations, after the creation of the model selected in S602 (reference time). These data may include new input data that no longer fits the model due to changes in circumstances since the model was created. If the modified data is the training data with which the model was trained, then those data are also included. However, if it overlaps with the corrected learning data information 506, it may be excluded.

Finally, the information to be displayed in the other input data information area 505 is extracted. Based on the reference time, the data for which the correction result 404 is not registered after the reference time is extracted from the inference result management table 133, and the number thereof is registered as the new data information 509 (S607).

FIG. 7 is an example of a flowchart showing details of S606. The processing of S606 is executed for each generation model included in the dependency list. First, for the model to be processed (for example, the latest model), the corrected data set ID 303 column of the model management table 134 is cleared (S702).

A new ID is assigned to the cleared correction data set ID 303 column to create a new data set, and the ID of the data set is registered in the model management table 134 . A dataset with a cleared dataset ID may be cleared or left alone. When actually applied, it is desirable to keep the data set for a certain period of time for the purpose of preserving past records.

Next, for all the data in the inference result management table 133 (S703), the correction result 404 column is examined to check whether the label has been corrected (S704) and whether the correction time is later than the reference time. Examine (S705). If the condition is satisfied, the ID of the current data is added to the data list 202 of the data set indicated by the correction data set ID 303 column of the model management table 134. FIG.

For example, when model 1 shown in row 306 of FIG. 3 is selected in S602, in the learning data information display unit 503, the corrected learning data information 506 is 0, and the other data information 507 is the first registered learning data (1 50) are extracted (see Fig. 5).

As for the correction data information 508, two data 53 and 54 are extracted from the row 205 in FIG. Therefore, the data set shown in row 205 in FIG. 2 is created, and its ID 2 is recorded in the modified data set ID column 303 in FIG.

Also, since there is no model based on model 1 or a model based on model 1, only information about model 1 is displayed. Finally, in the other input data information 605, data shown in rows 406, 407, etc. in FIG. 4 are extracted as new data information 509 (see FIG. 5).

In the above description, all data registered in the inference result management table 133 are to be extracted. It may be possible. In that case, the inference result management table 133 can be dealt with by preparing a flag indicating confirmation by the user. If the user confirms the label by double-checking, etc., the confirmed flag is set. In the data extraction process (FIG. 7), only data for which the confirmed flag is set is extracted.

Also, in the flowchart shown in FIG. 6, the dependencies created in S604 are represented as a list, but a plurality of models may be created from a certain model. Such a case can be dealt with by recording IDs of a plurality of models in the next model ID of the model management table 134 and tracing all dependent models.

Next, the user uses the data extracted by the data set management unit 121 to create a new data set to be used for the next learning data. Screen 511 in FIG. 5 is an example of the data set output screen. Based on the extracted learning data information display section 503 and corrected data information display section 504, the user can select the number of data to be included in the new data set and the number of data finally created from each data group. Specify the number (512). When a new creation button 513 is pressed, a new data set is created by supplementing specified data and missing data with new data 509 and registered in the data set management table 201 .

FIG. 8 is an example of a flow chart of processing of the learning data output unit 122 . First, a specified number of data is added to the output data set from the data extracted in S603 (learning data with modified inference results) (S802). At this time, if the number specified for the number of data is small, a random selection is made from the data. These data are used as learning data, and are expected to be effective in reducing degradation.

Next, for the dependency list model created in S604, for the data extracted and recorded in S606 (data modified after the reference time), the specified number of data is output as an output data set. Add (S805). The number may be specified for each model. These data can be expected to be useful for learning corresponding to new input data.

Finally, if the number of data in the output data set does not satisfy the specified total number, the shortage is extracted from the data recorded in S607 (S807).

The data extracted and output as described above can be used as learning data to be used for the next re-learning. In the present embodiment, it is possible to generate learning data capable of coping with degradation and learning corresponding to new input data. Also, by adjusting the number of data points, it is possible to adjust the characteristics of the learning data.

In S802, S805, and S807, the method of selecting data at random may be changed to select new data or extract data with high priority. Higher priority means, for example, labels whose label errors were difficult to find, for example, labels whose errors were found later in the process or whose errors could only be found by skilled workers. These can be realized, for example, by adding information about process and label correctors to the inference result management table 133 and prioritizing them.

Next, the user re-learns the model in re-learning section 114 using the data set created in learning data management section 113 and the model managed in model management table 134 . When a model that satisfies evaluation criteria such as model recognition accuracy is created by re-learning, the newly created model and data set are registered in the model management table 134 . Also, the ID of the newly registered model is recorded in the next model ID 304 field of the learning source model. This makes it possible to manage the learning history of the model and associate the model with learning data.

In some cases, re-learning may not produce a model that satisfies the evaluation criteria. In that case, a new data set is created from a new combination of data and re-learning is repeated. If the newly created dataset drops the evaluation criteria significantly, it is likely that the newly adopted data is having a negative impact. Therefore, it is necessary to exclude these data in subsequent processing of the learning data management unit 113 . These are dealt with by providing the inference result management table 133 with a new flag for exclusion at the time of data extraction.

A button for setting exclusion information is prepared for the screen displayed by learning data management unit 113 . When the button is pressed, exclusion flags are set for data newly adopted in the created data set for the learning data of the selected model. Data set management unit 121 excludes data for which an exclusion flag is set from the extraction target at the time of data extraction.

A specific example will be described with reference to FIGS. 9A to 9C and FIGS. 10A to 10C. An operation example when the cycle of inference, label correction, data set extraction, and re-learning is repeated by the above processing is shown in the data set management table 132 (FIG. 9A), the model management table 134 (FIG. 9B), and the inference result management table 133. (FIG. 9C) and FIGS. 10A to 10C showing information displayed on the screen in FIG.

In this example, model 2 is generated by re-learning based on model 1, and model 3 is generated from model 2 (FIG. 9B). It should be noted that the values in the creation time columns in FIGS. 9B and 9C assume that time has elapsed in ascending order of numerical value.

First, a model 1 as an initial model and a data set 1 as its learning data are registered in the model management table 134 of FIG. 9B (911). In the data set management table 132 of FIG. 9A, data set 1 consists of data 1 to 100 (901).

Next, assume that the inference unit 110 makes an inference using the model 1, inputs the data 101 to 200, and then corrects the label of the data 102 by the inference result management unit 112 (921 in FIG. 9C). .

Here, when the processing of the learning data management unit 113 is executed to create a new model 2, the data set management unit 121 extracts the data shown in FIG. 10A. Also, in this example, in the generation of the data set by the learning data output unit 122, the number of generated data is set to 100, corrected data is always used, and 10 new data are always included. . This extracts the data shown in row 902 of FIG. When the model learned with this data set is adopted as model 2, row 912 of model management table 134 in FIG. 9B is registered.

Next, as in the case of model 1, model 2 is used in inference unit 110, data 201 to 300 are input, and then labels of data 103 and 203 are corrected by inference result management unit 112. (922, 923 in FIG. 9C). In this case, the data set management unit 121 extracts the data shown in FIG. 10B. Next, the learning data output unit 122 creates the data set 3 indicated by row 903 in FIG. 9A, and registers the model 3 learned with that data set.

The correction of the data 103 assumes that an error in the result when using the past model is discovered later and corrected. In FIG. 9C, the inference result of the model with model ID1, which is the previous generation, is corrected at time 103, which is the time after the model with model ID2 is activated. As a specific example, in a factory inspection, the error was not found at the manufacturing site and shipped, and then a new lot was updated and inspected. This is the case, for example, when there is a report of a defective product in one lot.

Next, in the same way as the models 1 and 2, the inference unit 110 uses the model 3, inputs the data 301 to 400, and then the inference result management unit 112 processes the data 104, 202, 203, 302, 303 shall be modified. In this case, the data set management unit 121 extracts the data shown in FIG. 10C.

FIG. 11 shows an example of a screen displayed by the learning data management unit 113 at this time. Here, in model 3, 3 pieces of learning data are corrected, and these 3 pieces are adopted as the next learning data. Also, 97 of the training data were not corrected, but 83 of them are adopted as the next training data. After the reference time, each generation model of model 3, model 2, and model 1 has 2, 1, and 1 data corrected, respectively, and these are adopted as the next learning data. In this embodiment, the numbers of corrected data displayed on display units 1108, 1109, and 1110 do not distinguish between learning data, test data, and operational data, but they can be displayed separately.

The user can arbitrarily set how many of each data displayed on the screen displayed by the learning data management unit 113 are adopted as the next learning data by adjusting the numerical values in the frame 511. .

Other learning data 507 includes data that has been learned in past generations and correctly recognized by the model. For example, if the following learning data is created by reducing the data of the other learning data 507, and if the learned model is degraded, it is presumed that the reduced data is the cause of the degradation. . Therefore, by using the other learning data 507, it is possible to cope with the deterioration. In this way, data that was useful in training models of past generations can be reflected in retraining.

As described above, in this embodiment, for the model used in inference, the data in which the label contained in the learning data is corrected, the label inferred by the model related to the model It is now possible to extract a list of modified data, and by easily creating a dataset that combines these data, the efficiency of data management by the user can be improved. Become.

FIG. 12 shows an example of a computer system in which the inference system 101 described in this embodiment is executed. The computer system comprises an arithmetic device 1201 , a storage device 1202 , a memory device 1203 , a communication device 1204 and an input/output device 1205 , which are connected via a bus 1206 . A program that configures the inference system 101 is held in the storage device 1202 and executed by the arithmetic device 1201 .

In addition, in the present embodiment, classification by labels for image data is explained as an example, but the inference processing is not limited to image data, and classifies voice data, text data, numerical data, etc. can also be applied to

In the above embodiments, the learning history of the model and the revision history of the label for the inference result data are managed. Using these, the data with corrected labels and the labels included in the training data for the inference results performed using the original model for re-learning and the generation model that was the basis of that model. extract the corrected data.

As described above, it is possible to extract data whose label is corrected, which is important for improving the accuracy of the model, based on the history of the model, and it is possible to improve the management efficiency of the learning data.

According to the above embodiment, efficient machine learning can be realized, so that energy consumption can be reduced, carbon emissions can be reduced, global warming can be prevented, and contribution can be made to the realization of a sustainable society.

Inference system 101, system user terminal 102, network 103, inference unit 110, data input unit 111, inference result management unit 112, learning data management unit 113, relearning unit 114, input/output unit 115, input data group 131, data Set management table 132, inference result management table 133, model group 135, model management table 134

Claims (15)

A first step of recording as inference result management information whether or not the inference result by the first machine learning model learned with the first learning data has been corrected;
A learning data management method that performs a second step of obtaining, as a first corrected data list, data obtained by correcting the inference result of the first machine learning model based on the inference result management information;
a third step of causing the second learning data to include at least a portion included in the first modified data list when learning the first machine learning model with the second learning data;
The learning data management method according to claim 1, wherein In the third step,
At least part of the first correction data list to be included in the second learning data is at least part of the first learning data,
The learning data management method according to claim 2. When re-learning the previous generation machine learning model to obtain the first machine learning model,
a fourth step of recording as the inference result management information whether or not the inference result by the previous generation machine learning model has been corrected;
a fifth step of obtaining, as a second corrected data list, data obtained by correcting the inference result of the previous generation machine learning model based on the inference result management information;
a sixth step of causing the second learning data to include at least a portion included in the second modified data list when learning the first machine learning model with the second learning data;
3. The learning data management method according to claim 2, wherein In the case of obtaining the previous generation machine learning model by relearning the previous generation machine learning model,
a seventh step of recording, as the inference result management information, whether or not the inference result by the second generation machine learning model has been corrected;
an eighth step of obtaining, as a third corrected data list, data obtained by correcting the inference result of the second-previous generation machine learning model based on the inference result management information;
a ninth step of causing the second learning data to include at least a portion included in the third modified data list when learning the first machine learning model with the second learning data;
5. The learning data management method according to claim 4, wherein Creating model management information that records the relationship between generations of the second generation machine learning model, the previous generation machine learning model, and the first machine learning model;
The learning data management method according to claim 5. The model management information includes information specifying learning data used for learning each machine learning model and information about the creation time of each machine learning model.
The learning data management method according to claim 6. When causing the second learning data to include at least a portion included in the first correction data list and at least a portion included in the second correction data list,
The number of data to be selected from the first correction data list and the number of data to be selected from the second correction data list can be arbitrarily specified,
The learning data management method according to claim 4. The inference result management information includes information specifying input data to be inferred, information specifying a machine learning model used for inference of the input data, an inference result by the machine learning model, and corrections to the inference result. including results and correction time information,
The learning data management method according to claim 1. Equipped with an inference result management unit, a data set management unit, and a learning data output unit,
The inference result management unit
recording as inference result management information whether or not the inference result by the first machine learning model learned with the first learning data has been corrected;
The data set management unit
based on the inference result management information, extracting data obtained by correcting the inference result of the first machine learning model as a first corrected data list;
The learning data output unit
outputting data including at least one of the data included in the first correction data list as learning data;
Learning data generator. The inference result management unit
recording as inference result management information whether or not the inference result by the machine learning model of the generation prior to the first machine learning model has been corrected;
The data set management unit
based on the inference result management information, extract data in which the inference result of the previous generation machine learning model has been corrected as a second corrected data list;
The learning data output unit
outputting data including at least one of the data included in the second correction data list as learning data;
11. The learning data generation device according to claim 10. The learning data output unit
at least one of the first learning data included in the first modified data list and the second learning data obtained by learning the previous generation machine learning model included in the second modified data list; , output as the learning data,
12. The learning data generation device according to claim 11. Equipped with a data set management section,
The data set management unit
displaying the number of data included in the first correction data list and the number of data included in the second correction data list;
a GUI that enables designation of the number of data included in the first correction data list and the number of data included in the second correction data list to be included in the learning data output by the learning data output unit. indicate,
13. The learning data generation device according to claim 12. The data set management unit
Further displaying the number of the first learning data, and displaying a GUI that enables specification of the number of the first learning data to be included in the learning data output by the learning data output unit.
14. The learning data generation device according to claim 13. Learning a machine learning model using the learning data output from the learning data output unit of the learning data generation device according to claim 10,
machine learning method.

Images