JP2023165464A - Machine learning model update device, machine learning model update method, and machine learning model update program - Google Patents

Abstract

To appropriately determine whether to update a learning model.SOLUTION: A machine learning model update device 1 includes: input means 11 that receives operation data C; evaluation means 13 that evaluates whether or not a sample CS constituting the operation data C is included in a set of first learning data D that is used for learning a first learning model M; calculation means 14 that obtains an update determination index value θ1 for determining whether or not to update the first learning model M to a second learning model M by using a result obtained by the evaluation means 13; and update means 17 that updates, when the update determination index value θ1 satisfies an update condition, the first learning model M to the second learning model M.SELECTED DRAWING: Figure 3

Description

The present invention relates to a machine learning model updating device, a machine learning model updating method, and a machine learning model updating program.

An inference device using machine learning updates a learning model to improve inference results. For example, it may be determined whether or not to update a learning model using operational data used for inference. The device disclosed in Patent Document 1 evaluates operational data, which is a set of a plurality of samples, using an average and a variance. The device disclosed in Patent Document 1 uses evaluation values based on the average and variance to determine whether or not to update the learning model.

For example, Patent Document 1 discloses an apparatus having a function of updating a machine learning model. This device executes index value calculation, index value calculation, and learning model update in this order.

JP2020-86778A

Operational data does not necessarily follow a normal distribution. When operational data follows a normal distribution, a set of multiple samples can be appropriately evaluated using the mean and variance. Therefore, it is possible to appropriately determine whether or not to update the learning model. However, if the operational data does not follow a normal distribution, a set of multiple samples cannot be appropriately evaluated based on the mean and variance. As a result, it may not be possible to appropriately determine whether or not to update the learning model.

The present invention provides a machine learning model updating device, a machine learning model updating method, and a machine learning model updating program that can appropriately determine whether to update a learning model.

A machine learning model updating device that is an embodiment of the present invention includes an input means for receiving operational data and elements constituting the operational data, which are included in a first learning data set used for learning a first learning model. an evaluation means for evaluating whether or not the first learning model is updated to the second learning model; and a calculation means for obtaining an update determination index value for determining whether to update the first learning model to the second learning model using the results of the evaluation means. and updating means for updating the first learning model to the second learning model when the update determination index value satisfies the update condition.

The machine learning model update device updates the first learning model with the second learning model to determine whether each element constituting the operational data is included in the first learning data set used for learning the first learning model. This is used as a condition for whether or not to update to the learning model. By evaluating whether each element is included in the set of learning data, it is possible to determine which aspects need to be updated and which ones do not need to be updated, regardless of the distribution of elements included in the operational data. Able to appropriately judge aspects. In other words, the machine learning model updating device can appropriately determine whether to update the learning model.

The updating means of the machine learning model updating device described above includes an update judgment means that uses an update judgment index value to judge whether or not to update the first learning model to a second learning model, and an update judgment means that uses operational data. learning data updating means for updating the first learning data to the second learning data; and learning model updating means for updating the first learning model to the second learning model using the second learning data. May have. According to this configuration, the first learning model can be suitably updated to the second learning model.

The evaluation means of the machine learning model updating device described above evaluates whether or not each element is included in the first set of learning data, and if the element is not included in the first set of learning data, A flag indicating that the element is not included in the first learning data set may be attached to the element. According to this configuration, it is possible to determine whether to update based on the presence or absence of the flag.

The evaluation means of the machine learning model updating device described above may evaluate whether or not an element is included based on the distance from the first learning data set to the element. According to this configuration, it is possible to determine whether to update based on the presence or absence of the flag.

The calculation means of the machine learning model updating device described above may obtain the ratio of the number of flagged elements to the total number of elements included in the operational data as the update determination index value. According to this configuration, it becomes possible to easily determine whether to update.

The update condition in the above machine learning model update device may be a threshold value. The update determination means may determine to update the first learning model to the second learning model when the update determination index value exceeds a threshold value. According to this configuration, it becomes possible to make an update determination more easily.

Another aspect of the present invention is a machine learning model updating method, in which the step of receiving operational data and the elements constituting the operational data are included in a first learning data set used for learning a first learning model. and a step of obtaining an update determination index value for determining whether to update the first learning model to the second learning model using the result of the evaluation. , updating the first learning model to the second learning model when the update determination index value satisfies the update condition. Similar to the machine learning model update device, the machine learning model update method appropriately distinguishes between aspects that need to be updated and aspects that do not need to be updated, regardless of the distribution of elements included in operational data. I can judge.

A machine learning model update program according to yet another embodiment of the present invention includes a step of receiving operational data and elements constituting the operational data included in a first learning data set used for learning the first learning model. and a step of obtaining an update determination index value for determining whether to update the first learning model to the second learning model using the results of the evaluation. and updating the first learning model to the second learning model when the update determination index value satisfies the update condition. Similar to the machine learning model update device, the machine learning model update program also appropriately identifies aspects that need to be updated and aspects that do not need to be updated, without being influenced by the distribution of elements included in operational data. I can judge.

The machine learning model updating device, the machine learning model updating method, and the machine learning model updating program of the present invention can appropriately determine whether to update a learning model.

FIG. 1 is a functional block diagram schematically showing how a machine learning model updating device is used. FIG. 2 is a configuration diagram of a computer that implements the machine learning model updating device. FIG. 3 is a functional block diagram of an information providing device including a machine learning model updating device. FIG. 4(a) is an example of learning data before updating. FIG. 4(b) is an example of operational data. FIG. 4(c) is a schematic diagram for explaining the determination process. FIG. 4(d) is an example of updated learning data. FIG. 5 is a flowchart of information provision processing including a machine learning model updating method executed by the machine learning model updating device. FIG. 6 is a flowchart of a machine learning model updating method executed by the machine learning model updating device.

Hereinafter, a machine learning model updating device, a machine learning model updating method, and a machine learning model updating program according to embodiments will be described in detail with reference to the accompanying drawings. In addition, in the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description will be omitted.

FIG. 1 is a functional block diagram showing an example of how the machine learning model updating device 1 is used. The machine learning model updating device 1 is used, for example, to monitor or control the target equipment 201. The target equipment 201 is provided with a plurality of sensors 202, 203, and 204. For example, sensor 202 measures temperature. Sensor 203 measures pressure. Sensor 204 measures the flow rate. The sensors 202, 203, and 204 output the measured values to the information providing device 100.

The information passed from the sensors 202, 203, and 204 to the information providing device 100 is referred to as "operational data C." That is, the operational data C includes at least one type of measurement value associated with time. In the example of FIG. 1, the operational data C includes data in which temperature and time are associated, data in which pressure and time are associated, and data in which flow rate and time are associated. Further, for example, data in which temperature and time are associated includes a plurality of temperature values measured at a certain time. In other words, the data includes multiple numerical values. Each numerical value indicating these temperatures, flow rates, and pressures is called a "sample" (element).

The information providing device 100 uses the operational data C to infer values regarding the target equipment 201 and outputs the inferred results. For example, the value regarding the target equipment 201 may be a value for monitoring the operating status of the target equipment 201. In this case, the information providing device 100 outputs the inference result R to an external device 206 such as a display. Further, the information providing device 100 may transmit the inference result R to an upstream device via a communication line.

The values related to the target equipment 201 are not limited to those used for monitoring the target equipment 201. The value related to the target equipment 201 may be a value for controlling the target equipment 201. In this case, the information providing device 100 outputs the inference result R to the controller 205. Controller 205 generates control signal H using inference result R. Then, the controller 205 outputs the control signal H to the target equipment 201.

The information providing device 100 includes a machine learning model updating device 1, a storage device 2, and an inference device 3. These functional blocks are realized by executing the machine learning model update program PG on the computer 300 (see FIG. 2).

The machine learning model updating device 1 generates a so-called trained model. Hereinafter, the learned model will be simply referred to as "learning model M." The machine learning model updating device 1 generates a learning model M using operational data C, for example. The generation of the learning model M executed by the machine learning model updating device 1 includes the operation of generating the first learning model M. The operation of generating the first learning model M is referred to as a learning phase, for example. Furthermore, the generation of the learning model M executed by the machine learning model updating device 1 also includes the operation of generating a new learning model M by updating the already generated learning model M. The operation of updating the learning model M is referred to as an update phase, for example. The machine learning model updating device 1 outputs the generated learning model M to the storage device 2.

The storage device 2 stores the learning model M. Note that the storage device 2 may store various information other than the learning model M.

The inference device 3 uses the learning model M and the operational data C to infer values regarding the target equipment 201. The operation of inferring values regarding the target equipment 201 is referred to as an inference phase. The inference device 3 outputs the inference result R to the controller 205 and the external device 206.

The hardware configuration of the machine learning model updating device 1 will be described with reference to FIG. 2. The machine learning model updating device 1 is realized, for example, by the computer 300 executing the machine learning model updating program PG. The machine learning model update program PG is for causing the computer 300 to execute a machine learning model update method.

Machine learning model updating device 1 includes one or more computers 300. The computer 300 includes a CPU (Central Processing Unit) 301 that is a processor, a main storage section 302, an auxiliary storage section 303, a communication control section 304, an input device 305, and an output device 306. The machine learning model updating device 1 is configured by one or more computers 300 configured by these hardware and software such as programs.

When the machine learning model updating device 1 is configured by a plurality of computers 300, these computers 300 may be connected locally or may be connected via a communication network such as the Internet or an intranet. Through this connection, one machine learning model updating device 1 is logically constructed.

The CPU 301 executes an operating system, application programs, and the like. The main storage unit 302 is composed of a ROM (Read Only Memory) and a RAM (Random Access Memory). The auxiliary storage unit 303 is a storage medium composed of a hard disk, flash memory, and the like. The auxiliary storage unit 303 generally stores a larger amount of data than the main storage unit 302. The communication control unit 304 is configured by a network card or a wireless communication module. The input device 305 includes a keyboard, a mouse, a touch panel, a voice input microphone, and the like. The output device 306 includes a display, a printer, and the like.

The auxiliary storage unit 303 stores in advance the machine learning model update program PG and data necessary for processing. The machine learning model update program PG causes the computer 300 to execute each functional element of the machine learning model update device 1. For example, the machine learning model update program PG is read by the CPU 301 or the main storage unit 302, and controls at least one of the CPU 301, the main storage unit 302, the auxiliary storage unit 303, the communication control unit 304, the input device 305, and the output device 306. make it work. For example, the machine learning model update program PG reads and writes data in the main storage unit 302 and the auxiliary storage unit 303.

The machine learning model update program PG may be provided after being recorded on a tangible recording medium such as, for example, a CD-ROM, a DVD-ROM, or a semiconductor memory. The machine learning model update program PG may be provided as a data signal via a communication network.

The machine learning model updating device 1 will be described in detail below.

As shown in FIG. 3, the machine learning model updating device 1 includes several functional components. These functional components are realized by the computer 300 executing the machine learning model update program PG.

The machine learning model updating device 1 includes an input means 11 , a learning model generation means 12 , an evaluation means 13 , a calculation means 14 , an update determination means 15 , and a learning data update means 16 . The learning model generating means 12, the update determining means 15, and the learning data updating means 16 constitute an updating means 17. Note that, apart from these functional components, the machine learning model updating device 1 may include functional components for additional operations required to update the learning model M, as necessary.

The input means 11 receives operational data C from outside the information providing device 100 . The input means 11 passes the received operational data C to the storage device 2. The storage device 2 stores operational data C. The input means 11 may correspond to, for example, the communication control unit 304 or the input device 305 shown in FIG. 2. The input means 11 may pass the operational data C to the storage device 2 without performing any data processing on the operational data C. Further, the input means 11 may perform some data processing on the operational data C and then deliver the processed operational data C to the storage device 2.

The learning model generation means 12 generates a learning model M. The learning model generation means 12 receives learning data D from the storage device 2. The learning model generating means 12 performs learning using the learning data D. Then, the learning model generating means 12 transfers the learning model M that has been trained to the storage device. The learning model generating means 12 can generate a new learning model M, or can generate a new learning model M based on the learning model M already stored in the storage device 2. That is, the learning model generating means 12 can obtain an updated learning model M by using the learning data D updated by the learning data updating means 16, which will be described later, as teacher data. Therefore, the learning model generation means 12 also functions as the learning model updating means described in the claims.

As the learning model M, various models can be adopted. For example, a neural network model may be adopted as the learning model M. Further, as the learning model M, a decision tree model may be adopted. In the machine learning model updating device 1 of this embodiment, it is determined whether or not to update the learning model M using the result of evaluating whether a sample of the operational data C is included in the learning data D. The decision tree model is well suited for such judgments.

The evaluation means 13 flags the samples forming the operational data C. The evaluation means 13 receives learning data D and operational data C from the storage device 2. The learning data D that the evaluation means 13 receives from the storage device 2 at this time is the data that was used to generate the learning model M that is currently being used. Further, the operational data C that the evaluation means 13 receives from the storage device 2 may be used in the inference processing that is currently being performed. In other words, the learning data D received by the evaluation means 13 may be earlier in time than the operational data C received by the evaluation means 13.

The evaluation means 13 evaluates whether the sample CS (see FIG. 4(b)) constituting the operational data C is included in the set of learning data D (see FIG. 4(a)). Specifically, the evaluation means 13 selects one sample CS constituting the operational data C, and evaluates whether the selected sample CS is included in the set of learning data D. That is, the evaluation means 13 evaluates each sample CS to see whether it is included in the set of learning data D (see FIG. 4(c)). Therefore, in principle, the number of sample CSs included in the operational data C does not need to be plural, and the number of sample CSs may be one. For a set of samples, a sufficient number of samples are required in order to obtain evaluation values such as the average and variance that well represent the characteristics of the set. However, since the evaluation means 13 processes the sample CS one by one, the number of sample CS does not necessarily have to be plural.

Various outlier detection algorithms may be used to evaluate whether a value is included in a set. Examples of algorithms for outlier detection include the local outlier factor method (LOF) and the k-nearest neighbor method (kNN). These outlier detection algorithms are distance-based detection techniques. Note that the outlier detection employed by the evaluation means 13 does not necessarily have to be a detection method based on distance. For example, the evaluation means 13 may evaluate whether the sample CS is included in the set of learning data D using a neural network.

When the evaluation means 13 evaluates that the sample CS is included in the set of learning data D, it attaches an inclusion flag to the sample CS indicating that it is included in the set of learning data D. For example, the inclusion flag indicating inclusion in the set of learning data D may be indicated by a numerical value of "0".

When the evaluation means 13 evaluates that the sample CS is not included in the set of learning data D, it attaches an inclusion flag indicating that the sample CS is not included in the set of learning data D to the sample CS. For example, the inclusion flag indicating that it is not included in the set of learning data D may be indicated by a numerical value of "1". Note that the evaluation means 13 may attach an inclusion flag only to the sample CS that is evaluated as not being included in the set of learning data D.

The evaluation means 13 passes the flagged operational data CF including the samples to which the inclusion flag is attached to the calculation means 14 .

The calculation means 14 receives the flagged operation data CF from the evaluation means 13. The calculation means 14 obtains the update determination index value θ1 using the flagged operation data CF. The update determination index value θ1 may be, for example, the number of samples to which an inclusion flag (1: not included in the set of learning data D) is attached. Further, the update determination index value θ1 may be, for example, the ratio of samples to which the inclusion flag (1: not included in the set of learning data D) is attached to the total number of samples included in the flagged operational data CF. The calculation means 14 passes the update determination index value θ1 to the update determination means 15.

The update determination means 15 receives the update determination index value θ1 from the calculation means 14. The update determination means 15 determines whether the update determination index value θ1 satisfies an update condition. The update condition may be a threshold value. For example, when the update determination index value θ1 is the ratio of samples to which the inclusion flag (1) is attached to the total number of samples, and the ratio is larger than a predetermined value (update condition ), it may be determined that the learning model M is to be updated. The update determining means 15 may determine not to update the learning model M if the update conditions are not satisfied. The update determining means 15 passes the determination result θ2 to the learning data updating means 16.

The learning data updating means 16 receives the determination result θ2 from the update determining means 15. The learning data updating means 16 updates the learning data D when receiving the determination result θ2 that the update determination index value θ1 satisfies the update condition. Specifically, the learning data updating means 16 receives learning data D and operational data C from the storage device 2. The learning data updating means 16 updates the learning data D using the operational data C. The learning data updating means 16 then passes the updated learning data D to the storage device 2.

There are no particular restrictions on the method of updating the learning data D. For example, updated learning data D may be obtained by adding only the sample CS to which the inclusion flag (1: not included in the set of learning data D) is attached to the current learning data D. In addition, updated learning data D can be obtained by adding all samples with connotative flags to the current learning data D, regardless of the contents of the connotative flags. (See FIG. 4(d)).

Next, a machine learning model updating method will be described with reference to the flowcharts shown in FIGS. 5 and 6.

First, as shown in FIG. 5, first operational data C is received (step S1). Step S1 is executed by the input means 11. Next, a first learning model M is obtained using the first operational data C (step S2). Step S2 is executed by the learning model generation means 12. These steps S1 and S2 are what is called a learning phase. Next, second operational data C is received (step S3). Step S3 is executed by the input means 11. Then, an inference result R is obtained using the second operational data C and the first learning model M (step S4). Step S4 is executed by the inference device 3. This step S4 is a so-called inference phase.

Here, the machine learning model updating method (step S5) uses the second operational data C. Therefore, the machine learning model updating method may be performed after the inference phase is completed. Note that the machine learning model updating method may be performed in parallel with the inference phase. After the machine learning model update method (step S5), third operational data C is received (steps S6 and S7). Then, in the second inference phase, depending on the execution result of the machine learning model updating method (step S5), a process using the updated second learning model M (step S8) or a process using the unupdated first learning model M Any one of the processes using (step S9) is executed.

The machine learning model updating method will be described in detail below with reference to FIG. It is now assumed that the storage device 2 stores first learning data D and a first learning model M.

First, the second operational data C is received (step S51). Step S51 is executed by the input means 11. Next, evaluation processing is executed. The evaluation process is executed by the evaluation means 13. Specifically, when the operational data C includes M samples, it is evaluated whether the m-th sample CS is included in the first learning data set D (step S52). If included (step S52: YES), a inclusion flag (0) is attached to the m-th sample (step S53). If it is not included (step S52: NO), an inclusion flag (1) is attached to the m-th sample (step S54). Next, it is determined whether inclusion flags have been attached to all sample CS (step S55). If the inclusion flag is not attached to all the samples CS (step S55: NO), step S52 is executed again. If the inclusion flag is attached to all the samples CS (step S55: YES), the next step S56 is executed. The above steps S52 to S55 constitute evaluation processing.

An update determination index value θ1 is obtained (step S56). Step S56 is executed by the calculation means 14. Next, it is determined whether the update determination index value θ1 satisfies the update conditions (step S57). Step S57 is executed by the update determination means 15.

If the update determination index value θ1 does not satisfy the update condition (step S57: NO), the learning data D is not updated. In this case, the machine learning model update process ends. After that, step S7 and step S9 shown in FIG. 5 are executed.

If the update determination index value θ1 satisfies the update condition (step S57: YES), the first learning data D is updated to obtain the second learning data D (step S58). Next, a second learning model M is obtained using the second learning data D (step S59). Step S59 is executed by the learning model generation means 12. Thereafter, the machine learning model update process (step S5) is ended. After that, step S6 and step S8 shown in FIG. 5 are executed.

The machine learning model updating device 1 includes an input means 11 for receiving operational data C, and a sample CS constituting the operational data C, which is included in a set of first learning data D used for learning the first learning model M. an evaluation means 13 for evaluating whether or not the first learning model M is updated, and a calculation means 14 for obtaining an update determination index value θ1 for determining whether or not to update the first learning model M using the results of the evaluation means 13; , updating means 17 for updating the first learning model M to the second learning model M when the update determination index value θ1 satisfies the update condition.

The machine learning model updating device 1 determines, for each sample CS constituting the operational data C, whether or not it is included in the set of first learning data D used for learning the first learning model M. This is used as a condition for determining whether or not to update the learning model M to the second learning model M. By evaluating whether each sample CS is included in the set of learning data M, it is possible to determine the aspects that need to be updated and the updates, regardless of the distribution of the sample CS included in the operational data C. It is possible to appropriately judge whether or not this is necessary. In other words, the machine learning model updating device 1 can appropriately determine whether or not to update the learning model M.

The updating means 17 of the machine learning model updating device 1 described above includes an update determining means 15 that determines whether or not to update the first learning model M using the update determination index value θ1, and an update determining means 15 that uses the operational data C. learning data updating means 16 that updates the first learning data D to the second learning data M using the learning data D; and learning that updates the first learning model M to the second learning model M using the second learning data D. model generation means 12. According to this configuration, the first learning model M can be suitably updated to the second learning model M.

The evaluation means 13 of the machine learning model updating device 1 configures the operational data C as a result of evaluating whether each sample CS constituting the operational data C is included in the first set of learning data M. If the sample CS is not included in the first set of learning data D, an inclusion flag (1) indicating that it is not included in the first set of learning data D is attached to the sample CS. According to this configuration, it is possible to determine whether to update based on the presence or absence of the inclusion flag.

The evaluation means 13 of the machine learning model updating device 1 described above evaluates whether or not the sample CS is included based on the distance from the first set of learning data D to the sample CS. According to this configuration, it is possible to determine whether to update based on the presence or absence of the inclusion flag.

The calculation means 14 of the machine learning model updating device 1 described above obtains the ratio of the number of sample CSs to which the connotation flag (1) is attached to the total number of sample CSs included in the operational data C as the update determination index value θ1. According to this configuration, it becomes possible to easily determine whether to update.

In the machine learning model updating device 1 described above, the update condition is a threshold value. The update determination means 15 determines to update the first learning model M when the update determination index value θ1 exceeds the threshold value. According to this configuration, it becomes possible to make an update determination more easily.

The machine learning model update device, machine learning model update method, and machine learning model update program of the present invention are not limited to the embodiments described above.

The calculation means 14 uses the content of the inclusion flag attached to the flagged operational data CF to obtain the update determination index value θ1. Specifically, the calculation means 14 of the embodiment obtained the update determination index value θ1 using the information indicated by the binary value of “included: 0” or “not included: 1”. The evaluation means 13 generates several values in internal processing for determining the inclusion flag. For example, the evaluation means 13 uses a score expressed as a continuous value as an internal parameter in order to obtain a determination result of "includes: 0" or "does not include: 1". If the score is smaller than a predetermined numerical value, the evaluation means 13 determines that "includes: 0". If the score is larger than a predetermined numerical value, the evaluation means 13 determines that "not inclusive: 1". The evaluation means 13 may output scored operational data by assigning a score to the sample in addition to the inclusion flag indicated by "0" or "1". Then, the calculation means 14 may obtain the update determination index value θ1 using the score.

There are no particular restrictions on the operational data C. For example, the operational data C may be data obtained in one day (24 hours). The operational data C may be data obtained in one week (24 hours x 7 days).

1 Machine learning model updating device 11 Input means 12 Learning model generation means (learning model updating means)
13 Evaluation means 14 Calculation means 15 Update determination means 16 Learning data updating means 17 Updating means C Operational data D Learning data M Learning model R Inference result

Claims (8)

an input means for receiving operational data;
an evaluation means for evaluating whether the elements constituting the operational data are included in the first learning data set used for learning the first learning model;
calculation means for obtaining an update determination index value for determining whether to update the first learning model to a second learning model using the results of the evaluation means;
A machine learning model updating device comprising: updating means for updating the first learning model to the second learning model when the update determination index value satisfies an update condition. The updating means includes:
update determination means for determining whether or not to update the first learning model to the second learning model using the update determination index value;
learning data updating means for updating the first learning data to second learning data using the operational data;
The machine learning model updating device according to claim 1, further comprising a learning model updating unit that updates the first learning model to the second learning model using the second learning data. The evaluation means evaluates whether or not each element is included in the first learning data set, and if the element is not included in the first learning data set, the first The machine learning model updating device according to claim 1, wherein a flag indicating that the element is not included in a set of learning data is attached to the element. The machine learning model updating device according to claim 1, wherein the evaluation means evaluates whether or not the element is contained based on a distance from the first learning data set to the element. The machine learning model updating device according to claim 3, wherein the calculation means obtains, as the update determination index value, a ratio of the number of the flagged elements to the total number of the elements included in the operational data. The update condition is a threshold,
The machine learning model updating device according to claim 2, wherein the update determination means determines to update the first learning model to the second learning model when the update determination index value exceeds the threshold value. a step of receiving operational data;
a step of evaluating whether or not elements constituting the operational data are included in a first learning data set used for learning the first learning model;
obtaining an update determination index value for determining whether to update the first learning model to a second learning model using the result of the evaluation;
A machine learning model updating method, comprising: updating the first learning model to the second learning model when the update determination index value satisfies an update condition. a step of receiving operational data;
a step of evaluating whether or not elements constituting the operational data are included in a first learning data set used for learning the first learning model;
obtaining an update determination index value for determining whether to update the first learning model to a second learning model using the result of the evaluation;
A machine learning model update program that causes a computer to execute the step of updating the first learning model to the second learning model when the update determination index value satisfies an update condition.

Images