JP7307215B2 - Operation support system and method - Google Patents

Description

The present invention relates generally to assisting in the operation of learning models.

From the viewpoint of cost reduction or operational efficiency improvement, awareness of operational reform using AI (for example, machine learning) is increasing in various fields. In particular, in the product inspection process, there is also the problem that it is becoming difficult to stabilize inspection quality, train inspectors, and secure skilled personnel. Therefore, expectations are growing for automation of the inspection process by AI.

In order to automate the inspection process, it is conceivable to operate a learning model created by machine learning.

However, learning models can deteriorate. Patent Literature 1 discloses a prediction model as an example of a learning model. The prediction model maintenance system disclosed in Patent Literature 1 creates a new prediction model and selects the prediction model and the new prediction model, whichever has the higher evaluation, as the operational prediction model.

Japanese Patent Application Laid-Open No. 2019-87101

As a result of earnestly studying the operation of a learning model for automating the product inspection process, the inventor of the present application has obtained the following findings.

A learning model to which product inspection data is input performs an inspection of the product using the inspection data, and outputs inspection result data representing the result of the inspection (for example, whether the product is normal or abnormal). . If the learning model in operation is degraded, inspection result data representing a defective product (abnormal product) is output even though inspection data representing a non-defective product (normal product) is input. There may be a case and a second case in which inspection result data representing a non-defective product is output in spite of input of inspection data representing a defective product.

When the first case occurs, it is possible that the number of non-shipped non-defective products continues to increase.

When the second case occurs, even if the shipment of defective products can be avoided by the post-stage inspection (for example, the final inspection immediately before shipping), which is the inspection after the product inspection by the learning model, the product inspection by the learning model It is possible that defective products will continue to be manufactured until the post-stage inspection is performed.

Such problems, that is, output data indicating that the target is abnormal despite inputting input data indicating that the target is normal, or input data indicating that the target is abnormal The problem that the output data indicating that the target is normal is output even though the is input, that is, the content of the input data and the content of the output data are different Continuing problems can also arise when learning models are used in fields other than the manufacturing inspection process.

The prediction model maintenance system disclosed in Patent Document 1 is a system that is separated from the operating environment of the prediction model, and evaluates the prediction model at timing that does not depend on the certainty of the prediction model. Therefore, the above problem cannot be solved.

The system monitors the certainty of the learning model in operation in real time (for example, each time output data is output). Specifically, for example, the system performs relearning, which is to create one or a plurality of learning models with different versions, and also performs operation monitoring, which is monitoring of the learning model during operation. In operation monitoring, the system performs a first confidence comparison to determine whether the confidence of the learning model is less than a first threshold each time a learning model in operation to which input data is input outputs output data. . If the result of the first confidence comparison is true, the system selects the learning model to be operated as one or more candidate learning models (one or more learning models of a version different from the version of the learning model in operation). Among them, the learning model in operation is replaced with one of the candidate learning models having higher certainty than the learning model in operation from which the true result was obtained.

It is possible to reduce the continuous occurrence of differences between the content represented by the input data and the content represented by the output data for the target.

1 is a diagram showing an overview of an operation support system according to Embodiment 1; FIG. It is a figure which shows the structure of an operation-monitoring apparatus. FIG. 4 is a diagram showing the structure of a model management table; FIG. 4 is a diagram showing the configuration of an operation management table; FIG. FIG. 4 is a diagram showing the configuration of a teaching data management table; FIG. FIG. 4 is a diagram showing multiple phases in Embodiment 1; It is a figure which shows the flow of the background processing which a learning part performs. FIG. 10 is a diagram showing a flow of re-learning processing; It is a figure which shows the flow of the process which a monitoring part performs. FIG. 10 is a diagram showing part of the flow of processing performed by the operation monitoring device according to the second embodiment; It is explanatory drawing of the technical significance of _Th1 and _Th2 . FIG. 12 is a diagram showing part of the flow of processing performed by the operation monitoring device according to the third embodiment; FIG. 12 is a diagram showing an overview of an operation support system according to Embodiment 4;

In the following description, an "interface device" may be one or more interface devices. The one or more interface devices may be at least one of the following:
- One or more I/O (Input/Output) interface devices. An I/O (Input/Output) interface device is an interface device for at least one of an I/O device and a remote display computer. The I/O interface device to the display computer may be a communications interface device. The at least one I/O device may be any of a user interface device, eg, an input device such as a keyboard and pointing device, and an output device such as a display device.
- One or more communication interface devices. The one or more communication interface devices may be one or more of the same type of communication interface device (e.g., one or more NICs (Network Interface Cards)) or two or more different types of communication interface devices (e.g., NIC and It may be an HBA (Host Bus Adapter).

Also, in the following description, "memory" may be one or more memory devices, typically a main memory device. At least one memory device in the memory may be a volatile memory device or a non-volatile memory device.

Also, in the following description, a "persistent storage device" is one or more persistent storage devices. A permanent storage device is typically a non-volatile storage device (for example, an auxiliary storage device), and more specifically, for example, an HDD (Hard Disk Drive) or SSD (Solid State Drive).

Also, in the following description, "storage" may be at least memory of memory and persistent storage.

Also, in the following description, a "processor" is one or more processor devices. The at least one processor device is typically a microprocessor device such as a CPU (Central Processing Unit), but may be another type of processor device such as a GPU (Graphics Processing Unit). At least one processor device may be single-core or multi-core. At least one processor device may be a processor core. At least one processor device may be a broadly defined processor device such as a hardware circuit (for example, FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit)) that performs part or all of processing.

In the following description, the expression "xxx table" may be used to describe information that provides an output for an input. A learning model such as a neural network that generates Therefore, the "xxx table" can be called "xxx information". Also, in the following description, the configuration of each table is an example, and one table may be divided into two or more tables, or all or part of two or more tables may be one table. may

In addition, in the following description, the function may be described using the expression “kkk unit”, but the function may be realized by executing one or more computer programs by a processor, or may be realized by executing one or more computer programs. It may be realized by the above hardware circuits (FPGA or ASIC, for example). When a function is realized by executing a program by a processor, the defined processing is performed using a storage device and/or an interface device as appropriate, so the function may be at least part of the processor. good. A process described with a function as the subject may be a process performed by a processor or a device having the processor. Programs may be installed from program sources. The program source may be, for example, a program distribution computer or a computer-readable recording medium (for example, a non-temporary recording medium). The description of each function is an example, and multiple functions may be combined into one function, or one function may be divided into multiple functions.

Also, in the following description, the "operation support system" may be one or more physical computers, or a software-defined system implemented by at least one physical computer executing predetermined software. Alternatively, it may be a system implemented on a computing resource group (for example, a cloud platform) that is a plurality of computing resources. The operation support system may include at least one of an edge device that transmits data and a core device that receives data.

Also, in the following description, the "ID" of an element is identification information, and may be information expressed in alphanumeric characters or the like, or alternatively or additionally may include the name of the element.

Also, in the following description, the unit of time is year, month, day, but the unit of time may be coarser or finer.

In addition, in the following description, common parts of the reference numerals are used when similar elements are described without distinction, and reference numerals are used when similar elements are described separately. . For example, when the edge devices are not distinguished, the term "edge device 92" is used, and when the edge devices are distinguished, the term "edge device 92A" and "edge device 92B" are used.

Also, in the following description, "learning model" is abbreviated as "model". In the following embodiments, a neural network created by deep learning is used as a model, but other types of models, such as an autoregressive moving average model (for example, ARIMA (Autoregressive Integrated Moving Average) model) may be adopted.

Also, in the following embodiments, a product inspection process is taken as an example. For this reason, the input data input to the learning model in operation is product inspection data (for example, data having measurement data including one or more measured values respectively measured for one or more measurement items of a product). ). The output data output from the learning model is inspection result data representing the result of processing using the inspection data (for example, the result of whether the product is normal or abnormal). The present invention may be applied to fields other than product inspection processes. In addition, "product" generally means a manufactured item, that is, a so-called finished product. can mean Therefore, in the following description, for example, "products" include both items before being put into the manufacturing system and items in the manufacturing system (so-called "semi-finished products"), and all relevant processes in the manufacturing system. It may be any finished product that has undergone a process and is ready for shipment.
[Embodiment 1]

FIG. 1 is a diagram showing an outline of an operation support system according to Embodiment 1. FIG.

The operation support system 50 performs operation monitoring to monitor the reliability of the model 79 in operation in real time (for example, each time inspection result data is output). Therefore, the operation support system 50 can immediately detect deterioration of the operating model 79 (when the certainty factor of the operating model 79 falls below the threshold value). When the deterioration of the model 79 is detected, the operation support system 50 takes countermeasures such as replacing the model 79 with another version of the model with a higher degree of certainty. As a result, it is possible to prevent the content (for example, normal or abnormal) represented by the inspection data of the product from continuing to differ from the content (for example, normal or abnormal) represented by the inspection result data of the product.

The operation support system 50 includes an annotation section 51 , a preprocessing section 53 , a learning section 55 , an operation section 57 , a monitoring section 59 , a display section 71 , an annotation management table 140 and a model management table 110 .

The annotation unit 51 receives annotation data representing annotations (for example, definitions) for data input to the model, and registers the annotation data in the annotation management table 140 . Annotations may be based, for example, on the knowledge of those involved in product inspection. The annotation may be at least one of a non-defective condition that is a condition of inspection data representing a non-defective product (normal product) and an abnormal condition that is a condition of inspection data representing a defective product (abnormal product). Meeting the non-defective condition may be meeting the non-defective condition itself or not meeting the abnormal condition.

The pre-processing unit 53 transfers inspection data input from a data source (not shown) such as an edge device such as a product inspection device to the operation unit 57, and the annotation management table 140 determines whether the inspection data is a non-defective product. If the conditions are met, the inspection data is input to the learning unit 55 . Inputting inspection data corresponding to non-defective product conditions to the learning unit 55 means that the data management table (at least one of a teacher data management table and an inspection data management table, which will be described later) that the learning unit 55 refers to for creating a model. You can register.

The learning unit 55 creates a new model or re-learns the model (creates one or more learning models of a version different from the version of the model). The learning unit 55 registers the created model in the model management table 110 .

The operation unit 57 operates any model registered in the model management table 110 . The model can be operated by the operation department 57 by deploying it to the operation department 57 . Operation unit 57 may be called an inference unit.

The monitoring unit 59 performs operation monitoring, which is monitoring of the model 79 in operation, in real time (for example, each time inspection result data is output, or all the time). The monitoring unit 59 includes a certainty comparison unit 61 , a factor determination unit 63 and a model replacement unit 65 .

The certainty comparison unit 61 performs a certainty comparison to determine whether the certainty of the model is less than a predetermined threshold each time the operating model 79 to which inspection data is input outputs inspection result data. Further, the certainty comparison unit 61 compares the certainty of the model 79 in operation with the certainty of the replacement candidate model of the model 79 . In addition, the certainty comparison unit 61, for the same model, the certainty of the model when the inspection data input for creating the model is input, and the inspection data newer than the inspection data are input. Then, compare with the confidence of the model in the case of

The factor determination unit 63 determines whether the factor that causes the certainty factor of the model 79 in operation to be less than a predetermined threshold is the data input to the model or the model itself.

The model replacement unit 65 replaces the model to be operated from the model 79 in operation with a model of a version different from the version of the model 79 .

The display unit 71 displays information about the result of operation monitoring (for example, information including information indicating the current confidence of the model 79 in operation and the confidence of the replacement candidate model) and information about the result of re-learning. (For example, information representing each version of one or more models obtained for the model 79 in operation) and information representing annotations are displayed on the computer 81 communicably connected to the operation support system 50. Send to. These information may be at least some of the information represented by at least one of tables 110-150 (see FIG. 2) and information resulting from various operations.

The present embodiment will be described in detail below.

FIG. 2 is a diagram showing the configuration of the operation support device.

With respect to the manufacturing inspection process, there is generally provided a plurality of edge devices each transmitting inspection data and a core device receiving inspection data from each of the plurality of edge devices. At least one edge device may be a sensor device that measures a predetermined measurement item regarding a product and transmits inspection data including the measured values, or a relay device that receives inspection data from one or more sensor devices (e.g., an edge device). server or gateway). The relay device may transfer the received inspection data as it is to the core device, or process the inspection data (for example, create inspection data as a summary of inspection data for a certain period of time) and send the processed inspection data. It may be transferred to the core device.

The operation support device 90 is an example of a core device and corresponds to the operation support system 50 itself according to this embodiment. The operation support device 90 may be a computer system having the interface device 10, the storage device 20, and the processor 30, or may be an annotation unit 51, a preprocessing unit 53, and a learning unit 55 realized on a computational resource group including those computational resources. , the operation unit 57, the monitoring unit 59, and the display unit 71, at least the learning unit 55 and the monitoring unit 59 may be provided.

The interface device 10 is a device for communicating with an external device via a network (for example, the Internet). External devices include, for example, a client computer such as the computer 81 shown in FIG. 1 and the above-described edge device.

The storage device 20 stores management data and one or more programs (not shown). The management data includes, for example, a model management table 110, an operation management table 120, a teacher data management table 130, an annotation management table 140, and an inspection data management table 150.

One or more programs in the storage device 20 are executed by the processor 30, thereby realizing functions such as the annotation section 51, the preprocessing section 53, the learning section 55, the operation section 57, the monitoring section 59, and the display section 71. be done.

FIG. 3 is a diagram showing the structure of the model management table 110. As shown in FIG.

The model management table 110 holds information representing models. Each record in the model management table 110 holds information such as model ID 111, version 112, learning start 113, learning end 114, product ID 115, product name 116, and certainty 117, for example. Records correspond to models. In the following, one record is taken as an example (in the description of FIG. 3, the model corresponding to the record is called "attention model").

The model ID 111 represents the ID of the noted model. Version 112 represents the version of the model of interest. A model is identified from a set of model ID 111 and version 112 . Learning start 113 represents the start date of learning of the model of interest. A learning end 114 represents the end date of the learning of the model of interest. Product ID 115 represents the ID of the product inspected by the model of interest. Product name 116 represents the name of the product inspected by the model of interest. The certainty factor 117 represents the certainty factor of the attention model at the end of learning.

FIG. 4 is a diagram showing the configuration of the operation management table 120. As shown in FIG.

The operation management table 120 holds information on model operation. Each record in the operation management table 120 holds information such as model ID 121, version 122, deployment date 123, activation 124, and number of inspection results 125, for example. Records correspond to models. In the following, one record is taken as an example (in the description of FIG. 4, the model corresponding to the record is called "attention model").

The model ID 121 represents the ID of the noted model. Version 122 represents the version of the model of interest. The deployment date 123 represents the date when the model of interest was deployed to the operation unit 57 . Activation 124 indicates whether the model of interest is a model in operation. "1" means in operation, and "0" means not in operation. The number of inspection results 125 represents the number of products inspected by the model of interest.

FIG. 5 is a diagram showing the configuration of the teacher data management table 130. As shown in FIG.

The teacher data management table 130 holds information about teacher data. Each record in the teaching data management table 130 holds information such as product ID 131, product name 132, manufacturing number 133, performance 134, and measurement data 135, for example. A record corresponds to training data (for example, it may be training data itself). In the following, one record is taken as an example (in the explanation of FIG. 5, the teacher data corresponding to the record is called "notable teacher data"). Note that the “teaching data” may be data including inspection data, and more specifically, may be inspection data prepared for model learning.

All of the information 131 to 135 are information possessed by the noted teacher data. Information other than the track record 134 in the noted teacher data may be information included in the inspection data. In other words, the attention teacher data may be data including the performance 134 and inspection data. Product ID 131 represents the ID of the product. The product name 132 represents the name of the product. The serial number 133 represents the serial number of the product. Result 134 indicates whether the product is good or bad. Metrology data 135 includes one or more measurements about the product. In the present embodiment, for example, the teacher data whose track record 134 is “defective product” does not correspond to annotations represented by the annotation management table 140, and as a result, is excluded from being input to the learning unit 55 by the preprocessing unit 53. you can

Although not shown, the configuration of the inspection data management table 150 may be the same as the configuration of the teacher data management table 130 . The inspection data management table 150 holds information about inspection data received by the operation support device 90 during operation. A record in the inspection data management table 150 may be inspection data itself.

An example of the processing performed in this embodiment will be described below.

FIG. 6 is a diagram showing multiple phases in this embodiment. In the following description, inspection data obtained in period X (the X-th period) is referred to as "inspection data X".

It can be roughly divided into three phases, a pre-learning phase, an operation phase and a re-learning phase.

The pre-learning phase is a phase in which a model with a new model ID is created (learned). In the pre-learning phase, the learning unit 55 inputs the inspection data T to the model to be learned, and the track record of the teacher data including the inspection data T matches the inspection result data as a result of processing the inspection data. The model is trained so that When the model has been learned, information representing the model is registered in each of the model management table 110 and the operation management table 120 . The model is deployed to the operation department 57 . The created model has model ID "M100" and version "Ver.1". Hereinafter, the model with model ID "P" may be described as "model P", and the model P with version "Q" may be described as "model P/Q". Also, the certainty of version "Ver.K" may be described as "C _K ". In the example of FIG. 6, _C1 is "90%".

The operational phase is the phase in which the deployed model is operational (that is, the inference phase). In the operation phase, the operation department 57 uses the deployed model M100/Ver. 1 is operated. Specifically, for example, the period (T+1) next to the period T is at least part of the period in operation, and the inspection data (T+1) is the model M100/Ver. 1 and outputs inspection result data as a result of processing inspection data (T+1).

The relearning phase is a phase in which a model with the same model ID and a different version is created. The re-learning phase may be started periodically, or may be started according to the relationship between the confidence of the model in operation and the threshold. In the re-learning phase, the learning unit 55 converts the inspection data (T+1) (or teacher data prepared for re-learning) into the model M100/Ver. 1 (or another model with the model ID "M100"), the model M100/Ver. Relearn 1. As a model after re-learning, model M100/Ver. 2 (that is, a model with the same model ID as model M100/Ver.1 and a different version). Information representing the model is registered in each of the model management table 110 and the operation management table 120 . In the example of FIG. 6, _C2 is "95%".

The monitoring unit 59 detects the model M100/Ver. 1 is being monitored. The monitoring unit 59 monitors the model M100/Ver. In the example of FIG. 6, it is determined whether or not _C1 is less than Th (threshold value) each time inspection result data is output from 1. If a true result is obtained as a result of this determination, the monitoring unit 59 converts the model M100 to be operated to the model M100/Ver. ₁ to model _M100 /Ver. Replace with 2.

FIG. 7 is a diagram showing a flow of background processing performed by the learning unit 55. As shown in FIG.

Background processing is processing performed in the background of model operation, and is processing for the model created in the pre-learning phase. In the following description, the model M100 is taken as an example of the model.

The learning unit 55 determines whether or not the conditions for starting the re-learning process for the model M100 are met (S701). Any condition may be employed as the start condition. For example, in the present embodiment, the start condition is that a certain period of time has elapsed since the most recent relearning process for model M100 was completed. That is, in the present embodiment, the relearning process is periodically performed in the background of model operation.

If the determination result of S701 is true (S701: Yes), the learning unit 55 performs re-learning processing of the model M100 (S702). For example, in one re-learning process, the learning unit 55 creates one model of a different version from the created version for the model M100, and stores information representing the created model in the model management table 110 (and , operation management table 120).

By repeating the re-learning process for the model M100, a model of a version different from the version of the created model (the model registered in the model management table 110) is created.

FIG. 8 is a diagram showing the flow of the re-learning process (S702).

Based on the annotation management table 140, the learning unit 55 determines whether inspection data (in the description of FIG. 8, inspection data of interest) as one of input candidates for re-learning is correct data (S801). . An example of "whether or not the inspection data of interest is correct data" is whether or not the inspection data of interest corresponds to the non-defective product definition represented by the annotation of the product to be inspected of the model M100. If the determination result of S801 is true (S801: Yes), S805 is performed.

If the determination result of S801 is false (S801: No), the learning unit 55 determines whether or not the inspection data of interest is unknown data (S802). An example of "unknown data" may be inspection data actually obtained during operation, rather than inspection data as teacher data (which may be called training data).

If the determination result of S802 is false (S802: No), it means that there is inappropriate test data, so the learning unit 55 performs reporting processing (S803). In this reporting process, the learning unit 55 may transmit information indicating that inappropriate inspection data (for example, nonstandard data) has occurred to the person in charge of the manufacturing line of the product represented by the inspection data of interest. .

If the determination result of S802 is true (S802: Yes), the learning unit 55 determines whether or not the inspection data of interest conforms to a predetermined rule (rule represented by the annotation) based on the annotation management table 140 ( S804). If the determination result of S804 is false (S804: No), S803 is performed.

If the determination result of S804 is true (S804: Yes), the learning unit 55 re-learns the model M100 by inputting the inspection data of interest to the model M100 of a certain version (for example, the version in operation) ( S805).

After S805, the learning unit 55 determines whether or not the inspection data of interest is the last data as an input candidate (S806). If the determination result of S806 is false (S806: No), the learning unit 55 selects another inspection data and performs S801 for the other inspection data. If the determination result in S806 is true (S806: Yes), a new version of the model M100 has been completed for the model M100. 110 (and the operation management table 120).

In the re-learning process, only data determined to be suitable for re-learning the model M100, such as inspection data corresponding to annotations, is used for re-learning. Therefore, it is possible to maintain the reliability of the model M100. Note that S801 to S803 may be performed by the pre-processing unit 53 instead of the learning unit 55. FIG.

FIG. 9 is a diagram showing the flow of processing performed by the monitoring unit 59. As shown in FIG.

Assume that the version of the model M100 in operation is (N-1). Each time inspection result data is output from the operating model M100/(N-1) to which inspection data is input, the certainty comparison unit 61 determines whether C _(N-1) < Th ( S901). Th is a threshold.

If the determination result of S901 is false (S901: No), the certainty comparison unit 61 refers to the operation management table 120 and checks whether a certain period of time has passed since the deployment date 123 of the model M100/(N−1). Determine (S902). If the determination result of S902 is true (S902: Yes), the certainty comparison unit 61 refers to the model management table 110 and the operation management table 120, and the activation 124 is "0" (that is, not in operation). ) Select model M100/N (S905).

If the determination result of S901 is true (S901: Yes), the factor determination unit 63 determines whether concept drift occurs in the input inspection data (S903). Determination of “whether or not concept drift occurs” is an example of determination of whether or not the factor of C _(N-1) < Th is data. An example of concept drift is that the average value of the measurement values represented by the inspection data has changed by a certain value or more. If the determination result of S903 is true (S903: Yes), the factor determination unit 63 performs reporting processing (S904). In this reporting process, the factor determination unit 63 may transmit information indicating that a concept drift has occurred to the person in charge of the manufacturing line of the product represented by the inspection data input to the model in operation.

If the determination result of S903 is false (S903: No), the certainty comparison unit 61 performs S905, that is, selects model M100/N.

The certainty comparison unit 61 determines whether or not C _N >C _(N−1) (S906). C _N is the confidence of model M100/N and C _(N-1) is the confidence of model M100/(N-1). C _N and C _(N−1) may be values represented by the confidence factor 117 of the model management table 110 . Alternatively, the certainty comparison unit 61 inputs inspection data to each of the model M100/N and the model M100/(N−1), and compares the certainty of the model M100/N and the model M100/(N -1) may obtain C _N and C _(N-1) . Also, S906 and S905 may be integrated. That is, a higher confidence version of model M100 than C _(N-1) is selected in S905, and S906 may be omitted.

If the determination result of S906 is false (S906: No), the certainty comparison unit 61 increments N by 1 (S908) and performs S905.

If the determination result of S906 is true (S906: Yes), the certainty comparison unit 61 inputs each of the inspection data T and the inspection data (T+1) to the model M100/N, and _{CN:T≤CN :} It is determined whether or not _(T+1) (S907). CN _:T is the confidence factor of model M100/N when inspection data T is input. _{CN: (T+1)} is the confidence of model M100/N when inspection data (T+1) is input. Inspection data T is an example of old inspection data, for example, inspection data used in pre-learning of model M100. The inspection data (T+1) is an example of inspection data newer than the inspection data T, and is, for example, inspection data received during operation of the model M100. If the determination result of S907 is false (S907: No), the certainty comparison unit 61 performs S908.

If the determination result of S907 is true (S907: Yes), the model replacement unit 65 deploys the model M100/N (S909). That is, the model replacement unit 65 replaces the model M001 to be operated from the model M100/(N-1) with the model M100/N.
[Embodiment 2]

A second embodiment will be described. At that time, the points of difference from the first embodiment will be mainly explained, and the explanations of the points in common with the first embodiment will be omitted or simplified.

FIG. 10 is a diagram illustrating part of the flow of processing performed by the operation monitoring device according to the second embodiment;

In this embodiment, the condition for starting the relearning process is that C _(N-1) (certainty of model M100/(N-1)) is smaller than Th ₂ (second threshold). In other words, in the first embodiment, the processing related to the relearning processing is background processing, whereas in the present embodiment it can be said to be real-time processing.

The certainty comparison unit 61 determines whether or not C _(N-1) < Th ₂ (S1001). If the determination result of S1001 is true (S1001: Yes), the learning unit 55 performs re-learning processing of the model M100 (S1002).

After that, the certainty comparison unit 61 determines whether or not C _(N−1) <Th ₁ (S1003). Th ₁ is the first threshold. The technical significance of _Th1 and _Th2 will be described later.

If the determination result of S1003 is false (S1003: No), S1002 is performed.

If the determination result of S1003 is true (S1003: Yes), the processes after S903 in FIG. 9 are performed.

FIG. 11 is an explanatory diagram of the technical significance of _Th1 and _Th2 .

For the model M100, Th ₁ and Th ₂ are set based on Th _B (reference confidence factor) determined based on the past confidence factor distribution (or other information). For example, the certainty comparison unit 61 may periodically calculate Th _B and change at least one of Th ₁ and Th ₂ based on the calculated Th _B.

Th ₁ includes Th _1U , which means Th ₁ as upper limit, and Th _1D , which means Th ₁ as lower limit. Th ₂ includes Th _2U , which means Th ₂ as upper limit, and Th _2D , which means Th ₂ as lower limit. Th _2U <Th _1U and Th _2D >Th _1D . That is, _Th2 is closer to _ThB than _Th1 . The confidence usually varies in the confidence range from Th _2U to Th _2D . When the certainty of the model M100/(N−1) in operation exceeds (falls below or exceeds) _Th2 , the learning unit 55 prepares the model M100/N by re-learning the model M100. . When the certainty factor of the model M100/(N-1) in operation exceeds (falls below or exceeds) _Th1 , the model replacement unit 65 deploys the model M100/N.

As described above, with Th ₁ and Th ₂ , model preparation and replacement can be performed at appropriate timing.
[Embodiment 3]

A third embodiment will be described. At that time, differences from Embodiments 1 and 2 will be mainly described, and descriptions of common points with Embodiments 1 and 2 will be omitted or simplified.

12 is a diagram illustrating part of the flow of processing performed by the operation monitoring device according to the third embodiment; FIG.

The certainty comparison unit 61 determines whether or not C _(N-1) < Th ₂ (S1201).

If the determination result of S1201 is true (S1201: Yes), model evaluation, that is, S905 to S907: Yes is performed.

Thereafter, the certainty comparison unit 61 determines whether or not C _(N−1) <Th ₁ (S1202).

If the determination result of S1202 is true (S1202: Yes), the factor determination unit 63 determines whether concept drift has occurred (S1203). If the determination result of S1203 is true (S1203: Yes), the factor determination unit 63 performs reporting processing (S1205).

If the determination result of S1203 is false (S1203: No), the model replacement unit 65 deploys the model M100/N (S1204).
[Embodiment 4]

FIG. 13 is a diagram showing an overview of an operation support system according to the fourth embodiment.

In Embodiments 1 to 3, functions such as the annotation unit 51, the preprocessing unit 53, the learning unit 55, the operation unit 57, the monitoring unit 59, and the display unit 71 are integrated into an operation support device that is an example of a core device. , these functions may be centralized in the edge device instead of the core device, or may be distributed between the edge device and the core device. That is, the operation support system may be one or both of an edge device that transmits data and a core device that receives data.

According to the example shown in FIG. 13, there are a plurality of edge devices 92A, 92B, . An edge device 92 (for example, 92A) has an annotation unit 51, a preprocessing unit 53, a learning unit 55, an operation unit 57, a monitoring unit 59 and a display unit 71, and has a teacher data management table 130, an annotation management table 140 and The inspection data management table 150 is stored. A core device 96 provides a plurality of storage spaces 98A, 98B, . . . respectively corresponding to a plurality of edge devices 92A, 92B, . A storage space 98 (for example, 98A) stores a model management table 110 and an operation management table 120 corresponding to the edge device 92 corresponding to the storage space 98 . Information representing the model created by re-learning in the edge device 92 is registered in the tables 110 and 120 in the storage space 98 corresponding to the edge device 92 . The model itself may also be stored in storage space 98 .

The above description of Embodiments 1 to 4 can be summarized, for example, as follows.

The operation support system 50 monitors the reliability of the model M100/(N−1) in operation in real time. Specifically, the operation support system 50 includes a learning unit 55 that performs re-learning by creating one or a plurality of models with different versions, and a monitoring unit 59 that performs operation monitoring, which is monitoring of models in operation. and A monitoring unit 59 includes a certainty comparison unit 61 and a model replacement unit 65 . Each time the model M100/(N−1) in operation to which inspection data (an example of input data) is input, the certainty comparison unit 61 outputs inspection result data (an example of output data), C _{(N−1 )} <Th Make a first confidence comparison whether or not ₁ . When C _(N−1) <Th ₁ , the model replacement unit 65 performs model replacement, that is, replaces the model M100 to be operated with one or more candidate models (the model M100 in operation/(N -Among one or more models M100) of a version different from the version of -1), any candidate model M100 that has a higher confidence than the model M100/(N-1) in operation for which the true result was obtained /N from the model M100/(N-1) in operation. As a result, it is possible to reduce the fact that the content represented by the inspection data (eg, good or defective) and the content represented by the inspection result data (eg, good or defective) are different from each other for the product (an example of the target). Note that “C _(N-1) < Th ₁ ” means that C _(N-1) exceeds Th 1 depending on the type of Th ₁ (for example _, depending on whether Th ₁ is the upper limit or the lower limit). or falling below.

The operation support system 50 may include the factor determination section 63 . When C _(N−1) <Th ₁ , the factor determining unit 63 determines that the inspection data input to the model M100/(N−1) in operation from which the result was obtained is the factor of the result. A factor determination may be made as to whether or not If the result of the factor determination is false, the model replacement unit 65 may perform the above-described model replacement. During the operation of the model M100/(N-1), inappropriate inspection data (for example, data with different features (such as trends) from the inspection data used in learning to create the model M100/(N-1)) Operation support system 50 may receive. C _(N−1) may be less than Th ₁ due to inappropriate inspection data. Therefore, it is determined whether or not the factor of C _(N-1) < Th ₁ is data, and if it is specified that the factor is not data, in other words, the model M100/(N-1) Model substitution occurs when one is identified. This makes it possible to avoid unnecessary model replacement. The factor determination is based on the inspection data input to the model M100/(N-1) in operation for which C _(N-1) < Th ₁ was obtained, and whether or not the concept drift was identified. good. It can be expected to avoid the continuation of inspection data entering the model M100/(N−1) after concept drift has occurred.

Each time the operating model M100/(N−1) to which the inspection data is input outputs inspection result data, the certainty comparison unit 61 performs a second check of whether C _(N−1) < Th ₂ . Confidence comparisons may be performed. When C _(N-1) < Th ₂ , the certainty comparison unit 61 inputs evaluation data to the candidate model M100/N (an example of at least one candidate model among one or more candidate models). Then, model evaluation may be performed to evaluate the candidate model M100/N. That is, _CN may be evaluated between model M100/N selection and deployment. As a result, the reliability of the model M100/N after replacement is expected. In the model evaluation, for example, the certainty comparison unit 61 selects a model M100/N (an example of one or more candidate models) and inputs inspection data (an example of evaluation data) to the candidate model M100/N. Thus, _CN may be evaluated. If C _(N-1) < Th ₁ and the result of the factor determination is false, in the model replacement, the model after replacement has a higher confidence than C _(N-1) in model evaluation It may be the candidate model M100/N for which _CN was obtained. The inspection data used in the model evaluation may be the inspection data T used to create the candidate model M100/N and the inspection data (T+1) newer than the inspection data T used. A candidate model M100/N may be a model such that _CN:T ≤ _CN:(T+1) . Thus, the model after replacement is expected to be a model suitable for recent feature inspection data. _CN:T is the degree of confidence when the inspection data T used to create the model M100/N is input to the model M100/N. C _{N: (T+1)} is the degree of confidence when inspection data (T+1) newer than inspection data T is input to the model M100/N.

The learning unit 55 may perform re-learning periodically or irregularly asynchronously with operation monitoring. As a result, it is possible to increase models of different versions, that is, prepare for model replacement without depending on the results of operation monitoring.

When C _(N-1) < Th ₂ , the learning unit 55 may perform re-learning to create a different version of the model M100/(N-1) from the model M100/(N-1) in operation. As a result, it is possible to start the preparatory process for creating candidates for models after replacement at an appropriate timing when C _(N−1) <Th ₂ is specified.

The operation support system 50 may include a preprocessor 53 . The pre-processing unit 53 inputs inspection data corresponding to data conditions (for example, an example of conditions represented by annotations) indicating that the product is non-defective into the model, and does not input inspection data that does not meet the conditions into the model. OK. During the operation of the product inspection process, usually more inspection data representing non-defective products is obtained than inspection data representing defective products. Therefore, it is expected that the accuracy of the model will be improved by inputting only the inspection data representing non-defective products. It should be noted that the condition of the data indicating that the product is non-defective may be an example of a predetermined condition regarding the data indicating the object. Predetermined conditions, in place of or in addition to data conditions indicating that the product is non-defective, may include other conditions according to the field or purpose to which the operation support system is applied (e.g., that the product is defective). conditions for the data to be represented).

The model replacement unit 65 may select the model M100/N and perform model evaluation. If C _(N-1) < Th ₁ and the result of the factor determination is false, then the reliability of the model M100/N after replacement is expected. Note that the model M100/N after replacement may be a model satisfying _CN:T ≤ _CN:(T+1) . Thus, the model after replacement is expected to be a model suitable for recent feature inspection data.

Although several embodiments have been described above, these are examples for explaining the present invention, and are not intended to limit the scope of the present invention only to these embodiments. The invention can also be implemented in various other forms. For example, as described above, the present invention is applicable to fields other than manufacturing inspection processes. For example, the present invention can be applied to the field of marketing. In this case, an example of input data may be data representing sales performance, an example of a model may be a demand forecast model, and an example of output data may be a required number of purchases. data that represents

50 Operation support system

Claims (18)

A learning unit that performs re-learning by creating one or more learning models with different versions;
a monitoring unit that monitors the operation of the learning model during operation ;
factor determination unit and
display and
with
The monitoring unit
a certainty comparison unit that performs a first certainty comparison to determine whether the certainty of the learning model in operation is less than a first threshold;
When the result of the first confidence comparison is true, the learning model to be operated is one or more candidate learning models whose version is different from the version of the learning model in operation. A model replacement unit that replaces the learning model in operation with one of the candidate learning models that has a higher certainty than the learning model in operation from which the true result was obtained. and
The factor determining unit, when the result of the first certainty comparison is true, determines whether the input data input to the learning model in operation from which the true result is obtained is the factor of the result. or factor determination,
When the result of the factor determination is false, the display unit displays the confidence of the candidate learning model to be replaced by the model replacement unit;
Operation support system. The certainty comparison unit performs a second certainty comparison as to whether the certainty of the learning model in operation is less than a second threshold closer to the reference certainty than the first threshold,
When the result of the second confidence comparison is true, the confidence comparison unit inputs data for evaluation of at least one of the one or more candidate learning models to the candidate learning model. Perform model evaluation, which is to evaluate by
When the result of the first confidence comparison is true and the result of the factor determination is false, in the model replacement, one of the candidate learning models is the learning model in operation in the model evaluation is a candidate learning model with a higher confidence than the confidence of
The operation support system according to claim 1 . The data for evaluation is input data newer than the input data used to create the at least one candidate learning model and the input data;
For any of the candidate learning models, the confidence when input data newer than the input data used to create the candidate learning model is input to the candidate learning model is used to create the candidate learning model. is a candidate learning model with the same or higher confidence when the input data is input to the candidate learning model,
The operation support system according to claim 2 . The factor determination is a determination of whether or not a concept drift has been identified based on the input data input to the learning model in operation from which a true result was obtained as a result of the first confidence comparison.
The operation support system according to any one of claims 1 to 3 . The certainty comparison unit evaluates at least one of the one or more candidate learning models by inputting evaluation data into the candidate learning model to perform model evaluation,
When the result of the first confidence comparison is true and the result of the factor determination is false, in the model replacement, one of the candidate learning models is the learning model in operation in the model evaluation is a candidate learning model that has a higher confidence than
The operation support system according to any one of claims 1 to 4. The data for evaluation is input data newer than the input data used to create the at least one candidate learning model and the input data;
Any of the above candidate learning models is determined by the input data used to create the candidate learning model rather than the confidence when the input data used to create the candidate learning model is input to the candidate learning model. is a candidate learning model with a higher confidence when new input data is input to the candidate learning model,
The operation support system according to claim 5. A learning unit that performs re-learning by creating one or more learning models with different versions;
A monitoring unit that monitors the operation of the learning model during operation
display and
with
The monitoring unit
a certainty comparison unit that performs a first certainty comparison to determine whether the certainty of the learning model in operation is less than a first threshold;
When the result of the first confidence comparison is true, the learning model to be operated is one or more candidate learning models whose version is different from the version of the learning model in operation. A model replacement unit that replaces the learning model in operation with one of the candidate learning models having a higher certainty than the learning model in operation from which the true result was obtained. and
with
The certainty comparison unit performs a second certainty comparison as to whether the certainty of the learning model in operation is less than a second threshold closer to the reference certainty than the first threshold,
When the result of the second confidence comparison is true, the learning unit performs the relearning by creating one or more learning models of a version different from the version of the learning model in operation. stomach,
the previous display unit displays version information of the one or more learning models of the different versions;
Operation support system. 8. The operation support system according to claim 1, wherein said first threshold is a threshold based on a reference degree of certainty determined based on past distribution of certainty. A pre-process that inputs input data that satisfies a predetermined condition regarding data representing an object to a learning model for re-learning , and does not input input data that does not satisfy the predetermined condition to the learning model for re-learning. part,
The operation support system according to any one of claims 1 to 8, further comprising: The target is a product to be inspected in product inspection,
The predetermined condition is a condition of data representing that the product is non-defective,
The operation support system according to claim 9 . The learning unit periodically or irregularly re-learns asynchronously with the operation monitoring,
The operation support system according to any one of claims 1 to 10. The certainty comparison unit performs the first certainty comparison each time the learning model in operation to which input data is input outputs output data.
The operation support system according to any one of claims 1 to 11. The computer performs re-learning by creating one or more learning models with different versions,
The computer performs a first confidence comparison as to whether the confidence of the learning model in operation is less than a first threshold,
When the result of the first certainty comparison is true, the computer determines whether or not the input data input to the learning model in operation from which the true result was obtained is a factor of the result. and
When the result of the factor determination is false, the computer selects the learning model to be operated from among one or more candidate learning models that are one or more learning models of a version different from the version of the learning model in operation. , displaying the confidence of any candidate learning model that is higher than the confidence of the learning model in operation from which the true result was obtained;
Operational support method. The computer performs re-learning by creating one or more learning models with different versions,
The computer performs a first confidence comparison as to whether the confidence of the learning model in operation is less than a first threshold,
When the result of the first confidence comparison is true, the computer selects the learning model to be operated as one or more candidates that are one or more learning models of a version different from the version of the learning model in operation. Among the learning models, perform model replacement, which is to replace the learning model in operation with one of the candidate learning models with a higher confidence than the learning model in operation from which the true result was obtained,
A computer performs a second confidence comparison as to whether the confidence of the learning model in operation is less than a second threshold closer to the reference confidence than the first threshold,
When the result of the second confidence comparison is true, the computer performs the re-learning, which is to create one or more learning models of a version different from the version of the learning model in operation, display version information for one or more learning models of different versions,
Operational support method. The first threshold is a threshold based on the reference confidence determined based on the past confidence distribution,
The operation support method according to claim 13 or 14. Perform re-learning by creating one or more learning models with different versions,
performing a first confidence comparison as to whether the confidence of the learning model in operation is less than a first threshold;
If the result of the first confidence comparison is true, determine whether the input data input to the learning model in operation from which the true result was obtained is a factor of the result, When the result of the factor determination is false, the learning model to be operated is selected from one or more candidate learning models that are one or more learning models of a version different from the version of the learning model in operation. Displaying the confidence of any candidate learning model that is higher than the confidence of the learning model in operation for which the result of
A computer program that makes a computer do something. Perform re-learning by creating one or more learning models with different versions,
performing a first confidence comparison as to whether the confidence of the learning model in operation is less than a first threshold;
When the result of the first confidence comparison is true, the learning model to be operated is one or more candidate learning models whose version is different from the version of the learning model in operation. Among them, perform model replacement, which is to replace the learning model in operation with one of the candidate learning models having a higher confidence than the learning model in operation from which the true result was obtained,
Perform a second confidence comparison as to whether the confidence of the learning model in operation is less than a second threshold closer to the reference confidence than the first threshold,
If the result of the second confidence comparison is true, perform the re-learning, which is to create one or more learning models of a version different from the version of the learning model in operation, and the different version of the display version information for one or more learning models,
A computer program that makes a computer do something. The first threshold is a threshold based on the reference confidence determined based on the past confidence distribution,
18. A computer program as claimed in claim 16 or 17.

Images