JP2022117730A - Information processing device, control system and report output method - Google Patents

Abstract

To provide a method for easily evaluating the validity of a trained model itself.SOLUTION: An information processing device connected to a control device for controlling an object to be controlled is provided. The control device has at least one trained model that receives feature values calculated from information collected from the object to be controlled and outputs a score that indicates the likelihood of a predetermined event. The information processing device includes an acquisition unit in which time-series data of information collected from the object to be controlled is acquired from the control device, and a generating unit in which a report on the trained model is generated based on the acquired time-series data. By changing at least one of the event, trained model and target period, the report includes multiple report elements including a temporal change in feature values input to the trained model corresponding to a specific event, a temporal change in score output from the trained model, and a performance index of the trained model.SELECTED DRAWING: Figure 6

Description

The present technology relates to an information processing device connected to a control device, a control system including the control device, and a report output method used in the control system.

At various production sites, there is a need to improve productivity and facility operating rates by utilizing machine learning. In general machine learning, events that have occurred in the past are collected to generate a trained model, but it is not easy to evaluate the validity of the generated trained model. Moreover, when a trained model is generated for each arbitrary condition, it is difficult to determine which trained model to adopt.

For example, Japanese Unexamined Patent Application Publication No. 2020-042668 (Patent Document 1) discloses an inspection device or the like that facilitates selection of a learning model that satisfies required specifications for a user in the field.

Japanese Patent Application Laid-Open No. 2020-042668

The prior art documents mentioned above are directed to techniques for selecting an appropriate trained model from among pre-generated trained models, and do not discuss evaluation of the validity of the trained model itself. not

This technology provides a method for easily evaluating the validity of the trained model itself.

According to an example of the present technology, an information processing device connected to a control device for controlling a controlled object is provided. The control device has at least one trained model that receives feature values calculated from information collected from the controlled object and outputs a score indicating the possibility of a predetermined event. The information processing device includes an acquisition unit that acquires time-series data of information collected from the controlled object from the control device, and a generation unit that generates a report regarding the trained model based on the acquired time-series data. The report includes temporal changes in feature values input to a trained model corresponding to a specific event, temporal changes in scores output from the trained model, and performance indicators of the trained model. A plurality of elements are included with at least one of an event, a learned model, and a target period being different.

According to this configuration, a report is generated that includes a plurality of report elements in which at least one of the event, learned model, and target period is different. can easily assess the adequacy of

In the report, each of the report elements may have information aligned in a first direction and information aligned in a second direction between the report elements. According to this configuration, a plurality of report elements are represented in a two-dimensional array, so that the contents can be understood more easily.

The information processing device further includes a first calculator that calculates a performance index of the corresponding trained model based on the time-series data. The performance index may include at least one of a correct answer rate, an overlook rate, and an oversight rate. According to this configuration, the validity of the learned model itself can be evaluated numerically.

Each of the report elements may include thresholds set for the corresponding trained model. According to this configuration, it is possible to more easily evaluate the validity of the setting of the learned model based on the output temporal change and the threshold value.

Reports may be configured to allow sorting or filtering of report elements. According to this configuration, only the report elements desired by the user can be easily selected or extracted.

A report may include code to implement sorting or filtering of report elements. This configuration eliminates the need for a dedicated application for sorting or filtering report elements, thereby enhancing secondary usability.

The information processing device may further include a second calculation unit that calculates the feature value input to the corresponding trained model and the score output from the trained model based on the time-series data. . According to this configuration, a report can be output even when the feature amount or score cannot be obtained from the control device.

The information processing device may further include a search unit that searches for similar temporal changes included in the report. According to this configuration, it is possible to easily extract or select report elements to be compared.

A control system according to another example of the present technology includes a control device for controlling a controlled object, and an information processing device connected to the control device. The control device includes at least one trained model that is input with feature amounts calculated from information collected from the controlled object and that outputs a score indicating the possibility of a predetermined event. The information processing device includes an acquisition unit that acquires time-series data of information collected from the controlled object from the control device, and a generation unit that generates a report regarding the trained model based on the acquired time-series data. The report includes temporal changes in feature values input to a trained model corresponding to a specific event, temporal changes in scores output from the trained model, and performance indicators of the trained model. A plurality of elements are included with at least one of an event, a learned model, and a target period being different.

According to still another example of the present technology, there is provided a report output method for outputting a report regarding a control device for controlling a controlled object. The control device has at least one trained model for outputting a score indicating the possibility of a predetermined event by inputting a feature amount calculated from information collected from the controlled object, and a report output method. includes the steps of obtaining time-series data of information collected from the controlled object from the control device, and generating a report on the trained model based on the obtained time-series data. The report includes temporal changes in feature values input to a trained model corresponding to a specific event, temporal changes in scores output from the trained model, and performance indicators of the trained model. A plurality of elements are included with at least one of an event, a learned model, and a target period being different.

According to this technology, the validity of the trained model itself can be evaluated more easily.

1 is a schematic diagram showing an example of the overall configuration of a control system according to an embodiment; FIG. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram for explaining an example of main configuration and operation mode of a control system according to an embodiment; 2 is a block diagram showing an example hardware configuration of a control device that constitutes the control system according to the present embodiment; FIG. FIG. 2 is a block diagram showing an example hardware configuration of a support device that configures the control system according to the present embodiment; FIG. FIG. 4 is a schematic diagram showing an outline of processing for performing abnormality detection of the control system according to the present embodiment; FIG. 4 is a schematic diagram for explaining contents of report output processing of the control system according to the present embodiment; FIG. 9 is a schematic diagram showing another example of a report output by the control system according to the embodiment; It is a figure which shows an example of the utilization method of the report output by the control system which concerns on this Embodiment. FIG. 10 is a schematic diagram showing another display example of a time waveform in a report output by the control system according to the embodiment; FIG. 3 is a schematic diagram showing a functional configuration example for realizing report output processing according to the present embodiment; 4 is a flow chart showing a processing procedure relating to report output processing according to the present embodiment;

Embodiments of the present invention will be described in detail with reference to the drawings. The same or corresponding parts in the drawings are denoted by the same reference numerals, and the description thereof will not be repeated.

<A. Application example>
First, an example of a scene to which the present invention is applied will be described. As an application example of machine learning, an example of determining whether or not an abnormality has occurred in equipment will be described below, but the present invention is not limited to abnormality detection and can be applied to arbitrary machine learning.

Any device to which a controller is associated is also referred to herein as a "controlled object". The term "controlled object" is not limited to any device directly controlled by the control device or the device from which the control device acquires information, but also any device or facility including those devices, as well as the units that make up the control device etc. can also be included.

In this specification, an anomaly detection engine reads a learned model generated by machine learning as an example of an entity that performs processing such as judgment. One or more predetermined feature amounts are input to the anomaly detection engine, and an output is calculated based on the input feature amounts. A value calculated by the anomaly detection engine using the read trained model is also called a “score”.

In this specification, an event that can be judged by applying a trained model generated by machine learning is also called an "event". For a trained model related to anomaly detection, an event typically means a specific anomaly that can occur in the controlled object. Note that the term “event” can include not only cases of direct determination using a trained model, but also events that can be determined indirectly. Therefore, the result (score) output from the trained model is a value that directly or indirectly indicates the possibility (occurrence possibility) of a given event.

FIG. 1 is a schematic diagram showing an example of the overall configuration of a control system 1 according to this embodiment. Referring to FIG. 1, a control system 1 according to the present embodiment includes, as main components, a control device 100 for controlling a controlled object, and a support device which is an example of an information processing device connected to the control device 100. and device 200 .

The control device 100 may be embodied as a kind of computer such as a PLC (programmable controller). The control device 100 is connected to the field device group 10 via the fieldbus 2 . Fieldbus 2 preferably employs an industrial communication protocol. EtherCAT (registered trademark), EtherNet/IP (registered trademark), DeviceNet (registered trademark), CompoNet (registered trademark), and the like are known as such communication protocols.

The field device group 10 includes devices that collect input data from control targets or control-related manufacturing devices, production lines, and the like (hereinafter also collectively referred to as “fields”). An input relay, various sensors, and the like are assumed as a device for collecting such input data. Field device group 10 further includes devices that give some effect to the field based on commands (hereinafter also referred to as “output data”) generated by control device 100 . Output relays, contactors, servo drivers and servo motors, and any other actuators are contemplated as devices that exert some action on such fields. These field device groups 10 exchange data including input data and output data with the control device 100 via the fieldbus 2 .

In the configuration example shown in FIG. 1 , field device group 10 includes remote I/O (Input/Output) device 12 , relay group 14 , servo driver 18 and servo motor 20 .

The remote I/O device 12 includes a communication unit that communicates via the fieldbus 2 and an input/output unit (hereinafter also referred to as "I/O unit") for collecting input data and outputting output data. including. Input data and output data are exchanged between the control device 100 and the field via such an I/O unit. FIG. 1 shows an example in which digital signals are exchanged as input data and output data via the relay group 14 .

The I/O units may be directly connected to the fieldbus. FIG. 1 shows an example in which an I/O unit 16 is directly connected to the fieldbus 2 .

The servo driver 18 drives the servo motor 20 in accordance with output data (for example, position commands, etc.) from the control device 100 .

As described above, input data and output data are exchanged between the control device 100 and the field device group 10 via the fieldbus 2. These exchanged data are on the order of hundreds of microseconds. It will be updated in a very short cycle on the order of several tens of milliseconds. Note that such processing for updating exchanged data is sometimes referred to as "I/O refresh processing".

The control device 100 has a PLC engine (the PLC engine 130 shown in FIG. 2) that executes control calculations for controlling controlled objects such as equipment and machines. The PLC engine corresponds to a control calculation unit, and determines output data by executing control calculation based on input data. The control device 100 has a time series database (hereinafter referred to as "TSDB ( (also referred to as Time Series Data Base)”) 140. Hereinafter, data stored in the TSDB 140 is also referred to as "time-series data".

The control device 100 is an anomaly detection engine that detects anomalies in the controlled object by referring to a pre-prepared learned model (hereinafter also simply referred to as "model 160") based on information collected from the controlled object. has 150.

A feature amount calculated from information collected from the controlled object is input to the model 160 . Output from the model 160 is a score that indicates the likelihood of a given event. Note that basically, a model 160 is prepared for each event. However, the logic for determining each of a plurality of events can also be integrated into one model. Thus, the control device 100 (anomaly detection engine 150) has at least one model 160. FIG.

As the information collected from the controlled object, time-series data (or feature values calculated from the time-series data) stored in the TSDB 140 are typically used. That is, the anomaly detection engine 150 determines the occurrence or possibility of an anomaly in the controlled object based on the time-series data stored in the TSDB 140 .

The control device 100 may be connected to the server 400 via the host network 6 or may be connected to one or more display devices 500 via the fieldbus 4 . Note that the server 400 and the display device 500 are optional components and not essential components of the control system 1 .

The server 400 is in charge of processing such as providing arbitrary information to the control device 100 or collecting data from the control device 100 .

The display device 500 receives an operation from a user, transmits a command according to the user's operation to the control device 100 , and graphically displays a calculation result of the control device 100 .

The support device 200 is an information processing device (an example of a computer) that assists preparations necessary for the control device 100 to control a control target. Specifically, the support device 200 includes a development environment (program creation and editing tool, parser, compiler, etc.) for the user program executed by the control device 100, parameters of the control device 100 and various devices connected to the control device 100 ( configuration), a function to send the generated user program to the control device 100, a function to modify/change the user program executed on the control device 100 online, and the like.

Further, the support device 200 has a function of supporting the generation and adjustment of the model 160 referenced by the anomaly detection engine 150 implemented in the control device 100 .

Here, what is related to the object of anomaly detection is any information that can be used to generate learning data for learning the detection logic of the anomaly detection engine 150 (that is, the model 160). may Items related to anomaly detection targets include, for example, works to be produced (products or semi-finished products) and production facilities themselves.

Next, an example of the configuration and operation of the abnormality detection engine 150 of the control device 100 will be described. The support device 200 acquires the time-series data stored in the TSDB 140 and determines parameters defining the model 160 that the anomaly detection engine 150 refers to by data mining. The support device 200 then transfers the determined parameters to the control device 100 . Thereby, the abnormality detection engine 150 of the control device 100 is effectively configured.

After operation, the support device 200 adjusts the parameters defining the model 160 based on the time-series data collected after operation. That is, the model 160 will be regenerated or adjusted. The support device 200 then transfers the adjusted parameters to the control device 100 . This further optimizes the abnormality detection engine 150 of the control device 100 .

FIG. 2 is a diagram for explaining an example of the main configuration and operation mode of the control system 1 according to the present embodiment. Referring to FIG. 2, the abnormality detection process in control system 1 is typically realized in the order of data collection phase 30, data analysis phase 32, and operation phase .

In the data collection phase 30, time-series data related to the object of anomaly detection is collected. In the data analysis phase 32, by data mining of the time-series data collected in the data collection phase 30, the feature amount suitable for the target anomaly detection is examined, and the model 160 according to the examined feature amount is created. generated. Utilizing the model 160 generated in the data analysis phase 32, the operational phase 34 begins.

After the start of the operation phase 34, if an insufficient detection or an erroneous detection occurs such that an abnormality that should be detected is overlooked or an abnormality that should not be detected is detected, the model 160 may be updated by re-learning. be. Alternatively, the model 160 may be updated by re-learning even when some change occurs in the target of abnormality detection (for example, aging, change in operating conditions, etc.).

Considering such a processing procedure, the support device 200 may include a data mining tool 240 , a model generation tool 250 and a report output tool 260 . Data mining tools 240 primarily provide the necessary processing during data analysis phase 32 . The model generation tool 250 primarily provides the necessary processing during the operations phase 34 .

The data mining tool 240 typically provides feature generation processing 242 , feature selection processing 244 , model generation processing 246 and threshold setting processing 248 .

Model generation tool 250 provides model update process 252 and threshold adjustment process 254 .

The report output tool 260 outputs various information obtained by the processing of the data mining tool 240 and the model generation tool 250 (report output), as will be described later.

On the other hand, the control device 100 has the PLC engine 130, the TSDB 140 and the abnormality detection engine 150 as described above.

The PLC engine 130 periodically executes control operations defined by a user program 132 that is arbitrarily created according to the object to be controlled. The AI library 134 is available for the user program 132 . Alternatively, the AI library 134 may be embedded in the user program 132 . The content, properties, behavior, etc. of AI library 134 may be determined using data mining tools 240 in data analysis phase 32 .

The PLC engine 130 executes the part of the AI library 134 included in the user program 132 to calculate one or more feature amounts 152 from the data specified in advance, and the calculated feature amounts 152 are sent to the anomaly detection engine 150. Provided.

The PLC engine 130 has a data management unit 136 and holds input data, output data, and internal data in a form that can be referenced from the user program 132 . The value of data managed by the data management unit 136 is updated every predetermined cycle (I/O refresh cycle) by the I/O refresh process. Also, among the data managed by the data management unit 136, data designated in advance is stored in the TSDB 140 at predetermined intervals. As a result, the TSDB 140 outputs the value change of the specified data every predetermined period, that is, the time-series data 142 .

The anomaly detection engine 150 refers to the model 160 based on the feature quantity 152 from the PLC engine 130 to determine the occurrence or possibility of an anomaly. The model 160 may be determined using the data mining tools 240 during the data analysis phase 32 and may be determined using the model generation tools 250 during the operational phase 34 .

Typically, the anomaly detection engine 150 reads the model 160 and outputs a value indicating the degree of anomaly (degree of separation from normal) when one or more feature quantities 152 are input. Parameters defining the model 160 may include each coefficient defining the function that is the entity of the model 160 and a threshold for evaluating the value output from the model 160 .

The control system 1 according to the present embodiment can output information related to machine learning as a report, and the user can easily evaluate the validity of the trained model by using such report output. In addition, it is possible to easily grasp the tendency of equipment deterioration, the tendency of production status, and the like.

<B. Hardware configuration example>
Next, an example of hardware configuration of main devices that constitute the control system 1 according to the present embodiment will be described.

(b1: Hardware configuration example of control device 100)
FIG. 3 is a block diagram showing a hardware configuration example of the control device 100 forming the control system 1 according to the present embodiment. Referring to FIG. 3, control device 100 includes a processor 102 such as a CPU (Central Processing Unit) or MPU (Micro-Processing Unit), a chipset 104, a main storage device 106, a secondary storage device 108, Host network controller 110, USB (Universal Serial Bus) controller 112, memory card interface 114, internal bus controller 122, field bus controllers 118, 120, I/O units 124-1, 124-2, . including.

The processor 102 implements the PLC engine 130 and the abnormality detection engine 150 by reading out various programs stored in the secondary storage device 108, developing them in the main storage device 106, and executing them. The chipset 104 controls data transmission between the processor 102 and each component.

The secondary storage device 108 stores a system program 131 for realizing the PLC engine 130 and the anomaly detection engine 150 as well as a user program 132 that is executed using the PLC engine 130 . A partial area of the secondary storage device 108 may be used as the TSDB 140 .

The host network controller 110 controls exchange of data with other devices via the host network 6 . The USB controller 112 controls data exchange with the support device 200 via a USB connection.

The memory card interface 114 is configured such that a memory card 116 can be attached/detached, and data can be written to the memory card 116, and various data (user program, trace data, etc.) can be read from the memory card 116. ing.

The internal bus controller 122 is an interface that exchanges data with the I/O units 124-1, 124-2, .

The fieldbus controller 118 controls data exchange with other devices via the fieldbus 2 . Similarly, the fieldbus controller 120 controls data exchange with other devices via the fieldbus 4 .

FIG. 3 shows a configuration example in which necessary functions are provided by the processor 102 executing a program. (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array), etc.). Alternatively, the main part of the control device 100 may be implemented using hardware conforming to a general-purpose architecture (for example, an industrial personal computer based on a general-purpose personal computer). In this case, virtualization technology may be used to execute a plurality of OSs (Operating Systems) for different purposes in parallel, and necessary applications may be executed on each OS.

(b2: Hardware configuration example of support device 200)
Support device 200 according to the present embodiment is implemented, for example, by executing a program using hardware conforming to a general-purpose architecture (for example, a general-purpose personal computer).

FIG. 4 is a block diagram showing a hardware configuration example of the support device 200 that configures the control system 1 according to this embodiment. Referring to FIG. 4, support device 200 includes processor 202 such as a CPU or MPU, optical drive 204, main storage device 206, secondary storage device 208, USB controller 212, network controller 214, input A portion 216 and a display portion 218 are included. These components are connected via bus 220 .

The processor 202 reads various programs stored in the secondary storage device 208, develops them in the main storage device 206, and executes them, thereby realizing various processes including report output processing, which will be described later.

The secondary storage device 208 is composed of, for example, an HDD (Hard Disk Drive) or an SSD (Flash Solid State Drive). The secondary storage device 208 typically contains an OS 222 and a PLC interface program 224 for exchanging data relating to anomaly detection functions between the control device 100 and a user program to be executed in the support device 200. , a development program 226 for debugging the created user program, defining the system configuration, setting various parameters, etc.; a data mining program 228 for realizing the data mining tool 240; and the model generation program 230 are stored. The secondary storage device 208 may store necessary programs other than the programs shown in FIG.

The support device 200 has an optical drive 204, and from a recording medium 205 (for example, an optical recording medium such as a DVD (Digital Versatile Disc)) that stores a computer-readable program non-transitory, The stored program is read and installed in the secondary storage device 208 or the like.

Various programs to be executed by the support device 200 may be installed via the computer-readable recording medium 205, or may be installed by being downloaded from a server device or the like on the network. Also, the functions provided by the support apparatus 200 according to the present embodiment may be realized by using some of the modules provided by the OS 222. FIG.

The USB controller 212 controls data exchange with the control device 100 via a USB connection. Network controller 214 controls the exchange of data to and from other devices over any network.

The input unit 216 is composed of a keyboard, a mouse, and the like, and receives user operations. A display unit 218 includes a display, various indicators, a printer, and the like, and outputs processing results from the processor 202 and the like.

FIG. 4 shows a configuration example in which necessary functions are provided by the processor 202 executing a program. Alternatively, it may be implemented using an FPGA, etc.).

<C. Report output processing>
Next, report output processing according to this embodiment will be described.

FIG. 5 is a schematic diagram showing an outline of processing for performing abnormality detection of the control system 1 according to the present embodiment. Referring to FIG. 5, anomaly detection engine 150 has model 160 . One or more feature quantities 152 are input to the model 160 , and a score calculated from the input feature quantity 152 is output from the model 160 . The score output from the model 160 is evaluated using, for example, a threshold (threshold evaluation process 154). Although FIG. 5 shows an example in which N feature amounts are input, the number of feature amounts to be input is not particularly limited, and may be one or more.

The feature amount 152 is information calculated by an arbitrary method from the information from the field (part of the time-series data). Any value such as a maximum value, a minimum value, a median value, an average value, and a standard deviation can be used as the feature quantity 152 . It should be noted that the time-series data can be used directly as the feature amount.

In the example shown in FIG. 5, model 1 corresponding to event 1 is used, but model 160 may be generated for each event to be detected (event to be monitored).

The scores output from model 160 are typically one-dimensional values, but multi-dimensional values may be employed. When adopting a multidimensional score, it is preferable to use a multidimensional value also for the threshold used for evaluation.

The threshold may be one or plural. For example, assuming anomaly detection, two types of information may be prepared: caution indicating that the anomaly has progressed to a certain extent, and damage indicating that an anomaly has occurred.

The report output process according to this embodiment outputs information related to the anomaly detection engine 150 as shown in FIG.

FIG. 6 is a schematic diagram for explaining the content of report output processing of the control system 1 according to the present embodiment. Referring to FIG. 6, support device 200 outputs user interface screen 270 regarding abnormality detection processing.

User interface screen 270 shown in FIG. 6 includes score display area 272 and histogram display area 274 . In FIG. 6, feature amounts calculated for each of the three events are displayed.

That is, in the score display area 272, temporal changes in the feature amount calculated for each event are displayed. More specifically, the horizontal axis of the graph displayed in the score display area 272 is divided into predetermined time intervals called frames. Feature values calculated from time-series data collected in each time interval are displayed.

It should be noted that the score display area 272 may be switched to display temporal changes in one or more selected feature amounts.

Furthermore, a set threshold value 276 is displayed in the score display area 272 . A user can adjust the threshold 276 to any value.

The histogram display area 274 displays a histogram of temporal changes in the feature amount (feature amount in each time interval) displayed in the corresponding score display area 272 .

While referring to the user interface screen 270, the user can confirm the validity of the selected feature amount, the validity of the model 160 for calculating the score, the validity of the set threshold value 276, and the like. The user can also change the threshold 276 as desired.

The support device 200 can output, for example, a report 278 corresponding to the user interface screen 270 shown in FIG. Report 278 shown in FIG. 6 is output in tabular form and includes one or more report elements 280 .

As shown in FIG. 6, in the report 278, each of the report elements 280 has information arranged side by side in the horizontal direction (first direction) on the page, and the information is arranged in the vertical direction (second direction) between the report elements 280 on the page. direction).

The report 278 will typically include a plurality of report elements 280 with at least one of events, trained models, and time periods differing according to predetermined settings or user settings. In the example shown in FIG. 6, report 278 includes report elements 280 with different events (multiple event-specific report elements 280).

However, even for the same event, the report 278 may include a plurality of report elements 280 with different target periods, or the models 160 may be different (for example, when the model 160 is generated or when the model version is different). Different) report elements 280 may be included in the report 278 .

Each of the report elements 280 includes a score display term 281, a feature quantity display term 282, threshold terms 283 and 284, a correct answer rate term 285, an oversight rate term 286, an oversight rate term 287, and an abnormality probability. Includes section 288 and comment section 289 . Information corresponding to each item is stored for each report element 280 .

In the score display field 281, the temporal change in the calculated score is pasted as an image for each event to be monitored.

In the feature amount display field 282, the temporal change of the calculated feature amount is pasted as an image for each event to be monitored. Note that when a score is calculated from a plurality of feature amounts, only the temporal change of a specific feature amount may be displayed, or the temporal changes of a plurality of feature amounts may be displayed in the same display area. may be

The correct answer rate term 285, the overlook rate term 286, and the oversight rate term 287 store values indicating the abnormality detection performance of the currently set model 160. FIG. More specifically, the correct answer rate term 285 includes the sum of cases where an abnormal state is correctly judged as an abnormal state (TP: true positive) and cases where a normal state is correctly judged as a normal state (TN: true negative). is stored. The miss rate term 286 stores the total rate of cases where an abnormal state is erroneously determined to be a normal state (FN: false negative). The oversight rate term 287 stores the total rate of cases in which a normal state is erroneously determined to be an abnormal state (TN: true negative).

It should be noted that it is not necessary to use all of the correct answer rate, the overlook rate, and the overwatch rate as the performance index of the model 160, and at least one of them may be included. On the other hand, the content of report 278 shown in FIG. 6 is an example, and other or additional performance indicators for model 160 may be output. Performance indicators of the model 160 include, for example, sensitivity, detection rate, specificity, accuracy, precision, ROC curve, PR curve, and the like.

In this way, each of the report elements 280 represents changes over time in the feature quantity input to the model 160 corresponding to a specific event (feature quantity display term 282) and changes in score over time (score display term 281). , and the performance indicators of model 160 (correct answer rate term 285, overlook rate term 286, and overwatch rate term 287).

The threshold terms 283 and 284 store the thresholds set for the corresponding model 160 . More specifically, threshold terms 283 and 284 store the value of threshold 276 for evaluating the score. The example shown in FIG. 6 shows a state in which each value is stored when two threshold values 276 are set.

The anomaly probability term 288 stores the rate of occurrence of samples judged to be in an anomalous state.

The comment item 289 may store special notes. Basically, when the user uses the report 278, he/she will add it as necessary.

Furthermore, additional items may be added to facilitate various types of analysis and management.

FIG. 7 is a schematic diagram showing another example of the report 278 output by the control system 1 according to this embodiment. Report 278 shown in FIG. 7 includes an event ID term 290, a model ID term 291, and a time stamp term 292 in addition to report 278 shown in FIG.

The event ID field 290 stores identification information for specifying the corresponding event. The identification information stored in the event ID field 290 may be determined according to a predetermined rule so that the event can be easily identified. The model ID item 291 stores identification information for specifying the corresponding model 160 . For the identification information stored in the model ID item 291, a hash value or the like calculated from the corresponding model 160 itself may be used. The timestamp field 292 stores information (timestamp) of the date and time when the corresponding information was collected.

A report 278 may include multiple pieces of information (eg, different collection times) for the same event. Reports 278 may also include multiple pieces of information (eg, different collection times and/or model versions) for the same event. In such cases, the information stored in one or more terms may be used to extract and aggregate the information of interest.

Any term contained in the title column 294 of the report 278 is selectable, and when any term is selected, the selected term can be sorted (e.g. sorted in ascending or descending order). . Alternatively, one or more selected terms may be filtered under arbitrary conditions.

The report 278 shown in FIG. 7 may further include a flags term 293 for identifying similar information. For example, it may be determined whether the temporal change in score (score display item 281) and/or the temporal change in feature amount (feature amount display item 282) are similar to each other. The similarity of temporal changes (time waveforms) can be evaluated using, for example, the standard deviation of time waveforms, the peak value of time waveforms, the correlation between time waveforms, and the like. In that case, the information associated with the information of the selected event is searched and the flags term 293 of the searched information is checked.

Alternatively, the user may arbitrarily check the similarity instead of automatically determining the similarity as described above. In this case, it is assumed that the user checks the information displayed in the report 278, checks the items of interest, and collectively outputs the checked information.

In the feature amount display field 282, the temporal change of the calculated feature amount is pasted as an image for each event to be monitored. Note that when a score is calculated from a plurality of feature amounts, only the temporal change of a specific feature amount may be displayed, or the temporal changes of a plurality of feature amounts may be displayed in the same display area. may be

A plurality of groups of events output as the report 278 may be arbitrarily set, or may be grouped under conditions such as those related to a common production line.

FIG. 8 is a diagram showing an example of how to use the report 278 output by the control system 1 according to this embodiment. For convenience of explanation, the threshold terms 283 and 284, the correct answer rate term 285, the overlook rate term 286, and the overwatch rate term 287 shown in FIGS. ing.

FIG. 8A shows an example of filtering using the values stored in the event ID field 290 of the report 278. FIG. For example, if the report 278 contains multiple pieces of information about a specific event, filtering or sorting by the event ID term 290 will identify the specific event (ID in the example shown in FIG. 8A). is "00") can be collectively displayed.

By collectively displaying information about such a specific event, the validity of the model 160 and the like can be displayed over time. It is also possible to check the process of generating the model 160 corresponding to a specific event after the fact.

FIG. 8B shows an example of filtering using the value stored in the timestamp field 292 of report 278 . For example, if the report 278 contains information over a period of time, filtering or sorting by the timestamp term 292 will aggregate information about events that occurred at a particular time period or at a particular time. can be displayed.

By collectively displaying information about events that have occurred in the same target institution, it is possible to check information related to a certain event at a glance, and to facilitate investigation of the cause and determination of countermeasures for the event that has occurred. .

FIG. 8C shows an example of filtering using the flag term 293 of the report 278 . For example, the flag term 293 is checked for information in which the temporal change in score (score display term 281) and/or the temporal change in feature quantity (feature quantity display term 282) are similar to each other. Therefore, by extracting only the information whose flag item 293 has been checked, it is possible to collectively display information indicating similar time waveforms.

Collecting learning data used for updating or generating the model 160 can be easily performed by collectively displaying information with similar tendencies.

As shown in FIGS. 8A-8C, report 278 may be configured to allow sorting or filtering of report elements 280 .

FIG. 9 is a schematic diagram showing another display example of the temporal waveform in the report 278 output by the control system 1 according to the embodiment. Referring to FIG. 9, for the information of the selected event, the temporal change in score (score display item 281) and the temporal change in feature amount (feature amount display item 282) may be superimposed and displayed. By displaying the score and the feature amount used to calculate the score on a common time axis in this way, it is possible to easily confirm the selection of the feature amount and the appropriateness of the score.

The report 278 may be configured on a paper medium according to an instruction from the support device 200, or may be output in the data format of a spreadsheet application (for example, Microsoft's Excel (registered trademark), etc.).

When the report 278 is generated and output in the data format of the spreadsheet application, the functions such as selection as described above may be realized using the program execution environment provided by the spreadsheet application. More specifically, report 278 may include, in addition to the information described above, programs (eg, program code referred to as macros, etc.) for sorting, filtering, searching, and the like. That is, report 278 may include code to implement sorting or filtering of report elements 280 .

Alternatively, the display function provided by the support device 200 may be used to realize sorting, filtering, searching, etc. as described above.

By using such a report 278, it is possible to easily grasp the performance of the model 160 generated by machine learning. In particular, for each of a plurality of events, by outputting the corresponding information as one report 278, it becomes easy to identify events with relatively low detection performance, and it is necessary to update the model 160. makes it easier to identify events with Additionally, reports 278 can be used to facilitate updating models 160 and generating new models 160 .

<D. Functional configuration example>
Next, a functional configuration example for realizing report output processing according to the present embodiment will be described.

FIG. 10 is a schematic diagram showing a functional configuration example for realizing report output processing according to the present embodiment. Each function shown in FIG. 10 is typically implemented by processor 202 of support device 200 executing development program 226 (FIG. 2).

10, support apparatus 200 includes a time-series data acquisition unit 261, a statistical data calculation unit 262, a feature value score calculation unit 263, a model management unit 263, and a It includes a unit 264 , a model performance calculation unit 265 , a report generation unit 266 , a report management unit 267 , a search unit 268 and a display editing unit 269 .

The time-series data acquisition unit 261 corresponds to an acquisition unit for acquiring from the control device 100 time-series data of information collected from the controlled object. More specifically, the time-series data acquisition unit 261 acquires time-series data from the TSDB 140 ( FIG. 2 ) of the control device 100 . The time-series data acquired by the time-series data acquisition unit 261 may be selected according to conditions preset by the user, or may be appropriately selected according to processes such as updating and generating the model 160 and outputting the report 278. good too.

The statistical data calculation unit 262 analyzes the acquired time-series data and calculates statistical data (maximum value, minimum value, median value, average value, standard deviation of time waveform). The statistical data calculation unit 262 can also acquire information such as a threshold associated with the acquired time-series data.

The feature amount score calculation unit 263 corresponds to a calculation unit for calculating the feature amount input to the corresponding model 160 and the score output from the model 160 based on the time series data. That is, the feature amount score calculation unit 263 calculates the feature amount and score from the time-series data acquired by the time-series data acquisition unit 261 . When the time-series data acquisition unit 261 acquires time waveforms of feature amounts and scores as time-series data, the feature amount score calculation unit 263 does not necessarily need to calculate the feature amounts and scores. In this case, the feature values and scores calculated by the control device 100 during operation are used.

The model management unit 264 manages the version of the generated model 160 and the like. The model management unit 264 holds and updates a model generation history 162 as information regarding management of the model 160 .

The model performance calculator 265 corresponds to a calculator for calculating the performance index of the model 160 based on the time-series data. As described above, the performance index may include correct answer rate, overlook rate, oversight rate, abnormality probability, sensitivity, detection rate, specificity, precision, precision, ROC curve, PR curve, and the like.

The report generator 266 corresponds to a generator for generating a report on the model 160 based on the acquired time-series data. More specifically, the report generation unit 266 generates the statistical data output from the statistical data calculation unit 262 and the feature score output from the feature value score calculation unit 263 (or obtained by the time-series data acquisition unit 261). A report 278 is generated and output using the quantities and scores and performance indicators of the model 160 output from the model performance calculator 265 .

Report management unit 267 manages generated report 278 . For example, the report management unit 267 gives the date and time (time stamp) when the report 278 was generated, or holds the version and identification information of the model 160 used in each report 278 .

The search unit 268 searches for similar temporal changes in feature quantity and/or temporal changes in scores included in the report 278 . More specifically, the searching unit 268 implements the processing described with reference to FIG. 8(C).

The display editor 269 provides processing for displaying or editing the report 278 . However, if the report 278 is output in the data format of a spreadsheet application, a dedicated function for displaying the report 278 need not be implemented.

<E. Processing procedure>
Next, a processing procedure related to report output according to the present embodiment will be described.

FIG. 11 is a flow chart showing a processing procedure for report output processing according to the present embodiment. Each step shown in FIG. 11 is typically implemented by processor 202 of support device 200 executing development program 226 (FIG. 2).

Referring to FIG. 11, upon receiving a report output request from the user (YES in step S100), support device 200 acquires necessary time-series data from control device 100 (step S102). That is, the support device 200 acquires time-series data of information collected from the controlled object from the control device 100 . The support device 200 then analyzes the acquired time-series data and calculates statistical data (step S104).

Further, the support device 200 determines whether or not the acquired time-series data includes the time waveforms of the feature amount and score (step S106).

If the acquired time-series data does not include feature amounts and score time waveforms (NO in step S106), support device 200 calculates feature amounts and score time waveforms from the acquired time-series data (step S108). ). If the acquired time-series data includes the feature amount and the time waveform of the score (YES in step S106), the process of step S108 is skipped.

The support device 200 selects target models 160 (step S110), and calculates a performance index for each of the selected models 160 (step S112).

The support device 200 generates and outputs a report 278 using the statistical data, the feature amount and score, and the performance index of the model 160 (step S114). That is, the support device 200 generates the report 278 regarding the model 160 based on the acquired time-series data.

At the same time, the support device 200 adds information about the generated report 278 as management information (step S116).

By the procedure described above, the processing related to report output is completed.

<F. Variation>
In the above description, a configuration example in which the support device 200 independently executes the report output process has been described, but the report output process may be realized by cooperation between the support device 200 and another device. It may be realized by a device (for example, a computing resource on the cloud) different from the support device 200 . In other words, any kind of hardware may be used to implement the report output process according to this embodiment.

Although an example of a tabular report 278 has been described above, the report 278 may be of any style or format as long as it can present the information as described above.

<G. Note>
The present embodiment as described above includes the following technical ideas.
[Configuration 1]
An information processing device (200) connected to a control device (100) for controlling a controlled object, wherein the control device receives a feature amount calculated from information collected from the controlled object, at least one trained model (160) that outputs a score indicating the possibility of a predetermined event, the information processing device comprising:
an acquisition unit (261) that acquires time-series data of information collected from the controlled object from the control device;
a generation unit (266) that generates a report (278) on the trained model based on the acquired time-series data;
The report contains temporal changes (282) in feature values input to a trained model corresponding to a specific event, temporal changes (281) in scores output from the trained model, and performance indicators (285, 286, 287; 296) of the information processing apparatus, wherein at least one of an event, a learned model, and a target period is different.
[Configuration 2]
In said report,
each of the report elements is arranged side by side in a first direction;
The information processing apparatus according to configuration 1, wherein the report elements are arranged side by side in the second direction.
[Configuration 3]
Further comprising a first calculation unit (265) for calculating a performance index of the corresponding trained model based on the time-series data,
3. The information processing apparatus according to configuration 1 or 2, wherein the performance index includes at least one of a correct answer rate, an overlook rate, and an oversight rate.
[Configuration 4]
4. The information processing apparatus according to any one of configurations 1 to 3, wherein each of said report elements includes a threshold value (283, 284) set in a corresponding trained model.
[Configuration 5]
5. The information processing apparatus according to any one of configurations 1 to 4, wherein the report is configured to enable sorting or filtering of the report elements.
[Configuration 6]
6. The information processing apparatus according to configuration 5, wherein the report includes code for implementing sorting or filtering of the report elements.
[Configuration 7]
Configurations 1 to 6, further comprising: a second calculation unit (263) for calculating, based on the time-series data, a feature value input to a corresponding trained model and a score output from the trained model The information processing device according to any one of .
[Configuration 8]
6. The information processing apparatus according to any one of configurations 1 to 5, further comprising a search unit (268) that searches for similar temporal changes included in the report.
[Configuration 9]
a control device (100) for controlling a controlled object;
An information processing device (200) connected to the control device,
The control device includes at least one trained model (160) that receives feature values calculated from information collected from the controlled object and outputs a score indicating the possibility of a predetermined event,
The information processing device is
an acquisition unit (261) that acquires time-series data of information collected from the controlled object from the control device;
a generation unit (278) that generates a report on the trained model based on the acquired time-series data;
The report includes temporal changes (282) in feature values input to a trained model corresponding to a specific event, temporal changes (281) in scores output from the trained model, and performance indicators (285, 286, 287; 296) of the control system, wherein at least one of an event, a learned model, and a period of interest is different.
[Configuration 10]
A report output method for outputting a report on a control device (100) for controlling a controlled object, wherein the control device is input with a feature amount calculated from information collected from the controlled object, and a predetermined at least one trained model (160) that outputs a score indicating the likelihood of an event, and the report output method includes:
a step of acquiring time-series data of information collected from the controlled object from the control device (S102);
generating a report on the trained model based on the acquired time-series data (S114);
The report includes temporal changes (282) in feature values input to a trained model corresponding to a specific event, temporal changes (281) in scores output from the trained model, and performance index (285, 286, 287; 296), wherein at least one of an event, a learned model, and a target period is different.

<H. Advantage>
The information processing apparatus according to the present embodiment generates a report including a plurality of report elements in which at least one of an event, a learned model, and a target period is different. By using such a report, the validity of the trained model itself can be evaluated more easily. Since it is output as a report, for example, the generated learned model can be used for sharing among users, reporting to superiors and customers, and the like. Furthermore, by accumulating reports, it becomes easier to manage the history of trained models. By using the report in this way, it is possible to easily explain information such as the data used in the trained model and the set threshold value to a third party.

In addition, since the report output by the information processing apparatus according to the present embodiment can be sorted or filtered for a plurality of report elements, it is possible to easily extract the report elements to be analyzed. This makes it possible to easily regenerate or update the learned model.

It should be considered that the embodiments disclosed this time are illustrative in all respects and not restrictive. The scope of the present invention is indicated by the scope of the claims rather than the above description, and is intended to include all modifications within the scope and meaning of equivalents to the scope of the claims.

1 control system, 2, 4 field bus, 6 host network, 10 field device group, 12 remote I/O device, 14 relay group, 16, 124 I/O unit, 18 servo driver, 20 servo motor, 30 data acquisition phase , 32 data analysis phase, 34 operation phase, 100 control device, 102, 202 processor, 104 chipset, 106, 206 main storage device, 108, 208 secondary storage device, 110 upper network controller, 112, 212 USB controller, 114 memory card interface, 116 memory card, 118, 120 fieldbus controller, 122 internal bus controller, 130 PLC engine, 131 system program, 132 user program, 134 library, 136 data management section, 142 time series data, 150 anomaly detection engine, 152 feature quantity, 154 evaluation process, 160 model, 162 model generation history, 200 support device, 204 optical drive, 205 recording medium, 214 network controller, 216 input unit, 218 display unit, 220 bus, 222 OS, 224 interface program, 226 development program, 228 data mining program, 230 model generation program, 240 data mining tool, 242 feature amount generation process, 244 feature amount selection process, 246 model generation process, 248 setting process, 250 model generation tool, 252 model update process, 254 adjustment processing, 260 report output tool, 261 time-series data acquisition unit, 262 statistical data calculation unit, 263 feature value score calculation unit, 264 model management unit, 265 model performance calculation unit, 266 report generation unit, 267 report management unit, 268 search unit, 269 display editing unit, 270 user interface screen, 272 score display area, 274 histogram display area, 276 threshold value, 278 report, 280 report element, 281 score display item, 282 feature amount display item, 283,284 Threshold term, 285 Correct answer rate term, 286 Missed rate term, 287 Overwatch rate term, 288 Abnormal probability term, 289 comment term, 290 event ID term, 291 model ID term, 292 time stamp term, 293 flag term, 294 title string, 296 performance index term, 400 server, 500 display device.

Claims (10)

An information processing device connected to a control device for controlling a controlled object, wherein the control device receives a feature amount calculated from information collected from the controlled object and determines the possibility of a predetermined event. and at least one trained model that outputs a score indicating
an acquisition unit that acquires time-series data of information collected from the controlled object from the control device;
a generation unit that generates a report on the trained model based on the acquired time-series data,
The report includes temporal changes in feature values input to a trained model corresponding to a specific event, temporal changes in scores output from the trained model, and performance indicators of the trained model. An information processing apparatus comprising a plurality of report elements with at least one of an event, a learned model, and a target period being different. In said report,
each of the report elements is arranged side by side in a first direction;
2. The information processing apparatus according to claim 1, wherein the report elements are arranged side by side in the second direction. Further comprising a first calculation unit that calculates a performance index of the corresponding trained model based on the time-series data,
3. The information processing apparatus according to claim 1, wherein said performance index includes at least one of a correct answer rate, an overlook rate, and an oversight rate. 4. The information processing apparatus according to any one of claims 1 to 3, wherein each of said report elements includes a threshold value set for a corresponding trained model. The information processing apparatus according to any one of claims 1 to 4, wherein said report is configured to enable sorting or filtering of said report elements. 6. Information processing apparatus according to claim 5, wherein said report includes code for implementing sorting or filtering of said report elements. 7. The method according to any one of claims 1 to 6, further comprising a second calculation unit that calculates a feature value input to a corresponding trained model and a score output from the trained model based on the time-series data. 1. The information processing apparatus according to 1. 6. The information processing apparatus according to any one of claims 1 to 5, further comprising a search unit that searches for similar temporal changes included in said report. a control device for controlling a controlled object;
and an information processing device connected to the control device,
The control device includes at least one trained model that receives feature values calculated from information collected from the controlled object and outputs a score indicating the possibility of a predetermined event,
The information processing device is
an acquisition unit that acquires time-series data of information collected from the controlled object from the control device;
a generation unit that generates a report on the trained model based on the acquired time-series data,
The report includes temporal changes in feature values input to a trained model corresponding to a specific event, temporal changes in scores output from the trained model, and performance indicators of the trained model. A control system comprising a plurality of report elements with at least one of an event, a learned model, and a target period being different. A report output method for outputting a report relating to a control device for controlling a controlled object, wherein the control device is input with a feature amount calculated from information collected from the controlled object, and detects the possibility of a predetermined event. and at least one trained model that outputs a score indicating sexuality, and the report output method includes:
a step of acquiring time-series data of information collected from the controlled object from the control device;
generating a report on the trained model based on the acquired time series data;
The report includes temporal changes in feature values input to a trained model corresponding to a specific event, temporal changes in scores output from the trained model, and performance indicators of the trained model. A report output method including a plurality of report elements with at least one of an event, a learned model, and a target period being different.

Images