JP2021105807A - Computer system and method for analyzing operating environment of business system - Google Patents

Abstract

To present information superior in readability for grasping an operating environment of an AI system (business system).SOLUTION: A computer system for analyzing an operating environment of a business system having an inference unit 111 that performs inference includes: an interpretation index calculation unit 510 that calculates an interpretation index for interpreting an inference result 501 obtained by inputting data 500 containing a plurality of feature amounts into the inference unit; and an analysis unit 113 that analyzes the operating environment of the current business system based on the interpretation index and outputs an analysis result. The interpretation index is a grounds vector 502 whose component is a degree of influence of each of the plurality of feature amounts contained in the data input into the inference unit on the inference result.SELECTED DRAWING: Figure 5

Description

The present invention relates to a system and a method for analyzing the operating environment of a system utilizing AI.

In recent years, systems utilizing AI have been provided in various fields such as medical care and finance. For example, in the medical field, AI is used to predict the incidence of diseases and identify symptoms. In the financial field, AI is used to conduct credit screening. In this specification, a system utilizing AI is referred to as an AI system (business system).

The AI model is generated by machine learning using training data. When the AI system is operated for a long period of time, the operating environment of the AI system changes due to changes in the target and changes in the world situation. Therefore, there is a problem that the expected accuracy of AI prediction is lowered due to a deviation between the assumed operating environment of the AI system and the actual operating environment of the AI system.

Therefore, in order to operate the AI system for a long period of time, it is necessary to detect changes in the operating environment, analyze the factors of the changes, and perform re-learning using the analysis results. In particular, a mechanism for detecting changes in the operating environment is important.

On the other hand, LML (Lifelong Machine Learning), which realizes continuous learning of AI, is attracting attention. LML has a function of detecting the appearance of unprecedented observation data as an environmental change. As a technique using LML, for example, a technique such as Patent Document 1 is known.

Patent Document 1 states that "a hierarchical machine learning system including a separate identification subsystem and an application subsystem, in which the first node of the first layer processes the first input and at least a part of the first input. To generate the first feature vector, and the second node of the second layer processes the second input including at least a part of the first feature vector to generate the second feature vector. The node generates the first sparse feature vector from the first feature vector, the second node generates the second sparse feature vector from the second feature vector, and the third node of the identification subsystem , At least one of the first sparse feature vector and the second sparse feature vector is processed to determine the output. "

U.S. Patent Application Publication No. 2019/0279105

Marco Tulio Ribeiro et al., "Why Should I Trust You?": Explaining the Predictions of Any Classifier, KDD '16 Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, August 2016, Pages 1135-1144 Scott M Lundberg et al., "A Unified Approach to Interpreting Model Predictions", Advances in Neural Information Processing Systems 30, December 2017, Pages 4765-4774 Ramprasaath R.Selvaraju et al., "Grad-CAM: Why did you say that? Visual Explanations from Deep Networks via Gradient-based Localization", [Searched on December 19, 1991], Internet <URL: https: // arxiv .org / abs / 1610.02391v1 ＞

In the prior art, the user needs to interpret the internal information of the model such as the feature vector and detect the change in the operating environment. Since the internal information of the model is not information that directly expresses the relationship with the input data, it is difficult to define the interpretation method. In addition, when re-learning is performed, the model changes, so the interpretation method used so far cannot be used as it is.

That is, since the internal information of the model is not readable, there is a problem that it takes labor and time to use the information. Further, in the prior art, the model needs to have a structure capable of acquiring the internal information of the model, and the applicable model is limited.

The present invention provides a method and a system for outputting information for grasping the operating environment of an AI system having excellent readability without being limited to the structure of the model.

A typical example of the invention disclosed in the present application is as follows. That is, it is a computer system that analyzes the operating environment of a business system having an inference unit that performs inference, and includes at least one computer having a processor, a memory connected to the processor, and a network interface connected to the processor. , An interpretation index calculation unit that calculates an interpretation index for interpreting the inference result obtained by inputting data including a plurality of feature quantities into the reasoning unit, and the current business system based on the interpretation index. It is provided with an analysis unit that analyzes the operating environment of the above and outputs the result of the analysis, and the interpretation index is for the inference result of each of the plurality of feature quantities included in the data input to the inference unit. It is a basis vector whose component is the degree of influence.

According to the present invention, it is possible to output information for grasping the operating environment of an AI system having excellent readability without being limited to the structure of the model. Issues, configurations and effects other than those mentioned above will be clarified by the description of the following examples.

It is a figure which shows the configuration example of the computer system of Example 1. FIG. It is a figure which shows an example of the hardware configuration of the computer of Example 1. FIG. It is a figure which shows an example of the data structure of the case data management information of Example 1. FIG. It is a figure which shows an example of the data structure of the basis vector management information of Example 1. FIG. It is a figure which shows the process flow of the computer system of Example 1. FIG. It is a flowchart explaining an example of the model generation processing executed by the learning part of Example 1. FIG. It is a flowchart explaining an example of the basis vector generation processing executed by the basis vector generation part of Example 1. FIG. It is a flowchart explaining an example of the environment analysis processing executed by the analysis part of Example 1. FIG. It is a figure which shows an example of the screen displayed based on the environment analysis information generated by the result output part of Example 1. FIG. It is a figure which shows an example of the screen displayed based on the environment analysis information generated by the result output part of Example 1. FIG. It is a figure which shows an example of the screen displayed based on the environment analysis information generated by the result output part of Example 1. FIG. It is a figure which shows an example of the screen displayed based on the environment analysis information generated by the result output part of Example 1. FIG. It is a figure which shows the process flow of the computer system of Example 2. FIG. It is a flowchart explaining an example of the environment analysis processing executed by the analysis part of Example 2. It is a figure which shows an example of the screen displayed based on the environment analysis information generated by the result output part of Example 2. FIG. It is a figure which shows the configuration example of the computer system of Example 3. It is a figure which shows the process flow of the computer system of Example 3. FIG. It is a flowchart explaining an example of the environment analysis processing which the analysis part of Example 3 executes. It is a flowchart explaining an example of the algorithm update process executed by the algorithm update part of Example 3. FIG.

Hereinafter, examples of the present invention will be described with reference to the drawings. However, the present invention is not construed as being limited to the description of the embodiments shown below. It is easily understood by those skilled in the art that a specific configuration thereof can be changed without departing from the idea or gist of the present invention.

In the configurations of the invention described below, the same or similar configurations or functions are designated by the same reference numerals, and duplicate description will be omitted.

The notations such as "first", "second", and "third" in the present specification and the like are attached to identify the components, and do not necessarily limit the number or order.

The position, size, shape, range, etc. of each configuration shown in the drawings and the like may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not limited to the position, size, shape, range, etc. disclosed in the drawings and the like.

FIG. 1 is a diagram showing a configuration example of the computer system of the first embodiment.

The computer system is composed of a plurality of computers 100-1, 100-2, 100-3, and a terminal 101. The plurality of computers 100-1, 100-2, 100-3, and the terminal 101 are connected to each other via the network 105. The network 105 is, for example, WAN (Wide Area Network) and LAN (Local Area Network). The connection method of the network 105 may be either wired or wireless.

In the following description, when computers 100-1, 100-2, and 100-3 are not distinguished, they are referred to as computer 100.

The terminal 101 is a computer operated by the user. The terminal 101 is, for example, a personal computer, a smartphone, a tablet terminal, or the like. The terminal 101 inputs evaluation target data 500 (see FIG. 5) and the like necessary for inference by AI (inference unit 111) based on the user's operation. The evaluation target data 500 is composed of values (features) of a plurality of items.

The terminal 101 includes a processor, a memory, a network interface, an input device, and an output device. The input device is a device such as a keyboard, a mouse, and a touch panel, and the output device is a device such as a touch panel and a display.

The computer 100-1 is a computer for constructing an AI (inference unit 111). The computer 100-1 includes a learning unit 110, and also holds case data management information 120.

The learning unit 110 executes a learning process for generating a model (algorithm) set in the inference unit 111. The present invention is not limited to the learning method of the model. Further, the present invention is not limited to the type and structure of the model. For example, the model is a neural network and a decision tree.

The case data management information 120 is information for managing the learning data used in the learning process. The learning data of this embodiment is data generated based on past cases. In the following description, the learning data is also described as case data.

The computer 100-2 is a computer that performs inference using the evaluation target data 500 based on an arbitrary model (algorithm) and outputs the inference result. The inference using the evaluation target data 500 is, for example, classification and inference of an event. The computer 100-2 includes an inference unit 111 that performs inference.

In the learning process, case data is input to the inference unit 111, and in the estimation process, the evaluation target data 500 is input to the inference unit 111. In the following description, when the evaluation target data 500 and the case data are not distinguished, they are described as input data.

The computer 100-3 is a computer that analyzes the operating environment of the AI system using the inference unit 111. The computer 100-3 includes a basis vector generation unit 112, an analysis unit 113, and a result output unit 114, and also holds the basis vector management information 121.

The rationale vector generation unit 112 generates the rationale vector 502 (see FIG. 5) as an index for interpreting the inference result. The basis vector 502 is a vector whose component is the degree of influence of each feature amount included in the data (case data or evaluation target data 500) input to the inference unit 111 on the inference result.

The analysis unit 113 analyzes the operating environment of the AI system using the basis vector 502.

The result output unit 114 generates display data based on the analysis result of the analysis unit 113, and transmits the display data to the terminal 101.

In addition, any one of the computers 100-1, 100-2, and 100-3 has an operation reception unit that provides an API (Application Programming Interface) for receiving a request from the terminal 101.

Here, the hardware configuration of the computer 100 will be described. FIG. 2 is a diagram showing an example of the hardware configuration of the computer 100 of the first embodiment.

The computer 100 has a processor 201, a main storage device 202, a sub storage device 203, and a network interface 204. Each piece of hardware is connected to each other via an internal bus. The computer 100 does not have to have the sub-storage device 203. Further, the computer 100 may have an input device and an output device.

The processor 201 executes a program stored in the main storage device 202. When the processor 201 executes the process according to the program, it operates as a functional unit (module) that realizes a specific function, such as the basis vector generation unit 112. In the following description, when the process is described with the functional unit as the subject, it is shown that the processor 201 is executing the program that realizes the functional unit.

The main storage device 202 stores a program executed by the processor 201 and information used by the program. The main memory 202 also includes a work area that the program temporarily uses.

The main storage device 202 of the computer 100-1 stores a program for realizing the learning unit 110. The main storage device 202 of the computer 100-2 stores a program for realizing the inference unit 111. The main storage device 202 of the computer 100-3 stores a program for realizing the basis vector generation unit 112, the analysis unit 113, and the result output unit 114. Further, the main storage device 202 of any of the computers 100-1, 100-2, and 100-3 stores a program for realizing the operation reception unit.

The sub-storage device 203 permanently stores data such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive).

The sub-storage device 203 of the computer 100-1 stores the case data management information 120. The sub-storage device 203 of the computer 100-2 stores model information (not shown) generated by the learning unit 110. The sub-storage device 203 of the computer 100-3 stores the basis vector management information 121.

Regarding each functional unit included in each computer 100, a plurality of functional units may be combined into one functional unit, or one functional unit may be divided into a plurality of functional units for each function. Further, the functions of each computer 100 may be combined into one computer 100.

FIG. 3 is a diagram showing an example of the data structure of the case data management information 120 of the first embodiment.

The case data management information 120 stores a plurality of entries including the ID 301, the feature amount 302, and the correct answer value 303. One entry corresponds to one case data.

ID301 is a field for storing identification information of case data. A number is stored in the ID 301 of the first embodiment.

The feature amount 302 is a field group for storing the feature amount which is a value of an item constituting the case data. Items are, for example, age, station distance, building materials, and the like.

The correct answer value 303 is a field for storing the correct answer value of the inference for the case data. The value stored in the correct answer value 303 is given in advance.

The user or the computer 100 may register the evaluation target data 500 associated with the correct answer value in the case data management information 120 as case data.

FIG. 4 is a diagram showing an example of the data structure of the basis vector management information 121 of the first embodiment.

The rationale vector management information 121 stores a plurality of entries including ID 401, influence degree 402, inference result 403, and cluster 404. One entry corresponds to one input data basis vector 502.

ID 401 is a field for storing the identification information of the input data.

The influence degree 402 is a group of fields for storing the influence degree indicating the magnitude of the influence of the feature amount of each item on the inference result. In the first embodiment, the vector whose component is the value of each field included in the influence degree 402 is treated as the basis vector 502.

The inference result 403 is a field for storing the inference result output by the inference unit 111.

The cluster 404 is a field for storing the identification information of the cluster to which the input data belongs. As will be described later, clustering is performed based on the basis vector 502.

FIG. 5 is a diagram showing a processing flow of the computer system of the first embodiment.

The arrows in the figure indicate the data flow. The solid line shows the data flow in the process for generating the basis vector 502 of the case data. The dotted line shows the data flow in the process for outputting the inference result of the evaluation target data 500. The alternate long and short dash line indicates the data flow in the process for outputting the analysis result of the operating environment of the AI system.

First, the flow of processing for calculating the basis vector 502 of the case data will be described.

When the operation reception unit receives the generation request of the inference unit 111 from the terminal 101, the operation reception unit outputs a model generation instruction to the computer 100-2. Further, when the operation reception unit receives the request for generating the basis vector 502 of the case data from the terminal 101, the operation reception unit outputs the generation instruction of the basis vector 502 of the case data to the computer 100-3.

When the computer 100-2 receives the generation instruction of the inference unit 111, the computer 100-2 executes the learning process and generates the model to be set in the inference unit 111.

When the basis vector generation unit 112 of the computer 100-3 receives the generation instruction of the basis vector 502 of the case data, it calculates the influence degree of each feature amount included in the case data and generates the basis vector 502. As a method for calculating the degree of influence, it is conceivable to use the techniques of Non-Patent Document 1 to Non-Patent Document 3.

When the calculation method described in Non-Patent Document 2 is adopted, the following processing is executed. The index calculation unit 510 generates calculation data from the case data, inputs the calculation data to the inference unit 111, and acquires the inference result. The distribution unit 511 calculates the degree of influence of each feature amount included in the case data on the inference result (correct answer value) by using the calculation data and the inference result. Further, the distribution unit 511 generates a basis vector 502 having an influence degree of each feature amount as a component and registers it in the basis vector management information 121.

The index calculation unit 510 and the distribution unit 511 refer to the calculation algorithm 520 for calculating the degree of influence, if necessary. The calculation algorithm 520 is, for example, an algorithm that realizes any one of Non-Patent Document 1 to Non-Patent Document 3.

Next, the flow of processing for outputting the inference result of the evaluation target data 500 will be described.

When the operation reception unit receives an inference request including the evaluation target data 500 from the terminal 101, the operation reception unit outputs an inference instruction including the evaluation target data 500 to the computer 100-2. Further, the operation reception unit outputs an instruction to generate the basis vector 502 of the evaluation target data 500 to the computer 100-3.

When the inference unit 111 of the computer 100-2 receives the inference instruction, it makes an inference using the evaluation target data 500 and outputs the inference result 501 to the operation reception unit. The operation reception unit transmits the inference result 501 to the terminal 101.

When the basis vector generation unit 112 of the computer 100-3 receives the calculation instruction of the basis vector 502 of the evaluation target data 500, the basis vector 502 of the evaluation target data 500 is generated. Since the method of generating the basis vector 502 of the evaluation target data 500 is the same as the method of generating the basis vector 502 of the case data, the description thereof will be omitted. The basis vector generation unit 112 registers the basis vector 502 of the evaluation target data 500 in the basis vector management information 121.

Next, the flow of processing for outputting the analysis result of the operating environment of the AI system will be described.

The analysis unit 113 of the computer 100-3 executes the environment analysis process periodically when receiving an environment analysis instruction from the operation reception unit, satisfying the execution conditions, or periodically. For example, the calculation of the basis vector 502 of the evaluation target data 500 can be set as the execution condition. The analysis unit 113 outputs the processing result to the result output unit 114.

The result output unit 114 of the computer 100-3 generates the environment analysis information 503 for displaying the result of the analysis process. The result output unit 114 outputs the environmental analysis information 503 to the operation reception unit. The operation reception unit transmits the environmental analysis information 503 to the terminal 101.

The basis vector 502 can be treated as information indicating the characteristics of the inference unit 111 (model). On the other hand, the degree of influence is not information acquired from the inside of the inference unit 111. That is, the basis vector 502 can be used as highly readable information that can be uniquely interpreted regardless of the model update. Therefore, the computer system of the first embodiment can automatically analyze the operating environment of the AI system by using the basis vector 502 even when the model is updated.

Next, the specific contents of the processing will be described. First, a process for generating a model to be set in the inference unit 111 will be described.

FIG. 6 is a flowchart illustrating an example of a model generation process executed by the learning unit 110 of the first embodiment.

The learning unit 110 acquires case data from the case data management information 120 (step S101). The number and range of case data to be acquired can be set arbitrarily.

The learning unit 110 executes a learning process using the case data (step S102). Since the learning process is a known technique, detailed description thereof will be omitted.

The learning unit 110 determines whether or not the end condition is satisfied (step S103). For example, when the accuracy of the inference using the evaluation data is higher than the threshold value, the learning unit 110 determines that the end condition is satisfied. Further, when the number of executions of the learning process is larger than the threshold value, the learning unit 110 determines that the end condition is satisfied.

If it is determined that the end condition is not satisfied, the learning unit 110 returns to step S102 and executes the same process.

When it is determined that the end condition is satisfied, the learning unit 110 ends the model generation process.

FIG. 7 is a flowchart illustrating an example of the basis vector generation process executed by the basis vector generation unit 112 of the first embodiment. Here, the basis vector generation process will be described using the calculation method described in Non-Patent Document 2 as an example.

The basis vector generation unit 112 acquires the input data (step S201). When calculating the basis vector 502 of the case data, the index calculation unit 510 acquires the case data from the case data management information 120. When calculating the basis vector 502 of the evaluation target data 500, the index calculation unit 510 acquires the evaluation target data 500 included in the generation instruction.

Next, the basis vector generation unit 112 generates calculation data from the input data (step S202).

Specifically, the index calculation unit 510 generates calculation data by changing the feature amount of some items of the input data based on the background data set of the feature amount. Here, it is assumed that K pieces of calculation data are generated from one input data. Identification information is added to each calculation data.

The background data set of the feature quantity is stored in the calculation algorithm 520. Further, a rule for determining the number of feature quantities to be changed, the amount of change, the number of calculation data to be generated, and the like are stored in the calculation algorithm 520.

Next, the basis vector generation unit 112 sets the initial value “1” in the variable k (step S203).

Next, the basis vector generation unit 112 acquires the inference result of the calculation data corresponding to the variable k (step S204).

Specifically, the index calculation unit 510 acquires the inference result by inputting the calculation data corresponding to the variable k into the inference unit 111.

Next, the basis vector generation unit 112 stores a set of calculation data and an inference result in the storage area of the main storage device 202 (step S205).

Next, the basis vector generation unit 112 determines whether or not the variable k matches K (step S206). That is, it is determined whether or not the inference results of all the calculation data have been acquired.

When it is determined that the variable k does not match K, the basis vector generation unit 112 sets the value obtained by adding 1 to the value of the variable k in the variable k (step S207). After that, the basis vector generation unit 112 returns to step S204 and executes the same process.

When it is determined that the variable k matches K, the rationale vector generation unit 112 generates the rationale vector 502 (step S208).

Specifically, the distribution unit 511 performs statistical analysis such as multiple regression analysis using a set of calculation data and inference results to determine the degree of influence of the features of each item of the input data on the inference results. calculate. Further, the distribution unit 511 generates a basis vector 502 whose component is the degree of influence of each feature amount.

Next, the basis vector generation unit 112 registers the basis vector 502 together with the inference result in the basis vector management information 121 (step S209). After that, the basis vector generation unit 112 ends the basis vector generation process. At this point, the cluster 404 of the entry corresponding to the registered rationale vector 502 is blank.

In the calculation method described in Non-Patent Document 3, the index calculation unit 510 calculates the feature map, and the distribution unit 511 multiplies the average value of each coordinate of the feature map by the importance of each class channel. Calculate the sum as the degree of influence.

FIG. 8 is a flowchart illustrating an example of the environment analysis process executed by the analysis unit 113 of the first embodiment.

The analysis unit 113 acquires the basis vector 502 from the basis vector management information 121 (step S301).

Note that the analysis unit 113 may acquire only the basis vector 502 corresponding to the blank entry in the cluster 404.

The analysis unit 113 generates a cluster of input data by executing clustering using the basis vector 502 (step S302).

At this time, the analysis unit 113 updates the value of the cluster 404 of each entry of the basis vector management information 121 based on the result of clustering. Further, the analysis unit 113 generates cluster information based on the result of clustering. The data structure of the cluster information is the same as that of the basis vector management information 121.

Next, the analysis unit 113 sorts the input data based on the sort conditions (step S303). It is assumed that the sort conditions are set in advance. The analysis unit 113 may receive the sort condition from the terminal 101 via the operation reception unit.

As the sorting condition, a time series, a feature amount, an influence degree, an inference result, identification information of input data, and the like can be considered. A sort condition that combines a plurality of viewpoints may be used. Further, a sort condition using the result of clustering may be used.

The analysis unit 113 determines whether or not the cluster information generated in the past exists (step S304). The case where the cluster information generated in the past does not exist is the case where the environment analysis process is executed for the first time.

When it is determined that the cluster information generated in the past does not exist, the analysis unit 113 analyzes the characteristics of each cluster (step S305).

For example, the analysis unit 113 specifies a range of features that have a large influence on the input data belonging to the cluster. Further, the analysis unit 113 identifies the tendency of the inference result of the input data belonging to the cluster.

Next, the analysis unit 113 outputs the cluster information, the sort result, and the analysis result to the result output unit 114 (step S306). After that, the analysis unit 113 ends the environment analysis process. That is, when the environment analysis process is executed for the first time, information indicating the current operating environment of the AI system is output from the result output unit 114.

If it is determined in step S304 that the cluster information generated in the past exists, the analysis unit 113 compares the cluster information generated in the past with the cluster information generated this time, and has a new cluster appeared? It is determined whether or not (step S307).

When a plurality of cluster information generated in the past exists, the analysis unit 113 compares the cluster information with the latest time series with the cluster information generated this time.

When it is determined that no new cluster has appeared, the analysis unit 113 outputs the cluster information and the sort result to the result output unit 114 (step S308). After that, the analysis unit 113 ends the environment analysis process. In this case, information indicating the current operating environment of the AI system is output from the result output unit 114.

The analysis unit 113 may analyze the characteristics of the cluster in which the number of input data to which it belongs has changed, and output the analysis result.

When it is determined that a new cluster has appeared, the analysis unit 113 analyzes the characteristics of the input data belonging to the new cluster (step S309). The analysis method is the same as in step S305.

Next, the analysis unit 113 outputs the cluster information, the sort result, the analysis result, and the environmental change notification to the result output unit 114 (step S310). After that, the analysis unit 113 ends the environment analysis process. In this case, the result output unit 114 outputs information indicating the current operating environment of the AI system and information indicating that a change in the operating environment has been detected.

The analysis unit 113 may analyze the characteristics of the cluster in which the number of input data to which it belongs has changed, and output the analysis result.

In this embodiment, clustering is performed on the feature space centered on the component of the basis vector 502. The basis vector 502 is information representing the behavior of the inference unit 111 with respect to the evaluation target data 500. Therefore, the appearance of a new cluster indicates that the inference unit 111 behaves differently than before. That is, the appearance of a new cluster indicates that the evaluation target data 500, which was not assumed in the operating environment so far, has been input. In this way, by using the basis vector 502, the operating environment of the AI system can be accurately grasped.

Further, since the basis vector 502 is not the information generated from the internal information of the inference unit 111, the handling does not change even if the inference unit 111 is updated. Therefore, no effort or time is required to interpret the rationale vector 502. Furthermore, it is not limited to the structure of the model.

Next, the display based on the information output from the analysis unit 113 will be described.

9, FIG. 10A, FIG. 10B, and FIG. 11 are diagrams showing an example of a screen displayed based on the environmental analysis information 503 generated by the result output unit 114 of the first embodiment.

FIG. 9 shows an example of the screen 900 displayed by the environment analysis information 503 generated based on the cluster information and the environment change notification.

On the screen 900, a graph 910 showing clusters in the feature space centered on the component of the basis vector 502 is displayed. The basis vector 502 is plotted on the graph 910. Further, in the graph 910, three clusters 911-1, 911-2, and 911-3 are displayed based on the cluster information. A blowout 912 is displayed on each cluster 911 based on the analysis result.

Clusters 911-1 and 911-2 are existing clusters. Cluster 911-3 is a new cluster. Cluster 911-3 is highlighted based on the environmental change notification.

On the screen displayed by the environment analysis information 503 generated only from the cluster information, all the clusters 911 are displayed in the same display format.

10A and 10B show an example of the screen 1000 displayed by the environmental analysis information 503 generated based on the sort result.

A graph 1010 showing the sort result is displayed on the screen 1000. The vertical axis of the graph 1010 in FIG. 10A shows the estimation result, and the horizontal axis shows the cluster. The vertical axis of the graph 1010 in FIG. 10B shows the estimation result, and the horizontal axis shows the time.

In the graph 1010 of FIG. 10A, the sorting result of the input data using the icon 1011 representing the basis vector 502 of the input data is displayed. For the icon 1011 the degree of influence is represented by the size of the area within the icon 1011. The white portion of the icon 1011 represents the degree of positive influence, and the shaded portion represents the degree of negative influence. The size of the icon 1011 is determined based on the inference result. In the graph 1010, the blowout 1012 is displayed based on the analysis result. Further, the icon 1011 may be given information indicating the cluster to which the icon 1011 belongs. In FIG. 10A, the outer frame of the icon 1011 is changed according to the cluster to which it belongs.

The user can grasp the characteristics of the input data in the cluster based on the screen 1000 of FIG. 10A.

The icon 1011 and the pointer 1013 are displayed in the graph 1010 of FIG. 10B. The pointer 1013 indicates the estimated time when the operating environment has changed.

The user can grasp the time when the operating environment has changed based on the screen 1000 of FIG. 10B.

FIG. 11 shows an example of the screen 1100 displayed by the environment analysis information 503 generated based on the cluster information.

On the screen 1100, the icon 1101 representing the basis vector 502 of the input data is displayed. When the user moves the pointer 1102 to a predetermined area, the feature amount and the degree of influence of the input data in the area are displayed.

The screens described with reference to FIGS. 9, 10A, 10B, and 11 are merely examples and are not limited thereto. Any screen that can present similar information will do.

As described above, the computer system of the first embodiment can present information about the cluster generated by the clustering using the basis vector 502. The basis vector 502 is information indicating the relationship between the input data and the inference unit 111, and is excellent in readability. Therefore, the information about the cluster can be treated as highly readable information for grasping the operating environment of the AI system.

Further, in the prior art, it is difficult to automatically detect environmental changes because it is necessary to change the interpretation method of internal information as the model is updated. On the other hand, in the first embodiment, since the basis vector 502 that does not depend on the internal information of the model and shows the behavior of the model is used, it is possible to automatically detect the environmental change.

The computer system of the second embodiment detects a decrease in the inference accuracy of the inference unit 111. Hereinafter, Example 2 will be described with a focus on the differences from Example 1.

The system configuration of the second embodiment is the same as that of the first embodiment. The hardware configuration and software configuration of the computer 100 of the second embodiment are the same as those of the first embodiment. The information handled in the second embodiment is the same as that in the first embodiment. Further, the processing executed by the learning unit 110, the inference unit 111, the basis vector generation unit 112, and the result output unit 114 of the second embodiment is the same as that of the first embodiment.

In the second embodiment, the processing executed by the analysis unit 113 is partially different. FIG. 12 is a diagram showing a processing flow of the computer system of the second embodiment.

In the second embodiment, the correct answer data 504 associated with the evaluation target data 500 is input to the analysis unit 113. The correct answer data 504 includes the identification information of the evaluation target data 500 and the correct answer of the reasoning for the evaluation target data 500.

The correct answer data 504 may be input before the execution of the environment analysis process, or may be input when the determination result in step S307 is YES.

FIG. 13 is a flowchart illustrating an example of the environment analysis process executed by the analysis unit 113 of the second embodiment.

The processing flow from step S301 to step S309 is the same as that in the first embodiment. After the process of step S309 is executed, the analysis unit 113 calculates the inference error of the evaluation target data 500 belonging to the new cluster (step S351).

When the inference result output by the inference unit 111 is a numerical value, the analysis unit 113 calculates the difference between the inference result and the correct answer as an inference error. When the inference result output by the inference unit 111 is a value other than a numerical value, the analysis unit 113 sets "1" if the inference result and the correct answer match, and "0" if the inference result and the correct answer do not match. Calculate as. The above-mentioned calculation method is an example and is not limited to this.

Next, the analysis unit 113 determines whether or not the average value of the inference error is equal to or greater than the threshold value (step S352). It is assumed that the threshold value is set in advance. That is, it is determined whether or not the inference accuracy of the inference unit 111 is lowered.

If it is determined that the average value of the inference error is smaller than the threshold value, the analysis unit 113 proceeds to step S310. The process of step S310 of the second embodiment is the same as that of the first embodiment.

When it is determined that the average value of the inference error is equal to or greater than the threshold value, the analysis unit 113 outputs the cluster information, the sort result, the analysis result, the environmental change notification, and the alert to the result output unit 114 (step S353). After that, the analysis unit 113 ends the environment analysis process. In this case, the result output unit 114 outputs information indicating the current operating environment of the AI system, information indicating that a change in the operating environment has been detected, and information indicating that the inference accuracy of the inference unit 111 has deteriorated. ..

The analysis unit 113 may analyze the characteristics of the cluster in which the number of input data to which it belongs has changed, and output the analysis result.

In the analysis of the characteristics of the cluster, the analysis unit 113 may calculate the average value of the inference errors of the evaluation target data 500 belonging to the cluster.

FIG. 14 is a diagram showing an example of a screen displayed based on the environment analysis information 503 generated by the result output unit 114 of the second embodiment.

FIG. 14 shows an example of the screen 900 displayed by the environment analysis information 503 generated based on the cluster information, the environment change notification, and the alert.

In the second embodiment, the outlet 912 includes the average value of the inference error of the evaluation target data 500 belonging to the cluster 911. If the average value of the inference error is equal to or greater than the threshold value, the average value of the inference error of the outlet 912 is highlighted based on the alert.

According to the second embodiment, it is possible to output highly readable information for grasping the operating environment of the AI system. In addition, the computer system can detect a decrease in inference accuracy. This makes it possible to automate re-learning and recommend re-learning to users.

The computer system of the third embodiment updates the calculation method (calculation algorithm 520) of the basis vector 502 in accordance with the change in the operating environment of the AI system. Hereinafter, Example 3 will be described with a focus on the differences from Example 1.

FIG. 15 is a diagram showing a configuration example of the computer system of the third embodiment.

As shown in FIG. 15, the system configuration of the third embodiment is the same as that of the first embodiment. The hardware configuration of the computer 100 of the third embodiment is the same as that of the first embodiment. The software configurations of the computers 100-1 and 100-2 of the third embodiment are the same as those of the first embodiment.

In the third embodiment, the software configuration of the computer 100-3 is partially different. Specifically, the computer 100-3 of the third embodiment includes an algorithm update unit 115 that updates the calculation algorithm 520.

The processing executed by the learning unit 110, the inference unit 111, the basis vector generation unit 112, and the result output unit 114 of the third embodiment is the same as that of the first embodiment.

In the third embodiment, the processing executed by the analysis unit 113 is partially different. FIG. 16 is a diagram showing a processing flow of the computer system of the third embodiment.

In the third embodiment, the analysis unit 113 outputs the analysis result of the input data belonging to the cluster to the algorithm update unit 115. The algorithm update unit 115 updates the calculation algorithm 520 based on the analysis result.

FIG. 17 is a flowchart illustrating an example of the environment analysis process executed by the analysis unit 113 of the third embodiment.

The processing flow from step S301 to step S309 is the same as that in the first embodiment. After the process of step S309 is executed, the analysis unit 113 outputs the analysis result to the algorithm update unit 115 (step S361). Then, the process proceeds to step S310. The process of step S310 of Example 3 is the same as that of Example 1.

FIG. 18 is a flowchart illustrating an example of the algorithm update process executed by the algorithm update unit 115 of the third embodiment.

When the algorithm update unit 115 receives the analysis result from the analysis unit 113, the algorithm update unit 115 executes the algorithm update process.

The algorithm update unit 115 acquires the definition information of the calculation algorithm 520 from the basis vector generation unit 112 (step S401).

The algorithm update unit 115 updates the definition information of the calculation algorithm 520 based on the analysis result (step S402).

The algorithm update unit 115 outputs the updated definition information of the calculation algorithm 520 to the basis vector generation unit 112 (step S403). After that, the algorithm update unit 115 ends the algorithm update process.

Here, a method of updating the calculation algorithm 520 will be described using a specific example. Here, a method of updating the calculation algorithm 520 that realizes the processing described in Non-Patent Document 2 and the calculation algorithm 520 that realizes the processing described in Non-Patent Document 3 will be described.

(1) Non-Patent Document 2
In the process described in Non-Patent Document 2, calculation data is generated using a background data set. When the process described in Non-Patent Document 2 is adopted, the background data set is included in the calculation algorithm 520.

Here, it is assumed that the average value of each feature is defined as a background data set. In this case, the basis vector generation unit 112 generates calculation data by increasing or decreasing each feature amount within the range of the average value. For example, the feature quantity of the variable x ₁ of the input data is 8, when the average value is 5, basis vector generation unit 112, the calculation data obtained by changing the characteristics of the variable x ₁ in the range 3 13 Generate.

The algorithm update unit 115 calculates the average value of each feature amount based on the analysis result. The algorithm update unit 115 updates the calculation algorithm 520 by reflecting the calculation result in the background data set.

(2) Non-Patent Document 3
Classification score S ^c to each class of feature map of CNN output is calculated by the equation (1). C represents a class. A _ij represents a feature map and w _k ^c represents a weight. Further, i and j represent the coordinates of the pixel. Z represents the size of the feature map.

The weight is understood as the importance of equation (2) for classifying the input data as class C.

Here, the order of the finite sum of the equation (1) is changed and transformed as in the equation (3).

At this time, the term in parentheses in the equation (3) can be understood as the magnitude of the influence that the pixel (i.j) is classified into the class C. Here, the weight can be calculated as in the equation (4). Here, y ^C represents the inference result 501 of the inference unit 111.

In Non-Patent Document 3, the degree of influence (importance) as shown in the equation (5) is visualized as a weighted sum of each channel of the feature map.

As shown in equation (4), the weight is affected by the operating environment of the AI system, especially the change in the distribution of features. Therefore, the algorithm update unit 115 corrects the weight based on the change in the distribution of the feature amount.

Specifically, the algorithm update unit 115 calculates the average value of each feature amount based on the analysis result. The algorithm update unit 115 generates calculation data from the input data based on the calculation result. The algorithm update unit 115 inputs the calculation data to the inference unit 111 and acquires the inference result 501.

Algorithm updating unit 115 corrects the weighting using the inference results 501 of the input data ^{(y c)} and the inference result 501 (y ^'c) of the calculation data. Specifically, the weight is calculated using the equation (6).

Incidentally, the algorithm updating unit 115 generates different input data and the calculation data pairs, by plotting the y ^c -y ^'c space may calculate the slope of the equation (7).

According to the third embodiment, the calculation algorithm 520 for calculating the basis vector 502 can be automatically updated according to the change in the operating environment of the AI system. As a result, it is not necessary to develop an algorithm for monitoring the operating environment of the AI system and define an interpretation.

The present invention is not limited to the above-described examples, and includes various modifications. Further, for example, the above-described embodiment describes the configuration in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the described configurations. In addition, a part of the configuration of each embodiment can be added, deleted, or replaced with another configuration.

Further, each of the above configurations, functions, processing units, processing means and the like may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. The present invention can also be realized by a program code of software that realizes the functions of the examples. In this case, a storage medium in which the program code is recorded is provided to the computer, and the processor included in the computer reads out the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and the program code itself and the storage medium storing the program code itself constitute the present invention. Examples of the storage medium for supplying such a program code include a flexible disk, a CD-ROM, a DVD-ROM, a hard disk, an SSD (Solid State Drive), an optical disk, a magneto-optical disk, a CD-R, and a magnetic tape. Non-volatile memory cards, ROMs, etc. are used.

In addition, the program code that realizes the functions described in this embodiment can be implemented in a wide range of programs or script languages such as assembler, C / C ++, perl, Shell, PHP, and Java (registered trademark).

Further, by distributing the program code of the software that realizes the functions of the examples via the network, it is stored in a storage means such as a hard disk or memory of a computer or a storage medium such as a CD-RW or a CD-R. , The processor provided in the computer may read and execute the program code stored in the storage means or the storage medium.

In the above-described embodiment, the control lines and information lines show what is considered necessary for explanation, and do not necessarily indicate all the control lines and information lines in the product. All configurations may be interconnected.

100 Computer 101 Terminal 105 Network 110 Learning unit 111 Reasoning unit 112 Ground vector generation unit 113 Analysis unit 114 Result output unit 115 Algorithm update unit 120 Case data management information 121 Ground vector management information 201 Processor 202 Main storage device 203 Sub storage device 204 Network Interface 500 Evaluation target data 501 Inference result 502 Basis vector 503 Environmental analysis information 504 Correct answer data 510 Index calculation unit 511 Distribution unit 520 Calculation algorithm 900, 1000, 1100 screens

Claims (14)

A computer system that analyzes the operating environment of a business system that has an inference unit that makes inferences.
Includes a processor, memory connected to the processor, and at least one calculator having a network interface connected to the processor.
An interpretation index calculation unit that calculates an interpretation index for interpreting the inference result obtained by inputting data including a plurality of features into the inference unit, and an interpretation index calculation unit.
It is provided with an analysis unit that analyzes the current operating environment of the business system based on the interpretation index and outputs the result of the analysis.
The computer system is characterized in that the interpretation index is a basis vector whose component is the degree of influence of each of the plurality of feature quantities included in the data input to the inference unit on the inference result. The computer system according to claim 1.
The analysis unit
Perform clustering using the rationale vector
Based on the result of the clustering, it is determined whether or not a change in the operating environment of the business system has occurred.
A computer system characterized by outputting the result of the clustering and the result of the determination. The computer system according to claim 2.
The analysis unit
Cluster information is generated based on the result of the clustering.
By comparing the cluster information generated in the past with the newly generated cluster information, it is determined whether or not a new cluster has appeared.
A computer system characterized in that when a new cluster appears, it is determined that a change in the operating environment of the business system has occurred. The computer system according to claim 3.
The analysis unit
By analyzing the accuracy of the inference result of the data belonging to the new cluster, it is determined whether or not the inference accuracy of the inference unit is lowered.
A computer system characterized in that when it is determined that the inference accuracy of the inference unit is reduced, information for notifying the decrease in the inference accuracy of the inference unit is output. The computer system according to claim 2.
A computer system characterized in that the analysis unit sorts the data based on sort conditions and outputs the result of the sort. The computer system according to claim 5.
A computer system characterized in that the sort condition is at least one of the feature amount, the basis vector, the time series of the data, the cluster to which the data belongs, the inference result, and the accuracy of the inference result. The computer system according to claim 1.
The interpretation index calculation unit calculates the interpretation index based on the calculation algorithm.
The analysis unit is a computer system characterized in that the calculation algorithm is updated based on the result of the analysis. It is an analysis method of the operating environment of a business system that has an inference unit that performs inference, which is executed by a computer system.
The computer system includes a processor, a memory connected to the processor, and at least one computer having a network interface connected to the processor.
The method of analyzing the operating environment of the business system is as follows.
A first step in which the at least one computer calculates an interpretation index for interpreting an inference result obtained by inputting data including a plurality of feature quantities into the inference unit.
The at least one computer analyzes the current operating environment of the business system based on the interpretation index, and includes a second step of outputting the result of the analysis.
The interpretation index is a basis vector whose component is the degree of influence of each of the plurality of feature quantities included in the data input to the inference unit on the inference result. analysis method. The method for analyzing the operating environment of the business system according to claim 8.
The second step is
A third step in which the at least one computer performs clustering using the rationale vector,
A fourth step in which the at least one computer determines whether or not a change in the operating environment of the business system has occurred based on the result of the clustering.
A method for analyzing an operating environment of a business system, wherein the at least one computer includes a fifth step of outputting the result of the clustering and the result of the determination. The method for analyzing the operating environment of the business system according to claim 9.
The third step includes a step in which the at least one computer generates cluster information based on the result of the clustering.
The fourth step is
A step in which the at least one computer determines whether or not a new cluster has appeared by comparing the cluster information generated in the past with the newly generated cluster information.
A method for analyzing the operating environment of a business system, which comprises a step of determining that a change in the operating environment of the business system has occurred when the new cluster appears. The method for analyzing the operating environment of the business system according to claim 10.
The fourth step is
A step in which the at least one computer analyzes the accuracy of the inference result of the data belonging to the new cluster to determine whether or not the inference accuracy of the inference unit is lowered.
When it is determined that the inference accuracy of the inference unit is low, the at least one computer includes a step of outputting information notifying that the inference accuracy of the inference unit is low. How to analyze the operating environment of the business system to be used. The method for analyzing the operating environment of the business system according to claim 9.
The second step is
A step in which the at least one computer sorts the data based on the sorting conditions.
A method for analyzing an operating environment of a business system, wherein the at least one computer includes a step of outputting the result of the sort. The method for analyzing the operating environment of the business system according to claim 12.
The operation of the business system, wherein the sort condition is at least one of the feature amount, the basis vector, the time series of the data, the cluster to which the data belongs, the inference result, and the accuracy of the inference result. How to analyze the environment. The method for analyzing the operating environment of the business system according to claim 8.
The first step includes a step in which the at least one computer calculates the interpretation index based on a calculation algorithm.
The analysis method is a method for analyzing an operating environment of a business system, wherein the at least one computer includes a step of updating the calculation algorithm based on the result of the analysis.

Images