JP2021149893A - Neural network analysis apparatus, neural network analysis method, and program - Google Patents

Abstract

To provide a neural network analysis apparatus, a neural network analysis method, and a program that facilitate interpretation of a learned model while suppressing deterioration of prediction accuracy.SOLUTION: A neural network analysis apparatus according to an embodiment comprises a node used index pattern acquisition unit, and a clustering unit. The node used index pattern acquisition unit acquires a node used index pattern being information indicating a node used index that is a value indicating a degree for a node of a neural network to be used according to an input into an input layer for each node of an input layer, an intermediate layer and an output layer of the neural network, for each test data that is input into a learned model represented by an analysis object neural network which is a neural network of an analysis object. The clustering unit performs clustering of the test data and the node used index pattern on the basis of the node used index pattern acquired by the node used index pattern acquisition unit.SELECTED DRAWING: Figure 1

Description

Embodiments of the present invention relate to a neural network analyzer, a neural network analysis method and a program.

In recent years, it has become popular to perform machine learning using a neural network and predict the result using a trained model. However, since the trained model generated by machine learning using a neural network is a non-linear model, it may be difficult to interpret the trained model. Interpreting the trained model means knowing the basis of the prediction result of the trained model.

Therefore, a method of approximating the trained model with a linear model (see Patent Document 1) and a method of approximating the trained model so as to simplify the combination of nonlinear models (see Non-Patent Documents 1 and 2) are used. was suggested. However, none of these methods was a method for analyzing the state of the neural network nodes, such as whether or not each node was activated. Therefore, the methods proposed so far may not be able to achieve both prediction accuracy and ease of interpretation.

JP-A-2018-169960

Leo Breiman and Nong Shang, "BORN AGAIN TREES", [online], [Searched on March 19, 2nd year of Reiwa], Internet <URL: https://www.stat.berkeley.edu/users/breiman/BAtrees. pdf> Marco Tulio Ribeiro, Sameer Singh and Carlos Guestrin, "Why Should I Trust You?" Explaining the Predictions of Any Classifier, [online], [Search on March 19, 2nd year of Reiwa], Internet <URL: https: // www.kdd.org/kdd2016/papers/files/rfp0573-ribeiroA.pdf>

An object to be solved by the present invention is to provide a neural net analysis device, a neural net analysis method, and a program that facilitate interpretation of a trained model while suppressing a decrease in prediction accuracy.

The neural network analysis device of the embodiment has a node used index pattern acquisition unit and a clustering unit. The node used index pattern acquisition unit is a value indicating the degree to which the node of the neural network is used according to the input to the input layer. The node used index is the node used index of each node of the input layer, the intermediate layer, and the output layer of the neural network. The node usage index pattern, which is the information indicating the above, is acquired for each test data input to the trained model represented by the neural network to be analyzed, which is the neural network to be analyzed. The clustering unit clusters the test data and the node used index pattern based on the node used index pattern acquired by the node used index pattern acquisition unit.

The first explanatory diagram explaining the outline of the neural network analysis apparatus 1 of an embodiment. The second explanatory diagram explaining the outline of the neural network analysis apparatus 1 of an embodiment. The figure which shows an example of the hardware composition of the neural network analysis apparatus 1 of embodiment. The figure which shows an example of the functional structure of the control part 10 in embodiment. FIG. 5 is a flowchart showing an example of a processing flow in the analysis processing initialization stage executed by the control unit 10 in the embodiment. FIG. 5 is a flowchart showing an example of a processing flow in the analysis processing update stage executed by the control unit 10 in the embodiment.

Hereinafter, the neural network analysis device, the neural network analysis method, and the program of the embodiment will be described with reference to the drawings. First, the outline of the neural network analysis apparatus of the embodiment will be described with reference to FIGS. 1 and 2.

FIG. 1 is a first explanatory diagram illustrating an outline of the neural network analysis device 1 of the embodiment. The neural network analysis device 1 uses a trained model represented by the neural network to be analyzed (hereinafter referred to as “analysis target model”) as a linear model based on a node usage index pattern when test data is input to the analysis target model. Approximate with. Hereinafter, the neural network to be analyzed is referred to as an analysis target neural network.

The node usage index pattern is information indicating the node usage index for each node of the input layer, the intermediate layer, and the output layer of the neural network. The node usage index is a value indicating the degree to which a node is used according to the input to the input layer. The node usage index is, for example, the value of the slope of the activation function of the node. For example, the node usage index may be the slope of the linear function in the node where the relationship between the input and the output is represented by the linear function. The node usage index is represented by, for example, the following equations (1) and (2).

In equations (1) and (2), σ is an activation function with f (x) as a variable. f (x) is a function representing a value input to a node, and is a function that outputs a value corresponding to x. x is the output value of the node of the previous layer. Hereinafter, x is referred to as a previous node output value. The previous node output value x is the value itself input to the input layer in the input layer, which is a layer in which the previous layer does not exist. The value input to the input layer is, for example, a feature amount of data input to the analysis target model. In equations (1) and (2), w and b are weight parameters in the neural network to be analyzed. In equations (1) and (2), x, w and b are represented by tensors such as matrices and vectors.

Hereinafter, for the sake of simplicity, the neural network analysis device 1 will be described by taking the case where the node usage index is the value of the slope of the activation function of the node as an example.

The node usage index is, for example, a value of 0 or 1 when the activation function of the node is ReLU. When the activation function is ReLU, the node usage index of 1 means that the node has been activated.

The node usage index of the node of the input layer is a value that has a bijective relationship with the value input to the node. For example, the node usage index of a node in the input layer is the same value as the value input to the node. For the sake of simplicity of the following description, the neural network analysis device 1 will be described by taking the case where the node usage index of the node of the input layer is the same value as the value input to the node.

The node usage index of the intermediate layer and the output layer is not necessarily a value that has a bijective relationship with the value input to the node. For example, when the activation function of the nodes of the intermediate layer and the output layer is ReLU, there are two candidates for the node usage index of the nodes of the intermediate layer and the output layer, 0 or 1. In such a case, for example, the condition for the node used index to be 1 is the condition that the value input to the node is larger than 0, and the condition for the node used index to be 0 is input to the node. The condition is that the value is 0 or less. As described above, the node usage index of the intermediate layer and the output layer is not necessarily a value having a bijective relationship with the value input to the node.

In the neural network, the output of the node in the first stage (hereinafter referred to as "node output value") is input to the node in the second stage. For example, the node output value of the node of the input layer is input to the node of the first layer of the intermediate layer. For example, the node output value of the node of the nth layer of the intermediate layer is input to the node of the (n + 1) th layer of the intermediate layer (n is a natural number). For example, the node output value of the node of the last layer of the middle layer is input to the node of the output layer.

FIG. 2 is a second explanatory diagram illustrating an outline of the neural network analysis device 1 of the embodiment. FIG. 2 shows four types of node usage index patterns P1, P2, P3 and P4. The neural network to be analyzed showing the node usage index pattern in FIG. 2 has one node in the output layer, and the value candidates indicated by the node usage index of the node in the output layer are 4 such as 0, 1, 2, and 3. Two values.

In FIG. 2, the node usage index of the node of the output layer of the node usage index patterns P1, P2, P3 and P4 is different from each other. The line connecting the node of the first layer and the node of the second layer (hereinafter referred to as "node connection line") is an image line for explanation described for simplicity of explanation. The node connection line indicates the node of the previous layer that is dominant in determining the node usage index of the node of the lower layer. The dominant node in the first layer is a node in which a change in the node usage index causes a change of a predetermined magnitude or more in the node usage index of the node in the second layer.

The neural network to be analyzed in FIG. 2 determines which of the four types of node-used index patterns P1, P2, P3, and P4 is appropriate according to the input value. The analysis target neural network of FIG. 2 determines the output of the analysis target neural network using the node usage index pattern of the determination result. Since there are four types of node-used index patterns in FIG. 2, it is highly possible that the user of the neural network to be analyzed can easily interpret the operation of the neural network. However, as the number of types of node-used index patterns increases, it becomes more difficult to interpret the operation of the neural network to be analyzed. Therefore, the neural network analysis device 1 classifies the node usage index pattern of the neural network to be analyzed.

In the case of the neural network to be analyzed in FIG. 2, for example, the neural network analysis device 1 sets the node-used index pattern P1 and the node-used index pattern P3 to be the same type of node-used index pattern. Classify patterns into one or more sets. After the classification, the neural network analysis device 1 acquires the node index information for each set. The node index information is information indicating a node index for each node of the neural network to be analyzed. The node index is an index representing the node probability distribution. The node probability distribution is a probability distribution for each node of the neural network to be analyzed, and is a probability distribution using the node usage index as a random variable.

The node index is, for example, the expected value of the node probability distribution. The node index may be, for example, the median value of the node probability distribution. The node index may be, for example, an index represented by the distance function of the Kullback-Leibler. In FIG. 2, the information EV1 is the node index information in the set including the node used index pattern P1 and the node used index pattern P3. Therefore, the values of the five nodes of the information EV1 in FIG. 2 are, for example, the expected values of the node probability distribution in each node. Specifically, the probability represented by the probability distribution is the probability (burden rate) of the appearance of each node usage index pattern in each cluster.

The neural network analysis device 1 generates one linear model representing the distribution of the node used index parameters belonging to the set for each set after classification based on the node index information. After the linear model is generated, the neural network analysis device 1 updates the generated linear model every time test data is input. Hereinafter, a linear model generated for each set is referred to as a representative linear model. The set of representative linear models of each set is the approximation result model.

In this way, the neural network analysis device 1 classifies the node-used index patterns of the neural network to be analyzed and reduces the types of the node-used index patterns. Then, the linear model for each type is acquired. Therefore, the neural network analysis device 1 facilitates the user's interpretation of the operation of the neural network.

The linear model is a model (that is, a linear map) in which the output value corresponding to the sum of a plurality of inputs matches the sum of the output values of each input. The neural network analysis device 1 may approximate the analysis target model with one linear model, or may approximate it with a plurality of linear models. Approximating the model to be analyzed with a linear model means acquiring a linear model (hereinafter referred to as "approximation result model") in which the relationship between the input and the output is somewhat similar to the model to be analyzed.

The test data is the data input to the analysis target model. The test data may be any data as long as it can be input to the analysis target model.

Returning to the description of FIG. In FIG. 2, the outline of the operation of the neural network analysis device 1 has been described using four neural networks in which the candidates for the node usage index of the node of the output layer are four. However, the more candidates for the node usage index of the node, the more difficult it is to explain. Therefore, for the sake of simplicity of the explanation below, the neural network analysis device 1 will be described by taking the case where there are two candidates for the node usage index of the node as an example. .. The neural network in which there are two candidates for the node usage index of the node is specifically a neural network that satisfies the following input information condition, input layer condition, and ReLU firing condition.

The input condition is that the information input to the neural network indicates a value input to each node provided in the input layer, and the value input to each node is 0 or 1. The information input to the neural network is, for example, test data.

The input layer condition is a condition that the node usage index of each node of the input layer is the same as the value input to each node.

The ReLU firing condition is a condition that the activation function of the nodes in the intermediate layer and the output layer is ReLU. Therefore, the ReLU ignition condition includes a condition that the node used index shows 1 when the input value is larger than 0, and the node used index shows 0 when the input value is 0 or less.

Hereinafter, among the nodes in the intermediate layer, the node showing a node usage index of 1 is referred to as an intermediate layer excitation node. Hereinafter, among the nodes of the output layer, the node showing a node usage index of 1 is referred to as an output layer excitation node. Hereinafter, among the nodes in the intermediate layer, the node in which the node usage index shows 0 is referred to as an intermediate layer non-excited node. Hereinafter, among the nodes of the output layer, the node whose node usage index shows 0 is referred to as an output layer non-excited node.

Hereinafter, among the nodes of the input layer, the node whose node usage index is 1 is referred to as an input layer excitation node. Hereinafter, among the nodes of the input layer, the node whose node usage index is 0 is referred to as an input layer non-excited node.

Hereinafter, for the sake of simplicity, a node that is an input layer excitation node, an intermediate layer excitation node, or an output layer excitation node is referred to as an excitation node. That is, the excited node is a node having a node usage index of 1. Hereinafter, for the sake of simplicity, a node that is an input layer non-excited node, an intermediate layer non-excited node, or an output layer non-excited node is referred to as a non-excited node. That is, the non-excited node is a node having a node usage index of 0.

Since the excitation pattern is information indicating the node usage index of each node of the input layer, the intermediate layer, and the output layer of the neural network, the node usage when the input information condition, the input layer condition, and the ReLU ignition condition are satisfied. The index pattern is information indicating the excitation node.

The node used index pattern Ex1 and the node used index pattern Ex2 in FIG. 1 are examples of node used index patterns in a neural network that satisfy the input information condition, the input layer condition, and the ReLU firing condition, respectively. The node used index pattern Ex1 and the node used index pattern Ex2 differ from each other in at least one of an input layer excitation node, an intermediate layer excitation node, and an output layer excitation node. In this way, the node usage index pattern has a bijective relationship with the information input to the input layer such as test data.

FIG. 3 is a diagram showing an example of the hardware configuration of the neural network analysis device 1 of the embodiment. The neural net analysis device 1 includes a control unit 10 including a processor 91 such as a CPU (Central Processing Unit) connected by a bus and a memory 92, and executes a program. The neural network analysis device 1 functions as a device including a control unit 10, a communication unit 11, a storage unit 12, and a user interface 13 by executing a program.

More specifically, in the neural network analysis device 1, the processor 91 reads out the program stored in the storage unit 12, and stores the read program in the memory 92. When the processor 91 executes the program stored in the memory 92, the neural net analysis device 1 functions as a device including the control unit 10, the communication unit 11, the storage unit 12, and the user interface 13.

The control unit 10 controls the operation of each functional unit included in the neural network analysis device 1. The control unit 10 executes, for example, a process of acquiring an approximation result model of the analysis target model. The control unit 10 records the acquired approximation result model in the storage unit 12. The control unit 10 controls, for example, the operation of the communication unit 11.

The communication unit 11 includes a communication interface for connecting the neural network analysis device 1 to an external device. The communication unit 11 transmits, for example, the approximation result model to the external device of the communication destination. The external device is, for example, a printer that outputs an approximation result model. The communication unit 11 acquires test data from, for example, an external device.

The storage unit 12 is configured by using a storage device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 12 stores various information related to the neural network analysis device 1. The storage unit 12 stores, for example, a program for controlling the operation of each functional unit included in the neural network analysis device 1 in advance. The storage unit 12 stores, for example, information representing the neural network to be analyzed in advance. The information representing the analysis target model includes, for example, hyperparameters of the neural network representing the analysis target model. The storage unit 12 stores, for example, an approximation result model. The storage unit 12 stores, for example, information indicating each set that classifies the node usage index pattern. The cluster described later is an example of a set that classifies node usage index patterns. The storage unit 12 stores, for example, a representative linear model. The storage unit 12 stores the node usage index pattern.

The user interface 13 includes an input unit 131 that receives input to the neural network analysis device 1, and an output unit 132 that displays various information about the neural network analysis device 1. The user interface 13 is, for example, a touch panel. The input unit 131 receives an input to its own device. The input unit 131 is an input terminal such as a mouse, a keyboard, or a touch panel. The input unit 131 may be configured as, for example, an interface for connecting these input terminals to its own device. The input received by the input unit 131 is, for example, information representing the analysis target model. The input received by the input unit 131 is, for example, test data.

The output unit 132 is a display device such as a liquid crystal display or an organic EL (Electro Luminescence) display. The output unit 132 may be configured as, for example, an interface for connecting these display devices to its own device. The output unit 132 may be an audio output device such as a speaker. The information output by the output unit 132 is, for example, an approximation result model. Hereinafter, for the sake of simplicity, the neural network analysis device 1 will be described by taking the case where the output unit 132 is a display device as an example.

FIG. 4 is a diagram showing an example of the functional configuration of the control unit 10 in the embodiment.

The control unit 10 includes a test data acquisition unit 101, a node used index pattern acquisition unit 102, a clustering unit 103, a node index information acquisition unit 104, a representative linear model acquisition unit 105, an operation control unit 106, and an output control unit 107.

The test data acquisition unit 101 acquires test data via the communication unit 11 or the user interface 13.

The node used index pattern acquisition unit 102 inputs the test data into the analysis target model, and acquires the node used index pattern corresponding to the input test pattern.

The clustering unit 103 includes an initialization stage clustering unit 301 and an update stage clustering unit 302.

The initialization stage clustering unit 301 clusters test data into a predetermined number of K clusters (K is an integer) based on the node used index pattern acquired by the node used index pattern acquisition unit 102. Since the node-used index pattern has a bijective relationship with the test data, the node-used index pattern is also clustered by clustering the test data based on the node-used index pattern. In this way, the initialization stage clustering unit 301 clusters the test data and the node used index pattern based on the node used index pattern acquired by the node used index pattern acquisition unit 102.

The update stage clustering unit 302 operates when at least the condition that K clusters for classifying test data have already been generated is satisfied. That is, the update stage clustering unit 302 operates when a predetermined condition including the condition that K clusters for classifying the test data have already been generated is satisfied. The update stage clustering unit 302 clusters all the test data input to the analysis target model including the additional test data into a plurality of clusters based on the output of the analysis target model when additional test data is input. ..

For example, the update stage clustering unit 302 clusters all of the test data and the node usage index pattern into a plurality of clusters by executing the belonging cluster determination process. In the affiliation cluster determination process, the update stage clustering unit 302 first outputs the output result of the analysis target model when additional test data is input and the output of the representative linear model of each cluster when additional test data is input. Compare with the results. The output result of the representative linear model is, for example, the node output value of each node in the output layer of the representative linear model.

The output result of the analysis target model when the additional test data is input is, for example, the node output value of each node of the output layer of the analysis target model when the additional test data is input.

In the affiliation cluster determination process, the update stage clustering unit 302 then determines that the cluster to which the highly similar model belongs is the cluster to which the additional test data belongs. The highly similar model is a representative linear model that outputs the output result with the smallest difference from the output result of the analysis target model when additional test data is input. In the belonging cluster determination processing, the update stage clustering unit 302 does not change the belonging cluster for the test data that has already been clustered before the execution of the belonging cluster determination processing.

The following equation (3) is referred to as a degree of similarity between the cluster to which the additional test data belongs and the analysis target model when the additional test data is input in the belonging cluster determination process (hereinafter referred to as “class similarity”). ) Is an example of a function. The condition that the difference in the affiliation cluster determination process is the smallest may be the condition that the equation (3) is large.

q represents the class similarity. In the following, the underscore represents a subscript. Specifically, A_B represents _{A B.} In the following, the hat represents a superscript. Specifically, A ^ B represents ^{A B.}

n indicates a data index number. The data index is an identifier for identifying each data. n is an integer of 1 or more and N or less. N is an integer. The definition of N is the number of data. k is a node used index pattern index. The node usage index pattern index is an identifier for identifying each cluster generated by executing the belonging cluster determination process. The definition of u_k ^ (n) is the probability that it will be 1 if the nth data belongs to the kth node used index pattern, and 0 if the nth data does not belong to the kth node used index pattern. ..

p (A | B) is the probability that the condition A is satisfied (that is, the conditional probability) when the condition B is satisfied. The definition of y is the teacher signal of the data, and the definition of y ^ (n) is the teacher signal of the nth data. Φ is one of the model parameters. The model parameter means a parameter related to the structure of the model represented by the neural network. The parameters related to the structure of the model are, for example, parameters representing the structure of the model. Parameters related to the structure of the model are, for example, weight parameters in the model. Specifically, Φ is a weight vector which is one of the model parameters. g is the region index. The area index is an identifier for identifying the area. The domain represents a domain in analysis, and specifically represents a connected open subset on the feature space. f is a function for obtaining the region index from the node used index pattern index k. For example, an identity map. φg is a model parameter whose node used index pattern index is g. The definition of x ^ (n) is different from the definition of the previous node output value x itself and is the nth input data.

m is a feature index of the node used index pattern. The feature amount index of the node-used index pattern is an identifier for identifying each element of the vector of the node-used index pattern. The feature amount index of the node-used index pattern is, for example, a number indicating one element of the vector of the node-used index pattern. m is an integer of 1 or more and M or less. M is an integer. The definition of M is the number of elements of the vector indicating the node usage index pattern feature quantity.

η is one of the model parameters. Specifically, η represents the representative node used index pattern, which is one of the model parameters. s represents a node usage index pattern. (S_m) ^ (n) means the m-th element of the feature vector of the node-used index pattern of the n-th data. The definition of η_ (k, m) is expressed by the following equation (4).

Π_m = 1 ^ M (η_k, m) ^ ((s_m) ^ (n)) (1-η_k, m) ^ (1- (s_m) ^ (n)) is the node usage index of the nth data. Represents the probability of occurrence in the kth representative node used index pattern of the pattern. α is one of the model parameters. Specifically, α represents the weight of the representative node used index pattern, which is one of the model parameters. The symbol represented by the following equation (5) is a mathematical symbol representing a product.

The definition of α_k is the definition represented by the following equation (6).

In this way, the update stage clustering unit 302 is a process for updating the cluster. Updating the cluster specifically means updating the conditions for the test data to belong to each cluster.

The node index information acquisition unit 104 acquires node index information for each cluster based on the node used index pattern to which it belongs. Hereinafter, for the sake of simplicity, the neural network analysis device 1 will be described by taking the case where the node index is the expected value of the node probability distribution as an example.

The representative linear model acquisition unit 105 executes a process of acquiring a representative linear model for each classified cluster based on the node index information. The process of acquiring the representative linear model when the representative linear model has not been generated yet is the process of generating a new representative linear model. The process of acquiring the representative linear model when the representative linear model has already been generated for each cluster may be the process of updating the generated representative linear model or the process of generating a new representative linear model. May be good.

In the process of acquiring the representative linear model for each cluster after classification based on the node index information, specifically, the activation function of each node of the neural network is fixed at the value indicated by the node index information and the representative linear model is calculated. It is a process. Fixing the activation function of each node with the value indicated by the node index information means constant gradient when the output is differentiated at the input, that is, linear modeling. Gradient constantization is specifically the process of calculating the gradient vector of the activation function given a certain input and replacing the activation function of the model with the Hadamard product function of the input and the gradient vector. be. Since the gradient of the model does not change depending on the input value, the model with constant gradient becomes a linear model. After fixing the activation function of each node with the value indicated by the node index information, the process of calculating the representative linear model is specifically the weight matrix of the layer immediately before the layer to which each node belongs, the node index information, and the activation. This is a process of calculating the matrix product with the weight matrix of the layer after the function. The left side of the above equation (4) (that is, η_ (k, m)) represents a representative linear model.

The following equation (7) is an equation representing an example of processing for acquiring a representative linear model.

β is a hyperparameter that defines the update width. L is a loss function with y, x and Φ_g as variables. L is, for example, a square error. L may be an identification error. ΔL represents the amount of change in L when the variable Φ_g in the loss function is changed.

The operation control unit 106 controls the operations of the node used index pattern acquisition unit 102, the initialization stage clustering unit 301, the node index information acquisition unit 104, the representative linear model acquisition unit 105, and the update stage clustering unit 302. The operation control unit 106 has the node used index pattern acquisition unit 102, the initialization stage clustering unit 301, and the node index until a predetermined condition for acquiring the linear model is satisfied (hereinafter referred to as “analysis processing initialization stage”). The information acquisition unit 104 and the representative linear model acquisition unit 105 are operated.

The operation control unit 106 operates the node used index pattern acquisition unit 102, the node index information acquisition unit 104, the representative linear model acquisition unit 105, and the update stage clustering unit 302 in the analysis processing update stage. That is, the motion control unit 106 operates the update stage clustering unit 302 in place of the initialization stage clustering unit 301 that was operated in the analysis process initialization stage in the analysis process update stage. The analysis processing update stage is after a predetermined condition (hereinafter referred to as “operation change condition”) relating to the acquisition of the linear model is satisfied.

The operation change condition is, for example, a condition that the number of test data used for acquiring the representative linear model has reached a predetermined number. The operation change condition may be, for example, a condition that the change of the representative linear model due to the addition of new test data is less than a predetermined change.

The operation change condition may be, for example, a condition in which test data to which information indicating that the data is used in the analysis process update stage is added in advance is input. Hereinafter, the neural network analysis device 1 will be described by taking as an example a case where the operation change condition is a condition that the number of test data used for acquiring the linear model has reached a predetermined number for the sake of simplicity.

The storage unit 12 stores information indicating whether or not the operation change condition is satisfied.

The output control unit 107 controls the operation of the output unit 132. The output control unit 107 controls the operation of the output unit 132 to cause the output unit 132 to output, for example, an approximation result model.

FIG. 5 is a flowchart showing an example of a processing flow in the analysis processing initialization stage executed by the control unit 10 in the embodiment.

The test data acquisition unit 101 acquires one or a plurality of test data (step S101). Next, the node used index pattern acquisition unit 102 inputs each test data acquired in step S101 into the analysis target model, and acquires a node used index pattern for each test data (step S102). Next, the initialization stage clustering unit 301 clusters the test data and the node used index pattern based on the acquired node used index pattern (step S103). Node usage index patterns are also clustered by clustering test data.

Next, the node index information acquisition unit 104 acquires node index information for each cluster based on the node used index pattern to which it belongs (step S104). That is, the node index information acquisition unit 104 acquires the expected value of the node probability distribution for each node of the neural network to be analyzed for each cluster based on the node used index pattern to which it belongs.

Next, the representative linear model acquisition unit 105 acquires a representative linear model for each classified cluster based on the node index information (step S105). Next, the operation control unit 106 determines whether or not the operation change condition is satisfied (step S106). When the operation change condition is satisfied, the processing in the analysis processing initialization stage executed by the control unit 10 ends. On the other hand, if the operation change condition is not satisfied, the process returns to the process of step S101. The "update stage" shown in FIG. 5 means the analysis process update stage.

FIG. 6 is a flowchart showing an example of a processing flow in the analysis processing update stage executed by the control unit 10 in the embodiment. The processing in the analysis processing update stage is executed when there is test data that is not used in the processing in the initialization stage after the processing in the initialization stage is completed. The process in the analysis process update stage is executed, for example, when test data is input via the user interface 13 or the communication unit 11 after the process in the initialization stage is completed. The processing in the analysis processing update stage includes, for example, test data that has already been input via the user interface 13 or the communication unit 11 at the end of the processing in the initialization stage but has not been used in the processing in the initialization stage. May be executed if.

The test data acquisition unit 101 acquires one or a plurality of test data (step S201). Next, the node used index pattern acquisition unit 102 inputs each test data acquired in step S201 into the analysis target model, and acquires a node used index pattern for each test data (step S202). Next, the update stage clustering unit 302 updates the cluster based on the node usage index pattern acquired in S202 (step S203).

Next, the node index information acquisition unit 104 acquires node index information for each cluster based on the node used index pattern to which it belongs (step S204). That is, the node index information acquisition unit 104 acquires the expected value of the node probability distribution for each node of the neural network to be analyzed for each cluster based on the node used index pattern to which it belongs.

Next, the node index information acquisition unit 104 acquires node index information for each cluster based on the node used index pattern to which it belongs (step S204). That is, the node index information acquisition unit 104 acquires the expected value of the node probability distribution for each node of the neural network to be analyzed for each cluster based on the node used index pattern to which it belongs.

Next, the representative linear model acquisition unit 105 acquires a representative linear model for each classified cluster based on the node index information (step S205). The set whose elements are the representative linear model of each acquired cluster is the approximation result model. Next, the output control unit 107 causes the output unit 132 to display the approximation result model (step S206).

The neural network analysis device 1 configured in this way includes a control unit 10 that approximates the analysis target model with a linear model based on the node usage index pattern. Therefore, the neural network analysis device 1 configured in this way facilitates the interpretation of the trained model while suppressing a decrease in prediction accuracy.

(Modification example)

The neural network analysis device 1 does not necessarily have to include the update stage clustering unit 302. In such a case, the clustering in the analysis process update stage may be executed by the initialization stage clustering unit 301 in the same manner as the clustering in the analysis process initialization stage. Specifically, in such a case, even in the analysis processing update stage, each time test data is added, the processing of steps S102 to S105 is performed using the newly added test data and the already used test data. Is executed. Then, the execution result of step S105 is displayed as an approximation result model.

The process performed by the neural network analysis device 1 is a process of acquiring the equation represented by the equation (10) based on the equations represented by the following equations (8) and (9). Specifically, the process performed by the neural net analysis device 1 is a process of approximating the analysis target model with a linear model based on the node usage index pattern when the test data is input to the analysis target model.

Ω represents a set of model parameters as represented by equation (9). G represents a set of clusters. The left side of Eq. (8) shows the output value when the cluster set and model parameters are given, the node usage index pattern, and the simultaneous probability of class affiliation. The function of equation (10) shows the appearance rate of the output value y when the parameter Φ_g and the data x are given.

The functional unit included in the neural network analysis device 1 does not necessarily have to be mounted in one housing. The neural network analysis device 1 may be implemented by using a plurality of information processing devices that are communicably connected via a network. In this case, each functional unit included in the neural network analysis device 1 may be distributed and mounted in a plurality of information processing devices. For example, the initialization stage clustering unit 301 and the update stage clustering unit 302 may be mounted on different information processing devices.

According to at least one embodiment described above, the neural network analyzer 1 inputs a node usage index, which is a value indicating the degree to which a node of the neural network is used, according to an input to the input layer. The node usage index pattern, which is information indicating each node of the layer, the intermediate layer, and the output layer, is acquired for each test data input to the trained model represented by the analysis target neural network, which is the analysis target neural network. It includes a usage index pattern acquisition unit 102 and a clustering unit 103 that clusters test data and a node usage index pattern based on the node usage index pattern acquired by the node usage index pattern acquisition unit 102. Therefore, the neural network analysis device 1 configured in this way facilitates the interpretation of the trained model while suppressing a decrease in prediction accuracy.

All or part of each function of the control unit 10 may be realized by using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a flexible disk, a magneto-optical disk, a portable medium such as a ROM or a CD-ROM, or a storage device such as a hard disk built in a computer system. The program may be transmitted over a telecommunication line.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, as well as in the scope of the invention described in the claims and the equivalent scope thereof.

1 ... Neural net analysis device, 10 ... Control unit, 11 ... Communication unit, 12 ... Storage unit, 13 ... User interface, 131 ... Input unit, 132 ... Output unit, 101 ... Test data acquisition unit, 102 ... Node usage index Pattern acquisition unit, 103 ... Clustering unit, 104 ... Node index information acquisition unit, 105 ... Representative linear model acquisition unit, 106 ... Motion control unit, 301 ... Initialization stage clustering unit, 302 ... Update stage clustering unit

Claims (8)

The node usage index, which is a value indicating the degree to which the nodes of the neural network are used according to the input to the input layer, is the information indicating each node of the input layer, the intermediate layer, and the output layer of the neural network. A node-used index pattern acquisition unit that acquires a pattern for each test data input to the trained model represented by the neural network to be analyzed, which is the neural network to be analyzed.
A clustering unit that clusters the test data and the node-used index pattern based on the node-used index pattern acquired by the node-used index pattern acquisition unit.
A neural network analyzer equipped with. A representative linear model acquisition unit that acquires a representative linear model that is a linear model based on the distribution of the node used index parameters to which each cluster is generated by clustering by the clustering unit.
The neural network analysis apparatus according to claim 1. The clustering unit receives the output result of the trained model for the newly added test data and each of the newly added test data when a predetermined condition including the condition that the cluster has already been generated is satisfied. Based on the output result of the representative linear model, the cluster to which the representative linear model belongs that outputs the output result having the smallest difference from the output result of the trained model when the additional test data is input. It is determined that the cluster belongs to the newly added test data, and the cluster is updated based on the determination result.
The neural network analysis apparatus according to claim 2. The representative linear model acquisition unit acquires a node index obtained by using the node used index as a random variable for each node of the analysis target neural network for each cluster, and based on the acquired node index, the representative linear model To get,
The neural network analysis apparatus according to claim 2 or 3. The node index is an expected value of the random variable.
The neural network analysis apparatus according to claim 4. An output control unit that controls the operation of the output unit that outputs the representative linear model and causes the output unit to output the representative linear model.
The neural network analysis apparatus according to any one of claims 2 to 5. It is a neural network analysis method executed by a computer.
The node usage index, which is a value indicating the degree to which the nodes of the neural network are used according to the input to the input layer, is the information indicating each node of the input layer, the intermediate layer, and the output layer of the neural network. A node usage index pattern acquisition step that acquires a pattern for each test data input to the trained model represented by the analysis target neural network, which is the analysis target neural network,
A clustering step for clustering the test data and the node used index pattern based on the node used index pattern acquired in the node used index pattern acquisition step, and a clustering step.
Neural network analysis method having. The node usage index, which is a value indicating the degree to which the nodes of the neural network are used according to the input to the input layer, is the information indicating each node of the input layer, the intermediate layer, and the output layer of the neural network. A node usage index pattern acquisition step that acquires a pattern for each test data input to the trained model represented by the analysis target neural network, which is the analysis target neural network,
A clustering step for clustering the test data and the node used index pattern based on the node used index pattern acquired in the node used index pattern acquisition step, and a clustering step.
A program that causes a computer to run.

Images