JP7148445B2 - Information estimation device and information estimation method - Google Patents

Description

The present invention relates to an information estimation device and an information estimation method that perform estimation processing using a neural network (NN).

Compared to other estimators, estimators using neural networks are expected to be applied to various fields because they can process a large amount of information such as images and sensor signal data as input data and perform estimation. there is In the learning phase, training data is input to the neural network, and the weights of the neural network are optimized so that the expected output result is output. In the test phase, if some unknown input data is input to the neural network, the result estimated by the neural network is output as an output value.

The data used by the neural network, learning data and test data, is point data consisting of a vector _xin with a certain number of dimensions _nxin . The term "point data" as used here means that it takes a certain fixed value. In addition, they are input to a neural network, and linear and/or nonlinear processing is performed in each layer as neuron values. It is point data consisting of Then, whether it is a regression problem or a classification problem, the data output as the final result is also point data consisting of fixed values, and is a vector or scalar value having a certain number of dimensions.

As conventional techniques related to estimation techniques using neural networks, for example, methods described in Non-Patent Documents 1 and 2 are known. Non-Patent Document 1 describes a method in which the learned weighting parameters are considered as distributions rather than fixed values, and the output is calculated as a distribution based on the distribution of the learned parameters for the input data during estimation. ing. In Non-Patent Document 2, as a technique for calculating ideal input data by back propagation, a technique for visualizing which neurons in which layers in a neural network play an important role in estimation results is known. It is

2016, “Uncertainty in Deep Learning”, Yarin Gal, Thesis http://mlg.eng.cam.ac.uk/yarin/thesis/thesis.pdf 2009, “Visualizing Higher-Layer Features of a Deep Network”, Yoshua Bengio, University de Montreal 2009, “Fast Dropout”, Sida Wang, Christopher Manning; Proceedings of the 30th International Conference on Machine Learning, PMLR 28(2):118-126, 2013. Jean Daunizeau,"Semi-analytical approximations to statistical moments of sigmoid and softmax mappings of normal variables", 2017, arXiv:1703.00091 [stat.ML] I. J. Goodfellow, J. Shlens, and C. Szegedy. “Explaining and harnessing adversarial examples”, 2015.

However, the processing of the conventional neural network as described above is a point data unit process that can only be calculated as point data consisting of vectors for the input of vector point data, and the output estimation result is also a point data unit. was the data.

Conventional neural networks cannot directly handle input data with errors such as observed data, such as distribution data in which the error between a certain value and some plus or minus value exists as a Gaussian distribution. rice field. If you dare to handle distribution data with a conventional neural network, you can sample a large amount of point data from the distribution in a Monte Carlo manner, and input that large amount of point data into the neural network for calculation. However, there is a problem that the processing takes a long time. In addition, the greater the number of dimensions of the vector of point data, the greater the number of possible variations, so there is also the problem that the required number of samples increases.

In addition, when the output estimation result is point data, if the estimation result is only a calculated fixed value instead of the distribution of possible values, how confident was the value estimated? , there is a problem that the probability or certainty of the value cannot be evaluated. That is, there is a problem that the variance of the estimated value and the confidence interval for a certain estimated value cannot be obtained.

As a countermeasure against this problem, for example, a technique as described in Non-Patent Document 1 has been proposed. In the method of Non-Patent Document 1, as described above, the learned weighting parameters are considered as a distribution rather than a fixed value, and even during estimation, the output is distributed based on the distribution of the parameters learned for the input data. is calculated as In particular, in the method of Non-Patent Document 1, the integral calculation for calculating the output distribution from the weight parameter distribution, which is originally unsolvable, is approximately calculated by giving a dropout. proposed to obtain similar results. However, the method of Non-Patent Document 1 considers the learned weighting parameters not as fixed values but as distributions, and is different from the method according to the present invention.

In view of the above problems, the present invention provides an information estimation device and an information estimation method capable of calculating the feature amount distribution of data in each layer of a neural network for outputting a certain estimation result of a trained neural network. intended to provide

In order to achieve the above object, according to the present invention, there is provided an information estimation device that performs estimation processing using a neural network,
a weight learning unit that learns weights in the neural network, which includes a dropout layer that performs dropout to drop a part of data using learning data consisting of a vector of fixed values;
Data propagated in each layer of the neural network is a random variable having a multivariate Gaussian distribution defined by distribution parameters including a mean value, a variance value, and a covariance value, and the dropout is performed in the neural network for input data. a propagation calculation unit that performs propagation calculation by giving
a cost function calculation unit that calculates a cost function representing the difference between output data from the neural network and correct data;
a distribution parameter updating unit that performs back propagation calculation by giving the dropout to the minute error obtained by differentiating the cost function in the neural network and updating the distribution parameter in each layer of the neural network;
an optimization parameter output unit that outputs the distribution parameter as an optimized distribution parameter when the cost function is equal to or less than a predetermined threshold;
There is provided an information estimation device comprising:

Further, in order to achieve the above object, according to the present invention, there is provided an information estimation method executed by an information estimation device that performs estimation processing using a neural network, comprising:
A weight learning step of learning weights in the neural network provided with a dropout layer that performs dropout to lose a part of data using learning data consisting of a vector of fixed values;
Data propagated in each layer of the neural network is a random variable having a multivariate Gaussian distribution defined by distribution parameters including a mean value, a variance value, and a covariance value, and the dropout is performed in the neural network for input data. a propagation calculation step of providing and performing propagation calculation;
a cost function calculation step of calculating a cost function representing the difference between output data from the neural network and correct data;
A distribution parameter updating step of performing back propagation calculation by giving the dropout in the neural network to the minute error obtained by differentiating the cost function, and updating the distribution parameter in each layer of the neural network;
an optimization parameter output step of outputting the distribution parameter as an optimized distribution parameter when the cost function is equal to or less than a predetermined threshold;
An information estimation method is provided.

According to the present invention, in a trained neural network, the distribution parameter can be optimized, and it is possible to calculate the feature quantity distribution of data in each layer of the neural network in order to output a certain estimation result.

In the first to third embodiments of the present invention, it is a diagram schematically showing the relationship between input and output data in an arbitrary layer l of the neural network, (a) is input data in layer l of the neural network (b) from the parameters of the Gaussian distribution of the output data to the parameters of the Gaussian distribution of the input data at an arbitrary layer l of the neural network. FIG. 11 illustrates backpropagation of an error to be updated to . It is a figure which shows an example of a structure of the neural network in the 1st Embodiment of this invention. 4 is a flow chart showing an example of processing for calculating a feature quantity distribution of data according to the first embodiment of the present invention; 1 is a block diagram showing an example of the configuration of an information estimation device according to first to third embodiments of the present invention; FIG. 4 is a graph showing the relationship between the value of the cost function and the number of updates in the first embodiment of the present invention and the comparative example, wherein (a) shows the relationship between the value of the cost function and the number of updates in the comparative example; 2B is a graph showing the relationship between the value of the cost function and the number of updates in the first embodiment of the present invention. In the first embodiment of the present invention, it is a diagram showing the optimized feature amount distribution in the class of number 9, (a) is the feature amount of the input image in the input layer of the neural network of FIG. 3B is a diagram showing the distribution, and (b) is a diagram showing the feature quantity distribution for 10×10=100 neurons in the first FC layer of the neural network of FIG. 2; FIG. FIG. 6A is a diagram showing an example of an input image and softmax output values in the first embodiment of the present invention, and FIG. FIG. 7B is a diagram showing an example, and FIG. 7B is a diagram showing softmax output values when the input image of FIG. 7A is propagated through a neural network; FIG. 4 is a diagram showing a plurality of examples of MNIST augmented images input to the neural network in the first embodiment of the present invention; FIG. FIG. 9 is a diagram showing an example of a feature amount distribution of an input image when a correct label (number 0) is calculated in relation to the MNIST augmented image shown in FIG. 8; FIG. 10 is a diagram showing an example of an input image and softmax output values in the first embodiment of the present invention, where (a) is a diagram showing an example of an input image sampled from the feature amount distribution of FIG. 10(b) is a diagram showing softmax output values when the input image of FIG. 10(a) is propagated through a neural network. In the second embodiment of the present invention, graphs showing the posterior probability distribution and input data, (a) is a graph showing the posterior probability distribution and input point data, (b) is a graph showing the posterior It is a graph showing probability distribution and input distribution data. FIG. 10 is a flowchart showing an example of processing for calculating an approximate expression of a posterior probability distribution in the second embodiment of the present invention; FIG. A histogram showing the posterior probability value of each class in the second embodiment of the present invention and a comparative example, wherein (a) is a temporary value of the correction constant in the comparative example (non-optimized correction constant) is a histogram showing the posterior probability values of each class in the case of (b) is a histogram showing the posterior probability values of each class when the correction constant is optimized in the second embodiment of the present invention be. FIG. 10 is a diagram showing an example of data related to experiments in the second embodiment of the present invention, (a) is a diagram showing an image of number 7 which is test data, and (b) is a diagram showing (a ) is input data, and the logarithm of the posterior probability with the optimized correction constant is calculated for each class. FIG. 10 is a diagram showing an example of data related to experiments in the second embodiment of the present invention, in which the left column shows the posterior probability values calculated (using the provisional correction constants) before obtaining the correction constants; The right column shows the accuracy rate of each class when the posterior probability value is calculated after obtaining the correction constant. FIG. 10 is a diagram showing an example of data related to experiments in the third embodiment of the present invention, (a) is a diagram showing an image of number 4 as test data, and (b) is a diagram showing test data and (c) shows the softmax value calculated by the conventional method for the test data of (a). (d) is a diagram showing the softmax value calculated by the conventional method for the test data of (b), and (e) is the value of the present invention for the test data of (a) and (f) is a diagram showing the logarithm of the posterior probability calculated by the calculation method according to the present invention for the test data of (b). be.

First to third embodiments of the present invention will be described below with reference to the drawings.

First, the outline of the present invention will be explained.

In the neural network, in order to output a certain output (a certain output value for a regression problem, a certain score value for each class for a classification problem), the distribution of vector values, which are input point data for each layer, is calculated. We propose a method to In the present invention, a distribution is calculated that indicates what kind of distribution vector values, which are data input to each layer, should have and from which distribution they are sampled.

By looking at the shape of the distribution, it is possible to grasp which neurons in each layer are important for producing an estimation result. For example, if the distribution of a certain neuron value in a certain layer is wide, it means that any value of that neuron value will produce the desired output result. means that For example, it is possible to simplify the calculation by determining that such neuron values do not require calculation.

Conversely, if the distribution width of a certain neuron value in a certain layer is narrow, it means that the neuron value must take a value within this narrow distribution width to obtain the desired output result. That is, it turns out that the treatment of the layer there is important. This makes it possible to identify which layer is making important decisions in judging the results estimated by the neural network, and is an important clue to accountability for the estimation results.

The present invention is significantly different from the technique of Non-Patent Document 1. As described above, in the method of Non-Patent Document 1, the learned weight parameters are considered as a distribution rather than a fixed value, and at the time of estimation, based on the distribution of the learned weight parameters, the output for the input data is calculated. Calculated as a distribution.

In contrast, the present invention leaves the learned weighting parameter at a fixed value and determines what kind of characteristic distribution the values of the input data vector should have in order to maximize the desired output. , the distribution of the input data is calculated by backpropagation.

The present invention is also significantly different from the technique of Non-Patent Document 2. In the method of Non-Patent Document 2, in order to activate the layer behind the neural network, it is calculated by back propagation what kind of input data should be present, but the input data calculated by back propagation is It is point data of a certain fixed value to the last.

The present invention is on the same level as the visualization technique of "what does a neural network look at in input data to make a decision?" However, the present invention calculates what kind of distribution sampled data can activate the back layer of the neural network and produce the desired output. and visualize it. That is, the method of Non-Patent Document 2 calculates the input data (point data) itself that produces the desired output result, whereas the present invention calculates the distribution (feature distribution) that such point data can take. is very different in terms of calculating

In the present invention, it is assumed that the feature distribution that the input data can take is a single-mode multivariate Gaussian distribution. Σ (variance value Var and covariance value Cov) is obtained using back propagation.

The distribution of actual data is not a single-mode Gaussian distribution but a complex distribution, and the complex distribution propagates while being mapped from layer to layer. And because of its complexity, it is difficult to compute backpropagation of the distribution. However, as a breakthrough technique in the present invention, noise can be added by dropout to allow calculation of the backpropagation of its parameters as a single-mode Gaussian distribution.

As described above, the present invention proposes an information estimation method that includes an approximation method for calculating parameters of a distribution that is assumed to be a single-mode Gaussian distribution, and provides an information estimation apparatus that includes an estimator using this approximation method. suggest. With conventional technology, it is difficult to perform backpropagation calculations assuming a single-mode Gaussian distribution, but the inventors discovered that adding noise due to dropout enables the above calculations. As a result, the present invention proposes an estimator that utilizes the effects and properties of the method.

In relation to the technique of adding dropout, for example, Non-Patent Document 3 describes a technique called analytical dropout. Analytical dropout technology does not need to perform dropout many times in a Monte-Carlo way, but analytically calculates the resulting variance, and applies the resulting variance to the distribution taken by the neuron values. By adding it, an effect equivalent to dropout can be obtained. In Non-Patent Document 3, this analytical dropout technique is used to calculate weights during neural network learning. On the other hand, in the present invention, the data distribution is a single-mode Gaussian It has the point that it can be calculated even if it is assumed to be a distribution, and these points are not disclosed in Non-Patent Document 3.

In order to calculate back propagation in a conventional neural network that handles point data, forward propagation (called forward propagation or simply propagation) is calculated first, and then back propagation (reverse propagation) is performed using the result. (called propagation or backpropagation) must be performed. In the present invention, the data to be handled is not point data but distribution parameters, but it is necessary to perform both forward and backward calculations as in the conventional art.

Calculation formulas used in the first to third embodiments of the present invention will be described in detail below.

(Calculation during propagation)
First, it will be described how Gaussian-distributed input data is input to each layer of the neural network and how it is output in forward propagation calculations.

As shown in FIG. 1(a), an arbitrary layer is denoted by l, and an n _in ^l -dimensional multivariate Gaussian distribution, which is input data input to the layer l, is defined as follows.

μ _in ^l is the n _in ^l dimensional mean vector, Var _in ^l is the n _in ^l dimensional variance vector, and Cov _in ^l is the n _in ^l dimensional covariance matrix, which are the parameters of the Gaussian distribution.

Also, the n _out ^l -dimensional multivariate Gaussian distribution, which is the output data output from the layer l, is defined as follows.

μ _out ^l is the n _out ^l -dimensional mean vector, Var _out ^l is the n _out ^l -dimensional variance vector, and Cov _out ^l is the n _out ^l -dimensional covariance matrix, which are the parameters of the Gaussian distribution.

As shown in FIG. 1A, for a given layer l, the mapping of the parameters of the Gaussian distribution, which is the input data, to the parameters of the Gaussian distribution, which is the output data, is calculated according to the following equation.

In the first to third embodiments of the present invention, computation is performed by constructing a neural network having a structure as shown in FIG. 2, for example. The neural network shown in FIG. 2 is constructed including a fully connected layer (hereinafter referred to as an FC layer), a dropout layer (Dropout Layer), a sigmoid layer (Sigmoid Layer), and a softmax layer (Softmax Layer) It is A neural network having the same structure was used in the experiments described later, but the neural network according to the present invention is not limited to this structure.

Hereinafter, the calculation of the mapping during forward propagation in each layer l included in the neural network will be described for each type.

(DF layer: calculation during propagation)
If the layer l to be treated is a layer consisting of a dropout layer and an FC layer, these layers are collectively referred to herein as the DF layer and denoted l=DF.

The dropout layer is a layer that performs dropout-related calculations. Dropout means multiplying the values of individual neurons in a layer and each connect that connects neurons in two layers independently by a value sampled from a distribution such as Bernoulli distribution or Gaussian distribution. . Sampling is done independently for each propagation and backpropagation.

When dropout is performed by multiplying each neuron in the DF layer by a value sampled from the Bernoulli distribution, the values of each n _in ^DF elements of the vector of input data are independently determined by the designer. is calculated to be zero with probability p _dropout ^DF (0<p _dropout ^DF <1). Therefore, the remaining non-zero values are represented by the following formula as M out of n _in ^DF (M is the M bar described in Equation 4).

On the other hand, in the FC layer, the weight W _i,j of the matrix consisting of n _out ^l × n _in ^l dimensions and the bias term b _i of the vector consisting of n _out ^l dimensions are set, and the input data is multiplied by the weight to A calculation is made to add the bias term.

When the Gaussian distribution represented by Equation 1 is input to the DF layer, what kind of Gaussian distribution represented by Equation 2 is output, that is, the parameters of the Gaussian distribution shown in Equation 5 below (average, We will explain how the transformation of variance, covariance) is performed.

The output average μ _out ^DF is calculated as follows.

The output variance Var _out ^DF is calculated as follows.

The output covariance is computed as follows.

The latter two terms of Equation 11 can be calculated from the above equations. Also, the first term of Equation 11 can be calculated as follows.

(Sigmoid layer: calculation during propagation)
Next, the case where the layer l to be treated is a nonlinear layer, especially a sigmoid layer, will be described. Here, the covariance is not considered as an approximation, and the multivariate Gaussian distribution X _in consisting of the parameters of the single-mode mean vector μ _in and the variance vector Var _in is input to the sigmoid layer, and the output X _out is also the single-mode mean vector μ It is approximated to be a multivariate Gaussian distribution with parameters _out and variance vector Var _out . In this case, it is defined as follows.

In this case, we describe how the parameters of the single-mode multivariate Gaussian distribution of the input X _in are used to compute the parameters of the single-mode multivariate Gaussian distribution of the output X _out .

The average value of the output is calculated as follows.

The variance of the output is calculated as follows.

(Final output layer: calculation during propagation)
A case where the layer l to be treated is the final output layer will be described. Let function π be a function for obtaining output from input in the final output layer.

In the case of the regression problem, there is no special layer for outputting the final output value. layer) and a nonlinear layer (a layer calculated by an arbitrary nonlinear function Nonlinear) directly become the final output layer, and the output value from this layer becomes the estimated value.

On the other hand, for the classification problem, the final output layer is the softmax layer (l=Softmax) and the function π is represented by the softmax function.

Conventionally, the input and output are point data, the input is a K-dimensional vector X, and the output is also a K-dimensional vector π(X). Note that the number of dimensions K means the number of classes c _max .

On the other hand, in the present invention, the input and output are not point data but Gaussian distributions, and the calculation is performed by approximating the single Gaussian distribution of the input to the single Gaussian distribution of the output as well. Specifically, the forward computation only computes the mean of the Gaussian output from the softmax layer. This is because this layer is the final output layer, and the values output from the final output layer are used only to compare with the correct value in the case of learning and to be adopted as the estimated value in the case of estimation. , variance and covariance values are not required. According to Non-Patent Document 4, the average value of the Gaussian distribution output from the softmax layer can be calculated as follows.

(Calculation during back propagation)
Next, calculations during backpropagation of input data will be described. In the following, going back from the final output layer, the neuron value becomes a Gaussian distribution, how the parameters of the Gaussian distribution propagate back, and what kind of input data becomes in the input layer, the Gaussian distribution We will explain how to calculate the parameters of

Non-Patent Literature 2 describes a method of obtaining an error in the value of input data by back-propagating certain point data. Similar to this method, in the present invention, the error of the parameters of the Gaussian distribution that the input data can have, that is, the error δμ of the mean value, the error δVar of the variance value, and the error δCov of the covariance value is calculated by back propagation. go.

As shown in FIG. 1(b), in a certain layer l, the parameter error of the Gaussian distribution of the input in is calculated inversely from the parameter error of the Gaussian distribution of the output out.

Once the error in the parameters of the Gaussian distribution of input in is determined, the parameters of the Gaussian distribution of input in can be updated with the value of the designer-defined learning rate LearningRate, as in conventional learning methods.

The calculation for back-propagating the error δ _out of the rear layer (output side) to the error δ _in of the front layer (input side) will be described below.

In order to solve the above equation, besides the error δ, it is necessary to calculate the derivative between each parameter, ie the Jacobian J ^l . The Jacobian at some layer l is shown in Equation 33 below.

A differential calculation method for constructing the Jacobian will be described below for each type of each layer l included in the neural network.

(DF layer: calculation during back propagation)
Differential calculation in the DF layer composed of the dropout layer and the FC layer described above will be described. By inserting dropout, the present invention enables calculation of back propagation even for single-mode Gaussian distribution propagation.

The derivative of the average output value with respect to the average input value is as follows.

The derivative of the output mean value with respect to the input variance value is as follows.

The derivative of the output mean value with respect to the input covariance value is as follows.

Differentiation of the output variance value with respect to the input mean value is as follows.

The Var(ListWμ _in ^DF _i ) portion of Equation 37 above is expressed as follows.

This derivative can be calculated as follows.

The 2nd and 3rd items in the above formula 37 are zero because they do not depend on the input average value, and finally the following formula is obtained.

The derivative of the variance of the output with respect to the variance of the input is as follows.

The derivative of the variance of the output with respect to the covariance of the input is as follows.

The derivative of the covariance value of the output with respect to the average value of the input is as follows.

The derivative of the covariance value of the output with respect to the variance value of the input is as follows.

The derivative of the output covariance value with respect to the input covariance value is:

(Sigmoid layer: calculation during backpropagation)
Next, the case of nonlinear layers, in particular sigmoid layers, will be described.

The derivative of the average output value with respect to the average input value is as follows.

The derivative of the output mean value with respect to the input variance value is as follows.

Differentiation of the output variance value with respect to the input mean value is as follows.

The derivative of the output variance with respect to the input variance is as follows.

(Final output layer: calculation during backpropagation)
Differentiation of the function π (see Equation 21) when the layer l of the neural network is the final output layer will be described.

In the case of the regression problem, the mean value of the Gaussian distribution and the derivative of the variance-covariance value in the FC layer and the nonlinear layer described above are used for backpropagation.

On the other hand, for classification problems, the final output layer is the softmax layer. A method of calculating the differentiation, that is, the Jacobian, when the input of the softmax layer is a Gaussian distribution will be described below. The output from the final output layer is used to calculate the cost function by the difference from the correct label scalar value. Therefore, it is only necessary to consider only the derivative of the average value of the distribution of the output from the final output layer without depending on the derivative of the variance or the covariance.

The derivative of the average output value with respect to the average input value is as follows.

The derivative of the output mean value with respect to the input variance value is as follows.

Above, for a neural network of arbitrary structure, the calculation of the Jacobian elements of the parameters of a single Gaussian distribution for the DF layer, the sigmoid layer, and the softmax layer, that is, the small error of the parameters of the Gaussian distribution in backpropagation I explained how to calculate it.

(cost function)
Finally, the starting point of backpropagation, that is, the method of setting the cost function in the final output layer will be described.

For an n-dimensional final output layer with n neurons, we only consider the average value of the distribution of outputs from the final output layer. The average value of the distribution from the final output layer is compared with the desired value of the learned correct data, and the difference is set as the cost function. Although the case where the cost function is set by the squared error will be described below, the calculation is not limited to the squared error as long as the calculation can express the difference.

In the case of a regression problem, n=1 when the output value is a scalar as shown below, and the cost function is defined from the difference from the correct value y.

In the case of the classification problem, the difference between the hot vector h of the correct label and the output vector π(X) from the softmax layer defines the cost function as follows in the total number of classes c _max as follows: .

In order to reduce the value of this cost function, the cost function is differentiated, and the minute error δμ _in of the mean value of the parameters of the input Gaussian distribution and the minute error δVar _in of the variance value can be calculated as follows.

By using the above-mentioned Jacobian, it is possible to sequentially calculate the small error δ in each layer by back propagation, and by going back to the input layer, it is possible to calculate the parameters of the Gaussian distribution in the input layer and each layer. can.

Further, as regularization, by providing an arbitrary regular term R in the cost function (see Equations 52 and 53), it is possible to control the solution to be optimized. For example, if you want to suppress the average value μ ^l of the distribution of input data in a certain layer l to between 0 and 1, the following regular term R is applied to the cost function that depends on the μ ^l value as follows: It is possible to attach

When the regular term R is provided, it is necessary to calculate the differential of the regular term R with respect to μ and Var, and add the respective calculation results to Equations 54 and 55, respectively.

The above is the calculation method proposed in the present specification for obtaining the distribution parameter in each layer for maximizing the desired output through back propagation.

In this specification, a distribution related to this optimized distribution parameter is called a "feature quantity distribution". This feature quantity distribution can be calculated in relation to the values of input data in neurons of all layers on the neural network. Here, let x be the value of the input data of a neuron in a certain layer l, and for the characteristic Gaussian distribution function P _l (x) to be taken by that value, use the mean vector μ and the variance-covariance matrix Σ that are the parameters of the Gaussian distribution. is defined as the following formula.

In the case of a regression problem, the feature Gaussian distribution P _l,y (x) for outputting a certain value y is expressed as follows. A value y is a value chosen by the user from a finite or infinite range.

In the case of a classification problem, the feature Gaussian distribution P _l,c (x) for classification into a certain class c is expressed as follows. There are as many classes c as the total number of classes c _max .

It should be noted here that the denominator in Equations 58 and 59 is a scalar value common to all neurons, that is, a constant, independent of the input data x.

A neural network tester can evaluate the importance of a visualized neuron from the variance, which is the width of the feature distribution P _l (x) in each neuron. The variance value of the variance-covariance matrix Σ of the n-dimensional feature quantity distribution in a certain layer l in Equations 58 and 59 represents the variance value of each of n neurons. A large variance value means that any value is acceptable, and therefore the neuron can be considered to have a low influence on the final output decision.

Conversely, if a neuron has a small variance value, the value it takes must be within that range to produce the desired final result, and the neuron has a significant influence on the estimation decision. I understand. In addition, in terms of accountability for the estimation results of the neural network, it is possible to indicate which neurons play an important role in the estimation results by looking at the magnitude of the variance value. Furthermore, it is possible to simplify the calculation process by omitting the calculation for neurons that are judged to be unimportant.

(First embodiment)
In the first embodiment of the present invention, processing for calculating the feature quantity distribution of data will be described. FIG. 3 is a flow chart showing an example of processing for calculating the feature amount distribution of data in the first embodiment of the present invention.

In FIG. 3, first, learning data, which is point data, is used as in the conventional art, and dropouts are given to the neural network for learning, thereby optimizing weights and bias terms (step S101). Note that providing a dropout means setting a probability p _dropout ^DF in the dropout layer in the neural network and setting the data passing through the dropout layer to zero with a certain probability p _dropout ^DF . When the learning in step S101 is completed, the weights and bias terms determined by this learning are treated as fixed values.

Next, a process of optimizing the input data is performed. In the process of optimizing the input data, propagation of the input data and back propagation of the calculation results (Gaussian distribution parameters) obtained from the propagation are repeatedly executed while the learned weights and bias terms are fixed. A process of optimizing the parameters of the Gaussian distribution related to the data, that is, a process of optimizing the input data based on what kind of Gaussian distribution should be input is performed.

First, the input data propagated by the neural network is assumed to have a multivariate Gaussian distribution, and the initial values of the parameters (mean vector and variance-covariance matrix) of the multivariate Gaussian distribution of this input data are set (step S102). In this initial setting, an arbitrary value such as a random number is set as an initial value. For the average value, for example, when the input is an image, the initial value may be set to the central value of the range of possible pixel values. Also, the variance value may be, for example, an appropriate random number of 1 or more. Additionally, the covariance value may be zero. However, the variance-covariance value must be a positive definite matrix.

Next, the parameters of the Gaussian distribution of the input data are propagated from the input layer to the final output layer with dropouts, the cost function is calculated from the results obtained in the final output layer (step S103), and the value of the cost function is It is determined whether or not it has sufficiently approached zero (step S104).

Whether or not the value of the cost function has sufficiently approached zero is determined, for example, by comparing the value of the cost function with a threshold (a value close to zero) predetermined by the user. By comparing the value of the cost function with a predetermined threshold, for example, when the value of the cost function is equal to or less than the predetermined threshold, it can be determined that the value of the cost function has sufficiently approached zero. The state in which the value of the cost function is sufficiently close to zero is the state in which a user-selected desired value is maximally output in the case of a regression problem, and the state in which a user-selected desired class is output in the case of a classification problem. It means that the maximum softmax value is output.

If it is not determined that the value of the cost function is sufficiently close to zero ("No" in step S104), the minute error of the Gaussian distribution parameter is calculated from the cost function (step S105). Specifically, as shown in Equations 54 and 55 described above, the minute error δ can be calculated by calculating the differentiation of the cost function.

Next, for example, by using the above-mentioned Jacobian, the minute errors are back-propagated with dropouts, and the minute errors of the parameters of the Gaussian distribution in each layer are sequentially calculated (step S106). As a result, it is possible to trace back to the input layer and calculate the minute error. Then, the Gaussian distribution parameters in each layer are updated using the minute error of the Gaussian distribution parameters of the input data (step S107). At this time, the parameter is updated by, for example, the minute error multiplied by the learning rate LearningRate.

When the parameters of the Gaussian distribution of the input data are updated in step S107, the process returns to step S103, the updated parameters are propagated from the input layer to the final output layer with dropouts, and the processes after step S103 described above are repeated. be Then, when it is determined that the value of the cost function has sufficiently approached zero ("Yes" in step S104), the optimization processing of the parameters of the Gaussian distribution of the input data ends.

The parameters of the Gaussian distribution of the input data at the end of the optimization process are the optimized data, and given input data sampled from this Gaussian distribution, the neural network will produce the desired result ( desired output value, or in the case of a classification problem, the softmax value of the desired class). The Gaussian distribution of input data means not only the distribution of data in the input layer, which is the first layer of the neural network, but also the distribution of data in each neuron of each layer. That is, the parameters of the Gaussian distribution in an arbitrary layer are optimized by the processing shown in FIG.

Next, the configuration of the information estimation device according to the first embodiment of the present invention will be described. FIG. 4 is a block diagram showing an example of the configuration of the information estimation device according to the first embodiment of the present invention. The information estimation device 1 shown in FIG. The information estimation device 1 shown in FIG. 4 also shows constituent elements used in second and third embodiments of the present invention, which will be described later.

The block diagram shown in FIG. 4 merely represents functionality associated with the present invention, which in actual implementation may be realized in hardware, software, firmware, or any combination thereof. Software-implemented functions may be stored as one or more instructions or code on any computer-readable medium, and these instructions or code may be processed by a hardware-based processor, such as a CPU (Central Processing Unit). executable by the unit. Also, functions related to the present invention may be implemented by various devices including an IC (Integrated Circuit), an IC chipset, and the like.

In the first embodiment of the present invention, the weight learning unit 10 and the data feature distribution calculation unit 20 are used, and the processing of the flowchart shown in FIG. executed.

The weight learning unit 10 has a function of learning weights in the above-described neural network having a dropout layer that performs dropout to drop out part of data using learning data consisting of vectors of fixed values.

The data feature quantity distribution calculator 20 includes a propagation calculator 21 , a cost function calculator 22 , a distribution parameter updater 23 and an optimization parameter outputter 24 .

The propagation calculation unit 21 treats the data propagated in each layer of the neural network as a random variable having a multivariate Gaussian distribution defined by distribution parameters including the mean value, the variance value, and the covariance value, and the input data in the neural network. It has a function to give dropout and perform propagation calculation.

The cost function calculator 22 has a function of calculating a cost function representing the difference between the output data from the neural network and the correct data.

The distribution parameter updating unit 23 has a function of giving dropouts to minute errors obtained by differentiating the cost function in the neural network, performing back propagation calculation, and updating the distribution parameters in each layer of the neural network.

The optimization parameter output unit 24 has a function of outputting a distribution parameter when the cost function is equal to or less than a predetermined threshold as an optimized distribution parameter.

Experiments related to the first embodiment of the present invention will be described below. In this experiment, the neural network shown in FIG. 2 was constructed and handwritten digit data MNIST was used. The neural network of FIG. 2 is constructed including an FC layer, a dropout layer, a sigmoid layer, and a softmax layer. Input data (input image) is an image of 28×28=784 pixels and is vector data of 784 dimensions. Also, a score representing any number from 0 to 9 is output as an estimation result from the softmax layer of the final output layer. The number attached to each layer is the number of dimensions of each layer. For example, when the number of dimensions is converted, the numbers of dimensions before and after conversion are described, such as "784→100".

In this experiment, in the neural network, 60,000 handwritten digit data MNIST images are given dropouts for learning, and weights are calculated. An input image of MNIST is an image of 28×28=784 pixels, and one image corresponds to one point data that is a 784-dimensional vector.

After learning is completed, the weights and bias terms are set to fixed values, and the input data are now propagated as random variables sampled from some unknown Gaussian distribution with dropout. If the parameters of the Gaussian distribution (that is, mean vector μ, variance-covariance matrix Σ) are unknown at the initial stage, appropriate initial values are set.

In the softmax layer of the final output layer, the small error of the Gaussian distribution parameter is back-propagated so that it matches the correct label, and the Gaussian distribution parameter is updated. At that time, the DF layer always gives a dropout. Thus, propagation and backpropagation calculations are performed to update the Gaussian parameters.

Evaluation of the parameters of the updated Gaussian distribution is done Monte Carlo as follows. A Gaussian distribution based on the parameters of the updated Gaussian distribution is set, and 5000 point data are sampled from the Gaussian distribution. Using the point data as input data, forward computation is performed by a neural network, and a cost function is computed for evaluating whether or not the output of the softmax layer matches the correct label. It is assumed that optimization progresses as the value of the cost function decreases.

5A and 5B show plots representing the relationship between the value of the cost function (vertical axis) and the number of updates (horizontal axis). FIG. 5(a) shows a case (comparative example of the present invention) in which no dropout is given during backpropagation of Gaussian distribution parameters, and FIG. 5(b) shows Gaussian A case (experimental example according to the present invention) in which dropout is given during backpropagation of distribution parameters is shown.

As shown in FIG. 5(a), when dropout is not given, the cost function does not converge to near zero even if the optimization is repeated many times, and the parameters of the Gaussian distribution of the input data are optimized. Can not do it. On the other hand, as shown in FIG. 5(b), in the experiment according to the present invention, by providing a noise source called dropout, the cost function can be converged near zero, and a Gaussian distribution of correct data can be created. .

Next, the operation after calculating the parameters of the feature quantity distribution obtained by giving dropout will be described. 6(a) and 6(b) show the optimized feature quantity distribution in the class of number 9. FIG. These feature quantity distributions represent the distribution of grayscale values in each pixel. However, in FIGS. 6A and 6B, only the distributions that can be represented by the mean values and the variance values are shown, and the covariance values, which are the correlations between the distributions, are not shown.

FIG. 6(a) shows the feature quantity distribution of the input image in the input layer of the neural network of FIG. Note that the input image has 28×28=784 pixels, and the feature quantity distribution for each of the 784 pixels is obtained. The volume distribution is shown (every other pixel is shown).

As shown by the distribution of the pixel values of the input image shown in FIG. has a small width and a small variance. On the other hand, near the four corners and edges of the image, the width of the distribution is wide and the dispersion is large. Therefore, it is clearly shown that the pixels near the corners and edges of the image are not important in determining the estimation result.

FIG. 6(b) shows the feature quantity distribution for 10×10=100 neurons in the first FC layer of the neural network of FIG. Since this FC layer is the next layer after the input layer, the input image is abstracted, but it is still divided into neurons with a narrow distribution width and neurons with a wide distribution width. A neuron with a narrow distribution is believed to have a greater influence on the final decision of the estimation than a neuron with a wide distribution.

Also, FIG. 7(a) shows an example of an input image sampled from the Gaussian distribution of number 9 in FIG. 6(a). Also, FIG. 7(b) shows a bar graph of softmax values (the horizontal axis is the class, and the vertical axis is the softmax value) when the neural network calculates the image of FIG. 7(a) as input data. . FIG. 7(a) is an image sampled from the Gaussian distribution of the number 9. Although it is an image that does not look like the number 9, the neural network produces the maximum softness only in the class of the number 9. Output the max value (estimation result).

No matter what kind of data is sampled from the feature quantity distribution of FIG. . From this, it can be said that the method according to the present invention is a method of calculating one of the distributions that can sample the input data "Adversarial Example" that produces the desired estimation results mentioned in Non-Patent Document 5. can also

In addition, the MNIST as shown in FIG. It can be observed remarkably when an extended image (MNIST extended image) is learned. The MNIST augmented image of FIG. 8 is composed of true data related to the correct label located in the upper right, irrelevant dummy data in two places of the upper left and lower right, and an empty black part in the lower left. ing. In this MNIST-extended image, the correct label data is always only the upper right number, and the remaining area is dummy data that does not depend on the correct label. Even with such data, if a conventional neural network is trained as point data, the correct answer can be estimated.

FIG. 9 shows the feature amount distribution of the input image when the correct label (number 0) is calculated by the method according to the present invention. Note that the input image has (28×2)×(28×2)=3136 pixels, and the feature quantity distribution for each of the 3136 pixels is obtained. =784 pixels are shown (every other pixel is displayed). In the feature quantity distribution shown in FIG. 9, the width of the distribution is small only in the upper right area, indicating that the feature quantity distribution in this upper right area is important.

FIG. 10(a) shows an example of an input image sampled from the feature amount distribution of FIG. FIG. 10(b) shows a bar graph of softmax values (horizontal axis is class, vertical axis is softmax value) when the neural network calculates the image of FIG. 10(a) as input data. . For the image of FIG. 10(a) sampled from the feature quantity distribution of FIG. Output the maximum estimation result. As shown in FIGS. 9 and 10(a) and (b), referring to the feature quantity distribution obtained by the method of the present invention, it can be seen that the neural network only looks at the upper right portion of the input image for judgment. , the feature distribution enables visualization of the neural network.

(Second embodiment)
Next, a second embodiment of the invention will be described.

In the above-described first embodiment of the present invention, the method of calculating the feature quantity distribution of data has been described. In the second embodiment of the present invention, learning data is used, and the feature quantity distribution obtained in the first embodiment of the present invention is treated as a likelihood distribution to calculate the posterior probability distribution. If the posterior probability distribution is calculated, as shown in the graph of FIG. 11(a) (the horizontal axis is the input value and the vertical axis is the posterior probability value), By calculating the value of the posterior probability distribution with , it is possible to calculate the probability value of the output estimation result, that is, the certainty of the estimation, which indicates the degree of certainty of the estimation.

A posterior probability distribution can be computed on the neuron values of all layers in the neural network. Therefore, it is possible to calculate the posterior probability value in any layer for new input data, and it is possible to grasp what kind of output it will be without calculating up to the final output layer. Simplification is expected.

As will be described below, the output value of the aforementioned feature quantity distribution (see Equations 58 and 59) can be regarded as the likelihood when certain input point data x _in is input.

In the case of a regression problem, the likelihood L _l (x _in |y) of the input data x _in that gives an output value of y is defined as expressed by Equation 60 below. Here, the output value y is compared as to which y value among the plurality of output values y ₁ , . . . , y _max .

In the case of a classification problem, the likelihood L _l (x _in |c) of input data that falls into a certain class c is defined as expressed by Equation 61 below.

The denominator of the likelihood formulas of Formulas 58 and 59 above is the determinant of the variance-covariance matrix. When calculating the determinant of a matrix, a very high computational precision is required when the number of dimensions of the matrix is large. Depending on the layer, the number of neurons is a huge number of dimensions of 1000 or more, and it may be difficult to calculate the determinant of the matrix. Furthermore, in order to estimate which output value y/class c is given when unknown data x _in is given, the posterior probability Post _l must be calculated from the likelihood L _l and the prior probability Prior _l . not.

In the case of a regression problem, for the output value y, the posterior probability Post _l (y | x _in ) calculated from the likelihood L _l (x _in | y) and the prior probability Prior _l (y) is defined as follows: do.

In the case of the classification problem, for class c, the posterior probability Post _l (c|x _in ) calculated from the likelihood L _l (x _in |c) and the prior probability Prior _l (c) is defined as follows: .

Since the denominators of the above Expressions 62 and 63 are constants between the output value y/class c, it is sufficient to calculate only the numerators without calculating the denominators. Furthermore, even if the prior probability Prior _l of Equations 62 and 63 is not calculated, a value proportional to the posterior probability Post _l can be calculated by calculating only the numerator of the likelihood L _l . This will be explained below.

In the learning data Data _train , in the case of a regression problem, there is learning data for each of the output values y ₁ , . . . , y _max . Also, in the case of a classification problem, there is learning data for each class, which is the value of c among a plurality of classes c ₁ , . . . , c _max .

The present invention proposes that Equations 62 and 63, which are posterior probabilities Post _l , can be approximated in a simple form by multiplying the numerator of the likelihood L _l of Equations 60 and 61 by a correction constant.

In the regression problem, the posterior probability _Postl is represented by the following _formula using correction constants _ay and by.

Also, in the classification problem, the posterior probability _Postl is represented by the following equation using correction constants a _c and b _c .

The correction constant a is a bias value, the correction constant b is a scale value, and the correction constants a and b represent the denominator of the likelihood L _l in Equations 60 and 61, the prior distribution prior, and the calculation error. The correction constant can be calculated by calibration using the macroscopic features of the data, which will be described later. This will be explained below.

For each prior probability of output value y/class c, let N _y and N _c be the number of output value y/class c assumed to exist in the test phase, then the number of output value y/class c is the prior probability value , the regression problem can be expressed as Equation 66 below, and the classification problem can be expressed as Equation 67 below.

Therefore, we prepare a number of data that reflects the prior probability of the test phase. However, a large number of data N _y and N _c are prepared. For example, it is desirable to prepare at least 1000 data items. Let the data be learning data Data _train .

For all learning data Data _train , the value of the posterior probability Post _l is calculated for each output value y/class c, and the set of values of the calculation result is denoted as the set TotalP _l (x _in ) (where x _in is an element of Data _train ). Although this set has a distribution, it can be considered Gaussian due to the white effect, since it is a reasonably large number.

For regression problems, the set of posterior probability values whose output value is y is represented by the following equation.

As for the classification problem, the set of posterior probability values whose class is c is represented by the following equation.

The right sides of Equations 68 and 69 mean Gaussian distributions whose parameters are the mean value Total μ and variance value Total Σ of each output value y/class c.

The average value Totalμ depends on the output value y/class c. Also, the average value _Totalμ should be a value proportional to the data numbers Ny and _Nc of the learning data. Because the posterior probability value is the estimated probability value of which output value y/class c will be, and the average of that set of values in the training data is proportional to the number of known correct labels in the training data. Because there should be.

Based on the assumption that the number of data is sufficiently large, the variance value TotalΣ is considered to be the same value for any output value/class. This is because the variance, which is the variation of data, should converge to a certain constant value when the number of data is sufficiently large.

The correction constants a and b in Equations 64 and 65 can be obtained from the above conditions relating to the average value Total μ and variance value Total Σ.

FIG. 12 shows the processing from defining the approximation formula of the posterior probability distribution as described above to calculating the correction constants a and b. FIG. 12 is a flow chart showing an example of processing for completing an approximation of the posterior probability distribution in the second embodiment of the present invention.

In this process, as shown in FIG. 12, first, the feature amount distribution of data is calculated by the process according to the first embodiment of the present invention (step S201).

Next, learning data Data _train having the same ratio as the prior probability value for each output value y/class c at the time of estimation is prepared (step S202). Further, the feature amount distribution calculated in step S201 is regarded as a distribution formula for calculating the likelihood, and the posterior probability is obtained by subtracting the correction constant a from the numerator formula of the likelihood and dividing it by the correction constant b. A distribution approximation formula is obtained (step S203).

Next, in step S203, the posterior probability distribution formula defined by the correction constants a and b is used to calculate a set of posterior probability values for each output value y/class c for the prepared learning data Data _train (step S204 ). Furthermore, for each output value y/class c posterior probability value, the average value and variance value of the set are calculated (step S205).

Then, for the posterior probability value for each output value y/class c, each output value y/ For the posterior probability values in class c, correction constants a and b are calculated so that the set has the same variance (step S206). The correction constants a and b are obtained by the above calculation, and the approximation formula of the posterior probability distribution can be completed using the correction constants a and b.

Next, the configuration of the information estimation device according to the second embodiment of the present invention will be described. In the second embodiment of the present invention, the weight learning unit 10, the data feature distribution calculation unit 20, the posterior probability distribution calculation unit 30, and the posterior probability value estimation unit 40 included in the information estimation device 1 shown in FIG. be done. The weight learning unit 10 and the data feature quantity distribution calculator 20 are the same as in the first embodiment of the present invention, and output the feature quantity distribution in an arbitrary layer l of the neural network, that is, the optimized distribution parameters. do.

The posterior probability distribution calculation unit 30 uses the distribution related to the optimized distribution parameters output from the data feature quantity distribution calculation unit 20 as a likelihood distribution, and multiplies the likelihood distribution by the prior probability as the posterior probability distribution formula and calculate the posterior probability value of the training data with the approximation formula of the posterior probability distribution, using the formula obtained by subtracting the bias value from the numerator of the likelihood distribution expressed as a fraction and dividing it by the scale value as the approximation formula of the posterior probability distribution, Correction constants a, b ( bias value and scale value). The posterior probability distribution calculation unit 30 executes the processing of the flowchart shown in FIG.

The posterior probability value estimating unit 40 uses an approximate expression of the posterior probability distribution including the correction constants a and b (bias value and scale value) optimized by the posterior probability distribution calculating unit 30 to obtain an input consisting of a vector of fixed values. It has the ability to calculate posterior probability values for data.

Experiments related to the second embodiment of the present invention will be described below. In this experiment, handwritten numeral data MINST is used as a continuation from the first embodiment of the present invention described above.

In the case of MNIST, 10 classes from 0 to 9 (max=10) are prepared in equal numbers for both learning data and test data. Therefore, in the estimating environment using MNIST, the prior distributions can be considered the same.

Also, a suitable number of learning data is prepared. In this case, since the MNIST learning data are evenly distributed, the learning data can be used as they are as data _train for calibration correction.

A sufficient number of 60,000 learning data exist. Therefore, regarding the distribution of the set of posterior probability values Post _l (c|x _in ) (where x _in is an element of the Data _train ), the average Total μ _c and variance value Total Σ _c are simply the same in each class. should be.

In the point data of the learning data Data _train , the set (c|x _in ) of the posterior probability Post _l values defined by Equation 65 in the first FC layer of the neural network in FIG. 2 (where x _in is Data _train element) was calculated at each output value y/class c. It should be noted that the correction constants are assumed to be temporary values of a _c =0 and b _c =1.

FIG. 13(a) shows the posterior probability values of the set as a histogram. FIG. 13(a) shows histograms for each class out of all 10 classes. From the histogram of FIG. 13(a), it can be seen that the correction constants are set as temporary values as described above and are not correct values, so that the distributions are scattered among the classes.

On the other hand, the correction constants a _c and b _c are obtained so that the mean value and the variance value related to the distribution of the set of values of these posterior probabilities are the same, respectively, and the posterior probability Post _l (c| x _in ) values are shown in FIG. 13(b) as a histogram. In this case, it can be seen that the posterior probability distributions, strictly speaking, the distributions of probabilities proportional to the posterior probabilities, have similar shapes in all classes and are well aligned.

As described above, if the correction constants a _c and b _c that make up the posterior probability distribution are calculated using the learning data, even if unknown test data comes at the time of estimation, the posterior probability distribution A posterior probability value (ie, the probability of estimation) is calculated for each class, and the class with the highest value of probability of estimation can be determined to be the estimated class.

In the experiment, for 10,000 MNIST test data images, which class has the highest posterior probability value was calculated. Then, it was determined whether or not this calculation result matched the correct label.

As an example of this experiment, FIG. 14(a) shows an image of MNIST digit 7 used as test data. FIG. 14(b) shows the logarithmic value (Log value) of the posterior probability for each class when the image of FIG. 14(a) is used as input data. As shown in FIG. 14(b), the logarithm value of the posterior probability in the class of number 7 is the largest, indicating that the probability is the highest and the estimation is correct.

Similarly, FIG. 15 shows the accuracy rate in each class obtained by performing this judgment on 10,000 test data images. The left side of FIG. 15 (before correction) shows the accuracy rate when correction constants are not obtained from the learning data (when temporary values a _c =0 and b _c =1 are set), and the right side of FIG. 15 (after correction ) is the accuracy rate when the posterior probability value is calculated after obtaining the correction constant by the method described above. In the case of the provisional correction constant, the average accuracy rate was 0.759, whereas in the case of obtaining the correction constant, the average accuracy rate increased to 0.907. Note that the correct answer rate is 0.92 when forward calculation is performed using the conventional test data as point data without calculating the posterior probability distribution from the learning data, and estimation is performed using the softmax layer. Therefore, the accuracy rate of the estimation result estimated from the posterior probability value in the second embodiment of the present invention is approximately the same level as the accuracy rate of the estimation result output from the softmax layer, which is the final layer, and is high. It can be seen that the accuracy rate can be estimated from the distribution.

Furthermore, from the posterior probability value, it is possible to calculate not only the class estimation but also the probability of estimation and the probability value. It can also be applied to estimators.

(Third Embodiment)
Next, a third embodiment of the invention will be described.

In the above-described second embodiment of the present invention, a method for calculating the posterior probability distribution has been described. In the third embodiment of the present invention, it will be further shown that calculation can be performed even when the input data is not point data but distribution data consisting of random variables having a certain distribution. Specifically, the third embodiment of the present invention proposes a method of analytically calculating the posterior probability value by calculating the degree of overlap between distributions. If the input data is a distribution, it means that there are uncertainties and errors in the data values. Even in such cases, the neural network can handle them. In other words, according to the third embodiment of the present invention, an estimator based on a neural network can be applied even to input data with uncertainty expressed as a distribution of values.

Here, it is assumed that input data is a vector consisting of Gaussian-distributed random variables represented by an average vector μ _in and a variance-covariance matrix Σ _in as represented by the following equations.

In the expression of the posterior probability Post _l (x _in ) of Expressions 64 and 65, the fact that the input data x _in is a distribution (that is, the input distribution data) is shown in the graph of FIG. value, the vertical axis is the posterior probability value), it means calculating the expected value of the output by sampling from the Gaussian distribution, that is, the expected value of the posterior probability value. This is represented by the following formula.

A method of calculating the expected value of Equation 73 is shown below.

As a result, even if the input data is distribution data, that is, a random variable based on a certain distribution, it is possible to perform estimation using a neural network, and it is possible to calculate the likelihood of estimation. As an application, even if the input data is a signal with a measurement error distribution or data with a variance calculated by a conventional stochastic method such as the Kalman filter, it can be handled by the neural network. In addition, even for missing data with unknown values, instead of performing calculations by substituting appropriate temporary fixed values, set them as random variables with a moderately wide range of variance values and perform estimation calculations. becomes possible.

Also, if conventional point data is considered to be distribution data with a small variance value such as a delta function, then all data can be regarded as distribution data. This makes it possible to process missing data in which point data and distribution data coexist. Calculation results using MNIST images are shown in FIGS.

FIG. 16(a) shows an image of a normal MNIST number 4, and FIG. 16(b) shows a defective image in which part of the image of the number 4 is painted in gray. In the image shown in FIG. 16(b), the portion painted in gray is the missing portion where the grayscale value of the pixel is unknown.

The images shown in FIGS. 16(a) and 16(b) are used as input images, and softmax values estimated using the conventional method for each input image are shown in FIGS. 16(c) and 16(d). On the other hand, the images shown in FIGS. 16A and 16B are used as input images, and the posterior probability values calculated using the calculation method according to the present invention for each input image are shown in FIGS. (f). 16(c) to (f) show the estimation results for each class in bar graphs.

The input image is a grayscale image with grayscale values ranging from 0 to 1. In this experiment, when the input image was treated as point data, the intermediate value of 0.5 was adopted for pixels in the gray missing portion in FIG. 16(b). On the other hand, when the input image is treated as distribution data, the gray missing pixels in FIG. The part is also treated as distribution data as described above, and by setting the average value to the grayscale value of the pixel and the variance value to 0.001 (temporary small variance value), the point data is expressed as delta function distribution data. and performed the calculations.

When the input image is treated as point data as shown in FIG. 16(c), the softmax value of the image of number 4 (the image of FIG. 16(a)) with no defects shows a large score for number 4. ing. However, as shown in FIG. 16(d), the softmax value of the missing image (the image in FIG. 16(b)) shows that the scores for numbers 8 and 9 are higher, and the score for number 4 is lower. do not have. That is, in the conventional method, the value of the pixel in the gray portion is fixed at 0.5 for calculation, so the possibility that this input image is the number 4 image is not recognized.

On the other hand, FIG. 16E shows a case where the input image is treated as distribution data. The posterior probability value of the image of number 4 (the image in FIG. 16( a )) without defects is recognized as number 4 . Furthermore, as shown in FIG. 16(f), it is possible that the gray portion of the defective image (the image of FIG. 16(b)) may take various values within the range of 0 to 1. It is considered and the possibilities of images of number 0, number 4, number 8, number 9 are recognized and these possibilities are output as a score.

In general, in sensor networks, when a neural network makes judgments based on the data obtained from various sensors, it is not often possible to make judgments with complete data obtained from all sensors. no. Rather, there are many cases where judgments must be made with data missing or unknown from some sensors. In a conventional neural network, it is necessary to set appropriate values, or to train the neural network assuming that each data is missing. On the other hand, according to the technique of the present invention, even with regard to distribution data consisting of random variables, data is lost simply by calculating the posterior probability distribution obtained in the above-described second embodiment of the present invention using learning data. It is possible to deal with the state where the data is unknown and the state where the data is unknown, and obtain an effective estimation result.

INDUSTRIAL APPLICABILITY The present invention is capable of calculating the feature quantity distribution of data in each layer of a neural network so that a trained neural network can output certain estimation results, and is applicable to all neural network techniques.

1 information estimation device 10 weight learning unit 20 data feature distribution calculation unit 21 propagation calculation unit 22 cost function calculation unit 23 distribution parameter update unit 24 optimization parameter output unit 30 posterior probability distribution calculation unit 40 posterior probability value estimation unit

Claims (12)

An information estimation device that performs estimation processing using a neural network,
a weight learning unit that learns weights in the neural network, which includes a dropout layer that performs dropout to drop a part of data using learning data consisting of a vector of fixed values;
Data propagated in each layer of the neural network is a random variable having a multivariate Gaussian distribution defined by distribution parameters including a mean value, a variance value, and a covariance value, and the dropout is performed in the neural network for input data. a propagation calculation unit that performs propagation calculation by giving
a cost function calculation unit that calculates a cost function representing the difference between output data from the neural network and correct data;
a distribution parameter updating unit that performs back propagation calculation by giving the dropout to the minute error obtained by differentiating the cost function in the neural network and updating the distribution parameter in each layer of the neural network;
an optimization parameter output unit that outputs the distribution parameter as an optimized distribution parameter when the cost function is equal to or less than a predetermined threshold;
information estimation device. 2. The information estimation device according to claim 1, wherein the variance value included in the optimized distribution parameter output from the optimization parameter output unit represents the importance of each neuron in each layer of the neural network. 3. The information estimating apparatus according to claim 2, which simplifies calculation in neurons determined to have low importance based on the importance of each neuron in each layer of the neural network. The distribution related to the optimized distribution parameter output from the optimization parameter output unit is assumed to be a likelihood distribution, and the likelihood distribution multiplied by the prior probability is assumed to be a posterior probability distribution formula, expressed as a fraction The formula obtained by subtracting the bias value from the numerator of the likelihood distribution and dividing it by the scale value is used as the approximation formula of the posterior probability distribution, and the posterior probability value of the learning data is calculated by the approximation formula of the posterior probability distribution. The bias value and the scale value so that the mean value and the variance value of a distribution consisting of a set of probability values are the same for all classes in the case of a classification problem, and between a plurality of output values in the case of a regression problem. 4. The information estimation device according to any one of claims 1 to 3, further comprising a posterior probability distribution calculator for optimizing . 5. A posterior probability value estimating unit that calculates a posterior probability value for input data consisting of a vector of fixed values using an approximate expression of the posterior probability distribution including the optimized bias value and the scale value. The information estimation device according to . The input data for which the posterior probability values are to be calculated is a random variable having a predetermined distribution, and a plurality of posterior probability values are calculated using a plurality of values sampled from the distribution of the input data. 6. The information estimation device according to claim 5, wherein the posterior probability value for the input data is calculated by calculating the expected value. An information estimation method executed by an information estimation device that performs estimation processing using a neural network,
A weight learning step of learning weights in the neural network provided with a dropout layer that performs dropout to lose a part of data using learning data consisting of a vector of fixed values;
Data propagated in each layer of the neural network is a random variable having a multivariate Gaussian distribution defined by distribution parameters including a mean value, a variance value, and a covariance value, and the dropout is performed in the neural network for input data. a propagation calculation step of providing and performing propagation calculation;
a cost function calculation step of calculating a cost function representing the difference between output data from the neural network and correct data;
A distribution parameter updating step of performing back propagation calculation by giving the dropout in the neural network to the minute error obtained by differentiating the cost function, and updating the distribution parameter in each layer of the neural network;
an optimization parameter output step of outputting the distribution parameter as an optimized distribution parameter when the cost function is equal to or less than a predetermined threshold;
information estimation method. 8. The information estimation method according to claim 7, wherein the variance value included in the optimized distribution parameter output in the optimization parameter output step represents the importance of each neuron in each layer of the neural network. 9. The method of estimating information according to claim 8 , wherein calculations in neurons determined to be less important based on the importance of each neuron in each layer of the neural network are simplified. A distribution related to the optimized distribution parameter is defined as a likelihood distribution, a formula obtained by multiplying the likelihood distribution by a prior probability is defined as a posterior probability distribution formula, and a bias value is obtained from the numerator of the likelihood distribution represented by a fraction. The formula obtained by subtracting and dividing by the scale value is used as the approximation formula of the posterior probability distribution, and the posterior probability value of the learning data is calculated by the approximation formula of the posterior probability distribution, and the average value of the distribution consisting of the set of posterior probability values and a posterior probability distribution calculation step of optimizing the bias value and the scale value such that the variance value is the same for all classes in the case of a classification problem and among a plurality of output values in the case of a regression problem. The information estimation method according to any one of claims 7 to 9, comprising: Information estimation according to claim 10, wherein the approximation of the posterior probability distribution including the optimized bias value and the scale value is used to calculate the posterior probability value for input data consisting of a vector of fixed values. Method. The input data for which the posterior probability values are to be calculated is a random variable having a predetermined distribution, and a plurality of posterior probability values are calculated using a plurality of values sampled from the distribution of the input data. 12. The information estimation method according to claim 11, wherein the posterior probability value for the input data is calculated by calculating its expected value.

Images