JP2019139482A - Information estimation device and information estimation method - Google Patents

Abstract

To embody a new auto-encoder equipped with probabilistic elements in an estimation technology using a neural network.SOLUTION: In an information estimation device comprising an auto-encoder configured by an encoder and a decoder utilizing a neural network, at least one integration layer composed of a combination of a dropout layer for dropping out a part of input data out and a full coupling layer (FC layer) for calculating weight is provided as a final layer of the encoder. Accordingly, output values (latent variables) in a latent space which are output values from the encoder become multi-dimensional probability variable vectors, allowing calculating parameters relating to probability distribution in the latent space while preventing the number of dimensions (the number of neurons) in the latent space from growing, and moreover allowing performing estimation by analytically calculating shapes of the probability distribution in the latent space.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an information estimation apparatus and an information estimation method that perform an estimation process using a neural network. In particular, the present invention relates to an information estimation apparatus and an information estimation method obtained by improving a variational auto encoder that is a kind of auto encoder.

  Estimators using neural networks (NNs) can process and estimate a large amount of information such as images and sensor signal data as input data compared to other estimators. It is expected to be applied.

  Neural networks include what is called an auto-encoder. Auto-encoders are unsupervised learners based on neural networks.Typically, in the neural network structure of auto-encoders, the number of neurons in the input layer means the number of dimensions, and the number of neurons in the subsequent layers gradually decreases. In the layer that represents the latent space in the central part, the number of dimensions is compressed the most and the number of neurons is reduced. On the other hand, after the layer representing the latent space in the central portion, the number of neurons increases conversely, and the last output layer has a structure in which the number of neurons is the same as that of the input layer. That is, the number of dimensions of the input layer and the number of dimensions of the output layer are the same, and the number of dimensions of the layer representing the latent space in the central portion is set to be smaller than the number of dimensions of the input layer and the output layer. The first half from the input layer to the layer representing the latent space is called an encoder, and the second half from the layer representing the latent space to the output layer is called a decoder.

When unlabeled learning data (n _Xin- dimensional vector x) is input, first, the encoder compresses the data into latent space with a reduced number of dimensions ( _{nz z-} dimensional vector z: also called a latent variable). In the latent space, they exist in a plurality of chunks according to the similarity of the original data. Furthermore, the compressed space data z can pass through the decoder to reconstruct the input x. This is a classic auto encoder, and the value output from the auto encoder based on the input x that is a fixed value is uniquely determined to be a fixed value as in the case of the input x, and is deterministic.

  On the other hand, as a stochastic auto encoder that includes a stochastic element, that is, the output value changes every calculation for a certain fixed input x, Non-Patent Document 1 discloses a variational auto An encoder (Variational AutoEncoder, hereinafter abbreviated as VAE) has been proposed.

In the classic auto encoder described above, the vector data z in the compressed _nz- dimensional latent space is uniquely determined for the input vector data x. On the other hand, in the VAE, The compressed vector data x is not uniquely determined in the compressed _nz- dimensional latent space, but is determined as a vector of random variables having a certain posterior probability distribution p (z | x). The posterior probability distribution p (z | x) is represented by, for example, an _nz- dimensional multivariate Gaussian distribution. Hereinafter, the theory proposed in Non-Patent Document 1 will be described.

  In VAE, a given data x is described by integrating all z values of the potential factors that caused it. It is described mathematically as follows:

Here, p _θ means the probability that the distribution shape is determined by a certain parameter θ. The larger the probability of the data x explained by integrating all zs on the right side, the more data x is explained.

When data x is given, we want to find a posterior probability distribution p (z | x) that represents what kind of distribution the latent random variable z that caused the data x takes. However, since this posterior probability distribution p (z | x) cannot be calculated analytically, for example, a variational method is used. That is, assuming that there is a certain proposed function q _φ (probability distribution whose distribution shape is determined by a certain parameter φ) that is assumed to be close to p (z | x), the following relational expression is established. The proposed function q _φ can be obtained and used as an approximate solution of p (z | x).

  Here, the left side of the above equation (1) is a log likelihood representing the likelihood of how much the above-described given data x can be explained.

D _{KL in} the first term on the right-hand side of the above equation (1) means KL divergence, and is a function that returns a value of zero or more indicating a distance as to how close two functions are. In order to obtain the proposed function q _φ that approximates the posterior probability distribution p (z | x), it is determined what function the distribution is represented by, and parameters θ and φ of the function are determined. For a large amount of data x, if the above equation is made up of parameters θ and φ in a more optimal state, the logp _θ (x) of the left-side likelihood can be explained, so it should be high, Since the proposed function q _φ approaches the posterior probability distribution p (z | x) that cannot be known, it can be considered that D _{KL of} the first term on the right side approaches zero.

  On the other hand, when the second term on the right side is written as L (θ, φ; x), the second term on the right side is represented by two terms as follows.

The first term in the above equation (2) is a term that means regularization, and the second term in the above equation (2) means whether the input data can be reconstructed in the output. Term. In order to increase logp _θ (x) representing the likelihood, it is necessary to maximize L (θ, φ; x), and it is necessary to maximize the first and second terms of the above equation (2). There is. The optimization in learning is to obtain parameters θ and φ that maximize the objective function L (θ, φ; x) for a large amount of learning data x. For that purpose, it is optimal to use a neural network having a large amount of data processing capability, and it is used as a parameter optimization calculation tool.

In the VAE proposed in Non-Patent Document 1, q _φ (z | x) is considered as an _nz- dimensional multivariate Gaussian distribution, and the parameter φ that determines its shape is set to be equal to the mean μ _{z of the} Gaussian distribution and the variance. The calculation is performed assuming that there are two variance diags (Σ _z ) of the variance matrix Σ _z . Note that diag means a diagonal term of the matrix. Further, the remaining non-diagonal portion offdiag (Σ _z ) is set to zero in Non-Patent Document 1, and therefore the covariance value offdiag (Σ _z ) is calculated in the VAE proposed in Non-Patent Document 1. Not specified. That is, in the VAE proposed in Non-Patent Document 1, conditions such as the following formula are set.

The parameter φ is calculated as an output value of the encoder, and the number of neurons in the latent space layer is _nz dimension × 2. That is, the following _nz dimensions × 2 parameter values are output in order from the encoder.

  As described above, in the optimization calculation, the objective function L (θ, φ; x) needs to be maximized. For this purpose, the first term of the above equation (2) that means regularization is used.

  Need to be maximized. Maximizing this term means that

The distribution q _φ (z | x) to be obtained must have a shape as close as possible to the distribution p _θ (z). p _θ (z) means the prior distribution p _θ (z) of z. According to Non-Patent Document 1, the average μ ₀ is a vector of zero values, and the variance value Σ ₀ is a unit vector. Calculate as a standard Gaussian distribution.

  From the above formula, the first term of the above formula (2), which means regularization, is expressed as the following formula.

According to Non-Patent Document 1, another parameter θ means the output value of the decoder. In the decoder, a specific value of z is sampled, and the probability distribution q _φ (z | x) obtained as described above, that is, the probability of approaching the posterior probability p (z | x) that cannot be known as much as possible. Restore from the distribution q _φ (z | x). The second term of the above formula (2) relating to the above-described restoration means log likelihood indicating whether the restored x has the same value corresponding to the input data x.

That is, as described above, the value output from the final layer of the decoder is not x itself, but is a parameter θ that determines the shape of the probability distribution p _θ (x | z) taken by x. If the data x is a black and white image, the probability distribution is set as the Bernoulli distribution, and the probability p _θ (x | z) that is the same as the input x is calculated using the parameter θ that determines the Bernoulli distribution. Further, log [p _θ (z | x)] is calculated by taking the log. The expected value part of the second term of the above formula (2) related to the above restoration

  Is considered to have an equivalent expected value calculation by processing with multiple samples of the batch.

FIG. 1 is a diagram schematically illustrating an example of a VAE in the prior art. As shown in FIG. 1, an input X (an n _Xin- dimensional vector) passes through an encoder constituted by a neural network, and an average (N _z dimension) of a Gaussian distribution and a variance value (n _z dimension) are output from the encoder. Is output. Further, a specific value of z is sampled based on the output result of the encoder and input to a decoder configured by a neural network, and an n _Xout- dimensional vector is output from the decoder. Note that the output from the decoder is optimized to be the same as the input X, and the number of dimensions of the input and the output is the same (n _Xin = n _Xout ).

International Publication No. WO2014155866A1

“Auto-Encoding Variational Bayes”, Diederik P. Kingma, Max Welling: December 20, 2013 (available from https://arxiv.org/abs/1312.6114) “APPROXIMATING THE KULLBACK LEIBLER DIVERGENCE BETWEEN GAUSSIAN MIXTURE MODELS”, John R. Hershey and Peder A. Olsen: April 15-20, 2007 ({HYPERLINK "http://ieeexplore.ieee.org/document/4218101/", available from http://ieeexplore.ieee.org/document/4218101/})

Although the VAE proposed in Non-Patent Document 1 has a stochastic element, the output in the latent space of the neural network is not a value of z itself, but a parameter that determines the shape of the probability distribution of values that z can take. It is. As described above, in the VAE proposed in Non-Patent Document 1, q _φ (z | x) is considered as an _nz- dimensional multivariate Gaussian distribution, and the parameter φ in the latent space layer of the VAE is an _nz average. And n _z variance values, and the covariance values are all simplified to zero.

  However, when a designer designs to make a more complicated distribution, more parameters are required to determine the distribution shape. For example, when the distribution of the latent space is a 10-dimensional multivariate Gaussian distribution, the number of parameters determining the shape is (10 × 10 −10) / 2 in addition to 10 average values and 10 variance values. = 45 covariance values are required. Further, when the latent space distribution is changed to a mixed Gaussian distribution or the like, it becomes more complicated.

  In order to solve the above-described problems, an object of the present invention is to provide an information estimation apparatus and an information estimation method that realize a new auto encoder having a stochastic element.

  In order to achieve the above object, according to the present invention, the output z in the latent space of the VAE encoder in the prior art is not a parameter that determines the distribution of the output z, but is similar to the classic auto encoder described above. The output z value itself, and the output z value is not a deterministic value like a classic auto-encoder, but a random variable sampled from a probability distribution. An information estimation apparatus and an information estimation method are provided.

In order to achieve the above object, for example, an information estimation apparatus according to the present invention is an information estimation apparatus that performs an estimation process using a neural network,
An auto encoder composed of an encoder and a decoder is provided, and the encoder and the decoder sequentially perform calculation processing based on input data input to the auto encoder, and output data from the auto encoder as a result of the estimation processing An auto-encoder calculator configured to
At least one integrated layer composed of a combination of a dropout layer that drops out a part of data and a fully connected layer that performs weight calculation on the data output from the dropout layer, By providing as a layer, the output value in the latent space, which is the output value from the encoder, is configured to be a multidimensional random variable vector.

In order to achieve the above object, for example, an information estimation method according to the present invention is an information estimation method performed by an information estimation apparatus that performs an estimation process using a neural network,
Using an auto encoder configured by an encoder and a decoder, the encoder and the decoder sequentially perform calculation processing based on input data input to the auto encoder, and output data from the auto encoder is obtained as a result of the estimation processing. An auto encoder calculation step to output,
At least one integrated layer composed of a combination of a dropout layer that drops out a part of data and a fully connected layer that performs weight calculation on the data output from the dropout layer, By providing as a layer, the output value in the latent space, which is the output value from the encoder, is set as a multidimensional random variable vector.

  The present invention realizes a new auto-encoder equipped with a stochastic element and can cope with an arbitrary probability distribution shape in the latent space while suppressing an increase in the number of dimensions (number of neurons) in the latent space. It has the effect. Further, the present invention can estimate the shape of the probability distribution in the latent space by analytical calculation, and thus has an effect that the state of separation of input data in the latent space can be more accurately evaluated.

It is a figure which shows typically an example of VAE in a prior art. It is a figure which shows typically the 1st example of the auto encoder in the 1st Embodiment of this invention. It is a figure which shows the detail of DF layer regarding the 1st example of the auto encoder in the 1st Embodiment of this invention. It is a figure which shows the 2nd example of the auto encoder in the 1st Embodiment of this invention. It is a figure which shows the detail of DF layer regarding the 2nd example of the auto encoder in the 1st Embodiment of this invention. It is a block diagram which shows an example of a structure of the information estimation apparatus containing the calculation processing function of the auto encoder in the 1st Embodiment of this invention. It is a flowchart which shows an example of the calculation process in the 1st Embodiment of this invention. (A) is a figure for demonstrating the display method which shows the ellipse of the contour line of (sigma) showing the width | variety of a Gaussian distribution, and also the scatter diagram of the Monte Carlo sample sampled according to the distribution, (b) ) Is a diagram for explaining a display method that shows an ellipse of a contour line of σ representing the width of the Gaussian distribution, and further, a center value of the Gaussian ellipse, that is, a point of an average value. It is a figure which shows distribution of the value of z in the latent space when the dimension number _nz of the latent space is obtained by experiment using the information estimation apparatus in the 1st Embodiment of this invention, and _nz = 2. FIG. 9 is a diagram drawn by the display method of FIG. It is a figure which shows distribution of the value of z in the latent space when the dimension number _nz of the latent space is obtained by experiment using the information estimation apparatus in the 1st Embodiment of this invention, and _nz = 2. FIG. 9 is a diagram drawn by the display method of FIG. (A) is the figure produced in order to evaluate the experimental result using the information estimation apparatus in the 1st Embodiment of this invention, Comprising: The state which the auto encoder of the state before learning decompress | restored the input image (B) is a diagram created to evaluate the experimental results using the information estimation apparatus according to the first embodiment of the present invention, and is input by the auto encoder in a state after learning It is a figure which shows the state which decompress | restored the image. FIG. 9 is a diagram showing experimental results when the input image is an image with the letter “H” in the upper right, in which the posterior probability distribution (Gaussian distribution) of FIG. The analytically calculated mixed Gaussian distribution is indicated by contour lines, and is displayed by superimposing Monte Carlo distributions represented by a scatter diagram. The posterior probability distribution (Gaussian distribution) shown in FIG. 9 is expanded to the case of the mixed Gaussian distribution according to the second embodiment of the present invention. The mixed Gaussian distribution calculated analytically is indicated by contour lines, and is displayed by superimposing the distributions represented by the scatter chart in a Monte Carlo manner.

  Hereinafter, first and second embodiments of the present invention will be described with reference to the drawings.

<First Embodiment>
In the first embodiment of the present invention, the output z in the latent space of the auto encoder is not used as a parameter that determines the distribution of the output z, but the value of the output z itself is the same as the classic auto encoder described above. And let the value of the output z be a random variable sampled from a probability distribution, rather than a deterministic value as in a classic auto-encoder.

  Specifically, in the first embodiment of the present invention, by adding a dropout layer in the neural network that constitutes the encoder, a value output from the encoder is obtained for input data that is a fixed value. Convert to random variable. Furthermore, the distribution shape of the random variable is calculated by analytically calculating in what form the Bernoulli distribution due to dropout propagates on the neural network, and regularized as in the case of VAE in the prior art. Used for calculation.

  The structure of the auto encoder according to the embodiment of the present invention will be described below with reference to FIGS. FIG. 2 is a diagram schematically illustrating a first example of the auto encoder according to the first embodiment of the present invention, and FIG. 3 illustrates a first example of the auto encoder according to the first embodiment of the present invention. FIG. 4 shows details of the DF layer for an example. FIG. 4 is a diagram showing a second example of the auto encoder according to the first embodiment of the present invention, and FIG. 5 is a second example of the auto encoder according to the first embodiment of the present invention. Is a diagram showing details of the DF layer. 2 and 3, the encoder is provided with one dropout layer, and in the examples shown in FIGS. 4 and 5, the encoder is provided with two dropout layers.

  In the auto encoder according to the first embodiment of the present invention, the encoder of the classic auto encoder performs calculation of a dropout layer, a dropout layer, and a weight that cause randomness by missing a part of input data. A fully connected (FC) layer is provided. Furthermore, the distribution of output values is analytically calculated from the dropout layer and the FC layer, and used as a regularization condition. In this specification, for the sake of simplicity, an integrated layer combining a dropout layer and an FC layer is called a DF layer, and the calculation process in the dropout layer and the calculation process in the FC layer are described as being performed together. .

First, the case where one dropout layer is provided in the encoder will be described. FIG. 2 shows the case where one dropout layer is provided in the encoder. In the conventional VAE shown in FIG. 1, the number of dimensions of the value in the latent space is the number of parameters of the probability distribution of z, whereas in the auto encoder shown in FIG. 2, the first embodiment of the present invention is used. Then, the dimension number of the value in the latent space is the z dimension number n _z itself.

FIG. 3 shows the DF1 layer of the encoder when one dropout layer is provided in the encoder. FIG. 3 shows the dropout layer and FC layer included in the encoder of FIG. The input value Xin ^DF1 to the DF1 layer in FIG. 3 is a fixed value, and its output Xout ^DF1 is a random variable converted by the dropout layer. The probability distribution of the output Xout ^DF1 can be calculated using, for example, the calculation method proposed in Patent Document 1. The calculation method will be described below.

Assume that the input to the DF1 layer is Xin ^DF1 , the output is Xout ^DF1, and the dropout rate (probability of missing data randomly) set in the dropout layer of the DF1 layer is p _Drop ^DF1 . Further, the weight set in advance for the FC layer of the DF1 layer is W _{i, j} ^DF1 , and the bias is b _i ^DF1 . The subscripts i and j are integers satisfying 1 ≦ i ≦ n _Xout ^DF1 and 1 ≦ j ≦ n _Xin ^DF1 . The notation n _Xin ^DF1 in the specification indicates that the subscript of n is Xin ^DF1 , and the notation n _Xout ^DF1 in the specification indicates that the subscript of n is Xout ^DF1 .

The input Xin ^DF1 to the DF1 layer is a fixed value and is an n _Xin ^DF1 dimensional vector composed of constants, and is expressed as follows.

On the other hand, the output Xout ^DF1 from the DF1 layer is expressed as follows.

The output Xout ^DF1 from the DF1 layer is an n _Xout ^DF1- dimensional vector, and the i-th element of this vector Xout ^DF1 is as follows.

Here, due to the dropout in the dropout layer, the W _{i, j} ^DF1 Xin ^DF1 _j term (1 ≦ j ≦ n _Xin ^DF1 ) on the right side disappears at random with the probability p _drop ^DF1 (becomes zero). Accordingly, the Xout ^DF1 _{i on the} left side, which is the sum of the terms, can be calculated as a “sampling sum”. From this, it is assumed that the output Xout ^DF1 is a random variable, for example, a random variable that follows an n _Xout ^DF1- dimensional multivariate Gaussian distribution as follows.

However, μ _out ^DF1 is an n _Xout ^DF1 dimensional vector indicating an average value, and Σ _out ^DF1 is an n _Xout ^DF1 × n _Xout ^DF1 variance-covariance matrix. The average value μ _out ^DF1 and the variance covariance matrix Σ _out ^DF1 are obtained from the following equations.

The output from the DF1 layer in FIG. 3 is the output from the encoder of the auto encoder in FIG. 2, and corresponds to the probability distribution q _φ (z | x) of the value z in the latent space output from the encoder. From this, Xout ^DF1 can be replaced with z, μ _out ^DF1 with μ _z , Σ _out ^DF1 with Σ _z , n _Xin ^DF1 with n _h , and n _Xout ^DF1 with n _z. The value z in the output latent space is represented as the following multivariate Gaussian distribution.

Here, μ _z is an _nz- dimensional vector, and Σ _z is an _nz × _nz variance-covariance matrix.

  Next, a case where two dropout layers are provided in the encoder will be described. FIG. 4 shows the case where two dropout layers are provided in the encoder as a more complicated case. FIG. 5 shows the DF1 layer, ReLu (Rectified Linear Unit) layer, and DF2 layer of the encoder when two dropout layers are provided in the encoder. FIG. 5 shows two dropout layers and an FC layer included in the encoder of FIG. 4 and a part of the ReLu layer sandwiched between them. Hereinafter, a calculation method when there are two DF layers will be described.

  In the case of FIG. 5, two DF layers, that is, a DF1 layer and a DF2 layer are provided with a ReLu layer interposed therebetween. Inputs and outputs to the first DF1 layer are as described above. In addition, as a method of calculating a nonlinear function such as a ReLu layer between the DF1 layer and the DF2 layer, for example, a method of calculating as a multivariate Gaussian approximation as described in Patent Document 1, or a simple Gaussian function is negative. Method of Approximating and Calculating by Judgment of Being in Region or Positive Region (Details Related to Patent Application (Japanese Patent Application No. 2017-196740) Inventor as Inventor Although Not Published at the Time of Application) The calculation method described in the document and drawings) can be used, but the present invention is not limited to these calculation methods.

Hereinafter, input and output to the second DF2 layer will be described. The input to the DF2 layer is Xin ^DF2 , the output is Xout ^DF2, and the dropout rate of the DF2 layer is p _Drop ^DF2 . The weight of the FC layer of the DF2 layer is W _{i, j} ^DF2 , and the bias is b _i ^DF2 . The subscripts i and j are integers satisfying 1 ≦ i ≦ n _Xout ^DF2 and 1 ≦ j ≦ n _Xin ^DF2 . The notation n _Xin ^DF2 in the specification indicates that the subscript of n is Xin ^DF2 , and the notation n _Xout ^DF2 in the specification indicates that the subscript of n is Xout ^DF2 .

Both the input Xin ^DF2 and the output Xout ^DF2 to the DF2 layer are random variables according to a multivariate Gaussian distribution, and are expressed as follows.

However, μ _in ^DF2 is an n _Xin ^DF2 dimensional vector, Σ _in ^DF2 is an n _Xin ^DF2 × n _Xin ^DF2 covariance matrix, μ _out ^DF2 is an n _Xout ^DF2 dimensional vector, and Σ _out ^DF2 is n _Xout ^DF2 Xn _Xout ^DF2 variance-covariance matrix.

  The average value can be calculated as follows.

  The variance-covariance matrix can be calculated as follows.

  The first term on the right side can be calculated as follows.

The output from the DF2 layer in FIG. 5 is the output from the encoder of the auto encoder in FIG. 4, and corresponds to the probability distribution q _φ (z | x) of the value z in the latent space output from the encoder. Thus, as with the drop-out layer is present one, the title, the Xout ^DF2 to z, the mu _out ^DF2 in mu _z, the sigma _out ^DF2 to sigma _z, the n _Xin ^DF2 to n _h, n _Xout Each of ^DF2 can be replaced with _nz, and the value z in the latent space output from the encoder is expressed as the following multivariate Gaussian distribution.

  In addition, although the case where two dropout layers exist is described here, three or more dropout layers may exist. For example, an output value from the DF2 layer may be input to a further dropout layer (third dropout layer), and in this case, the calculation method similar to the calculation method in the DF2 layer described above may be used. The output value from the dropout layer can be obtained.

As described above, in the first embodiment of the present invention, input data that is a fixed value is converted into a random variable by dropout to generate a probability distribution, and the probability distribution is calculated by an analytical calculation method. Also, this calculation result is used as a regularization condition as in the case of VAE in the prior art. In other words, a condition for keeping the same shape is imposed so that the probability distribution q _φ (z | x) expressed by the following expression is not so different from the prior distribution p _θ (z) expressed by the following expression.

For example, in order to determine whether the probability distribution q _φ (z | x) and the prior distribution p _θ (z) remain in the same shape, as described above, the KL divergence of the multivariate Gaussian distribution is used, Set a cost function that minimizes the distance of the multivariate Gaussian distribution. The formula is shown below.

  The calculation method according to the first embodiment of the present invention is greatly different from the calculation method according to the prior art disclosed in Non-Patent Document 1 in that a covariance value is calculated. That is, in Non-Patent Document 1, the covariance value is not obtained, and the covariance value is set to zero, or in order to obtain the covariance value, it is necessary to further increase the number of neurons. On the other hand, in the first embodiment of the present invention, the covariance value is also calculated by the analysis calculation described above, although the number of neurons of the encoder is smaller.

  Further, according to the calculation method in the first embodiment of the present invention, it is possible to more easily determine the condition of whether the output of the auto encoder can reproduce the input data than the determination of the VAE according to the related art. According to the prior art, since the output value of the encoder is a parameter of the probability distribution of z, for example, in order to obtain a value to be input to the decoder, the probability distribution must be further generated and the value of z must be sampled. I must. On the other hand, in the first embodiment of the present invention, the output of the encoder itself is the value of z, that is, the output value of the encoder can be used as it is as the input value of the decoder. The processing in the decoder after obtaining the value of z is the same in both the first embodiment of the present invention and the prior art.

  In the first embodiment of the present invention, the dropout rate is used to express the probability distribution of z generated by the encoder. For example, when there is one dropout layer, the dropout rate is Is preferably a relatively large value (for example, a value of 0.7 or more).

  Next, an information estimation apparatus capable of executing the process according to the first embodiment of the present invention will be described. FIG. 6 is a block diagram showing an example of the configuration of the information estimation apparatus according to the first embodiment of the present invention. The information estimation apparatus 10 in FIG. 6 is an estimator that performs an estimation process using a neural network, and includes an auto encoder calculation unit 20, an encoder output distribution shape calculation unit 30, a cost function calculation unit 40, and a parameter optimization calculation unit 50. Have.

  The block diagram shown in FIG. 6 only represents the functions related to the present invention, and in actual implementation, may be realized by hardware, software, firmware, or any combination thereof. Functions implemented by software are stored as one or a plurality of instructions or codes in an arbitrary computer-readable medium, and these instructions or codes are stored in a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit: GPU). It can be executed by a hardware-based processing unit such as a graphics processing unit. The functions related to the present invention may be realized by various devices including an IC (Integrated Circuit) and an IC chip set.

  The auto encoder calculation unit 20 includes an auto encoder including an encoder and a decoder configured by a neural network, and has a function of performing calculation processing on the input data X by the encoder and decoder and outputting the output data X. As described with reference to FIGS. 2 to 5, the auto encoder used for the calculation in the auto encoder calculation unit 20 is provided with one or more dropout layers in the encoder. A part of the data is lost at random. Thereby, the value z of the output from the encoder of the auto encoder (output in the latent space) can be used as a random variable.

  The encoder output distribution shape calculation unit 30 has a function of analytically calculating what kind of probability distribution shape the input data x has become due to dropout at the encoder. The encoder output distribution shape calculation unit 30 can calculate the distribution shape of the output z in the latent space from, for example, the input data x, the dropout rate in the dropout layer, and parameters (eg, weight and bias in the FC layer). .

  The cost function calculation unit 40 calculates whether the regularization condition is satisfied from the distribution shape (distribution shape of the output z in the latent space) calculated by the encoder output distribution shape calculation unit 30 by dropout, and further, the auto encoder calculation unit By calculating how much the output x calculated from 20 is similar to the input x, it has a function of calculating the value of the total cost function combining these two calculation results.

  The parameter optimization calculation unit 50 calculates to which value the weight and bias referred to by the auto encoder calculation unit 20 are optimized so that the value of the cost function calculated by the cost function calculation unit 40 is optimized. It has the function to do. The parameter optimization calculation unit 50 calculates parameters (weights and biases) so that the value of the cost function is minimized, and the parameters obtained as a result of this calculation are supplied to the auto encoder calculation unit 20 and parameters of the auto encoder are calculated. Is updated.

  In the information estimation apparatus 10 configured as described above, optimization is performed so that an optimal solution can be obtained from the auto encoder by repeatedly performing optimization on a large amount of input data X.

  Next, an example of processing in the information estimation apparatus 10 illustrated in FIG. 6 will be described with reference to FIG. FIG. 7 is a flowchart showing an example of processing of the information estimation apparatus according to the first embodiment of the present invention.

  In the flowchart shown in FIG. 7, first, the auto encoder calculation unit 20 initializes parameters (weights and biases) of the auto encoder (step S101). When the learning data X is input as the input X of the auto encoder (step S102), the auto encoder calculation unit 20 calculates the value z in the latent space in the encoder of the auto encoder (step S103).

  Also, the encoder output distribution shape calculation unit 30 calculates the distribution shape of the value z in the latent space from the dropout rate, the input data X, and the parameters (weights, bias) (step S104). Information related to the distribution shape of the value z in the latent space calculated by the encoder output distribution shape calculation unit 30 is supplied to the cost function calculation unit 40.

  The auto encoder calculation unit 20 further calculates the output X of the decoder of the auto encoder using the value z in the latent space (step S105). The output X of the decoder of the auto encoder calculated by the auto encoder calculation unit 20 is supplied to the cost function calculation unit 40.

  The cost function calculation unit 40 calculates whether the regularization condition is satisfied based on the information related to the distribution shape of the value z in the latent space, and further calculates how much the output X is similar to the input X Then, the total cost function value obtained by combining these two calculation results is calculated (step S106).

  The parameter optimization calculation unit 50 calculates parameters (weights and biases) so that the value of the cost function calculated by the cost function calculation unit 40 is minimized, and based on the calculation results, the auto encoder calculation unit 20 The encoder parameters are updated (step S107).

  If unprocessed new learning data X exists (“Yes” in step S108), the process returns to step S102, and the same processing (processing in steps S103 to S107) is executed for the new learning data X. That is, the processing of steps S103 to S107 is repeatedly executed for a large amount of learning data X. On the other hand, when all the learning data X is processed and there is no unprocessed new learning data X (“No” in step S108), the processing ends.

Next, an experimental result when learning optimization calculation is actually performed using the information estimation apparatus according to the first embodiment of the present invention will be described. In the experiment described below, the auto encoder shown in FIGS. 2 and 3 is adopted, and one dropout layer is provided in the encoder. Also, the number of dimensions n _z value z in latent space to a n _z = 2. Furthermore, an auto encoder is constructed that restores the input MNIST data at the output by performing learning using the MNIST data (image set of handwritten numerals from 0 to 9) used in the technical field according to the present invention. doing.

  The algorithm for optimization uses a root mean square (RMS) method, and calculates the weight and bias of the auto encoder at a learning rate of 0.001. Further, the above prior distribution is calculated as follows.

  Of course, the part of the non-diagonal term of the variance-covariance matrix, that is, the covariance value can be set to a value other than 0 to have a positive correlation or a negative correlation.

FIG. 9 and FIG. 10 show the value of z in the latent space when the dimension number n _z of the latent space is n _z = 2 obtained by the experiment using the information estimation apparatus in the first embodiment of the present invention. The distribution of. As a method for visualizing and displaying a two-dimensional Gaussian distribution, for example, as shown in FIG. 8 (a), a σ contour ellipse representing the width of the Gaussian distribution, and a Monte Carlo-like dot scattered according to the distribution. A display method showing a scatter diagram of sampled points (repeating trials many times), an ellipse of a contour line of σ representing the width of a Gaussian distribution, and the Gaussian as shown in FIG. There is a display method that shows the center value of the ellipse, that is, the average value point. FIG. 9 is a diagram showing the experimental results by the display method of FIG. 8A, and FIG. 10 is a diagram showing the experimental results by the display method of FIG. 8B.

  The experimental results shown in FIG. 9 and FIG. 10 show the distribution of the value of z in the latent space when the Monte Carlo sampling is performed 400 times with 5000 optimization learning using the MNIST data. ing. A distribution (ellipse) of one z value exists in the latent space for one piece of image input data that is one of the handwritten numerals 0 to 9 of the MNIST data. 9 and 10, z values in the latent space corresponding to the different handwritten numerals 0 to 9 of the MNIST data image are represented by different colors.

  Note that, in the technical field according to the present invention, for example, the value z in the latent space in the VAE is displayed in different colors, for example, corresponding to the handwritten numerals 0 to 9 of the MNIST data. FIG. 9 and FIG. 10 are also made in accordance with such a convention so that those skilled in the art can easily understand, and are originally color drawings, but it is difficult to express colors in monochrome drawings. Referring to FIGS. 9 and 10, the handwritten numerals 0 to 9 and the colors associated with the numerals are schematically described. The value z in the latent space is red when the handwritten numeral is 0, and green when the handwritten numeral is 1. 2 for blue, 3 for yellow, 4 for light blue, 5 for purple, 6 for orange, 7 for pink, 8 for gray, 9 for black It corresponds. Also, although not necessarily an accurate representation, the red point is the direction of 1 o'clock, the green point is the direction of 9 o'clock, the blue point is the direction of 12 o'clock, and the center of FIGS. 9 and 10 is yellow. Point is 5 o'clock, light blue is 5 o'clock, purple is 6 o'clock, orange is 5 o'clock, pink is 6 o'clock, gray is 11 o'clock The direction and the black dot form a lump in the 4 o'clock direction and have a spread. As described above, in FIGS. 9 and 10, the same colors, that is, the same handwritten numerals spread in a two-dimensional latent space, forming a lump. Therefore, it can be seen that the input MNIST data can be automatically classified by the unsupervised learning without the correct answer label as to which of the handwritten numerals 0-9.

For example, in FIG. 9, based on the Gaussian distribution parameters (mean value, variance covariance value) of the z value in the latent space, obtained by the analytical calculation in the first embodiment of the present invention. The posterior probability distribution corresponding to the input of each handwritten digit is represented by an ellipse q _φ (z | x). Furthermore, in order to visually verify whether the posterior probability distribution (ellipse) obtained by analytical calculation is correct or not, 400 points are distributed in a stochastic manner by dropout in a Monte Carlo manner for each ellipse. Plotted as a scatter plot. This was done in order to evaluate that the ellipse obtained by the analytical calculation certainly captured the probability distribution caused by the dropout. No sample calculation is required to draw fine points.

  On the other hand, in the VAE according to the prior art disclosed in Non-Patent Document 1, as described with reference to FIG. 1, what can be calculated regarding the latent space at the center of the auto encoder is not the value of z itself but z Is a distribution parameter. Therefore, in the VAE according to the prior art, a scatter diagram of z values as shown in FIGS. 9 and 10 cannot be directly drawn. Thus, since the VAE according to the prior art does not calculate the covariance value, the shape of the probability distribution that can be understood only by using all of the mean, variance, and covariance, that is, shown in FIG. 9 and FIG. I can't draw an elliptical shape. Therefore, in the VAE according to the related art, it is impossible to see whether the actual individual z values overlap each other in different input handwritten numeral images or can be separated properly in the latent space.

Further, if it is intended to display the distribution as shown in FIGS. 9 and 10 using the result obtained by the conventional VAE, the average value μ _z and the variance value diag (Σ It is necessary to prepare not only _z ) but also the output of the covariance value offdiag (Σ _z ) in the latent space to learn the weights, perform sampling from the completed distribution after learning, and display it as a scatter diagram. That is, when trying to calculate the covariance value by the conventional VAE, more parameters for determining the distribution shape are required, and it is necessary to design a more complicated structure.

  FIGS. 11A and 11B are diagrams created in order to evaluate the experimental results using the information estimation apparatus according to the first embodiment of the present invention. 11 (a) and 11 (b) show the results of sampling each 20 × 20 grid in a two-dimensional latent space and restoring the values of each grid into an image of a handwritten numeral by the decoder. It is the figure plotted side by side reflecting FIG. 11A shows an output obtained when the number of optimization learnings of the auto encoder is zero (learning number = 0, that is, before learning), and FIG. 11B shows the output. The output obtained when the number of optimization learnings of the auto encoder is 5000th (learning number = 5000, that is, after learning) is shown.

  When the number of times of optimization learning is zero, the output from the auto encoder cannot be restored from the input handwritten numeral image, and is merely random noise as shown in FIG. On the other hand, when the number of optimization learnings is 5000, it can be seen that the output from the auto encoder can restore the input handwritten numeral image as shown in FIG. In addition, numbers having similar shapes are present in similar places in the latent space, and the same results as the VAE according to the prior art are obtained.

<Second Embodiment>
Next, a second embodiment of the present invention will be described. In the first embodiment described above, the calculation is performed assuming that the probability distribution q _φ (z | x) of the value of z in the latent space is a multivariate Gaussian distribution. However, if there is a term in the xin ^DF _j Wi _{, j} ^DF term for calculating the output Xout ^DF from the DF layer that deviates from other terms and has a large value, the above-mentioned An approximation in which the output Xout ^DF from the DF layer is a multivariate Gaussian distribution as in the first embodiment does not hold. In that case, as described in Patent Document 1, for a deviated xin ^DF _j Wi _{, j} ^DF term called a peak term, a case where the peak term is dropped out and a case where it is not dropped out are shown. By considering them individually, they are not random variables but are regarded as constants under conditional probabilities, and can be calculated as multivariate Gaussian distributions as in the first embodiment described above under each case. In that case, since the multivariate Gaussian distribution is obtained under the condition probabilities for each of a plurality of cases, the output Xout ^DF from the DF layer is a multivariate “mixed” Gaussian distribution.

In the first embodiment described above, the term corresponding to the calculation of the weight of the output Xout ^DF from the DF layer is described as W _{i, j} ^DF Xin ^DF _j , but in the second embodiment, , Xin ^DF _j Wi _{, j} ^DF . Both represent different terms and the same term.

For the DF layer consisting of a dropout layer and a fully coupled layer, the i-th element Xout ^DF _i of the output vector is the sum of the product of the weight W and the input Xin ^DF plus the bias term b _i ^DF . Is expressed as the following equation.

  When one of the terms is a peak term (j = peak) that deviates from the other terms and has a large absolute value, that is, when the following formula is satisfied, a mixture in which two Gaussian distributions are mixed: Gaussian distribution.

  The inequality sign “>>” in the above expression means that the value on the left side is larger than the value on the right side.

Hereinafter, as a more general case, how the probability distribution of the output vector Xout ^DF from the DF layer (for example, the DF1 layer in FIG. 3) is calculated as a multivariate mixed Gaussian distribution will be described.

Just as in the first embodiment, the n _Xout ^DF- dimensional output vector Xout ^DF is a random variable vector having n _Xout ^DF elements, and the i-th element (1 ≦ i ≦ n _Xout ^DF ) And Xout ^DF _i . Each element Xout ^DF _i is an expression having n _Xin ^DF xW terms represented by an index j (1 ≦ j ≦ n _Xin ^DF ) as in the following expression.

Here, the aforementioned peak term (j = peak) is not an xW term that deviates and has a large value in one index i-th row (Xout ^DF _i ), but 1 ≦ i ≦ n _Xout ^DF Is the value of the most deviating xW term with a common index j in all rows in the range, and is the j th column (“column”). Therefore, the peak term cannot be determined only from one row specified by a certain index i. For example, the peak term (j = peak) You need to find the column.

First, for every n _Xin ^DF columns, a box PeakScore _j (1 ≦ j ≦ n _Xin ^DF ) indicating a deviation degree is prepared, and the initial value is set to zero as follows.

Next, the peak term in a certain i-th row is searched. That is, the average value xWMean _i of all xW _j terms (1 ≦ j ≦ n _Xin ^DF ) in the i-th row is calculated.

The right side means that the average value of the xW _j terms of all indexes j is calculated in a certain i-th row. Further, for each xW _j term (1 ≦ j ≦ n _Xin ^DF ) in a certain i-th row, a value xWDevision _{i, j} indicating how far from the average value is calculated. This value is calculated as an absolute value of a difference from the average value, for example, as in the following equation.

As a result, a score (deviation) indicating how much the j-th xW _j term in an i-th row deviates from the average value can be calculated. The above calculation is performed for all rows (all indexes i), and the score for each index j is accumulated cumulatively. For example _, the value of xWDDev _{i, j} is added to the above-described column box PeakScore _j as follows.

The above calculation is repeated for all rows (all indexes i: 1 ≦ i ≦ n _Xout ^DF ), and PeakScore _j is updated to finally obtain the deviation degree of each column (each index j). be able to. Then, finally obtained PeakScore _j (1 ≦ j ≦ n _Xin ^DF ) is arranged in descending order, and a predetermined number (for example, K) of indexes j are recorded in order from the largest PeakScore _j. To do. Thus, K indexes j (j _{k = 1} , j _{k = 2} ,..., J _{k = K} ) are specified as candidates for the peak term xW _j as a column.

Next, for each peak term xW _j , a combination of cases where it is dropped out / not dropped out is considered, and a mixed Gaussian distribution is created. When K peak terms are considered, the number of mixtures in the mixed Gaussian distribution is 2 ^K.

  Note that the true probability distribution can be calculated more accurately as the number (K) recorded as peak terms is larger. On the other hand, increasing the value of K increases the calculation load. Therefore, the value of K may be specified in advance by the user within a range that can be calculated by a trade-off with the calculation load. The number of peak terms (K value) can be 1 or an integer greater than or equal to 2, and when the number of peak terms (K value) is zero, the first aspect of the present invention described above can be used. The calculation is the same as in the embodiment.

Hereinafter, in the calculation according to the first embodiment, when all K peak terms xW _j (j = j _{K = 1} , j _{K = 2} ,..., J _{k = K} ) are dropped out / dropped A calculation method for calculating the probability distribution of the output Xout ^DF approximated as a Gaussian distribution under the conditional probabilities in each case will be described using a specific example.

Here, as a specific example, the number of peak terms is two (K = 2), and the indexes j (j = j _{K = 1} , j _{K = 2} ) of the two peak terms xW _j calculated from the aforementioned PeakScore _j. Suppose that j _{K = 1} = 3 and j _{K = 2} = 5. That is, the peak terms are xW _{j = 3} and xW _{j = 5} .

There are 2 ^{K = 2} = 4 combinations in the following cases (1) to (4) when the two peak terms xW _{j = 3} and xW _{j = 5} are dropped out / not dropped out. .

(1) xW _{j = 3} was dropped out, xW _{j = 5} was dropped out (2) xW _{j = 3} was dropped out, xW _{j = 5} was not dropped out (3) xW _{j = 3} was not dropped out, xW _{j = 5} was dropped out (4) xW _{j = 3} was not dropped out, xW _{j = 5} was not dropped out

Considering the above four cases (1) to (4), the probability distribution of the output Xout ^DF becomes four multivariate mixed Gaussian distributions. The probability that each of cases (1) to (4) may occur is as follows when the dropout rate in the DF layer is p _Drop ^DF .

_Xin ^DF _j Wi _{, j} ^DF term (1 ≦ j, j ≠ 3, j ≠ 5 ≦ n _Xin ) in all indexes j other than the index j _{K = 1} = 3 and j _{K = 2} = 5 corresponding to the peak term ^DF ) is a random variable that fluctuates and disappears due to dropout. On the other hand, since the peak terms xin ^DF _{j = 3} Wi _{, j = 3} ^DF and xin ^DF _{j = 5} Wi _{, j = 5} ^DF are considered when each term is dropped out / not dropped out. , And can be treated as a fixed value under each condition. Therefore, in the second embodiment, in the calculation according to the first embodiment, among the xin ^DF _j Wi _{, j} ^DF term groups considered as random variables in a certain i-th row, the peak term xin ^DF _{j = 3} W _{i, j = 3} ^DF and peak term xin ^DF _{j = 5} W _{i, j = 5} ^DF are removed and calculation is performed as follows.

  Therefore, in each case (1) to (4), the average value is as follows.

  In addition, the variance value can be calculated by the same formula as in the first embodiment as follows.

However, with respect to ListW ^DF x ^DF _i , since the two peak terms are treated as constants instead of random variables, the peak term xin ^DF _{j = 3} W _{i, j = 3} ^DF and the peak term xin ^DF _{j = 5} as well as the bias term. _{Wi, j = 5} ^DF can be ignored. Accordingly, as shown in the following expression, the index _{j K = 1 = 3, j} K = xin DF j W i except 2 = _{5, j} list of ^DF claim ListW ^DF x ^DF _{j ≠ 3} corresponding to the peak _{section, j ≠ 5, i} is used for the calculation.

In this way, using ListW ^DF x ^DF _i excluding the peak term, the variance value Var (Xout ^DF _i ) is obtained from the above-described equation. The variance value Var (Xout ^DF _i ) is the same in all cases (1) to (4).

  Also, the covariance value is obtained in the same manner as in the first embodiment.

  The covariance values are all the same in cases (1) to (4).

  Finally, the variance-covariance matrix has the same value in all cases (1) to (4).

  As described above, for the four cases (1) to (4), the probability value in which each case can occur and the average value, variance value, and covariance value in each case can be calculated. By simply adding the probability values as weights, the probability distribution of the output values can be calculated as a multivariate mixed Gaussian distribution in which four Gaussian distributions are mixed as in the following equation.

In the first embodiment, the probability of the output value distribution q _phi | to (z x) to determine whether the condition is satisfied regularization probability multivariate Gaussian distribution q φ _(z | x ) And the prior distribution p _θ (z). On the other hand, in the second embodiment, the output value probability distribution q _φ (z | x) is a mixed Gaussian distribution. There is no analytical solution for the calculation of the mixed Gaussian KL divergence, but it can be calculated by various approximation calculation methods such as the variational approximation method described in Non-Patent Document 2.

With the calculation method according to the second embodiment described above, as an extension of the first embodiment, the probability distribution q _φ (z | x) of the value of z in the latent space is calculated as a multivariate mixed Gaussian distribution. be able to. As a result of the calculation, FIGS. 12 and 13 each show a mixed Gaussian distribution composed of 2 ^{K = 4} = 16 Gaussian distributions with 4 peak terms (K = 4), and the latent space of the latent variable. 2 shows a two-dimensional plot of the probability distribution q _φ (z | x) of the value of z. In this case, as the input image, an image of the letter “H” shown small in the upper right of the figure is put. As in the case of the Gaussian distribution shown in FIG. 9, the Monte Carlo distribution (scatter chart or one-dimensional histogram) and the analytical distribution (two-dimensional contour lines, one-dimensional function shape) coincide, and the distribution is analytically determined. It can be seen that it can be calculated as a mixed Gaussian.

Further, even when a plurality of dropout layers are provided as shown in FIG. 5, the same calculation as that of the first embodiment is performed individually for each Gaussian distribution under the condition probability of the mixed Gaussian distribution. By doing so, the probability distribution of the output value q _φ (z | x) can be calculated. However, when the calculation is performed in the DF layer provided in the encoder, the Gaussian distribution is further divided into a plurality of mixed Gaussian distributions, so that the number of mixtures increases every time a plurality of DF layers are propagated. Therefore, for example, calculation may be performed while reducing the number of Gaussian distributions by performing a process such as merging similar Gaussian distributions using existing technology.

Moreover, the information estimation apparatus according to the second embodiment of the present invention is realized by extending the configuration of the information estimation apparatus according to the first embodiment of the present invention (configuration illustrated in FIG. 6). Is possible. For example, the auto encoder calculation unit 20 is provided with a data analysis unit having a function of determining the peak term of the xW term that is the product of the weight W that appears when calculating the output value Xout ^DF _i of the DF layer and the input Xin ^DF Just do it. Then, the auto encoder calculation unit 20 is expanded to perform the above-described calculation for the K peak terms specified by the data analysis unit, so that the value of z according to the multivariate mixed Gaussian distribution in the latent space. Can be output. Further, the calculation related to the regularization condition may be extended so that the auto encoder calculation unit 20 executes the above-described calculation.

  The present invention is applicable to an estimation technique using a neural network, and can realize a new auto encoder having a stochastic element.

DESCRIPTION OF SYMBOLS 10 Information estimation apparatus 20 Auto encoder calculation part 30 Encoder output distribution shape calculation part 40 Cost function calculation part 50 Parameter optimization calculation part

Claims (12)

An information estimation device that performs an estimation process using a neural network,
An auto encoder composed of an encoder and a decoder is provided, and the encoder and the decoder sequentially perform calculation processing based on input data input to the auto encoder, and output data from the auto encoder as a result of the estimation processing An auto-encoder calculator configured to
At least one integrated layer composed of a combination of a dropout layer that drops out a part of data and a fully connected layer that performs weight calculation on the data output from the dropout layer, An information estimation apparatus configured so that an output value in a latent space which is an output value from the encoder becomes a multidimensional random variable vector by being provided as a layer. The auto encoder calculation unit causes the dropout layer to drop out a part of data input to the integrated layer according to a predetermined dropout rate, and from the dropout layer to the total coupling layer. It is configured to calculate a biased sum of a list of terms obtained by multiplying the value of the vector of output data by a matrix of weights,
The information estimation apparatus according to claim 1, wherein a part of each item included in the list becomes zero according to the dropout rate.   Based on the data input to the integration layer, the dropout rate, the weight, and the bias, an average value of a probability distribution followed by a multidimensional random variable vector that is an output value in the latent space, a variance value, The information estimation apparatus according to claim 2, further comprising an encoder output distribution shape calculation unit that calculates a covariance value. The encoder output distribution shape calculator
Multiply the sum of each term in the list by the ratio that remains without being dropped out, and add a bias to calculate the average value of the distribution that the list sum follows,
By calculating the variance value of the list and calculating the variance value of the sample mean, the variance value of the distribution followed by the sum of the list is calculated,
Calculating from the variance values of the distribution followed by the sum of the list a covariance value indicating the correlation of two elements with the sum of the list;
4. The shape of a probability distribution followed by a multidimensional random variable vector that is an output value in the latent space is analytically calculated from the average value, the variance value, and the covariance value. Information estimation device. Regularization processing for regularizing the probability distribution followed by the multidimensional random variable vector, which is an output value in the latent space, to remain in the same shape as the prior distribution, and the output data output from the auto encoder to the auto encoder A cost function calculation unit that calculates a cost function for evaluating a restoration process for restoring the input data that is input;
Based on the cost function, calculate a parameter for optimizing the regularization process and the restoration process, and update a parameter used in the calculation of the auto encoder with the optimization parameter;
The information estimation device according to any one of claims 1 to 4. A term obtained by multiplying the value of a vector of data output from the dropout layer by a matrix of weights used in calculating each element of multidimensional random variable vector data that is data output from the integration layer In the list of, a term specified by a common index included in each element of the multi-dimensional random variable vector is referred to, and a predetermined number of indexes of terms having terms larger than terms specified by other indexes are used. A data analysis unit that extracts and identifies as a peak term with a value greater than other terms;
The auto encoder calculation unit is divided into a case where the peak term is dropped out in the dropout layer and a case where the peak term is not dropped out in the dropout layer, and an average value of the Gaussian distribution in each case. The multivariate mixed Gaussian distribution is calculated by calculating the variance value and the covariance value, and further calculating the mixture sum of the Gaussian distribution in each case using the probability value in which each case occurs. The information estimation apparatus according to claim 2. An information estimation method performed by an information estimation apparatus that performs an estimation process using a neural network,
Using an auto encoder configured by an encoder and a decoder, the encoder and the decoder sequentially perform calculation processing based on input data input to the auto encoder, and output data from the auto encoder is obtained as a result of the estimation processing. An auto encoder calculation step to output,
At least one integrated layer composed of a combination of a dropout layer that drops out a part of data and a fully connected layer that performs weight calculation on the data output from the dropout layer, An information estimation method in which an output value in a latent space, which is an output value from the encoder, is provided as a layer and a multidimensional random variable vector is used. The auto encoder calculation step causes the dropout layer to drop out a part of the data input to the integrated layer according to a predetermined dropout rate, and from the dropout layer to the total coupling layer. Calculate the biased value of the sum of the list of terms multiplied by a weight matrix multiplied by the vector value of the output data,
The information estimation method according to claim 7, wherein a part of each item included in the list becomes zero according to the dropout rate.   Based on the data input to the integration layer, the dropout rate, the weight, and the bias, an average value of a probability distribution followed by a multidimensional random variable vector that is an output value in the latent space, a variance value, The information estimation method according to claim 8, further comprising an encoder output distribution shape calculation step for calculating a covariance value. The encoder output distribution shape calculation step
Multiplying the sum of each term contained in the list by the ratio that remains without being dropped out, and adding a bias to calculate an average value of the distribution followed by the sum of the list;
Calculating a variance value of the list by calculating a variance value of the list by calculating a variance value of the sample average; and
Calculating a covariance value indicating the correlation between two elements of the list sum from the variance value of the distribution followed by the list sum;
Analytically calculating the shape of a probability distribution followed by a multidimensional random variable vector that is an output value in the latent space from the mean value, the variance value, and the covariance value;
The information estimation method according to claim 9. Regularization processing for regularizing the probability distribution followed by the multidimensional random variable vector, which is an output value in the latent space, to remain in the same shape as the prior distribution, and the output data output from the auto encoder to the auto encoder A cost function calculating step for calculating a cost function for evaluating a restoration process for restoring the input data inputted;
A parameter optimization calculation step of calculating a parameter for optimizing the regularization process and the restoration process based on the cost function, and updating a parameter used in the calculation of the auto encoder with the optimization parameter;
The information estimation method according to any one of claims 7 to 10. A term obtained by multiplying the value of a vector of data output from the dropout layer by a matrix of weights used in calculating each element of multidimensional random variable vector data that is data output from the integration layer In the list of, a term specified by a common index included in each element of the multi-dimensional random variable vector is referred to, and a predetermined number of indexes of terms having terms larger than terms specified by other indexes are used. Having a data analysis step to extract and identify as a peak term with a value greater than the other terms;
The auto encoder calculation step is divided into a case where the peak term is dropped out in the dropout layer and a case where the peak term is not dropped out in the dropout layer, and an average value of the Gaussian distribution in each case. The multivariate mixed Gaussian distribution is calculated by calculating a variance value and a covariance value, and further calculating a mixture sum of the Gaussian distributions in each case using a probability value in which each case occurs. Information estimation method described in 1.