JP5713877B2 - I / O model estimation apparatus, method, and program - Google Patents

Description

  The present invention relates to an input / output model estimation device, method, and program, and in particular, input / output model estimation for estimating a parameter of a model for predicting output data from input data based on the input data set and the output data set. The present invention relates to an apparatus, a method, and a program.

In supervised learning, when a set of input and output pairs (x, y) is given as training data {(x _n , y _n )} _{n = 1} ^N , the output is predicted to be unknown. Learn the model. However, it may be difficult or costly to obtain an input / output pair.

  Therefore, a technique for finding a pair from an input-only set and an output-only set is known (Non-Patent Document 1).

Novi Quadrianto, Le Song, Alex J. Smola. “Kernelized Sorting.” Advances in Neural Information Processing Systems 21, pp. 1289-1296, 2009.

  However, the non-patent document 1 has a problem that the input set and the output set need to have the same size.

  The present invention has been made in view of the above circumstances, and an input / output model capable of estimating a parameter of a model for predicting output data for input data from an input data set and an output data set whose association is unknown An object is to provide an estimation apparatus, a method, and a program.

  In order to achieve the above object, the input / output model estimation apparatus of the present invention is an input / output for estimating a parameter of a model for predicting the output data from the input data when output data is obtained for the input data. A model estimation apparatus, wherein the input data is a set of input data, the output data is a set of output data, and an input data set that is not yet associated with the input data; Based on the input data set and the output data set, and approximating an average for each first latent variable of the input data and an average for each second latent variable of the output data, and Approximating the covariance matrix for each first latent variable of the input data and the covariance matrix for each second latent variable of the output data. The first latent variable for each of the input data and a first weight variable for mapping the first latent variable to the input data, and the second latent variable for each of the output data and the second latent variable Model estimation means for estimating a second weight variable for mapping a variable to the output data as a parameter of the model.

  According to the input / output model estimation device of the present invention, the input unit, which is a set of input data, and the set of the output data, and the correspondence between the input data and the input data are unknown by the reading unit. Read the input output data set.

  Then, based on the input data set and the output data set, the model estimation means approximates the average of each of the input data with respect to the first latent variable and the average of each of the output data with respect to the second latent variable, And the first latent variable of each of the input data and the covariance matrix of each of the input data are approximated to the covariance matrix of each of the input data and the second latent variable of each of the output data. A first weight variable for mapping a first latent variable to the input data, and a second latent variable for each of the output data and a second weight variable for mapping the second latent variable to the output data; Estimated as a parameter of the model.

  In this way, each first latency of the input data is approximated with the mean and covariance matrix for each first latent variable of the input data and the mean and covariance matrix for each second latent variable of the output data. By estimating the variable, the first weight variable, and the second latent variable and the second weight variable of each of the output data, the output data for the input data is predicted from the input data set and the output data set whose association is unknown. Model parameters can be estimated.

  Also, the input / output model estimation method of the present invention provides an input / output in an input / output model estimation device that estimates a parameter of a model for predicting the output data from the input data when output data is obtained for the input data. A model estimation method, in which an input data set that is a set of the input data and a set of the output data and an association between the input data and an unknown input are input by a reading unit Reading an output data set; and by means of model estimation means, based on the input data set and the output data set, an average for each first latent variable of each of the input data and a second latent variable of each of the output data And a covariance matrix for each first latent variable of the input data and the output data A first latent variable for mapping each first latent variable of the input data and the first latent variable to the input data so as to approximate a covariance matrix for each second latent variable of Estimating each second latent variable of the output data and a second weight variable for mapping the second latent variable to the output data as parameters of the model.

  The input / output model estimation program of the present invention is a program for causing a computer to function as each means constituting the input / output model estimation device.

  As described above, according to the input / output model estimation device, method, and program of the present invention, the average of each first latent variable of input data and the average of each second latent variable of covariance matrix and output data. And the first latent variable and the first weight variable for each of the input data, and the second latent variable and the second weight variable for each of the output data so as to approximate the covariance matrix and the covariance matrix. From the unknown input data set and output data set, it is possible to estimate the model parameters for predicting the output data for the input data.

It is a block diagram which shows the functional structure of the input-output model estimation apparatus of embodiment of this invention. It is a flowchart which shows the content of the input-output model estimation process routine in the input-output model estimation apparatus of embodiment of this invention. It is a flowchart which shows the flow of the process which estimates the parameter set in the input-output model estimation apparatus of embodiment of this invention. It is a figure which shows the accuracy comparison result of the input / output model transition apparatus of this Embodiment, and another method. It is a figure which shows the accuracy comparison result of the input / output model transition apparatus of this Embodiment, and another method. It is a figure which shows the accuracy comparison result of the input / output model transition apparatus of this Embodiment, and another method.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Outline of model>
Assume that the input data set is X = (x _n ) _{n = 1} ^N1 and the output data set is Y = (y _n ) _{n = 1} ^N2 . Here, it is assumed that the input data and the output data are multidimensional vectors x _n = (x _nm ) _{m = 1} ^M1 and y _n = (y _nm ) _{m = 1} ^M2 . The output data set Y is a set of output data obtained based on a predetermined relationship with the input data, and the association with the input data is unknown. For example, the input data is a patent document, and the output data is a category for the patent document. Note that the output data of the output data set Y does not have to be output data for the input data included in the input data set, and is obtained based on the same relationship with respect to similar input data not included in the input data set. Any output data may be used.

  Assume that the input data set X and the output data set Y can be approximated by the following equations (1) and (2), respectively.

Here, u _in = (u _ink ) ^K _{k = 1} , v _im = (v _imk ) ^K _{k = 1} , u _1n is a latent variable of input data x _n , and v _1m is a latent variable v It is a weight variable for mapping _1m to input data. u _2n is a latent variable of the output data y _n , and v _2m is a weight variable for mapping the latent variable v _2m to the output data. The latent variables u _1n and u _2n and weight variables v _1m and v _2m are model parameters for predicting output data from input data.

  The above equations (1) and (2) are equivalent to approximating a matrix representing an input data set and a matrix representing an output data set by the product of two low rank matrices (matrix decomposition).

When the input data set and the output data set are separately subjected to matrix decomposition, the characteristics of the elements of u _1n and u _2n of the obtained low rank matrix are not related. So, in order to relate them, the average of U ₁ = (u _1n ) _{n = 1} ^{N1 is} the same as the average of U ₂ = (u _2n ) ^N2 _{n = 1} and U ₁ = (u _1n ) Assume that the covariance matrix of _{n = 1} ^{N1 and} the covariance matrix of U ₂ = (u _2n ) ^N2 _{n = 1} are the same. That is, it is assumed that the following formulas (3) and (4) hold.

Here, μ _i is an average and is expressed by the following equation (5).

C _i is a covariance matrix and is represented by the following equation (6).

In the above description, the case where U ₁ and U ₂ have the same mean and covariance matrix has been described as an example. However, the present invention is not limited to this, and U ₁ and U ₂ have the same mean and covariance matrix. It may be assumed that it is generated from a certain normal distribution.

In the present embodiment, the parameter U ₁ = (u _1n ) so that the approximation expressed by the above equations (1), (2), (3), and (4) is satisfied as much as possible in the sense of least squares. _{n = 1} ^N1 , U ₂ = (u _2n ) _{n = 1} ^N2 , V ₁ = (v _1m ) _{m = 1} ^M1 , and V ₂ = (v _2m ) _{m = 1} ^M2 are estimated. That is, the objective function to be minimized is expressed by the following equation (7).

Here, λ ₁ , λ ₂ , α, and β are hyperparameters indicating the importance of each approximation. η is a hyperparameter for regularization. The hyper parameter may be given in advance or may be estimated from the data by cross validation or the like. As an evaluation index at that time, it is possible to use how much approximation of the above formulas (1) and (2) is satisfied.

Also, if a parameter associated with the feature of each element can be estimated, the m-th element of the output for a certain input x _n can be predicted by u ^T _1n v _2m .

Although it is assumed that both input and output can be approximated by a linear sum of latent variables u _in as _in the above equations (1) and (2), as shown in the following equations (8) and (9) Assuming that it can be approximated by an arbitrary function, the parameters may be estimated so as to satisfy these approximations.

When approximated by the above equations (8) and (9), a nonlinear function can be modeled by using a Gaussian process or the like. In this case, the output can be predicted using f ₂ (u _1n ; v _2m ).

<System configuration>
An input / output model estimation device 100 according to an embodiment of the present invention includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM that stores a program for executing an input / output model estimation processing routine described later. (Read Only Memory). This computer can be functionally represented by a configuration including a reading unit 10, a storage unit 20, an estimation unit 30, and a prediction unit 40, as shown in FIG.

  The reading unit 10 includes an input set reading unit 11 and an output set reading unit 12.

  The input set reading unit 11 reads the input data set X that has been input. The output set reading unit 12 reads the input output data set Y.

  The storage unit 20 includes an input set storage unit 21 for storing the read input data set X, an output set storage unit 22 for storing the read output data set Y, and a parameter set for storing the parameter set Θ. A storage unit 23 is provided.

  The estimation unit 30 includes an initialization unit 31, a parameter estimation unit 32, and a writing unit 33.

The initialization unit 31 initializes each parameter of the parameter set Θ by setting values at random. In addition, the hyper parameter is initialized by setting an appropriate value. Parameter set Θ is U ₁ = (u _1n ) _{n = 1} ^N1 , U ₂ = (u _2n ) _{n = 1} ^N2 , V ₁ = (v _1m ) _{m = 1} ^M1 , V ₂ = (v _2m ) _{m = 1} ^M2 .

  The parameter estimation unit 32 estimates the parameter set so that the approximations of the expressions (1) to (4) are satisfied as much as possible. For example, by obtaining a parameter set that minimizes the objective function of the equation (7), a parameter set that satisfies the approximations of the equations (1) to (4) can be obtained. The objective function can be minimized using methods such as the quasi-Newton method and the gradient method. The gradient for each parameter required at that time is expressed by the following equations (10) to (13).

  The parameter estimation unit 32 calculates the parameter set Θ based on the parameter set Θ one time ago so as to minimize the objective function of the equation (7) using the equations (10) to (13). Update.

  By updating the parameter set Θ by the parameter estimator 32 until the end condition is satisfied, the parameter set Θ that satisfies the approximation as much as possible is estimated. The parameter set Θ is an example of model parameters.

  The termination condition is, for example, that the objective function indicating how much the current estimated value of the parameter set Θ satisfies the approximation converges.

  Further, as the termination condition, for example, it is possible to use the convergence of the likelihood indicating how much the current estimated value of the parameter set Θ can explain the input data set and the output data set. Further, as the termination condition, it can be used that the number of repetitions, the calculation time, etc. reach a predetermined value.

  Finally, the writing unit 33 writes the estimated parameter set Θ.

The prediction unit 40 includes an output prediction unit 41 and a writing unit 42. Output prediction unit 41, the input data x _n of the prediction target among the input data set, and latent variables u _1n the input data x _n of the parameter set Θ estimated, weighting variables v _2m (m output data = 1,..., M ₂ ) and predicting the m th (m = 1,..., M ₂ ) element of the output using the following equation (15), Predict output data y for input data _xn .

  The writing unit 42 writes the predicted output data into the storage unit 20.

<Operation of input / output model estimation device>
When the input data set X and the output data set Y are input to the input / output model estimation device 100 of the present embodiment, the input / output model estimation processing routine shown in FIG. The

  First, in step S 101, the input unit 10 reads the input data set X and the output data set Y inputted by the reading unit 10 and stores them in the input set storage unit 21 and the output set storage unit 22.

  In the next step S102, the estimation unit 30 estimates the parameter set Θ for the input data set X and the output data set Y.

  Here, step S102 is realized by the processing routine shown in FIG.

In step S110, the initialization unit 31 sets the latent variable u _1n to be each of the input data x of the input data set X, the weight variable v _1m , and the latent variable u _2n to be each of the output data y of the output data set Y, A random value is set to the weight variable v _2m to initialize the parameter set Θ. Also, appropriate values are set for the hyperparameters λ1, λ2, α, β, and η.

  In the next step S111, the parameter estimation unit 32 uses the parameter set Θ initialized or the parameter set Θ updated last time in step S111, according to the above formulas (10) to (13). ) Update the parameter set Θ so as to minimize the objective function of the expression.

  In step S112, it is determined whether or not a repetitive end condition is satisfied. For example, it is determined whether the value of the objective function calculated based on the parameter set Θ obtained in step S111 has converged. If the value of the objective function has not converged, the process returns to step S111 and the process of step S111 is repeated. On the other hand, if it is determined that the value of the objective function has converged, the parameter set Θ finally estimated by the writing unit 33 is stored in the parameter set storage unit 23 in step S113, and the above-described step S102 is performed. The process ends.

  Next, in step S103 of FIG. 2, the output prediction unit 41 predicts output data for the input data to be predicted according to the above equation (15), using the parameter set Θ estimated in step S102. The writing unit 42 writes the predicted output data in the storage unit 20 and ends the process.

  As described above, according to the input / output model estimation device according to the embodiment of the present invention, the mean and covariance matrix for each latent variable of input data and the mean and covariance matrix for each latent variable of output data Are estimated from the input data set and the output data set whose correspondence is unknown, by estimating each latent variable and weight variable of the input data and each latent variable and weight variable of the output data. Model parameters for predicting output data for the data can be estimated.

  Even if there is no pair of input data and output data, output data for the input data to be predicted can be predicted.

<Example>
In order to evaluate the present invention, an experiment using artificial data was performed.

The artificial data was created by the following procedure. First, an average of u _n is sampled from a normal distribution having an average of 0 and a covariance matrix I. The average 0, the triangular matrix L on each element from a normal distribution of dispersion 1 was sampled, what was LL ^T was covariance matrix of u _n. U = (u _n ) _{n = 1} ^N was generated from the normal distribution of the mean and covariance matrix. Further, V _i , i = 1, 2 were generated from the normal distribution of mean 0 and covariance matrix I. An input data set X and an output data set Y were generated by x _nm = u ^T _n v _1m and ynm = u ^T _n v _2m . Here, N = 300, M1 = 300, M2 = 300, and K = 3.

A mean square error between the correct value y _nm of the output data and the value ^ u ^T _1n ^ v _2m estimated and predicted from the input data set X and the output data set Y is used as an evaluation index. As a baseline method, a method using an average 1 / NΣ _{n = 1} ^N y _nm of data of each element of output data as a predicted value was used. Hyper parameters in the present invention is λ1 = 1, λ2 = 1, α = 1,10,10 2 ···, 10 7, β = 1,10,10 2 ···, 10 7, and the eta = 1. The experiment was conducted with K = 1, 2,.

  The mean square error at the best hyperparameter for each K is shown in FIG. The error was lower than the baseline at all K, and the method proposed in the present invention was found to be effective. The error is the lowest when K = 3 used for data generation.

FIG. 5 shows the mean square error when α is changed. Here K = 3, and a beta = 10 ^2. FIG. 6 shows the mean square error when β is changed. Here, K = 3 and β = 10 ⁶ . If the weights of the approximations on the mean and covariance matrices are too low (α = 1, β = 1), the error is large, suggesting the importance of these approximations for prediction. Also, if the weights of approximations for the mean and covariance matrix are too high (α = 10 ⁷ , β = 10 ⁷ ), the error is large, which is relatively similar to the approximation of the above formulas (1) and (2). It is thought that it gets worse because the weight becomes lower.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the gist of the present invention.

  For example, when some elements of the input data or output data are missing, the approximation regarding the missing part in the corresponding input data or output data may be removed from the objective function. Thereby, it is possible to deal with missing data.

Further, assuming that U ₁ and U ₂ are generated from a normal distribution having the same mean and covariance matrix, the parameter set may be estimated by Bayesian estimation. In this case, the Markov chain Monte Carlo method and the variational Bayes method described in the reference (Ruslan Salakhutdinov, Andriy Mnih. “Bayesian Probabilistic Matrix Factorization using Markov Chain Monte Carlo.” International Conference on Machine Learning 2008) can be used.

  In addition, when it is known that the input data and the output data are non-negative, the parameter set can be estimated by putting a non-negative constraint on the parameter as in the non-negative matrix decomposition method.

  In addition, when input data and output data are discrete values instead of continuous values, it is not the least square, but the reference (David M. Blei and John Lafferty. “Correlated topic models.” Advances in Neural Information Processing Systems 17, As described in Cambridge, MA, 2005), parameter sets can be estimated by maximum likelihood estimation, posterior probability maximization estimation, or Bayesian estimation using a probabilistic model assuming generation from a multinomial distribution. .

  In the above embodiment, the case where the parameter set is estimated using the two data of the input data and the output data has been described as an example. However, the present invention is not limited to this, and three or more data are used. Thus, the parameter set may be estimated so that the mean and the covariance matrix regarding the latent variables of the data are the same.

  In addition, the input / output model estimation apparatus described above has a computer system inside, but the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. Shall be.

  In the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium.

DESCRIPTION OF SYMBOLS 10 Reading part 20 Storage part 30 Estimation part 31 Initialization part 32 Parameter estimation part 40 Prediction part 100 Input / output model estimation apparatus

Claims (8)

An input / output model estimation device for estimating a parameter of a model for predicting the output data from the input data when output data is obtained with respect to the input data,
Reading means for reading an input data set that is a set of the input data, and a set of output data that is an input data set that is an association with the input data and is unknown;
Based on the input data set and the output data set, approximate an average for each first latent variable of the input data and an average for each second latent variable of the output data, and each of the input data The first latent variable of each of the input data and the first latent variable are input to the input so as to approximate a covariance matrix of the first latent variable and a covariance matrix of each second latent variable of the output data. A first weight variable for mapping to data, and a second latent variable for each of the output data and a second weight variable for mapping the second latent variable to the output data are estimated as parameters of the model. Model estimation means;
Input / output model estimation device including   Prediction means for predicting output data for the input data to be predicted based on the first latent variable estimated for the input data to be predicted of the input data set and the estimated second weight variable. The input / output model estimation apparatus according to claim 1, further comprising: The model estimation means includes
Initialization means for setting an initial value for each of the first latent variable and the first weight variable of each of the input data and each of the second latent variable and the second weight variable of each of the output data;
Predetermined for approximating the mean for the first latent variable and the mean for the second latent variable and approximating the covariance matrix for the first latent variable and the covariance matrix for the second latent variable Parameter estimation for estimating each of the first latent variable and the first weight variable of each of the input data and each of the second latent variable and the second weight variable of each of the input data based on the objective function Means,
By repeating estimation by the parameter estimation means, the first latent variable and the first weight variable of each of the input data when the value of the objective function is minimized, and the second of each of the output data 3. The input / output model estimation apparatus according to claim 1, wherein the latent variable and the second weight variable are estimated results of the parameters of the model.   4. The mean and covariance matrix for the latent variable is the mean and covariance matrix of the latent variable or a normal distribution mean and covariance matrix for the latent variable. 5. Input / output model estimation device. An input / output model estimation method in an input / output model estimation device for estimating a parameter of a model for predicting the output data from the input data when output data is obtained with respect to input data,
A step of reading an input data set that is a set of the input data and an output data set that is a set of the output data and whose association with the input data is unknown by reading means; When,
Approximating an average for each first latent variable of the input data and an average for each second latent variable of the output data by the model estimation means based on the input data set and the output data set; and The first latent variable and the first of each of the input data are approximated with a covariance matrix for each of the first latent variables of the input data and a covariance matrix of each of the second latent variables of the output data. A first weight variable for mapping a latent variable to the input data, a second latent variable for each of the output data and a second weight variable for mapping the second latent variable to the output data; Estimating as a parameter of
I / O model estimation method including   Based on the input data to be predicted in the set of input data by the prediction means, the first latent variable estimated for the input data, and the estimated second weight variable, the input of the prediction target The input / output model estimation method according to claim 5, further comprising the step of predicting output data for the data. The step of estimating by the model estimating means includes
A step of setting an initial value for each of the first latent variable and the first weight variable of each of the input data and each of the second latent variable and the second weight variable of each of the output data by an initialization unit; When,
The parameter estimation means approximates the mean related to the first latent variable and the mean related to the second latent variable, and approximates the covariance matrix related to the first latent variable and the covariance matrix related to the second latent variable. Each of the first latent variable and the first weight variable for each of the input data and each of the second latent variable and the second weight variable for each of the input data based on a predetermined objective function for Estimating
By repeating estimation by the parameter estimation means, the first latent variable and the first weight variable of each of the input data when the value of the objective function is minimized, and the second of each of the output data The input / output model estimation method according to claim 5, wherein the latent variable and the second weight variable are estimated results of the parameters of the model.   An input / output model estimation program for causing a computer to function as each means constituting the input / output model estimation device according to any one of claims 1 to 4.

Images