JP2009075737A - Semi-supervised learning method, device, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a general-purpose, practical and high-accuracy classifier having a small number of learning parameters, and allowing application of supervised learning as a lower learning machine. <P>SOLUTION: Training data are stored. Unlabeled data and test data are stored. The supervised learning is performed by the training data. A gradient is stored. The training data, the unlabeled data, and the test data are combined, and the supervised learning is repeatedly performed. A learned discriminant function is stored. A label of the test data is predicted by use of the discriminant function. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a semi-supervised learning method, a semi-supervised learning apparatus, and a semi-supervised learning program with high prediction accuracy by learning not only labeled data but also unlabeled data using ensemble learning.

  A learning method for predicting the label of the test data after causing the learning machine to learn the labeled data as training data is called supervised learning.

  As supervised learning methods, boosting, bagging, support vector machines, and the like are well known and applied to various data. Non-patent documents 1 to 3 describe boosting, bagging, and support vector machines.

  However, a sufficient amount of training data is required to construct a highly accurate classifier in supervised learning. In this regard, there is a problem that it takes a lot of time and labor to obtain a sufficient amount of training data in order to label the data manually. Moreover, when the distribution of the obtained data is largely biased, it has been pointed out that the prediction accuracy for the test data is low.

  A technique called semi-supervised learning has been proposed for these problems. Non-Patent Documents 4 to 7 describe examples in which semi-supervised learning is realized by extending conventional boosting and support vector machine methodologies.

  Here, semi-supervised learning refers to a learning method that considers not only training data but also the distribution of unlabeled data or test data, and constitutes a high-precision classifier even when the number of training data is small. This is a learning method for this purpose.

In the following description, unlabeled data and test data are simply referred to as test data.
Y Freund, RE Schapire, A decision-theoretic generalization of on-line learning and an application to boosting, Journal of Computer and System Sciences, 1997, 23-27. L. Breiman, Bagging Predictors, Machine Learning, 1996, 123-140. JP Vert, K. Tsuda and B. Scholkopf, "A primer on kernel methods", in Kernel Methods in Computational Biology, MIT Press, 2004, 35-70. F.d'Alche Buc, Y. Grandvalet, and C. Ambroise, Semi-supervised margin boost, Advances in Neural Information Processing Systems 14, MIT Press, 2002. Y. Grandvalet, F.d'Alche Buc, and C. Ambroise, Boosting mixture models for semi-supervised learning, ICANN, Springer-Verlag, 2002, 41-48. KP Benett, A. Demiriz, and R. Macin, Exploiting unlabeled data in ensemble methods, SIGKDD, ACM, 2002. O. Chapelle and A. Zien, Semi-supervised classification by low density separation, In Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, 2005. J Friedman, Greedy Function Approximation: A Gradient Boosting Machine, The Annals of Statistics, 2001, 1189-1232. Boaz Leskes, The Value of Agreement, a New Boosting Algorithm, Master Thesis, Section Computational Science and Institute for Logic, Language & Computation, University of Amsterdam, 2005. A. Buja and YS Lee, Data Mining Criteria for Tree-Based Regression and Classification, Proceedings of KDD 2001, 27-36. UCI Machine Learning Repository Content Summary [Searched on September 12, 2007], Internet <http://mlearn.ics.uci.edu/MLSummary.html>

  However, the methods described in Non-Patent Documents 4 to 7 have the following problems.

  First, the methods described in Non-Patent Documents 4 and 5 require a learning machine that performs semi-supervised learning as a lower learning machine used for boosting, and cannot use a conventional supervised learning method. Therefore, there is a problem that applicable data formats are limited and versatility is poor. In addition, there is a problem that it takes a calculation time for large-scale data.

  On the other hand, in the method described in Non-Patent Document 6, it is possible to use a supervised learning machine as a boosting low-order learning machine that was a problem in Non-Patent Documents 4 and 5. However, it has been reported that there is a problem in the formulation of learning, so that the effect of improving the classification accuracy is poor and the performance may deteriorate.

  In the method described in Non-Patent Document 7, a loss function related to test data is introduced into the objective function of the support vector machine. However, since the calculation method and the optimization method are complicated, tuning and calculation of many learning parameters are required. There is a problem that takes time.

  Therefore, in view of the above-described problems, the present invention is based on learning theory, and by using boosting to minimize the variance of predicted values for unlabeled data while minimizing the loss function of training data, It is an object of the present invention to provide a practical high-precision classifier that can be applied to supervised learning and has a small number of learning parameters.

  According to a first aspect of the present invention, in a semi-supervised learning method based on ensemble learning as a method, a data storage step for storing storage data, test data and unlabeled data, and training data recorded in the data storage step An initial learning step for performing supervised learning based on the above, a calculation result storing step for storing data calculated in the initial learning step, and data obtained by combining attributes of training data and unlabeled data recorded in the data storing step And a learning step for repeatedly performing supervised learning based on the data stored in the calculation result storing step.

  Furthermore, according to the second aspect of the present invention, in the semi-supervised learning device based on ensemble learning as the first device, a training data storage unit that stores training data, and a test that stores unlabeled data and test data A data storage unit, an initial learning unit that performs supervised learning using training data, a parameter storage unit that stores gradients, and a learning unit that combines training data, unlabeled data, and test data to repeatedly perform supervised learning And a discriminant function storage unit for storing the learned discriminant function, and a discriminator unit for predicting the label of the test data using the discriminant function.

  Further, according to a third aspect of the present invention, in the semi-supervised learning device based on ensemble learning as the second device, data storage means for storing storage data, test data and unlabeled data, and the data storage means Initial learning means for performing supervised learning based on the training data recorded in the above, calculation result storage means for storing data calculated in the initial learning means, training data recorded in the data storage means and unlabeled data There is provided a semi-supervised learning apparatus comprising learning means for repeatedly performing supervised learning based on data obtained by combining attributes and data stored in the calculation result storage means.

  Furthermore, according to the fourth aspect of the present invention, in a semi-supervised learning program based on ensemble learning as a first program, a training data storage function for storing training data, and a test for storing unlabeled data and test data Data storage function, initial learning function that performs supervised learning using training data, parameter storage function that stores gradients, and learning function that combines training data with unlabeled data and test data to perform supervised learning repeatedly And a discriminant function storage function for storing the learned discriminant function, and a discriminant function for predicting the label of the test data using the discriminant function.

  Furthermore, according to the fifth aspect of the present invention, in the semi-supervised learning program based on ensemble learning as the second program, a data storage function for storing storage data, test data and unlabeled data, and the data storage function An initial learning function for performing supervised learning based on the training data recorded in the above, a calculation result storage function for storing data calculated in the initial learning function, and training data and unlabeled data recorded in the data storage function. Provided is a semi-supervised learning program that causes a computer to realize a learning function that repeatedly performs supervised learning based on data combined with attributes and data stored in the calculation result storage function .

  According to the present invention, not only training data but also the distribution of unlabeled data is learned, so that a high-precision classifier can be configured from a small number of training data.

  Next, embodiments of the present invention will be described with reference to the drawings.

  The present invention uses not only training data but also test data to perform learning by boosting, and uses an arbitrary supervised learning machine as a subordinate learning machine, so that even if there is little training data, high-precision classification It is a new way to configure the vessel. EMBODIMENT OF THE INVENTION Below, the form for implementing this invention is demonstrated with reference to drawings.

  Referring to FIG. 1, an embodiment of the present invention includes an input device 1 such as a keyboard, a data processing device 2 that operates under program control, a storage device 3 that stores information, and an output device such as a display device and a printing device. 4 is provided.

  The data processing device 2 includes an initial learning unit 21, a learning unit 22, and a determination unit 23.

  The initial learning unit 21 is a part that performs learning by boosting using only training data and calculates the gradient of the test data. The learning unit 22 is a part that performs learning by boosting using the gradients of the training data and test data as labels and updates the discriminant function. The discriminating unit 23 is a part that predicts the label of the test data using the learned discriminant function.

  The storage device 3 includes a training data storage unit 31 that stores training data, a test data storage unit 32 that stores test data, a parameter storage unit 33 that stores a gradient of a loss function, and a discriminant function storage that stores a discriminant function. The unit 34 is provided.

  Next, the operation of the embodiment for carrying out the present invention will be described with reference to FIGS.

  First, an execution instruction is given by the input device 1, and training data and test data are input from the training data storage unit 31 and the test data storage unit 32 to the data processing device 2 (FIG. 2, step A1).

  Next, supervised learning of the discriminant function F using training data is performed by the initial learning unit 21 (step A4 in FIG. 2). The specific operation (step A2 and step A3) of the initial learning unit 21 will be described below.

  Then, the discriminant function is iteratively updated (FIG. 2, step A5), and the gradient of the training data by the discriminant function is calculated (FIG. 2, step A6).

  Thereafter, the calculated gradient is stored in the parameter storage unit 33 and the discriminant function is stored in the discriminant function storage unit 34.

  A specific operation of the initial learning unit 21 will be described with reference to FIG.

  First, training data and test data are input from the training data storage unit 31 and the test data storage unit 32 to the data processing device 2 (FIG. 2, step A1).

Next, the initial learning unit 21 sets t _{1 as} the number of iterations and ν ₁ as a reduction parameter (FIG. 2, step A2). Further, the instruction parameter T for counting the number of iterations is initialized to 1 (FIG. 2, step A3).

  Supervised learning of the discriminant function F using training data is performed (FIG. 2, step A4).

The discriminant function F _T-1 obtained in the Round T-1, by adding a discriminant function F obtained by learning, updating a discriminant function F _T (FIG. 2, step A5). The formula is shown below.

ここで、ν_１（０＜ν_１≦１）は縮小パラメータであり、オーバーフィッティングを防ぐ正則化のために導入されている。ν１が導入されていることにより汎化誤差を最小にするようなＦ_Ｔに収束することとなる。

Here, ν ₁ (0 <ν ₁ ≦ 1) is a reduction parameter, and is introduced for regularization to prevent overfitting. ν1 becomes possible to converge the generalization error F _T that minimizes by is introduced.

Next, using the discriminant function F _T, determining the gradient of the training data to minimize the loss function (Fig. 2, step A6).

  Here, the boosting loss function in supervised learning is designed for the purpose of simultaneously minimizing the error rate of the test data by minimizing the error rate of the training data. Non-Patent Document 8 describes a method of searching for a minimizing direction of a loss function by differentiating the loss function with a discriminant function by a method called gradient boosting.

  As a specific loss function, there is a function represented by the following formula.

ここで、Ｌは損失関数、ｙｉはｉ番目のデータのクラスラベル、ｙｉ∈（−１，１）、Ｆ（ｘｉ）は判別関数であり、ｉ番目のデータの属性ｘｉが与えられたときの出力を表す。損失関数は凸関数であるので、損失関数ＬをＦ（ｘｉ）で微分し、負をとることによって、損失関数の最小化方向が得られる。

Here, L is a loss function, yi is a class label of the i-th data, yiε (−1,1), F (xi) is a discriminant function, and an attribute xi of the i-th data is given. Represents the output. Since the loss function is a convex function, the loss function L can be minimized by differentiating the loss function L with F (xi) and taking the negative value.

ここで、ｇｉはデータｘｉの勾配であり、Ｄｌａｂｅｌｅｄは訓練データの属性とラベルの組の集合である。

Here, gi is the gradient of data xi, and Labeled is a set of attribute data and label pairs of training data.

  Next, 1 is added to the instruction parameter T for the number of iterations (FIG. 2, step A7). When the value of T reaches a predetermined constant t, the learning is finished. On the other hand, if the constant t has not been reached, the process returns to supervised learning (FIG. 2, step A4) of the discriminant function with the gradient as the label (FIG. 2, step A8).

Then, to calculate the gradient of the test data by using the discriminant function F _T (FIG. 2, step A9).

  Here, the test data is unlabeled data. Therefore, the same loss function and gradient as the training data cannot be used for the test data. However, Non-Patent Document 9 shows that the generalization performance of boosting can be improved by reducing the variance of predicted values of unlabeled data obtained from a set of lower learning machines. That is, the performance can be improved if a gradient that minimizes the variance of the predicted value of unlabeled data is obtained for the test data.

  Therefore, a gradient capable of sequentially minimizing the variance is derived as follows. Assuming that the variance of the predicted values when the number of lower learning machines is L and the number of lower learning machines is L + 1 is VL and VL + 1, respectively.

と書ける。ここで、ｆｌはｌ番目に学習された判別関数であり、ｘｉはテストデータである。ここで、Ｌ＋１番目の下位学習機械で求めるべき勾配をｇとすると、

Can be written. Here, fl is the l-th learned discriminant function, and xi is test data. Here, if the gradient to be obtained by the (L + 1) th subordinate learning machine is g,

と書き直すことができる。ｇはΔＶ＝ＶＴ−ＶＴ＋１を最大化するように求めればよい。そこで、ｇによりΔＶの微分が０になるようにｇを定める。

Can be rewritten. g may be obtained so as to maximize ΔV = VT−VT + 1. Therefore, g is determined so that the derivative of ΔV becomes 0 by g.

ここで、ΔＶがｇに関する２階微分が常に負であることから、凹関数であり、大域的な最大値を求めることができることが保証されている。また、上記のｇをΔＶに代入すると、ΔＶ＞０であるので、単調に分散を減少させることができる。

Here, since ΔV is always negative in the second derivative with respect to g, it is a concave function, and it is guaranteed that a global maximum value can be obtained. Further, when the above g is substituted for ΔV, ΔV> 0, so that dispersion can be monotonously reduced.

The gradient of training data and test data obtained by the initial learning unit 21 is stored in the parameter storage unit 33, and the learned discriminant function _FT is stored in the discriminant function storage unit 34.

  The initial learning unit 21 is the same algorithm as gradient boosting, and learns the gradient of test data in order to obtain the variance of predicted values from a plurality of discriminant functions.

  Next, the operation of the learning unit 22 will be described with reference to FIG.

  First, training data and test data are input to the learning unit 22 from the training data storage unit 31 and the test data storage unit 32. Further, the gradient of the training data and the test data obtained by the initial learning unit 21 is input from the parameter storage unit 33 to the learning unit 22 (FIG. 3, step B1).

The number of iterations t ₂ and the reduction parameter ν ₂ are set (FIG. 3, step B2). The instruction parameter T for counting the number of iterations is initialized to 1 (FIG. 3, step B3), and supervised learning of the discriminant function F using the training data and test data is performed using the gradient of the training data and test data as a label (FIG. 3). 3, Step B4).

Then, the discriminant function F _T-1 obtained in the Round T-1, by adding a discriminant function F obtained by learning, updating a discriminant function F _T (FIG. 3, step B5).
Obtained using a discriminant function F _T, calculate the slope of the training data and the test data (FIG. 3, step B6).

  Similarly to the initial learning unit 21, the gradients of the training data and the test data are obtained, and then 1 is added to the instruction parameter T for the number of iterations (FIG. 3, step B7).

Once you reach a constant t ₂ where T is a predetermined, it ends the learning. On the other hand, if not reached, the process returns to supervised learning (FIG. 3, step B4) of the discriminant function with the gradient as a label (FIG. 3, step B8). The learned discriminant function is stored in the discriminant function storage unit 34.

  In the discriminative learning unit 23, test data is input from the test data storage unit 32, and a discriminant function is input from the discriminant function storage unit 34 to predict the label of the test data.

  As the label information, for example, in the medical / biological field, the presence / absence of a disease or a drug effect, the progress of a disease state, and the like can be used. As a supervised learning method, for example, ensemble learning such as bagging or boosting, a support vector machine, a decision tree, or a survival tree can be used. Note that the above-described label information and supervised learning method are merely examples, and other label information and supervised learning methods can be used.

  The prediction result of the test data is output from the output device 4.

  The semi-supervised learning device can be realized by hardware, software, or a combination thereof.

  Next, examples of the present invention will be described.

  As data used for implementation, clinical information on diabetes was obtained from the UCI repository of machine learning benchmark data (see Non-Patent Document 11).

  Performance evaluation was performed based on 8 items of clinical information among 768 patients examined for the presence or absence of diabetes onset (attribute name: diabets). Of the 768 patients, 268 have been diagnosed with diabetes and 500 have been diagnosed with no onset. The missing values in the attributes of clinical information were supplemented by the most frequent category for category data and the median for numerical data.

The reduction parameters ν ₁ and ν ₂ = 1 were used as learning parameters of the present invention, and the Adaboost type exponential function of gradient boosting was used as the loss function of training data. CART, which is one of the decision trees, was used as the lower learning machine. Details of CART are described in Non-Patent Document 10. For convenience of explanation, the method of the present invention is referred to as SSBoost (Semi-Supervised Boosting).

  As a control method of the present invention, gradient boosting using only training data and CART were used. Details of gradient boosting are described in Non-Patent Document 8. For convenience, the control method is referred to as Adaboost. SSBoost performed learning about training data 20 times (t1 = 20) in exactly the same manner as Adaboost, and then learned 20 times using the gradient of test data (t2 = 20).

  The number of Adaboost iterations was 40. SSBoost and Adaboost performed a total of 40 repetitive learnings, and performed a fair performance comparison under the same conditions. The results of the performance comparison are shown in FIG.

  From the results shown in FIG. 4, it can be seen that the method SSBoost of the present invention always has high classification performance compared to Adaboost and CART.

  As described above, by implementing the present invention, it is possible to realize a highly accurate classifier to which supervised learning can be applied as a lower learning machine.

It is a figure showing the basic composition of the embodiment of the present invention. It is a flowchart which shows the process sequence of the initial learning part in embodiment of this invention. It is a flowchart which shows the process sequence of the iterative learning part in embodiment of this invention. It is a figure showing the result of the performance comparison using the gradient boosting which used only training data, and CART as a control method of embodiment of this invention.

Explanation of symbols

1 Input device 2 Data processing device 3 Storage device 4 Output device 21 Initial learning unit 22 Learning unit 23 Discriminating means 31 Training data storage unit 32 Test data storage unit 33 Parameter storage unit 34 Discriminant function storage unit

Claims (32)

In a semi-supervised learning method based on ensemble learning,
A data storage step for storing stored data, test data and unlabeled data;
An initial learning step for performing supervised learning based on the training data recorded in the data storage step;
A calculation result storing step for storing data calculated in the initial learning step;
A learning step for repeatedly performing supervised learning based on data obtained by combining the attributes of the training data recorded in the data storage step and unlabeled data and the data stored in the calculation result storage step. Semi-supervised learning method. The semi-supervised learning method according to claim 1,
The semi-supervised learning method, wherein the supervised learning in the learning step is performed using the gradient of the training data and the gradient of the test data calculated in the initial learning step as labels. The semi-supervised learning method according to claim 1 or 2,
A semi-supervised learning method, wherein the supervised learning method in the initial learning step is boosting or gradient boosting. The semi-supervised learning method according to any one of claims 1 to 3,
A semi-supervised learning method, wherein a loss function related to training data in the initial learning step is a convex function. The semi-supervised learning method according to any one of claims 1 to 4,
A semi-supervised learning method, wherein a loss function related to training data in the initial learning step is an exponential function. The semi-supervised learning method according to any one of claims 1 to 5,
A semi-supervised learning method, further comprising an initial regularization step of multiplying a discriminant function learned in each round in the initial learning step by a reduction parameter. The semi-supervised learning method according to any one of claims 1 to 6,
A semi-supervised learning method, wherein the lower learning machine in the initial learning step is a decision tree. The semi-supervised learning method according to any one of claims 1 to 7,
A semi-supervised learning method, wherein the supervised learning method in the learning step is boosting or gradient boosting. The semi-supervised learning method according to any one of claims 1 to 8,
The semi-supervised learning method, wherein the unlabeled data in the learning step is test data. The semi-supervised learning method according to any one of claims 1 to 9,
A semi-supervised learning method, wherein a loss function related to training data in the learning step is a convex function. The semi-supervised learning method according to any one of claims 1 to 10,
A semi-supervised learning method, wherein a loss function related to training data in the learning step is an exponential function. The semi-supervised learning method according to any one of claims 1 to 11,
A semi-supervised learning method characterized in that, in the learning step, a variance of predicted values of a lower learning machine with respect to unlabeled data is minimized. The semi-supervised learning method according to any one of claims 1 to 12,
A semi-supervised learning method, further comprising a regularization step of multiplying a discriminant function learned in each round in the learning step by a reduction parameter. The semi-supervised learning method according to any one of claims 1 to 13,
A semi-supervised learning method, wherein the lower learning machine in the learning step is a decision tree. The semi-supervised learning method according to any one of claims 1 to 14,
A semi-supervised learning method, further comprising a prediction step of predicting a label of test data based on a learning result obtained in the learning step. In a semi-supervised learning device based on ensemble learning,
A training data storage unit that stores training data, a test data storage unit that stores unlabeled data and test data, an initial learning unit that performs supervised learning using training data, a parameter storage unit that stores gradients, and training data A learning unit that repeatedly performs supervised learning, a discriminant function storage unit that stores the learned discriminant function, and a discriminant that predicts the label of the test data using the discriminant function A semi-supervised learning apparatus characterized by comprising a unit. In a semi-supervised learning device based on ensemble learning,
Data storage means for storing stored data, test data and unlabeled data;
Initial learning means for performing supervised learning based on the training data recorded in the data storage means;
Calculation result storage means for storing data calculated in the initial learning means;
Learning means for repeatedly performing supervised learning based on data obtained by combining the attributes of the training data recorded in the data storage means and unlabeled data and the data stored in the calculation result storage means. A semi-supervised learning device. The semi-supervised learning device according to claim 17,
The semi-supervised learning apparatus, wherein the supervised learning in the learning means is performed using the gradient of the training data and the gradient of the test data calculated by the initial learning means as labels. The semi-supervised learning device according to claim 17 or 18,
A semi-supervised learning apparatus, wherein the supervised learning method in the initial learning means is boosting or gradient boosting. The semi-supervised learning device according to any one of claims 17 to 19,
A semi-supervised learning apparatus, wherein a loss function related to training data in the initial learning means is a convex function. The semi-supervised learning device according to any one of claims 17 to 20,
A semi-supervised learning apparatus, wherein a loss function related to training data in the initial learning means is an exponential function. The semi-supervised learning device according to any one of claims 17 to 21,
A semi-supervised learning apparatus, further comprising first regularization means for multiplying a discriminant function learned in each round in the initial learning means by a reduction parameter. The semi-supervised learning device according to any one of claims 17 to 22,
A semi-supervised learning apparatus, wherein the lower learning machine in the initial learning means is a decision tree. The semi-supervised learning device according to any one of claims 17 to 23,
A semi-supervised learning apparatus, wherein the supervised learning method in the learning means is boosting or gradient boosting. The semi-supervised learning device according to any one of claims 17 to 24,
A semi-supervised learning apparatus, wherein the unlabeled data in the learning means is test data. The semi-supervised learning device according to any one of claims 17 to 25,
A semi-supervised learning apparatus, wherein a loss function related to training data in the learning means is a convex function. The semi-supervised learning apparatus according to any one of claims 17 to 26,
A semi-supervised learning apparatus, wherein a loss function relating to training data in the learning means is an exponential function. The semi-supervised learning device according to any one of claims 17 to 27,
The semi-supervised learning device characterized in that in the learning means, the variance of the predicted value of the lower learning machine with respect to unlabeled data is minimized. The semi-supervised learning device according to any one of claims 17 to 28,
A semi-supervised learning apparatus, further comprising second regularization means for multiplying a discriminant function learned in each round in the learning means by a reduction parameter. The semi-supervised learning device according to any one of claims 17 to 29,
A semi-supervised learning apparatus, wherein the lower learning machine in the learning means is a decision tree. In a semi-supervised learning program based on ensemble learning,
Training data storage function for storing training data, test data storage function for storing unlabeled data and test data, initial learning function for supervised learning using training data, parameter storage function for storing gradients, and training data A learning function that combines unlabeled data and test data to perform supervised learning repeatedly, a discriminant function storage function that stores learned discriminant functions, and a discriminant that predicts test data labels using discriminant functions A semi-supervised learning program characterized by having a computer realize the function. In a semi-supervised learning program based on ensemble learning,
A data storage function for storing stored data, test data and unlabeled data;
An initial learning function for performing supervised learning based on the training data recorded in the data storage function;
A calculation result storage function for storing data calculated in the initial learning function;
The computer realizes a learning function that repeatedly performs supervised learning based on data obtained by combining the attributes of the training data recorded in the data storage function and unlabeled data and the data stored in the calculation result storage function. A semi-supervised learning program characterized by

Images