JP2010003106A - Classification model generation device, classification device, classification model generation method, classification method, classification model generation program, classification program and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a high-accuracy classification model related to a target classification system by efficiently using not only data on the target classification system but also data on an auxiliary classification system. <P>SOLUTION: This classification device 1 uses not only already classified data in the target classification system but also already classified data in the auxiliary classification system, estimates weight so as to minimize expected error as total of a product of an error function and the weight, generates the classification model by use of the estimated weight and two kinds of already classified data to effectively use not only the data of the target classification system but also the data of the auxiliary classification system, and can generate the high-accuracy classification model related to the target classification system. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention uses not only data of a classification system (hereinafter referred to as “target classification system”) for classifying classification target data but also data of another classification system (hereinafter referred to as “auxiliary classification system”). The present invention relates to a technique for learning a classification model and classifying classification target data in a target classification system using the learned classification model.

  When the number of learning data (learning data) is small, generally the performance of the classification model is low. Therefore, it is preferable that the performance of the classification model can be improved by using data with a class label (hereinafter referred to as “class label” or simply “label”) in the auxiliary classification system. In this case, for example, it is assumed that a certain Web page is classified into a class of a certain target classification system (hereinafter also referred to as “target class”). It is desirable if many web pages have already been classified into an auxiliary classification system different from the target classification system by a large number of users in a directory search engine or a social bookmark site, and such information can be utilized.

In addition, for example, a technique related to prediction (classification) in consideration of a purchase order for products such as online shopping is known (see Non-Patent Document 1).
Tomoharu Iwata, Takeshi Yamada, Nobuo Ueda, “Collaborative Filtering Considering Purchase Order”, Artificial Intelligence and Knowledge Processing Study Group, AI2007-3,13-18,2007

  However, the class label is generally different between the auxiliary classification system and the target classification system, and even if the same label exists, the meaning may be different. Therefore, there is a problem that data of a class of an auxiliary classification system (hereinafter also referred to as “auxiliary class”) cannot be used by using a conventional supervised learning technique (Non-Patent Document 1 or the like).

  Therefore, the present invention has been made in view of the above problems, and an object thereof is to generate a highly accurate classification model related to a target classification system by effectively using data of an auxiliary classification system. Another object is to use the generated classification model to classify the classification target data with high accuracy in the target classification system.

  In order to solve the above problems, the present invention provides one or more already classified data already classified in a target classification system that is a classification system for classifying classification target data, and a classification system different from the target classification system. Classification for classifying the data to be classified into one of a plurality of classes in the target classification system by performing learning using one or more already classified data already classified in a certain auxiliary classification system A classification model generation device for generating a model, wherein storage means for storing information and prediction that each individual classification data in the two types of classification data described above is classified into one of the classes of the target classification system Error function of the classification model at the time of the prediction, and each individual fraction in the two types of previously classified data at the time of the prediction Each weight indicating the degree of influence of the data on the classification model, and the sum of products of the error function value and the weight for each of the already classified data in the two types of previously classified data. In order to minimize the expected error, the weight is estimated, the weight estimation unit that stores the weight in the storage unit, the weight stored in the storage unit, and the two types of already classified data described above, And a model construction unit for generating the classification model.

  According to this invention, not only the already classified data in the target classification system but also the already classified data in the auxiliary classification system is used, and the weight is estimated so as to minimize the expected error that is the sum of the products of the error function and the weight. By generating a classification model using the estimated weight and two types of already-classified data, it is possible to effectively use the data of the auxiliary classification system and generate a highly accurate classification model for the target classification system. it can.

  Further, the present invention provides the target classification system for approximating a probability distribution model when the weight estimation unit integrates the target classification system and the auxiliary classification system to a probability distribution model of the target classification system. And the auxiliary classification system are used to estimate the posterior probabilities related to the two types of already classified data using a mixing ratio indicating the ratio of the degree of influence on the classification model for each class, and the storage means stores the posterior probabilities The mixture ratio is estimated using the posterior probability estimation unit stored in the storage means and the posterior probability stored in the storage means so as to maximize the likelihood for the already classified data of the target classification system, and the likelihood And a mixture ratio estimation unit that estimates the weight from the mixture ratio when the value is maximized and stores the weight in the storage means.

  According to this invention, since the weight estimation unit includes the posterior probability estimation unit and the mixture ratio estimation unit, for example, the posterior probability estimation unit performs the E (Expectation) step in the EM (Expectation-Maximization) algorithm. And the mixture ratio estimation unit performs the M (Maximization) step, thereby obtaining a global optimum solution for the mixture ratio and determining (estimating) the weight from the obtained mixture ratio.

  Further, according to the present invention, the model construction unit uses the weight stored in the storage unit and the two types of already classified data, and the classification target data in the classification model is included in the target classification system. A model parameter estimating unit for estimating model parameters for classification and storing the model parameters in the storage unit is provided.

  According to this invention, a model parameter estimation part can estimate a model parameter using the formula (10) described later, for example.

  Further, according to the present invention, there is provided a classification unit that classifies the classification target data into any one of a plurality of classes in the target classification system using a model parameter stored in the storage unit of the classification model generation device. It is characterized by providing.

  According to this invention, the classification unit classifies the classification target data into any one of a plurality of classes in the target classification system using the estimated model parameter, that is, a high-precision classification using a high-precision classification model. Can be realized.

  In addition, the present invention is a program for causing a computer to function as each unit of the classification model generation device or the classification device. Thereby, the computer installed with this program can realize each function based on this program.

  The present invention also provides a computer-readable recording medium in which the program is recorded. Thereby, the computer equipped with this recording medium can realize each function based on the program recorded on this recording medium.

  According to the present invention, it is possible to generate a highly accurate classification model related to the target classification system by effectively using the data of the auxiliary classification system. Further, using the generated classification model, the classification target data can be classified with high accuracy in the target classification system.

  Hereinafter, the best mode for carrying out the present invention (hereinafter referred to as “embodiment”) will be described in detail. FIG. 1 is a block diagram showing the configuration of the classification apparatus according to this embodiment. As shown in FIG. 1, the classification device 1 includes a calculation unit 2, an input unit 3, a storage unit 4, and an output unit 5. Each means 2 to 5 is connected to the bus line 11. The classification device 1 has a function as a classification model generation device that generates a classification model (hereinafter, also simply referred to as “model”), and a function as a classification device that classifies the classification target data using the generated classification model. However, it may be realized as having only one of the functions.

  The computing means 2 is a main control device composed of, for example, a CPU (Central Processing Unit) and a RAM (Random Access Memory). As shown in FIG. 1, the calculation unit 2 includes a weight estimation unit 21, a model construction unit 22, a classification unit 23, and a memory 24. In addition, although description of each part 21-23 is mentioned later, since the main characteristic in this embodiment at the time of comparing with the conventional method is the weight estimation part 21, it demonstrates in detail regarding the weight estimation part 21 in particular. Further, the model construction unit 22 and the classification unit 23 can be applied without greatly changing the conventional method, and thus detailed description thereof is omitted.

  The input unit 3 includes, for example, a keyboard, a mouse, a disk drive device, and the like. The input means 3 inputs various data and stores them in the storage means 4 (details will be described later).

  The storage means 4 is composed of, for example, a general hard disk device or the like, and stores various programs and various data used by the calculation means 2. The storage means 4 stores a weight estimation program 41, a model construction program 42, and a classification program 43 as programs in the program storage unit 40a. And the calculating means 2 reads these programs 41-43 from the memory | storage means 4, expand | deploys to the memory 24, and performs them, and each function of the above-mentioned weight estimation part 21, the model construction part 22, and the classification | category part 23 is carried out. Realize.

  The storage unit 4 stores the input data 44, the weight 45, the model parameter 46, and the test data 47 in the data storage unit 40b. Here, the input data 44 is data input from the input means 3 and is a learning sample. The weight 45 is data relating to the weight estimated by the calculation processing of the weight estimation unit 21 of the calculation means 2 (details will be described later). The model parameter 46 is data calculated by the calculation process of the model construction unit 22 of the calculation unit 2 (details will be described later). The test data 47 is a test sample (details will be described later). The input data 44, the weight 45, the model parameter 46, and the test data 47 are appropriately omitted below.

  The output means 5 is, for example, a graphic board (output interface) and a monitor connected thereto. This monitor is composed of, for example, a liquid crystal display or the like, and displays an operation processing result (classification result of classification target data, etc.).

  In the present embodiment, not only the data of the target classification system (already classified data; hereinafter also referred to as “target data”) but also the data of the auxiliary classification system (hereinafter referred to as already classified data, also referred to as “auxiliary data”) is used. Learn the classifier (classification model). The target class set is Z, the auxiliary class set is A, and all class sets are Y = {Z, A}.

As the learning data, target data D _z = {x _n , y _n } ^Nz _{n = 1} (In this specification, “ ^Nz _{n = 1} ” substitutes “1” to “N _z ” for “n”. Meaning the same for other characters)
_Given auxiliary data D _a = {x _n , y _n } ^N _{n = Nz + 1} , class y∈Z of sample x (classification target data; test data 47 described later) whose class is unknown is predicted. Learn classification models.

Here, in the case of Web page data, a sample is represented by, for example, a word appearance frequency vector x _n = (x _n1 ,..., X _nw ) (x _nw represents the number of times the word w has appeared in the nth sample. ).

Further, y _n εZ (if 1 ≦ n ≦ N _z ), y _n εA (if N _z + 1 ≦ n ≦ N), and in the case of a Web page, y _n is a category in which the nth sample is classified. Represents. Y is a discrete value. Moreover, although x is treated as a discrete variable, it can be easily extended to the case of a continuous variable.

In the present embodiment, the model M is learned by minimizing the weighted experience error E (M) (equation (1)) for all data including auxiliary data (data of the auxiliary classification system) in the target data.

Here, w (z | y) represents a weight representing how much a sample of class yεY is useful for model learning of target class zεZ. In Expression (1), bold characters (here, _xn and Z) indicate that they have a plurality of components, and the same applies to the other expressions below. In addition, the characters in the text are not shown in bold but are consistent with the respective expressions.

J (x _n , z; M) represents an error function of the model M when the class of the sample x is predicted as z. As an example of the error function,
Negative log likelihood J (x, z; M) = − logP (z | x; M),
0-1 loss function J (x, z; M) = 0 (if f (x) = y),
J (x, z; M) = 1 (otherwise) can be considered. In this specification, the logarithm is a natural logarithm, that is, the base of the logarithm log is “e”.

The weight is determined as follows (the action subject will be described later, and so on). First, a model distribution P ^~ (x | y) approximating an empirical distribution in class y (in this specification, a symbol " ^~ " meaning an empirical distribution is a symbol added on the immediately preceding letter. The same applies to “^” described later (formula (2)). Where δ (x, x _n ) represents the Kronecker delta and N (y) represents the number of samples with class y.

Next, the mixture ratio P _z (y) is estimated so that the mixture of all classes of the model distribution approximates the true distribution P (x | z) of the target class z∈Z (formula (3)). Here, the mixture ratio is the class of the target classification system and the auxiliary classification system for approximating the probability distribution model when the target classification system and the auxiliary classification system are integrated to the probability distribution model of the target classification system. The ratio of the degree of influence on the classification model is shown.

The mixing ratio _P z (y), and the set P of the mixed ratio _P z (y) is
P = {{P _z (y)} _y∈Y } _z∈Z (0 ≦ P _z (y) ≦ 1, Σ _y∈Y P _z (y) = 1).

Then, the weight w (z | y) is set (calculated) (formula (4)). Note that P (z) is a probability that a class z is selected for a certain sample.

At this time, the weighted error E (M) is an approximation of the expected error (formula (5)).

  In Expression (5), the expression modification from the first line to the second line on the right side is obtained by changing the summation expression for n to the summation expression for x and y. The expression transformation from the second line to the third line on the right side uses Expression (4). Expression transformation from the third line to the fourth line on the right side uses Expression (2). Expression transformation from the fourth line to the fifth line on the right side uses Expression (3). Formula transformation from the 5th line to the 6th line on the right side uses a conditional probability formula (definition), and P (x, z) indicates the probability that x and z occur simultaneously.

The expression transformation from the 6th line to the 7th line on the right side uses the expected value formula (definition), and ε _z [J (x, z; M)] is the expected value of the error for the target class z. Indicates. For this reason, it can be expected that a robust (high-precision) model can be estimated by minimizing the weighted error E (M) using auxiliary data.

The set P satisfying the approximation of Expression (3) is estimated by maximizing the log likelihood L (P) for the target data using an EM (Expectation-Maximization) algorithm (Expression (6)). The EM algorithm is an algorithm that performs maximum likelihood estimation of parameters (here, set P) by repeating two procedures of an E (Expectation) step and an M (Maximization) step until a convergence condition is satisfied.

Here, P ^~ _-n (x | y) represents a model distribution estimated using data excluding the nth sample. When the mixture ratio is estimated using the samples used for estimation of the model distribution, overlearning occurs, and an obvious solution of P _z (z) = 1 and P _z (y ≠ z) = 0 is obtained. The 1eave-one-out (LOO) method is used as shown in Equation (6). When P ^~ _-n (x | y) is estimated and fixed using data of class y, since L (P) is convex upward with respect to P, the global optimality of the solution is guaranteed. Let P ^(τ) be the estimated value at the τ-th step in the EM algorithm. Here, τ indicates the number of times (τ = 0, 1, 2,...) That the two procedures of E step and M step are repeated. In addition, when τ = 0, a predetermined initial value of the estimated value is shown. At this time, the conditional expected value Q (P | P ^(τ) ) of the complete data log likelihood to be maximized can be expressed as Equation (7).

The calculation in the E step can be expressed as shown in Equation (8). Note that y ′ in the denominator on the right side of Equation (8) is the same meaning as y except that the symbol is changed for the sake of distinction from y in other parts of Equation (8).

The calculation in the M step can be expressed as Equation (9).

  By repeating the calculation at the E step and the calculation at the M step until the convergence condition is satisfied, an estimated value of the set P is obtained.

  Note that the set P can also be estimated by maximizing Equation (6) using another optimization method such as the quasi-Newton method instead of the EM algorithm.

<Weight estimation>
The configuration of the weight estimation unit 21 will be described with reference to FIG. FIG. 2 is a diagram including a block diagram of a weight estimation unit according to the present embodiment. As shown in FIG. 2, the weight estimation unit 21 includes an input data reading unit 211, a posterior probability estimation unit 212, a mixture ratio estimation unit 213, and a weight writing unit 214.

First, the input data reading unit 211 reads the input data 44. Then, the posterior probability estimation unit 212 estimates the posterior probability of all the learning samples with respect to all times using the equation (8), and the mixture ratio estimation unit 213 estimates the mixture ratio using the equation (9). The posterior probability estimation and the mixture ratio estimation are alternately repeated until the expression (6) converges.
The weight is set (calculated) as w (z | y) = P (z) P _z (y) / N (y) and stored in the weight 45. The stored weight 45 is used by the model construction unit 22.

<Model construction>
The configuration of the model construction unit 22 will be described with reference to FIG. FIG. 3 is a diagram including a block diagram of the model construction unit according to the present embodiment. As shown in FIG. 3, the model construction unit 22 includes an input data reading unit 221, a weight reading unit 222, a model parameter estimation unit 223, and a model parameter writing unit 224.

なお、式（１０）の左辺においてＭに付した記号「＾（ハット）」は、そのＭがargmin関数の引数を最小化させることを示すものである。 First, the input data reading unit 221 reads input data 44. Further, the weight reading unit 222 reads the weight 45. Then, the model parameter estimation unit 223 estimates the model parameter M ^ using Expression (10).

The symbol “記号 (hat)” attached to M on the left side of Expression (10) indicates that M minimizes the argument of the argmin function.

  The model parameter writing unit 224 stores the model parameter estimated by the model parameter estimation unit 223 in the model parameter 46. The stored model parameter 46 is used by the classification unit 23.

  The configuration of the classification unit 23 will be described with reference to FIG. FIG. 4 is a diagram including a block diagram of a classification unit according to the present embodiment. As shown in FIG. 4, the classification unit 23 includes a test data reading unit 231, a model parameter reading unit 232, and a classification result output unit 233.

  First, the test data reading unit 231 reads unclassified test data 47. The model parameter reading unit 232 reads the model parameter 46. Then, the classification result output unit 233 calculates the classification result using the test data and the model parameters, and outputs the classification result.

  The operation of the classification apparatus 1 shown in FIG. 1 will be described with reference to FIG. 5 (refer to FIG. 1 as appropriate). FIG. 5 is an explanatory diagram showing the flow of processing of the classification device according to the present embodiment.

  First, the classification device 1 uses the weight estimation unit 21 to estimate weights based on the input data 44 stored in advance in the storage unit 4 (see FIG. 1) (step S10: weight estimation step). The estimated weight is stored in the storage unit 4 as the weight 45. Next, the classification device 1 uses the model construction unit 22 to construct a model based on the input data 44 and weights 45 stored in advance in the storage unit 4 (see FIG. 1) (step S20: model construction step). The constructed model is stored in the storage unit 4 as the model parameter 46. Steps S10 and S20 are processes related to model learning.

  Subsequently, the classification device 1 classifies the unclassified test data 47 (classification target data) stored in advance in the storage unit 4 (see FIG. 1) based on the model parameter 46 by the classification unit 23 (step). S30: Classification step). This step S30 is processing relating to the classification of the classification target data.

  Next, the weight estimation step of step S10 described above will be described with reference to FIG. 6 (refer to FIGS. 1 to 5 as appropriate). FIG. 6 is a flowchart showing the weight estimation step.

  First, as shown in FIG. 6, the weight estimation unit 21 reads the input data 44 from the storage unit 4 (see FIG. 1) by the input data reading unit 211 (step S1). Next, the weight estimation unit 21 estimates the model distribution by the posterior probability estimation unit 212 (step S2). Specifically, a model distribution that satisfies the above-described equation (2) is estimated.

Thereafter, the weight estimation unit 21 performs initialization by the posterior probability estimation unit 212 (step S3). Specifically, the posterior probability estimation unit 212 sets the number of repetitions τ of the two steps of the E step and the M step of the EM algorithm to 0, and randomly sets the distribution of the mixture ratio P _z (y).

  Next, the weight estimation unit 21 executes the E step of the EM algorithm by using the posterior probability estimation unit 212 (step S4). Specifically, the posterior probability estimation unit 212 estimates the posterior probability by the above-described equation (8). Subsequently, the weight estimation unit 21 executes the M step of the EM algorithm by the mixture ratio estimation unit 213 (step S5). Specifically, the mixture ratio estimation unit 213 estimates the mixture ratio according to the equation (9). Next, the weight estimation unit 21 determines whether or not the convergence condition is satisfied by the mixture ratio estimation unit 213 (step S6). Specifically, the mixture ratio estimation unit 213 determines whether or not the likelihood L (P) shown in the above equation (6) has converged. This determination of convergence can be made by using a threshold value, a change rate, or the like.

When the convergence condition is satisfied, that is, when the likelihood L (P) shown in the above equation (6) has converged (step S6: Yes), the mixture ratio estimation unit 213 calculates the weight w (z | y). (Step S8). Specifically, the mixture ratio estimation unit 213
The weight is calculated using the formula w (z | y) = P (z) P _z (y) / N (y). Then, the weight estimation unit 21 writes the weight as the weight 45 in the storage unit 4 (see FIG. 1) by the weight writing unit 214 and ends the processing.

  On the other hand, when the convergence condition is not satisfied in step S6, that is, when the likelihood L (P) shown in the above equation (6) is not converged (step S6: No), the weight estimation unit 21 determines that E “1” is added to the number of repetitions τ of the step and the M step (τ = τ + 1) (step S7), and the process returns to step S4.

  According to this embodiment, the classification device 1 uses not only the already classified data in the target classification system but also the already classified data in the auxiliary classification system, and an expected error (formula (5)) that is the sum of the products of the error function and the weight. ) See)) to minimize the weight, and generate a classification model using the estimated weight and two types of already classified data. A highly accurate classification model related to the classification system can be generated.

  Further, since the weight estimation unit 21 includes the posterior probability estimation unit 212 and the mixture ratio estimation unit 213, for example, the posterior probability estimation unit 212 performs the E step in the EM algorithm, and the mixture ratio estimation unit When 213 performs M steps, a global optimum solution for the mixture ratio can be obtained, and a weight can be determined (estimated) from the obtained mixture ratio.

  Further, for example, the model parameter estimation unit 223 can estimate the model parameter using Expression (10).

  Further, the classification unit 23 classifies the classification target data into one of a plurality of classes in the target classification system using the estimated model parameters, that is, realizes high-precision classification by using a high-precision classification model. be able to.

  The classification device 1 can also be realized by causing a general computer to execute the above-described processing programs. This program can be distributed via a communication line, or can be written on a recording medium such as a CD-ROM (Compact Disc-Read Only Memory) for distribution.

  Although description of this embodiment is finished above, the aspect of the present invention is not limited to these. For example, the present invention can use any error function and model. In addition, specific configurations such as hardware and flowcharts can be appropriately changed without departing from the spirit of the present invention.

<Examples of artificial data>
In order to evaluate the classification apparatus 1 of the present embodiment, a two-class classification experiment using artificial data was performed. This two-class classification experiment is an experiment for classifying test data into one of two classes based on a classification model generated from target data and auxiliary data.

It is assumed that target data is generated from two 100-dimensional normal distributions having different averages. Here, the averages of classes c ₁ and c ₂ are μ ₁ = (− 1, 0, 0,..., 0) and μ ₂ = (1, 0, 0,..., 0), respectively. Both covariance matrices are assumed to be unit matrices. Then, the following three patterns are considered as auxiliary data. Note that the average after the third dimension is all 0 as in the target data, and the covariance matrix is all the unit matrix. FIG. 7A shows target data, and FIGS. 7B to 7D show the first and second dimensions of each auxiliary data generation model. FIGS. 7A to 7D do not particularly show axes or scales, but represent a two-dimensional coordinate plane, and the central portion is the origin. Each circle represents a standard deviation line.

The same auxiliary data shown in FIG. 7B is generated from the same generation model as the target data, and the averages of classes c ₃ and c ₄ are μ ₃ = (− 1, 0, 0,..., 0), respectively. , Μ ₄ = (1, 0, 0,..., 0).

The correlation auxiliary data shown in FIG. 7C is generated from a generation model having a correlation between the target data and the class relationship, and the averages of the classes c ₃ and c ₄ are μ ₃ = (− √0.5, √0, respectively. .5,0, ..., 0),
μ ₄ = (√0.5, −√0.5, 0,..., 0).

The mixed auxiliary data shown in FIG. 7D is a combination (mixed) of the same auxiliary data and auxiliary data in which the relationship between the target data and the class is orthogonal, and the class c ₃ , c ₄ , c ₅ , c ₆ The average is μ ₃ = (− 1, 0, 0,..., 0), μ ₄ = (1, 0, 0,..., 0),
μ ₅ = (0, 1, 0,..., 0) and μ ₆ = (0, −1, 0,..., 0). Of the auxiliary data, only this mixed auxiliary data is 4 auxiliary classes, and the others are 2 auxiliary classes.

  Each class 2, 4, 8, 16, 32, 64, 128, 256 samples (input data 44) as target data, each class 256 samples (input data 44) as auxiliary data, each class 100 samples (test data) as test data 47) was produced. Based on these, a classification model was generated, and the correct answer rate for the classification of test data when the auxiliary data was not used (target data only) and when each auxiliary data was used was calculated. As a result, it became as shown in Table 1. In Table 1, the numbers in the right four columns indicate the percentage of the average correct answer rate, and the numbers in parentheses indicate the standard deviation. It can be seen that the use of auxiliary data based on the classification method of the classification device 1 of the present embodiment improves the correct answer rate compared to the case where auxiliary data is not used.

<< Example of text data >>
In order to evaluate the classification device 1 of the present embodiment, a classification experiment was performed using text data.

ここで、Ｖは総語彙数、θ_ｙｊはクラスｙのときｊ番目の単語が出現する確率、ｘ_ｎｊはｎ番目のサンプルにおけるｊ番目の単語の出現頻度を表す。 <Model distribution>
An arbitrary distribution such as a normal distribution or a multinomial distribution can be assumed as the model distribution P ^~ (x | y). Here, text data is assumed as the input data 44 and the test data 47, x is considered as a word appearance frequency vector, and a multinomial distribution P ^to (x _n | y) (formula (11)) is used as a model distribution.

Here, V is the total number of vocabularies, θ _yj is the probability that the j-th word will appear in class y, and x _nj represents the frequency of appearance of the j-th word in the n-th sample.

The LOO maximum likelihood estimation value θ ^ _{−n, yj} when the nth sample of the parameter _θyj of the multinomial distribution is removed is obtained by Expression (12).

Here, in order to avoid the zero probability problem, smoothing is performed using the LOO maximum likelihood estimate and the linear sum of the uniform distribution (Equation (13)).

Here, 0 ≦ α ≦ 1 is a hyper parameter. The hyperparameters may be set manually, but by using the generalized EM algorithm, the set of mixing ratios is maximized so as to maximize the following Q (P, α | P ^(τ) , α ^(τ) ) It is also possible to estimate P and hyperparameter α from the data at the same time (formula (14)).

The E step is expressed by equation (8), and the update of the mixing ratio in the M step is expressed by equation (9), which can be realized in the same manner as a normal EM algorithm. The hyperparameters in the M step are updated using the Newton method (Formula (15)).

Here, the first-order partial differentiation with respect to α in the equation (14) described in the equation (15) becomes the equation (16).

Further, the second-order partial differentiation with respect to α in the equation (14) described in the equation (15) becomes the equation (17).

As is clear from equation (17), the second-order partial derivative is always negative,
Q (P, α | P ^(τ) , α ^(τ) ) is convex upward with respect to α. In this experiment, the set P of mixing ratios and the hyperparameter α were estimated from the data using a generalized EM algorithm.

<Classification model>
A case where a naive Bayes model, which is a typical text classification model, and a logistic regression model are used as the model M will be described.

(Naive Bayes model)
In the naive Bayes model, when a class is given, it is assumed that each word in the document is generated independently, and the distribution P (x | z) of the word appearance frequency vector x in the class z is expressed by a multinomial distribution ( Formula (18)).

Here, φ _zj represents the probability that the j-th word appears in a document of class z. Using a negative log likelihood as an error function, _{also, φ = {{φ zj}} V j = 1} Dirichlet P as the prior probability of ^{_{z∈Z (φ) αΠ z∈Z Π V}} j = 1 φ β _{When zj} is used, the weighted error function E (M _NB ) is expressed as Equation (19).

The estimated value φ ^ _zj of _φzj that minimizes the equation (19) is obtained by the equation (20).

(Logistic regression model)
In the logistic regression model, when a word appearance frequency vector x is given, the probability P (z | x) belonging to the class z is expressed as in Expression (21).

Here, λ _z represents an unknown parameter vector related to class z, and λ _z ^T represents transposition of λ _z . When a negative log likelihood is used as an error function, and a normal distribution of mean 0 and covariance matrix γ ⁻¹ I (I is a unit matrix) is used as a prior probability of λ _z , a weighted error (expected error) E (M _LR ) is expressed as in Expression (22).

The unknown parameter vector {λ _z } _zεZ can be estimated by minimizing the value of Equation (22) using a quasi-Newton method or the like. When a logistic regression model is used, only the error function of each sample is added, so that many classification models that have been proposed so far can be applied with slight modification.

<Comparison method>
This method using the naive Bayes model as a classification model (method by the classification apparatus 1 of this embodiment) (CA-NB), this method using a logistic regression model as a classification model (CA-LR), and auxiliary data Four methods, a method using a naive Bayes model not used (NB) and a method using a logistic regression model (LR), were compared. The estimated value of NB is obtained by setting w (z | z) = 1 and w (z | y ≠ z) = 0 as the weights of Equation (20), which is an estimated value. Similarly, the estimated value of LR is obtained by minimizing the weight of Equation (22), which is a weighted error in this method, as w (z | z) = 1 and w (z | y ≠ z) = 0. It is done.

In each experiment, 100 evaluation data sets were created and evaluated using the average correct answer rate. Also, one development data set is created separately from the evaluation data set, and the hyperparameter (β or γ) of the classification model that maximizes the correct answer rate of the development data set in each method is set to {10 ⁻³ , 10 ^-2 , 10 ^-1 , 1}.

<Toy data>
Using the data set created from 20Newsgroups (20news), the effect of this method is evaluated when the distribution of each auxiliary class is the same as that of a certain target class. 20news consists of about 20,000 English documents submitted to 20 discussion groups. The word appearance frequency was used as the feature value of each document. At this time, stop words (words that are generally called function words, such as prepositions and articles that have no semantic content, and are not useful for search) and words whose appearance frequency is 1 or less are omitted, and the total number of vocabularies Was 52,647.

Of the 20 groups, classify into 5 groups (graphics, os.ms-windows.misc, sys.ibm.pc.hardware, sys.mac.hardware, windows.x) that have computers (comp) in their parent directories. About the problem
The target class set is Z = {c ₁ ,..., C ₅ },
Let the auxiliary class set be A = {c ₆ ,..., C ₁₀ }.

Then, the article graphics to the target class c ₁ or the auxiliary class c _6, the articles os.ms-windows.misc the target class c ₂ or auxiliary class c _7, target class articles sys.ibm.pc.hardware to c ₃ or auxiliary class _{c 8,} articles sys.mac.hardware the target class _{c 4} or auxiliary class _{c 9,} the articles windows.x the target class _{c 5} or auxiliary class _{c 10,} randomly assigned, the target Data and auxiliary data were created.

  At this time, 100 samples of each class were used as test data, 2, 4, 8, 16, 32, 64, 128, 256 samples were used as target data, and all remaining samples were used as auxiliary data. The total number of learning samples was 4,363. The correct answer rate at this time is shown in Table 2. In Table 2, the numbers in the four right columns indicate the percentage of the average correct answer rate, and the numbers in parentheses indicate the standard deviation.

  The correct answer rate of CA-NB and CA-LR, which are the present methods, is extremely high even when the number of learning samples is small, and it can be said that robust (high-accuracy) model estimation can be performed by appropriately using auxiliary data. .

<20Newsgroups data>
Of 20 groups of 20news, 4 groups of comp.graphics, rec.sport.baseba11, sci.electronics, talk.religion.misc are used as target classes, and the other 16 groups are used as auxiliary classes to create data. evaluated. 100 samples for each class as test data 47, 2, 4, 8, 16, 32, 64, 128, 256 samples for target data (input data 44), and all samples for auxiliary data (input data 44). The number of auxiliary samples was 15,211. The correct answer rate at this time is shown in Table 3. In Table 3, the numbers in the right four columns indicate the percentage of the average correct answer rate, and the numbers in parentheses indicate the standard deviation. The correct answer rate of CA-NB which is this method is the highest.

<Web page data>
This method was evaluated using data of a Japanese directory search engine goo (registered trademark) category search (acquired in September 2003) and yahoo (registered trademark) category (acquired in March 2003). Words were extracted by morphological analysis, and words having an appearance count of 10 or more in both categories were used as feature quantities. At this time, the total number of vocabulary was 43,200. There are classes with the same class label in goo (registered trademark) and yahoo (registered trademark), and classes that seem to be related, but there are classes that are difficult to clearly associate, and the number of classes is also different (goo (Registered trademark): 13 classes, yahoo (registered trademark): 14 classes).

  The class of the goo (registered trademark) directory is the target class, 100 samples for each class as test data 47, each class 2, 4, 8, 16, 32, 64, 128, 256 samples for target data (input data 44), auxiliary All samples included in the yahoo (registered trademark) directory were used as data (input data 44). The total number of auxiliary samples was 51,728. The correct answer rate at this time is shown in Table 4. In Table 4, the numbers in the right four columns indicate the percentage of the average correct answer rate, and the numbers in parentheses indicate the standard deviation. The correct answer rate of CA-NR and CA-LR which are the present methods is generally high.

It is a block diagram which shows the structure of the classification device which concerns on this embodiment. It is a figure containing the block diagram of the weight estimation part which concerns on this embodiment. It is a figure containing the block diagram of the model construction part which concerns on this embodiment. It is a figure containing the block diagram of the classification | category part which concerns on this embodiment. It is explanatory drawing which shows the flow of a process of the classification device concerning this embodiment. It is a flowchart which shows the process of a weight estimation step. (A) is target data, (b)-(d) is a figure which shows the 1st, 2nd dimension of the production | generation model of each auxiliary data.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Classifier 2 Calculation means 3 Input means 4 Storage means 5 Output means 11 Bus line 21 Weight estimation part 22 Model construction part 23 Classification part 24 Memory 40a Program storage part 41 Weight estimation program 42 Model construction program 43 Classification program 40b Data storage part 44 Input data 45 Weight 46 Model parameter 47 Test data 211 Input data reading unit 212 A posteriori probability estimating unit 213 Mixing ratio estimating unit 214 Weight writing unit 221 Input data reading unit 222 Weight reading unit 223 Model parameter estimation unit 224 Model parameter writing 231 Test data reading unit 232 Model parameter reading unit 233 Classification result output unit

Claims (11)

One or more already classified data already classified in the target classification system which is a classification system for classifying the classification target data, and one already classified in the auxiliary classification system which is a classification system different from the target classification system A classification model generation device that generates a classification model for classifying the classification target data into one of a plurality of classes in the target classification system by performing learning using the above-described already classified data,
Storage means for storing information;
The error function of the classification model when it is predicted that each individual classification data in the two types of classification data described above is classified into any class of the target classification system, and the above-described 2 when the prediction is performed. The weights indicating the degree of influence on the classification model of each individual classified data in the types of already classified data, and the error function of each of the already classified data in the two types of already classified data A weight estimation unit that estimates the weight and stores the weight in the storage unit so as to minimize an expected error that is a sum of products of the value and the weight;
A model construction unit that generates the classification model using the weight stored in the storage unit and the two types of already-classified data;
A classification model generation device comprising: The weight estimation unit includes:
The classification for each class of the target classification system and the auxiliary classification system for approximating a probability distribution model when the target classification system and the auxiliary classification system are integrated to a probability distribution model of the target classification system A posterior probability estimating unit that estimates the posterior probability of the two types of already-classified data, and stores the posterior probability in the storage unit, using a mixture ratio indicating a ratio of the degree of influence on the model;
The mixture ratio when the likelihood is maximized is estimated by using the posterior probability stored in the storage means so as to maximize the likelihood for the already classified data of the target classification system. Estimating the weight from a ratio, and storing the weight in the storage means;
The classification model generation device according to claim 1, further comprising: The model building unit
Using the weights stored in the storage means and the two types of already classified data, estimating model parameters for classifying the classification target data in the target classification system in the classification model, The classification model generation apparatus according to claim 1, further comprising a model parameter estimation unit that stores model parameters in the storage unit. A classification unit that classifies the classification target data into one of a plurality of classes in the target classification system using model parameters stored in the storage unit of the classification model generation device according to claim 3. Classification device. One or more already classified data already classified in the target classification system which is a classification system for classifying the classification target data, and one already classified in the auxiliary classification system which is a classification system different from the target classification system Generation of a classification model by a classification model generation device that generates a classification model for classifying the classification target data into one of a plurality of classes in the target classification system by performing learning using the above-described already classified data A method,
The classification model generation device includes storage means for storing information, a weight estimation unit, and a model construction unit,
The weight estimation unit is configured to calculate an error function of the classification model when the individual classification data in the two types of classification data is predicted to be classified into any class of the target classification system, and the prediction. Using the respective weights indicating the degree of influence of the individual classified data in the two types of already classified data described above on the classification model, the individual previously classified data in the two types of previously classified data Performing a weight estimation step of estimating the weight and storing the weight in the storage means so as to minimize an expected error that is a sum of products of the value of the error function for each and the weight.
The model construction unit executes a model construction step of creating the classification model using the weights stored in the storage means and the two types of already-classified data described above. Method. The weight estimation unit includes a posterior probability estimation unit and a mixture ratio estimation unit,
In the weight estimation step,
The posterior probability estimation unit is configured to approximate the probability distribution model obtained by integrating the target classification system and the auxiliary classification system to the probability distribution model of the target classification system, and the auxiliary classification system. Using a mixture ratio indicating the ratio of the degree of influence on the classification model for each class with the system, estimating the posterior probability for the two types of already-classified data, and storing the posterior probability in the storage means;
The mixture ratio estimation unit estimates the mixture ratio using the posterior probability stored in the storage unit so as to maximize the likelihood for the already classified data of the target classification system, and the likelihood is maximized. The classification model generation method according to claim 5, wherein the weight is estimated from the mixture ratio at the time of conversion into the storage unit, and the weight is stored in the storage unit. The model construction unit includes a model parameter estimation unit,
In the model building step,
The model parameter estimation unit is a model for classifying the classification target data into the target classification system in the classification model using the weight stored in the storage means and the two types of already classified data. The classification model generation method according to claim 5, wherein a parameter is estimated and the model parameter is stored in the storage unit. Using the model parameters stored in the storage means by the classification model generation method according to claim 7,
The classification unit in the classification device for classifying the classification target data,
Classifying the classification target data into any one of a plurality of classes in the target classification system.   The classification model production | generation program for functioning a computer as each part of the classification model production | generation apparatus as described in any one of Claims 1-3.   The classification program for functioning a computer as a classification | category part of the classification device of Claim 4.   A computer-readable recording medium in which the classification model generation program according to claim 9 or the classification program according to claim 10 is recorded.

Images