JP2010170424A - Distribution estimation apparatus, clustering apparatus, estimation method for distribution estimation apparatus, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To optimize the number of discrete areas of each dimension and the positions of discretization of each dimension by a locally independent mixed discrete distribution when estimating a distribution. <P>SOLUTION: A distribution estimation apparatus includes an initialization means for initializing data, an expectation calculation means for calculating a posterior probability expectation of cluster assignment in the data initialized by the initialization means, a minimization means for calculating a parameter minimizing an information criterion expectation to the expectation of cluster assignment calculated by the expectation calculation means, and calculating a locally independent mixed discrete distribution, and an optimality decision means for determining optimality of the information criterion calculated by the minimization means, and repeating the expectation calculation by the expectation calculation means if determining that optimization is incomplete. The number of discrete areas of each dimension and the positions of discretization of each dimension are optimized. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a multivariate data distribution estimation device, a clustering device, a distribution estimation device estimation method, and a program for optimizing the number of mixtures and the discretization method simultaneously with the distribution estimation.

  Estimating the distribution of data, the predicted distribution of labels attached to the data (hereinafter referred to as label distribution), or the simultaneous distribution of data and labels in a multidimensional space is an industrially important technology (hereinafter referred to as the following). These are collectively called simply distribution).

  For example, when data such as body weight, blood pressure, and medical expenses are given as data, a multidimensional distribution related to three attributes is estimated, so that there are a plurality of relationships such as weight and blood pressure that are likely to cause high medical expenses. It is possible to analyze the relationship of attributes.

  Also, for example, when sensor values such as the engine speed and speed of a car are given as data, and the type of failure for each data is given as a label, new data whose failure type is unknown can be estimated by estimating the distribution of the label. Is acquired, it is possible to calculate the probability of occurrence of each failure and specify the failure type.

  Similarly, the relationship between data and labels can be analyzed by estimating the simultaneous distribution of data and labels.

  When estimating a distribution, the use of a discrete distribution represented by a histogram is that 1) there is no need to assume a specific distribution when the true distribution of data is unknown, and 2) by appropriate discretization. Since the analysis of the properties of data and labels is easy, the range of applications is wide.

  2) will be explained more concretely. Considering the case where the distribution of weight is represented by a histogram, the probability that the weight is 50 kg or more and less than 60 kg is 30%, 60 kg or more but less than 70 kg is 45%, and the other is 25%. There is an advantage that the distribution of is estimated in a form that can be interpreted by humans (for example, FIG. 1).

  Also, considering the distribution of labels taking the engine speed of an automobile as an example, if the engine speed is 0 or more and less than 1000, the probability of failure 1 is 70%, the probability of failure 2 is 10%, and other failures are 20%. The relationship of failure to data is easy for humans to understand (for example, FIG. 2).

  Conventionally, various methods for learning such a discrete distribution have been proposed. However, there is a problem in that the amount of calculation increases if an attempt is made to optimize the discrete distribution by giving dependency to each dimension. It was. In order to avoid this problem, there is a method of handling each dimension independently. In this case, however, there is a problem that it becomes impossible to grasp the dependency relationship of each dimension.

  In order to solve these problems, a technique called a latent class model proposed in Non-Patent Document 1 can be used. This model is characterized in that the distribution is represented by a locally independent mixed distribution.

  The normal distribution will be described as an example. As shown in FIG. 10, each dimension is independent in each cluster (three perfect circles in FIG. 10, local independence), and the dependency of each dimension is expressed by a plurality of clusters. To do. Although FIG. 10 has been described by taking a continuous distribution as an example, the same applies to a discrete distribution.

  Many latent class models are defined for data distribution, but it is possible to consider similar models assuming local independence for label distribution and simultaneous distribution of data and labels.

Goodman, L.A., Exploratory latent structure analysis using both identifiable and unidentifiable models, Biometrika, Aug 1974, Vol.61, No.2, pp.215-231

  However, in order to estimate the latent class model for a discrete distribution, it is necessary to specify how many regions are required to discretize each dimension into which region, where each dimension is to be discretized, and how many mixtures are required. However, these could not be optimized.

  Therefore, the present invention has been made in view of the above problems, and in estimating the distribution, the number of discretized regions in each dimension and the discretized position in each dimension are optimized by a locally independent mixed discrete distribution. With the goal.

  In order to solve the above problems, a distribution estimation apparatus according to the present invention includes an initialization unit that initializes data, and an expectation for calculating an expected value for a posterior probability of cluster assignment in the data initialized by the initialization unit. A value calculating means, and a minimizing means for calculating a parameter for minimizing the expected value of the information criterion for the expected value of the cluster assignment calculated by the expected value calculating means, and calculating a locally independent mixed discrete distribution; Determining the optimality of the information criterion calculated by the minimizing means and, if determined not optimal, the optimality determining means for performing the expected value calculation again by the expected value calculating means, and each dimension The number of discretized regions and the discretized position in each dimension are optimized.

  A clustering apparatus according to the present invention includes the distribution estimation apparatus described above.

  Further, the estimation method of the distribution estimation apparatus according to the present invention includes an initialization step for initializing data, and an expected value for calculating an expected value for the posterior probability of cluster assignment in the data initialized in the initialization step. A calculation step, a minimization step for calculating a parameter that minimizes an information criterion expected value for the expected value of the cluster assignment calculated in the expected value calculation step, and a minimization step for calculating a locally independent mixed discrete distribution; and a minimum And determining the optimality of the information criterion calculated in the conversion step, and if it is determined that the determination is not optimal, the optimality determination step in which the expected value calculation in the expected value calculation step is performed again. It is characterized by optimizing the number of discretization regions and the discretization position of each dimension.

  The program according to the present invention includes an initialization process for initializing data, an expected value calculation process for calculating an expected value for the posterior probability of cluster assignment in the data initialized by the initialization process, and an expectation The parameter that minimizes the expected value of the information criterion with respect to the expected value of the cluster assignment calculated by the value calculation process is calculated by the minimization process that calculates the locally independent mixed discrete distribution and the minimization process. And determining the optimality of the information amount criterion, and if it is determined that it is not optimal, an optimality determination process for performing the expected value calculation again by the expected value calculation process. It is characterized by causing a computer to optimize the number and the discretized position of each dimension.

  According to the present invention, when estimating a distribution, it is possible to optimize the number of discretized regions in each dimension and the discretized position in each dimension together with the distribution estimation by using a locally independent mixed discrete distribution model.

It is explanatory drawing (the 1) of discrete distribution. It is explanatory drawing (the 2) of discrete distribution. It is a block diagram of the distribution estimation apparatus which concerns on embodiment of this invention. It is a flowchart figure of the distribution estimation apparatus which concerns on embodiment of this invention. It is a block diagram of the distribution estimation apparatus which concerns on other embodiment of this invention. It is a flowchart figure of the distribution estimation apparatus which concerns on other embodiment of this invention. It is a block diagram of the clustering apparatus which concerns on embodiment of this invention. It is a block diagram of the outlier detection apparatus which concerns on embodiment of this invention. It is a block diagram of the data classification device which concerns on embodiment of this invention. It is explanatory drawing of a prior art.

  Next, the best mode for carrying out the invention will be described in detail with reference to the drawings.

[First Embodiment]
FIG. 4 is a configuration diagram of the data distribution estimation apparatus in the embodiment of the present invention. The distribution estimation apparatus 100 includes a data input unit 110, an initialization processing unit 120, a cluster assignment expected value calculation processing unit 130, an expected description length minimization processing unit 140, an optimality determination processing unit 150, and a distribution estimation. And a result output unit 160, which calculates a distribution in which the number of discretizations and the discretization breakpoints for each dimension are optimized, and outputs a distribution estimation result 190 including the calculated distributions.

The i-th data entered in the following _{x i = (x i1, x} i2, ..., x iD) and, and y _i labels corresponding to x _i, if the label is simultaneously input. The number of labels (label type) is expressed as C. The distribution to be estimated is the data distribution P (X) (Equation 1), the label distribution P (Y | X) (Equation 2), and the data and label simultaneous distribution P (X, Y) ( It is expressed as Equation 3).

  X = (X1, X2,..., XD) represents D-dimensional data, Y represents a label, and Z = (Z1, Z2,..., Zk) represents a random variable for the cluster to which X belongs. Zk is a random variable that takes 1 when X belongs to the k-th cluster and takes 0 otherwise, and is called a cluster assignment. Z cannot be observed directly, so it is generally called a hidden variable.

  K represents the number of mixtures (number of clusters). P (Xd | Zk = 1) represents a histogram for the dth dimension for the kth cluster, and P (Y | X_d, Zk = 1) is the value Xd of the dth dimension data for the kth cluster. Represents the probability of label Y with the condition.

  The data input unit 110 is a functional unit for inputting the input data and the label 180. At this time, it is possible to simultaneously input parameters necessary for distribution estimation. In the present embodiment, the number of mixtures in the mixture distribution is treated as a parameter input together with data.

  The initialization processing unit 120 is a device for initializing cluster assignments and the like. As an initialization method, ΣP (Zk = 1) = 1, ΣP (Zk =) in (Equation 1) to (Equation 3), such as a method in which all the clusters are made to have a uniform ratio or a random setting method using random numbers. It is possible to initialize with any value satisfying 1 | Xd) = 1.

  The cluster assignment expected value calculation processing unit 130 calculates an expected value for the posterior probability of cluster assignment for the data X. That is, the value of Zk for P (Z | X, Y) is calculated when estimating the distribution of data, and P (Z | X) when estimating the distribution of labels and the simultaneous distribution of data and labels.

  As an example of the calculation method, when calculating the distribution of data (Equation 4), when calculating the probability of the label (Equation 5), when calculating the simultaneous distribution of the data and the label (Equation 5) 6).

  Here, zi represents a cluster assignment for xi.

  The expected description length minimization processing unit 140 calculates a parameter that minimizes the expected value of the description length for the expected value of the cluster assignment calculated by the cluster assignment expected value calculation processing unit 130.

  The description length is an optimization standard for calculating a distribution from data or labels, and is expressed by the sum of the size when the data is compressed by the calculated distribution and the size when the distribution itself is compressed. .

  To explain intuitively, in the first embodiment of the present invention, the compression rate of input data can be increased by increasing the number of discretizations or arbitrarily changing the position of discretization, Accordingly, since the distribution becomes complicated, the compression ratio of the distribution decreases, and it is possible to select a distribution that optimizes the two trade-offs.

  Hereinafter, examples of the expected description length calculation method and the minimization method will be specifically described. In the following, Mkd is the number of discretized regions with respect to dimension d for cluster k.

[When discretization width is constant in each distribution]
When the width of the discretization is constant in each distribution, in the discrete distribution for the dimension d of the cluster k, each discretized region is defined as a region that equally divides the data value range into Mkd. In this case, the expected description length is calculated when the distribution of data is estimated (Equation 7), when the probability of the label is calculated (Equation 8), and when the simultaneous distribution of the data and the label is calculated (Equation 8). It is possible to calculate by equation (9).

However, the base of the logarithm is 2, and e represents the base of the natural logarithm. In addition, log ^* represents a number obtained by adding log ^* a = log c + log a + log log a + log log log a + ... to a natural number a until a negative number appears. However, log ^* 0 = 0 is defined. C is a constant not depending on a.

Also, E _{Z | X} [log (Σzik)], E _{Z | xi, Y} [log (Σzik)], and E _{Z | X, Y} [log (Σzik)] are the posterior of Z of the sum of cluster assignments. Represents the expected value for the distribution. The expected value can be calculated by an arbitrary method. For example, the calculation can be performed by a method of approximation by the Monte Carlo method or by Taylor expansion of the log function around the mean value of Σzik.

  Next, the procedure for minimizing the expected description length will be described taking the case of estimating the data distribution as an example. First, regarding the distribution of cluster k with respect to dimension d, the number Mkd of discretized regions is fixed. Next, the parameters of the distribution are estimated. Examples of the estimation method include a maximum likelihood estimation method.

  Next, regarding Mkd and the estimated parameter, the expected value of the description length of the data and the expected value of the description length of the parameter for the corresponding distribution, and when estimating the distribution of the data (Equation 10), the probability of the label is When calculating (Equation 11), when calculating the simultaneous distribution of data and label, it is calculated by (Equation 12).

  When the distribution of cluster k with respect to dimension d is changed by changing Mkd to estimate the distribution of data (Equation 10), when calculating the probability of label (Equation 11), the simultaneous distribution of data and label is calculated. In this case, Mkd and its parameters that minimize (Equation 12) are adopted. This process is performed on the distribution of each dimension in each cluster, and the optimum distribution for each dimension is calculated.

Of (Equation 7) to (Equation 9), terms that do not correspond to (Equation 10) to (Equation 12) do not depend on the number and parameters of the discretization regions of each distribution. The distribution for each dimension is calculated.

  Finally, the ratio P (Zk) of the mixture of each cluster is calculated when calculating the distribution of data (Equation 13), when calculating the probability of the label (Equation 14), When calculating, it calculates by (Equation 15).

  The optimality determination processing unit 150 compares the description length DL (not an expected value) for the calculated distribution with the description length DL_old calculated in the previous loop, and determines whether DL and DL_old are sufficiently close to each other. Determine if the calculated distribution is optimal. Specifically, it is conceivable that DL_old-DL is 0 or a value sufficiently close to 0 as a criterion for determination.

  The DL is calculated by (Equation 16) when estimating the distribution of data (Equation 16), when calculating the probability of the label (Equation 17), and when calculating the simultaneous distribution of data and labels (Equation 18). To do.

  The distribution estimation result output unit 160 outputs the calculated distribution estimation result 190 to an external medium such as a monitor or a hard disk.

  With reference to the flowchart shown in FIG. 5, the operation in the distribution estimation apparatus 100 according to the present embodiment will be described.

  Data 180 is input to the data input unit 110 (step S100). Next, the initialization processing unit 120 performs an initialization process for the distribution to be calculated (step S101).

  Next, an expected value for the posterior probability of cluster assignment for each data is calculated in the cluster assignment expected value calculation processing unit 130 (step S102).

  Next, the expected description length minimization processing unit 140 calculates a distribution that minimizes the expected description length (step S103).

  Next, the optimality determination processing unit 150 determines whether or not the currently calculated distribution is optimal (step S104).

  If it is determined that it is not optimal, the processing from step S102 to step S104 is repeated. On the other hand, if it is determined to be optimal, the distribution estimation result output unit 160 outputs the result (step S105).

  According to the present embodiment, the distribution of data, the predicted distribution of labels attached to the data, or the simultaneous distribution of data and labels is represented as a mixed distribution of locally independent discrete distributions, the number of discretized regions in each dimension, It is possible to optimize the discretized position together with the distribution estimation.

  In this embodiment, the distribution estimation device based on the description length (probabilistic complexity) has been described. However, as the information amount criterion, a model such as an Akaike information amount criterion, a generalized information amount criterion, or the like is used. Similar distribution estimation can be performed on any criterion for making a selection and can be easily analogized.

[Second Embodiment]
FIG. 6 is a configuration diagram of a data distribution estimation apparatus according to the second embodiment of the present invention. The distribution estimation device 200 includes a mixture number setting unit 210, a mixture number loop end determination processing unit 220, and an optimum distribution selection processing unit 230, as compared with the distribution estimation device 100 according to the first embodiment. Is different. In addition, about the function part same as 1st Embodiment, the same code | symbol is described and detailed description is abbreviate | omitted.

  The mixture number setting unit 210 selects a candidate value that has not been subjected to the process of minimizing the expected description length from the candidate number of the mixture number set in advance or input together with the input data 180, and is initialized. The data is output to the processing unit 120.

  The initialization processing unit 120 operates in the same manner as in the case of the distribution estimation apparatus 100, but the number of mixtures is input from the number of mixtures setting unit 210.

  When the optimality determination processing unit 150 determines that the distribution is optimal, the description length regarding the calculated distribution is stored together with the number of mixtures and the estimated parameter.

  The mixture number loop determination processing unit 220 determines whether or not a distribution that minimizes the expected description length has been calculated for all of the mixture number candidate values set in advance or input together with the data 180.

  The optimum distribution selection processing unit 230 compares the description lengths related to the distributions optimized for each number of mixtures, and selects the distribution with the smallest description length as the optimum distribution.

  With reference to FIG. 7, the operation in the distribution estimation apparatus 200 according to the present embodiment will be described.

  Data 180 is input to the data input unit (step S200). Next, the mixture number setting unit 210 sets one candidate value for the mixture number for which the calculation of the expected description length has not been completed (step S201).

  Next, the initialization processing unit 120 performs an initialization process for the distribution to be calculated (step S202).

  Next, the cluster assignment expected value calculation processing unit 130 calculates the expected value of the cluster assignment for each data (step S203).

  Next, the expected description length minimization processing unit 140 calculates a distribution that minimizes the expected description length (step S204).

  Next, the optimality determination processing unit 150 determines whether or not the currently calculated distribution is optimal (step S205).

  If it is determined that it is not optimal, the processing from step S203 to step S205 is repeated. On the other hand, if it is determined to be optimal, the mixture number loop end determination processing unit 220 determines whether the calculation of the description length has been completed for the candidate values for the total number of mixtures (step S206).

  If there is a candidate value for which calculation has not ended, the processing from step S201 to step S206 is repeated. When the calculation of the description length is completed for all candidate values, the optimal distribution selection processing unit 230 selects the number of mixtures that minimizes the description length from the candidate values (step S207). Next, the distribution estimation result output device 160 outputs the result (step S208).

  If this embodiment is used, the distribution estimation unit 100 can also calculate the optimum distribution with respect to the number of mixtures given from the outside as a parameter by optimizing the number of mixtures.

  In addition, the distribution estimation apparatuses 100 and 200 according to the embodiment of the present invention can be used for clustering, abnormality detection, data classification, and the like.

  For example, in the case of use for clustering, the most posterior probability for the posterior distribution of the cluster assignment calculated by the cluster assignment expected value calculation processing unit 130 for the optimal distribution calculated as shown in FIG. Clustering data into clusters with high

  For example, when used for abnormality detection, the occurrence probability (or probability density) of new input data is calculated for the optimal distribution calculated as shown in FIG. What is necessary is just to detect.

  For example, when used for data classification, the label probability P (Y | X) of new input data is calculated for the optimal distribution calculated as shown in FIG. Should be classified.

  Several industrial applications that can be achieved by using the present invention will be described.

[Automobile failure data analysis example]
When a sensor is provided with various sensors such as engine speed, vehicle speed, and engine oil temperature obtained from the vehicle's Electric Control Unit (ECU), and a failure label, the distribution estimation device proposed in the present invention learns the distribution. By doing so, it becomes possible to analyze the relationship between the failure and the sensor value, for example, failure 1 is likely to occur when the rotation speed is 3000 rotations or more and the vehicle speed is 30 km or less.

  By inputting normally running vehicle data into the distribution estimation device and configuring an outlier detection device, it is possible to analyze the characteristics of normal data, and new input data deviates from normal. It can be applied to a vehicle abnormality detection system, for example, by determining whether or not the vehicle is in an abnormal state.

  By estimating the label distribution from the failure data, it is possible to configure a failure diagnosis device that quickly identifies the cause of a failure when a failure has occurred for which a new cause cannot be identified.

  According to the embodiment of the present invention, it is possible to automatically optimize the number and position of discretizations and the number of mixtures, so that it is difficult for an expert to make a model or the number of sensors. Even when it is difficult to specify a discretization method for all variables, an appropriate distribution can be estimated.

  Although the embodiments have been described above, various modifications and changes can be made to these embodiments and specific examples without departing from the broad scope and scope of the present invention defined in the claims.

DESCRIPTION OF SYMBOLS 100 Distribution estimation apparatus 110 Data input part 120 Initialization processing part 130 Cluster assignment expected value calculation processing part 140 Expected description length minimization processing part 150 Optimality judgment processing part 160 Distribution estimation result output part 210 Mixing number setting part 220 Mixing number Loop end determination processing unit 230 Optimal distribution selection processing unit

Claims (7)

Initialization means for initializing data;
Expected value calculation means for calculating an expected value for the posterior probability of cluster assignment in the data initialized by the initialization means;
Minimizing means for calculating a parameter that minimizes the expected value of the information amount criterion for the expected value of the cluster assignment calculated by the expected value calculating means, and calculating a locally independent mixed discrete distribution;
Determining the optimality of the information amount criterion calculated by the minimizing means, and if it is determined not to be optimal, the optimality determining means for performing the expected value calculation again by the expected value calculating means, and
A distribution estimation apparatus characterized by optimizing the number of discretized areas in each dimension and the discretized position in each dimension. A mixture number setting means for selecting a candidate value that has not been minimized from the mixture number candidate values and inputting data to the initialization means;
For all candidate values of the number of mixtures in the data determined to be optimal by the optimality determining means, it is determined whether a parameter that minimizes the expected value of the information amount criterion is calculated, and all candidate values are minimized. If not, a mixture number determining means for transmitting data to the mixture number setting means;
An optimum parameter selection means for comparing information amount criteria related to parameters optimized for each number of mixtures and selecting a parameter having the smallest information amount criterion as an optimum parameter;
The distribution estimation apparatus according to claim 1, wherein the number of discretized regions in each dimension, the discretized position in each dimension, and the number of mixtures are optimized.   3. The distribution estimation apparatus according to claim 1, wherein any one of a description length, an Akaike information amount criterion, and a generalized information amount criterion is used as the information amount criterion.   4. The distribution estimation apparatus according to claim 1, wherein calculation is performed for any one of data distribution, label distribution on condition of data, and simultaneous distribution of data and label.   A clustering apparatus comprising the distribution estimation apparatus according to claim 1. An initialization step for initializing the data;
An expected value calculation step of calculating an expected value for the posterior probability of cluster assignment in the data initialized in the initialization step;
A minimizing step of calculating a parameter for minimizing the expected value of the information criterion with respect to the expected value of the cluster assignment calculated in the expected value calculating step, and calculating a locally independent mixed discrete distribution;
Determining the optimality of the information criterion calculated in the minimization step, and if not determined optimal, the optimality determination step for performing the expected value calculation again by the expected value calculation step, and
An estimation method for a distribution estimation apparatus, wherein the number of discretized areas in each dimension and the discretized position in each dimension are optimized. An initialization process to initialize the data,
An expected value calculation process for calculating an expected value for a posterior probability of cluster assignment in the data initialized by the initialization process;
A parameter that minimizes the expected value of the information criterion for the expected value of the cluster assignment calculated in the expected value calculation process, and a minimizing process that calculates a locally independent mixed discrete distribution;
Determining the optimality of the information amount criterion calculated in the minimization process, and if it is determined not to be optimal, the optimality determination process for causing the expected value calculation by the expected value calculation process to be performed again.
A program that causes a computer to execute optimization of the number of discretized regions in each dimension and the discretized position in each dimension along with distribution estimation.

Images