JPH052600A - System for learning probability distribution and probabilistic rule - Google Patents

Abstract

PURPOSE:To attain the learning of reliable and explainable probability distribution or probabilistic rule by outputting a specific expression as the best expression. CONSTITUTION:An expression capable of minimizing the sum of a product between the number of bits of the expression and the a-th power (0<a<1) of the number of input data and the minus logarithm likelihood of input data determined by probability distribution is outputted as the best expression. Namely a sample consisting of the restricted number of events and a parameter (a) in NIC definition are received as inputs and the expression of probability distribution is outputted based upon the class H (hypothetic space) of the expression of a certain determined probability distribution. In details, a hypothesis minimizing the value of an expression I in the hypothetic space H for input sample S = (x1 to xn) is outputted as the best expression. In the expression I, 1(h) is the description length of a hypothesis (h) and (m) is the number of input samples (number of events). When many events are applied in an environment including noise, the probability distribution or probabilistic rule capable of highly reliably explaining the events can be learned.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a learning method for a stochastic rule or probability distribution which inputs a number of cases and outputs a probabilistic rule or a probability distribution expression.

[0002]

2. Description of the Related Art An unknown stochastic information source (hereinafter referred to as a true model) is input with data generated from the information source, and a model that optimally approximates the information source is selected from a given hypothesis space. As the selection criterion of the hypothesis in the statistical estimation problem to be output, the description length minimum principle of Rissanen (or Schwarz's Bayesian information criterion), Akaike's information criterion, Takeuchi's information criterion, Hanan ( Hanan) and Quinn's criteria are known. Among them, the one that proves the agreement, that is, the convergence to an asymptotic true model is Risannen's minimum description length principle, And Hanan and Quinn criteria,
Furthermore, for some distance functions, the one that has been proved to have a rapid convergence is Rissanen.
Is the minimum description length principle. This criterion is based on the article "Modeling by Showtest Data Description" written by Risanen in the 14th issue of the Automatica magazine published in 1978.
(Modeling by shortestdata
description). The criterion is the hypothesis that minimizes the sum of the half of the product of the number of real-valued parameters of the hypothesis and the logarithm of the number of data and the negative logarithmic likelihood of the data determined by the hypothesis. Also,
The rapid convergence of the Hellinger distance, which is based on the principle of minimum length, is explained in the 1990 issue of the US magazine "Procedures of the Third Annual Workshop on Computational Learning Theory".
(Proceedings of the Third
Annual Workshop on Comput
"Learning Criterion Force Stochastic Rules" (ALearn), written by Yamanishi, published by National Learning Theory.
ing Criterion for Stochas
tic Rules).

[0003]

As described above, the principle of the minimum description length is as follows regarding the Hellinger distance.
The rapid convergence has been proved, but it has not been proved yet regarding the KL information content which is the most widely used distance function in statistics. Minimizing the amount of KL information is also minimizing the average code length in code theory, and is more explanatory as an evaluation criterion of statistical validity.
Furthermore, the KL information amount is a stricter evaluation criterion than many distance functions including the Hellinger distance, that is, the KL information amount causes the Hellinger amount.
It is known that many distance functions including distance can be suppressed from the top, but the reverse is not always true. So KL
There has been a demand for a rapid proof of convergence regarding the amount of information, or the discovery of an information standard that enables such proof. The present invention accomplishes exactly this.

[0004]

According to a first aspect of the present invention, there is provided a probability distribution learning method for a large number of input data, wherein a probability distribution expression for explaining the distribution of the data is given to a plurality of given probabilities. In a learning method of a probability distribution that is selected and output from a class including a distribution expression form, the probability distribution is determined by the product of the number of bits of the expression form and the a-th power (0 <a <1) of the number of input data. This is a probability distribution learning method that outputs the expression that minimizes the sum of the input data and the minus logarithmic likelihood as the best expression.

The probabilistic rule learning method of the second invention is such that, with respect to inputs of a large number of data and teacher signals, a probabilistic rule for classifying the data is given to a plurality of given probabilistic rule expressions. In a learning method of a probabilistic rule that selects and outputs from a class including, the input data determined by the probability distribution and the product of the number of bits of the expression and the number of input data to the power a (0 <a <1), and This is a probabilistic rule learning method that outputs the expression that minimizes the sum of the teacher signal and the minus logarithmic likelihood as the best expression.

The probability distribution learning method according to the third aspect of the present invention represents, with respect to a large number of input data and input accuracy parameters, an expression form of the probability distribution that explains the distribution of the data, and a plurality of given probability distribution expressions. In a probability distribution learning method for selecting and outputting from a class including a shape, the expression is a subclass of the expression type class, and the partial class is the expression for an arbitrary expression type in the expression type class. The probability that any representation in the subclass will give to any element on the region, including a representation approximating the shape within the precision parameter, is one-half the reciprocal of the precision parameter and the bits of the individual data. It is constrained from below by a polynomial power on the upper bound of the number, and the number of elements in the subclass is a reciprocal of the precision parameter of 2 and a polynomial power on the upper bound of the number of bits of the individual data. From the class having the feature of being suppressed from, the product of the number of bits of the expression and the number of input data to the power a (0 <a <1), and the negative logarithmic likelihood of the input data determined by the probability distribution. This is a probability distribution learning method that outputs the expression that minimizes the sum of the as the best expression.

The probabilistic rule learning method of the fourth aspect of the present invention uses a probabilistic rule for classifying a large number of data and teacher signals, and input precision parameters for the input data.
In a learning method of a probabilistic rule that selects and outputs from a class including a plurality of given probabilistic rule expressions, it is a partial class of the expression type class, and the partial class is
For any expression in the class of the expression, including an expression that approximates the expression well within the accuracy parameter, any expression in the subclass includes any element on the region and any teacher. The conditional probabilities given to the signal pairs are also constrained from below by a polynomial power of one half the reciprocal of the precision parameter and the upper bound on the number of bits of the individual data, and the number of elements in the subclass is Two
Of the class having the characteristic that the reciprocal of the precision parameter and the upper limit of the number of bits of individual data are suppressed from the above, and the number of bits of the expression and the number of input data to the power a (0 <a < It is a learning method of a stochastic rule that outputs the expression that minimizes the sum of the product of 1) and the minus log-likelihood of the input data and the teacher signal determined by the stochastic rule as the best expression.

The learning method of the probability distribution of the fifth invention is given a representation of the probability distribution which explains the distribution of the data, for a large number of input data and input accuracy parameters.
Dynamically calculate the upper limit of the description length of the target discrete model in the search process in the learning method of the probability distribution that selects and outputs from the class that includes multiple parametric representations of the probability distribution consisting of the discrete model and real-valued parameters However, the description length of the discrete model is a partial class whose expression length is less than or equal to the upper limit, and the partial class has a description length of the discrete model that is less than or equal to the upper limit. The probability that any expression in the subclass given to any element on the region includes an expression that approximates the expression within the precision parameter for any expression.
The reciprocal of the precision parameter is reduced by a factor of 1, the upper limit of the length of the expression description and the polynomial power of the upper limit of the number of bits of individual data, and the number of elements in the partial class is 2. From the class having the characteristic that the reciprocal of the precision parameter, the upper limit of the expression description length, and the polynomial power related to the upper limit of the number of bits of individual data are held from above, a. It is a learning method of a probability distribution that outputs the expression that minimizes the sum of the product of powers (0 <a <1) and the negative logarithmic likelihood of the input data determined by the probability distribution as the best expression. .

A learning method of a probabilistic rule according to a sixth aspect of the present invention uses a probabilistic rule for classifying a large number of data and teacher signals and input accuracy parameters into the classified data.
In the learning method of a probabilistic rule that selects and outputs from a given class that includes multiple parametric expressions of a probabilistic rule consisting of a discrete model and real-valued parameters, the description length of the target discrete model in the search process The upper limit is dynamically calculated, and is a partial class of the description model whose description length is less than or equal to the upper limit, and the partial class describes the description length of the discrete model in the expression class. For any expression whose value is less than or equal to the upper limit, including an expression that closely approximates the expression within the accuracy parameter, where any expression in the subclass is any element on the region and any teacher signal. The conditional probability given to the pair of? Is also reduced by a polynomial power with respect to the reciprocal of the precision parameter of 1/2, the representation length upper limit, and the upper limit of the number of bits of individual data. The feature is characterized in that the number of elements in the partial class is suppressed from above by a polynomial power related to the reciprocal of the precision parameter of 2, the upper limit of the expression description length, and the upper limit of the number of bits of individual data. Of the number of bits of the expression and the a-th power (0 <a <1) of the number of input data and the negative logarithmic likelihood of the input data and the teacher signal determined by the stochastic rule This is a learning method of probabilistic rules that outputs the expression that minimizes the sum as the best expression.

[0010]

The term "probability distribution on the region X" is generally called the probability density function in the probability statistical theory, but here, for simplicity, it is assumed that the region X is discrete, We use the term probability distribution. Further, in the region X, Σ ⁿ or U _{i = 1, ..., n} Σ ⁱ _{, where} Σ is a certain alphabet.
Suppose that Further, since n is considered to be a variable here, Σ ⁿ or U _{i = 1, ..., n} Σ ⁱ is written as X _n, and the family X = {X _n : nεN} of the region is considered.

The probabilistic rule is a function that gives a conditional probability that a class y of x takes a certain realization value when a certain attribute value x is given. Here, x is an element of the region X that satisfies the same assumption as above, and y is an element of the finite range Y. Given any xεX, class yεY
The conditional probability that is assigned is written as p (y / x).

The probability distribution expression or the stochastic rule expression is only assumed to be an expression that can be described in a binary alphabet by a fixed coding method. For example, a parametric expression that is often used consists of two parts, a discrete model and a real-valued parameter, but as long as it is output by a computer, the real-valued parameter is also described by a binary alphabet with a certain precision. Therefore, no particular distinction is made here. Here, the description length of the expression h of the probability distribution or the stochastic rule is the number of bits when described by the above coding method.

The first and second invention methods will be described in more detail using the above definitions. The learning method of the probability distribution of the first invention is a sample consisting of a finite number of cases as an input,
And a parameter a in the NIC definition, and outputs a probability distribution phenotype from a certain defined probability distribution phenotype class H (hereinafter, referred to as a hypothesis space). More specifically, the learning method is based on the input sample S =
For <x _{1, ...,} X _m >, the hypothesis that minimizes the following amount in the hypothesis space H is the best and is output.

[Equation 1] Here, it is assumed that a is given as an input, but when it is not given, there is no essential difference, for example, a = 3/4. Although m is the number of input samples (the number of cases), generally, the upper limit m of the number of samples used for learning is set, and when the number of samples exceeds m, the last m.
It is also possible to learn using individual data. The above method is particularly effective when the information source is changing with time and the learning method has a limited memory. The first term nic (S / h) above is the “minus logarithmic likelihood of the sample determined by h”, specifically, the sum of the minus logarithmic likelihood of individual data determined by h, and the second The term nic (h) is the product of the description length of h and the a-th power of the number of samples. In addition, what was described as nic above is'new information writer.
Abbreviation for'ion '(new information volume).

The probabilistic rule learning method of the second invention receives a sample consisting of a finite number of cases, that is, X × Y elements, and a parameter a in the NIC definition as an input, and determines a certain rule. The probabilistic rule expression is output from the given probabilistic rule expression class H (hereinafter referred to as a hypothesis space). More specifically, for the input sample S = << x ₁ , y ₁ >, _..., <x _m , y _m >>, the hypothesis that minimizes the following quantity in the hypothesis space H is the best. ,Output.

[Equation 2] Here, l (h) is the description length of the hypothesis h, and m is the number of input samples (the number of cases) as above. Further, although it is assumed that a is given as an input, if it is not given, for example, a = 3/4 may be set, and there is no essential difference.

Next, some definitions of the probability distribution and the representational form of the stochastic rule necessary for explaining the third and fourth inventions will be introduced. (Hereinafter, the probability distribution is regarded as a special case of stochastic rules, and only "stochastic rules" are defined.) Definition 1 H and H (ε) are classes that include a finite number of representations of stochastic rules. If there exists a polynomial q and h′εH (ε) such that for any nεN, the following two equations are satisfied for any hεH, then
H (ε) is said to be an ε-lower bound approximation of H.

[Equation 3]

[Equation 4] Definition 2 For any ε> 0, it is the ε-lower bound approximation of H, and the density is at most 2 to the power of poly (1 / ε, n) (where n is the upper limit of the character string length on the region, and poly is When there is an algorithm that constructs a subclass H (ε) of H, which is a polynomial) in polynomial time for the same parameter 1 / ε, n, H is said to be closed for ε-lower bound approximation.

Using the above definitions, the third and fourth invention methods will be described in more detail. The learning method of the probability distribution of the third invention receives a sample consisting of a finite number of cases, a parameter a in the NIC definition, and an accuracy parameter ε> 0 as inputs, and sets a certain ε-lower bound approximation. The probability distribution expression is output from the probability distribution expression class H (hereinafter, referred to as a hypothesis space) that is closed for. It is assumed here that the accuracy parameter ε is given, but if the accuracy parameter is not given, it is determined as a decreasing function of the number of samples (m), for example, ε = 1 / m. Better, there is no essential difference. Although a is given as an input, if a is not given, a = 3/4 may be set, for example. More specifically, the learning method is based on the input sample S = (x ₁ , ...
.., x _m ), and ε> 0, ε-lower bound approximation H of H that can be efficiently constructed in a partial class of the hypothesis space H
From (ε), the hypothesis that minimizes
Output.

[Equation 5] Here, m is the number of input samples (the number of cases).

In the learning method of the stochastic rule of the fourth invention, a sample consisting of a finite number of cases, that is, X × Y elements, a parameter a in the NIC definition, and an accuracy parameter ε are used as inputs. Receiving> 0, the probabilistic rule expression is output from a probabilistic rule expression class H (hereinafter referred to as a hypothesis space) H that is closed for a given ε-lower bound approximation. Similar to the above, when the accuracy parameter is not given, it is defined as a decreasing function of the number of samples (m), for example, ε = 1 / m, and there is no essential difference. Although a is given as an input, if a is not given, for example, a = 3/4
You should say Specifically, the input sample S = <<< x
_{1, y 1>, ···,} <x m, y m >>, and with respect to epsilon> 0, with partial class hypothesis space H, efficiently configurable for H .epsilon. lower bound approximation H (epsilon ), The hypothesis that minimizes the following quantity is the best and is output.

[Equation 6] Here, l (h) is the description length of the hypothesis h, and m is the number of input samples (the number of cases) as above.

Next, some definitions of parametric expressions of probability distributions and stochastic rules necessary for explaining the fifth and sixth inventions are introduced. (In the following,
Consider the probability distribution as a special case of probabilistic rules, and define only “probabilistic rules”. ) A parametric expression of a stochastic rule is an expression consisting of a pair of a discrete model and a real-valued parameter. here,
A discrete model is generally described in a binary alphabet with a finite description length, and is also described in a binary alphabet with a finite precision with real-valued parameters. For example, in the case of a probability automaton, the discrete model is a description of the number of states, and the real-valued parameter is the total transition probability. In the following, in general, when H is a class of a parametric expression form of a probabilistic rule, the partial class of H given by the upper limit s of the description length of the discrete model is written as H _s . Definition 3 Let H and H (ε) be a class of parametric representations of probabilistic rules, then there exists a polynomial q, and for any s, nεN, for any hεH _s If h′εH (ε) _s that satisfies the following two equations exists, then H (ε) is said to be an ε-lower bound approximation of H.

[Equation 7]

[Equation 8] Definition 4 For any n, sεN and ε> 0, it is an ε-lower bound approximation of H _s , with a concentration of at most 2 poly (1 /
epsilon, n, s) power (where, n represents the upper limit of the length of the string on the region, s is the upper limit of the description length of the discrete model, poly is some polynomial), H _s of partial class H (epsilon) _s H is said to be closed for the ε-lower bound approximation when there is an algorithm that constructs in the polynomial time for the same parameter 1 / ε, n, s.

The learning method of the probability distribution of the fifth invention receives a sample consisting of a finite number of cases, a parameter a in the NIC definition, and an accuracy parameter ε> 0 as an input, and determines a certain ε. An output form of a probability distribution is output from a class H (hereinafter, referred to as a hypothesis space) including a plurality of parametric expression forms of a probability distribution consisting of a discrete model closed for the lower bound approximation and real-valued parameters. It is assumed here that the accuracy parameter ε is given, but when the accuracy parameter is not given, a decreasing function of the number of samples (m), for example, ε = 1
/ M, etc., and there is no essential difference. Although a is given as an input, if a is not given, a = 3/4 may be set, for example. The learning method uses input samples S = <x ₁ , ..., X _m >, and ε
For> 0, the upper limit s of the description length of the target discrete model is dynamically calculated in the search process, and ε-lower bound approximation H (ε )
From among the following, the hypothesis that minimizes the following quantity is the best and is output.

[Equation 9] Here, l (h) is the description length of the hypothesis h, and m is the number of input samples (the number of cases) as above. Further, although the upper limit s of the description length that needs to be considered in the hypothesis space is dynamically calculated above, this can be realized in detail by the following procedure, for example. An arbitrary value (for example, logm) is used as the initial value of the upper limit s of the description length of the discrete model, and the hypothesis that the description length is the upper limit s or less and minimizes the above amount is the best hypothesis h to date. ^* And
Product of the current upper limit s and m ^a is, h ^* new information amount nic (S, h ^*) for the input samples s of it exceeds the so it can be seen description length of the discrete model of NIC minimum hypothesis is less than s , H ^* is output as the best hypothesis, and if it does not exceed,
Repeat the above process with s: = 2s.

The probabilistic rule learning method according to the sixth aspect of the present invention uses as input a sample consisting of a finite number of cases, that is, X × Y elements, a parameter a in the NIC definition, and an accuracy parameter ε> 0. Received, and a certain defined ε−
The probabilistic rule expression is output from a class H (hereinafter, referred to as a hypothesis space) that includes a plurality of parametric expression forms of a probabilistic rule consisting of a discrete model closed for the lower bound approximation and real-valued parameters. Similar to the above, when the accuracy parameter is not given, it is defined as a decreasing function of the number of samples (m), for example, ε = 1 / m, and there is no essential difference. Although a is given as an input, if a is not given, for example, a = 3/4
You should say The learning method is based on the input sample S = << x
₁ , y ₁ >, ..., <x _m , y _m >>, and ε> 0, the upper limit s of the description length of the target discrete model in the search process is dynamically calculated, and the hypothesis space is calculated. From the ε-lower bound approximation H (ε) of H that can be efficiently constructed in the subclass of H, the hypothesis that minimizes the following quantity is set as the best output.

[Equation 10] Here, l (h) is the description length of the hypothesis h, and m is the number of input samples (the number of cases) as above. Also, the upper limit s of the description length of the discrete model that needs to be considered in the hypothesis space above
Was calculated dynamically, but this can be calculated as described in the fifth invention above.

[0021]

Embodiments of the present invention will now be described in detail with reference to the drawings. Here, the probability distribution learning method of the fifth aspect of the invention will be described with reference to an embodiment in which a probability automaton is used as a hypothesis space. The probabilistic automaton here is a probabilistic automaton as a stochastic generator,
It is defined and explained in the above paper by Abe and Warmt. In the following, the class of stochastic automata whose number of states is at most s is written as H _S. For any constant ε> 0,
Assuming the existence of the procedure ε-BA that efficiently constructs the ε-lower bound approximation of the subclass given by the upper bound s of the description length of H, we obtain the H _s of H _s obtained by ε-BA (H, ε, s). The ε-lower bound approximation is written as H (ε) _s . Also, for simplicity, ε-BA
Let the ith automaton obtained by (H, ε, s) be ε-
Write BA (H, ε, s, i). It is also assumed that the procedure ε-BAC (H, ε, s) for calculating the concentration of H (ε) _s exists.

In the learning method of the probability distribution of the fifth invention, H _s
An example in which is a hypothetical space will be described below with reference to FIGS. 1 to 5. In addition, regarding the other five embodiments of the invention,
As will be described later, it can be obtained by partially modifying the fifth embodiment of the invention described below. First, the overall structure of the learning method of the fifth invention will be described with reference to FIG. The accuracy parameter ε as an input, the parameter a in the NIC definition, and the sample S = <x ₁ , ...,
x _m >. Here, it is assumed that a is given as an input, but when it is not given, there is no essential difference, for example, a = 3/4. First, in the input and initialization means (step 0), the upper limit of the description length is set to s: = lo
after gm and initialization, .epsilon. In lower bound approximation calculation means (step 1) constitutes the .epsilon. lower bound approximation H (epsilon) _s of H _s, to which the sequence H (epsilon). H _s has at most s states
In the present context, which is the probability automaton of, H (ε)
_s is the power of each parameter {1-ε / (2n)} and is at least ε / n.
It may be an arbitrary array of all probability automata of. (This is explained in the above-mentioned paper by Abe and Warmt.) There are several possibilities to realize ε-BA and ε-BAC in this case, but since they are obvious, we will explain them in detail here. Does not touch.

Next, in the NIC calculating means (step 2), for each hypothesis h in H (ε) _s , the NIC information amount of S of h, that is,

[Equation 11] To calculate. Next, in the NIC minimum hypothesis finding means (step 3), the above-mentioned amount nic of H (ε) _s
One hypothesis h that minimizes (S, h) is selected and designated as h ^* . In addition, the selection method in case of the same point is generally arbitrary. Finally, in the end judging means (step 4), after updating the value of s to s: = 2s, nic (S,
h ^*) <to determine whether it is a sm ^a, if so, to output the h ^* as the best hypothesis, otherwise, return to step 1.

Next, an implementation example of each of the above means will be described in detail with reference to FIGS. First, the ε-lower bound approximation calculating means (step 1) can be realized, for example, as shown in the flowchart of FIG. First, at 101, ε-BAC (H _s , ε)
Calculate the concentration of H (ε) _s by using
Also, i: = 1 is initialized. Next, at 102, it is determined whether the index i is equal to k, and if so,
The present means is terminated and the process proceeds to step 2. Otherwise 1
In 03, the i-th element of the array H (ε) _s is ε-BA
Let the i-th hypothesis calculated using (H _s , ε, i). Finally, add 1 to the index i and proceed to 102.

Next, the NIC calculation means (step 2)
For example, it can be realized as in the flow chart shown in FIG. First,
At 201, i: = 1 is initialized. Next, at 202, it is determined whether the index i is equal to k, and if so, the present means is terminated and the process proceeds to step 3. Otherwise, at 203, for the i-th hypothesis h of H (ε) _s , NI
The first term of C, that is,

[Equation 12] Is calculated and this is set as NIC1 (i). Then 204
And NIC for the i-th hypothesis h of H (ε) _s
The second term of

[Equation 13] Is calculated and this is set as NIC2 (i). Then 205
Then, the sum of NIC1 (i) and NIC2 (i) obtained above is set as NIC (i). Finally at 206, the index i is incremented by 1 and the process proceeds to 202.

Next, the NIC minimum hypothesis finding means (step 3) can be realized as shown in the flow chart of FIG. 4, for example. First, in 301, i: = 1, MINVAL: = 0,
Initialize as MININD: = 1. Next, in 302, it is determined whether the index i is equal to k, and if so, the present means is terminated, and in step 4, H (ε) (MINI
ND) is output. Else, MINV at 303
AL = 0 or MINVAL> NIC
(I), if not, go to 305, and if so, at 304, a new NIC minimum index was found, so MINVAL: = NIC
(I) and MININD: = i are updated. Then 3
At 05, 1 is added to the index i and the process proceeds to 302.

Finally, the end judging means (step 4)
For example, it can be realized as in the flow chart shown in FIG. 401
Then, s: = 2s is updated, and in 402, MINVAL <s
It is determined whether or not it is m ^a , and if so, the process proceeds to the output (step 5), and if not so, the process returns to step 1.

The sixth embodiment of the invention is obtained, for example, by replacing the stochastic automaton, which is the hypothetical space in the above embodiment, with a "stochastic acceptor" which is a stochastic acceptor. “Probability acceptor” as a probabilistic rule is, for example, the US magazine “Proceedings of the Force Annual Workshop on Computational Learning Theory” (Proceedings of the F).
ourth Annual Workshop C
operational Learning The
ory), Abe, Takeuchi, and Valmut's paper "Polynomial Lanablity of Probabilistic Concept with Respect to KL Divergence" (Polynomial Learna)
bilility of Probabilistic C
onscepts with respect to t
he Kullback-Leibler Diver
gen)). still,
The sixth embodiment of the invention is different from the above-mentioned embodiments in that the input sample S is 〈〈 x ₁ , y ₁ 〉, ..., 〈x _m , y
It is a sequence of pairs of elements and teacher signals in a region such as _m 〉〉. Therefore, i of H (ε) _s calculated in 203
For the th hypothesis h, the first term of NIC is given by the following equation.

[Equation 14]

The embodiment of the third invention is obtained by replacing the hypothesis space of the embodiment of the fifth invention with a class of probability automata whose number of states is at most s. In this case, the calculation of the ε-lower bound approximation can be similarly performed. The end determination means (step 4) is unnecessary.

The embodiment of the fourth invention is obtained by replacing the hypothesis space probability automaton of the embodiment of the third invention with a stochastic acceptor stochastic acceptor.

The embodiment of the first invention is obtained by replacing the hypothesis space of the embodiment of the fifth invention with a subclass which is an ε-lower bound approximation of a class of stochastic automata whose number of states is at most s. . The ε-lower bound approximation calculation means (step 1) and the end determination means (step 4) are unnecessary.

The embodiment of the second invention is obtained by replacing the stochastic automaton, which is the hypothesis space of the embodiment of the first invention, with a stochastic acceptor, which is a stochastic acceptor.

[0033]

According to the probability distribution learning method of the third aspect of the present invention, for a large number of data, an expression form of the probability distribution that explains the data distribution is expressed as a certain probability distribution given as a hypothesis space. In the learning method of the stochastic distribution that selects and outputs from the class of the form, when the input data is independently generated by a certain probability distribution on the region, as a hypothesis regarding an arbitrary hypothesis space closed for the ε-lower bound approximation The KL information amount between the output probability distribution and the true probability distribution is the KL information amount between the optimal distribution and the true distribution in the hypothesis space,
It is possible to ensure that the data converges rapidly as the number of data increases.

According to the probabilistic rule learning method of the fourth invention, a probabilistic rule for classifying data into a large number of data and teacher signals is given as a hypothetical space. In the learning method of the stochastic rule selected from the class of and output, when the input data and the teacher signal are independently generated by the probability distribution over the area and the simultaneous distribution of the true stochastic rules over the area, ε -For any hypothesis space closed about the lower bound approximation, the KL information amount of the probabilistic rule output as a hypothesis and the true stochastic rule is KL of the optimal rule and the true rule in the hypothesis space.
It is possible to guarantee that the amount of information converges rapidly as the number of data increases.

According to the learning method of the probability distribution of the fifth invention, the expression form of the probability distribution for explaining the distribution of the data is calculated for a large number of data from the discrete model and the real-valued parameter given as the hypothesis space. In a learning method of a probabilistic distribution that selects and outputs from a class of parametric representations of the probability distribution, if the input data is independently generated by a certain probability distribution on the region, an arbitrary closed ε-lower bound approximation For the parametric hypothesis space of, the KL information amount of the probability distribution output as the hypothesis and the true probability distribution is increased to the KL information amount of the optimal distribution and the true distribution in the hypothesis space. It is possible to guarantee that it will converge rapidly with.

According to the probabilistic rule learning method of the sixth invention, a probabilistic rule for classifying data into a large number of data and teacher signals is input to a discrete model and a real model given as a hypothesis space. In the learning method of the probabilistic rule that selects and outputs from the parametric expression form class of the probabilistic rule consisting of numerical parameters, the input data and the teacher signal are the probability distribution over the area and the true stochastic rule over the area. The KL information between the probabilistic rule output as a hypothesis and the true probabilistic rule for any parametric hypothetical space closed about the ε-lower bound approximation when independently generated by the joint distribution is It is possible to guarantee that the KL information amount between the optimum rule and the true rule in space converges rapidly as the number of data increases.

Regarding the performance guarantees regarding the learning methods of the third, fourth, fifth, and sixth inventions, the paper "Minimum Description Length Principle and Uniform Convergence Method" by Abe, published in the Proceedings of the 1991 National Conference on Artificial Intelligence. , ”But here is a brief description. Theorem 1 ε-When a class H of parametric expression forms of closed probability distributions or stochastic rules about the lower bound approximation is given, Hε is set to p (1
/ Ε, n, s), and the lower limit of probability is (1/2) q
Let ε be the ε-lower bound approximation of H held by the (1 / ε, n, s) th power (p and q are certain polynomials). At this time, the third, fourth, fifth, and sixth learning methods using Hε as a hypothetical space have an order of the number of data in the input sample with respect to an arbitrary true model.

[Equation 15] When the upper limit given by is exceeded, the probability is at least 1-δ
Then, a hypothesis that the KL information amount regarding the true model is at most ε larger than the optimal hypothesis in H is output. The KL information amount used as the distance between the probability distributions above is defined below. When D is a true probability distribution and h is a probability distribution as a hypothesis, the KL information amount d (D, h) of D of H (or Kullback-Leibler diverg
ence) is given by the following equation.

[Equation 16] The KL information amount as the distance between the probability rules is defined in almost the same manner. D is the probability distribution over the region, p is the true probability rule, h
Is a probability rule as a hypothesis, KL of h for p
Information amount d (p, h) (or Kullback-Le
The ibler divergence) is given by the following equation.

[Equation 17]

For example, the class of probability automata as a parametric expression of the probability distribution is closed in the ε-lower bound approximation, as shown in the above paper by Abe and Walmut. Also, the class of stochastic automata (stochastic acceptors) as parametric expressions of stochastic rules is closed for the ε-lower bound approximation, as shown in the above papers by Abe, Takeuchi, and Walmut. Therefore, we obtain the following two systems. System 1 In the probability distribution learning method of the fifth invention, when a probability automaton is used as the hypothesis space, the number of cases is poly (1 / ε, n, s) (where,
n is the upper limit of the character string length on the area, and s is the description length of the discrete model of the optimum model), the probability is at least 1-δ
Then, a probability automaton whose KL information amount regarding the true model is at most ε larger than the optimum automaton is output. System 2 In the learning method of the stochastic rule of the sixth invention, when the probability acceptor is a hypothesis space, the number of cases is poly (1 / ε, n, s) (where n is on the domain) for a certain polynomial poly. The upper limit of the character string length of s, and s is the description length of the discrete model of the optimal model), the probability is at least 1-
At δ, we output a stochastic acceptor whose KL information about the true model is at most ε greater than the optimal automaton.

[Brief description of drawings]

FIG. 1 is a configuration diagram of a learning method of a probability distribution and a stochastic rule according to the present invention.

FIG. 2 is a flowchart showing ε-lower bound approximation calculation means in step 1.

FIG. 3 is a flowchart showing a NIC calculating means in step 2.

FIG. 4 is a flowchart showing the NIC minimum hypothesis finding means of step 3;

FIG. 5 is a flowchart showing an end determination unit of step 4.

Claims (6)

[Claims] 1. Learning of a probability distribution for a large number of input data, which selects and outputs a probability distribution expression for explaining the distribution of the data from a class including a plurality of given probability distribution expressions. In the method, an expression form that minimizes the sum of the product of the number of bits of the expression form and the a-th power (0 <a <1) of the number of input data and the negative logarithmic likelihood of the input data determined by the probability distribution. , A probability distribution learning method characterized by outputting as the best expression. 2. Probabilistic rule for selecting and outputting a probabilistic rule for classifying a large number of data and teacher signals from a class including a plurality of given probabilistic rule expressions. In the rule learning method, a product of the number of bits of the expression and the number of input data to the power a (0 <a <1),
A learning method of a stochastic rule, wherein an expression form that minimizes the sum of the input data and the minus log likelihood of the teacher signal determined by the probability distribution is output as the best expression form. 3. With respect to a large number of input data and input accuracy parameters, a probability distribution expression that describes the distribution of the data is selected and output from a class that includes a plurality of given probability distribution expressions. In the learning method of the probability distribution, the partial class of the expression type class, and the partial class is
For any representation in the class of representations, including the representation that approximates the representation within the accuracy parameter, the probability that any representation in the subclass will give to any element on the region The reciprocal of the precision parameter and the reciprocal of the precision parameter of 2 and the number of elements in the subclass are suppressed from below by a polynomial power with respect to the reciprocal of the precision parameter of 1/2 and the upper limit of the number of bits of individual data. From the class having the characteristic of being constrained from above by a polynomial power related to the upper limit of the number of bits of the data, the product of the number of bits of the expression and the number a of the input data (0 <a <1), and the probability A learning method of a probability distribution, wherein an expression form that minimizes the sum of the input data and the minus logarithmic likelihood determined by the distribution is output as the best expression form. 4. A probabilistic rule for classifying a large number of data and teacher signals and input accuracy parameters is selected from a class including a plurality of given probabilistic rule expressions. In the learning method of the probabilistic rule to be output, the partial class is a subclass of the class of the expression, and the partial class sets the expression to the accuracy parameter for the arbitrary expression in the class of the expression. The conditional probability that any expression in the subclass gives to any pair of elements and any teacher signal in the region is It is constrained from below by a polynomial power on the upper limit of the number of bits of data, and the number of elements in the subclass is the reciprocal of the precision parameter of 2 and the upper limit of the number of bits of individual data. From the class having the characteristic of being pushed down from above by a polynomial power with respect to, the input data determined by the stochastic rule and the product of the number of bits of the expression and the power of a (0 <a <1) And a learning method of a probabilistic rule, which outputs the expression that minimizes the sum of the teacher signal and the minus logarithmic likelihood as the best expression. 5. A parametric representation of a probability distribution consisting of a discrete model and real-valued parameters, given a representation of a probability distribution that explains the distribution of the data for a large number of input data and input accuracy parameters. In the learning method of the probability distribution that selects and outputs from the class including multiple,
Dynamically calculating the upper limit of the description length of the target discrete model in the search process, which is a partial class of the representation model whose description length is less than or equal to the upper limit,
The partial class includes an expression form approximating the expression form within the accuracy parameter for any expression form whose description length in the class of expression forms is less than or equal to the upper limit,
The probability that any representation in the subclass will give to any element on the region is the reciprocal of the precision parameter, halved,
The expression description length upper limit, and is suppressed from below by a polynomial power with respect to the upper limit of the number of bits of individual data,
Moreover, from among the classes having the feature that the number of elements in the partial class is suppressed from above by the reciprocal of the precision parameter of 2, the representation length description upper limit, and a polynomial power related to the upper limit of the number of bits of individual data. , The expression that minimizes the sum of the product of the number of bits of the expression and the a-th power (0 <a <1) of the number of input data and the negative logarithmic likelihood of the input data determined by the probability distribution is best. A learning method of probability distribution, which is characterized by outputting as the expression form of. 6. Parametric of a probabilistic rule consisting of a discrete model and real-valued parameters, given a probabilistic rule for classifying a large number of data and teacher signals and input accuracy parameters In a learning method of probabilistic rules that selects and outputs from a class that includes multiple different expression types, the upper limit of the description length of the target discrete model in the search process is dynamically calculated, and A model is a partial class whose description length is less than or equal to the upper limit, and the partial class is the representation type for any representation type in which the description length of the discrete model is less than or equal to the upper limit. Of the conditional probabilities given to the pair of any element on the region and any teacher signal by any expression in the subclass is 2 The reciprocal of the precision parameter divided by 1,
The expression description length upper limit, and is suppressed from below by a polynomial power with respect to the upper limit of the number of bits of individual data,
Moreover, in the class having the characteristic that the number of elements in the partial class is suppressed from above by the reciprocal of the precision parameter of 2, the upper limit of the expression description length, and the polynomial power with respect to the upper limit of the number of bits of individual data. To minimize the sum of the product of the number of bits of the expression and the a-th power (0 <a <1) of the number of input data and the negative logarithmic likelihood of the input data and the teacher signal determined by the stochastic rule. A probabilistic rule learning method characterized by outputting the expression as the best expression.