JP3004254B2 - Statistical sequence model generation device, statistical language model generation device, and speech recognition device - Google Patents

Abstract

An apparatus is disclosed for generating a statistical class sequence model called class bi-multigram model from input strings of discrete-valued units, where bigram dependencies are assumed between adjacent variable length sequences of maximum length N units, and where class labels are assigned to the sequences. There are counted the number of times all sequences of units occur and the number of times all pairs of sequences of units co-occur in the input training strings of units, and an initial bigram probability distribution of all the pairs of sequences is computed as the counted number of times the two sequences co-occur divided by the counted number of times the first sequence occurs in the input training string. Then the input sequences are classified into a pre-specified desired number of classes. Further, an estimate of the bigram probability distribution of the sequences is calculated by using an EM algorithm to maximize the likelihood of the input training string computed with the input probability distributions, and the above processes are iteratively performed to generate a statistical class sequence model. <IMAGE>

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a statistical sequence model generation device for generating a statistical sequence model based on learning sequence data, and a statistical language model for generating a statistical language model based on learning text data. The present invention relates to a generation device and a speech recognition device that recognizes a speech signal of an input uttered speech sentence using the statistical language model.

[0002]

2. Description of the Related Art In recent years, a method of using a language model has been studied to improve the performance of a continuous speech recognition apparatus. This aims at improving the recognition rate and reducing the calculation time by predicting the next word and reducing the search space using a language model that is a sequence model. Here, the sequence is, specifically, a word in a sequence of characters and a phrase (or phrase) in a sequence of words. N-gram (N-gram; here, N-gram)
Is a natural number of 2 or more. ). It learns large-scale text data and statistically gives the transition probability from the previous N-1 words to the next word. Generation probability P of a plurality of L word strings w ₁ ^L = w ₁ , w ₂ ,..., W _L
(W ₁ ^L ) is expressed by the following equation.

[0003]

(Equation 1)

[0004] Here, w _t represents a t-th one word of the word string w ₁ ^L, w _i ^j represents the j-th word string from the i-th. In the above _equation 1, the probability P (w _t |
wt _{+ 1-} ^Nt-1 ) is a word sequence wt _{+ 1-} ^Nt-1 composed of N words.
Is the probability that the word w _t will be uttered after is uttered, and similarly, the probability P (A | B) means the probability that the word A will be uttered after the word or word string B has been uttered. Also,
“Π” in Equation 1 represents the probability P (w _t from t = 1 to L
| W _{t + 1−N} ^t−1 ).

In recent years, techniques for improving the performance of continuous speech recognition using the above-described statistical language model N-gram have been actively proposed. It uses a method that takes advantage of dependencies. These models have in common the traditional N
Used to mitigate the fixed-length dependency assumption found in the -gram model, covering a variety of broader assumptions.

[0006] Phrases are expressed in a purely statistical manner (ie,
Stochastic Context Free Gramma
rs), it is necessary to use various criteria in order to derive it in a manner that does not use grammatical rules, for example, the following criteria have been proposed. (A) Prior art document 1 "K. Ries et al.," Class phra
se models for languagemodeling ”, Proceedings of I
Leave-one-out likelihood disclosed in CSLP 96, 1996, and (b) Prior art document 2 “H. Masataki et al., Variable
-order n-gram generation by word-class splitting a
nd consecutive word grouping.Proceedings ofICASSP
96, 1996 ".

[0007]

In these methods, by using the likelihood criterion in a statistical framework, the E
M (Expectation Maximum; ie, maximization of expected value)
Although an optimization method using an algorithm can be used, it tends to be over-learned. Further, in the optimization processing, for example, the related art document 3 “S. Matsunaga et a
l., ”Variable-length language modeling integrating
global constraints ”, Proceedings of EUROSPEECH 9
7,1997], the convergence and optimization of statistical language models are not theoretically guaranteed.

Here, for example, prior art document 1
The following describes problems when the likelihood criterion proposed in is used. <Problem 1> Since the frequency probability of a sequence of words can be obtained by a greedy algorithm,
Monotonic convergence towards optimal conditions is not guaranteed. <Problem 2> This method is definite. That is,
If the sequence [bcd] is a list of sequences (invent
ory), even if "bcd" occurs in the input character string,
This is [bc] + [d], [b] + [cd], [b] +
It is not divided into sub-sequences such as [c] + [d]. In other words, there is no degree of freedom in analyzing the sequence. <Problem 3> The definition of the class of the sequence is based on the class classification of the preceding word. That is, first the words are classified, and then each sequence of labels of the class of words is used to define the class of the sequence. Therefore, sequences with different lengths cannot be included in the same class. For example, “thank you for” and “tha
nk you very much for ”does not belong to the same class.

In order to solve this problem, the present inventor has proposed a technique disclosed in prior art document 4 “S. Deligne et al.,” Introducing statis.
tical dependencies and structural constraints in v
ariable-length sequence models ”, In Grammatical In
ference: Learning Syntaxfrom Sentences, Lecture No
tes in Artificial Intelligence 1147, pp.156-167, Sp
ringer, 1996 ", for a statistical language model using a multigram that is a variable-length sequence, only the possibility of calculating those parameters using Equation (16) of the related art document 4 is shown. , The (1)
Equation (6) is not in a format that can be actually calculated using a digital computer, and has a problem that it cannot be put to practical use. Here, a multigram is a variable-length sequence that does not specify dependence on other sequences.

[0010] An object of the present invention is to solve the above-mentioned problems, to assure monotonous convergence toward an optimum state as compared with the conventional example, to provide a degree of freedom in the analysis result, and to obtain a variable-length sequence. A statistical sequence model generation device, a statistical language model generation device, and a speech recognition device capable of handling statistical information in the same class and generating a statistical model by practically performing high-speed processing using a digital computer. Is to do.

[0011]

According to the present invention, there is provided a statistical sequence model generating apparatus based on input data including a sequence which is a unit sequence of one or more units.
A multigram that is a unit sequence of variable-length natural numbers N ₁ ,
A is bi bigram between multigram a natural number N ₂ pieces of unit columns of variable length - a statistical sequence model generating device for generating a statistical sequence model of multi-gram, based on the input data, Initializing means for counting the frequency probabilities of the bigrams of all combinations of unit strings under the constraint of predetermined maximum values of N ₁ and N _{2, and} the bigram counted by the initializing means Based on the frequency probabilities of each class, the classes are merged so that the loss of mutual information when the pairs of classes are merged is minimized, and the frequency probabilities of each class are updated to classify them into a predetermined number of classes. By calculating the unit sequence included in the classified class, the frequency probability of the conditional unit sequence of the classified class, and the frequency probability of the bigram between the classified classes Classifying means, and a unit sequence included in the classified class output from the classification processing means, a frequency probability of a conditional unit sequence of the classified class, and a bigram between the classified classes. Based on the frequency probabilities, re-estimation is performed using the EM algorithm to obtain a maximum likelihood estimation value. Here, using the forward / backward algorithm, each unit sequence to be processed is time-series , The forward likelihood of the unit sequence of the processing target that can be taken forward, the frequency probability of the unit sequence when the unit sequence immediately before the unit sequence is a condition, and the backward possibility of the unit sequence that can be taken backward in chronological order. By re-estimating the frequency probability of the bigram between the sequences using an expression indicating the frequency probability of the bigram between the sequences based on the likelihood, Re-estimating means for generating and outputting a statistical sequence model of the chigram; and control means for controlling the processing of the classifying means and the processing of the re-estimating means to be repeatedly executed until a predetermined end condition is satisfied. It is characterized by the following.

In the above-mentioned statistical sequence model generating apparatus, the initialization means may further remove, from among the counted frequency counts of the bigram, data of a combination of bigrams having a predetermined frequency probability or less. I do.

Further, in the statistical sequence model generation device, the classification means classifies the plurality of classes using a Brownian algorithm based on the frequency probability of the bigram counted by the initialization means. It is characterized by.

In the above-mentioned statistical sequence model generating apparatus, the above equation may be a bigram between a sequence of unit sequences when a second unit sequence as the unit sequence follows the first unit sequence in the input data. Is a formula for calculating the frequency probability of each of the unit strings to be processed in the input data. The frequency probability of the bigram between the sequences is obtained by dividing all of the segmentation including the first and second unit strings. Is obtained by dividing the sum of likelihoods by the sum of likelihoods in all the segmentations including the first unit sequence. In this case, the above formula is obtained by calculating a denominator indicating an average number of times each unit sequence occurs in the input data, and an average for each unit sequence when the second unit sequence follows the first unit sequence in the input data. A numerator indicating the number of times, the numerator, for each unit sequence of the processing target, the forward likelihood,
The sum of the product of the frequency probability of the unit sequence and the backward likelihood when the unit sequence immediately before the unit sequence is a condition, and the denominator is the forward likelihood for each unit sequence to be processed. ,
This is the sum of the product of the frequency probabilities of all the unit columns and the above-mentioned backward likelihood when the unit column immediately before the unit column is used as a condition.

Further, in the above-mentioned statistical sequence model generating apparatus, the termination condition is that the number of repetitions of the processing of the classifying means and the processing of the re-estimating means has reached a predetermined number. Features.

In the statistical language model generating apparatus according to the present invention, in the statistical sequence model generating apparatus, the unit is a character of a natural language, the sequence is a word, and the classifying means is a character string. Is classified into a plurality of word strings, and the statistical sequence model is a statistical language model.

Further, in the statistical language model generating apparatus according to the present invention, in the statistical sequence model generating apparatus, the unit is a word of a natural language, the sequence is a phrase, and the classification means includes a word string. Into a series of phrases, and the statistical sequence model
It is characterized by being a statistical language model.

Still further, the speech recognition apparatus according to the present invention is a speech recognition apparatus provided with speech recognition means for recognizing a speech using a predetermined statistical language model based on an input speech signal of an uttered speech sentence. , The voice recognition means,
The speech recognition is performed by referring to the statistical language model generated by the statistical language model generation device.

[0019]

Embodiments of the present invention will be described below with reference to the drawings. In the following embodiments, the unit is a character, an example of classifying a character string that is a sequence of characters into a word string, and the unit is a word,
An example of classifying a word sequence, which is a sequence of words, into phrases (phrases) has been described. However, the present invention is not limited to this, and the unit is DNA, and the sequence of DNA, D
You may comprise so that an NA row | line may be classified into a predetermined DNA sequence. Further, the unit may be a base, and a base sequence as a base sequence may be classified into predetermined codons.

FIG. 1 is a block diagram of a continuous speech recognition apparatus according to an embodiment of the present invention. The continuous speech recognition apparatus according to the present embodiment uses the working RAM 30 to generate a variable-length bi-multigram language model based on text data that is a character string stored in the learning text data memory 21. Language model generation unit 20
Here, as shown in FIG. 3, the processing of the statistical language model generation unit 20 can be roughly classified into a classification processing using the Brown algorithm (step S3),
And a re-estimation process using a multigram (step S4).

That is, the statistical language model generating apparatus according to the present embodiment uses a variable-length natural number N based on input data including a character string sequence composed of one or more characters.
A is bi bigram between _one string and a natural number N ₂ pieces of string length - a statistical language model generating device for generating a statistical language model of a multi-gram, wherein
As shown in FIG. 3, (a) based on the input data,
Under the constraint of the predetermined maximum value of N ₁ and N ₂ ,
Initialization processing (step S2) for counting the frequency probabilities of the bigrams of all combinations of character strings, and (b) merging pairs of each class based on the frequency probabilities of the bigrams counted by the initialization processing Character strings included in the classified classes by merging so as to minimize the loss of mutual information when doing so and updating the frequency probability of each class and classifying it into a predetermined number of multiple classes A classification process (step S3) for calculating and outputting the frequency probability of a conditional character string of the classified class and the frequency probability of a bigram between the classified classes (step S3); EM based on the character strings included in the classified class, the frequency probability of the conditional character string of the classified class, and the frequency probability of the bigram between the classified classes. The algorithm is re-estimated so as to obtain the maximum likelihood estimation value. Here, using the forward / backward algorithm, for each character string to be processed, Based on the forward likelihood for the character string, the frequency probability of the character string under the condition of the character string immediately before the character string, and the backward likelihood for the character string that can be backward in time series, By re-estimating the frequency probability of the bigram between the sequences using the equation (equation 22-equation 24) indicating the frequency probability of the bigram, the statistical sequence model of the bi-multigram as the re-estimation result is obtained. A re-estimation process (step S4) to generate and output, and (d) a process of controlling to repeatedly execute the classification process and the re-estimation process until predetermined end conditions are satisfied. Step S5)
It is characterized by including.

This embodiment focuses on a phrase-based method as opposed to a word N-gram-based method. Here, the plurality of sentences are formed into phrases, and the frequency probabilities are assigned to the phrases in place of the words. Regardless of whether the models are based on N-grams or phrases, they correspond to either deterministic or statistical models. In a phrase-based framework, uncertainty is introduced into a phrase through ambiguity in the parsing of the sentence. That is, this means that, in fact, even though the phrase "abc" is registered as a phrase, there is no probability that the analysis result of the character string is, for example, [ab] [c]. In contrast, in the deterministic approach, the simultaneous occurrence of all a, b, and c is systematically interpreted as the occurrence of the phrase [abc].

In this embodiment, the processing of the statistical language model is executed by using a bi-multigram, and the language model of the bi-multigram is a statistical model based on phrases, and its parameter is Estimated according to likelihood criteria.

First, the theoretical formulation of the multigram will be described. In the multigram framework, a sentence consisting of T words

[Mathematical formula-see original document] W = w ₍₁₎ w ₍₂₎ ... W _(T) is assumed to be a sequence (sequence) of phrases having a maximum length of n words. here,
S indicates segmentation into T _s phrases, and s _(t) is the time index (indicating the serial number from the first word ₎ in the segmentation S. Can be represented by the following equation.

## EQU3 ## (W, S) = s ₍₁₎ ... S _(Ts)

Here, a dictionary composed of a plurality of segmented phrases is formed by combining words from 1, 2,... To n from the vocabulary, and is represented by the following equation. .

Ds = {s _j } _{j The} sentence likelihood is calculated as the following equation as the sum of likelihoods for each segmentation.

[0026]

(Equation 5)

According to the decision-oriented method of the model, the sentence W is
Analyzed according to the most likely segmentation, the following approximation is obtained.

[0028]

(Equation 6)

Here, assuming an n-gram correlation between phrases, the likelihood value of the result of the specific segmentation S is calculated as follows.

[0030]

(Equation 7)

Here, the symbol n represents the degree of dependence between a plurality of phrases, and is used as n in the conventional n-gram notation. The code n _max indicates the maximum length of the phrase. Therefore, an example of calculating the likelihood is shown in the following equation. In this example, the bi-multigram model (n _max = 3, n
= 2) indicates the likelihood of “abcd”. The symbol # represents an empty sequence.

[0032]

Equation 8 Likelihood = p ([a] | #) p ([b] | [a]) p ([c] | [b]) p ([d] | [c]) + p ([a] | #) P ([b] | [a]) p ([cd] | [b]) + p ([a] | #) p ([bc] | [a]) p ([d] | [bc] ) + P ([a] | #) p ([bcd] | [a]) + p ([ab] | #) p ([c] | [ab]) p ([d] | [c]) + p ( [Ab] | #) p ([cd] | [ab]) + p ([abc] | #) p ([d] | [abc])

As is apparent from the above equation 8, the likelihood represents the sum of the frequency probabilities of all combinations when segmenting the sequence “abcd”.

Next, the estimation of the language model parameters will be described. The multigram n-gram model is completely defined by a set of parameters Θ, and the parameter の in the following equation is obtained using a dictionary Ds:

９ = {p (s _in | s _i1 ... s _in-1 ) | s _i1 ... s _in {Ds} ng related to any combination of n phrases
ram. The estimate of the set of parameters Θ is obtained, for example, as the maximum possible likelihood value obtained from incomplete data, ie, the Maximum Likelihood Estimation, where the unknown data is based on This is the segmentation S to be made. Therefore, the iterative maximum likelihood estimate of the parameter Θ is calculated using the well-known EM algorithm (Expectation Maximization Alg
orithm). Where Q
Let (k, k + 1) be an auxiliary function of the following equation, which is calculated using the likelihood of the iteration number parameters k and k + 1.

[0035]

(Equation 10)

As shown in the known EM algorithm,

If Q (k, k + 1) ≧ Q (k, k), then

L ^{(k + 1)} (W) ≧ L ^(k) (W) Therefore, the following re-estimation formula for the number of iterations parameter (k + 1)

## EQU13 ## p ^{(k + 1)} (s _in | s _i1 ... S _in-1 ) is a constraint condition of the following equation.

[Equation 14] By maximizing the auxiliary function Q (k, k + 1) for the model parameter Θ ^{(k + 1)} under the following equation, the following equation can be directly derived. In this specification, the subscript notation and the subscript notation of the superscript are not possible, so the subscript notation of the lower layer is omitted.

[0037]

(Equation 15)

[0038] _{Here, c (s i1 ... s in} , S) indicates the number of occurrences of the combination of a plurality of phrases s _i1 ... s _in the segmentation S. The re-estimation equation of equation (15) is described in detail later for the bi-multigram (n = 2), by using a forward backward algorithm (forward backward algorithm).
algorithm) (hereinafter also referred to as the FB method). In a decision-oriented method, the re-estimation equation is simplified as:

[0039]

## EQU16 ## p ^{(k + 1)} (s _in ... S _in-1 ) = ｛c (s _i1 .
_in-1 s _in , S ^{* (k)} )｝ /｝ c (s _i1 ... s _in-1 ,
S ^{* (k)} )｝

Here, S ^{* (k)} is an analysis result of a sentence that maximizes L ^(k) (S | W), and is derived by the Viterbi algorithm. Each iteration has a likelihood L ^(k) (W)
Improve the language model in the sense of increasing and eventually converge to a critical point (perhaps a local maximum). The set of model parameters Θ is initialized using the learning corpus, ie, the relative frequency of all phrase combinations observed in the training text data.

Next, clustering of variable-length phrases (classification processing) will be described. According to the prior art document 1, a model based on class-phrases has recently attracted attention, but it usually assumes conventional word clustering. Typically, each word is first assigned the label C _k of the class to which the word belongs, and a variable-length phrase [C _k1 , C _k2 ... C _kn ] of the word-class label is derived.
By each variable length phrase, “<[C _k1 , C _k2 ... C _kn ]
The label of the class to which the phrase denoted as >> is defined. However, with this approach, only phrases of the same length can be assigned the same phrase-class label. For example, "thank you for"
And the phrase "thank you very much for" cannot be assigned to the same class label. In the present embodiment, as a solution to such a limitation, a method of directly clustering phrases instead of words is proposed. To this end, assuming a bigram correlation (n _max = 2) between the two phrases, and modifying the bi-multigram model learning method described above, each iteration has the following two Be composed of stages.

(I) Step SS1: Class assignment (corresponding to step S3 in FIG. 3)

[Equation 17] {p ^(k) (s _j | s _i )} → {p ^(k) (C _{k (sj)}
| C _{k (sj)} ), p ^(k) (s _j | C _{k (sj)} )｝ (II) Step SS2: Re-estimation of multigram (FIG. 3
Corresponds to step S4. )

_１８p ^(k) (C _{k (sj)} | C _{k (si)} ), p ^(k) (s _j
| C _{k (sj)} )｝ → {p ^{(k + 1)} (s _j | s _i )}

In step SS1, the frequency probability of the phrase bigram is input and the frequency probability of the class bigram is output. The class assignment is performed, for example, according to the conventional technique 5 “PF Brown et al.,” “Class-based n-gram mod.
els of natural language ”, Computational Linguistic
s, Vol. 18, No. 4, pp. 467-479, 1992 ", this is performed by maximizing correlation information between adjacent phrases. Here, the clustering candidates are not words but phrases. As described above, {p ⁽⁰⁾ (s _j | s _i )}
Is initialized using the relative frequency of simultaneous appearance of phrases in the learning text data. The above step SS
In step 2, the phrase frequency probability is re-estimated using the multigram re-estimation equation (Equation 15) or its approximate equation (Equation 16). Here, the only difference is that the likelihood of the analysis result is calculated by the following equation.

[0044]

[Equation 19]

This is, as described above, the frequency probability p ^(k)
(S _j | s _i ), the frequency probability p
^{(k) The} frequency probability p ^{(k + 1)} (s _j | s _i ) is _calculated based on (C _{k (sj)} | C _{k (si)} ) × p ^(k) (s _j | C _{k (sj)} ). Equivalent to re-estimation.

In summary, the above-mentioned step SS1 guarantees that the class assignment based on the mutual information criterion is optimized with respect to the current phrase distribution, and the above-mentioned step SS2 uses the frequency probabilities of the current class. According to Equation 19 above, it is ensured that the calculated likelihood is optimized by the frequency probability of the phrase. The training data is thus constructed iteratively at both a paradigmatic and a syntagmatic (each a linguistic term) level in a completely integrated manner. That is, the associative relationship between phrases represented by class assignments affects the re-estimation of phrase frequency probabilities, and the phrase frequency probabilities determine subsequent class assignments.

In the present embodiment, as described above, the forward and
The backward algorithm (FB method) is used. This will be described in detail below.

The above equation (15) uses the forward / backward algorithm, where n _max is the maximum length of the sequence and T is the number of words in the corpus (learning text data). n _max ²
T). Here, the complexity O (n _max ² T) corresponds to the order of the calculation cost.
That is, the calculation cost of Equation 15 is proportional to the square of the maximum length n _max of the sequence, and is proportional to the number of words in the corpus. In this embodiment, basically, the addition is performed over the time index (t) of the word, not the set of segmentation {S}, and the numerator and denominator of Expression 15 are calculated. Here, the calculation is based on the forward variable α in the following equation.
(T, l _i ) and the backward variable β (t, l _j ).

[0049]

Α (t, l _i ) = L (W ₍₁₎ ^(t-li) |
[W _{(t-li + 1)} ^(t) ])

Β (t, l _j ) = L (W _{(t + 1)} ^(T) | [W _{(t−lj + 1)} ^(t) ])

The forward variable α (t, l _i ) is the first t
Represents the likelihood of the words, where the last l _i words are
Limited to form one sequence. Also,
The backward variable β (t, l _j ) indicates the conditional likelihood of the last (Tt) words, and the last (Tt) words are the sequence [w _{(t-lj +1)} ... W _(t) ]. Here, for example, W ₍₁₎ ^(t-li) represents a word sequence composed of words from the time index (1) to (t-l _i). Then, assuming that the likelihood of the analysis result is calculated by Expression 7, Expression 15 is rewritten as the following expression.

[0051]

P ^{(k + 1)} (s _j | s _i ) = p _c / p _d where:

(Equation 23)

(Equation 24)

Here, l _i and l _j indicate the lengths of the sequences s _i and s _j , respectively. Kronecker function δ _k (t)
When the sequence of words that started at time index t is s _k whereas a 1, a 0 to become function otherwise. The variables α and β can be calculated by the following iterative formula (or induction formula). Here, start and end symbols are assumed at time indices t = 0 and t = T + 1, respectively.

For 1 ≦ t ≦ T + 1:

(Equation 25) here,

Α (0,1) = 1, α (0,2) =... = Α
(0, n _max ) = 0.

For 0 ≦ t ≦ T:

[Equation 27] here,

(28) β (T + 1,1) = 1, β (T + 1,2) =
.. = Β (T + 1, n _max ) = 0.

When the likelihood of the analysis result is calculated using the class assumption, that is, when it is calculated according to Equation 19, the term p ^(k) (s _j ^{) in} the re-estimation equation (Equation 22-Equation 24) | S
_i ) is the equivalent of that class, ie p ^(k) (C _{k (sj)} |
C _{k (si)} ) p ^(k) (s _j | C _{k (sj)} ). α
In the iterative expression, the term p ([W _{(t-li + 1)} ^(t) ] | [W
_{(t-li-l + 1)} ^(t-li) ]) is replaced by the corresponding class bigram probability multiplied by the conditional probability of the class of the sequence [W _{(t-li + 1)} ^(t) ]. . The same modification is performed for the variable β in the iterative equation.

Next, in the present embodiment, the forward
The re-estimation process using the backward algorithm will be described in detail below with reference to an example. The re-estimation process in the forward direction and the backward direction (hereinafter, referred to as the front-back direction) is represented by Equation 2
Rearrange the multiple terms in Equation 15 so that the addition of the numerator of 2 and the addition of the denominator are calculated for the time index t of the unit in the learning data, instead of the possible analysis result set {S}. Do. This method relies on the definition of the forward variable α and the backward variable β. (A) In the following paragraph << A1 >>, it is assumed that there is no class. (B) In the following paragraph << A1.1 >>, the variable α
And β are defined and examples are provided. (C) In the following paragraph << A1.2 >>, the variable α
The following describes an example of re-estimation in the front-rear direction with respect to the frequency probability using. (D) In the following paragraph << A1.3 >>, an example is given of a method of calculating the variables α and β by iteration (or induction). (E) In the following paragraph << A2 >>, the paragraphs << A1.2 >> and << A
1.3 shows a correction method. (F) All the examples below are based on the data shown in the following table.

[0057]

[Table 1] ――――――――――――――――――――――――――――――――― Input training data (below): onesixoneeightsixthre etwo 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Time index (unit above): ――――――――――――――――――― ―――――――――――――――― (Note) One character of the learning data corresponds to one time index.

<< A1.1 >> Definition of Forward Variable α and Backward Variable β Variable α (t, l) ends with a sequence of length l.
This is the likelihood of the data up to the time index (t). For example, the variable α (9,3) corresponds to the sequence “o ne sixo
_n_e ”. Also, the variable β (t, l) indicates that when the sequence of length l is known to end at the time index (t), the time index (t
+1) is the conditional likelihood of the data starting at +1). For example, for the variable β (9,3), the preceding sequence is “o_n_
e ", the sequence" eightsixthr
ee tw o ”. Variable α by iteration or induction
Examples of how to calculate and β are given in the following paragraphs <<
A1.3 >>.

<< A1.2 >> Re-estimation of Probability Based on Variables α and β As an example, the variables α and β
Is a formula for re-estimating the frequency probability p (o_n_e | s_i_x) using The general re-estimation formula (Equation 15) for the frequency probability p (o_n_e | s_i_x) has the following meaning. (A) The numerator is the sequence “o_n_
"e" is the average number of times following the sequence "s_i_x". (B) The denominator uses the sequence “s_i_
"x" is the average number of occurrences. (C) Here, the value of the average number is determined for all possible analysis results in the sequence of the learning data.

The numerator (Equation 23) and the denominator (Equation 24) of the re-estimation equation (Equation 22-24) using the forward / backward algorithm are equal to the numerator and the denominator of Equation 15, respectively. Rather than addition over
It is calculated by addition over a time index. In the numerator of the re-estimation equation (Equation 15), “s_i_x” and “o
Each time two sequences of “_n_e” occur consecutively, the likelihood of each possible analysis result is added. On the other hand, a re-estimation formula using the forward / backward algorithm (Equation 22)
In (Equation 24), two sequences of “s_i_x” and “o_n_e” continuously occur, and the sequence “o_n_e”
First, likelihood values of all analysis results such that “e” starts at the time index (t + 1) are grouped and added. The addition calculation is completed when the addition is performed up to the time index t.

In the above example, “s_i_x” and “o_n_e”
Sequences occur consecutively, and the sequence “o_n_e” starts only at the time index (7). Here, two sequences of “s_i_x” and “o_n_e” continuously occur, and the sum of likelihood values of all analysis results such that the sequence “o_n_e” starts at the time index (7) is , The sequence "one s_i_x o_n_e ei
ghtsixthre et wo ", which is
It is equal to:

(Equation 29)

Here, p (o_n_e | s_i_x) of the second term is
This is the frequency probability in the number of iterations parameter (k). According to the definition of the variable α in the forward direction, the variable α (6, 3) is the likelihood of the sequence “one s_i_x”. Further, according to the definition of the variable β in the backward direction, the variable β (9, 3) becomes When the sequence “o_n_e” is obtained, the sequence “ei
ghtsixthreetwo ”.

In the denominator of equation 15, the likelihood of each possible analysis result is added the same number of times as the sequence "s_i_x" occurs in this analysis. Equivalent, forward
In the forward-backward formulation using the backward algorithm, a sequence “s_i_x” is generated, the likelihood values of all analysis results ending at the time index (t) are first grouped, and then added, and the time index t The addition ends when the value exceeds.

In the above example, the sequence “s_i_x” is
It is generated to end at the time index (6) and the time index (17). The addition of the likelihood values of all the analysis results that occur so that the sequence “s_i_x” ends at the time index (6) is the sequence “one s_
i_x o_n_e eightsixthreetwo ”, which is equal to:

[0065]

[Equation 30]

Here, according to the definition of the forward variable α, the variable α (6, 3) is the likelihood of the sequence “ones_i_x”, and according to the definition of the backward variable β, the variable β (9, 3) )
Is the likelihood of the sequence “eightsixt hr eetwo” when the sequence “o_n_e” is given.

Next, at the time index (17), the addition of the likelihood values of all the analysis results at which the sequence “s_i_x” ends is determined by the sequence “onesixoneei”.
g ht s_i_x t_h_r_e_e two ”, which is equal to the following equation.

[0068]

(Equation 31)

Here, according to the definition of the variable α in the forward direction, the variable α (17, 3) is changed to the sequence “on esixoneei
ght s_i_x ”, and according to the definition of the variable β in the backward direction, the variable β (22, 5) corresponds to the sequence“ t_h_r_e_x ”.
The likelihood of the sequence "two" given "e".

Therefore, "onesixoneeight
In the learning data "sixthreetwo", the frequency probability p (o_n_e
| S_i_x) is given by the following equation using the forward / backward algorithm.

[0071]

(Equation 32) here,

[Equation 33]

(Equation 34)

As described above, the feature of the embodiment of the present invention is that Formula 22 including Formula 23 and Formula 24 is formulated using the forward-backward algorithm. Has the following meaning. The expression is used to calculate the frequency of the bigram between the sequences of unit columns when the second unit column, which is the unit column, follows the first unit column in the input data, and the processing target unit columns in the input data. Where the frequency probability of a bigram between the above sequences is
It is obtained by dividing the sum of the likelihoods in all the segmentations including the first and second unit strings by the sum of the likelihoods in all the segmentations including the first unit string. Further, the above expression shows a denominator indicating the average number of times each unit column occurs in the input data, and an average number of times for each unit column when the second unit column follows the first unit column in the input data. A numerator, the numerator is the forward likelihood for each unit sequence to be processed, the frequency probability of the unit sequence when a unit sequence immediately before the unit sequence is a condition, and the backward likelihood Where the denominator is the forward likelihood for each unit sequence to be processed, the frequency probabilities of all unit sequences on the condition of the unit sequence immediately before the unit sequence,
This is the sum of the products of the backward likelihood.

<< A1.3 >> Calculation Example of Forward Variable α and Backward Variable β As an example, the data “onesixoneeightsi
The variable α (9,3) and the variable β (9,3) for “xthreetwo” are calculated below. Here, the variable α (9,
3) is the likelihood of the sequence “onesix o_n_e”, which is the sequence up to the time index 9 and has a length 3 sequence at the end. Further, the variable β (9, 3) corresponds to the sequence “o_n_
sequence "eightsix" given "e"
This sequence is a conditional likelihood of "threetwo", and this sequence is a sequence after time index 9, and the preceding sequence "o_n_e" is known in advance.

The likelihood (forward variable) α (9, 3) up to the sequence “o_n_e” is calculated by the following equation. It is assumed that the maximum value of the length of a sequence is set to “5”.

Α (9,3) = added value below (a) For n_e_s_i_x: α (6,5) × p (o_n_e |
n_e_s_i_x) (b) Regarding e_s_i_x: α (6,4) × p (o_n_e | e_
s_i_x) (c) Regarding s_i_x: α (6,3) × p (o_n_e | s_i_
x) (d) For i_x: α (6,2) × p (o_n_e | i_x) (e) For x: α (6,1) × p (o_n_e | x)

The backward likelihood (backward variable) β (9, 3) under the condition of the sequence “o_n_e” is calculated by the following equation.

(36) β (9,3) = added value below (a) For e_i_g_h_t: p (e_i_g_h_t | o_n_e) ×
β (9 + 5,5) (b) For e_i_g_h: p (e_i_g_h
| O_n_e) × β (9 + 4,4) (c) For e_i_g: p
(E_i_g | o_n_e) × β (9 + 3, 3) (d) For e_i: p (e_i | o_n_e) × β (9 + 2, 2) (e) For e: p (e | o_n_e) × β (9 + 1, 1)

<< A2 >> Class Case In the case where a sequence belongs to a class, variables α and β are calculated by replacing the probability part of the bigram in the above example as follows. (A) p (o_n_e | n_e_s_i_x) is p (class of o_n_e
| Class of n_e_s_i_x) × p (o_n_e | class of o_n_
Replaced with e). (B) p (o_n_e | e_s_i_x) is p (class of o_n_e
| Class of e_s_i_x) x p (o_n_e | class of o_n_
Replaced with e). (C) p (o_n_e | s_i_x)
Is p (class of o_n_e | class of s_i_x) × p (o_n_
e | class of o_n_e). (D) p (o_n_e | i_x) is p (class of o_n_e | clas
s of i_x) × p (o_n_e | class of o_n_e). (E) p (o_n_e | x) is p (class of o_n_e | class
of x) × p (o_n_e | class of o_n_e). (F) p (e_i_g_h_t | o_n_e) is p (class of e_i_g
_h_t | class of o_n_e) × p (e_i_g_h_t | class of e_
i_g_h_t). (G) p (e_i_g_h | o_n_e) is p (class of e_i_g_h)
| Class of o_n_e) × p (e_i_g_h | class of e_i_g_
Replaced with h). (H) p (e_i_g | o_n_e) is p (class of e_i_g | cl
ass of o_n_e) × p (e_i_g | class of e_i_g). (I) p (e_i | o_n_e) is p (class of e_i | class
of o_n_e) × p (e_i | class of e_i). (J) p (e | o_n_e) is p (class of e | class of o
_n_e) × p (e | class of e).

<Statistical Language Model Generation Processing> FIG. 3 is a flowchart showing the statistical language model generation processing executed by the statistical language model generation unit 20 of FIG.
Here, the statistical language model generation unit 20 includes a working RAM 30 divided into the following memories 31 to 36, as shown in FIG. (A) Parameter memory 31: A memory for storing various setting parameters used in the generation processing. (B) Sequence frequency probability memory 32: A memory for storing the calculated frequency probability of each sequence. (C) Class definition memory 33: a memory for storing character strings belonging to each estimated class. (D) Class conditional frequency probability memory 34: A memory for storing the estimated frequency probability for each character string belonging to each class, that is, the frequency probability of a character string between class conditional classes. (E) Class bigram frequency probability memory 35: A memory that stores the frequency probability of a bigram of a class. (F) Segmented sequence memory 36: A memory for storing a segmented sequence (character string) after the re-estimation processing.

In FIG. 3, first, in step S1,
The text data is read from the learning text data memory 21. Here, the input learning text data is a sequence of discrete units, where the unit is, for example, a character, and the sequence is a character string that can be a word or a sentence. The following input parameters are set in advance and stored in the parameter memory 31. (A) the maximum length of the sequence (represented by the number of units),
(B) the number of classes after the re-estimation process; (c) a threshold value for the number of discarded sequences (that is, the minimum value of the number of occurrences of discarded sequences); and (d) termination conditions. Here, the termination condition is, for example, a threshold value of the number of repetitions k.

Next, in step S2, an initialization process is executed. In the input training text data,
The relative frequency of the sequence consisting of a plurality of units is counted, and the frequency probability of each sequence is initialized based on the relative frequency. Also, a sequence that is equal to or less than the set threshold value of the number of sequences to be discarded is discarded. And
Reset the iteration number parameter k to zero.

Next, in step S3, a classification process using the Brownian algorithm is executed. In this classification processing, based on the frequency probability of each sequence at the time of the iteration number parameter k, the class definition and the class condition with the class number at the time of the iteration number parameter k are set so as to minimize the loss of mutual information between classes. Frequency probability of the sequence between classes,
And the frequency probability of the class bigram is calculated and output to and stored in the memories 32 to 35, respectively. The classification criterion in this process is the mutual information amount between adjacent sequences, and the above-described algorithm is used. These mutual information and algorithms are proposed by Brown for the case of adjacent words, and in this embodiment, the Brown algorithm is used. However, the invention is not so limited and other classification algorithms based on unit frequency probabilities can be used.

Next, in step S4, the number 2 obtained by referring to the forward / backward algorithm
2—Re-estimation processing using bi-multigrams is performed using Equation 24. In this process, the immediately preceding step S3
Based on the class definition, the frequency probability of the sequence between classes with class conditions, and the frequency probability of the class bigram for the iteration number parameter k calculated in the above, the frequency of the bigram between the sequences for the next iteration parameter In order to obtain the maximum likelihood estimation value of the probability, the frequency probability of each sequence in the case of the number of iterations parameter (k + 1) is re-estimated and calculated, output to the memory 32, and stored. The processing criterion in this processing is expressed by using the above equations (22) to (24).
That is, the maximum likelihood estimation value that is the maximum value among the likelihoods of the analysis results calculated assuming the dependence of the classes of a plurality of sequences and the bigram is used as a reference value. EM algorithm is used.

Next, in step S5, it is determined whether or not a predetermined end condition is satisfied. If NO, the repetition number parameter k is incremented by 1 in step S6, and the processing in steps S3 and S4 is repeated. On the other hand, if “YES” in the step S5, data of the generated statistical language model is output to the statistical language model memory 22 and stored. Here, the data of the generated statistical language model is data on the frequency probability of each sequence,
Specifically, it is the following data. (A) data of each sequence having a maximum likelihood estimate when the input data is segmented into a plurality of sequences; (b) class definitions, ie, sequences in each class; and (c) frequency probabilities of the classes; The bigram probability for each class, the class conditional probability for each sequence.

FIG. 4 is a flowchart showing a classification process using the Brownian algorithm which is a subroutine of FIG. For automatic word classification, Brown et al. Have proposed an algorithm for use in automatic sequence classification (for example, see Prior Art Document 5).
In the present embodiment, this is used. Brown et al. Show that the division or segmentation of a sentence into classes that maximizes the likelihood is also the division or segmentation that maximizes the mutual information between adjacent words. They take as input a bigram distribution of words, split them into word classes, and output a class distribution (greedy algori
thm). On the other hand, the present inventor has applied this algorithm by employing the distribution of bi-multigram frequency probabilities (ie, the distribution of sequence bigram frequency probabilities) as input. The output is the segmentation of the sequences into classes and the distribution of the frequency probabilities of each sequence.

The clustering of words using mutual information used in this classification processing will be described in detail (for example,
Prior Art Document 6: Kenji Kita et al., "Speech Language Processing", Morikita Publishing, pp. 110-113, November 15, 1996.
Day issue ”. ). Here, a method of maximizing mutual information between classes will be described as a method of classifying words based on adjacent words. The theoretical basis for mutual information-based clustering is that the maximum likelihood method of dividing words into classes in a bigram class model is a class assignment that maximizes the average mutual information of adjacent classes. And The N-gram class model is an N-gram of a word class as shown in the following equation.
And the word's N
(This expression is the same as the HMM expression in morphological analysis if the word class is replaced by the part of speech. Therefore, this word classification method automatically determines the optimal part of speech system. It is also conceivable to ask for it.

P (w _i | w ₁ ^i-1 ) ≒ P (w _i | c _i ) P (c _i
| C _{i-n + 1} ^i-1 )

Here, it is assumed that V words are divided into C classes using a function π that maps the words w _i to classes c _i . Given a learning text t ₁ ^T , P (t
^{_{2 T | t 1) = P}} (T 2 | T 1) P (t 3 | t 2) ... P (t T |
The function π may be determined so as to maximize t _T-1 ). Although details are omitted, the log likelihood L (π) per word, the entropy H (w) of the word, and the average mutual information I (c ₁ ; c ₂ ) of the adjacent classes are approximately The relationship of the expression holds.

[0086]

(38)

Since H (w) does not depend on the division π, L (π) can be maximized by maximizing I (c ₁ ; c ₂ ). At present, there is no known algorithm for obtaining a partition that maximizes the average mutual information amount. However, even the following greedy algorithm used in the present embodiment can obtain a rather interesting cluster. Such a method of generating a cluster having an inclusion relation is called hierarchical clustering. On the other hand, a method of generating clusters having no overlap, such as the k-means algorithm, is called non-hierarchical clustering.

When the next merging is repeated V-1 times, all the words are in one class. That is, from the order in which the classes are merged, a binary tree having words as leaves is created. 1. Assign one class to all words. 2. Among the possible combinations of the two classes, the combination that minimizes the loss of the average mutual information is selected, and these are combined into one class. 3. By repeating Step 2 VC times, C classes are obtained.

In general, a hierarchical structure representing a process of forming a cluster is called a dendrogram. In natural language processing, this can be used instead of a thesaurus. To put it simply, this sub-optimal algorithm requires V ⁵ computations for V vocabulary. But,
(1) It is necessary to obtain only the change in the information amount when two clusters are merged. (2) It is used that the mutual information amount changes by merging two clusters is only a part of the whole. Then, the calculation cost of O (V ³ ), that is, the calculation cost of the order proportional to the cube of the number of repetitions V is sufficient.

In FIG. 4 showing the classification processing (or clustering processing), first, in step S11, an initial setting processing is executed, and each sequence is assigned to its own class. That is assigned to each class C _i respectively each sequence s _i. Thus, the distribution of the frequency probabilities of the initial bigrams of a class is equal to the distribution of the frequency probabilities of the bigrams of the sequence, and

P (s _i | C _i ) = 1.

Next, in step S12, for each pair of classes (C _k , C _l ), the mutual information loss when class C _k and class C _l are merged is calculated. Merge pairs of classes with the least loss of information. Then, in step S14, the distribution of the frequency probabilities of the classes stored in the memories 34 and 35 is updated according to the merge. Next, in step S15, it is determined whether or not the required number of classes set in the initialization processing in step S2 has been obtained. If NO, the process returns to step S12, and the above processing is repeated. On the other hand, if YES in step S15, the process returns to the main routine.

<Speech Recognition Apparatus> Next, the configuration and operation of the continuous speech recognition apparatus shown in FIG. 1 will be described. In FIG. 1, the phoneme HMM in the phoneme hidden Markov model (hereinafter, referred to as HMM) memory 11 connected to the word matching unit 4 is represented by including each state, and each state includes the following information. Having. (A) state number, (b) acceptable context class, (c) list of preceding and succeeding states, (d) parameters of output probability density distribution, and (e) self-transition probability and transition to succeeding state probability. Note that the phoneme HMM used in the present embodiment is generated by converting a predetermined speaker-mixed HMM because it is necessary to specify which speaker each distribution originates from. Here, the output probability density function is a Gaussian mixture distribution having a 34-dimensional diagonal covariance matrix. Further, the word dictionary in the word dictionary memory 12 connected to the word matching unit 4 stores a symbol string indicating a reading represented by a symbol for each word of the phoneme HMM in the phoneme HMM memory 11.

In FIG. 1, a uttered voice of a speaker is input to a microphone 1 and converted into a voice signal, and then input to a feature extracting unit 2. After performing A / D conversion on the input audio signal, the feature extraction unit 2 performs, for example, LPC analysis, and performs 34-dimensional feature parameters including logarithmic power, 16th-order cepstrum coefficient, Δlogarithmic power, and 16th-order Δcepstrum coefficient. Is extracted. The time series of the extracted feature parameters is input to the word matching unit 4 via the buffer memory 3.

The word collating unit 4 uses the one-pass Viterbi decoding method and the word hypothesis using the phoneme HMM 11 and the word dictionary 12 based on feature parameter data input via the buffer memory 3. Is detected, the likelihood is calculated and output. Here, the word matching unit 4 determines whether each HMM
The likelihood within a word and the likelihood from the start of utterance are calculated for each state. The likelihood is individually provided for each word identification number, word start time, and difference between preceding words. Further, in order to reduce the amount of calculation processing, the grid hypothesis of a low likelihood among the total likelihoods calculated based on the phoneme HMM 11 and the word dictionary 12 is reduced. The word collating unit 4 outputs the resulting word hypothesis and likelihood information to the word hypothesis narrowing unit 6 via the buffer memory 5 together with time information (specifically, a frame number, for example) from the utterance start time. .

The word hypothesis narrowing section 6 refers to the statistical language model in the statistical language model memory 22 based on the word hypothesis output from the word collating section 4 via the buffer memory 5 and determines the end time. For the word hypothesis of the same word having the same start time but different start times, the highest likelihood among the total likelihoods calculated from the utterance start time to the end time of the word is determined for each head phoneme environment of the word. After narrowing down word hypotheses so that they are represented by one word hypothesis,
The word string of the hypothesis having the maximum total likelihood among the word strings of all the narrowed word hypotheses is output as the recognition result. In the present embodiment, preferably, the first phoneme environment of the word to be processed is three phonemes including the last phoneme of the word hypothesis preceding the word and the first two phonemes of the word hypothesis of the word. I mean a line.

For example, as shown in FIG. 2, following the (i-1) -th word Wi-1, phoneme strings a1, a2,.
When the i-th word Wi consisting of
The six hypotheses Wa, Wb, Wc, Wd,
We and Wf exist. Here, the final phoneme of the former three word hypotheses Wa, Wb, Wc is / x /, and the final phoneme of the latter three word hypotheses Wd, We, Wf is / y /
And The hypothesis with the highest total likelihood (for example, 1 in FIG. 2) is the hypothesis in which the end time te is the same as the first phoneme environment (the top three word hypotheses in which the first phoneme environment is “x / a1 / a2” in FIG. 2). Delete the hypothesis). The fourth hypothesis from the top is not deleted because the first phoneme environment is different, that is, the last phoneme of the preceding word hypothesis is y instead of x. That is, only one hypothesis is left for each final phoneme of the preceding word hypothesis. In the example of FIG. 2, one hypothesis is left for the final phoneme / x /, and one hypothesis is left for the final phoneme / y /.

In the above embodiment, the head phoneme environment of the word is defined as a sequence of three phonemes including the last phoneme of the word hypothesis preceding the word and the first two phonemes of the word hypothesis of the word. Although defined, the present invention is not limited to this. The phoneme sequence of the preceding word hypothesis including the final phoneme of the preceding word hypothesis, and at least one phoneme of the preceding word hypothesis that is continuous with the final phoneme, And a phoneme sequence that includes a phoneme sequence that includes the first phoneme of the word hypothesis.

In the above embodiment, the feature extraction unit 2
The word collating unit 4, the word hypothesis narrowing unit 6, and the statistical language model generating unit 20 are constituted by a computer such as a digital computer, for example.
A phoneme HMM memory 11, a word dictionary memory 12,
The learning text data memory 21 and the statistical language model memory 22 are configured by a storage device such as a hard disk memory.

In the above embodiment, speech recognition is performed using the word collating unit 4 and the word hypothesis narrowing unit 6.
The present invention is not limited to this. For example, the present invention includes a phoneme matching unit that refers to the phoneme HMM 11 and a speech recognition unit that performs speech recognition of a word by referring to a statistical language model using, for example, the One Pass DP algorithm. Is also good.

[0100]

<Embodiment><First Embodiment of Statistical Language Model Generation Processing> An example in which input learning data is a 1000-character string as shown below, and segments from a unit character to a word. It is. "Onesixoneeightfivezer o
... "However, the odd-numbered word always follows the even-numbered word, and the even-numbered word always follows the odd-numbered word. The input parameters in this embodiment are as follows. (A) Maximum length of one sequence = 5, (b) number of classes = 2, and (c) threshold of sequence to be discarded = 10
0.

In the initialization processing (k = 0), relative count values of all combinations of characters observed more than 100 times in the learning data are set as initial values. Therefore, the counting result of the distribution of the frequency probability of the sequence at the repetition parameter k = 0 is as shown in the following table. Note that n of each sequence
b (·) represents a count value.

[0102]

[Table 2] ―――――――――――――――――――――――――――――――――― P (n | o) = nb (on) / nb (o) = 0.08 p (n e | o) = nb (one) / nb (o) = 0.06. . . p (n e s i x | o) = nb (onesix) / nb (o) = 0.005 p (e | o n) = nb (one) / nb (on) = 0.9 p (e s ｜ o n) = nb (ones) / nb (on) = 0.005. . . p (e s i x o ｜ o n) = nb (onesixo) / nb (on) = 0.001. . . p (s i x ｜ o n e) = nb (onesix) / nb (one) = 0.05. . . ――――――――――――――――――――――――――――――――――

In the classification processing in step S3, the input data is the distribution of the frequency probability of the sequence when the repetition parameter k = 0, and the output data in the classification processing is
It looks like this: (A) Class definition when iterative parameter k = 1

[Equation 40] class1 = ｛e s i x o; e; e t w o; n e
s i x; ......; f o u r; f o u r f; ...; g h t s; g h t o
n e; e i g h t｝

[Equation 41] class2 = ｛o n e; e s i x o; x; f i v;
f i v e; t s e v; s e v e n; ......; x n i; x n i n e; n
i n e; ...｝ class3 = ... (b) Distribution of class conditional frequency probabilities when the repetition parameter k = 1

[Mathematical formula-see original document] p (e s i x o | class 1), p (e | class 1),. . . p (o n e | class 2), p (e s i x o ｜ class
2),. . . (C) Distribution of frequency probability of class bigram when iterative parameter k = 1

P (class1 | class2) = 0.3 p (class2 | class1) = 0.1 p (class3 | class1) = 0.4. . .

In the re-estimation process in step S4, the class definition and the distribution of the class frequency probability when the iteration parameter k = 1 are used as input data, and the following iteration parameter k =
The distribution of the frequency probability of the sequence at 1 is output.

P (n | o) = 0.9 p (n e | o) = 0.8 p (n e s | o) = 0.05. . . p (n e s i x | o) = 0

[Equation 45] p (e | o n) = 0.02 p (e s ｜ o n) = 0.001. . . p (e s i x o ｜ o n) = 0. . . p (s i x ｜ o n e) = 0.5. . .

Thereafter, the same processing is executed, and the output result in the first embodiment is as follows. (A) Input character string segmented (ML segmentation) "o n es i xo n ee i g h tf i v ez e r o
... ”(b) Class definition

[Equation 46] class1 = ｛o n e; t h r e e; f i v e;
s e v e n; n i n e｝ class2 = ｛z e r o; t w o; f o u r; s i x; e i g
h t｝ (c) Distribution of frequency probabilities with class conditions

[Equation 47] p (o n e | class 1) = 0.2 p (t h r e e | class 1) = 0.2 p (f i v e | class 1) = 0.2. . . p (z e r o | class 2) = 0.2 p (t w o | class 2) = 0.2 (d) Distribution of frequency probability of class bigram

P (class1 | class2) = 1 p (class2 | class1) = 1

<Second Embodiment of Statistical Language Model Generation Processing> In the case where the input learning data is the following sentence based on natural language text data, that is, a word string, the unit word is a phrase. 7 is an embodiment for explaining a case where the segmentation is performed. Here, <s> is a symbol indicating the start, and </ s> is a symbol indicating the end. "<S> good afternoon new washington hotel may i he
lp you ... </ s>] Here, the input parameters are as follows. (A) maximum length of sequence = several words (eg, 1 to 5 words, 4 in the following embodiment), (b) number of classes = 1000, and (c) threshold value of initialization processing = 3
0.

In the initialization process (k = 0), relative count values of all combinations of words observed more than 30 times in the learning data are set as initial values. Therefore, the counting result of the distribution of the frequency probability of the sequence at the repetition parameter k = 0 is as shown in the following table.

[0108]

[Table 3]

The output result in the second embodiment is as follows. (A) Input character string segmented (ML segmentation) "good_afternoon new_washington_hotel may_i_help_y
ou "(b) Class definition

[Equation 49] class1 = ｛good afternoon; good mo
rning; hello; may i help you ...} ... class2 = ｛new washington hotel; sheraton ho
tel; plaza; ...｝ ... class1000 = ｛give me some; tell me｝ (c) Distribution of frequency probability with class condition

[Mathematical formula-see original document] p (good afternoon | class 1) = 0.003 p (good morning | class 1) = 0.002 p (hello | class 1) = 0.002. . . (D) Distribution of class bigram frequency probabilities

P (class2 | class1) = 0.04 p (class3 | class1) = 0.005. . .

<Experiment and Experimental Results> The present inventor conducted the following experiment in order to experiment the performance of the device of the embodiment. First, the protocol and database experiments and experimental results will be described. The purpose of learning bigram dependencies between variable length phrases is to improve the limitations of the conventional word bigram model while reducing the number of parameters in the model to that of a word trigram. Therefore, a suitable criterion for performing an evaluation of a bi-multigram model is to measure its predictive ability, number of parameters, and compare it to those of the conventional bigram, trigram model. The predictive ability is usually evaluated by measuring perplexity as:

[0111]

52 = PP = exp {-(1 / T) log (L (W))}

Here, T is the number of words in the sentence W. A lower perplexity PP indicates that the prediction of the model is more accurate. In the statistical model, there are actually two perplexity values PP and PP ^* , and L (W) in Equation 52 is calculated as follows.

[0113]

(Equation 53) as well as

L (W) = L (W, S ^* )

The difference between the two perplexities PP ^* -PP is always a positive number or zero, and the degree of ambiguity of the analysis result S of the sentence W, or the likelihood of the best analysis result like an utterance recognizer. If degrees are used to reach the likelihood of a sentence, measure the loss in prediction accuracy.

In the following, first, the loss (PP ^* -PP) in a certain estimation procedure is evaluated, and the forward-backward algorithm (Equation 15) or the deterministic method (Equation 16) is used for the influence of the estimation procedure itself. Consider using
Finally, these results are compared with those obtained using the conventional n-gram model. To achieve this goal, a well-known CM by Clarkson et al. (1997)
Use the U toolkit. As the experimental object, the data of “Travel arrangement” owned by the present applicant in the following table is used.

[0116]

[Table 4] Data on “Travel arrangements” owned by the applicant of the present invention ―――――――――――――――――――――――――――――――― ― Learning test ――――――――――――――――――――――――――――――――― Number of sentences 13650 2430 Number of tokens 167000 29000 (1% OOV ) Number of vocabulary 3525 + 280OOV ――――――――――――――――――――――――――――――― (Note) OOV is an abbreviation of Out Of Vocabulary Yes, words that are not in the vocabulary.

This database is a conversation about travel / accommodation information which is spontaneously performed between a hotel clerk and a customer. Words that are stagnant, and incorrect start, are mapped to a single marker " ^* uh ^* ". In this experiment, the maximum length of the phrase was changed from n = 1 word to 4 words (when n = 1, the bi-multigram corresponds to the conventional bigram). The frequency probabilities of all bi-multigrams are estimated in 6 training iterations, discarding all sentences that appear less than 20 times in initialization and less than 10 times in each iteration, and branching the phrase dictionary. I mowed. Here, when the threshold value in the initialization is in the range of 10-30, using a different pruning limit value in the data does not significantly affect the result. The threshold for iteration is about half of that.

However, since all the one-word phrases are maintained irrespective of the estimated number of occurrences (the phrases s _i and s _j are one-word phrases, the re-estimated value of the combination c (s _i , s _j )) Is zero, the combination c
(S _i , s _j ) is reset to 1. ), All word bigrams will appear in the final dictionary. In addition, the bigram probabilities of all n-grams and phrases are
Known by Witten et al. (1991)
Using the Witten-Bell discounting method, smoothing is performed by the well-known back-off smoothing method by Katz (1987). Here, the reason why the Witten-Bell counting method is selected is that the best perplexity score can be obtained when the conventional n-gram is used in the test data.

Next, an experiment in which clustering is not performed will be described. First, in terms of the degree of non-determinism, the perplexity value PP ^* obtained after learning by the forward-backward algorithm in a test on data on “travel arrangement” owned by the present applicant in Table 4 And PP are shown in the following table. The difference in perplexity values (PP ^* -PP) typically stays within about one point of perplexity. This means that relying on a single best phrase should not significantly impair the accuracy of the prediction.

[0120]

[Table 5] Degree of nondeterminism method ―――――――――――――――――――――――――――― n 1 2 3 4 ―――――― ――――――――――――――――――――――――― PP 56.0 43.9 44.2 45.0 PP ^* 56.0 45.1 45.4 46 ３ ――――――――――――――――――――――――――――――

Next, in the influence of the re-estimation procedure, the perplexity value PP ^* and the model size using either the forward-backward algorithm or the Viterbi estimation algorithm are shown in the following table.

[0122]

[Table 6] Effect of estimation method: Test perplexity value PP ^* ――――――――――――――――――――――――――――――― n 1 2 3 4 ――――――――――――――――――――――――――――― FB method 56.0 45.1 45.4 46. 3 Viterbi method 56.0 45.7 45.9 46.2 ―――――――――――――――――――――――――――――――――

[0123]

[Table 7] Influence of estimation method: Model size ――――――――――――――――――――――――――――――― n 1 2 3 4 ――――――――――――――――――――――――――――――― FB method 32505 44382 43672 43186 Viterbi method 32505 65141 67258 67295 ――――― ――――――――――――――――――――――――――――

As can be seen from Tables 6 and 7, as far as the perplexity values are concerned, the estimation method has little effect and it seems to be slightly advantageous to use the learning by the forward-backward algorithm. On the other hand, the size of the model, when measured as individual bi-multigrams at the end of training,
About 30% in backward algorithm learning
Also decreases. That is, for the same test perplexity value, the difference is about 40,000 to 60,000.

The bi-multigram results generally suggest that heuristics for pruning with phrase abandonment cannot completely avoid overlearning. Indeed, n = 3 (perhaps implying a dependency spanning 6 to 8 words)
The perplexity value of 4 is higher than that for n = 2 (dependency is limited to 4 words). Other methods, perhaps those that favor longer phrases over shorter ones, are considered successful.

Further, in comparison with n-gram, the perplexity value (PP) obtained from learning by the forward-backward algorithm, n-gram
The model size for m and the bi-multigram are shown in the following table.

[0127]

[Table 8] Comparison of n-gram ―――――――――――――――――――――――――――――――― Test perplexity value PP ― ――――――――――――――――――――――――――――――――― Value of n 1 2 3 4 ―――――――――― ―――――――――――――――――――――――― n-gram 314.2 56.0 40.4 39.8 Bi-multigram 56.0 43.9 44 .2 45.0 ――――――――――――――――――――――――――――――――――

[0128]

[Table 9] Comparison of n-gram ―――――――――――――――――――――――――――――――― Model size ―――― ―――――――――――――――――――――――――――――― n value 1 2 3 4 ―――――――――――――― ―――――――――――――――――――― n-gram 3526 32505 75511 112148 Bi-multigram 32505 44382 43672 43186 ――――――――――――――― ―――――――――――――――――――

As can be seen from Tables 8 and 9, the lowest bi-multigram perplexity score (43.
9) is still higher than the trigram value, but is higher than the bigram value (56.0).
4) The value is closer. In addition, the trigram score depends on the discounted method. In addition,
According to the linear counting method, the perplexity of the trigram in this test was 48.1.

The 5-gram perplexity value (not shown in the above table) is 40.8, which is slightly higher than the 4-gram score. This is consistent with the fact that the bi-multigram perplexity does not decrease when n> 2 (ie, where the dependency spans more than four words). Finally,
The number of entries in the bi-multigram model is smaller than the number of entries in the trigram model (75000 versus 45000), indicating a trade-off between model accuracy and model size achieved by multigram.

Further, an experiment using clustering and an experimental result will be described. In this experiment, perplexity scores were not improved by phrase clustering. The increase in perplexity is very small (less than 1 point) only for a small part of the phrase (10
(.About.20%) is the time when it becomes a cluster, beyond which perplexity deteriorates considerably. This effect
When the class estimate is not integrated with the word estimate, n-gram
In the framework of the report. However, by clustering phrases, some of the verbal fluency, such as the insertion of words that characterize spontaneous speech, can be handled naturally. To illustrate this point, the following table first lists the phrases that are integrated during learning of a model that handles phrases up to n = 4 words. Here, the phrase including “ ^* uh ^* ” indicating the stagnation is shown at the top of the table. Mainly, the differences in phrases due to the speaker's depressing are often integrated together.

[0132]

[Table 10] Examples of integrated phrases in a model that handles up to four-word sequences ―――――――――――――――――――――――――――――――― -(Yes that will; ^* uh ^* that would} {yes that will be; ^* uh ^* yes that's} { ^* uh ^* by the; and by the} {yes ^* uh ^* i; i see i} {okay i understand; ^* uh ^* yes please} {could you recommend; ^* uh ^* is there} { ^* uh ^* could you tell; and could you tell} {so that will; yes that will; yes that would; uh ^* that would} {if possible i'd like; we would like; ^* uh ^* i want} {that sounds good; ^* uh ^* i understand} { ^* uh ^* i really; ^* uh ^* i don't} { ^* uh ^* i'm staying; and i'm staying} {all right we; ^* uh ^* yes i} ――――――――――――――――――――――――――――――――――― {good morning this; good afternoon this} {yes i do; yes thank you} {we'll be looking forward; we look forward} {dollars a night; and forty yen} {for your help; for your information} {hold the line; want for a moment} {yes that will be; and could you tell} {please go ahead; you like to know} {want time would you; and you would} {yes there is; but there is} {join phillips in room; ms. suzuki in} {name is suzuki; name is ms. suzuki} {i'm calling from; a; also i'd like} {much does it cost; can reach you} {thousand yen room; dollars per person} {yes i do; yes thank you; i see sir} {you tell me where; you tell me what} {a reservation for the; the reservation for} {your name and the; you give me the} {amy harris in; is amy harris in} {name is mary phillips; name is kazuo suzuki} {hold on a moment; wait a moment} {give me some; also tell me} ――――――――――――――――――――――――――――――――――

According to Kawahara et al. (1997), the above table further provides other motivations for performing phrase search and clustering, apart from word prediction, namely topic identification and modeling of dialogue, and language. It shows how to deal with issues related to understanding. Certainly, the clustered phrases in this experiment were derived completely blind, ie, without any semantic / pragmatic information, but the in-class phrases had strong semantic correlations. It is shown. However, in order to be able to use this technique efficiently for speech understanding, the constraints must be a phrase clustering process using some higher level information, such as speech act tags. Must be set to

As described above, ng between phrases is used.
Algorithms for deriving variable-length phrases that assume ram dependence have been proposed and estimated for the task of language modeling. The language corpus for a particular task can greatly reduce the bigram perplexity value by composing sentences into phrases, while keeping the number of entries in the language model lower than in the trigram model. It indicates that there is. However, these results are further improved by a more efficient pruning method, which makes it possible to learn about longer dependencies without performing unnecessary learning. Furthermore, since the forms of inflection can be easily integrated into the framework, it is possible to assign a common label to phrases having different lengths. Since the semantic relations of phrases are integrated, this method is also used in the field of dialog modeling and language understanding. In this case, if semantic / pragmatic information is used, a process for obtaining a phrase class can be limited.

<Modification> In the above embodiment, the unit is English characters, the sequence is a word, and the classification process classifies a character string into a plurality of word sequences, and the statistical sequence model Is a statistical language model. The present invention is not limited to this, and the unit may be a character of another natural language such as Japanese. The unit may be a word in a natural language, the sequence may be a phrase, and the classification processing may classify the word string into a plurality of phrase strings, and the statistical sequence model may be a statistical language model. .

<Effects of Embodiment> As described above,
According to the embodiment of the present invention, the following specific effects are obtained. (A) The frequency distribution of a sequence of words can be calculated using the EM algorithm, and the ML criteria can be optimized. That is, if the algorithm of the present embodiment is used, the clustering process can always be monotonically converged, and an analysis result of the optimum value can be obtained. (B) Analysis of sequence classification can be freely performed. Specifically, since a non-deterministic method using the above-described forward-backward algorithm is used, a solution having a degree of freedom can be obtained. It should be noted that the nondeterminism technique can be used because the variables α and β can be determined. Therefore, by improving the likelihood of the input data, when the sequence [bcd] is in the input sequence, [bc] + [d], [b] + [cd],
Division into small sequences such as [b] + [c] + [d] is possible. In other words, even if a sequence is given to the input sequence, the analysis is not predetermined and everything depends on the likelihood of the input data, i.e. it is not deterministic but clusters on the frequency probabilities of the input data Is performed. (C) Automatic classification of variable-length sequences can be performed. Here, the classification of the sequence does not depend on the classification of the word. In addition, the sequences are directly and automatically classified, and can be classified into common class sequences having different lengths with high accuracy.

Therefore, according to the embodiment of the present invention,
Compared to the conventional example, monotonous convergence toward the optimal state can be guaranteed, there is a degree of freedom, variable-length sequences can be handled in the same class, and practically high-speed using a digital computer It is possible to provide a statistical sequence model generation device, a statistical language model generation device, and a speech recognition device capable of processing.

[0138]

As described above in detail, according to the statistical sequence model generating apparatus according to the present invention, a variable length natural number is determined based on input data including a sequence which is a unit sequence composed of one or more units. Statistical sequence model generation for generating a bi-multigram statistical sequence model which is a bigram between a multigram which is N ₁ unit sequences and a multigram which is a variable length natural number N ₂ unit sequence. An initialization means for counting the bigram frequency probabilities of all combinations of unit sequences under the constraint of predetermined maximum values of N ₁ and N ₂ based on the input data. And, based on the frequency probabilities of the bigrams counted by the initialization means, merging so that the loss of mutual information when the pairs of each class are merged is minimized. By updating the frequency probability of a class and classifying it into a predetermined number of classes, the unit sequence included in the classified class, the frequency probability of a conditional unit sequence of the classified class, and the classification Classifying means for calculating and outputting the frequency probability of the bigram between the classes,
On the basis of the unit sequence included in the classified class output from the classification processing unit, the frequency probability of the conditional unit sequence of the classified class, and the frequency probability of the bigram between the classified classes, Using the algorithm, re-estimate to obtain the maximum likelihood estimation value. Here, using the forward-backward algorithm, for each unit sequence to be processed, Based on the forward likelihood for the unit sequence, the frequency probability of the unit sequence assuming the unit sequence immediately before the unit sequence as a condition, and the backward likelihood for the unit sequence that can be backward in time series, By re-estimating the probabilities of the bigrams between the sequences using an expression indicating the probabilities of the bigrams of the bigram, the statistical sequence of the bi-multigram as the re-estimation result is obtained. Comprising a re-estimation means for generating and outputting a Sumoderu, and control means for controlling to repeatedly execute the processing of the processing and the re-estimation means of the classification means until a predetermined termination condition is satisfied. Therefore, according to the present invention, it is possible to guarantee monotonous convergence toward an optimal state as compared with the conventional example, and it is possible to handle a variable-length sequence with the same degree of freedom and the same class, It is possible to provide a statistical sequence model generation device capable of generating a statistical sequence model through practical high-speed processing using a computer.

Further, according to the statistical language model generating apparatus of the present invention, in the statistical sequence model generating apparatus, the unit is a character of a natural language, the sequence is a word, and the classifying means includes: The character string is classified into a plurality of word strings, and the statistical sequence model is a statistical language model. Therefore, according to the present invention, it is possible to guarantee monotonous convergence toward an optimum state as compared with the conventional example, and it is possible to handle a variable-length sequence with the same class with a degree of freedom, A statistical language model generation device capable of generating a statistical language model by practically performing high-speed processing using a computer can be provided.

Further, according to the statistical language model generating device of the present invention, in the statistical sequence model generating device, the unit is a word of a natural language, the sequence is a phrase, and the classifying means includes: The word sequence is classified into a plurality of phrase sequences, and the statistical sequence model is a statistical language model. Therefore, according to the present invention, it is possible to guarantee monotonous convergence toward an optimum state as compared with the conventional example, and it is possible to handle a variable-length sequence with the same class with a degree of freedom, A statistical language model generation device capable of generating a statistical language model by practically performing high-speed processing using a computer can be provided.

Further, according to the speech recognition apparatus of the present invention, based on the speech signal of the input speech sentence,
In a speech recognition apparatus provided with speech recognition means for recognizing speech using a predetermined statistical language model, the speech recognition means refers to a statistical language model generated by the statistical language model generation apparatus and performs speech recognition. I do. Therefore, according to the present invention, it is possible to guarantee monotonous convergence toward an optimum state as compared with the conventional example, and it is possible to handle a variable-length sequence having the degree of freedom in the same class,
A statistical language model can be generated by practically high-speed processing using a digital computer. Also, by performing voice recognition using the generated statistical language model, voice recognition can be performed at a higher voice recognition rate than in the conventional example.

[Brief description of the drawings]

FIG. 1 is a block diagram of a continuous speech recognition apparatus according to an embodiment of the present invention.

FIG. 2 is a timing chart showing a process of a word hypothesis narrowing section 6 in the continuous speech recognition device of FIG.

FIG. 3 is a flowchart showing a statistical language model generation process executed by a statistical language model generation unit 20 of FIG. 1;

FIG. 4 is a flowchart showing a classification process using the Brownian algorithm, which is a subroutine of FIG.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Microphone, 2 ... Feature extraction part, 3, 5 ... Buffer memory, 4 ... Word collation part, 6 ... Word hypothesis narrowing part, 11 ... Phoneme HMM memory, 12 ... Word dictionary memory, 20 ... Statistical language model generation 21: learning text data memory, 22: statistical language model memory, 30: working RAM, 31: parameter memory, 32: sequence frequency probability memory, 33: class definition memory, 34: class conditional frequency probability memory, 35: class bigram frequency probability memory; 36 ... segmented sequence memory.

──────────────────────────────────────────────────続 き Continuation of the front page (72) Inventor Hideharu Nakajima 5th Sanraya, Inaya, Koika-cho, Soraku-cho, Kyoto Prefecture ATIR Corporation Voice Translation and Communication Research Laboratories (56) References DELIGNES S. "LANGUA GE MODELING BY VAR IABLE LENGTH SEQUENCE NESS: THEORETALIC FORMULATION AND EVA LUTION OF MULTIGR AMS", ICASP 1995, Vol. 1, pp. 169-172 Deligne S .; "INFIRE NCE OF VARIABLE-LE NGTH ACOUSTIC UNITS FOR CONTINOUUS S PEECH RECOGNITION N", ICASSP 1997, Vol. 3, pp 1731-1734 Frederic B. et. al. Variable-Length Sequence Modeling: Multigrams ", IEEE Signal Processing Letters, Vol. 2, No. 6, pp 111-113, JUNE 1995 (58) Fields investigated (Int. Cl. ⁷ , DB name ⁷ ). G10L 3/00-9/20 C12N 15/00 JICST file (JOIS)

Claims (9)

(57) [Claims] 1. A multigram that is a unit string of variable-length natural numbers N ₁ and a multi-gram that is a variable-length natural number N ₂ based on input data including a sequence that is a unit string composed of one or a plurality of units. A statistical sequence model generation device for generating a bi-multigram statistical sequence model which is a bigram between a unit sequence and a multigram, wherein N ₁ and N are predetermined based on the input data. Under the constraint of the maximum value of ₂ , initialization means for counting the frequency probabilities of the bigrams of all combinations of unit sequences, based on the frequency probabilities of the bigrams counted by the initialization means, Class pairs are merged so that the loss of mutual information when merged is minimized, the frequency probability of each class is updated, and the classes are classified into a predetermined number of classes. Classifying means for calculating and outputting a unit sequence included in the classified class, a frequency probability of a conditional unit sequence of the classified class, and a frequency probability of a bigram between the classified classes; The EM algorithm is performed based on the unit sequence included in the classified class output from the processing unit, the frequency probability of the conditional unit sequence of the classified class, and the bigram frequency probability between the classified classes. Is used to re-estimate to obtain the maximum likelihood estimation value. Here, using a forward / backward algorithm, for each unit sequence to be processed, the unit sequence to be processed can be taken forward in time series. Based on the forward likelihood of the unit sequence, the frequency probability of the unit sequence on the condition of the unit sequence immediately before the unit sequence, and the backward likelihood of the unit sequence that can be backward in time series. By re-estimating the frequency probabilities of the bigrams between the sequences using an expression indicating the frequency probabilities of the bigrams between the cans, a statistical sequence model of the bi-multigram, which is the re-estimation result, is generated and output. A statistical sequence model generation apparatus, comprising: a re-estimation unit; and a control unit that controls the process of the classification unit and the process of the re-estimation unit to be repeatedly executed until a predetermined end condition is satisfied. 2. The statistical sequence according to claim 1, wherein said initialization means further removes data of a combination of bigrams having a predetermined frequency probability or less from the frequency probabilities of the counted bigrams. Model generator. 3. The classifying unit according to claim 1, wherein the classifying unit classifies into the plurality of classes using a Brownian algorithm based on the frequency probability of the bigram counted by the initialization unit. The described statistical sequence model generator. 4. The above formula is used to calculate the frequency probability of a bigram between sequences of unit columns when the second unit column, which is the unit column, follows the first unit column in the input data. Expression for calculating for each unit sequence to be processed, the frequency probability of the bigram between the sequences is the sum of the likelihood in all the segmentations including the first and second unit sequences, 4. The statistical sequence model generation device according to claim 1, wherein the statistical sequence model generation device is obtained by dividing by a sum of likelihoods in all segmentations including one unit sequence. 5. The above-mentioned formula is a denominator indicating an average number of occurrences of each unit string in the input data, and an average for each unit string when a second unit string follows the first unit string in the input data. A numerator indicating the number of times, the numerator, for each unit sequence to be processed, the forward likelihood, the frequency probability of the unit sequence when the unit sequence immediately before the unit sequence is a condition, The denominator is the sum of the products of the backward likelihoods.The denominator is the forward likelihood for each unit sequence to be processed, and the frequency probabilities of all the unit sequences under the condition of the unit sequence immediately before the unit sequence. 5. The statistical sequence model generation apparatus according to claim 4, wherein the sum is the sum of the products of the backward likelihood. 6. The method according to claim 1, wherein the ending condition is when the number of repetitions of the processing of the classifying means and the processing of the re-estimating means reaches a predetermined number. The statistical sequence model generation device according to one of the above. 7. The statistical sequence model generation device according to claim 1, wherein the unit is a character of a natural language, the sequence is a word, and the classifying unit includes a character string. Is classified into a plurality of word strings, and the statistical sequence model is a statistical language model. 8. The statistical sequence model generating apparatus according to claim 1, wherein the unit is a word in a natural language, the sequence is a phrase, and the classifying unit includes a word string. Are classified into a plurality of phrase columns, and the statistical sequence model is a statistical language model. 9. A speech recognition device comprising speech recognition means for recognizing speech using a predetermined statistical language model based on a speech signal of an input uttered speech sentence, wherein the speech recognition means is provided. Or a speech recognition device characterized by performing speech recognition with reference to the statistical language model generated by the statistical language model generation device according to 8.