JP2011039965A - Model parameter estimation device, method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To allow a model parameter to be estimated more accurately than conventional devices and methods. <P>SOLUTION: A device receives one or more lists i which comprise a plurality of symbol sequences f<SB>i, j</SB>which have importance degrees e<SB>i, j</SB>assigned thereto and are expressed by identity vectors, and receives correct answer symbol sequences f<SB>i, O</SB>of respective list i, which have importance degrees e<SB>i, O</SB>assigned thereto and are expressed by identity vectors, and estimates a model parameter w and includes an importance degree conversion part and a model parameter estimation part. The importance degree conversion part converts, per list, importance degrees e<SB>i, j</SB>so that a value of the degree of importance of a prescribed symbol sequence is relatively larger than values of degrees of importance of symbol sequences other than the prescribed sequence. The model parameter estimation part estimates the model parameter w from the symbol sequences f<SB>i, j</SB>, correct answer symbol sequences f<SB>i, O</SB>, and degrees of importance after the conversion. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a model parameter estimation apparatus, method, and program used for model learning in a symbol sequence rearrangement problem.

  In speech recognition and machine translation, multiple provisional recognition results and translation results (word sequences) are output, and by finding a sequence with few errors (close to the correct answer), recognition and translation accuracy are improved. be able to. When a word sequence of individual correct candidates output by a speech recognizer or machine translator is called a symbol series, and a set of output correct candidates is called a list, the extraction of the correct symbol series from such a list is generally performed. A score is assigned to each symbol series, and the symbol series in the list is rearranged in the order of score. In other words, the word sequence with the highest score is usually the recognition / translation result, and even if it is not, it is possible to efficiently extract the result of the symbol sequence close to the correct answer by sequentially verifying the symbol sequence with the highest score. (Refer to Non-Patent Documents 1, 2, and 5 for voice recognition, and Non-Patent Documents 3 and 4 for machine translation).

  When a target symbol series is extracted from a list of symbol series, a model obtained by learning in advance is generally used. Hereinafter, a method of finding a sequence close to the correct answer using a model prepared in advance will be described with reference to FIG.

First, a list consisting of a plurality of symbol sequences is read (S11). Each symbol series is generally represented by a feature (feature) vector, and the feature is an N-gram or co-occurrence of words, parts of speech, phonemes, etc., frequency of dependency obtained from the result of applying syntactic analysis or dependency analysis, boolean (A binary representation of presence / absence) is used. However, the form of the list is not necessarily limited to the column of feature vectors, and any form that can finally extract a feature vector may be used even if it is an expression form such as a network. Note that the symbol sequence can be expressed by a feature vector by the following method (see Non-Patent Document 3). For example, consider a method of representing a symbol sequence OOXX with a symbol set {◯, χ, △} by a feature vector. If a feature value is 1 when a symbol appears in the symbol series and 0 when it does not appear, 1 and △ do not appear in the symbol series XX So it becomes 0. The feature vector is a vector representation of such a feature as [1, 1, 0] ^T. When a natural language word string is handled as a symbol series, additional information such as a syntax analysis result of each symbol series and its score is added, and a feature vector may be created including the information.

Next, referring to the model obtained by learning, a score corresponding to the symbol series is given (S12). There are various ways to calculate the score. When the vector w is a model parameter obtained by learning in advance _, the score S _w (f _{i, j} ) of the symbol sequence f _{i, j} represented by the feature vector is, for example, S _w (f _{i, j} ) = w ^T · f _{i, j} can be calculated (i is an index of a list (i = 1, 2,..., N), j is an index of a symbol sequence in each list i (j = 1, 2,. .., n _i )), T is the transpose of the matrix).

Then, by rearranging the symbol series f _{i, j} in accordance with the assigned score, the symbol series in the list can be arranged in the order closest to the correct answer (S13).

  A method for estimating the model parameter w used for score calculation will be described below with reference to FIG.

  First, a plurality of lists consisting of a plurality of symbol series are read (S21). As the number of lists to be read increases, it can be expected that model parameters that function with high accuracy can be obtained for various data. Also, the correct symbol series of each list is read together. However, the same symbol series as the correct symbol series may or may not be included in each list.

  Next, the model parameter w is estimated by learning based on the read information (S22). The parameter is estimated so that the correct symbol series is given a higher score than the other symbol series. That is, the model parameter w may be determined so as to reduce the number ErrorCount of symbol series to which a score higher than the score given to the correct symbol series is given. For example, w that minimizes the expression (1) is obtained.

Here, I (x) is a symbol included 0 when the value of x is positive, the function which gives 1 in other cases, f _{i, 0} is correct symbol sequence, N is the number of lists, the n _i list i The number of series.

Of course, each symbol series f _{i, j} output from a speech recognizer or machine translator usually has an importance e _{i, j} based on an arbitrary evaluation measure (for example, the rank of word error rates in the list). Since it is given, the estimation accuracy can be increased by using this for parameter estimation. For example, according to the ExpLoss Boosting (ELBst) method disclosed in Non-Patent Document 3, w that minimizes the value of L in Equation (2) may be obtained.

In the equation (2), there is an algorithm for efficiently estimating w particularly when the feature value is binary of 0 and 1.

Z. Zhou, J. Gao, FKSoong, and H. Meng, "A Comparative Study of Discriminative Methods for Reranking LVCSR N-Best Hypotheses in Domain Adaptation and Generalization," Proceedings of ICASSP, 2006, Vol.1, p.141 -144 Akio Kobayashi, Shohei Sato, Kazuho Onoe, Shinichi Honma, Satoshi Imai, Toru Toki, “Speech Recognition by Discriminative Scoring of Word Lattice”, Proceedings of the Acoustical Society of Japan, September 2007, p.233-234 M. Collins and T. Koo, "Discriminative Reranking for Natural Language Parsing," Association for Computational Linguistics, 2005, Vol. 31, No. 1, p. 25-70 F.J.Och, "Minimum Error Rate Training in Statistical Machine Translation," Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, 2003, p.160-167 B.Roark, M.Saraclar and M.Collins, "Corrective Language Modeling For Large Vocabulary ASR with The Perceptron Algorithm," Association for Computational Linguistics, Proceedings of ICASSP, 2004, Vol.1, p.749-752

For parameter estimation method according to equation (2), in the calculation of Σ of each list i, while correct symbol sequence f _{i, 0} is appearing n _i or all sections other symbol sequence f _{i, j} is Each appears in the j-th term, that is, only one term. Therefore, the model parameter w that minimizes L should strongly reflect the tendency common to many lists. If there is a remarkable feature in the correct symbol sequence of a certain list, There arises a problem that a low model parameter w is calculated. That is, it is important to give a high score to the correct answer of one list rather than giving a low score to errors common to many lists.

  An object of the present invention is to provide a model parameter estimation apparatus, method, and program capable of reducing the influence of such a problem and estimating a model parameter with higher accuracy than before.

Model parameter estimation device of the present invention, importance e _{i respectively,} the plurality of symbol sequences f _i that is expressed by _j is assigned feature _vector, consisting of _j, 1 or more lists i (i is the index of the list (i = 1, 2, · · ·, n), j is the index (j = 1, 2 of the symbol sequence for each i, · · _{·, n} i)) and, correct symbol of each list i which are respectively represented by a feature vector A model parameter estimation device that receives a sequence f _{i, 0} and estimates a model parameter w, and includes an importance level conversion unit and a model parameter estimation unit.

The importance level conversion unit sets the importance level e _{i, j} to a relatively large value for the importance level of a predetermined symbol sequence for each list as compared to the importance level values of symbol sequences other than the predetermined symbol sequence. Convert as follows.

The model parameter estimation unit estimates the model parameter w from the symbol series f _{i, j} , the correct symbol series f _{i, 0,} and the importance after the conversion.

  According to the model parameter estimation apparatus, method, and program of the present invention, the importance assigned to each symbol series in the list is appropriately converted, and a large importance is given to some incorrect answer symbol series to correct the correct symbol series. By reducing the influence relatively, the model parameters can be estimated with higher accuracy than in the past.

The figure which shows the function structural example of the model parameter estimation apparatus 100. FIG. The figure which shows the example of a processing flow of the model parameter estimation apparatus 100. The image figure which shows the difference in the shape of a Gaussian function by the difference in the value of a parameter. The figure which shows the content of each set for learning, development, and evaluation used for verification. The figure which shows the comparison verification result of the word error rate of this invention and a prior art. The figure which shows the verification result of the change of the word error rate at the time of changing the shape of a Gaussian function. The figure which shows the example of the rearrangement process flow of a symbol series. The figure which shows the example of the processing flow of model learning.

FIG. 1 shows a functional configuration example of the model parameter estimation apparatus 100 of the present invention, and FIG. 2 shows a processing flow example thereof. Model parameter estimation device 100, respectively importance e _i, a plurality of symbol sequences expressed in _j is assigned feature vector f _i, consisting of _j, 1 or more lists i (i is the index of the list (i = 1, 2,..., N), j is the symbol sequence index (j = 1, 2,..., N _i )) for each i, and the correct symbol sequence f for each list i represented by a feature vector. _{i, 0} is input, and the apparatus estimates and outputs the model parameter w, and includes an importance conversion unit 101 and a model parameter estimation unit 102.

The importance level conversion unit 101 converts the importance levels e _{i, j so} that the importance level of a predetermined symbol series is relatively higher than that of other symbol series for each list, The degree of importance E (e _{i, j} ) after conversion is output (S1). Specifically, by giving a large importance to a part of the incorrect answer symbol sequences, the influence of the correct answer symbol sequences is relatively reduced. Therefore, the function E used for conversion is a function having a single or a small number of convex shapes, and corresponds to, for example, a Gaussian function such as Expression (3).

Here, (a, b, c) are parameters for determining the shape of the Gaussian function, and are parameters given in advance prior to learning. a is a variable for controlling the amplitude, b is a variable for controlling the center position of the Gaussian function, and c is a variable for controlling the width of the Gaussian function. FIG. 3 shows the shape of the Gaussian function when the values of (a, b, c) are changed. In FIG. 3, it is assumed that e _{i, j} is limited to 5000 (values higher than that do not appear in the learning data), and the subsequent lines are indicated by dotted lines.

As a method of selecting (extracting) a symbol series that gives a large importance, for example,
・ In each list, extract a predetermined number (for example, about 1 to 50) in descending order of importance value. ・ Extract those with importance level higher than a predetermined value. Randomly extract a predetermined number (for example, about 1 to 50) with equal probability. • It may be possible to sequentially extract at a regular interval in descending order of importance. Also, by combining these, for example, a predetermined number of symbol sequences having importance levels lower than a predetermined value and those obtained by extracting a symbol sequence having an importance level higher than a predetermined value are randomly or You may extract as a combination of the thing extracted sequentially at equal intervals from the thing with a big value.

  Selecting a symbol sequence that gives a large importance is equivalent to specifying a peak position of the filter. After the peak position is determined, an arbitrary function that connects the peak positions may be designed in a range that is in accordance with the gist of the present invention that gives a higher importance to some symbol sequences than others. However, when an infinite (close) value is given to the peak position, a negative infinite (close) value is given between peaks, or a parameter estimation method that does not allow the importance of negative values is returned, a negative value is returned. It goes without saying that functions that obviously have an adverse effect on parameter estimation, such as designing a filter, should not be set.

  In order to determine the function E to be used specifically, it is preferable to try various E, evaluate the development set, etc., and select the best one (value of (a, b, c) in the expression (3)). The best way to select the best E using the development set is to divide the learning set into several parts and use the development set as a cross-validation method, where one is considered as a development set and the other is used for learning. Similar results can be obtained.

  In the present invention, each symbol series is randomly selected in the list even when the importance is not attached to the symbol series, or even when the importance is attached to all the symbol series. It is possible to apply by considering the value given to, or the distance from the correct answer in the feature vector space as the importance.

The model parameter estimation unit 102 calculates and outputs a model parameter w from the symbol series f _{i, j} , the correct symbol series f _{i, 0,} and the converted importance E (e _{i, j} ) ( S2). For example, equation (2) based on the ELBst method disclosed in Non-Patent Document 3 may be transformed into equation (4) to obtain w that minimizes the value of L in equation (4).

In the equation (4), there is an algorithm for efficiently estimating w particularly when the feature value is binary of 0 and 1.

  Note that the best w is generally estimated by using a development set by giving initial values of learning, convergence conditions, hyper parameters, and the like.

<Verification of effects>
A Japanese spoken corpus (CSJ) is used to verify the effect of the present invention. CSJ is a database consisting of speech data and transcripts. For verification, a learning set and a development / evaluation set shown in FIG. 4 were prepared.

  The lecture was divided into utterance units, and a 5000-best list was created with a speech recognition system. That is, the number of lists matches the number of utterances. The symbol series is a speech recognition result, and there are a maximum of 5000 symbol series in each list. For the feature, uni, bi-, tri-gram boolean and speech recognition score were used. For the importance, the rank in the list of each symbol series (ascending order of word error rate) was used. Note that the word error rate shown in FIG. 4 is calculated for the recognition result having the largest recognition score in the 5000-best list output by the speech recognition system.

Formula (2) that uses model parameter w as the importance e _{i, j} as it is and Formula (4) that converts the importance as E (e _{i, j} ) = E _gauss (e _{i, j} , a, b, c) ), And rearranging the symbol series using these, the symbol series having the highest score in the end is used as the speech recognition result, and the word error rates of the two are compared (FIG. 5). In addition, the difference in word error rate when the shape (a, b, c) of the function E _gauss (e _{i, j} , a, b, c) was changed was also verified (FIG. 6).

FIG. 5 shows that the word error rate can be reduced by the importance conversion according to the present invention. Further, FIG. 6 shows that the word error rate can be reduced as the value of b of E _gauss (e _{i, j} , a, b, c) is larger and the value of c is smaller. As described above, since c is a variable that controls the width of the Gaussian function, it can be confirmed from this verification that a relatively large weight is given to a small number of symbol sequences in each list, leading to high-accuracy model generation. I can say.

  As described above, according to the model parameter estimation apparatus and method of the present invention, the importance given to each symbol series in the list is appropriately converted, and a large degree of importance is given to some incorrect answer symbol series to give a correct answer. By relatively reducing the influence of the symbol series, it is possible to generate model parameters with higher accuracy than in the past.

  When each of the above devices is realized by a computer, processing contents of functions that each device should have are described by a program. The processing functions are realized on the computer by executing the program on the computer. In this case, at least a part of the processing content may be realized by hardware. Further, the various processes described above are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. In addition, it can change suitably in the range which does not deviate from the meaning of this invention.

Claims (7)

One or more lists i (i is an index of the list (i = 1, 2,..., N) each consisting of a plurality of symbol sequences f _{i, j} each assigned an importance e _{i, j} and represented by a feature vector. ), J is input with the symbol sequence index (j = 1, 2,..., N _i )) for each i and the correct symbol sequence f _{i, 0 for} each list i expressed by a feature vector. A model parameter estimation device for estimating the model parameter w,
Importance for converting the importance e _{i, j so} that the importance value of a predetermined symbol sequence is relatively larger than the importance value of a symbol sequence other than the predetermined symbol sequence for each list A degree converter,
A model parameter estimator for estimating a model parameter w from the symbol series f _{i, j} , the correct symbol series f _{i, 0,} and the importance after the conversion;
A model parameter estimation device comprising: The model parameter estimation apparatus according to claim 1, wherein
The model parameter estimation apparatus according to claim 1, wherein the predetermined symbol series is obtained by extracting a predetermined number of symbol series f _{i, j} in descending order of importance. The model parameter estimation apparatus according to claim 1, wherein
The predetermined symbol series is obtained by extracting a symbol series f _{i, j} having an importance level equal to or higher than a predetermined value and a predetermined number of symbol series f _{i, j} having an importance level lower than a predetermined value. A model parameter estimation apparatus characterized by being extracted at equal intervals sequentially from a random or large importance value. One or more lists i (i is an index of the list (i = 1, 2,..., N) each consisting of a plurality of symbol sequences f _{i, j} each assigned an importance e _{i, j} and represented by a feature vector. ), J is an index of a symbol sequence in each i (j = 1, 2,..., N _i )) and a correct symbol sequence f _{i, 0 of} each list i expressed by a feature vector. A model parameter estimation method for estimating the model parameter w,
Importance for converting the importance e _{i, j so} that the importance value of a predetermined symbol sequence is relatively larger than the importance value of a symbol sequence other than the predetermined symbol sequence for each list A degree conversion step;
A model parameter estimating step for estimating a model parameter w from the symbol series f _{i, j} , the correct symbol series f _{i, 0} and the importance after the conversion;
Model parameter estimation method to execute. The model parameter estimation method according to claim 4, wherein
The model parameter estimation method, wherein the predetermined symbol series is obtained by extracting a predetermined number of symbol series f _{i, j} in descending order of importance. The model parameter estimation method according to claim 4, wherein
The predetermined symbol series is obtained by extracting a symbol series f _{i, j} having an importance level equal to or higher than a predetermined value and a predetermined number of symbol series f _{i, j} having an importance level lower than a predetermined value. A model parameter estimation method characterized by being extracted at equal intervals sequentially from a random or large importance value.   The program for functioning a computer as an apparatus in any one of Claims 1 thru | or 3.

Images