JP5610304B2 - Model parameter array device, method and program thereof - Google Patents

Description

  The present invention relates to a model parameter arrangement apparatus and method for arranging a set of model parameter elements of a model used in the field of statistical machine learning including natural language processing such as speech recognition, machine translation, and speech classification in order of importance. And about the program.

  For example, classification problems and ranking problems are extremely important problems in natural language processing. The classification problem is a problem of estimating which of a finite number of classes in which a certain word string (sample) is classified. For example, an utterance classification in which utterance contents are classified into greetings, questions, answers, and the like corresponds to this. The ranking problem is a problem in which, when a set of a finite number of samples is given, each sample is given a rank from a certain viewpoint. Speech recognition and machine translation, which are technologies for converting speech signals into word strings, can be regarded as equivalent to this.

  In general, in a classification problem, a score corresponding to a sample and a class is given, and a class to be classified is estimated based on the magnitude of the score. In the ranking problem, a score is given for each sample, and the ranking is determined based on the magnitude of the score.

  In order to assign a score, a conversion rule for expressing a feature pair of a sample and a class needs to be defined in advance. In addition, it is necessary to prepare a model in advance.

  The model is a language model for speech recognition, for example, and is a function that returns a score for an input (word string, sample) expressed as a feature. Here, preparing a model means that a function value is determined and specific values of parameters are determined in advance. In the statistical method, model parameter values are determined using a large amount of collected learning data.

  Usually, the process of determining the values of model parameters, ie learning, is not easy. Specialized knowledge in the target field, large-scale computing system, and a large amount of learning data are required. For this reason, it is not realistic for a general user who does not have specialized knowledge for classification and ranking to learn a model according to the usage environment.

  When an expert prepares a highly versatile model, the number of model parameters may increase. In that case, the general user is forced to prepare a computer having a scale corresponding to the general user. For example, the language model has a parameter corresponding to each pair of adjacent words. For this reason, when a general-purpose language model that supports various vocabularies is to be constructed, the set of model parameter elements becomes very large.

  When the model is difficult to use due to its large scale, it is necessary to perform learning after an expert adjusts in advance the number of model parameters suitable for the use environment. (Non-Patent Document 1).

Takashi Kitauchi, Takehito Utsuro, Yuji Matsumoto, "Probability Model Learning of Japanese Morphological Analysis Using Error-Driven Feature Selection", Transactions of Information Processing Society of Japan, Vol.40 No.5 May 1999

  In the past, model parameter elements had to be re-estimated each time the computer system changed. In order to avoid re-estimating the model parameter elements, it is necessary to arrange the model parameter elements in the order of importance in advance and use only a necessary number of model parameters. The importance needs to be determined so that the score returned by the model does not change before and after the reduction of the number of model parameters. However, conventionally, there has been no apparatus or method for rearranging the model parameter elements in the order of importance after satisfying the conditions.

  The present invention has been made in view of such a problem, and an object of the present invention is to provide a model parameter array device, a method thereof, and a program for rearranging model parameter elements in order of importance.

The model parameter array device of the present invention includes an initialization unit, an element selection unit, a subset evaluation value storage unit, and a parameter ranking unit. The initialization unit initializes the selected subset candidate with an empty set. The element selection unit receives the set Θ and the selected subset candidate, and outputs all sets of the selected subset candidate and other parameter elements as subset candidates. Subset evaluation value storage unit as input a subset candidate obtains the maximum value or minimum value of each score E _t of the subset candidate, a difference between the evaluation value E is the score value of the aggregate theta, is the difference The smallest or largest subset candidate is stored as a selected subset candidate. The parameter ranking unit outputs arranged model parameters in which the selected subset candidates are arranged in the selected order or vice versa.

  The model parameter arrangement device of the present invention outputs arranged model parameters in which model parameter elements of an existing model are arranged in order of importance. The importance is determined so that the score returned by the model does not change before and after the reduction of the number of model parameters. For this reason, it is possible to obtain model parameter elements of a subset of an arbitrary size by using the model parameters already arranged for the importance. In other words, by arranging the model parameter element set Θ in order of importance by the model parameter arrangement apparatus of the present invention, it is possible to easily obtain a subset of model parameter elements suitable for an arbitrary computer system.

The figure which shows the function structural example of the model parameter arrangement | sequence apparatus 100 of this invention. The figure which shows the operation | movement flow of the model parameter arrangement | sequence apparatus 100. FIG. The figure which shows the Eval-A evaluation experiment result which model-reduces using data (a2) and evaluates by data (a3). The figure which shows the Eval-B evaluation experiment result which produces an arranged model parameter using data (a2) and evaluates with data (o2). The figure which shows the Eval-C evaluation experiment result which produces an arranged model parameter using data (o1) and evaluates with data (o2).

    Embodiments of the present invention will be described below with reference to the drawings. The same reference numerals are given to the same components in a plurality of drawings, and the description will not be repeated. Prior to the description of the embodiments, the basic idea of the present invention will be described.

[Basic idea of the present invention]
The problem of giving a desired score to the sample s (for example, a word string in speech recognition) is a concept that is common in many fields such as pattern recognition and natural language processing, and is extremely important. A function that gives a desired score is a model Q _Θ (s), and a set of model parameter elements that control the function is Θ (hereinafter also referred to as an element set Θ). Collect and estimate specific values for each element in the element set Θ. This is called model parameter estimation or learning.

  In general, the greater the number of elements in the element set Θ, the more difficult it is to estimate and the greater the number of samples required. Further, there may be an element that adversely affects the estimation in the element set Θ. If an optimal subset Φ of the element set Θ, that is, an optimal subset having a smaller number of elements than the element set Θ can be found, a function that gives an appropriate score based on the smaller number of elements can be obtained. The problem of finding this subset is called a variable selection problem. However, the purpose of the variable selection problem is to find a subset of the element set Θ, and usually the value of each element of the parameter subset Φ is estimated after Φ is determined.

  It is known that the variable selection problem is an NP-hard problem (NP-hard), and therefore various approximation methods have been proposed. For example, forward selection (SFS: Sequential Forward Selection) or backward selection (SBS) is one of them. In forward selection and backward selection, learning and selection of model parameters are repeated, and a subset that reduces an index such as an error rate is obtained. In the forward selection, important model parameter elements are adopted in order, and in the backward selection, model parameters that are not important are deleted in order. Each time a model parameter is selected, it is necessary to re-determine the value of the model parameter, that is, to re-learn, so that the learned model parameters having the number specified by the user cannot be returned instantaneously.

An object of the present invention is to select a subset of the element set Θ so that the change in the result of scoring becomes as small as possible. Selection of the subset allows the sample s to be scored using a small model Q _Φ (s) as an alternative to the model Q _Θ (s). The basic idea to achieve this is to determine the importance of elements in the element set Θ by forward selection or backward selection so that the score returned by the model does not change before and after the reduction of the number of model parameters. It is to select the specified number of elements from the thing. In the present invention, assuming that the values of all elements of the element set Θ have already been learned, the difference between the score calculated from the Θ and the score calculated from the subset is used as the evaluation function. In the conventional variable selection problem, parameter estimates for each element are not normally used, but are actively used in the present invention.

  FIG. 1 shows a functional configuration example of a model parameter array device 100 of the present invention. The model parameter array device 100 includes an initialization unit 5, an element selection unit 10, a subset evaluation value storage unit 20, a parameter ranking unit 30, and a control unit 40. The function of each part of the model parameter array device 100 is realized by, for example, reading a predetermined program into a computer composed of ROM, RAM, CPU, etc., and executing the program by the CPU.

The initialization unit 5 receives the set Θ of model parameter elements as an input. In the case of forward selection, the initialization subset 5 initializes the selected subset candidate Φ _t with an empty set, and in the case of backward selection, the selection subset candidate Φ _t Initialize with.

The element selection unit 10 receives a set Θ of N model parameter elements and a selected subset candidate Φ _t as input, and in the case of forward selection, all combinations of the selected subset candidate Φ _t and other parameter elements as subset candidates. Is output.

In the case of backward selection outputs all set after removing the model parameter elements that belong to the selected subset candidate [Phi _t as a subset candidates.

  Here, the model element set Θ is, for example, a set of model parameter elements a, b,..., N when the model y is expressed linearly (formula (1)). Hereinafter, a set of model parameter elements is denoted by {*}.

For example, when the model y is y = ax + bx + cx, the first subset candidate in the forward selection is {a}, {b}, {c}, and the selected subset candidate Φ _t is Φ _t = {a} Subset candidates are {ab}, {ac}. In the backward selection, the first subset candidate is {ab}, {ac}, {bc}, and the subset candidate when the selected subset candidate Φ _t is Φ _t = {ab} is {a}, {b} It becomes.

The subset evaluation value storage unit 20 obtains a difference between the maximum value or the minimum value of each score E _t of the subset candidate and the score value E of the element set Θ, and selects a subset candidate whose difference is the minimum or maximum. It is stored as a candidate Φ _t.

The parameter ranking unit 30 outputs the arranged model parameter Φ in which the selected subset candidate Φ _t is arranged in the selected order or in the reverse order. The control unit 40 controls time-series operations between the above-described units.

  With the above operation, the model parameter arrangement device 100 outputs the arranged model parameter Φ in which the model parameter elements of the element set Θ of the existing model are arranged in the order of importance.

  The operation will be described in more detail with reference to the operation flow of the model parameter array device 100 shown in FIG. For example, the case where the element set Θ is Θ = {a, b, c} and the subset selection in the element selection unit 10 is performed by forward selection will be described as an example.

In the case of forward selection, the initialization unit 5 makes the selected subset candidate Φ _{t an} empty set (step S5). The element selection unit 10 selects {a}, {b}, {c} as subset candidates because the initially selected subset candidate Φ _t is not selected (step S10). The forward selection for all of the model parameters elements of theta-[Phi _t performs recursive processing.

Subset evaluation value storage unit 20, a subset candidate {a}, {b}, obtaining the difference between the score E in the case of using all the elements of the maximum value of the score E _t and, element set Θ of {c} ( Step S20). Then, storing the subset candidate whose difference is the smallest as a selective subset candidate [Phi _t (step S21).

As selected subset candidate [Phi _t, for example, when selecting a subset candidate {a}, a forward selection to create a set in which to add the {a} to the selected subset candidate [Phi _t. In response to the input of the selected subset candidate Φ ₁ = {a}, the element selection unit 10 outputs the subset candidates {ab}, {ac} to the subset evaluation value storage unit 20 (step S10).

The subset evaluation value storage unit 20 obtains a difference between the maximum value of the score E _t of the subset candidates {ab} and {ac} and the score E when all the elements of the element set Θ are used, and the difference is minimum. For example, if the subset candidate is {ab}, {b} is added to the selected subset candidate (step S21). In this example, since the ranking of the model parameter elements a, b, and c ends at this point, {c} is also added to the selected subset candidate as the last model parameter element at the same time. The processes in steps S10 to S21 are repeated until there is no unprocessed model parameter element.

When the process for all model parameter elements is completed (No in step S41), the parameter ranking unit 30 outputs the stored selected subset candidate Φ _t as an arrayed model parameter Φ arranged in the order of t. (Step S30).

Thus, the forward selection is a method of starting from an empty set and adding elements in the order of the most important parameter elements under a given evaluation function. In the case of backward selection, a process of removing model parameter elements having the lowest importance in order is performed. For retrospective selection initial value of the selected subset candidate [Phi _t is an element set theta. In other words, Φ _t = Θ, and Φ _{t + 1} = Φ _t- {a} is processed with respect to Φ _{t + 1} = Φ _t + {a} of the forward selection, but the permutation of the importance of the model parameter element is The point of determination is the same, and the operation flow is the same as in FIG.

  Note that the subset evaluation value storage unit 20 may use an arbitrary distance function for the process of obtaining a difference in scores. For example, a square error of the score difference (Equation (2)) may be introduced into the distance function.

[Evaluation experiment]
For the purpose of confirming the effect of the model reduction method of the present invention, an evaluation experiment was conducted using a Japanese spoken language corpus (CSJ). CSJ is a corpus consisting of lecture speech and transcriptions. CSJ includes conference lectures and mock lectures. The mock lecture includes speech in the form of lectures such as “My Town” and “The happiest event in my life”.

An error correction language model for speech recognition was constructed, and the relationship between model size and accuracy was evaluated by reducing the model. In general, a speech recognition system can output a word string as a plurality of recognition candidates. Each recognition candidate word string is given a score, and the word string having the highest score is a normal recognition result. The error correction language model is a model that reranks recognition results by adding a correction score to a recognition candidate word string. Assuming that a set of recognition candidate word strings is S, the final recognition result s ^* is given by Expression (3).

Here, f ₀ (s) is a score corresponding to the recognition candidate word string s given by the speech recognition system. Further, a ₀ is a constant for adjusting the scaling, and is determined using development data or the like when estimating the model parameter A of the error correction language model.

  Table 1 shows a breakdown of data used for learning and evaluation of the error correction language model.

  Data (a1) to (a3) are conference lectures, and data (o1) and (o2) are mock lectures. A speech recognition system was applied to each utterance, and 5000 recognition candidate word strings were generated.

  The parameter of the error correction language model was estimated using the data (a1). That is, the element set Θ was estimated. For the feature, the frequencies of all words 1-gram, 2-gram, 3-gram and part-of-speech 1-gram, 2-gram, 3-gram appearing in (a1) were used. An n-gram is a sequence of n tokens (words or parts of speech). The final number of parameters is about 10 million.

Round-robin duel discrimination was used for parameter estimation. The value of the constant a ₀ for adjusting the scaling was determined so that the word error rate (WER) was minimized using the data (a2) as a development set.

  Three environments were prepared for model reduction evaluation. A case where an arrayed model parameter is created using the data (a2) and evaluated by the data (a3) is described as Eval-A. Similarly, when model reduction is performed using data (a2) and evaluation is performed using data (a3), an arrayed model parameter is generated using data (a1) and evaluation is performed using data (o2) Is Eval-B. Similarly, a set of data (o1) and data (o2) is Eval-C. Since the error correction language model is learned at the conference lecture, Eval-B and Eval-C are evaluated in the task of out-of-domain.

  Each result is shown in FIGS. The horizontal axis is the number of elements used and is a logarithmic scale. The vertical axis represents the WER of the recognition result after applying the error correction language model. In the figure, the characteristic indicated by the broken line is a conventional method, and the solid line is a case where a specified number of model parameters higher than the arranged model parameters created by the model parameter arranging apparatus 100 of the present invention are used. The arranged model parameters were created by backward selection.

  Under either condition, the model has been successfully reduced more greatly while suppressing the increase in the word error rate than the broken line (+) indicating the word error rate of the conventional method. That is, even if the number of elements is reduced, deterioration of the word error rate can be reduced.

  Thus, according to the model parameter arrangement method of the present invention, it is possible to greatly reduce the number of elements while suppressing a change in the recognition result to be small. Although the evaluation experiment results have been explained by taking speech recognition as an example, the model parameter arrangement method of the present invention is also used for models in the field of image recognition and natural language processing, as well as machine translation and speech classification. Is possible.

  When the processing means in the above apparatus is realized by a computer, the processing contents of the functions that each apparatus should have are described by a program. Then, by executing this program on the computer, the processing means in each apparatus is realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used. Specifically, for example, as a magnetic recording device, a hard disk device, a flexible disk, a magnetic tape or the like, and as an optical disk, a DVD (Digital Versatile Disc), a DVD-RAM (Random Access Memory), a CD-ROM (Compact Disc Read Only). Memory), CD-R (Recordable) / RW (ReWritable), etc., magneto-optical recording medium, MO (Magneto Optical disc), etc., semiconductor memory, EEP-ROM (Electronically Erasable and Programmable-Read Only Memory), etc. Can be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Further, the program may be distributed by storing the program in a recording device of a server computer and transferring the program from the server computer to another computer via a network.

  Each means may be configured by executing a predetermined program on a computer, or at least a part of these processing contents may be realized by hardware.

Claims (7)

Given a set Θ of N model parameter elements as input,
And the initialization section for initializing an empty set a select subset candidates,
As input the set theta and select subset candidates, to the selection subset candidates, included in the set theta, and a subset candidates all sets have added one model parameter elements not included in the selected subset candidate An element selection unit to output as
As inputs the subset candidate, and the maximum value or minimum value of each score E _t of the subset candidate, the evaluation value E is the score value in the case of using all of the N model parameters elements of the set Θ And a subset evaluation value storage unit that stores a subset candidate having the smallest or largest difference as a selected subset candidate ,
In the element selection unit and the subset evaluation value storage unit, the process of updating the subset candidate and the selected subset candidate is repeated until there is no unprocessed model parameter element for the N model parameter elements,
A parameter ranking unit that outputs arranged model parameters in which the selected subset candidates are arranged in the selected order or vice versa ;
Model parameter array device. Given a set Θ of N model parameter elements as input,
An initialization unit for initializing the selected subset candidate above set theta,
As input selection subset candidates, the element selection unit for outputting the entire set obtained by removing a model parameter elements from the selected subset candidate as a subset candidates,
As inputs the subset candidate, and the maximum value or minimum value of each score E _t of the subset candidate, the evaluation value E is the score value in the case of using all of the N model parameters elements of the set Θ And a subset evaluation value storage unit that stores a subset candidate having the smallest or largest difference as a selected subset candidate ,
In the element selection unit and the subset evaluation value storage unit, the process of updating the subset candidate and the selected subset candidate is repeated until there is no unprocessed model parameter element for the N model parameter elements,
A parameter ranking unit that outputs arranged model parameters in which the selected subset candidates are arranged in the selected order or vice versa ;
Model parameter array device. In the model parameter arrangement device according to claim 1 or 2,
The distance function for obtaining the difference, the model parameter array device characterized by using the function of the square error. Given a set Θ of N model parameter elements as input,
And the initialization process to initialize an empty set a select subset candidates,
As input the set theta and select subset candidates, to the selection subset candidates, included in the set theta, and a subset candidates all sets have added one model parameter elements not included in the selected subset candidate Element selection process to output as
As inputs the subset candidate, and the maximum value or minimum value of each score E _t of the subset candidate, the evaluation value E is the score value in the case of using all of the N model parameters elements of the set Θ And a subset evaluation value storing process for storing a subset candidate having the smallest or largest difference as a selected subset candidate ,
In the element selection process and the subset evaluation value storage process, the process of updating the subset candidate and the selected subset candidate is repeated until there is no unprocessed model parameter element for the N model parameter elements,
Further comprising a parameter ranking process of outputting the arranged model parameters in which the selected subset candidates are arranged in the selected order or vice versa .
Model parameter array method. Given a set Θ of N model parameter elements as input,
An initialization step for initializing the selected subset candidate above set theta,
An element selection process in which a selected subset candidate is input and all sets obtained by removing one model parameter element from the selected subset candidate are output as subset candidates ;
As inputs the subset candidate, and the maximum value or minimum value of each score E _t of the subset candidate, the evaluation value E is the score value in the case of using all of the N model parameters elements of the set Θ And a subset evaluation value storing process for storing a subset candidate having the smallest or largest difference as a selected subset candidate ,
In the element selection process and the subset evaluation value storage process, the process of updating the subset candidate and the selected subset candidate is repeated until there is no unprocessed model parameter element for the N model parameter elements,
Further comprising a parameter ranking process of outputting the arranged model parameters in which the selected subset candidates are arranged in the selected order or vice versa .
Model parameter array method. In the model parameter arrangement method according to claim 4 or 5,
A model parameter arrangement method using a square error function as a distance function for obtaining the difference.   A program for causing a computer to function as the model parameter array device according to any one of claims 1 to 3.

Images