JP6343582B2 - Language model generation apparatus, method, and program - Google Patents

Description

  The present invention relates to a technique for constructing a language model. In particular, the present invention relates to a technique for constructing a new speech recognition language model called a latent word recurrent neural network language model.

  Speech recognition and machine translation require a language model for linguistic prediction. A language model can measure language likeness, and its performance affects the performance of speech recognition and machine translation. So far, various kinds of language models have been proposed.

As this language model, an N-gram language model is generally used. Since the learning method of the N-gram language model is a known technique, it is omitted here (for example, see Non-Patent Document 1). The N-gram language model can be easily learned if there is a learning text. Various learning methods have been proposed so far (see, for example, Non-Patent Document 2). The N-gram language model can directly model the word sequence of the learning data, and the current word w from the immediately preceding word N-1 word w _{i-N + 1} , ..., w _{i-1. i} prediction probability P (w _i | w _{i−N + 1} ,..., w _i−1 , θ _N-gram ). Note that θ _N-gram represents a model parameter of the N-gram language model.

As a model obtained by extending the N-gram model, there is a language model called a latent word language model (see Non-Patent Document 3, for example). The latent language model can consider words hidden behind observable words called latent words, and the model structure is a transition probability model that models a series of latent words and the output of words for each latent word. Divided into probabilistic models. Transition probability model is represented as a N-gram model for a potential word, potential word h _{i-N + 1} of the N-1 immediately preceding, ..., predicted probability P h _i-1 from the current potential word h _i (h _i | h _{i−N + 1} ,..., h _i−1 , θ _LWLM ). The output probability model is represented as a 1-gram model for the observation word for each latent word, and constitutes the prediction probability P (w _i | h _i , θ _LWLM ) for the observation word w _i of the latent word h _i . Θ _LWLM represents a model parameter of the latent language model. The advantage of the latent language model over the N-gram language model is robustness by considering latent words. It has been found that by considering latent words, it is possible to predict probability with high performance even from a small amount of learning data.

  On the other hand, there is an attempt to upgrade the language model by a method that does not use latent words. In recent years, a recurrent neural network language model has achieved high performance (for example, see Non-Patent Document 1).

The recurrent neural network language model can be learned from text data. In the probability prediction by the recurrent neural network, the immediately preceding word w _i-1 and the output s _i-1 of the intermediate layer in the immediately preceding network are input, and the prediction probability P (w _i | w _{i of the} current word w _i is input. ₋₁ , s _i−1 , θ _RNNLM ). Note that θ _RNNLM is a model parameter of the recurrent neural network. An obvious advantage of the recurrent neural network over the N-gram model is that the probability distribution can be modeled in consideration of the long distance context. In the N-gram model, only the immediately preceding n-1 word was considered as the context, but in the recurrent neural network language model, long _- distance context information is included in the output s _i-1 of the intermediate layer of the immediately preceding network. Therefore, more robust probability prediction can be realized.

Kenji Kita, “Language and Computation-4 Stochastic Language Model”, University of Tokyo Press, pp.57-62 S. F. Chen, and J. Goodman, “An Empirical Study of Smoothing techniques for language modeling”, Computer Speech & Language, vol.13, no.4, pp.359-383, 1999. K. Deschacht, J. D. Belder, and M-F. Moens, “The Latent Language Model”, Computer Speech and Language, vol.26, pp.384-409, 2012. Mikolov Tomas, Karafiat Martin, Burget Lukas, Cernocky Jan, Khudanpur Sanjeev: Recurrent neural network based language model, In: Proc Interspeech2010.

  Although the latent language model is described as consisting of a transition probability model and an output probability model, the transition probability model remains in the structure of the N-gram language model. The transition probability model models series information of latent words, but cannot model a long-distance latent word relationship. Therefore, the performance of probability prediction has been lowered.

  An object of the present invention is to provide a language model generation apparatus, method, and program for generating a language model with higher probability prediction performance than the conventional one.

In the language model generation device according to one aspect of the present invention, w ₁ , w ₂ ,..., W _L are words constituting the text data, and h ₁ , h ₂ ,..., H _L are w ₁ , w ₂ , …, W _L latent words, θ _LWLM is a model parameter of the latent language model, N is a predetermined positive integer, latent language language model learning is performed using text data, h ₁ , h ₂ , ... , h _L , latent word that generates probability distribution P (h _i | h _{i-N + 1} , ..., h _i-1 , θ _LWLM ) and probability distribution P (w _i | h _i , θ _LWLM ) Recurrent using the language model learning unit and s _i-1 as the output of the intermediate layer in the network immediately before h _i and θ _LWRNNLM as the model parameter of the recurrent neural network using h ₁ , h ₂ , ..., h _L Recurrent neural network learning unit that performs neural network learning and generates probability distribution P (h _i | h _i-1 , s _i-1 , θ _LWRNNLM ), probability distribution P (w _i | h _i , θ _LWLM ) and Probability distribution P Latent word recurrent neural network language model constructing unit that constructs a language model for calculating the probability of occurrence of an arbitrary word sequence using multiplication with (h _i | h _i-1 , s _i-1 , θ _LWRNNLM ) And.

  It is possible to generate a language model with higher probability prediction performance than before.

The block diagram for demonstrating the example of a language model production | generation apparatus. The flowchart for demonstrating the example of the language model production | generation method.

[Technical background]
The latent variable transition probability model in the latent language model is modeled by a recurrent neural network language model instead of the normal N-gram language model. Specifically, when learning an ordinary latent language model whose transition probability model is an N-gram model, a recurrent neural network language model is constructed using the predicted latent word sequence, and it is used as a transition probability model. To adopt. At this time, the output probability model uses the original latent language model. The model constructed in this way is called a latent word recurrent neural network language model.

  By introducing the latent word recurrent neural network language model, it is possible to construct a language model having a higher language prediction performance than a normal latent language language model. High recognition performance can be obtained by using this model for speech recognition, and high translation performance can be obtained by using it for machine translation.

[Embodiment]
An example embodiment of a language model generation apparatus and method for building a latent word recurrent neural network language model is described.

  The language model generation device includes, for example, a latent language language model learning unit 1, a recurrent neural network learning unit 2, and a latent word recurrent neural network language model configuration unit 3. The language model generation apparatus implements the language model generation method by performing the processing of each step illustrated in FIG.

  The input of the language model generation device is text data that understands word breaks, and the output of the language model generation device is a latent word recurrent neural network language model. Text data for which word breaks can be understood can be created from a text file without word breaks by using an arbitrary morphological analyzer. The detailed flow will be described below.

  Hereinafter, processing of each unit of the language model generation device will be described.

<Latent language model learning unit 1>
Input: Text data that understands word breaks Output: Information about latent language models, latent word sequences of learning data The latent language model learning unit 1 learns latent language models using text data that understands word breaks as learning data (Step S1). As a specific learning method, for example, an existing latent language model learning method described in Non-Patent Document 2 may be used.

The latent language model has two probability distributions, P (h _i | h _{i-N + 1} , ..., h _i-1 , θ _LWLM ) and P (w _i | h _i , θ _LWLM ). Has a probability distribution. Here, h _i is called a latent word and w _i is called an observation word. The latent word h _i is a latent variable in the latent language model, and the observed word w _i represents a word that actually appears in the text. P (h _i | h _{i-N + 1} ,…, h _i-1 , θ _LWLM ) is a general word N-gram language model form, P (w _i | h _i , θ _LWLM ) is a unigram language model It has become. Θ _LWLM represents a model parameter of the latent language model.

The learning of the latent language model is a problem of estimating the allocation of latent words for each word of the input learning text. That is, if there is a learning text (series of observation words) “w ₁ · w ₂ ... W _L ” (total number of words included in the learning text: L), “w ₁ ” “w ₂ ” ... “w It can be said that the latent words “h ₁ ”, “h ₂ ”... “H _L ” of each observation word of “ _L ” are estimated. If this assignment can be estimated, P (h _i | h _{i-N + 1} ,…, h _i− can be obtained by learning an N-gram language model for the latent word sequence “h ₁ · h ₂ ···· h _L ”. ₁ , θ _LWLM ), and by learning a unigram language model for “h ₁ → w ₁ ”, “h ₂ → w ₂ ” ... “h _L → w _L ”, P (w _i | h _i , θ _LWLM ) can be constructed. A specific latent word assignment can be estimated by a method called Gibbs sampling. Since Gibbs sampling is a known technique, the description thereof is omitted here.

The final output is a latent language model (specifically, two probability distributions P (h _i | h _{i−N + 1} ,..., H _i−1 , θ which are parameters of the latent language model) _LWLM ), P (w _i | h _i , θ _LWLM ) and latent word sequences of input learning data estimated at the time of modeling.

  The latent word language model is output to the latent word recurrent neural network language model configuration unit 3. The latent word sequence of the learning data is output to the recurrent neural network learning unit 2.

<Recurrent neural network learning unit 2>
Input: Latent word sequence of learning data Output: Recurrent neural network for latent words The recurrent neural network learning unit 2 generates a recurrent neural network from the latent word sequence of learning data obtained as an output of the latent language language model learning unit 1. Learning is performed (step S2). Normally, a recurrent neural network that learns from observed word sequences is learned from latent word sequences of learning data. The learning method itself is the same as when the observed word sequence is handled. That is, for example, by applying the method for learning the recurrent neural network from the observed word sequence described in Non-Patent Document 4 to the latent word sequence of the learning data instead of the observed word sequence, the recurrent neural network is Can learn.

Here, the recurrent neural network to be learned is a prediction of the current latent word h _i when the previous latent word h _i-1 and the output s _i-1 of the intermediate layer in the previous network are input. A probability distribution P (h _i | h _i−1 , s _i−1 , θ _LWRNNLM ) for the probability. Θ _LWRNNLM is a model parameter of the recurrent neural network for the latent word.

  The recurrent neural network for latent words generated by learning is output to the latent word recurrent neural network language model construction unit 3.

<Latent Word Recurrent Neural Network Language Model Configuration Unit 3>
Input: latent word language model, recurrent neural network for latent words Output: latent word recurrent neural network language model The latent word recurrent neural network language model configuration unit 3 includes a latent word language model and a recurrent neural network for latent words. A latent word recurrent neural network language model is constructed (step S3). Specifically, the parameter of P (w _i | h _i , θ _LWLM ) in the latent word language model and P (h _i | h _i-1 , s _i-1 , θ _LWRNNLM in the recurrent neural network for the latent word ) And a probability model in which the extracted parameters are paired. J is a set of possible values of h _i (i = 1, 2,..., L).

In this way, the latent word recurrent neural network language model constructing unit 3 performs the probability distribution P (w _i | h _i , θ _LWLM ) and the probability distribution P (h _i | h _i−1 , s _i−1 , θ _LWRNNLM ) And a language model for calculating the probability that an arbitrary word string appears.

[Modification]
The N-gram language model may be generated by approximation based on the latent word recurrent neural network language model. As a result, the model can be easily used for speech recognition and machine translation. The form of the N-gram language model is well-practical because it can be operated at high speed by speech recognition and machine translation.

  Therefore, the language model generation device may further include, for example, a pseudo learning text generation unit 4 and an N-gram language model generation unit 5.

<Pseudo-learning text generator 4>
Input: latent word recurrent neural network language model Output: pseudo learning text The pseudo learning text generation unit 4 generates pseudo learning text from the latent word recurrent neural network language model constructed by the latent word recurrent neural network language model configuration unit 3. Here, the purpose is to generate M words of pseudo-learning text “w ₁ · w ₂ ... · W _M ”. Basically, a latent word sequence “h ₁ , h ₂ ,..., H _M ” is first generated, and a pseudo learning text is generated therefrom. The parameter of P (h _i | h _i−1 , s _i−1 , θ _LWRNNLM ) is used to generate the latent word sequence. When generating a word from a latent word, the parameter of P (w _i | h _i , θ _LWLM ) is used.

First, an initial latent word h ₁ is randomly generated based on P (h ₁ | silS, s ₀ , θ _LWRNNLM ). Here, silS uses a leading symbol, and s ₀ uses a vector in which all elements are 0. If a random distribution is generated when a probability distribution is obtained, an algorithm called SampleOne is followed. The SampleOne algorithm is an algorithm that randomly samples one word from a certain discrete probability distribution, which will be described later. Once h ₁ is generated, decide which w ₁ to generate. This _obtains a probability distribution of P (w ₁ | h ₁ , θ _LWLM ), and generates based on the SampleOne algorithm based on this distribution. Next, in order to generate h ₂ , a probability distribution P (h ₂ | h ₁ , s ₁ , θ _LWRNNLM ) is obtained. H ₂ is generated by the SampleOne algorithm according to this probability distribution. After generating h ₂ , decide which w ₂ to generate. This obtains a probability distribution of P (w ₂ | h ₂ , h ₁ , θ _LWLM ). W ₂ is generated by the SampleOne algorithm according to this probability distribution. Next, h ₃ is generated based on a probability distribution of P (h ₃ | h ₂ , s ₂ , θ _LWRNNLM ), and the generation of... _Is repeated. By repeating this process until h _M is generated, the latent word sequence “h ₁ · h ₂ ... h _M ” and pseudo learning text “w ₁ · w ₂ ... w _M ” are generated. The value of M is determined manually. The larger this value, the more the pseudo-learning text that better represents the properties of the latent word recurrent neural network language model. This value should be equal to or greater than the number of words L included in the first learning text. If it is too small, performance will not be achieved.

Hereinafter, the SampleOne algorithm will be described.
Input: Probability distribution (multinomial distribution)
Output: Realized probability distribution
The SampleOne algorithm is an algorithm for determining one value at random from a probability distribution. In order to explain specifically, the case where P (h ₁ ) in the above example is an input will be treated.

P (h ₁ ) is in the form of a probability distribution called a multinomial distribution. Let J be a set of concrete realization values of h ₁ . J is automatically determined if a probability distribution is given. Specifically, the probability distribution that h is P (h ₁ ) is P (h ₁ = t ₁ ), P (h ₁ = t ₂ ),..., P (h ₁ = t _H ). Here, t ₁ , t ₂ ,..., T _H are specific realization values, and this set is J. At this time, P (h ₁ ) has the following properties.

At this time, SampleOne of h ₁ is based on a random number. Here, the random value is set to rand. P (h ₁ = t ₁ ), P (h ₁ = t ₂ ),..., P (h ₁ = t _H ) have specific numerical values. rand-P (h ₁ = t ₁ ), rand-P (h ₁ = t ₁ ) -P (h ₁ = t ₂ ), rand-P (h ₁ = t ₁ ) -P (h ₁ = t ₂ ) -P (h ₁ = t ₃ ), ... in order and output the value when that value is less than 0. For example,
rand-P (h ₁ = t ₁ )> 0
rand-P (h ₁ = t ₁ ) -P (h ₁ = t ₂ ) <0
If so, t ₂ is output. The SampleOne algorithm can be said to be a data sampling algorithm from an arbitrary multinomial distribution.

<N-gram language model generator 5>
Input: Pseudo learning text Output: Latent word recurrent neural network N-gram language model
The N-gram language model generation unit 5 counts the frequency of the combination of all N words in the learning text, forms an N-gram language model, and constructs a latent word recurrent neural network N-gram language model.

  In the case of speech recognition, generally N = 3 is often used. Since the learning method of the N-gram language model is a known technique described in Non-Patent Document 1, for example, it is omitted here. This makes it possible to construct an N-gram language model that inherits the properties of the recurrent neural network for latent words, and can be easily used for speech recognition and machine translation.

[Program and recording medium]
The processes described in the language model generation apparatus and method are not only executed in time series in the order described, but may also be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes or as necessary. .

  Further, when each process in the language model generation device is realized by a computer, the processing contents of the functions that the language model generation device should have are described by a program. Then, by executing this program on a computer, each process is realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used.

  Each processing means may be configured by executing a predetermined program on a computer, or at least a part of these processing contents may be realized by hardware.

  Needless to say, other modifications are possible without departing from the spirit of the present invention.

DESCRIPTION OF SYMBOLS 1 Latent language language model learning part 2 Recurrent neural network learning part 3 Latent word recurrent neural network language model structure part 4 Pseudo learning text generation part 5 Language model generation part

Claims (4)

w ₁ , w ₂ , ..., w _L are the words constituting the text data, h ₁ , h ₂ , ..., h _L are the latent words of w ₁ , w ₂ , ..., w _L respectively, and θ _LWLM is The latent language model learning is performed using the above text data with the model parameters of the latent language model, N being a predetermined positive integer, and h ₁ , h ₂ ,..., H _L and the probability distribution P (h _i | h _{i-N + 1} , ..., h _i-1 , θ _LWLM ) and probability distribution P (w _i | h _i , θ _LWLM ),
s _i-1 is the output of the intermediate layer in the network immediately before h _i , θ _LWRNNLM is the model parameter of the recurrent neural network, and h ₁ , h ₂ , ..., h _L is used to perform recurrent neural network learning, A recurrent neural network learning unit that generates a probability distribution P (h _i | h _i-1 , s _i-1 , θ _LWRNNLM );
Calculate the probability that an arbitrary word string appears using multiplication of probability distribution P (w _i | h _i , θ _LWLM ) and probability distribution P (h _i | h _i-1 , s _i-1 , θ _LWRNNLM ) A latent word recurrent neural network language model component that constitutes a language model for
Language model generation device including The language model generation device according to claim 1,
A pseudo-learning text generation unit that generates pseudo-learning text based on the configured language model;
An N-gram language model generation unit that generates an N-gram language model based on the pseudo-learning text;
A language model generation apparatus further comprising: The latent language model learning unit designates w ₁ , w ₂ ,..., W _L as words constituting the text data, and h ₁ , h ₂ ,..., H _L as w ₁ , w _{2 ,.} the latent word, the model parameters of the potential language model theta _LwLm, the N as the predetermined positive integer, performs latent language model learning using the text _{data, h 1, h 2, ...} , h L If the probability distribution _{P (h i | h i-} N + 1, ..., h i-1, θ LWLM) and the probability distribution _{P (w i | h i,} θ LWLM) and potential language model learning to generate a Steps,
The recurrent neural network learning unit uses s _i-1 as the output of the intermediate layer in the network immediately before h _i , θ _LWRNNLM as the model parameter of the recurrent neural network, and h ₁ , h ₂ , ..., h _L A recurrent neural network learning step for performing recurrent neural network learning and generating a probability distribution P (h _i | h _i-1 , s _i-1 , θ _LWRNNLM );
Latent word recurrent neural network language model component uses multiplication of probability distribution P (w _i | h _i , θ _LWLM ) and probability distribution P (h _i | h _i-1 , s _i-1 , θ _LWRNNLM ) A latent word recurrent neural network language model constructing step for constructing a language model for calculating the probability of occurrence of an arbitrary word string;
Language model generation method including   The program for functioning a computer as each part of the language model production | generation apparatus of Claim 1 or 2.

Images