JP2017016384A - Mixed coefficient parameter learning device, mixed occurrence probability calculation device, and programs thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a mixed coefficient parameter learning device for improving the accuracy of a mixed occurrence probability.SOLUTION: A mixed coefficient parameter learning device 30 includes: first occurrence probability request means 312 for requesting a hidden layer vector and an occurrence probability from a neural network language model calculation device 10; second occurrence probability input means 313 for requesting an occurrence probability from an other language model calculation device 20; first mixed coefficient calculation means 314 for calculating a mixed coefficient from the hidden layer vector; mapping vector update means 315 for updating a mapping vector by a probabilistic gradient descent method; update rate reduction means 317 for reducing an update rate; and termination condition determination means 316 for causing the mapping vector update means 315 to update the mapping vector until satisfying a termination condition.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a mixing coefficient parameter learning device that learns parameters necessary for calculating a mixing coefficient, a mixed occurrence probability calculation device that calculates a mixed occurrence probability between a neural network probability model and another probability model, and a program thereof. .

Statistical language model (hereinafter, "language model") and comprises means for calculating in a language or domain, the probability word sequence _w 1 _w 2 ... _{w n} are occurring _{p (w 1 w 2 ... w} n), and , Defined as a list of various statistics necessary for calculation by the means. Probabilistic modeling of language occurrence by language model is one of the most basic techniques of statistical natural language processing, and is used in various natural language processing techniques such as speech recognition and machine translation. .

The language model is a probability model of occurrence of an expression (word sequence) in a specific language or a specific field of the language, and is generally a predetermined language or a corpus of the language in the field. To learn from.
A corpus is an example of a word sequence observed in a certain language or a specific field of the language.
In _{addition, w 1, w 2, w} n represents the word.

The occurrence probability p (w ₁ w ₂ ... W _n ) of a word sequence is generally the product of the probabilities that each word of the sequence occurs with the previous word system as the previous context, that is, p (w ₁ ) × It is modeled as p (w ₂ | w ₁ ) × p (w ₃ | w ₁ w ₂ ) ×... × p (w _n | w ₁ w ₂ ... w _n−1 ). That is, it can be said that the language model is a prediction model for occurrence of the next word under the condition given the previous context.

  The most common implementation method of the language model is an n-gram language model. This n-gram language model restricts the preceding context as the condition to the nearest n-1 words (where n is an integer equal to or greater than 1), and from the learning corpus, the previous context is a string of n-1 words. The next word occurrence probability under each previous context condition is estimated based on the result of collecting the next word occurrence frequency for each difference.

  In the n-gram language model, it is necessary to refer to a long previous context (using a large value of n) in order to accurately estimate the occurrence probability of the next word. Also, in the n-gram language model, it is necessary to collect sufficient examples for each previous context, but the longer the previous context, the more the difference in the previous context increases, so it is very large to improve accuracy. A learning corpus needs to be prepared.

  In recent years, a language model realization method using a neural network has been proposed for this n-gram language model. This method uses a neural network to learn a mapping to a word expression vector with a fixed dimension and each dimension as a real value as representing each word, and the word expression corresponding to each word in the word sequence as the previous context A combination of vectors is used.

For example, NNLM (Neural Network Language Model) described in Non-Patent Document 1 constructs a neural network as shown in FIG. Hereinafter, regarding the language model, only a limited number of | V | types of words are handled, and each word is expressed as a numerical value of 1 to | V |. At this time, it is assumed that a special word representing the beginning of a sentence is included in the | V | type words. Here, it is assumed that a word expression vector having a predetermined fixed dimension number m corresponding to each word w is C (w). Also, regarding the occurrence of the word sequence w ₁ w ₂ ... W _t , an n × m-dimensional input vector x (t) obtained by concatenating n−1 word expression vectors representing n−1 previous contexts of the word w _t. = [C (w _{t−n + 1} ),..., C (w _t−2 ), C (w _t−1 )] to Hx (t) is a linear mapping from a vector having a predetermined eigendimension number h.
If the length of the previous context is less than n−1 (ie, t <n), the input vector x (t (t) is obtained by supplementing the word w _{1 with} the word representing the beginning of the sentence before the word w _1. ).

Further, a linear mapping from a hidden layer vector z (t) obtained by converting each dimension of the linear mapping Hx (t) with a nonlinear function f (for example, a hyperbolic tangent function tanh) to a | V | -dimensional vector y (t) is expressed as Uz ( t).
Also, a | V | -dimensional vector obtained by converting each dimension of y (t) with the function shown in Expression (1) is set as an output vector p (t). In this case, the probability that the next word is w _t is defined as in the following equations (1) to (3) (where y _i is the i-dimensional value of y).
Further, (t) of the input vector x (t) means the input vector x related to the occurrence probability of the next word w _t following the previous context w ₁ w ₂ ... W _t−1 (the same applies to other vectors). .
Further, “◯” in FIG. 5 represents a vector element.

n, m, h are set in advance, and for each word w _t in the learning corpus, the previous contexts w _{t−n + 1} ,..., w _t−2 , w _t−1 are input to the neural network to generate the next word occurrence. While outputting the probability distribution (forward propagation) and propagating the cross-entropy error between the output vector and the correct vector to the neural network in the backward direction, the word expression vector C as shown in the following equations (4) to (6): The weight H from the input layer to the hidden layer and the weight U from the hidden layer to the output layer are updated by the stochastic gradient descent method (where ε is the update rate). Learning is realized by repeating this several times in the entire learning corpus.
The correct vector is a vector in which the occurrence probability of the word w _t is 1, and the occurrence probabilities of other words are 0.

  As a result of learning the word expression vector C, similar words are mapped to the nearest word expression vector, and as a result of learning H, similar words are mapped to the nearest hidden layer vector, so when learning from a small learning corpus But you can get high accuracy.

Further, as a method different from NNLM, RNNLM (Recurrent Neural Network Language Model) described in Non-Patent Document 2 has been proposed. As described above, in NNLM, word predetermined (n-1) for the word _{w t w t-n + 1} , ..., w t-2, w hiding from _t-1 represents a previous context layer vector z (t ). On the other hand, this RNNLM, as in FIG. 6, a hidden layer vector z (t), 1 previous word _{w t-1} and the hidden layer vector z representing the previous context for this word _{w t-1 (t-1} ) This allows RNNLM to predict the occurrence of the next word reflecting a long previous context without giving an explicit previous context length n.

Generally, these neural network language models are used in combination with other language models (for example, n-gram language models). Specifically, the occurrence probability by the neural network language model is p _N , the occurrence probability by another language model is p _O , and the mixing ratio is λ. In this case, as shown in the following formula (7), a mixture of the occurrence probabilities p _N and p _O of the two language models at a ratio of λ: 1−λ is calculated as the mixed occurrence probability p.

  In Equation (7), λ represents a mixing coefficient. In general, the mixed coefficient λ is prepared by preparing a learned bilingual model, determining a value with the highest accuracy with respect to a separately prepared test corpus, and using the determined value in a fixed manner.

There are the following reasons for mixing the occurrence probabilities in this way.
1) Although there is no general method for estimating the occurrence probability of a word (unknown word) that did not appear in the learning corpus in the neural network language model, an appropriate occurrence probability can be assigned to the unknown word in the n-gram language model.
2) Since the neural network language model has a very large amount of calculation for learning compared to the n-gram language model, the neural network language model learns using a learning corpus specialized for small domains in the neural network language model. It is realistic to combine with an n-gram language model learned from a wide range of large-scale learning corpora.

A Neural Probabilistic Language Model, Yoshua Bengio et.al, Journal of Machine Learning Research 3, (2003), 1137-1155 Static Language Model based on Neural Network, Tomas Mikolov

  However, since the neural network language model uses a fixed mixing coefficient regardless of the previous context, there is a problem that the accuracy of the mixed occurrence probability is lowered. For example, in order to accurately predict the word appearing next to the previous context “I am”, it is necessary to learn with a huge learning corpus. On the other hand, the word that appears next to the previous context “I went to the mountain” can only be about “ta” and “te”, and can be accurately predicted even by learning with a small learning corpus. In other words, the neural network language model can improve the accuracy of the mixed occurrence probability by using different mixing coefficients depending on the previous context.

  This invention makes it a subject to provide the mixing coefficient parameter learning apparatus, the mixing occurrence probability calculation apparatus, and these programs which improve the precision of mixing occurrence probability in view of an above described subject.

  In view of the problems described above, the mixing coefficient parameter learning device according to the present invention determines the occurrence probability of the next element with respect to the previous element sequence obtained by each of the neural network probability model and the other probability models other than the neural network probability model. A mixing coefficient parameter learning device for learning parameters necessary for calculating a mixing coefficient when mixing, a first occurrence probability input means, a second occurrence probability input means, a first mixing coefficient calculation means, a mapping vector The update unit, the update rate reduction unit, and the end condition determination unit are provided.

  According to such a configuration, the mixed coefficient parameter learning apparatus receives the hidden layer vector of the neural network probability model and the occurrence probability obtained by the neural network probability model by the first occurrence probability input means.

  That is, when a previous element sequence is input to a learned neural network probability model, a generalized expression of the previous element is obtained as a hidden layer vector of the neural network probability model. Therefore, if a mapping vector from a hidden layer vector of a learned neural network probability model to a mixture coefficient is learned, a mixture coefficient corresponding to the previous element sequence can be obtained.

In the mixing coefficient parameter learning device, the occurrence probability obtained by the other probability model is input by the second occurrence probability input means.
The mixing coefficient parameter learning device linearly maps the hidden layer vector to a real-valued scalar according to a preset mapping vector by the first mixing coefficient calculation means, and nonlinearly converts the real-valued scalar by a sigmoid function. And calculating the mixing coefficient.

  The mixing coefficient parameter learning device uses the mapping vector update means to generate the probabilities of occurrence of the neural network probability model and the other probability models, the mixing coefficient, and a probabilistic value using a preset update rate. The mapping vector as the parameter is updated by a gradient descent method.

The mixing coefficient parameter learning device decreases the update rate according to a preset update rate decrease rule by the update rate decrease means.
The mixing coefficient parameter learning device determines whether or not a predetermined end condition is satisfied by an end condition determining unit, and the mapping vector updating unit reduces the update rate to the mapping vector update unit until the end condition is satisfied. Update the vector. For example, the termination condition is a condition that the occurrence probability does not change even if the update rate is decreased.

  Further, in view of the above-described problems, the mixed occurrence probability calculation device according to the present invention provides the occurrence of the next element with respect to the previous element sequence obtained by each of the neural network probability model and another probability model other than the neural network probability model. A mixed occurrence probability calculating device for calculating a mixed occurrence probability in which probabilities are mixed, comprising: a third occurrence probability input means; a fourth occurrence probability input means; a second mixing coefficient calculation means; and a mixed occurrence probability calculation means. It was set as the structure provided.

According to such a configuration, the mixed occurrence probability calculation device receives the hidden layer vector of the neural network and the occurrence probability obtained by the neural network probability model by the third occurrence probability input unit.
In the mixed occurrence probability calculation device, the occurrence probability obtained by the other probability model is input by the fourth occurrence probability input means.

  The mixed occurrence probability calculating device linearly maps the hidden layer vector to a real-valued scalar with the mapping vector learned by the mixed-coefficient parameter learning device according to the present invention by the second mixing coefficient calculating means, and converts the real-valued scalar into the real-valued scalar. By performing non-linear transformation with a sigmoid function, a mixing coefficient corresponding to the previous element series is calculated.

  The mixed occurrence probability calculating device mixes the occurrence probabilities of the next element obtained by the neural network probability model and the other probability models by using the mixing coefficient corresponding to the previous element series by the mixed occurrence probability calculating means. Thus, the mixed occurrence probability is calculated.

The present invention has the following excellent effects.
According to the present invention, a mapping vector from a hidden layer vector of a learned neural network probability model to a mixture coefficient is learned. Thereby, since the mixing coefficient according to the previous element series is obtained, the accuracy of the mixing occurrence probability can be improved.

It is explanatory drawing explaining the learning procedure of the mapping vector in this invention. It is a block diagram which shows the structure of the mixed occurrence probability calculation system which concerns on embodiment of this invention. It is a flowchart which shows operation | movement of the mixing coefficient parameter calculation apparatus of FIG. It is a flowchart which shows operation | movement of the mixed occurrence probability calculation apparatus of FIG. It is explanatory drawing explaining the process outline | summary of the conventional NNLM. It is explanatory drawing explaining the process outline | summary of the conventional RNNLM.

Hereinafter, the mixed occurrence probability calculation system 1 according to the embodiment of the present invention will be described.
First, the mapping vector learning procedure and the mixed occurrence probability calculation procedure will be described with reference to FIG. Thereafter, the configuration of the mixed occurrence probability calculation system 1 will be described.

  Here, it is assumed that there is a neural network language model calculation apparatus 10 that has an h-dimensional hidden layer vector representing the previous context (previous element series) and calculates the occurrence probability of the next word as a mapping from the hidden layer vector. . The neural network language model computing device 10 assumes that each statistic necessary for calculating the occurrence probability has been learned by a learning corpus or the like.

Further, it is assumed that there is another language model calculation device 20 (FIG. 2) that estimates the occurrence probability of the next word using another language model such as an n-gram language model. In this other language model calculation device 20, it is assumed that each statistic necessary for calculating the occurrence probability has been learned by a learning corpus or the like.
The learning corpus used for learning each statistic of the other language model calculation device 20 needs to be the same as the learning corpus used for learning each statistic of the neural network language model calculation device 10. There is no.

<Mapping vector learning procedure>
When the previous context is input to the neural network language model calculation device 10, a generalized expression of the previous context is obtained as a hidden layer vector. Therefore, in the present invention, as shown in FIG. 1, a process for mapping from the hidden layer vector z to the mixing coefficient λ is added to the neural network language model calculation apparatus 10 and the mapping vector S is learned.

Specifically, by adding the processing of the following equations (8) and (9) to the processing by the neural network language model, the occurrence probability of the next word w _t when the previous context... W _t−1 is given. The mixing coefficient λ (t) necessary for calculating is calculated.

Equation (8) represents a nonlinear conversion by a sigmoid function from a real-valued scalar s (t) to a mixing coefficient λ (t).
Equation (9) represents a linear mapping Sz (t) from a hidden layer vector z (t) to a real-valued scalar s (t). In Expression (9), b represents the bias value.

Learning of the mapping vector S and the bias value b, for each word _{w t} in some learning corpus, the occurrence probability _{p N (w t | ... w} t-1) and the occurrence probability _{p O (w t | ... w} t-1 ) And the mixing coefficient λ (t) defined by the equation (8) so that the mixed occurrence probability p (w _t |... W _t−1 ) is maximized according to the following equation (10). The following steps 1 to 3 are performed.

Note that the learning corpus may be the same as or different from the corpus used to learn the neural network language model or another language model.
The occurrence probability p _N (w _t |... W _t-1 ) is the occurrence probability of the next word w _t obtained by giving the previous context... W _t-1 to the neural network language model.
The occurrence probability p _O (w _t |... W _t-1 ) is the occurrence probability of the next word w _t obtained by giving the previous context ... w _t-1 to another language model.

Procedure 1. An update rate ε is set in advance.
Procedure 2. The following processes (a) to (c) are executed for each word w _t in the learning corpus.
(A) The forward context is input to the neural network language model arithmetic unit 10 as appropriate (when the previous context is fixed as in NNML, divided by its length) ... w _t-1 to perform forward propagation. Thus, the occurrence probability p _N (w _t |... W _t−1 ) of the hidden layer vector z (t) and the next word w _t is obtained. Similarly, occurrence probabilities p _O (w _t |... W _t−1 ) of other language models are obtained.

(B) The mixing coefficient λ (t) is obtained by performing forward propagation from the hidden layer. That is, the mixing coefficient λ (t) is obtained using the equations (8) and (9).

(C) Update the mapping vector S by the stochastic gradient descent method. That is, each dimension S _i of the h-dimensional mapping vector S is updated by a probabilistic gradient descent method in which the mixing coefficient λ (t) is reflected as in the following expressions (11) and (12).

  Further, the bias value b in the equation (9) is also a learning target. For this reason, the bias value b is also updated as in the following equations (13) and (14).

In the procedure 2 (c), when the mapping vector S is updated, regularization may be performed as shown in Expression (15) as an example in order to prevent over-learning by the neural network language model. Further, the bias value b may be regularized as with the mapping vector S.
In equation (15), β represents a regularization coefficient. For example, the regularization coefficient β is set to a value smaller than the update rate ε.

Procedure 3. The process returns to step 2 and repeats until a predetermined end condition is met. At this time, the update rate ε is decreased according to a predetermined update rate decrease rule.
Details of the termination condition and the update rate reduction rule will be described later.

<Procedure for calculating mixed occurrence probability>
The occurrence probability of the next word w _t is calculated by the following procedure 4 to procedure 6 using the learning result described above.

Procedure 4. Hidden layer by inputting forward context ... w _t-1 to the neural network language model arithmetic unit as appropriate (when the previous context is fixed like NNML, divided by its length) and performing forward propagation The occurrence probability p _N (w _t |... W _t−1 ) of the vector z (t) and the next word w _t is obtained. Similarly, occurrence probabilities p _O (w _t |... W _t−1 ) of other language models are obtained.
This procedure 4 is the same process as the learning procedure 2 (a) of the mapping vector S.

Procedure 5. By performing forward propagation from the hidden layer, the mixing coefficient λ (t) is obtained. That is, the learned mapping vector S and the bias value b are substituted into equation (9) to obtain the mixing coefficient λ (t). This procedure 5 is the same process as the learning procedure 2 (b) of the mapping vector S.
Procedure 6. The mixed occurrence probability p (w _t |... W _t−1 ) is obtained using Expression (16).

  With reference to FIG. 2, the structure of the mixed occurrence probability calculation system 1 according to the embodiment of the present invention will be described.

Mixed occurrence probability calculation system 1 is for calculating the occurrence probability p _N calculated in the neural network language model, the mixing probability P obtained by mixing the occurrence probability p _O obtained in other language models. As shown in FIG. 2, the mixed occurrence probability calculation system 1 includes a neural network language model calculation device 10, another language model calculation device 20, a mixing coefficient parameter learning device 30, and a mixed occurrence probability calculation device 40.

[Configuration of Neural Network Language Model Calculation Device]
The neural network language model computing device 10 computes the occurrence probability p _N using a neural network language model. For example, the neural network language model calculation apparatus 10 can use a neural network using a hidden layer (for example, NNLM, RNNLM).

Specifically, when the previous context w ₁ , w ₂ ,..., W _t−1 is input, the neural network language model calculation apparatus 10 receives the occurrence probability p _N (w of the word w _t following the previous context. _t |... w _t−1 ) is calculated. The neural network language model calculation device 10 stores the hidden layer vector z (t) calculated from the input layer vector x (t) of the neural network when calculating the output layer vector p (t) of the neural network, The stored hidden layer vector z (t) is output to the mixture coefficient parameter learning device 30 or the mixture occurrence probability calculation device 40.

In the case of NNLM, the neural network language model calculation apparatus 10 limits the length of the previous context that can be referred to from the end of the previous context to a predetermined number of words n−1 (n is an integer of 1 or more).
For example, prior context _{w 1,} w 2, _..., when the _{w t-1,} referable prior context _{w t-n + 1, w} t-n + 1, ..., a _{w t-1.}
The neural network language model calculation apparatus 10 stores the word expression vector C (w) corresponding to each word of the input previous context, and the previous contexts w _{t−n + 1} , w _{t−n + 1} ,. , W _t−1 are input, the word expression vectors C (w) corresponding to the respective words are connected and set to the input layer vector x (t) of the neural network. Then, the neural network language model calculation device 10 performs forward propagation and calculates the hidden layer vector z (t) and the output layer vector p (t) of the neural network.
The output layer vector p (t) is a vector having a number of different dimensions of the word, and the value of each dimension of the vector represents the occurrence probability of the word corresponding to that dimension. It should be noted that the hidden layer vector z (t) is referred to as a "pre-context _{w 1,} w 2, _..., hidden layer representation of _{w t-1".}

In the case of RNNLM, the neural network language model calculation apparatus 10 stores a hidden layer vector z having the word series w ₁ , w ₂ ,. In the initial state, the hidden layer vector z is set to an initial value unique to the neural network language model arithmetic unit 10.
When the i-th word w _i is input, the neural network language model calculation device 10 inputs a vector in which only the dimension corresponding to the word w _i is 1 and all other dimensions are 0 to the input layer x (i). Set. Then, the neural network language model calculation device 10 performs forward propagation from the input layer x (i) and the stored hidden layer vector z (i) of the previous input, and the neural network hidden layer vector z (i + 1) and An output layer vector p (i + 1) is calculated. When the input to the words w ₁ , w ₂ ,..., W _t−1 and the forward propagation are finished, the hidden layer vector z (t) is “previous context w ₁ , w ₂ when using the NNLM”. ,..., Hidden layer representation of w _t−1 ”. In other words, neural network language model calculating unit 10 calculates the output layer vector p (t) by forward propagating Using Hidden layer vector z (t), obtaining the occurrence probability p _N of the next word.

It is assumed that the neural network language model calculation device 10 has already been learned (the mapping matrix for forward propagation has been set to an appropriate value by the learning data), and the learning result is stored.
Further, since the neural network language model calculation device 10 has a general configuration, further description thereof is omitted.

[Configuration of other language model arithmetic unit]
The other language model calculation device 20 calculates the occurrence probability p _O using a language model other than the neural network language model (for example, an n-gram language model). Specifically, when the previous context w ₁ , w ₂ ,..., W _t−1 is input, the other language model calculation apparatus 20 receives the occurrence probability p _O (arbitrary word w _t following the previous context. w _t |... w _t−1 ) is calculated and output.

In the other language model calculation device 20, various parameters necessary for calculating the probability value are set in advance.
Further, since the other language model calculation device 20 has a general configuration, further description is omitted.

[Configuration of mixing coefficient parameter learning device]
The mixing coefficient parameter learning device 30 learns parameters necessary for calculating the mixing coefficient λ when the occurrence probabilities p _N and p _O obtained in the neural network language model and other language models are mixed. .

  As shown in FIG. 2, the mixing coefficient parameter learning apparatus 30 includes a mixing coefficient parameter storage unit 301, a learning parameter storage unit 302, a learning data storage unit 303, a mixing coefficient storage unit 304, an initialization unit 311, 1 occurrence probability request means (first occurrence probability input means) 312, second occurrence probability request means (second occurrence probability input means) 313, first mixing coefficient calculation means 314, mapping vector update means 315, and end Condition determining means 316 and update rate reducing means 317 are provided.

  The mixing coefficient parameter storage unit 301 is a storage unit such as a memory or a hard disk that stores a mixing coefficient parameter necessary for calculating the mixing coefficient λ. Specifically, the mixing coefficient parameter storage unit 301 stores mixing coefficient parameters such as the mapping vector S and the bias value b. This mapping vector S has the same dimensionality as the dimensionality h of the hidden layer vector z of the neural network.

  The learning parameter storage unit 302 is a storage unit such as a memory or a hard disk that stores parameters necessary for learning the mapping vector S. Specifically, the learning parameter storage unit 302 stores learning parameters such as the update rate ε and the regularization coefficient β.

The learning data storage unit 303 is a storage unit such as a memory or a hard disk that stores a word string that is learning data necessary for learning the mapping vector S. This learning data may not be the same as that used for learning in the neural network language model calculation device 10 and the other language model calculation device 20.
The mixing coefficient storage unit 304 is a storage unit such as a memory or a hard disk that stores the mixing coefficient λ.

  The initialization unit 311 initializes the mixing coefficient parameter and the learning parameter. Specifically, the initialization unit 311 initializes the value of each dimension of the mapping vector S of the mixing coefficient parameter storage unit 301 and the bias value b with random numbers. The initialization unit 311 initializes the update rate ε and the regularization coefficient β in the learning parameter storage unit 302 with preset values.

The first probability requesting means 312, by outputting the previous context of learning data storage unit 303 to the neural network language model calculating unit 10, and requests the hidden layer vector z and probability p _N. In response to this request, the first occurrence probability requesting unit 312 receives the hidden layer vector z and the occurrence probability p _N from the neural network language model calculation device 10. The first probability request means 312 outputs the input hidden layers vector z and probability p _N to the first mixing coefficient calculation means 314 and the mapping vector updating means 315.

The second occurrence probability requesting unit 313 requests the occurrence probability p _O by outputting the previous context of the learning data storage unit 303 to the other language model calculation device 20. Here, the second occurrence probability requesting unit 313 outputs the same previous context as that of the first occurrence probability requesting unit 312 to the other language model calculation device 20. In response to this request, the second occurrence probability request means 313 receives the occurrence probability p _O from the other language model calculation device 20. Then, the second occurrence probability request unit 313 outputs the input occurrence probability p _O to the mapping vector update unit 315.

  The first mixing coefficient calculating unit 314 uses the expression (9) to convert the hidden layer vector z input from the first occurrence probability requesting unit 312 into a real-valued scalar s by using the mapping vector S of the mixing coefficient parameter storage unit 301. Is a linear mapping. The first mixing coefficient calculation unit 314 calculates the mixing coefficient λ by nonlinearly converting the real-valued scalar s using a sigmoid function using Equation (8). Then, the first mixing coefficient calculation unit 314 stores the calculated mixing coefficient λ in the mixing coefficient storage unit 304.

The mapping vector update unit 315 includes an occurrence probability p _N from the first occurrence probability request unit 312, an occurrence probability p _O from the second occurrence probability request unit 313, a mixing coefficient λ of the mixing coefficient storage unit 304, and a learning parameter storage. The mapping vector S of the mixing coefficient storage unit 304 is updated by a probabilistic gradient descent method using the update rate ε of the unit 302. That is, the mapping vector update unit 315 updates the mapping vector S using the stochastic gradient descent method expressed by the equations (11) and (12).

  The end condition determining unit 316 determines whether or not a preset end condition is satisfied, and the map vector updating unit 315 is updated with an update rate ε decreased by an update rate decreasing unit 317 described later until the end condition is satisfied. The mapping vector S is updated. For example, the end condition determination unit 316 determines that the end condition is satisfied when the update rate ε is decreased by a preset number of times and the value of the mixed occurrence probability p does not change.

Here, when the end condition is not satisfied, the end condition determining unit 316 instructs the update rate reducing unit 317 to decrease the update rate ε. Thereafter, the end condition determination unit 316 instructs the first occurrence probability request unit 312, the second occurrence probability request unit 313, the first mixing coefficient calculation unit 314, and the mapping vector update unit 315 to re-execute processing.
On the other hand, when the end condition is satisfied, the end condition determining unit 316 ends the process.
In FIG. 2, the command signal from the end condition determination unit 316 is illustrated by a broken line.

  The update rate reduction means 317 reduces the update rate ε of the learning parameter storage means 302 as necessary according to a preset update rate reduction rule. For example, the update rate reduction rule includes a rule of subtracting a preset value from the value of the update rate ε.

[Configuration of mixed occurrence probability calculation device]
The mixed occurrence probability calculation device 40 calculates a mixed occurrence probability p obtained by mixing the occurrence probabilities p _N and p _O obtained in the neural network language model and other probability models, respectively. As shown in FIG. 2, the mixed occurrence probability calculating device 40 includes a target data storage unit 401, a mixed occurrence probability storage unit 402, a third occurrence probability requesting unit (third occurrence probability input unit) 411, and a fourth occurrence probability. Request means (fourth occurrence probability input means) 412, second mixing coefficient calculation means 413, and mixed occurrence probability calculation means 414 are provided.

The target data storage unit 401 is a storage unit such as a memory or a hard disk that stores a word string representing a previous context and a next word that are targets of calculation of the mixed occurrence probability p. The word string in the target data storage unit 401 is different from the word string in the learning data storage unit 303.
The mixed occurrence probability storage unit 402 is a storage unit such as a memory or a hard disk that stores the mixed occurrence probability p.

Third probability requesting unit 411, by outputting the previous context object data storage means 401 in the neural network language model calculating unit 10, and requests the hidden layer vector z and probability p _N. In response to this request, the third occurrence probability requesting means 411 receives the hidden layer vector z and the occurrence probability p _N from the neural network language model calculation device 10. The third probability requesting unit 411 outputs the input hidden layers vector z and probability p _N to the second mixing coefficient calculation means 413, and mixtures occurrence probability calculating unit 414.

The fourth occurrence probability requesting unit 412 requests the occurrence probability p _O by outputting the previous context of the target data storage unit 401 to the other language model calculation device 20. Here, the fourth occurrence probability requesting unit 412 outputs the same previous context as that of the third occurrence probability requesting unit 411 to the other language model calculation device 20. In response to this request, the fourth occurrence probability request means 412 receives the occurrence probability p _O from the other language model calculation device 20. Then, the fourth occurrence probability requesting means 412 outputs the input occurrence probability p _O to the mixed occurrence probability calculating means 414.

  The second mixing coefficient calculation unit 413 uses the expression (9) to calculate the hidden layer vector z input from the third occurrence probability requesting unit 411 using the mapping vector S of the mixing coefficient parameter storage unit 301 as a real-valued scalar s. Is a linear mapping. Further, the second mixing coefficient calculation unit 413 calculates the mixing coefficient λ by nonlinearly transforming the real-valued scalar s with a sigmoid function using Expression (8). Then, the second mixing coefficient calculation unit 413 stores the calculated mixing coefficient in the mixing coefficient storage unit 304.

The mixed occurrence probability calculation means 414 uses the mixing coefficient λ of the mixing coefficient storage means 304 and the occurrence probability p _N input from the third occurrence probability request means 411 and the occurrence input input from the fourth occurrence probability request means 412. By mixing the probability p _O , the mixed occurrence probability p is calculated. Then, the mixed occurrence probability calculating unit 414 stores the calculated mixed occurrence probability p in the mixed occurrence probability storage unit 402.

[Operation of mixing coefficient parameter learning device]
The operation of the mixing coefficient parameter learning device 30 will be described with reference to FIG. 3 (see FIG. 2 as appropriate).

The mixing coefficient parameter learning device 30 initializes the mixing coefficient parameters such as the mapping vector S and the bias value b by the initialization unit 311 (step S1).
The mixing coefficient parameter learning device 30 initializes learning parameters such as the update rate ε and the regularization coefficient β by the initialization unit 311 (step S2).
The mixing coefficient parameter learning device 30 initializes the value of the counter i to 1 (step S3).

Mixing coefficient parameter learning unit 30, the first probability requesting means 312, a word string _w 1 of the learning data storage unit _303, w 2, ..., _w of _N, i-1 or word string _w 1 from the head, w ₂ ,..., w _i−1 are output to the neural network language model arithmetic unit 10 as the previous context.
The mixing coefficient parameter learning device 30 uses the first occurrence probability requesting unit 312 to generate the occurrence probability p _N (w _i | w ₁ w of the hidden layer vector z (i) and the next word w _i from the neural network language model calculation device 10. ₂ ... W _i-1 ) is input (step S4).

The mixed coefficient parameter learning device 30 outputs the same previous contexts w ₁ , w ₂ ,..., W _i−1 as in step S 4 to the other language model calculation device 20 by the second occurrence probability requesting unit 313.
The mixed coefficient parameter learning device 30 receives the occurrence probability p _O (w _i | w ₁ w ₂ ... W _i-1 ) of the next word w _i from the other language model calculation device 20 by the second occurrence probability requesting unit 313. (Step S5).

  The mixing coefficient parameter learning device 30 performs mixing according to the equations (8) and (9) using the hidden layer vector z (i) and the mapping vector S input by the first mixing coefficient calculation unit 314 in step S4. The coefficient λ (i) is calculated (step S6).

The mixing coefficient parameter learning device 30 uses the mapping vector update unit 315 to generate the occurrence probability p _N (w _i | w ₁ w ₂ ... W _i−1 ) input in step S4 and the occurrence probability p input in step S5. _{Using O} (w _i | w ₁ w ₂ ... W _i−1 ), the mixing coefficient λ (i) calculated in step S6, and the update rate ε, the mapping vector is expressed by equation (11) and equation (12). S is updated (step S7).

The mixing coefficient parameter learning device 30 increments the counter i (step S8).
The mixing coefficient parameter learning device 30 determines whether or not the counter i is equal to or less than the maximum number N of words (step S9).
When the counter i is equal to or less than the maximum number N of words (Yes in step S9), the mixing coefficient parameter learning device 30 returns to the process of step S4.

If the counter i is not equal to or less than the maximum number N of words (No in step S9), the mixture coefficient parameter learning device 30 determines whether or not the end condition is satisfied by the end condition determining unit 316 (step S10).
If the end condition is satisfied (Yes in step S10), the mixing coefficient parameter learning device 30 ends the process.

  When the termination condition is not satisfied (No in step S10), the mixing coefficient parameter learning device 30 decreases the update rate ε as necessary according to the update rate decrease rule by the update rate decreasing unit 317 (step S11), and step The process returns to S3.

[Operation of mixed occurrence probability calculation device]
The operation of the mixed occurrence probability calculation device 40 will be described with reference to FIG. 4 (see FIG. 1 as appropriate).

The mixed occurrence probability calculation device 40 outputs the word string w ₁ , w ₂ ,..., W _t−1 of the target data storage unit 401 to the neural network language model calculation device 10 as the previous context by the third occurrence probability request unit 411. To do.
The mixed occurrence probability calculation device 40 receives the occurrence probability p _N (w _t | w ₁ w ₂ ... W of the hidden layer vector z and the next word w _t from the neural network language model calculation device 10 by the third occurrence probability request means 411. _t-1 ) is input (step S21).

The mixed occurrence probability calculation device 40 outputs the same previous contexts w ₁ , w ₂ ,..., W _t−1 as in step S 21 to the other language model calculation device 20 by the fourth occurrence probability requesting means 412.
The occurrence probability p _O (w _t | w ₁ w ₂ ... W _t−1 ) of the next word w _i is input to the mixed occurrence probability calculation device 40 from the other language model calculation device 20 by the fourth occurrence probability request unit 412. (Step S22).

  The mixed occurrence probability calculating device 40 uses the hidden layer vector z and the mapping vector S input in step S21 by the second mixing coefficient calculating unit 413, according to the equations (8) and (9). t) is calculated (step S23).

The mixed occurrence probability calculation device 40 uses the occurrence probability p _N (w _t | w ₁ w ₂ ... W _t−1 ) input in step 21 and the occurrence probability p input in step 22 by the mixed occurrence probability calculation means 414. The mixed occurrence probability p (w _t | w ₁ w ₂ ... W _t−1 ) with _O (w _t | w ₁ , w ₂ ,..., W _t−1 ) is calculated by Expression (16) (step S24). ).

[Action / Effect]
As described above, the mixed occurrence probability calculation system 1 learns the mapping vector S using the neural network language model, and obtains the mixing coefficient corresponding to the previous context using the learned mapping vector S. As a result, when the mixed occurrence probability calculation system 1 calculates the mixed occurrence probability p by mixing with another language model such as an n-gram language model, the accuracy of the mixed occurrence probability p can be improved as compared with the conventional case. it can.

(Modification)
As mentioned above, although each embodiment of this invention was explained in full detail, this invention is not limited to above-described embodiment, The design change etc. of the range which does not deviate from the summary of this invention are also included.

  In the embodiment described above, the mixing coefficient parameter learning device has been described as including the mixing coefficient parameter storage unit and the mixing coefficient storage unit, but the present invention is not limited to this. That is, the mixture occurrence probability calculation device may include a mixture coefficient parameter storage unit and a mixture coefficient storage unit.

  In the embodiment described above, an example in which the present invention is applied to a language model has been described. However, the probability model to which the present invention can be applied is not limited to this, and the occurrence probability model of a symbol that occurs following any symbol sequence in general. Can be applied.

In the above-described embodiment, it has been described that regularization is performed. However, the present invention may not be regularized.
In the above-described embodiment, the bias value b is used. However, the present invention may not use the bias value b. In this case, the following formula (17) is used instead of the above formula (9).

In the above-described embodiment, the mixing coefficient parameter learning device has been described as independent hardware, but the present invention is not limited to this. For example, the mixing coefficient parameter learning device can also be realized by a mixing coefficient parameter learning program for cooperatively operating hardware resources such as a CPU, a memory, and a hard disk included in a computer. This program may be distributed through a communication line, or may be distributed by writing in a recording medium such as a CD-ROM or a flash memory.
Further, the mixed occurrence probability calculation device can also be realized by a mixed occurrence probability calculation program in the same manner as the mixing coefficient parameter learning device.

DESCRIPTION OF SYMBOLS 1 Mixed occurrence probability calculation system 10 Neural network language model calculating apparatus 20 Other language model calculating apparatus 30 Mixed coefficient parameter learning apparatus 301 Mixed coefficient parameter storage means 302 Learning parameter storage means 303 Learning data storage means 304 Mixed coefficient storage means 311 Initialization means 312 First occurrence probability request means (first occurrence probability input means)
313 Second occurrence probability request means (second occurrence probability input means)
314 First mixture coefficient calculation means 315 Mapping vector update means 316 End condition determination means 317 Update rate reduction means 40 Mixed occurrence probability calculation device 401 Target data storage means 402 Mixed occurrence probability storage means 411 Third occurrence probability request means (third occurrence) Probability input means)
412 Fourth occurrence probability request means (fourth occurrence probability input means)
413 Second mixing coefficient calculating means 414 Mixed occurrence probability calculating means

Claims (5)

A mixing coefficient parameter for learning parameters necessary for calculating the mixing coefficient when mixing the occurrence probabilities of the next element with respect to the previous element sequence obtained by the neural network probability model and other probability models other than the neural network probability model. A learning device,
First occurrence probability input means for inputting a hidden layer vector of the neural network probability model and an occurrence probability obtained by the neural network probability model;
A second occurrence probability input means for inputting the occurrence probability obtained by the other probability model;
A first mixing coefficient calculating means for calculating the mixing coefficient by linearly mapping the hidden layer vector to a real-valued scalar according to a preset mapping vector, and nonlinearly transforming the real-valued scalar with a sigmoid function;
The mapping vector as the parameter is obtained by a stochastic gradient descent method using an occurrence probability obtained by each of the neural network probability model and the other probability model, the mixing coefficient, and a preset update rate. Map vector updating means for updating;
Update rate reduction means for reducing the update rate according to a preset update rate reduction rule;
Determining whether or not a preset end condition is satisfied, and until the end condition is satisfied, an end condition determining unit that causes the map vector updating unit to update the mapping vector at the reduced update rate;
A mixing coefficient parameter learning apparatus comprising:   2. The mixed coefficient parameter learning apparatus according to claim 1, wherein the map vector update unit performs regularization when the map vector is updated. A mixed occurrence probability calculation device for calculating a mixed occurrence probability obtained by mixing the occurrence probabilities of the next element with respect to the previous element series obtained by the neural network probability model and other probability models other than the neural network probability model,
A third occurrence probability input means for inputting the hidden layer vector of the neural network and the occurrence probability obtained by the neural network probability model;
A fourth occurrence probability input means for inputting the occurrence probability obtained by the other probability model;
The mixture coefficient is calculated by linearly mapping the hidden layer vector to a real-valued scalar using the mapping vector learned by the mixing coefficient parameter learning apparatus according to claim 1 and performing nonlinear conversion on the real-valued scalar using a sigmoid function. Second mixing coefficient calculating means for
A mixed occurrence probability calculating means for calculating the mixed occurrence probability by mixing the occurrence probabilities of the next element obtained by the neural network probability model and the other probability model using the mixing coefficient;
A mixed occurrence probability calculation device comprising:   A mixing coefficient parameter learning program for causing a computer to function as the mixing coefficient parameter learning apparatus according to claim 1.   A mixed occurrence probability calculation program for causing a computer to function as the mixed occurrence probability calculation device according to claim 3.

Images