JP2017117045A - Method, device, and program for language probability calculation - Google Patents

Abstract

PROBLEM TO BE SOLVED: To appropriately predict the next word by effectively utilizing information of a long text, that is, a topic and a style of the sentence, a word peculiar to a speaker and a way of speaking or the like.SOLUTION: A language probability calculation device 1 sequentially reads a vector showing a symbol, and calculates an activity vector of an input layer for each reading, and every time the activity vector of the input layer is calculated, the activity vector of the intermediate layer is calculated based on the activity vector calculated last time in the intermediate layer and the activity vector of the input layer. At this time, the language probability calculation device 1 calculates an average activity vector which is an average of the activity vector calculated previous to the prescribed number of times from among the activity vectors of the intermediate layer, and every time the activity vector of the intermediate layer is calculated, the activity vector of an output layer is calculated based on the activity vector of the intermediate layer and the average activity vector of the intermediate layer, and an appearance probability of the symbol is calculated based on the activity vector of the output layer.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a language probability calculation method, a language probability calculation device, and a language probability calculation program using a language model.

  2. Description of the Related Art Conventionally, a language model is known, which is a model that gives as a language probability how likely a symbol string, which is a string of characters or words, in a target language. Language models are used for various purposes. For example, in speech recognition, an acoustic probability indicating the degree of acoustic similarity to the pronunciation of an arbitrary word sequence for a certain input speech signal, and a language probability indicating the degree of linguistic validity as a word sequence In addition, it is possible to select a word string that is plausible acoustically and linguistically from among a large number of recognition candidates.

  There is an N-gram language model as a widely used language model. The N-gram language model makes the assumption that the appearance probability of a word depends only on the N-1 word preceding that word. That is, the chain probability of N words is estimated as the language probability of each word, and the accumulated value is used as the language probability for the word string. Generally, a value of about 2 to 4 is used for N.

  On the other hand, as a language model other than the N-gram language model, there is an RNN (Recurrent Neural Network) language model (see, for example, Non-Patent Document 1). RNN is a kind of multilayer neural network, and has recursive connections to neurons in the intermediate layer. By this recursive combination, all contexts from the beginning of the input word string to the word read immediately before can be stored in the activity vector of the intermediate layer, and a longer language-dependent language probability can be calculated.

T. Mikolov, M. Karafiat, L. Burget, J. Cernocky, S. Khudanpur, “Recurrent neural network based language model,” International Conference Interspeech 2010 Proceedings, pp. 1045-1048, 2010.

  However, the conventional language model cannot effectively predict the next word by effectively using long contextual information, that is, the topic and style of the sentence, the word and the way of speaking specific to the speaker, etc. was there.

  For example, when predicting a word using an N-gram language model, only the information for several words can be used for predicting a word as described above. May not be predicted properly.

  In addition, when predicting a word using the RNN language model, the influence of a newly appearing word becomes large, and the influence of a past word to some extent becomes very small. May not be able to predict the word properly. This is because the component transmitted by recursive combination is normalized between 0 and 1 by the activity function, so that the component for the previously read symbol stored in the activity vector reads a new symbol. This is because it decreases exponentially.

  The language probability calculation method of the present invention is a language probability calculation method for calculating a language probability using a neural network model having an input layer, an intermediate layer having recursively coupled neurons, and an output layer, A symbol vector reading step for sequentially reading vectors to be represented; an input layer activity vector calculating step for calculating an activity vector in the input layer based on the vector each time the vector is read by the symbol vector reading step; Each time the activity vector in the input layer is calculated by the input layer activity vector calculation step, the activity level in the intermediate layer is calculated based on the activity vector previously calculated in the intermediate layer and the activity vector in the input layer. An intermediate layer activity vector calculating step for calculating an activity vector, and an activity in the intermediate layer; An average activity vector calculating step of calculating an average activity vector that is an average of the activity vectors calculated up to a predetermined number of times before, and the activity in the intermediate layer by the intermediate layer activity vector calculating step An output layer activity vector calculation step of calculating an activity vector in the output layer based on the activity vector in the intermediate layer and the average activity vector in the intermediate layer each time a vector is calculated; And a symbol appearance probability calculating step of calculating an appearance probability of a predetermined symbol based on an activity vector in the output layer.

  The language probability calculation device of the present invention is a language probability calculation device that calculates a language probability using a neural network model having an input layer, an intermediate layer having recursively coupled neurons, and an output layer, A symbol vector reading unit that sequentially reads a vector representing a symbol, and an input layer activity vector calculation unit that calculates an activity vector in the input layer based on the vector each time the vector is read by the symbol vector reading unit; Each time the activity vector in the input layer is calculated by the input layer activity vector calculator, the intermediate level is calculated based on the activity vector previously calculated in the intermediate layer and the activity vector in the input layer. An intermediate layer activity vector calculating unit for calculating an activity vector in the layer; and an activity in the intermediate layer An average activity vector calculation unit that calculates an average activity vector that is an average of activity vectors calculated up to a predetermined number of times before the battle, and the activity vector in the intermediate layer by the intermediate layer activity vector calculation unit An output layer activity vector calculation unit that calculates an activity vector in the output layer based on the activity vector in the intermediate layer and the average activity vector in the intermediate layer, and the output A symbol appearance probability calculating unit that calculates an appearance probability of a predetermined symbol based on an activity vector in the layer.

  According to the present invention, it is possible to appropriately predict the next word by effectively using long context information, that is, the topic and style of the sentence, the word and the way of speaking peculiar to the speaker, and the like.

FIG. 1 is a diagram illustrating an example of an RNN language model. FIG. 2 is a flowchart illustrating an example of a method for calculating the degree of activity in the RNN language model. FIG. 3 is a diagram for explaining an example of a language probability calculation method in the RNN language model. FIG. 4 is a diagram illustrating an example of the configuration of the language probability calculation apparatus according to the first embodiment. FIG. 5 is a flowchart illustrating an example of processing in the language probability calculation apparatus according to the first embodiment. FIG. 6 is a diagram for explaining an example of a language probability calculation method in the language probability calculation apparatus according to the first embodiment. FIG. 7 is a diagram illustrating an example of the configuration of the speech recognition apparatus according to the second embodiment. FIG. 8 is a flowchart illustrating an example of processing in the speech recognition apparatus according to the second embodiment. FIG. 9 is a diagram illustrating an example of a computer in which a language probability calculation device or a speech recognition device is realized by executing a program.

  Hereinafter, embodiments of a language probability calculation method, a language probability calculation device, and a language probability calculation program according to the present application will be described in detail with reference to the drawings. The language probability calculation method, the language probability calculation device, and the language probability calculation program according to the present application are not limited by this embodiment.

[Outline of RNN language model]
First, an outline of the RNN language model will be described. The RNN has one input layer, one or more intermediate layers, and one output layer, and has a recursive connection in which neurons are connected to each other in at least one intermediate layer. Then, each symbol of the input symbol string is sequentially input to the RNN of the RNN language model, and the current symbol is obtained by using the vector representing the current previous symbol and the activity of each neuron in the intermediate layer at that time. The appearance probability of is calculated.

  Each layer in the RNN has a plurality of neurons, which are connected to neurons in the upper layer, the lower layer, or the same layer. Each neuron has an activity (real value) indicating the degree of firing, and a connection weight (real value) indicating the strength of connection is assigned between the connected neurons. A value obtained by multiplying the activity of each neuron by the connection weight is propagated to the connection destination neuron. In addition, the activity of neurons included in the same layer is collectively expressed as an activity vector.

  FIG. 1 is a diagram illustrating an example of an RNN language model. FIG. 1 shows an RNN having one input layer, one intermediate layer, and one output layer. As shown in FIG. 1, the intermediate layer has a recursive connection back to the same layer. The input layer is given the value of the input vector as the activity.

  As described above, in the RNN language model, an input symbol is expressed as a vector composed of 0 or 1 values. For example, as many neurons as the number (vocabulary size) of all the symbols to be considered are prepared in the input layer, and the activity is set so that only the neurons corresponding to the input symbols take 1 and the other neurons take 0. can do. In this case, if the types of symbols to be considered are A, B, and C, three neurons in the input layer are required, and the input vectors corresponding to the symbols A, B, and C are, for example, those of Expression (1). It is expressed as follows. However, the first dimension of the vector corresponds to the symbol A, the second dimension corresponds to the symbol B, and the third dimension corresponds to the symbol C.

  In FIG. 1, the elements in the first, second, and third dimensions of the vector correspond to the degree of activity in order from the top of the neurons in the input layer. In the intermediate layer, the activity is calculated in consideration of recursive coupling. However, when the first symbol is read, that is, when t = 1, the activity is assumed to be zero. Further, the activity vector of the neurons in the output layer is calculated using a softmax function or the like. Note that the neurons in the output layer also correspond to the symbols A, B, and C in order from the top as in the input layer.

Next, a method for calculating the activity will be described with reference to FIG. FIG. 2 is a flowchart illustrating an example of a method for calculating the degree of activity in the RNN language model. Here, the activity is calculated by the RNN consisting of the L layer, where the first layer is the input layer, the second to L-1 layers are the intermediate layers (L ≧ 3), and the L layer is the output layer. Some RNN is used. In addition, the nth layer (1 ≦ n ≦ L) of the RNN includes H _n neurons. In addition, a matrix having a connection weight w _{m, n} [k, j] from the j-th neuron in the m-th layer to the k-th neuron in the n-th layer is represented by w _{m, n} . However, the 1 ≦ m ≦ n ≦ L, 1 ≦ j ≦ H m, 1 ≦ k ≦ H n. Further, the activity vector of the neuron at the time t in the nth layer (1 <n ≦ L ⁾ is represented as h _n ^(t) .

First, t is set to 1 (step S101). Next, the t-th input symbol x _t is substituted into the activity vector h ₁ ^(t) of the input layer (step S102), and n is initialized to 2 (step S103). Here, when the n-th layer of the RNN is an intermediate layer with recursive connection (step S104, Yes), the recursively propagated component w _{n, n} · h _n ^(t−1) is substituted into z _n . (Step S105). On the other hand, when the n-th layer of the RNN is not an intermediate layer with recursive connection (No in step S104), z _n is set to a 0 vector (step S106).

Then, the component w _{n−1, n} · h _n−1 ^(t) propagated from the ⁽ _n−1 ⁾ _{th layer} is added to z _n (step S107). Next, the activation function f _n (•) is applied to z _n to obtain the n-th layer activity vector h _n ^(t) (step S108). Here, when L is the number of layers, if n <L (step S109, Yes), n is incremented by 1 (step S110), and the process returns to step S104 and is repeated. If n <L is not satisfied (step S109, No), if t <T (step S111, Yes), t is incremented by 1 (step S112), and the process returns to step S102 to repeat the process. If t <T is not satisfied (No in step S111), the process is terminated.

  The language probability calculation method in the RNN language model will be described with reference to FIG. FIG. 3 is a diagram for explaining an example of a language probability calculation method in the RNN language model. In the example of FIG. 3, L = 3, and only one intermediate layer is provided.

As shown in FIG. 3, for the symbols x ₁ , x ₂ ,..., X _t that are input in order, the intermediate layer activity vector h ₂ ^(t) is the input layer activity vector h for the current symbol. ₁ ^(t) and the intermediate layer activity vector h ₂ ^(t−1) for the previous symbol are calculated (steps S105, S106, S107, and S108 in FIG. 2). Further, the activity vector h ₃ ^{(t) of the} output layer is calculated based on h ₂ ^(t) . Then, the appearance probability of the symbol is calculated based on the output layer activity vector.

  Thus, since the activity of the neurons in the intermediate layer of the RNN is propagated again to the neurons in the intermediate layer by recursive connection, the activity of the neurons in the intermediate layer includes the characteristics of the input sequence read so far. Is memorized. Therefore, the RNN language model can obtain the symbol appearance probability depending on the history from the beginning of the input symbol string to the present.

  As a result, the RNN language model can determine the symbol appearance probability considering a longer context than the N-gram model (N is 3 or 4 at most) that predicts the next symbol from only the past N-1 symbols. Model. However, as described above, the influence of newly appearing words becomes large, and the influence of past words to some extent becomes very small. Therefore, some past words are not effectively used for word prediction.

[First Embodiment]
In the following embodiments, the configuration of the language probability calculation device according to the first embodiment, the language probability calculation method executed by the language probability calculation device, and the effects of the first embodiment will be described. In the following description, “RNN language model” indicates the RNN language model in the embodiment of the present invention. “Conventional RNN language model” indicates the RNN language model described above with reference to FIGS. And

[Configuration of First Embodiment]
First, the configuration of the language probability calculation apparatus according to the first embodiment will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of the configuration of the language probability calculation apparatus according to the first embodiment. As illustrated in FIG. 4, the language probability calculation device 1 includes a prediction unit 10, a learning unit 11, and a storage unit 12.

  The prediction unit 10 includes a symbol vector reading unit 101, an input layer activity vector calculation unit 102, an intermediate layer activity vector calculation unit 103, an average activity vector calculation unit 104, an output layer activity vector calculation unit 105, and a symbol appearance probability. A calculation unit 106 is included. The learning unit 11 includes a loss function definition unit 111 and a parameter estimation unit 112. In addition, the storage unit 12 includes an RNN language model storage unit 121 that stores an RNN language model used in the prediction unit 10 and the like.

First, each part of the prediction unit 10 will be described in detail, and a language probability calculation method executed by the language probability calculation device 1 will be described. Here, the language probability calculation device 1 is an RNN composed of L layers, where the first layer is an input layer, the second to (L-1) th layers are intermediate layers (L ≧ 3), and the Lth layer is an output layer. A language probability is calculated using a certain RNN. In addition, the nth layer (1 ≦ n ≦ L) of the RNN includes H _n neurons. Further, a matrix having the connection weights w _{m, n} [k, j] from the j-th neuron in the m-th layer to the k-th neuron in the n-th layer is represented by w _{m, n} . However, the 1 ≦ m ≦ n ≦ L, 1 ≦ j ≦ H m, 1 ≦ k ≦ H n. Further, the activity vector of the neuron at the time t in the nth layer (1 <n ≦ L ⁾ is represented as h _n ^(t) .

  The symbol vector reading unit 101 sequentially reads vectors representing symbols. Each symbol corresponds to a dimension of one vector, and a unique symbol is expressed by setting only an element of a corresponding dimension to 1 and setting an element of a dimension not corresponding to 0.

Every time a vector is read by the symbol vector reading unit 101, the input layer activity vector calculation unit 102 calculates an activity vector in the input layer based on the vector. In the first embodiment, the input layer activity vector calculation unit 102 determines the value of each dimension of the input symbol vector as the neuron activity of the input layer. Therefore, the number of dimensions of the vector of input symbols is equal to the number of types of input symbols that can appear and the number of neurons in the input layer. Here, the input symbol sequence _{X = x 1, x 2,} ..., x t, ..., when the t-th symbol _{x t} is read from the first _{x T} in order, representing the activity of the neurons in the input layer The activity vector h ₁ ^(t) is expressed by Equation (2).

The intermediate layer activity vector calculation unit 103 calculates the activity vector in the intermediate layer each time the input layer activity vector calculation unit 102 calculates the activity vector in the input layer. As shown in Expression (3), the intermediate layer activity vector calculation unit 103 uses the n-th layer (1 <n ≦ L) neuron activity vector h _n ^(t) to the neuron coupled to the neuron. A set of values obtained by multiplying the activity by the connection weight, that is, the product of the connection weight matrix and the activity vector is obtained, and the product of the obtained vector is calculated by normalizing between 0 and 1 by the activation function. When the intermediate layer is one layer, h _n−1 ^(t) is an activity vector in the input layer.

  Here, the sigmoid function shown in Formula (4) is used as the activation function. Here, x represents each element of the vector.

On the other hand, when there is a recursive connection with neurons in the same layer in the intermediate layer, as shown in the equation (5), the activity vector previously calculated in the intermediate layer, that is, the t−1th symbol x _t H _n ^(t) is calculated based on the activity vector in the intermediate layer when ₋₁ is read. In the second term in f _n (•) on the right side of Equation (5), the activity vector h _n ^(t−1) with the subscript (t−1) is assigned to the activity of the neurons in the same layer. Add with weighting w _{n, n} .

The average activity vector calculation unit 104 calculates an average activity vector that is an average of the activity vectors calculated up to a predetermined number of times among the activity vectors in the intermediate layer. That is, the average activity vector calculation unit 104 performs the activity vector h n−1 (1) of the ( n−1 ) th intermediate layer with respect to the symbol strings x 1 , x 2 ,. ), h n-1 (2 ), ..., h n-1 (t-1), when h n-1 (t) is calculated by an interlayer activity vector calculating unit 103, the intermediate layer is calculated An average activity vector, which is the average of past A activity vectors, is calculated by Equation (6).

At this time, by increasing the value of A, context information longer than the conventional context information is held in the average activity vector. Further, the average activity vector calculation unit 104 may calculate a plurality of average activity vectors by changing the predetermined number of times. In this case, the average activity vector calculating unit 104, different values of ^{A A (1), A (} 2), ..., A (m), ..., set the ^{A (M),} a plurality of average activity vector calculate. Here, M is an arbitrary constant, and A ^(m) is a function of m that returns an integer. At this time, an average activity vector for m = 1, 2,...

Here, w _{n−1, n} ^(m) is a connection weight matrix for calculating an average activity vector for A ^(m) . When M = 1, A ⁽¹⁾ = 1, w _{n−1, n} ⁽¹⁾ = w _{n−1, n} , the average activity vector is calculated by the intermediate layer activity vector calculation unit 103. Equal to the activity vector. When the n-th layer has a recursive combination, the average activity vector calculation unit 104, like the intermediate layer activity vector calculation unit 103, uses the equation (8) to calculate the activity vector h _n ^{( t-1)} is multiplied by the connection weights w _{n, n} and added.

  The output layer activity vector calculation unit 105 calculates an activity vector in the output layer every time the activity vector in the intermediate layer is calculated by the intermediate layer activity vector calculation unit 103. The output layer activity vector calculation unit 105 calculates the activity vector in the Lth layer, that is, the output layer, based on the activity vector or the average activity vector of the (L-1) th layer which is the intermediate layer closest to the output layer. Do. At this time, the output layer activity vector calculation unit 105 calculates the activity vector in the output layer using the softmax function shown in Expression (9) as the activation function in order to regard the activity as a probability.

Here, the denominator of the equation (9) is a normalization term for regarding the activity as a probability, and z _n [i] is the activity propagated with weight from the (n−1) th layer shown in the equation (10). This represents the i- _th element of the degree vector z _n .

In the RNN language model, each neuron in the output layer corresponds to a unique symbol, and the appearance probability of the predicted next symbol is obtained as the activity of the neuron corresponding to that symbol. Therefore, the symbol occurrence probability calculation unit 106, an input symbol sequence _{x 1,} x 2, ..., the probability that the symbol _{v k} based on the activity of vector _h ^{L (t)} in the output layer after reading _{x t} appears , Calculated by the equation (11). Here, the symbol v _k represents a symbol corresponding to the k-th neuron in the output layer.

  When the average activity vector calculation unit 104 calculates a plurality of average activity vectors by changing the predetermined number of times, the output layer activity vector calculation unit 105 calculates the activity vector in the intermediate layer and the plurality of averages in the intermediate layer. The weighted sum of each activity vector is calculated as the activity vector in the output layer. In this case, the output layer activity vector calculation unit 105 uses Expression (7) or Expression (8) in which n is set to L.

  The intermediate layer activity vector calculation unit 103 and the average activity vector calculation unit 104 calculate the activity by using the method described so far depending on whether or not the n−1th layer and the nth layer have recursive connections. Do.

  First, when there is no recursive connection in either the nth layer or the (n−1) th layer, the activity vector calculated by the equation (3) becomes the activity vector of the nth layer. Further, when there is a recursive connection in the (n−1) th layer and there is no recursive connection in the nth layer, the activity vector calculated by the equation (5) becomes the activity vector of the nth layer. Further, when there is no recursive connection in the (n−1) th layer and there is a recursive connection in the nth layer, the activity vector calculated by the equation (7) becomes the activity vector of the nth layer. Further, when there is a recursive connection in both the nth layer and the n−1th layer, the activity vector calculated by the equation (8) becomes the activity vector of the nth layer.

  Next, each part of the learning unit 11 will be described in detail, and a connection weight setting method for calculating the activity vector will be described. First, the coupling weight, which is basically an RNN parameter, is estimated by using the error back propagation method using the learning data of the symbol string.

As an example, a method of setting the connection weight will be described by taking L = 3, that is, a case of creating a three-layer RNN language model as an example. In this case, the first layer is an input layer, the second layer is an intermediate layer, and the third layer is an output layer. First, the connection weight matrices w ₁ , ₂ , w ₂ , ₂ , w ₂ , ₃ can be set by a known method such as BPTT (Back Propagation Through Time) (reference document: Williams, RJ, and Zipser, D. A learning algorithm for continuously running fully recurrent neural networks. Neural Computation. 1 (2), 270, 1989.).

On the other hand, the learning unit 20 sets the connection weights w _2,3 ⁽¹⁾ ,..., W _2,3 ^(M) in the average activity vector calculation unit 104 as follows. First, the learning unit 20 uses the functions of the prediction unit 10 or the like to use the matrices w _1,2 and w _2,2 to change the intermediate layer for the symbol strings x ₁ , x _{2 ,.} activity vector sequence ^{_{h 2 (1), h 2}} (2), ..., h 2 (t), ..., determine the ^{h 2 (T).} Then, the loss function definition unit 111 uses the learning data to define a loss function using the weight in the weighted sum as a parameter. That is, the loss function definition unit 111 defines a loss function based on the negative log likelihood shown in Expression (13) for the set of parameters shown in Expression (12).

Then, the parameter estimation unit 112 estimates parameters so that the loss function is minimized. The parameter estimation unit 112 estimates V so that E (V) in Expression (13) is minimized. Here, as shown in Expression (14), y _t [k] is an element of the k-th dimension of the symbol y _t that appears next to each symbol x _t expressed in vector of the learning data, that is, at time t + 1. Represent.

H ₃ ^(t) is an activity vector of the output layer and represents the probability distribution of the predicted next symbol. That is, E (V) becomes smaller as the RNN language model gives a higher probability to the symbol that actually appears next. Therefore, the parameter estimation unit 112 may obtain V that minimizes E (V). However, the second term on the right-hand side is a regularization term for controlling the individual elements w _2,3 ^(m) [k, j] of V so as not to become too large, and β is a weighting factor for the regularization term. Is a positive constant.

The parameter estimation unit 112 obtains the minimum value of E (V) using, for example, a gradient method. For example, as shown in Expression (15), the parameter estimation unit 112 obtains partial differentiation with respect to individual elements w _2,3 ^(m) [k, j] of E (V).

  Then, the parameter estimation unit 112 obtains a new parameter as shown in Expression (16), and further repeats partial differential calculation and parameter update using the obtained new parameter. Here, η represents a learning rate.

  The parameter estimation unit 112 uses a stochastic gradient method that divides the learning data into several small blocks (or individual words, etc.), reads the divided blocks in order, and repeats the partial differential calculation and the parameter update. Thus, the convergence of the loss function can be accelerated.

[Process of First Embodiment]
Next, processing of the language probability calculation device 1 will be described with reference to FIG. FIG. 5 is a flowchart illustrating an example of processing in the language probability calculation apparatus according to the first embodiment. As shown in FIG. 5, first, the symbol vector reading unit 101 sets t to 1 (step S201), and reads a symbol string. Then, the input layer activity vector calculation unit 102 sets the symbol x _t to the input layer activity vector h ₁ ^(t) (step S202), and sets n to 2 (step S203).

Here, when the n-th layer is an intermediate layer with recursive connection (step S204, Yes), the intermediate layer activity vector calculation unit 103 recursively propagates the components wn _{, n} · h _n ^{(t−1). )} is substituted into _{z n} (step S205). On the other hand, if the n-th layer is not an intermediate layer with a recursive connection (step S204, No), the intermediate layer activity vector calculating unit 103 sets the _{z n} to 0 vector (step S206).

Furthermore, when the (n−1) th layer is an intermediate layer with recursive connection (step S207, Yes), the average activity vector calculation unit 104 calculates an average activity vector. At this time, the average activity vector calculation unit 104 first sets m to 1 (step S209). Then, the average activity vector calculation unit 104 calculates an average activity vector for the past A ^(m) (step S210), adds the weight to z _n (step S211).

  If m <M (step S212, Yes), the average activity vector calculation unit 104 increases m by 1 (step S213), returns to step S209, and repeats the process. If m <M is not satisfied (step S212, No), the average activity vector calculation unit 104 ends the calculation of the average activity vector.

On the other hand, when the (n−1) th layer is not an intermediate layer with recursive connection (No in step S207), the intermediate layer activity vector calculation unit 103 determines the component w _{n−1, n} · h propagated from the (n−1) th layer. Add ^{_n-1 (t)} to _{z n} (step S208).

Next, the average activity vector calculation unit 104 applies the activation function f _n (·) to z _n to obtain the n-th layer activity vector h _n ^(t) (step S214). When n = L, the output layer activity vector calculation unit 105 applies the activation function. Here, when n <L (step S215, Yes), n is increased by 1 (step S216), and the process returns to step S204 and is repeated. If n <L is not satisfied (step S215, No), if t <T (step S217, Yes), t is increased by 1 (step S218), and the process returns to step S202 to repeat the process. If t <T is not satisfied (step S217, No), the process is terminated.

  A language probability calculation method in the language probability calculation apparatus 1 will be described with reference to FIG. FIG. 6 is a diagram for explaining an example of a language probability calculation method in the language probability calculation apparatus according to the first embodiment. In the example of FIG. 6, L = 3, and only one intermediate layer is provided.

As shown in FIG. 6, for the symbols x ₁ , x ₂ ,..., X _t that are sequentially input, the intermediate layer activity vector calculation unit 103 sets n to 2 and the activity of the input layer for the current symbol. An intermediate layer activity vector h ₂ ^(t) is calculated from the vector h ₁ ^(t) and the intermediate layer activity vector h ₂ ^(t−1) for the previous symbol (steps S205 and S206 in FIG. 5 ⁾ . , S208, S214).

Next, the average activity vector calculation unit 104 sets n to 3, the average of the activity vectors of the past two intermediate layers, the average of the past four intermediate layer activity vectors, and the past eight The average of the activity vectors of the intermediate layers is calculated (steps S210 and S211 in FIG. 5). At this time, the activity vector h ₂ ^(t) of the intermediate layer is regarded as an average of the activity vectors of the intermediate layer for the past one time.

The output layer activity vector calculation unit 105 applies the activation function to the average activity vector for the past one time, two times, four times, and eight times, and calculates the activity vector h ₃ ^(t) in the output layer ( Step S214 in FIG. It should be noted that the past number of activity vectors for which the average activity vector is calculated depends on the setting of the constant M and the function A ^(m) .

[Effect of the first embodiment]
The effects of the first embodiment will be described using the results of the evaluation of the language probability calculation device 1 using actual data. First, in order to obtain the parameters of the RNN language model, we used sentences that were written by a person as a learning data for academic conferences included in the “Japanese spoken corpus”.

In addition, the RNN language model has an input layer (H ₁ = 52,564), an intermediate layer (H ₂ = 400), and an output layer (H ₃ ) for words appearing in learning data (vocabulary sizes 52 and 564). = 52,564) and configured as a three-layer (L = 3) RNN. In addition, the stochastic gradient method is used to estimate the parameters (w _2,3 ^(m) [k, j]) of the RNN language model in the first embodiment, and M = 6, A ^(m) = 2 ^{(m -1)} , learning rate η = 0.1, β = 10 ⁻⁵ .

  The learning data is learned by the RNN language model in the first embodiment and the conventional RNN language model, and transcripts of 10 conference lectures different from the learning data are used as evaluation data, and test sets of each RNN language model are used. The results of calculating plexity are shown in Table 1.

Test set perplexity is known as an index representing the performance of a language model. When the evaluation data x ₁ , x ₂ ,..., X _τ ,..., X _R are given, the test set perplexity is a value obtained by raising 2 to the entropy shown in Expression (17) using a language model, that is, 2 Defined as ^H. Since the smaller the test set perplexity means that the performance of the language model is higher, it can be seen from Table 1 that the RNN language model in the first embodiment shows higher performance than the conventional RNN language model.

  In the language probability calculation apparatus 1, first, the symbol vector reading unit 101 sequentially reads vectors representing symbols. Next, every time a vector is read by the symbol vector reading unit 101, the input layer activity vector calculation unit 102 calculates an activity vector in the input layer based on the vector. Then, each time the input layer activity vector calculation unit 102 calculates the activity vector in the input layer, the intermediate layer activity vector calculation unit 103 calculates the activity vector previously calculated in the intermediate layer and the activity in the input layer. Based on the degree vector, the activity vector in the intermediate layer is calculated.

  Here, the average activity vector calculation unit 104 calculates an average activity vector that is an average of the activity vectors calculated up to a predetermined number of times among the activity vectors in the intermediate layer. The output layer activity vector calculation unit 105 calculates the activity vector in the intermediate layer and the average activity vector in the intermediate layer each time the activity vector in the intermediate layer is calculated by the intermediate layer activity vector calculation unit 103. Based on the above, the activity vector in the output layer is calculated. Then, the symbol appearance probability calculation unit 106 calculates the appearance probability of a predetermined symbol based on the activity vector in the output layer.

  As described above, the average activity vector calculation unit 104 can sufficiently affect the influence of the activity vector calculated up to an arbitrary number of times before the final symbol appearance probability. Therefore, according to the first embodiment, it is possible to appropriately predict the next word by effectively using long context information, that is, the topic and style of the sentence, the word and the way of speaking specific to the speaker, and the like. .

  In addition, the average activity vector calculation unit 104 may calculate a plurality of average activity vectors by changing the predetermined number of times. At this time, the output layer activity vector calculation unit 105 calculates the weighted sum of the activity vector in the intermediate layer and the plurality of average activity vectors in the intermediate layer as the activity vector in the output layer. Further, the loss function definition unit 111 uses the learning data to define a loss function using the weight in the weighted sum as a parameter. Then, the parameter estimation unit 112 estimates parameters so that the loss function is minimized.

  In this way, since the prediction accuracy can be improved by learning using a plurality of average activity vectors, information in a long context can be used more effectively.

[Second Embodiment]
Next, a case where the language probability calculation method of the present invention is applied to a speech recognition apparatus will be described as a second embodiment. The speech recognition apparatus outputs a recognition result in consideration of both acoustic validity and linguistic validity. In the second embodiment, the language probability calculation method of the present invention is used to determine linguistic validity.

[Configuration of Second Embodiment]
The configuration of the speech recognition apparatus according to the second embodiment will be described with reference to FIG. FIG. 7 is a diagram illustrating an example of the configuration of the speech recognition apparatus according to the second embodiment. As illustrated in FIG. 7, the speech recognition device 2 includes a speech signal input unit 21, a candidate sentence creation unit 22, an acoustic score calculation unit 23, a language probability calculation unit 24, a language score calculation unit 25, and a recognition result extraction unit 26. .

An audio signal to be recognized is input to the audio signal input unit 21. The candidate sentence creation unit 22 creates a plurality of candidate sentences that are sentence candidates that match the input voice signal. Here, the candidate sentence creation unit 22 creates Q candidate sentences X ₁ , X ₁ ,..., X _Q.

The acoustic score calculation unit 23 calculates an acoustic score representing an acoustic coincidence with the speech signal for each candidate sentence. The acoustic score calculation unit 23 calculates an acoustic score amscore (X _q ) for each of the Q candidate sentences. The acoustic score calculation unit 23 may calculate the acoustic score using a known method.

Here, when the conventional N-gram language model is used, assuming that the language score is lmscore (X _q ), score (X _q ) is calculated as shown in Expression (18).

The language score lmscore (X _q ) is calculated by the equation (19) based on the language probability of N grams. Here, the appearance probability calculated by the N-gram of the symbol x _{q, τ} in the symbol string x _{q, 1,} ... _, X _{q, τ−1} of the candidate sentence X _q is P _ngram (x _{q, τ} | x _{q, τ−N + 1} ... x _{q, τ−1} ). At this time, N is about 3-4.

In the speech recognition device 2, the RNN language model in the language probability calculation method of the first embodiment is used instead of the N-gram language model. The language probability calculation unit 24 calculates the appearance probability of the symbol for each candidate sentence by the same language probability calculation method as that of the first embodiment. That is, the language probability calculation unit 24 _generates the appearance probability P _rnn (x _{q, τ} | x _{q, 1} of the symbols x _{q, τ} in the symbol string x _{q, 1,} ... _, X _{q, τ−1} of the candidate sentence X _q. ... _{Xq, [tau] -1} ) is calculated.

  In addition, the language score calculation unit 25 calculates a language score for each candidate sentence based on the appearance probability. The language score calculation unit 25 calculates the language score of each candidate sentence based on the appearance probability of the symbol calculated by the language probability calculation unit 24 using Expression (20).

The recognition result extraction unit 26 extracts the candidate sentence having the largest sum of the acoustic score and the language score from the candidate sentences as a sentence that matches the voice signal. The recognition result extraction unit 26 calculates score (X _q ) of each candidate sentence by Expression (21) based on the acoustic score and the language score, and extracts a candidate sentence having the maximum score (X _q ) as a recognition result. To do. Here, λ is a positive constant representing a scaling coefficient with respect to the logarithmic probability.

[Process of Second Embodiment]
The processing of the second embodiment will be described with reference to FIG. FIG. 8 is a flowchart illustrating an example of processing in the speech recognition apparatus according to the second embodiment. As shown in FIG. 8, an audio signal is first input to the audio signal input unit 21 (step S301). Next, the candidate sentence creation unit 22 creates a candidate sentence of the speech recognition result (step S302). And the acoustic score calculation part 23 calculates the acoustic score of each candidate sentence (step S303).

  Here, the language probability calculation unit 24 calculates a language probability (step S304). The language score calculation unit 25 calculates a language score based on the language probability (step S305). The recognition result extraction unit 26 extracts a candidate sentence having the maximum sum of values obtained by multiplying the acoustic score and the language score by the scaling coefficient (step S306), and outputs the candidate sentence as a recognition result (step S307).

[Effects of Second Embodiment]
Table 2 shows respective word error rates when speech recognition is performed using the N-gram language model, the conventional RNN language model, and the RNN language model in the second embodiment. However, the number Q of candidates to be output first is set to 500. The word error rate represents the proportion of words that are mistakenly recognized among the actually spoken words. The smaller the word error rate, the higher the accuracy of speech recognition.

  As shown in Table 2, the word error rate in speech recognition using the N-gram language model was 14.8%. In addition, the word error rate was 13.9% in speech recognition using the conventional RNN language model. In the speech recognition using the RNN language model in the second embodiment, the word error rate is 13.5%. From this, it was shown that the RNN language model in the second embodiment realizes speech recognition with higher accuracy than the N-gram language model and the conventional RNN language model.

  In the speech recognition apparatus 2, first, the candidate sentence creation unit 22 creates a plurality of candidate sentences that are sentence candidates that match the input speech signal. And the acoustic score calculation part 23 calculates the acoustic score showing an acoustic coincidence with the audio | voice signal for every candidate sentence.

  In addition, the language probability calculation unit 24 calculates a symbol appearance probability for each candidate sentence by the same language probability calculation method as that of the first embodiment. And the language score calculation part 25 calculates the language score for every candidate sentence based on an appearance probability. And the recognition result extraction part 26 extracts a candidate sentence with the largest sum of an acoustic score and a language score as a sentence which corresponds to an audio | voice signal among candidate sentences.

[Other Embodiments]
In FIG. 6 and the like, the case where the intermediate layer is one layer has been described as an example. However, the intermediate layer in the present invention is not limited to one layer and may be a plurality. In that case, the intermediate layer activity vector calculation unit 103 of the language probability calculation device 1 calculates the activity level previously calculated in the intermediate layer every time the input layer activity vector calculation unit 102 calculates the activity vector in the input layer. The activity vector in the intermediate layer is calculated based on the activity vector in the intermediate layer below the intermediate layer as well as the vector and the activity vector in the input layer.

  The language probability calculation device 1 also includes an activity vector calculated by the intermediate layer activity vector calculation unit 103 and an intermediate level each time an activity vector in the intermediate layer is calculated by the intermediate layer activity vector calculation unit 103. An interlayer activity vector calculation unit is further provided for calculating an activity vector in the intermediate layer above the intermediate layer based on the average activity vector in the layer.

[System configuration, etc.]
Further, each component of each illustrated apparatus is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured. Further, all or any part of each processing function performed in each device is realized by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or hardware by wired logic. Can be realized as

  In addition, among the processes described in the present embodiment, all or part of the processes described as being automatically performed can be manually performed, or the processes described as being manually performed can be performed. All or a part can be automatically performed by a known method. In addition, the processing procedure, control procedure, specific name, and information including various data and parameters shown in the above-described document and drawings can be arbitrarily changed unless otherwise specified.

[program]
FIG. 9 is a diagram illustrating an example of a computer in which a language probability calculation device or a speech recognition device is realized by executing a program. The computer 1000 includes a memory 1010 and a CPU 1020, for example. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These units are connected by a bus 1080.

  The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to the display 1130, for example.

  The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the language probability calculation device or the speech recognition device is implemented as a program module 1093 in which a code executable by a computer is described. The program module 1093 is stored in the hard disk drive 1090, for example. For example, a program module 1093 for executing processing similar to the functional configuration in the language probability calculation device or the speech recognition device is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

  The setting data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the memory 1010 and the hard disk drive 1090 to the RAM 1012 and executes them as necessary.

  The program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read out by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and the program data 1094 may be read by the CPU 1020 from another computer via the network interface 1070.

DESCRIPTION OF SYMBOLS 1 Language probability calculation apparatus 10 Prediction part 11 Learning part 12 Storage part 101 Symbol vector reading part 102 Input layer activity vector calculation part 103 Intermediate layer activity vector calculation part 104 Average activity vector calculation part 105 Output layer activity vector calculation part 106 Symbol appearance probability calculation unit 111 Loss function definition unit 112 Parameter estimation unit 121 RNN language model storage unit

Claims (7)

A language probability calculation method for calculating a language probability using a neural network model having an input layer, an intermediate layer having recursively coupled neurons, and an output layer,
A symbol vector reading step for sequentially reading vectors representing symbols;
An input layer activity vector calculation step of calculating an activity vector in the input layer based on the vector each time the vector is read by the symbol vector reading step;
Each time an activity vector in the input layer is calculated by the input layer activity vector calculation step, the intermediate layer is based on the activity vector previously calculated in the intermediate layer and the activity vector in the input layer. An intermediate layer activity vector calculation step of calculating an activity vector in
An average activity vector calculating step of calculating an average activity vector that is an average of the activity vectors calculated up to a predetermined number of times among the activity vectors in the intermediate layer;
Each time the activity vector in the intermediate layer is calculated by the intermediate layer activity vector calculation step, the activity in the output layer is based on the activity vector in the intermediate layer and the average activity vector in the intermediate layer. An output layer activity vector calculation step for calculating a degree vector;
A symbol appearance probability calculating step of calculating an appearance probability of a predetermined symbol based on an activity vector in the output layer;
A language probability calculation method comprising: The average activity vector calculating step calculates a plurality of average activity vectors by changing the predetermined number of times,
The output layer activity vector calculation step calculates a weighted sum of the activity vector in the intermediate layer and the plurality of average activity vectors in the intermediate layer as the activity vector in the output layer. The language probability calculation method according to claim 1. A loss function defining step for defining a loss function using the weight in the weighted sum as a parameter, using learning data;
A parameter estimation step for estimating the parameter so that the loss function is minimized;
The language probability calculation method according to claim 2, further comprising: Having a plurality of intermediate layers,
In the intermediate layer activity vector calculation step, whenever the activity vector in the input layer is calculated by the input layer activity vector calculation step, the activity vector previously calculated in the intermediate layer and the intermediate layer 4. The activity vector in the intermediate layer is calculated based on the activity vector in the lower intermediate layer and the average activity vector in the intermediate layer. 5. Language probability calculation method. A candidate sentence creation step of creating a plurality of candidate sentences that are sentence candidates that match the input speech signal;
An acoustic score calculation step of calculating an acoustic score representing an acoustic coincidence with the voice signal for each candidate sentence, and
The symbol vector reading step sequentially reads a vector representing words constituting the candidate sentence,
A language score calculating step of calculating a language score for each candidate sentence based on the appearance probability calculated by the symbol appearance probability calculating step;
A recognition result extraction step of extracting a candidate sentence having the largest sum of the acoustic score and the language score as a sentence matching the voice signal among the candidate sentences;
The language probability calculation method according to claim 1, further comprising: A language probability calculation device that calculates a language probability using a neural network model having an input layer, an intermediate layer having recursively coupled neurons, and an output layer,
A symbol vector reading unit for sequentially reading vectors representing symbols;
An input layer activity vector calculation unit that calculates an activity vector in the input layer based on the vector each time the vector is read by the symbol vector reading unit;
Each time the activity vector in the input layer is calculated by the input layer activity vector calculation unit, the intermediate layer based on the activity vector previously calculated in the intermediate layer and the activity vector in the input layer An intermediate layer activity vector calculation unit for calculating an activity vector in
An average activity vector calculation unit that calculates an average activity vector that is an average of activity vectors calculated up to a predetermined number of times among the activity vectors in the intermediate layer;
Each time the activity vector in the intermediate layer is calculated by the intermediate layer activity vector calculation unit, the activity in the output layer is based on the activity vector in the intermediate layer and the average activity vector in the intermediate layer. An output layer activity vector calculation unit for calculating a degree vector;
A symbol appearance probability calculating unit that calculates an appearance probability of a predetermined symbol based on an activity vector in the output layer;
A language probability calculation apparatus comprising:   A language probability calculation program for causing a computer to function as the language probability calculation device according to claim 6.

Images