JP6691501B2 - Acoustic model learning device, model learning device, model learning method, and program - Google Patents

Description

  The present invention relates to a sequence modeling technique, and more particularly to a technique for interpolating weighting parameters of linear transformation used in a neural network.

  Examples of probabilistic sequence modeling include an n-gram model (see Non-Patent Document 1). A language model based on probabilistic sequence modeling is called, for example, a probabilistic language model. Further, as a series modeling using a neural network, for example, a simple recursive neural network (Elman Network) which is a type of recurrent neural network (RNN) or long short-term memory (LSTM) is used. Etc. (see Non-Patent Document 2). A language model based on sequence modeling using a neural network is called, for example, a neural language model.

Okumura Manabu, "Introduction to Machine Learning for Language Processing", Corona Publishing Co., Ltd., 2010 Hochreiter, S., Schmidhuber, J., "Long short-term memory." Neural computation, vol. 9, no. 8, pp. 1735-1780, 1997.

  Probabilistic language models can have the probability of the next word given the previous word as a categorical distribution. Therefore, a probabilistic language model can explicitly describe all parameters as a tensor with rank n if it is an n-gram model. At the same time, it is possible to give the parameter meaning of the probability value of the next word given the previous n-1 words. For example, in a 2-gram (bigram) model, all the information can be expressed as a rank-2 tensor (that is, a matrix). Each column of this matrix can be interpreted as the distribution of the next word given the corresponding previous word. For example, a practically important operation such as applying a stochastic language model learned using text data belonging to a domain A to the original stochastic language model using text data belonging to another domain B, It can be executed by a method such as linear interpolation between tensors of the probabilistic language model calculated from each.

  The neural language model requires, as the minimum element, a linear transformation for the input vector and a non-linear transformation (typically a softmax function) for the vector after the linear transformation. The vector after the non-linear conversion can be interpreted as the posterior probability value of the next word, but the weighting parameter itself used for the linear conversion cannot be given meaning. Therefore, for example, the operation of applying the original neural language model using the text data of another domain B to the neural language model learned using the text data of a certain domain A cannot be executed.

  An object of the present invention is to realize a model learning technique capable of performing model learning by using learning data having a domain different from that of learning data used at the time of learning in sequence modeling using a neural network.

  In order to solve the above problems, the acoustic model learning device according to the first aspect of the present invention uses an acoustic feature vector as an input, and a posterior probability vector for an output symbol corresponding to the acoustic feature vector, and the output symbol is an empty symbol. An empty model probability that represents a certain probability, and an acoustic model storage unit that stores an acoustic model using a neural network that outputs, and an a posteriori probability vector and an empty symbol probability by inputting an acoustic feature vector extracted from learning speech into the neural network. And a context-preserving vector that holds the posterior probability vector output by the neural network at the previous time or the posterior probability vector output by the neural network at the current time based on the empty symbol probability. The context-saving vector calculation part to calculate and the context-saving vector The context-preserving vector concatenation unit that linearly transforms the context-preserving vector using the weight parameter each time the toll is calculated and adds it to the output of the final hidden layer of the neural network and concatenates it to the output layer of the neural network, and A probabilistic model generator that generates a probabilistic model parameter using a probabilistic model from a set of text data, and a weight that calculates the weight parameter equivalent amount by performing inverse calculation of the nonlinear transformation included in the neural network for the probabilistic model parameter. It includes a parameter inverse calculation unit, and a weight parameter interpolation unit that interpolates the weight parameter used in the linear conversion and the weight parameter equivalent amount.

  A model learning device according to a second aspect of the present invention linearly transforms an input vector using a weighting parameter learned from first learning data that is a set of discrete value sequences belonging to a first domain, and after linear transformation Probability of generating a probabilistic model parameter using a probabilistic model from a neural network storage unit that stores a neural network that performs a non-linear transformation on the input vector and a second learning data that is a set of discrete value sequences belonging to the second domain A model generation unit, a weight parameter inverse calculation unit that performs a non-linear transformation inverse calculation on a stochastic model parameter, and calculates a weight parameter equivalent amount, and a weight that interpolates a weight parameter used in linear transformation and a weight parameter equivalent amount. And a parameter interpolation unit.

  According to the present invention, the weight parameter equivalent amount can be inversely calculated from the probabilistic model parameter and interpolated with the weight parameter of the linear transformation used in the neural network. Model learning can be performed using learning data of a domain different from the learning data.

FIG. 1 is a diagram illustrating a functional configuration of the model learning device. FIG. 2 is a diagram illustrating a processing procedure of the model learning method. FIG. 3 is a diagram illustrating a functional configuration of the acoustic model learning device. FIG. 4 is a diagram illustrating a functional configuration of the voice recognition device. FIG. 5 is a diagram illustrating a processing procedure of the acoustic model learning method. FIG. 6 is a diagram illustrating a processing procedure of the voice recognition method. FIG. 7: is a figure which illustrates the calculation method of a context preservation | save vector. FIG. 8 is a diagram illustrating a configuration in which context-preserving vectors are linearly converted and connected. FIG. 9 is a diagram illustrating a functional configuration of the weight parameter interpolation device. FIG. 10 is a diagram illustrating a processing procedure of the weight parameter interpolation method.

  The symbol "^" used in this specification is supposed to be written right above the character immediately after it, but is written immediately before the character due to the limitation of text notation. In the mathematical formula, these symbols are described at their original positions, that is, directly above the characters. For example, the notation "^ x" is

Is represented.

  Hereinafter, embodiments of the present invention will be described in detail. In the drawings, components having the same function are denoted by the same reference numerals, and duplicate description will be omitted.

[First embodiment]
The first embodiment is a model learning device and method for performing model learning on a neural language model that has been trained using text data of a certain domain, using text data of another domain. In the first embodiment, a model corresponding to 2-gram (bigram) is targeted. This model uses, for example, a tensor of rank 2 (that is, a matrix), has a dimension number equal to the vocabulary number, and inputs a vector having a dimension corresponding to the appearing word of 1 and other dimensions of 0, After performing the product of the input vector and the weight parameter matrix, the softmax function is applied and the posterior probability vector of the next word is output. However, the rank of the tensor can be a target even if it is an arbitrary natural number.

  As shown in FIG. 1, the model learning device of the first embodiment includes a first learning data storage unit 10, a neural language model generation unit 11, a neural language model storage unit 12, a second learning data storage unit 20, and a frequency tensor generation. It includes a unit 21, a probabilistic language model generation unit 22, a weight parameter inverse calculation unit 23, a weight parameter correction unit 24, and a weight parameter interpolation unit 25. However, the weight parameter correction unit 24 is not an essential component and can be omitted. The model learning device of the first embodiment realizes the model learning method of the first embodiment by performing the processing of each step shown in FIG.

  The model learning device is, for example, a special program configured by loading a special program into a known or dedicated computer having a central processing unit (CPU), a main storage device (RAM: Random Access Memory), and the like. It is a device. The model learning device executes each process under the control of the central processing unit, for example. The data input to the model learning device and the data obtained in each process are stored in, for example, the main storage device, and the data stored in the main storage device is read as needed and used for other processes. It Further, at least a part of each processing unit of the model learning device may be configured by hardware such as an integrated circuit. Each storage unit included in the model learning device is, for example, a main storage device such as a RAM (Random Access Memory), an auxiliary storage device configured by a semiconductor memory element such as a hard disk, an optical disk, or a flash memory (Flash Memory), or a relational storage device. It can be configured by middleware such as a database or key-value store. Each storage unit included in the model learning device has only to be logically divided, and may be stored in one physical storage device.

  The processing procedure of the model learning method executed by the model learning device of the first embodiment will be described below with reference to FIG.

  The first learning data storage unit 10 stores first domain learning data obtained by collecting a plurality of text data belonging to a certain domain (hereinafter referred to as a first domain). The second learning data storage unit 20 stores second domain learning data obtained by collecting a plurality of text data belonging to a domain different from the first domain (hereinafter referred to as a second domain). It is sufficient that each text data has a sufficient amount for learning the language model.

  In step S11, the neural language model generation unit 11 reads the first domain learning data stored in the first learning data storage unit 10 and learns the neural language model using the first domain learning data. The neural language model generation unit 11 stores the learned neural language model in the neural language model storage unit 12.

  In step S21, the frequency tensor generation unit 21 reads the second domain learning data stored in the second learning data storage unit 20, totals the concatenation frequencies of words included in the second domain learning data, and calculates the frequency tensor. To generate. The frequency tensor generation unit 21 outputs the generated frequency tensor to the stochastic language model generation unit 22. This frequency tensor becomes a frequency table in the case of a bigram model, for example.

  In step S22, the probabilistic language model generation unit 22 receives the frequency tensor from the frequency tensor generation unit 21, and generates a probabilistic model parameter tensor from the frequency tensor using the probabilistic language model. At this time, as the probabilistic language model, for example, a general one such as an n-gram model may be used. Further, at that time, a general operation such as smoothing may be included. The probabilistic language model generation unit 22 outputs the generated probabilistic model parameter tensor to the weight parameter inverse calculation unit 23.

  In step S23, the weight parameter inverse calculation unit 23 receives the probabilistic model parameter tensor from the probabilistic language model generation unit 22, performs inverse calculation of the nonlinear transformation included in the neural language model on the probabilistic model parameter tensor, and An amount corresponding to the weight parameter used in the linear conversion included in the language model (hereinafter referred to as a weight parameter equivalent amount) is calculated. The weight parameter inverse calculation unit 23 outputs the calculated weight parameter equivalent amount to the weight parameter correction unit 24. When the weight parameter correction unit 24 is omitted, the calculated weight parameter equivalent amount is output to the weight parameter interpolation unit 25.

  The inverse calculation of the weight parameter equivalent amount is performed as follows. First, the softmax function is shown in equation (1). Here, D is the number of dimensions.

This softmax function has the property that the output does not change even if a constant is added to the input. This relationship will be specifically shown. It is assumed that the output vector of the probabilistic model is a fixed vector with P = [p ₁ , p ₂ , ..., p _d , ..., p _D ] ^T. Here, [...] ^T represents the transpose of the vector. Substituting the output vector P into the softmax function of equation (1) yields equation (2).

  Since the denominator of equation (2) is a constant for dimension d,

Therefore, the equations (4) to (6) are established.

Equation (6) uses c as a parameter, and points [ln (p ₁ ), ln (p ₂ ) on the D-dimensional space of the input vector X = [x ₁ , x ₂ ,…, x _D ] ^T. ,…, Ln (p _D )] ^T is a straight line equation parallel to the vector [1, 1,…, 1] ^T. That is, it can be seen that any point on this straight line gives the same output vector P = [p ₁ , p ₂ , ..., p _D ] ^T. Here, an arbitrary vector can be used to determine an appropriate vector ^ X = [^ x ₁ ,, ^ x ₂ , ..., ^ x _D ] ^T , but here, using the square norm minimization criterion, || [x ₁ , x ₂ , ..., x _D ] ^T || The one that minimizes ₂ , that is, the one that has the shortest length is adopted. When the optimal parameter c is represented by ^ c, equations (7) to (9) are established.

  By substituting equation (9) into equation (6), the final inverse calculation equation (10) is determined.

That is, according to the following equation (11), weighting is performed from the probability model parameter tensor for each posterior probability vector P = [p ₁ , p ₂ ,…, p _D ] ^T of the next word when the previous word string is given. The inverse calculation to the parameter equivalent ^ X = [^ x ₁ , ^ x ₂ ,…, ^ x _D ] ^T can be performed.

  In step S24, the weighting parameter correction unit 24 receives the weighting parameter equivalent amount from the weighting parameter inverse calculation unit 23, and the weighting parameter of the neural language model stored in the neural language model storage unit 12 and the distribution parameter (average and The weight parameter equivalent amount is corrected so that the variance) is matched. The weight parameter correction unit 24 outputs the corrected weight parameter equivalent amount to the weight parameter interpolation unit 25.

Theoretically, the amount calculated by equation (11) is the amount corresponding to the weighting parameter, but in practice, the range of values may shift significantly. Therefore, heuristically adjust the mean and variance to the weighting parameters of the neural language model. Specifically, the weight parameter inverse calculation unit 23 normalizes the average of the weight parameter equivalent amounts to 0 and the standard deviation to 1, and multiplies the standard deviation calculated from the weight parameter of the corresponding neural language model to obtain the same weight. Correction is performed by adding the average calculated from the parameters. That is, the average of the weight parameter equivalents calculated by the weight parameter inverse calculation unit 23 is μ _p , the standard deviation is σ _p , the average of the corresponding weight parameters is μ _n , and the standard deviation is σ _n, and d = 1, 2 , ..., the D, the weighting parameter corresponding amount by calculating equation (12) ^ X = [^ x 1, ^ x 2, ..., ^ x D] to correct ^T.

  In step S25, the weight parameter interpolating unit 25 receives the weight parameter equivalent amount from the weight parameter correcting unit 24 or the weight parameter inverse calculating unit 23, and the weight parameter of the neural language model stored in the neural language model storage unit 12, A new weight parameter is obtained by linearly interpolating the input weight parameter equivalent amount. The weight parameter interpolation unit 25 updates the weight parameter of the neural language model stored in the neural language model storage unit 12 to a new weight parameter.

[Modification 1]
When a word that is not included in the second domain learning data and that is included in the neural language model (hereinafter, referred to as a mismatch word) exists, the frequency tensor generation unit 21 uses the following method to identify the mismatch word. A virtual frequency is defined and a frequency tensor is generated. It is desirable that the posterior probabilities output by the probabilistic language model for words not included in the second domain learning data do not significantly change from the posterior probabilities output by the original neural language model. Therefore, we use the constraint that there is no change in the posterior probability of mismatched words after the softmax function is multiplied by the weight parameter of the neural language model. This constraint can be easily solved when there is only one mismatch word. Specifically, assume that the input vector of the softmax function is X = [x ₁ , x ₂ , ,,, x _D ] ^T , and the last element x _D is the frequency of mismatch words, and we want to obtain a virtual value. . For the remaining elements x ₁ , x ₂ , ..., x _D-1, it is assumed that specific frequencies have been obtained by aggregating the second domain learning data. The output vector is Y = [y ₁ , y ₂ , ..., y _D ] ^T , and the last element y _D is the posterior probability of the mismatch word obtained by multiplying the weight parameter of the neural language model by the softmax function. Do not change. This relationship is expressed by equation (13).

  this is,

Becomes Equation (14) is calculated for each condition of previously appeared words, and the obtained x _D is used as the virtual frequency of mismatched words. When the number of mismatched words is not one but an arbitrary number, the simultaneous equations established from the above relationships may be solved and used as the virtual frequency of each mismatched word.

[Modification 2]
In the first embodiment, the weight parameter inverse calculation unit 23 uses the logarithmic function ln whose base is the natural logarithm, but a logarithmic function whose base is an arbitrary value may be used.

[Modification 3]
In the first embodiment, the weight parameter inverse calculation unit 23 uses the square norm minimization criterion to minimize a vector ^ X = [that minimizes || [x ₁ , x ₂ , ..., x _D ] ^T || ₂ . Although ^ x ₁ , ^ x ₂ , ..., ^ x _D ] ^T is defined as the weight parameter equivalent amount, any criterion may be used as the criterion for defining the vector ^ X. For example, an arbitrary Lp norm may be used, or the error may be reduced so as to approach the weight parameter of the neural language model.

[Modification 4]
The weighting parameter correction unit 24 may change the width that the value of the weighting parameter equivalent amount can take, using an arbitrary function such as a fixed function or a function using the parameters of the neural language model.

[Modification 5]
In the first embodiment, the weight parameter interpolating unit 25 is configured to linearly interpolate the weight parameter equivalent amount and the weight parameter of the neural language model. However, the interpolation may be performed by any general interpolation method, for example, geometric mean. You can go.

[Modification 6]
In the first embodiment, the concatenation frequency is totaled for the word string, but any discrete value series such as a character or a phoneme string may be targeted.

[Modification 7]
In the first embodiment, the neural language model, that is, the language model based on the neural network is targeted, but the weighting parameters of a part or all of the neural network for the purpose of discriminating other arbitrary discrete value series may be targeted.

[Modification 8]
In the first embodiment, the probabilistic language model generation unit 22 is configured to output the probabilistic model parameter tensor using the probabilistic language model. However, for example, RNN or LSTM predicts the next word from the previous word. The output probability value sequence may be used instead of the probabilistic model parameter tensor by using any modeling that outputs the probability of performing.

[Modification 9]
In the first embodiment, the configuration in which the model learning is performed by using the weight parameter of the neural network learned from the first domain learning data as an initial value has been described, but the weight parameter inverse calculation unit 23 calculates (that is, the second domain learning data). The model learning may be performed with the weight parameter equivalent amount (inversely calculated for the output vector of the probabilistic model learned from the above) as the initial value. Further, model learning may be performed by using a Gaussian distribution having the weight parameter equivalent calculated by the weight parameter inverse calculation unit 23 as an average as a prior distribution.

[Second embodiment]
In the first embodiment, as an example of sequence modeling, a general configuration example in which the present invention is applied to a language model by a neural network has been described. In the second embodiment, as a more practical configuration example, a configuration example in which the present invention is applied to an acoustic model using a CTC neural network will be described.

  Connectionist Temporal Classification (CTC), which is mainly used for speech recognition, is a kind of sequence conversion model by machine learning using neural network (NN: Neural Network) and is equivalent to Hidden Markov Model (HMM). It is a framework that allows the neural network to perform the function of. The NN-HMM hybrid method, which is commonly used in speech recognition, has a constraint that the length of the input sequence and the output sequence is one-to-one in the acoustic model that converts sounds into symbols. On the other hand, in CTC, the NN acoustic model can be transformed by introducing an empty symbol that represents a blank in addition to the usual output symbol. Therefore, in the case of speech recognition, it is possible to learn an acoustic model or a speech recognizer by directly inputting an acoustic feature vector for each unit time (hereinafter, also referred to as a frame) using phonemes, characters, words, etc. as output sequences. Yes (see Reference 1).

[Reference 1] Yajie Miao, Mohammad Gowayyed, and Florian Metze, "EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding", 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), IEEE , 2015.
The voice recognition system of the second embodiment includes, for example, an acoustic model learning device and a voice recognition device. The acoustic model learning device uses a learning data including learning voices and text information (for example, characters, phonemes, HMM states, and the like, symbol information of a voice conversion destination) related to each learning voice, and generates an acoustic sound generated from the learning voice. Along with the feature vector, text information is input as the correct sequence of the transformation destination of the acoustic feature vector, and the acoustic model by CTC is learned using this pair. The voice recognition device performs voice recognition of the input voice using the acoustic model learned by the acoustic model learning device.

  As shown in FIG. 3, the acoustic model learning device includes a learning data storage unit 30, a context preserving vector generation unit 31, a posterior probability calculation unit 32, a context preserving vector calculation unit 33, a context preserving vector connection unit 34, an acoustic model storage unit. 35, a text data storage unit 40, a frequency tensor generation unit 41, a probabilistic language model generation unit 42, a weight parameter inverse calculation unit 43, a weight parameter correction unit 44, and a weight parameter interpolation unit 45. However, the weight parameter correction unit 44 is not an essential component and can be omitted. The acoustic model learning method of the second embodiment is realized by the acoustic model learning device of the second embodiment performing the processing of each step shown in FIG.

  As shown in FIG. 4, the voice recognition device includes an acoustic model storage unit 35, a language model storage unit 36, and a voice recognition unit 37. The speech recognition method of the second embodiment is realized by the processing of each step shown in FIG. 6 by the speech recognition device of the second embodiment.

  In the acoustic model learning device and the speech recognition device, for example, a special program is loaded into a known or dedicated computer having a central processing unit (CPU), a main storage device (RAM: Random Access Memory), and the like. It is a special device configured. The acoustic model learning device and the speech recognition device execute each process under the control of the central processing unit, for example. The data input to the acoustic model learning device and the voice recognition device and the data obtained by each process are stored in, for example, the main storage device, and the data stored in the main storage device are read as necessary and It is used for processing. Further, at least a part of each processing unit of the acoustic model learning device and the voice recognition device may be configured by hardware such as an integrated circuit. Each storage unit included in the acoustic model learning device and the voice recognition device is, for example, a main storage device such as a RAM (Random Access Memory), an auxiliary device including a semiconductor memory device such as a hard disk, an optical disk, or a flash memory (Flash Memory). It can be configured by a storage device or middleware such as a relational database or a key-value store. Each storage unit included in the acoustic model learning device and the voice recognition device may be logically divided, and may be stored in one physical storage device.

  Hereinafter, the acoustic model learning method executed by the acoustic model learning device of the second embodiment will be described with reference to FIG.

  The learning data storage unit 30 of the acoustic model learning device stores learning data used for learning the acoustic model. The learning data includes learning voices and text information (for example, HMM state, phonemes, characters, words, etc.) regarding the content of each learning voice. The learning data may be collected manually or may be automatically generated using a known learning data generation technique. A sufficient amount of learning data is prepared in advance and stored in the learning data storage unit 30.

  An acoustic model based on CTC is stored in the acoustic model storage unit 35 of the acoustic model learning device. In the initial state, a conventional CTC acoustic model may be prepared and stored.

In step S31, the context-preserving vector generation unit 31 generates the context-preserving vector K ₀ = [k _0,1 , k _0,2 , k _0,3 , k _0,4 , ...] ^T. Each dimension of the context-preserving vector K ₀ is initialized to an arbitrary value (for example, 0 or 1). The context-preserving vector generation unit 31 can operate without input, but may use a value used for initialization (for example, 0 or 1) as input. The generated context-saving vector K ₀ is connected to an arbitrary position in the input layer or hidden layer of the CTC neural network in the acoustic model stored in the acoustic model storage unit 35. The concatenation is to use the information of the context-preserving vector to influence the parameters of the CTC neural network. For example, a context-preserving vector is connected before or after the hidden layer of the original CTC neural network, and a vector that is the sum of the length of the hidden layer vector and the length of the context-preserving vector is newly generated. Alternatively, a matrix for converting each context-preserving vector into the size of the hidden layer or the output layer is prepared separately, and the result of the conversion by the matrix is added to the vector of the output layer of the original CTC neural network.

In step S32, the posterior probability calculating unit 32 causes the acoustic feature vector X _{t + 1} = [x _{t + 1, at} time t + 1 (t ≧ 0) extracted from the learning voice stored in the learning data storage unit 30 _{. 1} , x _{t + 1,2} , x _{t + 1,3} , x _{t + 1,4} , ...] ^T and the context preservation vector K _{t at} the previous time t are stored in the acoustic model storage unit 35. acoustic model of type into CTC neural network, the posterior probability vector for the output symbol _{C t + 1 = [c t} + 1,1, c t + 1,2 you _{are, c t + 1,3, c t} + 1,4 , ...] ^T (hereinafter referred to as an output posterior probability vector) and an empty symbol probability φ _{t + 1} representing the probability that the output symbol is an empty symbol. The output posterior probability vector C _{t + 1} and the correct sequence are input to the error function of the CTC neural network and used to update the parameters of the CTC neural network. The posterior probability calculation unit 32 outputs the output posterior probability vector C _{t + 1} and the empty symbol probability φ _{t + 1} to the context-preserving vector calculation unit 33.

In step S33, the context-preserving vector calculation unit 33 receives the output posterior probability vector C _{t + 1} and the empty symbol probability φ _{t + 1} from the posterior probability calculation unit 32, and outputs the previous one based on the empty symbol probability φ _{t + 1.} Context-preserving vector K _t = [k _{t, 1} , k _{t, 2} , k _{t, 3} , k _{t, 4} ,…] ^T is updated to the context-preserving vector K at the current time t + 1. _{t + 1} = [k _{t + 1,1} , k _{t + 1,2} , k _{t + 1,3} , k _{t + 1,4} , ...] ^T is generated. For example, when the empty symbol probability φ _{t + 1} indicates that the empty symbol probability φ _{t + 1} is an empty symbol, the context-preserving vector calculation unit 33 calculates the output posterior probability vector when the CTC neural network finally outputs a symbol other than the empty symbol. If it holds and the empty symbol probability φ _{t + 1} indicates that it is not an empty symbol, the context-preserving vector K _{t + 1} is set so as to record the output posterior probability vector C _{t + 1} output by the CTC neural network this time. calculate. The context-saving vector calculation unit 33 outputs the calculated context-saving vector K _{t + 1} to the context-saving vector connecting unit 34.

The calculation of the context-preserving vector uses, for example, an update rule similar to a flip-flop circuit in an electronic circuit. The empty symbol probability φ _{t + 1} output by the CTC neural network takes a value from 0 to 1, and the closer it is to 1, the higher the possibility of an empty symbol. For simplicity, considering the case of both ends,

If it is not an empty symbol (φ _{t + 1} = 0), the content of the output posterior probability vector C _{t + 1 at} the current time t + 1 is recorded, and if it is an empty symbol (φ _{t + 1} = In 1), the contents of the context-preserving vector K _{t at} the previous time t are retained. Specifically, if the output empty symbol probability φ _{t + 1} is greater than or equal to a predetermined threshold value, then φ _{t + 1} = 1, and if the empty symbol probability φ _{t + 1} is less than the predetermined threshold value, then φ _{t + 1} The empty symbol probability φ _{t + 1} may be binarized by a means such as = 0, and an update rule such as equation (15) may be used. Further, Equation (16) may be defined and calculated as an update rule that is naturally and continuously expanded to include both ends.

However,

Represents the product of each element. [1,…, 1] ^T represents a vector in which 1s are arranged vertically. Φ _{t + 1} is a vertical vector in which empty symbol probabilities φ _{t + 1} are arranged by the number of dimensions of the output posterior probability vector C _{t + 1} , that is, Φ _{t + 1} = [φ _{t + 1} , ..., φ _{t + 1} ]. ^T.

  Equation (16) can also be rewritten as equation (17).

The context-preserving vector calculation unit 33 replaces the posterior probability vector C _{t + 1} for the output symbol with other internal layers such as the hidden layer of the neural network such as the input layer, the first hidden layer, and the final hidden layer shown in FIG. 7. A parameter or the input acoustic feature vector X _{t + 1} may be used. The context-preserving vector calculation unit 33 may reduce the dimension of the context-preserving vector K _{t + 1} by using a known dimension reduction unit and may output it, or may be fixed in advance such as averaging, normalization, or discretization. You may output after performing the conversion by the function.

In step S34, the context saved vector connecting portion 34, the context save vector receive context save vector K _{t + 1} from the calculation unit 33, an acoustic model context stored in the storage unit 35 relative to the CTC neural network stored vector K _t Concatenate ₊₁ . Concatenation of the context-preserving vectors is performed each time the context-preserving vector updated at each time is received. When connecting the context-preserving vectors, as shown in FIG. 8, after applying a linear transformation using a matrix representing the Markov property of the output symbol sequence, the CTC neural network is directly integrated into the output layer by addition or the like. To create. When applying a linear transformation using a matrix, the matrix into which the context-preserving vector is input is a matrix that represents the Markov property of the output symbol sequence that reflects information from an external language resource or the like. This matrix uses the symbol transition probabilities calculated by aggregating external language resources, etc., and performs information from external language resources, etc., by operations that affect the matrix values from the outside (for example, linear interpolation). Is configured to reflect.

The matrix representing the Markov property of the output symbol sequence used when linearly converting the context-preserving vector K _{t + 1} is the one learned by the processing of steps S41 to S45. The learning of the matrix representing the Markov property of the output symbol sequence may be performed at any timing before connecting the context-preserving vector K _{t + 1} to the CTC neural network. Hereinafter, the matrix representing the Markov property of the output symbol sequence is also referred to as a weight parameter.

  The text data storage unit 40 stores a plurality of text data including at least one word string. This text data may be manually collected, may be automatically collected from a language resource that can be obtained from the Internet, or may be automatically generated by using a known learning data generation technique. A sufficient amount of text data is prepared in advance and stored in the text data storage unit 40.

  In step S41, the frequency tensor generation unit 41 reads the text data stored in the text data storage unit 40, totals the concatenation frequencies of words included in the text data, and generates a frequency tensor. The frequency tensor generation unit 41 outputs the generated frequency tensor to the stochastic language model generation unit 42.

  In step S42, the probabilistic language model generation unit 42 receives the frequency tensor from the frequency tensor generation unit 41 and generates a probabilistic model parameter tensor from the frequency tensor using the probabilistic language model. At this time, as the probabilistic language model, for example, a general one such as an n-gram model may be used. Further, at that time, a general operation such as smoothing may be included. The probabilistic language model generation unit 42 outputs the generated probabilistic model parameter tensor to the weight parameter inverse calculation unit 43.

  In step S43, the weight parameter inverse calculation unit 43 receives the stochastic model parameter tensor from the stochastic language model generation unit 42, performs inverse calculation of the nonlinear transformation included in the CTC neural network on the stochastic model parameter tensor, and outputs the result. An amount (hereinafter, referred to as a weight parameter equivalent amount) corresponding to a matrix (that is, a weight parameter) representing the Markov property of the symbol sequence is generated. The inverse calculation of the weight parameter equivalent amount may be performed by the equation (11) as in the weight parameter inverse calculation unit of the first embodiment. The weight parameter inverse calculation unit 43 outputs the calculated weight parameter equivalent amount to the weight parameter correction unit 44. When the weight parameter correction unit 44 is omitted, the calculated weight parameter equivalent amount is output to the weight parameter interpolation unit 45.

  In step S44, the weighting parameter correction unit 44 receives the weighting parameter equivalent amount from the weighting parameter inverse calculation unit 43 so that the weighting parameter stored in the acoustic model storage unit 35 matches the distribution parameter (mean and variance). Then, the weight parameter equivalent amount is corrected. The correction of the weight parameter equivalent amount may be performed by the equation (12) as in the weight parameter correction unit of the first embodiment. The weight parameter correction unit 44 outputs the corrected weight parameter equivalent amount to the weight parameter interpolation unit 45.

  In step S45, the weight parameter interpolation unit 45 receives the weight parameter equivalent amount from the weight parameter correction unit 44 or the weight parameter inverse calculation unit 43, and inputs the weight parameter of the CTC neural network stored in the acoustic model storage unit 35 and the input. A new weight parameter is obtained by linearly interpolating the obtained weight parameter equivalent amount. The weight parameter interpolating unit 45 updates the weight parameter of the CTC neural network stored in the acoustic model storage unit 35 to a new weight parameter.

  Hereinafter, the voice recognition method executed by the voice recognition device of the second embodiment will be described with reference to FIG.

  The acoustic model storage unit 35 of the voice recognition device stores an acoustic model based on CTC learned by the acoustic model learning device.

  The language model storage unit 36 of the voice recognition device stores a language model used for voice recognition. Any type of language model may be used as long as it can be used when the voice recognition unit 37 performs voice recognition.

  In step S37, the voice recognition unit 37 performs voice recognition on the input voice using the acoustic model stored in the acoustic model storage unit 35 and the language model stored in the language model storage unit 36, and outputs the voice recognition result. Output. The voice recognition unit 37 may be a voice recognizer that performs voice recognition using a single CTC acoustic model, or a voice recognizer that combines the CTC acoustic model with a weighted finite state transducer (WFST). May be

  In the second embodiment, the speech recognition system in which the acoustic model learning device and the speech recognition device are configured as separate devices has been described. However, a single speech having all the functions of the acoustic model learning device and the speech recognition device is provided. It may be configured as a recognition device. That is, the learning data storage unit 30, the context preservation vector generation unit 31, the posterior probability calculation unit 32, the context preservation vector calculation unit 33, the context preservation vector connection unit 34, the acoustic model storage unit 35, the language model storage unit 36, the speech recognition unit. 37, a text data storage unit 40, a frequency tensor generation unit 41, a probabilistic language model generation unit 42, a weight parameter inverse calculation unit 43, a weight parameter correction unit 44, and a weight parameter interpolation unit 45. Is also possible.

  With the above configuration, the acoustic model learning device of the second embodiment uses the Markov property of the output symbol sequence by using text data that is an external language resource different from the learning data used for learning the acoustic model by CTC. You can learn the matrix that represents. Since the learning data is the learning voice and the text information of the transcription, the cost required for the collection is large, but since the existing resources such as the Internet can be used for the text data, it is possible to collect a sufficient amount at a small cost. It is possible. As a result, according to the acoustic model learning device of the second embodiment, it is possible to efficiently and highly accurately learn the acoustic model by CTC.

[Third embodiment]
A weight parameter interpolating device that configures the weight parameter inverse calculating unit, the weight parameter correcting unit, and the weight parameter interpolating unit described in each of the above embodiments as independent devices, and interpolates the weight parameter of the neural network using the probabilistic model parameters as inputs Can also be configured.

  As shown in FIG. 9, the weight parameter interpolation device of the third embodiment includes a weight parameter storage unit 50, a weight parameter inverse calculation unit 51, a weight parameter correction unit 52, and a weight parameter interpolation unit 53. However, the weight parameter correction unit 52 is not an indispensable component and can be omitted. The weighting parameter interpolation device of the third embodiment implements the weighting parameter interpolation method of the third embodiment by performing the processing of each step shown in FIG.

  The weight parameter interpolating device is a special program configured by loading a special program into a known or dedicated computer having a central processing unit (CPU), a main storage device (RAM: Random Access Memory), or the like. It is a device. The weight parameter interpolating device executes each process under the control of the central processing unit, for example. The data input to the weight parameter interpolating device and the data obtained in each process are stored in, for example, the main storage device, and the data stored in the main storage device is read out as needed and used for other processes. To be done. Further, at least a part of each processing unit of the weight parameter interpolating device may be configured by hardware such as an integrated circuit. Each storage unit included in the weight parameter interpolating device is, for example, a main storage device such as a RAM (Random Access Memory), an auxiliary storage device configured by a semiconductor memory device such as a hard disk, an optical disk, or a flash memory (Flash Memory), or It can be configured by middleware such as relational database and key-value store.

  Hereinafter, the processing procedure of the weight parameter interpolation method executed by the weight parameter interpolation device of the third embodiment will be described with reference to FIG. 10.

  The weight parameter storage unit 50 stores weight parameters used in linear conversion included in a neural network learned from learning data belonging to a certain domain (hereinafter, first learning data).

  In step S51, the weight parameter inverse calculation unit 51 performs inverse calculation of the nonlinear transformation of the neural network on the stochastic model parameter tensor input to the weight parameter interpolating device, and generates a weight parameter equivalent amount. The probabilistic model parameter tensor is a concatenation of learning data (hereinafter, second learning data) belonging to a domain different from that of the first learning data for each processing unit (for example, an arbitrary discrete value sequence such as a word, a morpheme, a character, a phoneme, etc.). A frequency tensor representing a frequency is generated, and a probability model is generated from the frequency tensor. The inverse calculation of the weight parameter equivalent amount may be performed by the equation (11) as in the weight parameter inverse calculation unit of the first embodiment. The weight parameter inverse calculation unit 51 outputs the calculated weight parameter equivalent amount to the weight parameter correction unit 52. When the weight parameter correction unit 52 is omitted, the calculated weight parameter equivalent amount is output to the weight parameter interpolation unit 53.

  In step S52, the weighting parameter correction unit 52 receives the weighting parameter equivalent amount from the weighting parameter inverse calculation unit 51 so that the weighting parameter stored in the weighting parameter storage unit 50 matches the distribution parameter (mean and variance). Then, the weight parameter equivalent amount is corrected. The correction of the weight parameter equivalent amount may be performed by the equation (12) as in the weight parameter correction unit of the first embodiment. The weight parameter correction unit 52 outputs the corrected weight parameter equivalent amount to the weight parameter interpolation unit 53.

  In step S53, the weight parameter interpolation unit 53 receives the weight parameter equivalent amount from the weight parameter correction unit 52 or the weight parameter inverse calculation unit 51, and the weight parameter stored in the weight parameter storage unit 50 and the input weight parameter. A new weighting parameter is obtained by linearly interpolating with The weight parameter interpolation unit 53 outputs the new weight parameter as an output of the weight parameter interpolation device.

[Points of the Invention]
Since the softmax function used in the neural network is not a single shot, there is no inverse function. However, the probability vector can be converted into an amount corresponding to the weighting parameter used in the neural network by considering an operation (that is, inverse calculation) corresponding to that. The model can be adapted to another domain by interpolating the weight parameter used in the original neural network and the parameter tensor of the probabilistic model calculated from the text data of another domain converted into an amount corresponding to the weight parameter. In addition, learning using this property, such as performing re-learning with the conversion amount as an initial value or a constraint, can improve a model that is not simply rescoring of output.

  Although the embodiments of the present invention have been described above, the specific configuration is not limited to these embodiments, and even if the design is appropriately changed without departing from the gist of the present invention, Needless to say, it is included in the present invention. The various kinds of processing described in the embodiments may be executed not only in time series according to the order described, but also in parallel or individually according to the processing capability of the device that executes the processing or the need.

[Program, recording medium]
When various processing functions in each device described in the above embodiments are realized by a computer, processing contents of functions that each device should have are described by a program. By executing this program on a computer, various processing functions of the above-described devices are realized on the computer.

  The program describing the processing contents can be recorded in a computer-readable recording medium. The computer-readable recording medium may be any recording medium such as a magnetic recording device, an optical disc, a magneto-optical recording medium, or a semiconductor memory.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or a CD-ROM in which the program is recorded. Further, the program may be stored in a storage device of a server computer and transferred from the server computer to another computer via a network to distribute the program.

  A computer that executes such a program first stores, for example, the program recorded on a portable recording medium or the program transferred from the server computer in its own storage device. Then, when executing the process, this computer reads the program stored in its own storage device and executes the process according to the read program. As another execution form of this program, a computer may directly read the program from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to this computer. Each time, the processing according to the received program may be sequentially executed. In addition, a configuration in which the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes a processing function only by the execution instruction and result acquisition without transferring the program from the server computer to this computer May be It should be noted that the program in this embodiment includes information that is used for processing by an electronic computer and that conforms to the program (data that is not a direct command to a computer but has the property of defining computer processing).

  Further, in this embodiment, the present apparatus is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be implemented by hardware.

10 First Learning Data Storage Unit 11 Neural Language Model Generation Unit 12 Neural Language Model Storage Unit 20 Second Learning Data Storage Unit 21 Frequency Tensor Generation Unit 22 Stochastic Language Model Generation Unit 23 Weight Parameter Inverse Calculation Unit 24 Weight Parameter Correction Unit 25 Weight parameter interpolation unit 30 Learning data storage unit 31 Context-preserved vector generation unit 32 Posterior probability calculation unit 33 Context-preserved vector calculation unit 34 Context-preserved vector connection unit 35 Acoustic model storage unit 36 Language model storage unit 37 Speech recognition unit 40 Text data storage Unit 41 frequency tensor generation unit 42 stochastic language model generation unit 43 weight parameter inverse calculation unit 44 weight parameter correction unit 45 weight parameter interpolation unit 50 weight parameter storage unit 51 weight parameter inverse calculation unit 52 weight parameter correction unit 53 weight parameter interpolation unit

Claims (8)

Store an acoustic model using a neural network that receives an acoustic feature vector as an input and outputs a posterior probability vector for an output symbol corresponding to the acoustic feature vector and an empty symbol probability that represents the probability that the output symbol is an empty symbol. An acoustic model storage unit that
A posterior probability calculation unit that inputs the acoustic feature vector extracted from the learning voice to the neural network to obtain the posterior probability vector and the empty symbol probability,
Context-preserving vector calculation for selecting and retaining the posterior probability vector output by the neural network at the previous time or the posterior probability vector output by the neural network at the current time based on the empty symbol probability Department,
Each time the context-preserving vector is calculated, the context-preserving vector is linearly transformed by using a weight parameter, added to the output of the final hidden layer of the neural network, and connected to the output layer of the neural network. Department,
A probabilistic model generation unit that generates a probabilistic model parameter using a probabilistic model from text data that is a set of word strings,
A weight parameter inverse calculation unit that performs inverse calculation of the non-linear transformation included in the neural network on the stochastic model parameter and calculates a weight parameter equivalent amount,
A weight parameter interpolating unit that interpolates the weight parameter used in the linear conversion and the weight parameter equivalent amount,
Acoustic model learning device including. A neural network that performs a linear transformation on the input vector using the weighting parameters learned from the first learning data, which is a set of discrete value sequences belonging to the first domain, and performs a nonlinear transformation on the input vector after the linear transformation. A neural network storage unit for storing,
A probabilistic model generation unit that generates a probabilistic model parameter using a probabilistic model from the second learning data that is a set of discrete value series belonging to the second domain,
An inverse calculation of the non-linear transformation is performed on the stochastic model parameter, and a weight parameter inverse calculation unit that calculates a weight parameter equivalent amount,
A weight parameter interpolating unit that interpolates the weight parameter used in the linear conversion and the weight parameter equivalent amount,
Model learning device including. The model learning device according to claim 2, wherein
A weight parameter correction unit that corrects the weight parameter equivalent amount so that the distribution parameter of the weight parameter equivalent amount matches the distribution parameter of the weight parameter,
Model learning device. The model learning device according to claim 2 or 3, wherein
The weighting parameter interpolating unit performs interpolation between the weighting parameter and the weighting parameter equivalent amount by linear interpolation or geometric mean.
Model learning device. The model learning device according to any one of claims 2 to 4,
The probability model generation unit, for the mismatch data included in the first learning data but not included in the second learning data, so that the appearance probability of the mismatch data after the nonlinear conversion of the weighting parameter does not change. The frequency of the mismatch data is determined, and the probability model parameter is generated.
Model learning device. The model learning device according to any one of claims 2 to 5,
The neural network is one in which the weight parameter equivalent amount is an initial value, or a Gaussian distribution in which the weight parameter equivalent amount is an average is learned as a prior probability.
Model learning device. The neural network storage unit performs a linear transformation on the input vector using the weighting parameter learned from the first learning data, which is a set of discrete value sequences belonging to the first domain, and nonlinearly transforms the input vector after the linear transformation. It stores the neural network that performs the conversion,
The probabilistic model generation unit generates a probabilistic model parameter using a probabilistic model from the second learning data that is a set of discrete value series belonging to the second domain,
The weight parameter inverse calculation unit performs the inverse calculation of the non-linear transformation on the stochastic model parameter to calculate the weight parameter equivalent amount,
A weight parameter interpolating unit interpolates the weight parameter used in the linear conversion and the weight parameter equivalent amount,
Model learning method.   A program for causing a computer to function as the acoustic model learning device according to claim 1 or the model learning device according to any one of claims 2 to 6.

Images