JP2021039215A - Learning device, learning method, and learning program - Google Patents

Abstract

To learn the parameters of a speech recognition model end-to-end by semi-supervised learning.SOLUTION: A learning device 20 learns, by a neural network (NN), the parameters of a speech recognition model that converts speech data into information that specifies a code string. A recognition error calculation unit 215 calculates, using data for learning, a speech recognition error that represents non-similarity between the estimation result of code string data estimated on the basis of information that specifies the code string obtained by converting speech data in the data for learning by the NN and the code string data in the data for learning. A restore error calculation unit 216 calculates, using unsupervised code string data, a restore error that represents non-similarity between the estimation result of code string data estimated on the basis of information that specifies the code string obtained by converting unsupervised code string data by the NN and the unsupervised code string data. An update unit 218 updates the parameters of the NN on the basis of the speech recognition error and the restore error.SELECTED DRAWING: Figure 3

Description

The present invention relates to a learning device, a learning method and a learning program.

Conventionally, a technique of learning a model using a neural network (hereinafter, may be referred to as NN) by machine learning is known. For example, there is known a method of learning a speech recognition model using end-to-end NN that can convert speech data into information that identifies a symbol string (posterior probability) by supervised learning (for example,). See Non-Patent Document 1).

Further, for example, a method of learning an NN that performs image conversion or an end-to-end NN that performs machine translation by unsupervised learning is known (see, for example, Non-Patent Document 2 or Non-Patent Document 3). ..

Further, for example, in an end-to-end NN speech recognition model, a method of learning RNN-LM (Recurrent Neural Network Language Model) used as a part of a decoder by unsupervised learning is known. (See, for example, Non-Patent Document 4).

J. Chorowski et al., “Attention-Based Models for Speech Recognition,” Advances in Neural Information Processing Systems 28 (NIPS 2015), pp. 577-585, 2015. (URL: http://papers.nips.cc/) paper / 5847-attention-based-models-for-speech-recognition) M.-Y. Liu, T. Breuel, and J. Kautz, “Unsupervised Image-to-Image Translation Networks,” in Neural Information Processing Systems, 2017, pp. 700-708. (URL: https: // papers. nips.cc/paper/6672-unsupervised-image-to-image-translation-networks.pdf) M. Artetxe, G. Labaka, E. Agirre, and K. Cho, “Unsupervised Neural Machine Translation,” International Conference on Learning Representation, 2018. (URL: https://arxiv.org/pdf/1710.11041.pdf) Takaaki Hori, Shinji Watanabe, Yu Zhang, William Chan, “Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM,” Interspeech 2017. (URL: https: // arxiv. org / pdf / 1706.02737.pdf)

Supervised learning has the advantage that the accuracy of the recognition model can be improved, but it has the disadvantage that it is difficult to prepare a large amount of supervised learning data. In addition, unsupervised learning has the disadvantage that the accuracy of the recognition model is inferior to that of supervised learning, but it has the advantage that training data can be easily prepared.

Semi-supervised learning is a learning method that has the merits of both supervised learning and unsupervised learning.

On the other hand, the techniques of Non-Patent Document 1 and Non-Patent Document 5 do not learn a speech recognition model using end-to-end NN by semi-supervised learning.

Further, in Non-Patent Document 3 and Non-Patent Document 4, end-to-end NN is learned by unsupervised learning, but the target end-to-end NN here is an input and output domain. Is the same. For example, in Non-Patent Document 3, both input and output are image data, and in Non-Patent Document 3, both input and output are text data.

On the other hand, in the end-to-end speech recognition model, since the input is speech data and the output is information that identifies text data (symbol sequence), the input and output domains are different. The unsupervised learning method of Non-Patent Document 3 and Non-Patent Document 4 cannot be applied to such end-to-end learning of NNs having different input and output domains.

In view of the above problems, it is an object of the present invention to provide a technique in which the parameters of a speech recognition model can be learned end-to-end by semi-supervised learning. Here, end-to-end learning is a method of learning all the parameters of the neural network at once based on the output data obtained by inputting the input data to the neural network.

In order to solve the above-mentioned problems and achieve the purpose, the learning device is a learning device that learns the parameters of a voice recognition model that converts voice data into information that identifies a symbol string by using a neural network. A symbol string estimated based on information that identifies a symbol string obtained by converting the unsupervised symbol string data by the neural network using the unsupervised symbol string data which is the symbol string data to which the data is not associated. A restoration error calculation unit that calculates a restoration error that represents the degree of dissimilarity between the data estimation result and the unsupervised neural network data, and a distribution of intermediate feature quantities obtained by converting the audio data given for learning. , The feature quantity error calculation unit that calculates the feature quantity error representing the dissimilarity with the distribution of the intermediate feature quantity obtained by converting the symbol string data given for learning, and the restoration error and the feature quantity error. It is characterized by having an update unit that updates the parameters of the neural network based on at least one of the above.

According to the present invention, the parameters of the speech recognition model can be learned end-to-end by semi-supervised learning.

FIG. 1 is a diagram showing an example of a configuration of a voice recognition device. FIG. 2 is a flowchart showing a processing flow of the voice recognition device. FIG. 3 is a diagram showing an example of the configuration of the learning device according to the first embodiment. FIG. 4 is a flowchart showing a processing flow of the learning device according to the first embodiment. FIG. 5 is a flowchart showing the flow of the recognition error calculation process according to the first embodiment. FIG. 6 is a flowchart showing the flow of the restoration error calculation process according to the first embodiment. FIG. 7 is a flowchart showing the flow of the feature amount error calculation process according to the first embodiment. FIG. 8 is a diagram showing an example of the configuration of the learning device according to the third embodiment. FIG. 9 is a diagram showing the results of the first experiment. FIG. 10 is a diagram showing the results of the second experiment. FIG. 11 is a diagram showing an example of a computer that executes a learning program. FIG. 12 is a diagram showing a configuration of a conventional learning device. FIG. 13 is a flowchart showing a processing flow of the conventional learning device.

Hereinafter, the learning device, the learning method, and the embodiment of the learning program according to the present application will be described in detail with reference to the drawings. The present invention is not limited to the embodiments described below.

[Voice recognition device configuration]
Here, before explaining the details of the learning device, a speech recognition device using an end-to-end speech recognition model (neural network) will be described. The voice recognition device performs voice recognition using a voice recognition model in which learned parameters are set in the learning device described later. As shown in FIG. 1, the voice recognition device 10 includes a control unit 11 and a storage unit 12.

The control unit 11 controls the entire voice recognition device 10. The control unit 11 is realized by a CPU (Central Processing Unit) or the like. The control unit 11 has a conversion unit 110 and a search unit 115. Further, the conversion unit 110 includes a voice feature amount coding unit 111, an intermediate feature amount coding unit 113, and a symbol string decoding unit 114. The storage unit 12 is a storage device such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive). The storage unit 12 stores the parameter Λ of the neural network.

The conversion unit 110 uses a neural network to convert voice data into intermediate features, and converts the intermediate features into "information that identifies a symbol string". At this time, the conversion unit 110 reads the learned neural network parameter Λ from the storage unit 12, and performs conversion according to the read parameter. Note that Λ is a set of parameters of a plurality of neural networks.

Here, the symbol string is a series of symbols including all symbols (symbols) such as alphabets, kanji, and spaces, words that are a series thereof, and symbols that indicate the end of the recognition result. The information that identifies the symbol string is the likelihood (posterior probability) when the input voice is converted into a certain symbol string.

Of the neural networks constituting the conversion unit 110, if the part that converts voice data into intermediate features is called the encoder layer and the part that converts intermediate features into posterior probabilities is called the decoder layer, the encoder layer is called the voice features. Corresponding to the coding unit 111 and the intermediate feature amount coding unit 113, the decoder layer corresponds to the symbol string decoding unit 114.

Here, the processing of each unit included in the conversion unit 110 will be described. The voice feature amount coding unit 111 converts (encodes) the input voice data x ″ into a voice feature amount. Further, the intermediate feature amount coding unit 113 converts (encodes) the voice feature amount into the intermediate feature amount. Further, the symbol string decoding unit 114 converts (decodes) the intermediate feature amount into information for specifying the symbol string. It is assumed that the parameters of the neural network used by the voice feature amount coding unit 111, the intermediate feature amount coding unit 113, and the symbol string decoding unit 114 are included in Λ.

The search unit 115 searches for the symbol string based on the information for specifying the symbol string. The symbol string that is the search result is the voice recognition result (estimated symbol string).

Here, the information for specifying the symbol string to be converted by the symbol string decoding unit 114 is the posterior probability p (y ″ ｜ x ″) of the estimated symbol string y ″ of the voice recognition result for the voice data x ″. Is. Therefore, the search unit 115 searches for a symbol string that maximizes the posterior probability p (y ″ | x ″) by beam search or the like.

For example, the search unit 115 selects one of the symbol string candidates whose likelihood magnitude of the first character is in the upper predetermined number, and further, in which the likelihood magnitude of the following symbol is in the upper predetermined number. By repeating the selection of, the symbol string to be output can be selected.

[Processing of voice recognition device]
The processing flow of the voice recognition device 10 will be described with reference to FIG. FIG. 2 is a flowchart showing a processing flow of the voice recognition device. As shown in FIG. 2, first, the voice recognition device 10 reads the parameters from the storage unit 12 (step S11). Further, the voice recognition device 10 accepts an input of voice data for recognition (step S12).

Here, the voice recognition device 10 converts the voice data into a voice feature amount (step S13). Next, the voice recognition device 10 converts the voice feature amount into the intermediate feature amount (step S14). Then, the voice recognition device 10 converts the intermediate feature amount into information for specifying the symbol string (step S15).

The voice recognition device 10 uses a neural network to perform the conversions in steps S13, S14, and S15. Further, the parameters of each neural network are read in step S11.

The voice recognition device 10 searches for the symbol string based on the information for identifying the converted symbol string (step S16). Then, the voice recognition device 10 outputs the symbol string obtained by the search as the voice recognition result (step S17).

[First Embodiment]
Next, the configuration of the learning device of the first embodiment will be described with reference to FIG. FIG. 3 is a diagram showing an example of the configuration of the learning device according to the first embodiment. As shown in FIG. 3, the learning device 20 has a control unit 21 and a storage unit 22.

The control unit 21 controls the entire learning device 20. The control unit 21 is realized by a CPU or the like. The control unit 21 includes a conversion unit 210, a recognition error calculation unit 215, a restoration error calculation unit 216, a feature amount error calculation unit 217, an update unit 218, and a determination unit 219. Further, the conversion unit 210 includes a voice feature amount coding unit 211, a symbol string feature amount coding unit 212, an intermediate feature amount coding unit 213, and a symbol string decoding unit 214. Of these, the voice feature amount coding unit 211, the intermediate feature amount coding unit 213, and the symbol string decoding unit 214 are the same neural networks as the neural network used in the conversion unit 110 of the voice recognition device 10. That is, the neural network of the learning device 20 is different in that it further includes a partial structure (symbol string feature amount coding unit 212) that is not used in the voice recognition device 10.

The storage unit 22 is a storage device such as an HDD and an SSD. The storage unit 22 stores the parameter Λ of the neural network. It is assumed that an appropriate value is set in advance for each initial value of the parameter Λ of the neural network.

Here, the data (learning data) input to the learning device 20 will be described. As learning data
(1) A supervised learning data set Z consisting of a set of audio data x'for learning and corresponding symbol string data y'(correct answer symbol string data), and
(2) A set T of learning data (unsupervised learning symbol string data) consisting of only symbol string data y for which there is no corresponding audio data, and
(3) A set S of learning data (unsupervised learning voice data) consisting of only voice data x having no corresponding symbol string data, and
Shall be given in advance.

The voice data (x'or x) and the symbol string data (y'or y) included in the learning data may both be represented by vectors. Further, the definitions of voice data and symbol strings follow the definitions of each term in the voice recognition device described above.

Next, the processing of each part of the learning device 20 will be described. The conversion unit 210 uses a neural network to convert voice data or symbol string data into intermediate feature quantities, and converts the intermediate feature quantities into information that identifies the symbol string. The conversion unit 210 is a neural network, and has an encoder layer that converts each of the learning data into an intermediate feature amount, and a decoder layer that converts the intermediate feature amount into "information for specifying a symbol string". The encoder layer includes a voice feature amount coding unit 211, a symbol string feature amount coding unit 212, and an intermediate feature amount coding unit 213, and the decoder layer includes a symbol string decoding unit 214.

In the following description, the operations (conversion processing) performed in each part (each layer) of the neural network are described as functions. For example, the operation of the neural network corresponding to the voice feature amount coding unit 211 is expressed as a function f (.). The output value of this function is the output of the voice feature amount coding unit 211.

Further, the operation of the neural network corresponding to the symbol string feature amount coding unit 212 is referred to as a function g (.). Further, the operation of the neural network corresponding to the intermediate feature amount coding unit 213 is referred to as a function e (.). Further, the operation of the neural network corresponding to the symbol string decoding unit 214 is referred to as a function d (.).

An operation corresponding to a multi-layer neural network composed of a plurality of layers can be expressed by a composite function of a function representing an operation corresponding to each layer. For example, the operation of the multi-layer neural network including the voice feature amount coding unit 211 and the intermediate feature amount coding unit 213 can be expressed as e (f (・)).

Here, what is called a "layer" of a neural network does not necessarily mean that it is physically one layer, and a plurality of layers may be collectively referred to as a "layer". In other words, a partial structure consisting of one or more layers in the entire neural network is called a "layer".

The voice feature amount coding unit 211 converts (encodes) the voice data in the input learning data into a voice feature amount. Specifically, the voice feature amount coding unit 211 sets the voice data x'in the supervised learning data as the voice feature amount f (x') and the voice data x in the unsupervised learning voice data as the voice feature. Convert to quantity f (x).

The symbol string feature amount coding unit 212 converts (encodes) the symbol string data y in the input unsupervised learning data into the symbol string feature amount. Here, the ultimate goal of the symbol string feature amount coding unit 212 is that the symbol string feature amount g (y) obtained by converting the input symbol string data y converts the voice data into the voice feature amount coding unit. The conversion to an intermediate feature amount that is close to the voice feature amount obtained by the conversion in 211 is performed. That is, as the learning progresses, the distribution of the intermediate feature amount e (g (y)) obtained by converting the symbol string data y is the intermediate feature amount e (f (x)) obtained by converting the voice data x. ) Will be similar to the distribution.

The intermediate feature amount coding unit 213 converts (encodes) the input voice feature amount or symbol string feature amount into the intermediate feature amount. Specifically, the intermediate feature amount coding unit 213 sets the voice feature amount f (x') as the intermediate feature amount e (f (x')) and the voice feature amount f (x) as the intermediate feature amount e (f). In (x)), the symbol string feature amount g (y) is encoded into the intermediate feature amount e (g (y)).

Hereinafter, the intermediate feature amount e (f (x')) obtained by converting the voice data x'of the supervised learning data is referred to as "intermediate feature amount corresponding to the supervised voice data", and the unsupervised learning symbol string data. The intermediate feature amount e (g (y)) obtained by converting y is the "intermediate feature amount corresponding to the unsupervised symbol string data", and the intermediate feature amount e obtained by converting the unsupervised learning audio data x ( f (x)) will be referred to as "intermediate feature amount corresponding to unsupervised audio data".

The symbol string decoding unit 214 converts (decodes) the input intermediate feature amount into information that identifies the symbol string. Here, the target intermediate features are the intermediate features e (f (x')) corresponding to the supervised speech data and the intermediate features e (g (y)) corresponding to the unsupervised symbol string data.

The symbol string decoding unit 214 is realized by a recurrent neural network, for example, a long short-term memory (Long Short Term Memory: LSTM) having an attention mechanism.

First, a case where the intermediate feature amount e (f (x')) corresponding to the supervised voice data is converted into the information for specifying the symbol string will be described as an example. The symbol string specified based on the estimation result of the information for specifying the symbol string is referred to as ^ w (hereinafter, also referred to as "estimated symbol string data"). In the t-step, the symbol string decoding unit 214 includes the state vector h _t-1 output from the neural network in the immediately preceding step (t-1 step) and the estimated symbol string data y obtained up to the t-1 step. ' _{1: t-1} and the feature amount corresponding to the intermediate feature amount e (f (x')) corresponding to the voice feature amount obtained by the intermediate feature amount coding unit 213 are input, and the following output symbols are used. It converted to the corresponding state vector h _t to. Then, based on the _{h t,} the estimated symbol ^ _{w t} is (hereinafter also referred to as "correct symbol") t-th symbols in the correct symbol string data _y't match to the posterior probability _{P (^ w t = y't} | y'1 _{: t-1} , e (f (x'))) is calculated. This is repeated recursively from the beginning.

Here are obtained posterior probability _{P (^ w t = y't} | y'1: t-1, e (f (x'))) is "information that identifies the symbol string".

Further, the symbol string decoding unit 214 performs the same processing on the intermediate feature amount e (g (y)) corresponding to the unsupervised symbol string data, and the posterior probability P (^ v _t = y _t | y _{1: 1). Calculate t-1} , e (g (y))). Here, the estimated symbol string data obtained by converting the intermediate feature amount e (g (y)) corresponding to the unsupervised symbol string data is expressed as ^ v.

The recognition error calculation unit 215 calculates the recognition error using the result (posterior probability) obtained by converting the voice data in the supervised learning data by the conversion unit 210. Specifically, the recognition error calculation unit 215 calculates the cross entropy loss of the equation (1) as _{a recognition error as a differentiable recognition error L pair.}

_{Here, y'1: t-1} represents the symbol string from y _'1 to _{y' t-1.} That is, recognition errors calculator 215, information identifying a symbol string obtained by converting the audio data x'supervised by the conversion section _{210 P (^ w t = y'} t | y'1: t-1, e (F (x'))) is used to calculate the recognition error.

(1) expression, information to identify the supervised symbol string obtained by converting the voice data _{x'P (^ w t = y'} t | y'1: t-1, e (f (x')) ), It can be said that it is a measure showing the closeness between the estimated symbol string data ^ w estimated based on) and the correct answer symbol string data y'preliminarily associated with the voice data x'in the training data.

_{The restoration error calculation unit 216 converts information P (^ v t} = y _t | y _{1: t-1} , e (g) obtained by converting the unsupervised learning symbol string data y by the conversion unit 210. The error (restoration error) between (y))) and the unsupervised learning symbol string data y is calculated.

As described above, the intermediate feature amount e (g (y)) output from the intermediate feature amount coding unit 213 via the symbol string feature amount coding unit 212 is the intermediate feature amount e (g (y)) corresponding to the voice data. It becomes similar to f (x)). Obtaining the posterior probability by the symbol string decoding unit 214 from the intermediate feature quantity e (g (y)) obtained by converting the symbol string data is the voice feature quantity obtained from the symbol string data given for learning. It means that the intermediate feature amount e (g (y)) similar to the above is substituted for the voice feature amount to obtain the voice recognition result. If sufficient learning progresses, the voice recognition result ^ v obtained by voice-recognizing the symbol string data should match the original symbol string data y. The restoration error can be said to be a measure of how close the recognition result ^ v obtained by voice-recognizing the symbol string data y to the original symbol string data y.

That is, by using the unsupervised symbol string data y as the correct answer and learning the neural network parameters so that the estimation result obtained by voice recognition of y approaches y, the neural network approaches the correct answer even without the teacher data. It makes it possible to train network parameters.

Specifically, the restoration error calculation unit 216 is an estimated symbol string specified based on the posterior probability obtained by converting the symbol string data y and the symbol string data y by the conversion unit 210, as in Eq. (2). The cross-entropy loss based on the probability that the data ^ v matches is calculated as the _{divisible restoration error L symbol.}

_{The feature amount error calculation unit 217 calculates the feature amount error L dom} based on the intermediate feature amount obtained by converting the audio data for learning and the intermediate feature amount obtained by converting the symbol string data for learning. To do.

The feature amount error is an index showing the degree of dissimilarity between the intermediate feature amount obtained by converting the voice data and the intermediate feature amount obtained by converting the symbol string data. For example, an index in a test based on the kernel method. An error based on MMD (Maximum Mean Discrepancy) can be used. In this case, the feature amount error calculation unit 217 calculates the feature amount error L _dom as in the equation (3). Here, in the equation (3), since the kernel k is differentiable, the L _dom is differentiable.

The update unit 218 is neural based on the recognition error calculated by the recognition error calculation unit 215, the restoration error calculated by the restoration error calculation unit 216, and the feature amount error calculated by the feature amount error calculation unit 217. Update each parameter of the network. Specifically, the update unit 218 updates the parameters so as to minimize the weighted sum L of the recognition error, the restoration error, and the feature amount error, as in the equation (4).

Here, α, β, and γ are parameters representing weights, and their values are assumed to be set in advance. In equation (4), L is differentiable because _{L pair} , L _symbol , and L _{dom are all differentiable.} Therefore, the update unit 218 can update the parameters of the neural network based on the differential value by a well-known error propagation learning method or the like. In order to improve accuracy, it is preferable that α, β, and γ are values larger than 0, respectively, but any of the weights α, β, and γ may be learned as 0. In this case, an error with a weight of 0 is used. Means not to consider.

The determination unit 219 determines whether or not the error calculated by the update unit 218 satisfies a predetermined criterion. Here, when the determination unit 219 determines that the parameters do not satisfy the predetermined criteria, the learning device 20 returns to the voice feature amount coding unit 211 and repeats the process. On the other hand, when the determination unit 219 determines that the parameter satisfies a predetermined criterion, the learning device 20 outputs the parameter of the current neural network as a learned parameter.

As predetermined criteria, for example, the number of iteration processes has reached a predetermined number of times, the error has fallen below a predetermined threshold value, and the parameter update amount (differential value of the error) has a predetermined threshold value. The following can be used.

[Processing of the learning device of the first embodiment]
The processing flow of the learning apparatus 20 of this embodiment will be described with reference to FIG. FIG. 4 is a flowchart showing a processing flow of the learning device according to the first embodiment. As shown in FIG. 4, first, the learning device 20 reads the parameters of each neural network from the storage unit 22 (step S21). Here, the learning device 20 accepts input of supervised voice data and symbol string data (step S22). Further, the learning device 20 accepts input of unsupervised voice data and symbol string data (step S23).

Then, the learning device 20 executes the recognition error calculation process, the restoration error calculation process, and the feature amount error calculation process in parallel processing (step S24). It should be noted that the learning device 20 does not necessarily have to perform each process of step S24 in parallel processing, and may end each process before proceeding to step S25.

When the error calculation process is completed, the learning device 20 updates the parameters so that each error becomes smaller (step S25). Then, the learning device 20 determines whether or not the parameters have converged (step S26). When the learning device 20 determines that the parameters have not converged (steps S26, No), the learning device 20 returns to step S21 and repeats the process. On the other hand, when it is determined that the parameters have converged (step S26, Yes), the learning device 20 outputs the parameters (step S27) and ends the process.

The flow of the recognition error calculation process will be described with reference to FIG. FIG. 5 is a flowchart showing the flow of the recognition error calculation process according to the first embodiment. As shown in FIG. 5, the learning device 20 first reads supervised voice data and symbol string data (step S101).

Next, the learning device 20 converts the read voice data into voice features (step S102). Then, the learning device 20 converts the voice feature amount into the intermediate feature amount (step S103). Further, the learning device 20 converts the intermediate feature amount into the information for specifying the symbol string (step S104). Here, the learning device 20 calculates the recognition error based on the read symbol string data and the information for identifying the converted symbol string (step S105).

The flow of the restoration error calculation process will be described with reference to FIG. FIG. 6 is a flowchart showing the flow of the restoration error calculation process according to the first embodiment. As shown in FIG. 6, the learning device 20 first reads unsupervised symbol string data (step S201).

Next, the learning device 20 converts the unsupervised symbol string data read into the symbol string features (step S202). Then, the learning device 20 converts the symbol string feature amount into the intermediate feature amount (step S203). At this time, the learning device 20 passes the intermediate feature amount to the feature amount error calculation process (step S204). Further, the learning device 20 converts the intermediate feature amount into information for specifying the symbol string (step S205). Here, the learning device 20 calculates the restoration error based on the read symbol string data and the information for identifying the converted symbol string (step S206).

The flow of the feature amount error calculation process will be described with reference to FIG. 7. FIG. 7 is a flowchart showing the flow of the feature amount error calculation process according to the first embodiment. As shown in FIG. 7, the learning device 20 first reads unsupervised voice data (step S301).

Next, the learning device 20 converts the read voice data (with or without a teacher) into voice features (step S302). Then, the learning device 20 converts the voice feature amount into the intermediate feature amount (step S303). At this time, the learning device 20 receives the intermediate feature amount from the restoration error calculation process (step S304). Here, the learning device 20 calculates a feature amount error between the intermediate feature amount converted from the voice feature amount and the received intermediate feature amount (step S305). The intermediate feature amount received in step S304 is the intermediate feature amount delivered in step S204 of FIG. 6, and is an intermediate feature amount obtained by converting the symbol string feature amount.

[Effect of the first embodiment]
As described above, the conversion unit 210 uses a neural network to convert each of the supervised voice data and the unsupervised symbol string data into intermediate features, and converts each intermediate feature into a symbol string. Convert to specific information. In addition, the recognition error calculation unit 215 converts the voice data with the teacher by the conversion unit 210 to specify the symbol string, and the symbol string data (correct answer symbol string data) associated with the voice data in advance. And, the recognition error which is an error based on is calculated. Further, the restoration error calculation unit 216 calculates the restoration error based on the information for specifying the symbol string obtained by converting the symbol string data without the teacher by the conversion unit 210. Further, the feature amount error calculation unit 217 is an error between the intermediate feature amount obtained by converting the audio data by the conversion unit 210 and the intermediate feature amount obtained by converting the symbol string data by the conversion unit 210. Calculate a feature error. The voice data and the symbol string data targeted by the feature amount error calculation unit 217 may be supervised or unsupervised. Further, the update unit 218 updates the parameters of the neural network so that a predetermined error based on the recognition error, the restoration error, and the feature amount error is minimized.

Restoration errors can be calculated based on unsupervised symbol string data. That is, restore errors can be minimized by unsupervised learning. Similarly, the feature amount error can be calculated based on the unsupervised symbol string data and the unsupervised voice data. Then, the recognition error can be calculated based on a small amount of supervised learning data.

Therefore, by updating the neural network parameters so that certain errors based on recognition errors, restore errors, and feature errors are minimized, a small amount of supervised learning data and unsupervised learning data can be used. Semi-supervised learning can be performed based on the data.

Further, the restoration error calculation unit 216 matches the unsupervised symbol string data with the estimated symbol string data specified based on the information for identifying the symbol string obtained by converting the unsupervised symbol string data by the conversion unit 210. Calculate the cross entropy loss based on the probability of doing as an error. As a result, the restoration error can be calculated as a differentiable error, so that the parameters can be optimized by a known method such as an error backpropagation method.

In addition, the feature amount error corresponds to the dissimilarity between the distributions of the intermediate feature amount obtained by converting the voice data and the distribution of the intermediate feature amount obtained by converting the symbol string data. .. The feature error in Eq. (3) is defined as the dissimilarity in the reproducing kernel Hilbert space by the kernel function k (・). As a result, the feature amount error can be calculated as a differentiable error, so that the parameters can be optimized by a known method such as an error backpropagation method.

Then, in the update unit 218, all the parameters of the neural network are updated so as to minimize a predetermined standard (weighted sum of each error) based on the recognition error, the restoration error, and the feature amount error. Therefore, the parameters of the neural network that composes the speech recognition model can be learned end-to-end with semi-supervised learning.

Specifically, each weight in the weighted sum of the recognition error, the restoration error, and the feature amount error determines which error is prioritized and minimized. Recognition errors are errors based on supervised learning data, and restore errors and feature errors are errors based on unsupervised learning data. That is, by adjusting the weight, it is possible to adjust whether the neural network is trained so as to be more suitable for the supervised learning data or the unsupervised learning data. For example, it can be said that it is possible to adjust the weight so that the error corresponding to the larger amount of data is prioritized according to the amount of learning data.

[Modification 1 of the first embodiment]
In the description of the above-described embodiment, it has been described that three types of learning data, supervised learning data, unsupervised symbol string data, and unsupervised audio data, are used. However, unsupervised audio data is not essential. When unsupervised voice data is not used, the feature error calculation unit 217 is based on the intermediate feature obtained by converting the supervised voice data and the intermediate feature obtained by converting the unsupervised symbol string data. , Feature error can be calculated. That is, the intermediate feature amount obtained by converting the unsupervised voice data may be replaced with the intermediate feature amount obtained by converting the supervised voice data.

[Modification 2 of the first embodiment]
In the description of the above-described embodiment, an example of learning by using the supervised learning data and the unsupervised learning data together from the beginning has been described, but the present invention is not limited to this. For example, first, learning is performed using only supervised learning data until a predetermined first criterion is satisfied, and then learning is repeated until the unsupervised learning data is also used and a predetermined second criterion is satisfied. good.

For example, the first predetermined number is learned using only supervised learning data. In this case, the update unit 218 repeatedly updates the parameters of the neural network based on the recognition error. Then, when the initial predetermined number is exceeded, the supervised learning data and the unsupervised learning data are used together, and as described above, the neural network parameters are repeated based on the recognition error, the restoration error, and the feature amount error. I will update it.

As a result, the accuracy of the trained speech recognition model can be improved, and the learning can be converged at an early stage.

<Supplement>
The supervised learning data and the unsupervised learning data may be divided into predetermined units (mini-batch), and the above-mentioned learning may be performed in the mini-batch units. In this case, the above processing may be performed by replacing the training data sets Z, T, and S with each mini-batch, and when the learning about the mini-batch is completed, the same processing may be repeated for the next mini-batch.

[Second Embodiment]
The configuration of the learning device 20 of the second embodiment is the same as that of the first embodiment. However, the processing of the feature amount error calculation unit 217 and the update unit 218 is different from that of the first embodiment. Hereinafter, the parts different from the first embodiment will be referred to as a feature amount error calculation unit 217'and an update unit 218', and details will be described.

In the learning device 20 of the first embodiment, the feature amount error calculation unit 217 calculates the feature amount error based on the MMD. The feature amount error calculation unit 217'of the second embodiment calculates the feature amount error using the two-class identification neural network.

That is, the learning device 20 of the second embodiment has a two-class identification neural network corresponding to the feature amount error calculation unit 217'in addition to the neural network constituting the same conversion unit 210 as the learning device 20 of the first embodiment. The difference is that it has a separate network. Therefore, the neural network parameter Λ stored in the storage unit 22 includes the neural network parameter included in the feature amount error calculation unit 217 in addition to the neural network parameter constituting the conversion unit 210. Appropriate values shall be set in advance for the initial values of these parameters.

The two-class identification neural network of the feature error calculation unit 217'will be described. The two-class discrimination neural network is a neural network that outputs a discrimination result that discriminates whether the input intermediate feature amount is obtained by converting voice data or symbol string data. It is a network. The function representing the operation of the two-class identification neural network is expressed as h (・).

For example, the two-class discrimination neural network takes the intermediate feature amount e (g (y)) as an input and outputs h (e (g (y))) as the discrimination result. Alternatively, the intermediate feature amount e (f (x)) is input and h (e (f (x))) is output as the identification result.

_{The feature amount error calculation unit 217'calculates the feature amount error L dom} by the equation (5) using the identification result obtained by inputting the intermediate feature amount into the two-class identification neural network.

Similar to the update unit 218, the update unit 218'updates each parameter of the neural network constituting the conversion unit 210 based on a predetermined error based on the recognition error, the restoration error, and the feature amount error.

Further, the update unit 218'updates the parameters of the two-class identification neural network based _{on the -L dom} in which the positive and negative _{of the feature amount error L dom are reversed.} Specifically, the value of each parameter of the neural network h (.) _{Is updated according to the gradient of −L dom.}

The feature amount error L _dom represents the degree of dissimilarity between the intermediate feature amount based on the voice data and the intermediate feature amount based on the symbol string data. That is, learning the parameters of the neural network so as to reduce the error calculated by the feature amount error calculation unit 217'is that the intermediate feature amount based on the voice data and the intermediate feature amount based on the information for specifying the symbol string are used. It means learning to minimize the distance between distributions. As the learning progresses, the symbol string feature amount output from the symbol string feature amount coding unit becomes closer to the voice feature amount output from the voice feature amount coding unit, and as a result, is it an intermediate feature amount based on the voice data? , It becomes difficult to identify whether the feature quantity is based on the symbol string data.

On the other hand, _{learning to minimize -L dom} causes the two-class identification neural network h to erroneously identify the intermediate features based on the voice data as those based on the symbol string data, and the symbol string data. It means trying to learn so that the intermediate feature amount based on is not mistakenly distinguished from the intermediate feature amount based on the voice data.

That is, the learning of the two-class discrimination neural network h (.) And the learning of the neural network constituting the conversion unit 210 are in a hostile learning relationship.

Finally, in hostile learning, learning is performed so that the intermediate features obtained by the neural network constituting the conversion unit 210 cannot be properly identified (sufficiently deceived) by the two-class discrimination neural network h. As a result, it is possible to learn a neural network that can be converted so that the symbol string feature amount output from the symbol string feature amount coding unit 212 is sufficiently close to (similar to) the voice feature amount output from the voice feature amount conversion unit. ..

[Third Embodiment]
In the first embodiment and the second embodiment, the voice feature amount (intermediate feature amount corresponding to the voice data) obtained by the voice feature amount coding unit 211 and the symbol obtained by the symbol string feature amount coding unit 212. The feature was to train the model in consideration of the feature amount error so that the distribution with the column feature amount (intermediate feature amount corresponding to the symbol string data) would be close. Then, the voice feature amount or the intermediate feature amount (referred to as the first intermediate feature amount) once converted to have a similar distribution is further referred to as the intermediate feature amount (the second intermediate feature amount) by the intermediate feature amount coding unit 213. ) Was explained as a configuration to be converted to.

Here, bringing the distributions of the first intermediate feature amount obtained by converting the voice data and the first intermediate feature amount obtained by converting the symbol string data close to each other means that the voice data is converted in the end. It can be considered that the distribution of the second intermediate feature amount obtained by the above and the second intermediate feature amount obtained by converting the symbol string data will be brought closer to each other.

Therefore, in the third embodiment, the learning device 20 does not include the intermediate feature amount coding unit 213. However, also in the third embodiment, the input / output of the conversion unit 210 of the learning device 20 is common to the conventional embodiments. Further, in the description of the third embodiment, the description of matters common to the first embodiment or the second embodiment will be omitted as appropriate.

[Structure of the learning device of the third embodiment]
The configuration of the learning device of the third embodiment will be described with reference to FIG. FIG. 8 is a diagram showing an example of the configuration of the learning device according to the third embodiment. As shown in FIG. 8, the learning device 20 has a control unit 21 and a storage unit 22. The control unit 21 includes a conversion unit 210, a supervised learning error calculation unit 220, an unsupervised learning error calculation unit 230, and a feature amount error calculation unit 240.

The conversion unit 210 uses a neural network to convert the voice data or the symbol string data into the intermediate feature amount, and converts the intermediate feature amount into the information for specifying the voice data or the symbol string. The conversion unit 210 is a neural network, and has an encoder layer that converts each of the learning data into an intermediate feature amount, and a decoder layer that converts the intermediate feature amount into "information for specifying a symbol string". The encoder layer includes a voice feature amount coding unit 2111 and a symbol string feature amount coding unit 2121, and the decoder layer includes a symbol string decoding unit 2141.

The intermediate feature amount here is a voice feature amount which is an output of the voice feature amount coding unit 2111 and a symbol string feature amount which is an output of the symbol string feature amount coding unit 2121. In the first embodiment and the second embodiment, it has been described that the voice feature amount and the symbol string feature amount are converted into the intermediate feature amount. On the other hand, in the third embodiment, the voice feature amount and the symbol string feature amount are the intermediate feature amount itself.

The calculation of the neural network corresponding to the voice feature amount coding unit 2111 is the same as the calculation of the multi-layer neural network including the voice feature amount coding unit 211 and the intermediate feature amount coding unit 213 of the first embodiment. Therefore, the operation of the neural network corresponding to the voice feature amount coding unit 2111 can be expressed as e (f (・)) in the form of a composite function.

Further, the calculation of the neural network corresponding to the symbol string feature amount coding unit 2121 is the same as the calculation of the multi-layer neural network including the symbol string feature amount coding unit 212 and the intermediate feature amount coding unit 213 of the first embodiment. Is. Therefore, the operation of the neural network corresponding to the symbol string feature amount coding unit 2121 can be expressed as e (g (・)) in the form of a composite function.

The speech recognition error calculation unit 222 of the supervised learning error calculation unit 220 performs the same processing as the recognition error calculation unit 215 in the first embodiment. That is, the voice recognition error calculation unit 222 of the supervised learning error calculation unit 220 uses the result (posterior probability) obtained by converting the voice data in the supervised learning data by the conversion unit 210 to generate a voice recognition error. calculate. Specifically, the voice recognition error calculation unit 222 calculates the cross entropy loss of the equation (1) as a _{differentiable voice recognition error L pair.}

Further, the symbol string restoration error calculation unit 232 of the unsupervised learning error calculation unit 230 performs the same processing as the restoration error calculation unit 216 in the first embodiment. That is, the symbol string restoration error calculation unit 232 is the estimated symbol string data specified based on the posterior probability obtained by converting the symbol string data y and the symbol string data y by the conversion unit 210 as in the equation (2). The cross-entropy loss based on the probability that ^ v matches is calculated as the _{divisible symbol string restoration error L symbol.}

The feature amount error calculation unit 240 converts the distribution of the intermediate feature amount obtained by converting the voice data given for learning by the neural network and the voice data or the symbol string data given for learning by the neural network. _{The feature amount error L dom} representing the degree of dissimilarity with the distribution of the intermediate feature amount obtained is calculated.

_{Specifically, the feature amount error calculation unit 240 calculates the feature amount error L dom} as the degree of dissimilarity between the intermediate feature amount used for the voice recognition task and the intermediate feature amount used for other auxiliary tasks. Here, the voice recognition task is a task of converting supervised voice data by a voice feature amount coding unit 211 and a symbol string decoding unit 214 to obtain information for specifying a symbol string, and a voice feature amount. Is an intermediate feature amount obtained by converting the supervised voice data by the voice feature amount coding unit 211. Further, in the third embodiment, the auxiliary task is a task of restoring the symbol string.

The symbol string restoration task is a task of obtaining a symbol string by converting the unsupervised symbol string data by the symbol string feature amount coding unit 212 and the symbol string decoding unit 214. The intermediate feature amount used in the task of symbol string restoration is an intermediate feature amount obtained by converting unsupervised symbol string data by the symbol string feature amount coding unit 212.

The feature error is an index showing the degree of dissimilarity between the first intermediate feature Fn ₁ (u) and the second intermediate feature Fn ₂ (v), and is, for example, an index in a test based on the kernel method. An error based on a certain MMD (Maximum Mean Discrepancy) can be used. In this case, the feature amount error calculation unit 240 calculates the feature amount error L _dom as in the equation (5). Here, in the equation (5), since the kernel k uses a differentiable one, the L _dom is differentiable.

The voice recognition task and each intermediate feature amount obtained by each auxiliary task can be the first feature amount and the second feature amount. Further, the function Fn ₁ (・) and the function Fn ₂ are either f (・) or g (・). For example, the first feature amount is the voice feature amount f (x') obtained by the voice recognition task, and the second feature amount is the symbol string feature amount g (y) obtained by the symbol string restoration task. Can be done.

[Processing of the learning device of the third embodiment]
The processing flow of the learning device 20 of the third embodiment is the same as the processing flow of the learning device 20 of the first embodiment shown in FIG. Further, the feature amount error calculation process of the third embodiment is the same as the flow of the feature amount error calculation process of the first embodiment shown in FIG. 7. However, the learning device 20 of the third embodiment calculates the feature amount error between the voice feature amount obtained in the voice recognition error calculation process and the symbol string feature amount obtained in the symbol string restoration error calculation process. Can be done.

[Effect of the third embodiment]
According to the third embodiment, the same effect as that of the first embodiment can be obtained. The symbol string restoration error can be calculated based on unsupervised audio data. That is, the symbol string restoration error can be minimized by unsupervised learning. Therefore, the speech recognition error can be calculated based on a small amount of supervised learning data.

Further, in the third embodiment, learning using the feature amount error can be performed by using the supervised learning data and the unsupervised learning data. Therefore, according to the third embodiment, semi-supervised learning can be performed by effectively utilizing both the supervised learning data and the unsupervised learning data.

[Experimental result]
Here, the experiments performed using the prior art and the embodiments will be described with reference to FIG. FIG. 9 is a diagram showing the results of the first experiment. In the experiment, the speech recognition model learned by using the semi-supervised learning of the first embodiment and the speech recognition model learned by using the conventional supervised learning method (see Non-Patent Document 1) are continuously used. Word recognition processing was performed. The experimental conditions are as follows.
<Conditions common to conventional technology and embodiments>
-Teached data: 15 hours, small dataset of 7138 utterances-Voice data: 80-dimensional FBANK per frame (input unit is 80 dimensions x utterance time, average distribution model normalization parameter based on training data Normalized with)
-Information for identifying symbol strings: Series of symbols in character units such as alphabets and numbers-Parameter update algorithm: AdaDelta (The number of mini-batch of utterances to be processed in parallel is 30)
<Conditions only for the embodiment>
-Feature quantity error: MMD of equation (3)
Unsupervised data: 81 hours, large dataset of 37416 utterances

As a result of the experiment, as shown in FIG. 9, the character error rate (CER) was lower in the embodiment. From this, it can be said that according to the method of the embodiment, a voice recognition model having higher recognition accuracy than the conventional technique can be obtained.

FIG. 10 is a diagram showing the results of the second experiment. The experimental conditions are as follows.
<Conditions common to conventional technology and embodiments>
-Teached data: 100 hours small data set-Voice data: 80-dimensional FBANK per frame (input unit is 80 dimensions x utterance time, normalized by the normalization parameter of the average variance model based on learning data Done)
-Information for identifying symbol strings: Series of symbols in character units such as alphabets and numbers-Parameter update algorithm: AdaDelta (The number of mini-batch of utterances to be processed in parallel is 30)
<Conditions only for the embodiment>
-Feature quantity error: MMD of equation (5)
-Unsupervised data: Large data set of 860 hours (voice only: 360 hours, symbol string only equivalent to 500 hours)

As a result of the experiment, as shown in FIG. 10, the character error rate (CER: Character Error Rate) and the word error rate (WER: Word Error Rate) were lower in the embodiment. The development CER and the development WE are the recognition results of the data used for constructing the development set, that is, the model. The evaluation CER and the evaluation WE are evaluation sets, that is, recognition results of data not used for building the model.

From FIG. 10, it can be said that according to the method of the embodiment, a voice recognition model having higher recognition accuracy than the conventional technique can be obtained.

Here, a learning device that performs end-to-end learning by a conventional supervised learning method will be described for comparison with the learning device of the embodiment. FIG. 12 is a diagram showing a configuration of a conventional learning device. As shown in FIG. 12, the conventional learning device 20a has a control unit 21a and a storage unit 22a. Further, the control unit 21a includes a voice feature amount coding unit 211a, a symbol string decoding unit 214a, a recognition error calculation unit 215a, an update unit 218a, and a determination unit 219a.

The learning device 20a receives input of information for specifying a symbol string associated in advance and voice data (supervised learning data). Here, the voice feature amount coding unit 211a converts the voice data into a predetermined feature amount using a neural network. Further, the symbol string decoding unit 214a converts the feature amount converted by the voice feature amount coding unit 211a into information for specifying the symbol string by using the neural network. It is assumed that the parameters of each neural network are stored in the storage unit 22a.

Further, the recognition error calculation unit 215a calculates an error between the information for specifying the symbol string converted by the symbol string decoding unit 214a and the information for specifying the input symbol string. The update unit 218a updates the parameters so that the error calculated by the recognition error calculation unit 215a becomes smaller. Then, the determination unit 219a determines whether or not the error updated by the update unit 218a has converged. Here, when the determination unit 219a determines that the parameters have not converged, the learning device 20a further repeats the process.

[Processing of conventional learning device]
The processing of the conventional learning apparatus will be described with reference to FIG. FIG. 13 is a flowchart showing a processing flow of the conventional learning device. As shown in FIG. 13, first, the learning device 20a reads the parameters from the storage unit 22 (step S11a). Next, the learning device 20a accepts input of supervised voice data and information for identifying a symbol string (step S12a).

Here, the learning device 20a converts the input voice data into a voice feature amount (step S13a). Next, the learning device 20a converts the voice feature amount into information for specifying the symbol string (step S14a). Then, the learning device 20a calculates the recognition error from the information for specifying the converted symbol string and the information for specifying the input symbol string (step S15a).

Here, the learning device 20a updates the parameters so that the recognition error becomes small (step S16a). Then, the learning device 20a determines whether or not the parameters have converged (step S17a). When the learning device 20a determines that the parameters have not converged (steps S17a, No), the learning device 20a returns to step S11a and repeats the process. On the other hand, when the learning device 20a determines that the parameters have converged (steps S17a, Yes), the learning device 20a ends the process.

[Other Embodiments]
The neural network corresponding to each of the voice feature amount coding unit 211, the intermediate feature amount coding unit 213, and the symbol string decoding unit 214 of the learning device 20 of the embodiment is the same as that used in the conventional learning device 20a. It may be. Therefore, for example, the initial values of the parameters of each neural network can be determined by supervised learning using the conventional learning device 20a. That is, the learning device 20 adds the neural network g (・) to the neural networks f (・), e (・), and d (・) learned by the learning device 20a (in the case of the second embodiment, further neural networks. In addition to h (・), further learning can be performed.

[System configuration, etc.]
Further, each component of each of the illustrated devices is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of the device is functionally or physically dispersed or physically distributed in arbitrary units according to various loads and usage conditions. Can be integrated and configured. Further, each processing function performed by each device may be realized by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware by wired logic.

Further, among the processes described in the present embodiment, all or part of the processes described as being automatically performed can be manually performed, or the processes described as being manually performed can be performed. All or part of it can be done automatically by a known method. In addition, the processing procedure, control procedure, specific name, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.

[program]
In one embodiment, the learning device 20 can be implemented by installing a learning program that executes the above learning process as package software or online software on a desired computer. For example, by causing the information processing device to execute the above learning program, the information processing device can function as the learning device 20. The information processing device referred to here includes a desktop type or notebook type personal computer. In addition, information processing devices include smartphones, mobile communication terminals such as mobile phones and PHS (Personal Handyphone System), and slate terminals such as PDAs (Personal Digital Assistants).

Further, the learning device 20 can be implemented as a learning server device in which the terminal device used by the user is a client and the service related to the above learning process is provided to the client. For example, the learning server device is implemented as a server device that provides a learning service that inputs voice data and symbol string data and outputs parameters. In this case, the learning server device may be implemented as a Web server, or may be implemented as a cloud that provides the above-mentioned services related to the learning process by outsourcing.

FIG. 11 is a diagram showing an example of a computer that executes a learning program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. Each of these parts is connected by a bus 1080.

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, the display 1130.

The hard disk drive 1090 stores, for example, the OS 1091, the application program 1092, the program module 1093, and the program data 1094. That is, the program that defines each process of the learning device 20 is implemented as a program module 1093 in which a code that can be executed by a computer is described. The program module 1093 is stored in, for example, the hard disk drive 1090. For example, the program module 1093 for executing the same processing as the functional configuration in the learning device 20 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD.

Further, the setting data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, a memory 1010 or a hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the memory 1010 and the hard disk drive 1090 into the RAM 1012 as needed, and executes the processing of the above-described embodiment.

The program module 1093 and the program data 1094 are not limited to those stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). Then, the program module 1093 and the program data 1094 may be read by the CPU 1020 from another computer via the network interface 1070.

10 Speech recognition device 11, 21 Control unit 12, 22 Storage unit 20 Learning device 110, 210 Conversion unit 111, 211, 2111 Voice feature amount coding unit 113, 213 Intermediate feature amount coding unit 114, 214, 2141 Symbol string decoding Chemical unit 212, 2121 Symbol string feature quantity coding unit 215 Recognition error calculation unit 216 Restoration error calculation unit 217, 240 Feature quantity error calculation unit 218 Update unit 219 Judgment unit 220 Supervised learning error calculation unit 222 Speech recognition error calculation unit 230 Unsupervised learning error calculation unit 232 Symbol string restoration error calculation unit

Claims (6)

A learning device that learns the parameters of a speech recognition model that converts speech data into information that identifies a symbol string using a neural network.
A symbol string estimated based on information that identifies a symbol string obtained by converting the voice data in the training data by the neural network using the training data in which the voice data and the symbol string data are associated with each other. A voice recognition error calculation unit that calculates a voice recognition error representing a dissimilarity between the data estimation result and the symbol string data corresponding to the voice data in the training data.
A symbol estimated based on information that identifies a symbol string obtained by converting the unsupervised symbol string data by the neural network using unsupervised symbol string data that is symbol string data to which voice data is not associated. A symbol string restoration error calculation unit that calculates a symbol string restoration error indicating the degree of dissimilarity between the estimation result of the column data and the unsupervised symbol string data, and
An update unit that updates the parameters of the neural network based on a predetermined reference calculated from the voice recognition error and the symbol string restoration error.
A learning device characterized by having. The neural network
A symbol string feature coding unit that converts symbol string data into intermediate features, and
A voice feature coding unit that converts voice data into intermediate features,
A symbol string decoding unit that converts the intermediate feature amount into information that identifies the symbol string, and
An audio decoding unit that converts the intermediate features into audio data,
Have,
The estimation result of the information for identifying the symbol string in the voice recognition error calculation unit is obtained by decoding the intermediate feature amount obtained by converting the voice data in the learning data by the voice feature amount coding unit. It was obtained by converting it by the conversion part.
The estimation result of the voice data in the symbol string restoration error calculation unit is obtained by converting the intermediate feature amount obtained by converting the unsupervised symbol string data by the symbol string feature quantity coding unit by the symbol string decoding unit. The learning device according to claim 1, wherein the learning device is obtained in the above manner. The distribution of the intermediate features obtained by converting the voice data given for learning by the neural network, and the distribution of the intermediate features obtained by converting the symbol string data given for learning by the neural network. It also has a feature error calculation unit that calculates feature errors that represent the dissimilarity of
The learning device according to claim 1 or 2, wherein the updating unit updates the parameters of the neural network based on the voice recognition error, the symbol string restoration error, and the feature amount error. The learning apparatus according to any one of claims 1 to 3, wherein the update of the parameters of the neural network is performed so as to minimize the predetermined reference. It is a learning method executed by a learning device that learns the parameters of a speech recognition model that converts speech data into information that identifies a symbol string by a neural network.
A symbol string estimated based on information that identifies a symbol string obtained by converting the voice data in the training data by the neural network using the training data in which the voice data and the symbol string data are associated with each other. A voice recognition error calculation process for calculating a voice recognition error representing a dissimilarity between the data estimation result and the symbol string data corresponding to the voice data in the training data, and
A symbol estimated based on information that identifies a symbol string obtained by converting the unsupervised symbol string data by the neural network using unsupervised symbol string data that is symbol string data to which voice data is not associated. A symbol string restoration error calculation step for calculating a symbol string restoration error indicating the degree of dissimilarity between the estimation result of the column data and the unsupervised symbol string data, and
An update process for updating the parameters of the neural network based on a predetermined criterion calculated from the voice recognition error and the symbol string restoration error, and
A learning method characterized by including. A learning program for operating a computer as the learning device according to any one of claims 1 to 4.

Images