JP2019159654A - Time-series information learning system, method, and neural network model - Google Patents

Abstract

To obtain a model that has both real-time property and a speech recognition ratio by approximating the recognition ratio of a first model to the recognition ratio of a second model having its structure more complexed, without changing the structure of the first model.SOLUTION: A learning system 10 comprises a first model 23, a second model 13 having its structure more complexed than that of the first model, and a first model learning section which learns the first model using knowledge distillation by having the first model as a student model and the second model as a teacher model. The learning section evaluates probability for each of label series candidate groups that are outputted from the first and second models and correspond to first time-series data including a plurality of frame data, and causes itself to learn the first model based on the evaluation result.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a technique for recognizing information input in time series using a neural network. More specifically, the present invention relates to a technique for learning a neural network model used in a system for processing time-series information, and a neural network model obtained by the learning.

  Audio information, moving image information, and the like are information accompanying temporal changes, and are information that is continuously input as time passes.

  There is a voice recognition system that acquires voice information from a voice uttered by a person or various sound sources and recognizes the voice information. The speech recognition system analyzes uttered speech waveforms and compares them with a database called an acoustic model, pronunciation dictionary, and language model, and outputs utterance contents (sentences).

  There is a DNN-HMM model as a conventional acoustic model. DNN-HMM is the mainstream method for neural network based speech recognition. The DNN-HMM model is a model of DNN (deep neural networks) that models the probability of which label (for example, phoneme) is high with respect to speech features at a certain time, and the time variation of the label. It is expressed by two models, HMM (Hidden Markov Model).

  The end-to-end model was proposed after DNN-HMM. In the end-to-end model, the acoustic model is expressed by one model without being divided into two models like DNN-HMM. Compared with DNN-HMM, the end-to-end model has the advantage that the speech recognition process is simple and fast because it does not use an HMM. Examples of end-to-end models include CTC (Connectionist Temporal Classification) and Attention models. Hereinafter, the CTC acoustic model will be described as an example of the end-to-end model.

  As a technology related to the above-described recognition technology, there is a DNN learning method called Knowledge Distillation (KD). Knowledge distribution is a technique used to copy learned complex and high-performance model information into a simple and low-performance model. For example, consider a case where there are a model having high performance but having a complicated structure and difficult to apply to a system, and a model having a simple structure but low performance. In knowledge distribution, the former is defined as a teacher model and the latter as a student model, and the student model is trained using the output of the teacher model instead of the correct answer label. Thereby, the knowledge of the teacher model can be propagated to the student model.

  In the following Patent Document 1 and Patent Document 2, a speech recognition apparatus using CTC is disclosed.

  Non-Patent Document 1 below discloses a technique related to CTC. Non-Patent Document 2 below discloses a technique related to knowledge distribution.

JP 2017-16131 A JP 2017-40919 A

Alex Graves, Santiago Fernandez, Faustino Gomez, and Jurgen Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks”, ICML2006, pp. 369-376, 2006 Geoffrey Hinton, Oriol Vinyals, and Jeff Dean, “Distilling the knowledge in a neural network”, in NIPS Deep Learning and Representation Learning Workshop, 2014

  Since CTC handles time-series data, it needs to have an internal RNN (Recurrent neural network). There are two types of RNNs: Unidirectional RNN that considers only past information and Bidirectional RNN that uses both past and future information. When Bidirectional RNN is used, the speech recognition rate is high, but since future information is used, real-time processing is difficult. If Unidirectional RNN is used, it can be applied to real-time processing, but the speech recognition rate decreases.

  The present invention achieves both real-time performance and speech recognition rate by changing the recognition rate of the model to be learned close to the recognition rate of the teacher model having a more complicated structure without changing the structure of the model to be learned. The challenge is to obtain a model.

  In order to solve the above problem, the learning system of the present embodiment is configured as follows. The learning system of this embodiment is a system that learns a neural network in order to configure a system that recognizes time-series information.

  The learning system of the present embodiment includes a first model having a neural network capable of expressing time series information therein and a neural network capable of expressing time series information therein, and has been learned by a correct answer label. The second model having a more complex structure than the first model, the first model as a student model, the second model as a teacher model, and learning the first model using knowledge distraction A first model learning unit.

  The first model learning unit inputs first time series data including a plurality of frame data to the first model, and outputs each of the label series candidate groups as a first output result of the first model. A first output unit that obtains the probability of the first time-series data including the plurality of frame data is input to the second model, and a label sequence is output as a second output result of the second model. Based on the second output unit for obtaining the respective probabilities of the candidate group, the evaluation unit for evaluating the difference between the first output result and the second output result, and the evaluation result in the evaluation unit, A first model learning unit that learns one model.

  The first model is learned by the extended knowledge distribution in the learning system of the present embodiment. That is, it has succeeded in improving the accuracy of learning by evaluating the probability value of the label sequence candidate group output in time series, not the probability value of the output value output for each frame.

  Therefore, the first model is a model having a simple structure as compared with the second model, which is a teacher model, and maintains high recognition accuracy. In addition, since the first model has a simpler structure than the second model, the circuit scale can be reduced when implemented as hardware. In addition, since the first model has a simple structure as compared with the second model, high performance is not required for resources such as CPU and memory when implemented as software. Therefore, the recognition system that implements the first model can also be used in terminals such as smartphones and tablets.

  In the time-series information learning system according to the present embodiment, the first model and the second model include a model having a recurrent neural network therein.

  In the time-series information learning system according to the present embodiment, the first model and the second model include a CTC (Connectionist Temporal Classification) model.

  In the time-series information learning system of the present embodiment, the first model is a Unidirectional-CTC model, and the second model is a Bidirectional-CTC model. Unlike the Bidirectional-CTC model, which is the second model, the learned first model, the Unidirectional-CTC model, does not require future input, and thus can realize processing with high real-time characteristics. The first model is advantageous in terms of hardware and software implementation.

  In the time-series information learning system according to the present embodiment, the time-series information is audio information. The learned first model can recognize voice information with a high recognition rate. In addition, real-time performance can be improved as compared with the case where an acoustic model having a complicated structure is used.

  The present embodiment is also intended for a neural network model learned in the time-series information learning system. By constructing a recognition system using the neural network model learned in the above-described time-series information learning system, a highly accurate recognition result can be obtained without imposing a high load on hardware or software.

  In addition, the learning method of the present embodiment includes the following steps. The learning system according to the present embodiment is a learning method for learning a first model that internally includes a neural network that can express time-series information in order to configure a system that recognizes time-series information.

  The learning system of the present embodiment (a) has a neural network capable of expressing time-series information therein, and learns a second model having a more complex structure than the first model using a correct label. And (b) a first model that learns the first model using a knowledge destination using the first model as a student model and the second model as a teacher model. A learning process.

  The first model learning step (b) includes (b-1) inputting first time-series data including a plurality of frame data to the first model, and (b-2) the plurality of the plurality of frame data. A step of inputting the first time-series data including the frame data into the second model; and (b-3) obtained in correspondence with the first time-series data including the plurality of frame data. As a first output result of the first model, a step of obtaining each probability of a label sequence candidate group, and (b-4) obtained corresponding to the first time series data including the plurality of frame data. A second output result of the obtained second model, a step of obtaining each probability of a label sequence candidate group, and (b-5) the first output result obtained in the step (b-3). And the first obtained in the step (b-4) An evaluation step of evaluating a difference between the output result based on the evaluation results of (b-6) said evaluation step, and a step to learn the first model.

  The first model learned by the learning system or the learning method of the present embodiment is a model that has a simple structure as compared with the second model that is a teacher model, but retains high recognition accuracy. In addition, since the first model has a simple structure as compared with the second model, performance requirements for computers and devices are required regardless of whether they are implemented as hardware or software. Can be lowered. Further, since the first model has a simple structure as compared with the second model, the real-time property can be improved as compared with the case where the recognition process is executed using the second model.

It is a figure which shows the flow of a process of the learning phase in the time series information processing system which concerns on this Embodiment, and a recognition phase. It is a block diagram of the learning system with which the time series information processing system concerning this embodiment is provided. It is a figure which shows the learning method of a neural network by a correct answer label. It is a figure which shows the structure of general DNN. It is a figure which shows Bidirectional-RNN which a teacher model has. It is a figure which shows the learning method of the neural network by a knowledge distribution. It is a figure which shows Unidirectional-RNN which a student model has. It is a figure which shows the learning method by the extended knowledge distribution by this Embodiment. It is a block diagram of the recognition system with which the time series information processing system concerning this embodiment is provided. It is a figure which shows the experiment example which shows the learning effect of this Embodiment.

  Hereinafter, a time-series information processing system according to the present embodiment will be described with reference to the accompanying drawings. The time-series information processing system according to the present embodiment is a system that inputs and learns time-series information, inputs time-series information, and outputs a recognition result of the time-series information. The time series information is information that is continuously input as time passes. In this embodiment, audio information is described as an example of time series information. However, the time-series information processing system of this embodiment can be used as a system that recognizes not only audio information but also other time-series information. For example, as other time-series information, moving image information that is continuously input with the passage of time, sensing information, and the like can be given.

  The time-series information processing system according to the present embodiment includes a learning system that learns a model for recognizing time-series information, and a recognition system that recognizes time-series information using a model learned by the learning system. It is configured.

  In the following description, audio information will be described as an example of time series information. That is, as the time-series information processing system of the present embodiment, the learning system 10 that learns the acoustic model 23 for recognizing speech information and the acoustic model 23 learned by the learning system 10 are used to obtain speech information. The recognition system 20 to be recognized will be described as an example.

{1. Flow of learning phase and recognition phase processing in a time-series information processing system}
FIG. 1 is a diagram showing an overall flow of the time-series information processing system according to the present embodiment. The time-series information processing system of the present embodiment has two phases, a learning phase and a recognition phase. In the learning phase, the learning system 10 learns the acoustic model 23. In the recognition phase, the recognition system 20 recognizes voice information using the learned acoustic model 23. The learning system 10 and the recognition system 20 may be mounted on the same computer or device, or may be mounted on different computers or devices.

  As shown in FIG. 1, the feature amount calculation unit 11 calculates the feature amount of the speech data input to the learning system 10. The feature amount of the speech data calculated by the feature amount calculation unit 11 is input to the acoustic model learning unit 12 for each frame. The acoustic model learning unit 12 learns the acoustic model 23. As will be described in detail later, the acoustic model learning unit 12 learns the acoustic model 23 using the already learned acoustic model 13 as a teacher model. The acoustic model learning unit 12 learns the acoustic model 23 by using a plurality of label sequences output from the acoustic model 13 that is a teacher model and the output probability thereof as a teacher.

  The feature amount of the speech data input to the recognition system 20 is calculated by the feature amount calculation unit 21. The feature amount of the audio data calculated by the feature amount calculation unit 21 is analyzed by the decoder 22 for each frame. The decoder 22 uses the acoustic model 23, the pronunciation dictionary 24, and the language model 25 learned in the learning phase to output a speech data recognition result.

{2. Structure of learning system}
Next, the configuration of the learning system and the learning processing method will be described with reference to FIGS.

  FIG. 2 is a functional block diagram of the learning system 10. As shown in FIG. 1, the learning system 10 includes a feature amount calculation unit 11 and an acoustic model learning unit 12. The acoustic model learning unit 12 includes a label estimation unit 121, a label estimation unit 122, a label sequence evaluation unit 123, and a learning unit 124. The learning system 10 also includes an acoustic model 13 and an acoustic model 23.

  The feature amount calculation unit 11 inputs audio data as time series information. The feature amount calculation unit 11 decomposes the waveform of the audio data into 20 ms to 30 ms frame data, and extracts the feature amount for each frame. The feature amount calculation unit 11 extracts the feature amount of the audio data by a conventionally performed method. Examples of the feature extraction method include mel filter bank analysis and mel frequency cepstrum analysis.

<2-1. Output of label (phoneme) by acoustic model 13 (teacher model)>
The feature amount of each frame calculated by the feature amount calculation unit 11 is input to the label estimation unit 121. The label estimation unit 121 converts the input frame data into a probability for each label using the acoustic model 13.

  The label is defined in advance in units of phonemes such as a and i, or kana, characters, and words. In the CTC model, in addition to the above, a label of “blank (−)” indicating that it does not apply to any other label is also included in the label set. In this embodiment, a set of labels is defined by phonemes, noises, and blanks. The label estimation unit 121 outputs the probability value of each label for each input frame.

  The acoustic model 13 uses Bidirectional-CTC in the present embodiment. Bidirectional-CTC is a CTC (Connectionist Temporal Classification) model having a Bidirectional-RNN (Recurrent neural network) that is a kind of DNN (Deep neural network) that handles time-series information. CTC is an example of an end-to-end model. CTC is a framework that can convert between input / output sequences of different lengths (in this embodiment, a speech frame sequence and a label sequence). In CTC, the label sequence which is a recognition result is output by deleting the same label and the blank label (−) with respect to the label allocated for each frame. For example, when “a a -k -i-” is assigned to input data of 7 frames, a label series “aki” is output as a recognition result.

  The present embodiment is characterized in that an end-to-end model is used as the acoustic model 13. In this embodiment, CTC, which is an example of an end-to-end model, is used as the acoustic model 13, but an attention model can be used as the end-to-end model.

  In the present embodiment, a model having Bidirectional-RNN inside is used as the acoustic model 13. However, the present invention is not limited to this, and any neural network that can handle time-series information is applicable. Examples of neural networks that can handle time-series information include Bidirectional-RNN, Unidirectional-RNN, and Time-delay neural network. The RNN described above can be replaced with a similar model such as LSTM (Long short term memory). However, the acoustic model 13 is a model having a more complicated structure and higher performance than the acoustic model 23, which leads to an effect using the learning method of the present embodiment.

  The acoustic model 13 is a teacher model that has already been learned. The acoustic model 13 which is a teacher model is learned in advance using learning data including a set of voice data and a correct label sequence.

  The acoustic model 13 is learned according to a conventional CTC learning method. That is, learning is input and learning is performed so that the probability of the correct phoneme sequence is maximized. A conventional forward-backward algorithm is used as the probability calculation method. Further, the error back propagation method is used for updating the model parameters.

  FIG. 3 is a diagram showing a method of learning a general neural network using correct answer labels. Learning data (corresponding to one frame of audio data in the example of this embodiment) is input to the input layer of the neural network. Then, the probability (probability distribution) of each label is output from the output layer as a calculation result of the neural network. At this time, the neural network is trained so that the distance between the two probability distributions becomes small, assuming that the probability probability of the label corresponding to the learning data is 1 and that the probability of the other label is 0 is a correct probability distribution. Cross entropy or Euclidean distance is used as the distance measure.

  FIG. 4 is a diagram showing a general DNN 33. The DNN 33 has an input layer, a plurality of intermediate layers (hidden layers), and an output layer. 4 includes an input layer 331, four intermediate layers 332, 333, 334, and 335, and an output layer 336. Here, a general DNN 33 is illustrated for simplification of the drawing, but the acoustic model 13 of the present embodiment uses an RNN having coupling between the front and rear frames.

  A feature value vector for one frame calculated by the feature value calculation unit 11 is input to the input layer 331. That is, the number of nodes 331 (1) to 331 (n1) in the input layer corresponds to the number of dimensions of the feature amount.

  In this embodiment, the intermediate layer 332 includes n2 nodes 332 (1), 332 (2)... 332 (n2), and the intermediate layer 333 includes n3 nodes 333 (1), 333 ( 2)... 333 (n3), the intermediate layer 334 includes n4 nodes 334 (1), 334 (2)... 334 (n4), and the intermediate layer 335 includes n5 nodes 335 (1 ), 335 (2)... 335 (n5). The number of nodes in each intermediate layer may be different. Further, the number of nodes in each intermediate layer may be different from that of the input layer.

  In the present embodiment, the output layer 336 includes a node corresponding to each label. The number of nodes in the output layer corresponds to the number of labels.

The feature amount of the frame data included in the audio data is input to the input layers 331 (1), 331 (2). As described above, a general DNN 33 is described here with reference to the drawings. However, the acoustic model 13 which is a Bidirectional-CTC used in the present embodiment is used for information on past frame data and future frames. While referring to data information, each intermediate layer performs an operation, and the output layer outputs a probability value for each label. For example,
a: 0.12
b: 0.05
c: 0.03
...
z: 0.09
Blank: 0.02
As described above, a probability value indicating which label the frame data corresponds to is calculated.

  FIG. 5 shows the Bidirectional-RNN processing that the acoustic model 13 has in this embodiment. In FIG. 5, the horizontal axis is time. Bidirectional-RNN at the time when there is one system of blocks arranged in the vertical direction is shown. Each layer of Bidirectional-RNN at each time is shown by one block. That is, each block in FIG. 5 represents each layer of a neural network composed of a plurality of nodes as shown in FIG.

  Frame data input to the input layer 131 at a certain time propagates to the intermediate layers 132, 133... And is output from the output layer 136. At this time, in the intermediate layers 132, 133..., The outputs of the intermediate layers 132, 133. .

  The example shown in FIG. 5 indicates that a phoneme candidate is output as a blank (-) at time t1. From the output layer 136, the probability for each label is output, and the probability of blank is the highest among them.

  Similarly, “a” is output as a phoneme candidate at time t2, “a” at time t3, “k” at time t4, “i” at time t5, and blank (−) at time t6. It shows that.

<2-2. Output of label (phoneme) by acoustic model 23 (student model)>
Reference is again made to FIG. The feature amount calculated by the feature amount calculation unit 11 is also input to the label estimation unit 122. Similar to the label estimation unit 121, the label estimation unit 122 receives the feature amount included in the speech data for each frame, and converts the frame data into a probability for each label using the acoustic model 23. The label estimation unit 122 outputs frame data included in the audio data as a probability value for each label.

  Similarly to the label estimation unit 121, the label estimation unit 122 outputs a probability value for each label. Here, the definition of the label is the same in the acoustic model 13, the label estimation unit 121, the acoustic model 23, and the label estimation unit 122.

  The acoustic model 23 is a model learned using the acoustic model 13 as a teacher model. The acoustic model 23 which is a student model is a model having a structural complexity smaller than that of the acoustic model 13. Here, the model having a complicated structure is a model having a large number of intermediate layers (hidden layers), for example. Alternatively, a model having a complicated structure is a model having a large number of nodes. In addition, as a model having a complicated structure, there are a model having a layer with a large amount of calculation processing such as a CNN (Convolution neural network) and a model having a recurrent structure.

  The acoustic model 23 which is a student model is not learned using the correct answer label. The acoustic model 23 is learned by knowledge distribution using the acoustic model 13 as a teacher model.

  The acoustic model 23 uses Unidirectional-CTC in the present embodiment. Unidirectional-CTC is a CTC model having Unidirectional-RNN inside. Similar to the acoustic model 13, the acoustic model 23 is an example of an end-to-end model. In addition to the CTC, an attention model can be used, and an internal neural network such as RNN, LSTM, Time-delay neural network, etc. Can be changed. However, the acoustic model 23 is a simpler and lower performance model than the acoustic model 13, which leads to the effect of using the learning method of the present embodiment.

  In this embodiment, a recurrent neural network (Bidirectional-CTC) that refers to information on the previous and subsequent times is used as a teacher model, whereas a recurrent neural network (Unidirectional) that refers to information on a past time as a student model. ) Is used. Therefore, the acoustic model 13 which is a teacher model is a model having a more complicated structure than the acoustic model 23 which is a student model. However, this is an example of the acoustic model 13 and the acoustic model 23. In the present embodiment, the acoustic model 13 and the acoustic model 23 are neural networks that can express time-series information, that is, are end-to-end model neural networks, and are compared with the acoustic model 13. Therefore, the acoustic model 23 having a complicated structure may be used, and other models may be used. For example, the acoustic models 13 and 23 may both use the Bidirectional-CTC and use the acoustic model 23 whose structure is less complicated than the acoustic model 13. Alternatively, the acoustic models 13 and 23 may both use the Unidirectional-CTC and use the acoustic model 23 whose structure is less complicated than the acoustic model 13.

  The acoustic model 23 which is a student model performs learning by transferring the recognition ability of the acoustic model 13 which is a teacher model. For example, the student model is a model with a relatively simple structure on the premise that the student model is used on a computer or a smartphone having a relatively small processing capability. By transferring the recognition ability of the complex acoustic model 13 having a structure learned using the correct model to the acoustic model 23, the high recognition accuracy of the teacher model can be transferred.

  FIG. 6 is a diagram for explaining a knowledge distribution in a general frame unit. One frame of audio data is input to the input layer of the student model. Also, the same frame data as the frame data input to the student model is input to the input layer of the teacher model.

  The input audio data of one frame propagates through the intermediate layer of the teacher model and the student model, respectively, and is output as the probability value of each label in the output layer. In the learning method based on knowledge distribution, the student model is learned so that the probability value (label probability distribution) for each label output from each model is close. As an index for measuring the proximity of the probability distribution, cross entropy or Cullback library divergence is used. When learning by conventional knowledge distribution is used for CTC learning as it is, the probability distribution of each frame using the probability distribution of the label output by the acoustic model 23 for each frame and the probability distribution output by the acoustic model 13 for each frame. The acoustic model 23 is learned so as to be close to each other. In other words, the conventional knowledge distribution is a frame-independent learning standard. However, in the learning system of the present embodiment, the probability value of each label for each frame is not simply evaluated, but the difference between the two models is evaluated by a new method (referred to as extended knowledge distribution). This evaluation method will be described in detail later.

  FIG. 7 is a diagram showing the Unidirectional-RNN that the acoustic model 23 has. In the figure, at time t1, a phoneme candidate is output as a blank (-). Also, “a” is output as a phoneme candidate at time t2, “a” at time t3, “k” at time t4, “i” at time t5, and blank (−) at time t6. Show.

<2-3. Learning process by extended knowledge distribution>
Refer to FIG. 2 again. The label series evaluation unit 123 inputs a probability value for each label of the frame data output from the label estimation unit 121. As described above, the label estimation unit 121 uses the teacher model 13 to output a probability value for each label for each frame data. Specifically, the label estimation unit 121 outputs a probability value for each label using a teacher model. The label series evaluation unit 123 inputs the probability value for each label output for each frame.

  FIG. 8 shows an example of probability values for each label input by the label series evaluation unit 123. The right side of FIG. 9 shows the probability value for each label output from the acoustic model 13 which is a teacher model. In the example of the figure, the probability value for each label is shown at each of the times t1, t2, and t3. At the time t1, the probability value of the label “a” is higher than the probability values of the other labels, indicating that the frame data at the time t1 is likely to be the label “a”. Similarly, at time t2, there is a high possibility that the label is “k”.

  Refer to FIG. 2 again. The label series evaluation unit 123 inputs a probability value for each label of the frame data output from the label estimation unit 122. As described above, the label estimation unit 122 outputs a probability value for each label for each frame using the student model 23. Specifically, the label estimation unit 122 outputs a probability value for each label using the student model. The label series evaluation unit 123 inputs the probability value for each label output for each frame.

  The left side of FIG. 8 shows the probability value for each label output from the acoustic model 23 which is a student model. In the example of the figure, the probability value for each label is shown at each of the times t1, t2, and t3. At the time t1, the probability value of the phoneme “a” is higher than the probability values of the other labels, indicating that there is a high possibility that the frame data at the time t1 is the label “a”. Similarly, at time t2, there is a high possibility that the label is “k”.

The label sequence evaluation unit 123 calculates the probability value of the label sequence candidate group from the probability value for each label output from the acoustic model 13 as the teacher model. In the example illustrated in FIG. 8, the label sequence evaluation unit 123 calculates the probability value of the following label sequence candidate group from the probability value for each label output from the acoustic model 13.
aki: 0.5
akai: 0.004
ai: 0.03
...

  Since it is practically difficult to expand all the label sequences, the label sequence evaluation unit 123 employs, for example, the top ten label sequences with the highest probability values as label sequence candidates.

The method for calculating the probability value of the label series is not particularly limited, but an example is shown. For example, there are many patterns of audio data corresponding to the label sequence “aki”. For example, assuming that the audio data is frame data of 7 frames, the audio data corresponding to the label sequence “aki” is
aakk--i
aa-kkki
akk--ii
akkii--
Many patterns exist. By expressing the probability value of each label sequence by multiplying the probability value for each label, the probability value of one pattern (for example, aakk-i) corresponding to the label sequence “aki” can be calculated. Therefore, as the probability value of the label series “aki”, there are conceivable methods such as using addition of probability values of individual patterns or adopting the probability value of the pattern having the highest probability value.

  When the label sequence evaluation unit 123 calculates the probability value of each label sequence “aki”, “akai”, “ai”, etc., as described above, for example, the top ten label sequences with the highest probability values are selected as label sequence candidates. Adopt as.

Similarly, the label series evaluation unit 123 calculates the probability value of the label series candidate group from the probability value for each label output from the acoustic model 23 as the student model. In the example shown in FIG. 8, the label sequence evaluation unit 123 calculates the probability value of the following label sequence candidate group from the probability value for each label output from the acoustic model 23.
aki: 0.3
akai: 0.1
ai: 0.05
...

  The method of calculating the probability value of the label series candidate group from the probability value for each label output from the acoustic model 23 as the student model is the same as in the case of the teacher model described above, and thus the description thereof is omitted.

  When the label sequence evaluation unit 123 calculates the probability value (label sequence probability distribution) of the label sequence candidate group for the acoustic model 13 and the acoustic model 23, respectively, the probability of the label sequence calculated for the acoustic model 13 and the acoustic model 23 The distance from the distribution is calculated using a loss function. Examples of the loss function include cross entropy and Cullback library divergence. What is important in the present embodiment is that the labels calculated for the acoustic model 13 and the acoustic model 23 are not evaluated, but the difference between the probability values of the labels for each frame calculated for the acoustic model 13 and the acoustic model 23 is not evaluated. It is to evaluate the difference between the probability values of the series candidate groups.

  The acoustic model used in the time series information processing system 1 of the present embodiment is an end-to-end model, which is a neural network capable of expressing time series information. Therefore, the inventors have confirmed that the accuracy of learning does not increase even if the probability value for each label output for each frame is evaluated. Therefore, the accuracy of learning can be improved by evaluating the probability value of the label sequence candidate group output at the sequence level, not the probability value of each label output for each frame.

  When the label sequence evaluation unit 123 calculates the distance of the probability distribution of the label sequence candidate group, the learning unit 124 learns the acoustic model 23 so as to minimize the distance. For learning, a conventionally used method such as an error back propagation method is used.

{3. Configuration of recognition system}
FIG. 9 shows a configuration of the recognition system 20 according to the present embodiment. The recognition system 20 includes a feature amount calculation unit 21, a decoder 22, an acoustic model 23, a pronunciation dictionary 24, and a language model 25.

  The feature amount calculation unit 21 inputs audio data as time series information. The feature amount calculator 21 decomposes the waveform of the audio data into frame data, and extracts the feature amount for each frame. The feature quantity calculation unit 21 extracts the feature quantity of the audio data by a conventional method. Examples of the feature extraction method include mel filter bank analysis and mel frequency cepstrum analysis. However, it is necessary to match the analysis conditions with the feature amount calculation unit 11 used at the time of learning.

  The feature amount for each frame calculated by the feature amount calculation unit 21 is input to the decoder 22. The decoder 22 includes an acoustic model 23 learned by the learning process described above. The decoder 22 inputs the feature amount of the audio data to the acoustic model 23 for each frame, and converts the frame into a probability value for each label. In the present embodiment, since the label is defined by phonemes, the decoder 22 converts to a probability value for each phoneme by the acoustic model 23.

  Based on the probability value for each phoneme obtained from the acoustic model 23, the decoder 22 refers to the pronunciation dictionary database 24 and the language model 25 and outputs a recognition result with the highest probability. The pronunciation dictionary database 24 is composed of words and phoneme strings constituting the words. For example, for the word "Hello", the phoneme string / k / o / N / n / i / ch / i / w / a / is defined.

  The language model 25 models the connection between words. For example, for the word "Hello", then what word is modeled or easy to appear. Language modeling methods include conventional n-gram and RNN models. Based on the probability value obtained from the acoustic model 23, the pronunciation dictionary database 24, and the probability value indicated by the language model 25, the decoder 22 outputs the word sequence having the highest probability as a speech recognition result. As a search method of the word sequence having the highest probability value, there is a beam search method or the like.

  Thus, in the recognition system 20, the acoustic model 23 learned by the learning system 10 is used. As described above, the acoustic model 23 is learned by the expanded knowledge distribution in the learning system 10 of the present embodiment. That is, the accuracy of learning has been successfully improved by evaluating the probability value of the label sequence candidate group output at the sequence level, not the probability value of each label output for each frame. Therefore, compared with the acoustic model 13 which is a teacher model, the model has a simple structure but retains high recognition accuracy. Further, since the acoustic model 23 has a simple structure compared to the acoustic model 13, the circuit scale can be reduced when implemented as hardware. In addition, since the acoustic model 23 has a simple structure, when implemented as software, high performance is not required for resources such as a CPU and a memory. Therefore, it is possible to use the recognition system 20 of the present embodiment also on a terminal such as a smartphone or a tablet. In addition, since the structure of the acoustic model 23 is simpler than that of the acoustic model 13, real-time performance can be improved.

  As described above, the learning system 10 according to the present embodiment changes the recognition rate of the acoustic model 23 without changing the structure of the acoustic model 23 that is a learning target, and the recognition rate of the acoustic model 13 that is a teacher model with a more complicated structure. The objective is to obtain a model that achieves both real-time performance and a speech recognition rate. More specifically, the end-point that achieves both real-time performance and voice recognition rate by changing the recognition rate closer to Bidirectional RNN-based CTC (Bi-CTC) without changing the structure of Unidirectional RNN-based CTC (Uni-CTC) It is possible to obtain a -to-end acoustic model.

{4. Experimental result}
FIG. 10 is a diagram illustrating an experimental result according to the learning method of the present embodiment. The evaluation data uses an English speech database called WSJ Corpus. For the feature amount, a 40-dimensional mel filter bank feature amount and its primary and secondary delta feature amounts were used (120 dimensions in total). The labels were defined by 72 phonemes, 2 noises, and a blank. Bidirectional-CTC with Bidirectional-LSTM was used for the teacher model, and Unidirectional-CTC with Unidirectional-LSTM was used for the student model. The number of each intermediate layer is 3, and the number of memory cells in each intermediate layer is 512. The upper part of the figure shows the experimental results when learning is performed using 15 hours of learning data called train_si84 in the WSJ corpus. The word error rate of the teacher model (Bidirectional-CTC) trained by the normal learning method using the correct answer label is 10.35%. The word error rate of the student model (Unidirectional-CTC) trained by the normal learning method using the correct answer label is 11.77%.

  On the other hand, the word error rate of the student model trained by the conventional knowledge distribution at the frame level was 16.04%, which was worse than the normal learning method. On the other hand, it can be seen that the word error rate of the student model trained by knowledge distribution at the sequence level (sequence unit) as the learning method of the present embodiment is 10.83%, and the performance difference is improved by 66.2%.

  The lower part of FIG. 10 shows the experimental results when learning is performed using 81 hours of learning data called train_si284 in the WSJ corpus. The word error rate of the teacher model (Bidirectional-CTC) trained by the normal learning method using the correct answer label is 8.70%. The word error rate of the student model (Unidirectional-CTC) trained by the normal learning method using the correct answer label is 10.37%.

  On the other hand, the word error rate of the student model learned by the conventional knowledge distribution at the frame level was 12.71%, which was worse than the normal learning method. On the other hand, it can be seen that the word error rate of the student model trained by knowledge distribution at the sequence level (sequence unit) which is the learning method of the present embodiment is 9.57%, and the performance difference is improved by 47.9%.

  The above experimental result is an example in which the teacher model and the student model each have three intermediate layers. However, it is not necessary to unify the number of intermediate layers and the number of nodes in the teacher model and the student model. For example, the same effect can be expected even under the condition that the middle layer of the teacher model is four layers and the middle layer of the student model is two layers.

  The specific configuration of the present invention is not limited to the above-described embodiment, and various changes and modifications can be made without departing from the scope of the invention.

1 Time-series information processing system 10 Learning system 13 Teacher model 20 Recognition system 23 Student model

Claims (6)

A system for learning a neural network to construct a system for recognizing time series information,
A first model having therein a neural network capable of expressing time series information;
A second model having a neural network capable of expressing time-series information therein and having been learned by a correct label and having a more complex structure than the first model;
A first model learning unit that uses the first model as a student model, the second model as a teacher model, and learns the first model using a knowledge destination;
With
The first model learning unit
A first output unit that inputs first time-series data including a plurality of frame data to the first model and obtains respective probabilities of label sequence candidate groups as a first output result of the first model When,
The first time series data including the plurality of frame data is input to the second model, and a second output result of the second model is used to obtain respective probabilities of label series candidate groups. An output section;
An evaluation unit that evaluates a difference between the first output result and the second output result;
A first model learning unit for learning the first model based on an evaluation result in the evaluation unit;
A learning system for time-series information. The time-series information learning system according to claim 1,
The time series information learning system, wherein the first model and the second model include a model having a recurrent neural network therein. The time series information learning system according to claim 2,
The time series information learning system, wherein the first model and the second model include a CTC (Connectionist Temporal Classification) model. The time-series information learning system according to claim 3,
The time-series information learning system, wherein the first model is a Unidirectional-CTC model and the second model is a Bidirectional-CTC model. The time-series information learning system according to any one of claims 1 to 4,
The time-series information learning system, wherein the time-series information includes voice information. In order to construct a system for recognizing time series information, a learning method for learning a first model having a neural network that can express time series information therein,
(A) a second model learning step in which a second model having a neural network capable of expressing time-series information and having a more complex structure than the first model is learned using a correct label;
(B) a first model learning step in which the first model is a student model, the second model is a teacher model, and the first model is learned using knowledge distrition;
With
In the first model learning step (b),
(B-1) inputting first time-series data including a plurality of frame data into the first model;
(B-2) inputting the first time-series data including the plurality of frame data into the second model;
(B-3) obtaining each probability of a label sequence candidate group as a first output result of the first model obtained corresponding to the first time-series data including the plurality of frame data When,
(B-4) A step of obtaining each probability of a label sequence candidate group as a second output result of the second model obtained corresponding to the first time-series data including the plurality of frame data. When,
(B-5) an evaluation step for evaluating a difference between the first output result obtained in the step (b-3) and the second output result obtained in the step (b-4); ,
(B-6) a step of learning the first model based on an evaluation result of the evaluation step;
For learning time-series information including

Images