JP7282363B2 - Language discrimination model training method and apparatus, and computer program therefor - Google Patents

Description

TECHNICAL FIELD The present invention relates to speech language identification technology, and more particularly to training a model for identifying speech language from short speech data using a neural network.

Spoken language identification (LID) is a technique for identifying in which language a speech is being made. Spoken language identification often serves as preprocessing for a wide range of multilingual speech processing systems, such as spoken language translation and multilingual speech recognition. In many of these applications, spoken language identification techniques are supposed to work in real-time, so it is very important to reliably identify the spoken language based only on short utterances.

Recently, spoken language identification technology has been extensively researched and developed. For example, i-vector techniques using conventional classifiers, such as support vector machines, probabilistic linear discriminant analysis (PLDA), and deep neural networks, have been validated in various systems, especially for relatively long utterances. It shows state-of-the-art performance in its task. However, it is known that in spoken language identification for short utterances, the i-vector based approach often suffers from severe performance degradation.

On the other hand, recently, attempts have also been made to use end-to-end methods such as convolutional neural networks (CNN), recurrent neural networks (RNN), and attention-based neural networks for spoken language identification. (Non-Patent Documents 1, 2 and 3 listed below). In spoken language recognition for short utterances, end-to-end approaches have been shown to perform better than i-vector based approaches. However, short utterances have more variations than long utterances, making it difficult to generalize the model. Therefore, it is required to enable sufficient generalization even for short utterances.

A. Lozano-Diez, R. Zazo Candil, J. G. Dominguez, D. T. Toledano and J. G. Rodriguez, “An end-to-end approach to language identification in short utterances using convolutional neural networks,” in Proc. of Interspeech, 2015. S. Fernando, V. Sethu, E. Ambikairajah and J. Epps, “Bidirectional Modeling for Short Duration Language Identification,” in Proc. of Interspeech, 2017. W. Geng, W. Wang, Y. Zhao, X. Cai and B. Xu, “End-to-End Language Identification Using Attention-Based Recurrent Neural Networks,” in Proc. of Interspeech, 2016. G. Hinton, O. Vinyals and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015. A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta and Y. Bengio, “Fitnets: Hints for thin deep nets,” in Proc. of ICLR, 2015.

On the other hand, Non-Patent Document 4 proposes a technique called Knowledge Distillation (hereinafter referred to as “KD”). This method can be called a knowledge compression method. Referring to FIG. 1, in this approach system 40 trains a compact student network 60 with the output of a more complex and sophisticated teacher network 50 . Although the teacher network 50 can operate with high precision, it takes a long time to operate due to its complicated configuration and many components, and is not suitable for real-time processing. On the other hand, the student network 60 has fewer layers and fewer neurons forming each layer than the teacher network 50 and operates faster than the teacher network 50 . However, since the student network 60 has a simple configuration, it cannot demonstrate sufficient performance with training using a normal method. To address this problem, Non-Patent Document 4 discloses a method of training the student network 60 by knowledge distillation using the language recognition knowledge possessed by the teacher network 50 .

The outline of the method disclosed in Non-Patent Document 4 is as follows. Referring to FIG. 1, a teacher network 50 is trained in a conventional manner.

That is, prepare training data for the teacher network 50 . Each training sample that constitutes the training data consists of utterance data of a fixed utterance time and a label indicating the language of the utterance. During training, the input of the teacher network 50 is the matrix x representing this speech data. On the other hand, a vector representing the language of the matrix x is given to the output of the teacher network 50 as a hard label. This hard-label vector is a one-hot vector with as many elements as there are languages to be identified. The teacher network 50 is trained by conventional backpropagation using such training data.

Training of the student network 60 using the teacher network 50 proceeds as follows. The training samples for training the student network 60 are the same training data (utterance data and language identification labels) as the training data for training the teacher network 50, and the utterance data of the training samples are input to the teacher network 50. It is a label (called a soft label) that is sometimes output by the teacher network 50 . The values output from the teacher network 50 for the utterance data x of the training samples are finally output as a vector having K elements in the softmax layer 54 . Each element represents the probability that the utterance is in the language corresponding to that element. The output vector of the teacher network 50 when the speech data x is input is vector z(x), the i-th element of vector z(x) is z _i (x), and the output vector q of the softmax layer 54 is i Assuming that the th element is _qi , Non-Patent Document 4 gives the following relationship between them.

このとき、ソフトマックス層５４の出力するラベルの値を「ソフト化」するために、ティーチャーネットワーク５０の出力層が出力する値を「温度」と呼ばれる値Ｔで除算５２してソフトマックス層５４への入力とする。温度Ｔは通常は１であるが、温度Ｔが大きくなると、ソフトマックス層５４の値の出力分布がある程度ならされる（ソフト化される）。

At this time, in order to "soften" the label value output by the softmax layer 54, the value output by the output layer of the teacher network 50 is divided 52 by a value T called "temperature" and sent to the softmax layer 54. input. The temperature T is normally 1, but as the temperature T increases, the output distribution of the values in the softmax layer 54 is smoothed out (softened) to some extent.

Training of the student network 60 using this soft label is performed as follows. Training sample utterance data x is input to the student network 60 . The following two types of losses (first loss 66 and second loss 70) are calculated for the output value of the final layer of the student network 60 at this time, and the final total loss 74 is obtained by the weighted sum of both to calculate Backpropagation updates the parameters of the Student's network 60 so that this total loss 74 approaches zero.

On the one hand, the cross entropy (CE) is calculated using the hard label 72 consisting of the one-hot vector representing the language of the utterance data x and the output vector obtained by applying the softmax layer 68 to the output of the student network 60. Calculate the second loss 70 due to On the other hand, the output of the student network 60 is divided 62 by the temperature T used in the soft label calculation of the teacher network 50, and the output of the softmax layer 64 with this value as input and the soft label obtained from the teacher network 50 are combined. From label 56, a first loss 66 is calculated. The weighted sum of this second loss 70 and the first loss 66 is the total loss 74 . Letting the total loss be L _KD , L _KD is represented by the following equation.

ここでＬ_ｈａｒｄが第２の損失７０であり、Ｌ_ｓｏｆｔが第１の損失６６を表す。λはハード損失とソフト損失とをバランスさせるための重みである。Ｎは訓練サンプルの数を表す。ｙはハードラベルを表す要素数Ｋのワンホットベクトルである。

where L _hard is the second loss 70 and L _soft represents the first loss 66 . λ is a weight for balancing hard loss and soft loss. N represents the number of training samples. y is a one-hot vector with K elements representing a hard label.

Here, using cross-entropy, a soft loss (first loss 66) and a hard loss (second loss 70) for training sample utterance data x are calculated according to the following equations.

ここで、ｐ（ｘ）は各クラスに対してスチューデントモデルが出力する確率を要素とするＫ次元ベクトルであり、ｙ^Ｔ及びｑ^Ｔはそれぞれ、ベクトルｙ及びｑに対する転置を表す。図１から明らかなように、ソフト損失を算出する際にもティーチャーネットワークの処理と同様、スチューデントネットワーク６０の出力は温度Ｔで除算６２された後にソフトマックス層６４に与えられる。

where p(x) is a K-dimensional vector whose elements are the probabilities output by the Student's model for each class, and ^yT and ^qT represent the transposes for vectors y and q, respectively. As is apparent from FIG. 1, the output of the student network 60 is divided 62 by the temperature T before being applied to the softmax layer 64, similar to the process of the teacher network when calculating the soft loss.

On the other hand, there is a technique called FitNets disclosed in Non-Patent Document 5, which is inspired by this KD. The network training method disclosed in Non-Patent Document 5 will be described with reference to FIGS. 2 to 5. FIG. Referring to FIG. 2, this system 100 uses a trained teacher network 110 to create a student network (referred to as FitNet in Non-Patent Document 5) that has more layers than the teacher network 110 but fewer neurons in each layer. ) 120. In Non-Patent Document 5, the number of layers of the student network 120 is greater than the number of layers of the teacher network 110 as shown in FIG. is assumed. However, the number of neurons in each hidden layer of student network 120 is less than that of teacher network 110 . That is, the Student Network 120 is generally a “slimer” network than the Teacher Network 110 .

In Non-Patent Document 5, the hidden layer 114 near the center of the teacher network 110 is the transfer source, and the hidden layer 124 near the center of the student network 120 is the transfer destination. According to Non-Patent Document 5, if the hidden layer 114 of the transfer source and the hidden layer 124 of the transfer destination are too far from the input, the network lacks flexibility. Both of the previous hidden layers 124 are chosen for the middle layer. Let h and g be natural numbers, the hidden layer 114 of the transfer source is the h-th layer, and the hidden layer 124 of the transfer destination is the g-th layer. In FIG. 2, the entire layer group from the first layer of the teacher network 110 to the hidden layer 114 of the transfer source is represented as the W _Hint layer 112, and the entire layer group from the first layer of the student network 120 to the hidden layer 124 of the transfer destination. is called the W _Guided layer 122 .

Further, since the number of neurons in the hidden layer of the teacher network 110 is larger than the number of neurons in the _student network 120, as shown in FIG. A regressor 126 is provided that has a number of inputs equal to the number of outputs of , and a number of outputs equal to the number of outputs of the W _Hint layer 112 comprising the source hidden layer 114 .

As shown in FIG. 3, in the technique of Non-Patent Document 5, the loss _LHT defined by the parameter W _Guided of the W _Guided layer 122 including the destination hidden layer 124 and the parameter Wr of the regressor 126 is minimized. to learn the value of W _Guided , and to minimize the loss L _KD defined for the output of the Student network 120 as shown in FIG. train.

More specifically, referring to FIGS. 2-4 and 5, training of student network 120 is performed as follows. In the parameter set W _Hint prepared corresponding to the W _Hint layer 112 of the teacher network 110 in FIG. 2, the parameter set {W _T ¹ , . . . , W _T ^h } (step 140). In addition, among the parameter sets of the _student network 120, the parameter set of the student network 120 {W _S ¹ , . . . , W _S ^g } (step 142). Here, the parameter set W _Guided includes hidden layers W _Guided ¹ , . . . , W _Guided ^g .

At step 144, the parameter set Wr of the regressor 126 is initialized with a small randomly chosen value. At step 146, the value of the parameter set W* _Guided corresponding to W _Guided is calculated according to the following equation. The parameter set W* _Guided includes hidden layers _WGuided ^*1 , . . . , _WGuided ^*g .

続くステップ１４８で、上の式により計算されたＷ_{Ｇｕｉｄｅｄ}の各隠れ層Ｗ_{Ｇｕｉｄｅｄ} ^*1，…，Ｗ_{Ｇｕｉｄｅｄ} ^*gの値をスチューデントネットワーク１２０の隠れ層｛Ｗ_Ｓ ^１，…，Ｗ_Ｓ ^ｇ｝に代入する。以上でスチューデントネットワーク１２０の第１段階の訓練は終了である。

In ^the _{following step} ¹⁴⁸ , _the values of each hidden layer W _Guided ^{*1 ,} _. substitute. The above completes the first-stage training of the student network 120 .

After the first-stage training of the student network 120 is completed in this manner, learning of KD for the student network 120 is performed in step 150 according to the following equation.

上記非特許文献５に記載された技術は、ティーチャーネットワーク１１０よりも層数の多いスリムなスチューデントネットワーク１２０を、訓練済のティーチャーネットワーク１１０のパラメータを使用して訓練することにより、スチューデントネットワーク１２０の精度を高くできるという意味で有用である。スチューデントネットワーク１２０はティーチャーネットワーク１１０よりも層の数は多いが、各層に含まれるニューロンの数は少なく、したがって高い精度を保ちながら比較的高速に動作する。

The technique described in Non-Patent Document 5 uses the parameters of the trained teacher network 110 to train the slim student network 120, which has more layers than the teacher network 110, thereby improving the accuracy of the student network 120. is useful in the sense that Although the student network 120 has more layers than the teacher network 110, it contains fewer neurons in each layer and therefore operates relatively fast while maintaining high accuracy.

However, when the technique described in Non-Patent Document 5 is used, after the first stage of training of the student network is performed using the parameters of the trained teacher network, the output of the teacher network is used to further train the student network There is a problem that it is necessary to perform the second stage of training, and the processing is complicated. In addition, there is a problem that the precision is lowered in the case of a short speech (for example, less than 4 seconds), and the deterioration of the precision becomes more serious as the speech is shorter.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a training method for a spoken language recognition model that can recognize a spoken language with high accuracy even if the utterance is short.

A language discrimination model training method according to a first aspect of the present invention provides a first neural network for language discrimination of utterances related to a set of a predetermined number of languages, trained with utterance data of a first utterance time. to train a second neural network for language identification of utterances for a set of languages with utterance data of a second utterance time shorter than the first utterance time. Each piece of speech data of the second speech time is any part of the speech data of the first speech time. The first neural network includes a first number of convolutional layer groups arranged to receive speech data of a first speech time as an input and propagate the output of each layer, and the first number of convolutional layer groups. and a first classification network that receives the output of and outputs language identification information. The second neural network includes a second number of convolutional layer groups arranged to receive speech data of a second speech time as an input and propagate the output of each layer, and the second number of convolutional layer groups. and a second classification network for receiving the output of and outputting language identification information. The layer with the first number of convolutional layers is the source layer from which knowledge is transferred to the second neural network, and the layer with the second number of convolutional layers is the knowledge destination. This is the transfer destination layer. The method comprises the steps of preparing a first neural network in operational form; speech data for a first speech time when training the first neural network; speech data for the first speech time; Prepare training data in a machine-readable format containing training data consisting of utterance data of the second utterance time included in and language information indicating the utterance language of the utterance data of the first utterance time in association with each other a step, the output of the source layer when the utterance data of the first utterance time of the training data is input to the first number of convolutional layers, and the utterance data of the first utterance time associated with the Using at least the output of the transfer destination layer when the speech data of the second speech time is input to the second number of convolutional layers and the linguistic information of the speech data of the second speech time, the and training two neural networks.

Preferably, the number of neurons in the source layer is the same as the number of neurons in the destination layer.

More preferably, the source layer is the topmost layer of the first number of convolutional layers and the destination layer is the topmost layer of the second number of convolutional layers.

（ただしλは重み係数、Ｌ_ｈａｒｄ（ｘ_ｓ，ｙ）は、前記第２のニューラルネットワークに発話データｘ_Ｓが与えられたときの前記第２のニューラルネットワークの出力と、当該発話データｘ_Ｓに関連付けられた前記言語情報ｙとの間に定義される損失関数）発話データｘ_Ｓが与えられたときの第２のニューラルネットワークの出力と、当該発話データｘ_Ｓに関連付けられた言語情報とを用いて誤差逆伝播法により第２のニューラルネットワークのパラメータを更新するステップとを含む。 More preferably, the _step of training the second neural network includes output u _{calculating T} (x _T ; Θ _{T )} , where Θ _T represents the parameter set of the source layer; calculating the output of the destination layer u _S (x _S ; Θ _S ), where Θ _S represents the parameter set of the destination layer when input to the group of layers; calculating a function L _FRKD ;

(where λ is a weighting factor, L _hard (x _s , y) is the output of the second neural network when speech data x _S is given to the second neural network, and the speech data x _S Loss function defined between the associated linguistic information y) Using the output of the second neural network when given the utterance data _xS and the linguistic information associated with the utterance data _xS and updating the parameters of the second neural network by backpropagation.

Preferably, the method of training a language discrimination model further comprises using the training data set and the development data set to convert the parameters of the first neural network and the parameters of the second neural network to the inputs of the development data set. to minimize a loss function defined using the error of the outputs of the two neural networks, the error of the first neural network on the training data set, and the error of the second neural network on the training data set. includes an interactive parameter adjustment step that adjusts with

More preferably, the interactive parameter adjustment step uses the mini-batch data sets extracted from the training data set and the development data set to determine the error of the output of the second neural network with respect to the input of the development data set and the training data set a first adjustment step of adjusting the parameters of the first neural network so as to minimize a first error function defined using the error with the first neural network from the training data set; a second adjustment step of adjusting the parameters of the second neural network to minimize a second loss function defined using the second neural network's error for the mini-batch data set obtained from the first repeating the adjusting step and the second adjusting step alternately with varying mini-batch data sets for the training data set and the development data set until a predetermined termination condition is met.

A language discrimination model training device according to a second aspect uses a first neural network for language discrimination of utterances related to a set of a predetermined number of languages, which has been trained with utterance data of a first utterance time, A language discrimination model training device for training a second neural network for language discrimination of utterances relating to a set of languages, using utterance data of a second utterance time shorter than the first utterance time. Each of the second speech time speech data may be part of any of the first speech time speech data. The first neural network includes a first number of convolutional layer groups arranged to receive speech data of a first speech time as an input and propagate the output of each layer, and the first number of convolutional layer groups. and a first classification network that receives the output of and outputs language identification information. The second neural network includes a second number of convolutional layer groups arranged to receive speech data of a second speech time as an input and propagate the output of each layer, and the second number of convolutional layer groups. and a second classification network for receiving the output of and outputting linguistic information. The layer with the first number of convolutional layers is the source layer from which knowledge is transferred to the second neural network, and the layer with the second number of convolutional layers is the knowledge destination. This is the transfer destination layer. This apparatus includes a model storage device for storing a first neural network in an operable form, speech data of a first speech time when the first neural network is trained, and data of the first speech time. training data in a machine-readable format containing training data consisting of utterance data of a second utterance time included in the utterance data and language information indicating an utterance language of the utterance data of the second utterance time in association with each other; A training data storage device to store, an output of a transfer source layer when speech data of a first speech time of training data is input to a first number of convolutional layer groups, and a speech of the first speech time The output of the transfer destination layer when the utterance data of the second utterance time associated with the data is input to the second number of convolutional layers, and the linguistic information of the utterance data of the second utterance time and training means for using at least training the second neural network.
A computer program according to a third aspect of the present invention causes a computer to function as each means of any one of the above devices.

FIG. 4 is a schematic diagram showing an outline of a training method for a student network 60 disclosed in Non-Patent Document 4; 1 is a diagram schematically showing the schematic configurations of a teacher network and a student network disclosed in Non-Patent Document 5; FIG. FIG. 10 is a diagram schematically explaining the first-stage training method of the student network in Non-Patent Document 5; FIG. 10 is a diagram schematically illustrating a second-stage training method for a student network in Non-Patent Document 5; 10 is a flow chart showing an outline of a control structure of a program that implements student network training in Non-Patent Document 5. FIG. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram schematically showing an outline of a training method for a spoken language discrimination model according to the first embodiment of this invention; BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing the configuration of a training device that implements a method of training a spoken language discrimination model according to a first embodiment of the present invention; 4 is a flow chart showing the control structure of a computer program that causes a computer to implement the method of training a spoken language discrimination model according to the first embodiment of the present invention; 9 is a flow chart showing the control structure of part of the program shown in FIG. 8; 9 is a flow chart showing the control structure of part of the program shown in FIG. 8; FIG. 4 is a diagram showing in tabular form an overview of training data for each language used in the experiment. It is a figure which shows the structure of the teacher network used by experiment. FIG. 4 is a diagram showing the configuration of a student network used in experiments; It is a figure which shows the result of experiment in tabular form in contrast according to conditions. FIG. 4 is a diagram showing the distribution of feature amounts extracted in the spoken language identification model according to the first embodiment of the present invention, in comparison with the distribution of feature amounts in the baseline, the teacher model, and the model using the conventional technique; be. It is a block diagram showing a schematic configuration of a training device according to a second embodiment. 9 is a flow chart showing a control structure of a program that causes a computer to function as a training device according to a second embodiment; FIG. 10 is a diagram schematically showing a procedure for parameter adjustment of a teacher network in the training device according to the second embodiment; FIG. 10 is a diagram schematically showing a procedure for parameter adjustment of a student's network in the training device according to the second embodiment; FIG. 9 is a diagram showing experimental results of the training device according to the second embodiment in tabular form, together with experimental results of the conventional technique and the training device according to the first embodiment; FIG. 10 is a diagram showing, in tabular form, experimental results for confirming the accuracy of the training device according to the second embodiment for different utterance lengths, together with experimental results for the training device according to the prior art and the first embodiment; . 1 is an external view of a computer system that implements a method of training a spoken language discrimination model according to each embodiment of the present invention; FIG. 23 is a block diagram showing the hardware configuration of the computer system shown in FIG. 22; FIG.

In the following description and drawings, identical parts are provided with identical reference numerals. Therefore, detailed description thereof will not be repeated.

[First embodiment]
<Configuration>
[Outline of training]
FIG. 6 shows an outline of the configuration of a training method 200 for a spoken language discrimination model according to the first embodiment of this invention. Referring to FIG. 6, a training method 200 according to this embodiment uses a trained teacher network 210 to train a student network 220 .

The teacher network 210 is trained using training data consisting in this embodiment of speech data 230 of four seconds of speech duration, and in response to four seconds of speech data input, the language of the speech data is a predetermined number ( It is trained to output an output vector 212 indicating which of 10 languages it is in, for example. The training itself uses normal techniques. The training samples used for this training include 4 seconds of speech data and a label representing the spoken language of the speech data.

The teacher network 210 includes a feature extraction unit 214 including a plurality of hidden layers each made up of convolution blocks, and two layers for classifying the spoken language based on the feature amount output by the feature extraction unit 214 and outputting an output vector 212 . and a fully connected layer network 216 consisting of fully connected layers of layers. A softmax layer (not shown) exists in the final stage of the fully connected layer network 216 . A deep neural network (Deep Convolution Neural Network) including a plurality of convolution blocks is hereinafter referred to as "DCNN".

Feature extraction unit 214 includes a plurality of convolution blocks, and in this embodiment, convolution block 218 at the highest layer transfers knowledge about feature extraction acquired through training to student network 220 in a manner described below. . This knowledge is represented in the output of convolution block 218 .

The student network 220, like the teacher network 210, consists of a DCNN. The student network 220 was trained using training data comprising speech data 232 with an speech time of 2 seconds shorter than the speech data 230, and the parameters of the convolution block 218 of the teacher network 210, resulting in speech data with an speech time of 2 seconds. In response to the input of the teacher network 210, an output vector 222 indicating in which of the 10 languages the utterance by the input utterance data was uttered, similar to the output vector 212 of the teacher network 210, is output.

Similar to the feature extraction unit 214 of the teacher network 210, the student network 220 classifies the input utterance based on the feature extraction unit 224 including a plurality of convolution blocks and the feature amount output by the feature extraction unit 224, and the output vector and a fully connected layer network 226 consisting of two fully connected layers for outputting 222 . The fully connected layer network 226 includes a softmax layer (not shown) in the final layer.

The feature extractor 224 also includes multiple convolutional blocks, but in this embodiment transfers the knowledge of the convolutional block 218 to the highest level convolutional block 228 . Therefore, the number of neurons in convolutional block 228 is the same as the number of neurons in convolutional block 218 . On the other hand, since the utterance length of the input to the student network 220 is half the utterance length of the input to the teacher network 210, the number of neurons in the lowest convolutional layer of the student network 220 is the same as that of the lowest neuron in the teacher network 210. Half. Similarly, the partial convolutional block in feature extractor 224 has fewer neurons than each convolutional block in feature extractor 214 . This configuration allows the student network 220 to operate faster than the teacher network 210 .

[Training Device for Spoken Language Discrimination Model]
FIG. 7 shows, in block diagram form, a schematic configuration of a training device that implements the method of training a spoken language discrimination model according to this embodiment. Referring to FIG. 7, this training apparatus 250 includes a teacher network training data storage unit 260 for storing training data including training samples used for training of teacher network 210, and teacher network 210 in operable form. Using the first language discrimination model storage unit 262 for storing and the training data stored in the teacher network training data storage unit 260, the teacher network 210 stored in the first language discrimination model storage unit 262 is normalized and a Teacher Network Trainer 264 for training using backpropagation. Each training sample includes, in this embodiment, utterance data with an utterance time of, for example, 4 seconds and a language label that identifies the language of the utterance.

The first language identification model storage unit 262 includes a teacher network program storage unit 280 for storing a program that defines the algorithm of the function implemented by the teacher network 210, parameters that determine the behavior of the function implemented by this algorithm, and and a teacher network parameter storage 282 for storing hyperparameters that define the configuration of the teacher network 210 . The hyperparameters here include information such as the number of layers of the network, the size and number of filters used by each convolutional layer, the window size and its stride for max pooling.

The training device 250 further transforms each of the training samples stored in the teacher network training data storage unit 260 to output training samples containing speech data of 2 seconds of speech time and the original 4 seconds of speech data. It includes a training data converter 266 and a student network training data storage 268 for storing the training samples output by the training data converter 266 in a machine-readable format. Each piece of utterance data with an utterance time of 2 seconds is associated with the original utterance data with an utterance time of 4 seconds.

The training data conversion unit 266 divides, for example, the 4-second utterance data of each training sample stored in the teacher network training data storage unit 260 into utterance data with an utterance time of 2 seconds each, and divides the resulting two utterances. New training samples are generated by correlating and combining each of the data with the original 4 seconds of speech data and labeling them with the original training samples. That is, two training samples for the student network 220 are obtained from one training sample for the teacher network 210, and each training sample consists of two seconds of speech data and four seconds of speech data from which the speech data is based. , and a label indicating the language of the speech data. In this way, if short-time speech data is a divisor of long-time speech data, it is convenient for generating short-time speech data. However, the present invention is not limited to such an embodiment, and the short speech duration need not be a divisor of the long speech duration.

The training device 250 further uses the training data of the student network 220 stored in the student network training data storage unit 268 and the language discrimination model consisting of the teacher network 210 stored in the first language discrimination model storage unit 262 to and a second language discrimination model storage unit 270 for storing the student network 220.

The second language identification model storage unit 270 includes a student network program storage unit 290 for storing a program that defines the algorithm of the function implemented by the student network 220, parameters that determine the behavior of the function implemented by this algorithm, and and a student network parameter storage unit 292 for storing hyperparameters that define the configuration of the student network 220 .

8-10 are flow charts showing the control structure of a computer program that controls a computer to function as the student network training unit 272 shown in FIG. Referring to FIG. 8, the program trains 320 the teacher network 210 using the training data for the teacher network 210 and trains the student network using the trained teacher network 210 and hard labels. 322.

FIG. 9 shows details of step 320 of FIG. Referring to FIG. 9, the program includes the steps of preparing 330 training data for the teacher network 210, initializing 332 the parameter set of the teacher network 210, and repeating the training process 336 for a planned number of training runs. 334 and a training process 338 that saves the resulting set of parameters for teacher network 210 from step 334 in teacher network parameter storage 282 (FIG. 7) and terminates the process.

The training process 336 includes a step of sampling 350 a mini-batch of m samples from the training data set and calculating the difference between the output of the teacher network 210 when inputting the mini-batch of speech data to the teacher network 210 and the hard labels in the training data. and a training process 336 by backpropagating the parameters of all layers of the teacher network 210 to minimize the following loss function L _T (x _T ,y) computed for the mini-batch: and step 354 of terminating. In the following equation, "T" on the right shoulder of label y represents transposition. On the other hand, the subscript “T” indicates that the value is for the teacher network 210 . Similarly, a loss function L _S (x _S ,y) for Student's network 220 is defined.

FIG. 10 shows details of step 322 of FIG. Referring to FIG. 10, the program prepares 360 training data for training the student network 220 from the trained teacher network 210 and hard labels, and the student network 220 stored in the student network parameter storage 292. and step 362 of initializing the parameter set Θ _S of . In this embodiment, the initialization in step 362 uses a method called Glorot's uniform distribution method. Of course, initialization is not limited to this technique, and other techniques may be used.

In step 360, as shown in FIG. 7, the training data conversion unit 266 generates training data with an utterance length of 2 seconds from the training data stored in the teacher network training data storage unit 260. are stored in the student network training data storage unit 268 together with the training data of . Here, the training data with an utterance length of 4 seconds is divided at the center to generate two training data with an utterance length of 2 seconds. However, the invention is not limited to such embodiments. From the original training data, a plurality of training data with overlapping utterance parts may be generated.

Further, this program uses the training data of the student network 220 prepared in step 360 to repeat the following training process 366 for a predetermined number of executions, and to perform a step 364 when the training in step 364 is completed. and a step 368 of saving the parameter set Θ _S of the student network 220 in the student network parameter storage unit 292 shown in FIG. 7 and ending the process. In this embodiment, the training is terminated when the training process 366 is repeated a specified number of times, but the repetition may be terminated when another termination condition is satisfied. For example, the iteration may be terminated when the absolute value of the mini-batch parameter update magnitude is equal to or less than a threshold for a predetermined number of consecutive times.

The training process 366 includes a step of sampling 380 a mini-batch of m training samples from the training data, and inputting 2 seconds of speech data from each of these m training samples to the student network 220, and outputting the student network 220 and a step 382 of calculating the difference between the hard label. At step 382, the output of convolutional block 228, which is the destination hidden layer of Student's network 220, is also computed and stored. The output of the destination convolution block 228 is a feature quantity denoted u _S (x _S ; Θ _S ). where x _S represents two seconds of speech data input to the student network 220, Θ _S represents the parameter set of the convolution block 228 to which knowledge is transferred, and u _S represents the feature up to the convolution block 228. A function defined by the extractor 224 is represented. It is desirable that the number of m training samples constituting each mini-batch is at least equal to or greater than the number of languages to be identified. For example, it is desirable to sample from each language by the same number for each language, or at a rate proportional to the number of samples so that the total is m.

The training process 366 further inputs 384 the four seconds of speech data into the teacher network 210 for each of the training data and computes 384 the output of the convolution block 218 from which the teacher network 210 transfers knowledge at that time. including. The output of convolution block 218 is a feature denoted u _T (x _T ; Θ _T ). where x _T represents the 4 seconds of speech data input to the teacher network 210, Θ _T represents the parameter set of the convolution block 218, and u _T is defined by the feature extractor 214 up to the source layer. represents a function.

The training process 366 further includes, following step 384, a step 386 of calculating a loss function L _FRKD represented by equation (6) below.

訓練処理３６６はさらに、ステップ３８６に続き、スチューデントネットワーク２２０の全ての層のネットワークのパラメータを、上記した損失関数を最小化するよう通常の誤差逆伝播法で更新し、対象となるミニバッチによる訓練処理３６６を終了するステップ３８８を含む。

The training process 366 further continues to step 386 to update the network parameters of all layers of the student network 220 by conventional backpropagation to minimize the loss function described above, and to train the target mini-batches. It includes a step 388 ending 366 .

<Action>
6-10, training device 250 according to this embodiment operates as follows. First, training data for teacher network 210 is prepared in teacher network training data storage unit 260 . The teacher network program storage unit 280 stores in advance a program that defines the algorithm of the teacher network 210 . Teacher network training unit 264 trains teacher network 210 using this training data and the program stored in teacher network program storage unit 280 (step 320 in FIG. 8).

In training the teacher network 210, as shown in FIG. 9, step 330 is first prepared to initialize the parameter set of the teacher network 210 (step 332). The training process 336 is then repeated for the planned number of executions of training (step 334). The parameter set of the teacher network 210 when this step 334 is completed is stored in the teacher network parameter storage unit 282 (FIG. 7). As a result, the teacher network parameter storage unit 282 stores parameter sets for each layer of the teacher network 210 . This parameter set includes the parameter set Θ _T of the source convolutional block 218 shown in FIG.

After the training of the teacher network 210 is completed, the training data conversion unit 266 converts the training data for the teacher network stored in the teacher network training data storage unit 260 to generate the training data for the student network 220 (step 360). The converted training data is stored in the student network training data storage unit 268 . Each training sample stored in the student network training data storage unit 268 is based on two seconds of speech data obtained by dividing the four seconds of speech data of the training sample stored in the teacher network training data storage unit 260. It contains 4 seconds of speech data and a label.

The student network program storage unit 290 stores a program (student network program) that defines the algorithm of the student network training unit 272 . Hyperparameters that define the configuration of the student network 220 are stored in the student network parameter storage unit 292 in advance, and a storage area for storing the parameters of the student network 220 is secured.

In training student network 220, student network trainer 272 prepares training data for student network 220 (step 360). Specifically, the student network training unit 272 accesses the student network training data storage unit 268 shown in FIG. 7, opens a training data file, and loads it into memory. Subsequently, the storage area for the set of student network parameters secured in the student network parameter storage unit 292 is initialized (step 362). This parameter set includes the parameter set Θ _S of the destination convolutional block 228 shown in FIG.

Subsequently, the student network training unit 272 executes step 364 for repeating the following process (training process 366 in FIG. 10) for a predetermined scheduled number of times.

In training process 366, student network trainer 272 samples a mini-batch of m training samples from the training data stored in student network training data storage 268 (step 380). Two seconds of utterance data x _S in each training sample included in this mini-batch is input to the student network 220, and the error between the identification result y output from the student network 220 and the hard label corresponding to each is calculated. . At the same time, the output u _S (x _S ; Θ _S ) of the convolution block 228, which is the highest layer of the feature extraction unit 224 of the student network 220, is calculated (step 382).

Subsequently, in step 384 , 4 seconds of utterance data x _T including 2 seconds of sampling data included in the mini-batch is input to the teacher network 210 in each training sample, and transferred to the top layer of the feature extraction unit 214 . Compute the output u _T (x _T ; Θ _T ) from the original convolution block 218 .

Further, the student network training unit 272 calculates the value of the loss function L _FRKD represented by Equation (6) (step 386).

At the end of the training process 366, all the parameters of the student network 220 are updated using conventional backpropagation to minimize the value of the loss function _LFRKD , and the training process 366 ends for one mini-batch. (step 388).

Training of the student network 220 is completed by repeating the above processing for the number of times of execution. The student network training section 272 saves the parameter set of the student network 220 at this time in the student network parameter storage section 292 (step 368), and completes the training of the student network 220. FIG.

<Experimental results>
An experiment was conducted using a student network trained by the training method according to the above embodiment, and it was confirmed whether or not the above training method was effective.

FIG. 11 shows an overview of the training data for each language used in this experiment. Referring to FIG. 11, this experiment used speech data in Burmese, Chinese, English, French, Indonesian, Japanese, Korean, Spanish, Thai, and Vietnamese. These speech data were prepared by the applicant.

The speech data for each language was divided into a training dataset, a validation dataset, and a test dataset. Information about these data for each language is shown in FIG. 11, and for each language consists of 4500 training datasets, 300 validation datasets and 1200 test datasets. However, the total time of each utterance data does not match each other as shown in FIG. The average duration of speech data is 7.6 seconds. All of these utterance data are data uttered by a native speaker.

Spoken language discrimination error rate (UER) was used to evaluate the model in the experiment.

In the experiment, a DCNN configured as shown in FIG. 12 was prepared as a teacher network, and a DCNN configured as shown in FIG. 13 was prepared as a student network. The example shown in FIG. 12 is for an input utterance length of 4 seconds, and the example shown in FIG. 13 is for an input utterance length of 2 seconds.

Referring to FIG. 12, the teacher network DCNN receives 4 seconds of speech as input. The audio signal is framed with a frame length of 25 msec and a frame interval (frame shift interval) of 10 msec. Since the utterance length of the input is 4 seconds, the total number of frames for one input is 4×1000/10=400 frames. Each frame is a 60-dimensional vector of 60 elements obtained by passing the speech signal through a 60-dimensional mel filter bank. That is, a 400×60 matrix is input to the DCNN. In FIG. 12, for example, "conv(7×7, 16, relu)" in the first line means that the filter size used in the convolution block is 7×7, the number of filters is 16, and the activation function is Indicates Rectified Linear Unit. "max-pooling (3 × 3, stride 2 × 2)" is a 3 × 3 area, and the max pooling is performed while moving two by two horizontally, and when it moves to the end of the line, the target area is moved vertically (downward) two times to perform the same processing. "BN" indicates that batch normalization processing is performed on the convolution block.

Referring to FIG. 13, the DCNN of student network 220 receives two seconds of speech as input. An input speech signal is framed and each frame is transformed into a 60-dimensional vector. This results in a 200x60 matrix as input to the DCNN. 13 differs from FIG. 12 in that, in addition to the difference in the number of input vectors (the number of neurons in the input layer), the stride of max-pooling in the third layer of the convolution block is 1×2 (2× 2) is the point. This is an adjustment to equalize the number of neurons in the transfer source layer and the transfer destination layer while the number of dimensions of the input data differs between FIG. 12 and FIG. 13 . Otherwise, the view of FIG. 13 is similar to that of FIG.

In this experiment, a DCNN having the same configuration as that shown in FIG. 13 was trained with training data having an utterance length of 2 seconds as a baseline. Furthermore, a DCNN with an utterance length of 2 seconds as an input is trained by the KD method (Non-Patent Document 1) using a teacher network with an utterance length of 4 seconds, and prepared as a comparative example with this embodiment. bottom. Comparative examples using the KD approach were constructed at combinations of temperatures T=3, 5, 7 and λ=0.1, 0.3, 0.5 and 0.7.

As the student network of this embodiment, a DCNN with an input utterance length of 2 seconds was prepared with the same configuration as in FIG. . At this time, λ was 0.1, 0.3, 0.5 and 0.7.

The results of the experiment are shown in FIG. Referring to FIG. 14, in the KD method, T=3 and λ=0.3 showed the highest performance (UER=5.69%) in the test. This is an improvement of 17.18% compared to baseline (UER=6.87%). On the other hand, in the network (FRKD) trained by the method of the present embodiment, the one that adopted the L1 norm as the loss function for knowledge transfer training with λ = 0.3 had the highest performance (UER = 5.28%). showed that. This is an 18.94% improvement compared to baseline and a 7.21% improvement compared to the KD method.

In this experiment, we also performed a performance comparison between the L1 norm fixed at λ=0.5 and the L2 norm. As a result, the former UER was 5.33%, while the latter UER was 5.89%. Therefore, it was found that the performance using the L1 norm is relatively better than the one using the L2 norm by 9.51%. The one using the L2 norm has lower performance than the best performance of the KD method (UER=5.69%).

From the above, the model trained by the method of this invention can obtain higher performance than that using the KD method, and that the L1 norm instead of the L2 norm for the loss function can achieve higher performance.

FIG. 15 shows the feature values obtained from the baseline model used in the experiment, the teacher network used for training the student network, the student network trained by the KD method, and the student network trained by the method according to the present invention. A compressed visualization is shown. This figure represents the distribution of the feature values output by each model for the verification data set like a two-dimensional scatter diagram. The teacher model has an utterance length of 4 seconds, and the baseline and two student models have an utterance length of 1 second.

As shown in FIG. 15B, in the diagram of the teacher network, the feature values are neatly distributed among 10 clusters. These 10 clusters correspond to each language. Therefore, it can be seen that ten languages can be identified with high accuracy using the teacher network. On the other hand, in the baseline shown in FIG. 15(A), the separation of the feature quantity is not well done, and it can be seen that the language discrimination performance is low. The KD method shown in FIG. 15(C) separates the feature quantity better than the baseline shown in FIG. 15(A), but is still not sufficient. On the other hand, in the network according to the present invention shown in FIG. 15(D), the feature values are well separated and are close to those of the teacher network shown in FIG. 15(B). Therefore, from FIG. 15 as well, it can be seen that the DCNN training method according to the present invention enables highly accurate speech language identification.

<Modification>
In the above embodiment, the teacher network was trained with four seconds of speech data, and the student network was trained with two seconds of speech data. However, the length of the speech data length is not limited to this combination. Also, it is sufficient if the utterance length of the student network is shorter than the utterance length of the teacher network. However, considering the purpose of the present invention, it is desirable that the utterance length for training the student network is 2 seconds or less. Furthermore, in the above embodiment, one is a multiple of the other, that is, the other is a divisor of the other, such as 4 seconds and 2 seconds. However, it is not limited to such combinations. However, considering the ease of preparation of training data and the ease of determination of hyperparameters of a convolutional network, it is realistic to prepare training data with utterance lengths that are in the relationship of multiples and divisors.

Also, in the above experiments, both the teacher network and the student network contain 7 layers of convolutional blocks. Also, both the source layer and the destination layer are the top hidden layers of these convolutional blocks. However, the invention is not limited to such embodiments. The number of convolution blocks may be other than seven layers, and the number of convolution blocks of the teacher network and the student network may not match. In addition, the transfer source layer or the transfer destination layer may not be the highest convolutional block of the convolutional block group from which the feature value is extracted, and the positions of both in the convolutional block group need to match. do not have.

In the above embodiments, training with the L _FRKD loss was applied to the DCNN for language identification. In this DCNN, what the convolutional block group outputs corresponds to the feature amount, and language identification is performed by inputting the feature amount to the fully connected layer. However, the invention is not limited to such embodiments. For example, speaker identification can be performed using features extracted by convolutional block groups trained by the same technique. However, in this case, it is practical to use the DCNN as a two-class classifier that uses features to determine whether an utterance is made by a specific speaker.

[Second embodiment]
The second embodiment described below improves the model obtained by the first embodiment by a new method of interactive training (interactive parameter adjustment) between the teacher network and the student network. is added. As will be described later, the Student's network trained in this manner showed even higher accuracy than the one trained in the first embodiment. In addition, it was confirmed that the accuracy at the same utterance time was higher. Therefore, if language discrimination is performed with the same accuracy, the necessary utterance length can be shortened.

<Configuration>
FIG. 16 shows a schematic configuration of a training device 400 according to the second embodiment in block diagram form. Referring to FIG. 16, training apparatus 400 includes teacher network training data storage unit 260, first language identification model storage unit 262, teacher network training unit 264, training data It includes a conversion unit 266 , a student network training data storage unit 268 , a student network training unit 272 and a second language discrimination model storage unit 270 .

The training device 400 further includes a development data storage unit 412 that stores development data sets input to the student network 220 for parameter adjustment of the teacher network 210 that constitutes the first language discrimination model, and a teacher network training data storage unit. 260, the student network training data stored in the student network training data storage unit 268, and the development data stored in the development data storage unit 412, the teacher network 210 and the student network 220 are developed. Training data storage device 414 for storing training data for adjusting the parameters of and interactive processing between the teacher network 210 and the student network 220 using this training data, the teacher network and the student network and an interactive model adjuster 410 for adjusting both parameters of , thereby making the accuracy of language identification by the student network higher compared to that according to the first embodiment.

FIG. 17 shows in flow chart form the control structure of a computer program for causing a computer to function as training device 400 . FIG. 18 schematically shows an overview of the parameter update of the loss function and teacher network 210 and student network 220 in the second embodiment.

Referring to FIG. 17, the program uses the teacher network training data to train a teacher network 210 that constructs a first language discrimination model, step 320, and the parameter set of the teacher network 210 trained by step 320. Among these are a step 450 of saving the parameter set Θ ⁰ _T of the top layer convolution block 218 (FIG. 18) to storage, and following step 450, training the Student network 220 using the Student network training data. 322, following step 322, the training data used for parameter adjustment of the teacher network 210 and the student network 220 are obtained using the teacher network training data, the student network training data, and the development data stored in the development data storage unit 412. and step 452 of preparing training data store 414 . The training data here is a set of {training data of the teacher network 210, training data of the student network 220, development data, and labels}. Among these, the set of {training data for teacher network 210, training data for student network 220, label} is used for parameter adjustment of student network 220, and the set of {training data for teacher network 210, development data, label} is It is used for parameter adjustment of the teacher network 210 . Thus, as shown in FIG. 18, training data that is input to student network 220 includes development data samples 520 . The development data set sample 520 is a set of utterance data with an utterance length of 2 seconds and a label y indicating the language of the utterance.

The program also includes a step 454 that executes the outer loop process 456 for a planned number of training runs, and stores the parameter set of the student network 220 at the completion of step 452 in a storage device such as the student network parameter storage 292 of FIG. and step 458 of storing to and terminating execution of the program.

The outer loop process 456 includes a step 480 of shuffling the training data and a step 482 of using the shuffled training data to perform an inner loop process 484 described below until all the training data has been processed.

The inner loop process 484 includes a step 500 of sampling a mini-batch of m samples from the training data mini-batch set, and among the mini-batches sampled in step 500, the tuple {training data, development data, label of teacher network 210}. and adjusting 502 the parameters of the teacher network 210 to minimize the following loss function ^̂L _T using .).

この式中で使用されている損失関数Ｌ_Ｔ及びＬ_Ｓはそれぞれ、ティーチャーネットワーク２１０の損失関数及びスチューデントネットワーク２２０の損失関数を表す。この損失関数としては、誤差逆伝播法が使用可能なものであればどのようなものを用いてもよいが、この実施の形態では従来の技術で紹介した式（３）と同様のものを用いる。γ及びξは実験により定めるパラメータである。ノルム１の項は、このパラメータ調整において、ティーチャーネットワーク２１０の最上位層の畳込みブロック２１８のパラメータが、元の畳込みブロック２１８のパラメータ（初期値Θ^０ _Ｔ）からあまり離れてしまわないようにするための制約である。

The loss functions L _T and L _S used in this equation represent the loss function of teacher network 210 and the loss function of student network 220, respectively. Any loss function can be used as long as the error backpropagation method can be used. . γ and ξ are parameters determined by experiments. The norm-1 term is used in this parameter adjustment so that the parameters of the top layer convolutional block 218 of the teacher network 210 do not deviate too far from the original convolutional block 218 parameters (initial value Θ ⁰ _T ). It is a constraint for

The program further uses {training data for teacher network 210, training data for student network 220, labels} in the mini-batches sampled in step 500 to minimize the following loss function ^ _LS A step 504 of adjusting the parameters of the network 220 is included.

式（８）中において、Ｌ_Ｓ及びＬ_ｋｔはそれぞれ第１の実施の形態で使用したものと同じである。定数λはＬ_ＳとＬ_ｋｔとのバランスを取るための定数であり、実験により定められる。

In equation (8), L _S and L _kt are the same as those used in the first embodiment. A constant λ is a constant for balancing L _s and L _kt and is determined by experiments.

18 and 19, inner loop processing 484 of FIG. 17 will be described. At step 502 of FIG. 17, teacher network 210 is provided with utterance data 230 (samples x _T ), the training data for teacher network 210 , for each sample in each mini-batch. Output vector 212 is output from fully connected layer network 216 of teacher network 210 . The loss function L _T computed between this output vector 212 and label y forms part of equation (7) as indicated by arrow 532 . Convolution block 218 of teacher network 210 outputs u _T (x _T ; Θ _T ). u _T (x _T ; Θ _T ) represents the output vector of convolution block 218 when the parameter set of convolution block 218 is parameter set Θ _T and the input of teacher network 210 is sample x _T . This vector forms part of equation (7) as represented by arrow 536 .

On the other hand, when development data samples 520 (x _S ) are input to student network 220 , student network 220 outputs output vector 522 . The loss function L _S computed between output vector 522 and label y forms part of equation (7) as indicated by arrow 534 .

In addition, with the same configuration as the teacher network 210, instead of the convolution block 218, a convolution block 528 having an initial parameter set Θ ⁰ _T stored in advance instead of the parameter set Θ _T is used to obtain a network 524. be done. Network 524 includes feature extractor 526 , the top hidden layer of which is convolution block 528 . This network 524 is otherwise identical to teacher network 210 . Speech data 230 (x _T ) is input to this network 524 . The output u _T (x _T ; Θ ⁰ _T ) of convolution block 528 then forms part of equation (7), as indicated by arrow 538 .

The parameters of teacher network 210 are adjusted so as to minimize the loss function of equation (7) thus obtained.

On the other hand, referring to FIG. 19, parameter adjustment of student network 220 is performed as follows. The teacher network 210 is provided with speech data 230 (x _T ). The student network 220 is provided with utterance data 232 (with an utterance length of 2 seconds) corresponding to the utterance data 230 (with an utterance length of 4 seconds). A loss function Ls defined between the output vector 222 output by the student's network 220 and the label y forms part of equation (8) as indicated by arrow 560 . Similarly, the loss function _Lkt used in the first embodiment is calculated from the output vector of convolution block 218 and the output vector of convolution block 228 . This loss function L _kt also forms part of equation (8) as indicated by arrow 562 .

In equations 8 and 9, λ, γ, and ξ are hyperparameters, and by varying all of them within the range of (0, 1), the optimum combination is obtained through experiments.

<Action>
Referring to FIG. 16, teacher network 210 and student network 220 are trained by teacher network training section 264, training data conversion section 266 and student network training section 272, as in the first embodiment. After completing these trainings, the teacher network 210 is stored in the teacher network training data storage unit 260 . A student network 220 is stored in the second language identification model storage unit 270 .

After that, the interactive model adjustment unit 410 further adjusts the parameters of the teacher network 210 and the student network 220 using the development data storage unit 412 . The development data storage unit 412 stores in advance the training data having the configuration described above.

Referring to FIG. 17, training of the language identification model by training device 400 is realized by causing a computer to function as follows. At step 320, the teacher network training data is used to train the teacher network 210 that constructs the first language discrimination model. At step 450, among the parameter sets of the teacher network 210 trained by step 320, the parameter set Θ ⁰ _T of the top layer convolutional block 218 (FIG. 18) is stored in storage. Further, at step 322, the student network 220 is trained using the student network training data. In step 452 , training data used for parameter adjustment of the teacher network 210 and the student network 220 are stored in the training data storage device using the teacher network training data, the student network training data, and the development data stored in the development data storage unit 412 . Prepare for 414.

The training data here is a set of {training data of teacher network 210, training data of student network 220, development data, label}. Among these, the set of {training data for teacher network 210, training data for student network 220, label} is used for parameter adjustment of student network 220, and the set of {training data for teacher network 210, development data, label} is It is used for parameter adjustment of the student network 220 .

Further, in step 454, the outer loop process 456 is executed for the planned number of training runs.

Step 480 of the outer loop process 456 shuffles the training data. In a subsequent step 482, this shuffled training data is used to execute an inner loop process 484, described below, until all training data has been processed.

Step 500 of the inner loop process 484 samples a mini-batch of m samples from the mini-batch set of training data. In the following step 502, the teacher network 210 is trained to minimize the loss function ̂L _T shown in equation (7) using the set {training data, development data, labels of the teacher network 210} in the sampled mini-batches. Adjust the parameters of Furthermore, in the following step 504, using {training data of the teacher network 210, training data of the student network 220, labels} in the mini-batch sampled in step 500, the loss function ^L _S shown in equation (8) is Adjust the parameters of the student network 220 to minimize

The inner loop process 484 is then repeated until all training data has been processed, and the outer loop process 456 is repeated for the planned number of training runs.

<Experimental results>
An experiment was conducted to confirm the performance of the student network trained by the training apparatus 400 according to the second embodiment. The results of the experiment are shown in FIG. 20 together with the results of the training device 250 according to the first embodiment and its perturbation. Here, the term “vertervation” means learning by superimposing random noise on the loss L _kt (L _kt =||U _T −U _S +Noise||) in the training apparatus according to the first embodiment. ing. According to experiments, there are cases where the accuracy is slightly improved by performing bartervation during learning of the training apparatus according to the first embodiment. In this experiment, in consideration of the results of the first embodiment shown in FIG. 20, λ was set to 0.3, and the value of γ was changed to 0.1, 0.2 and 0.3. rice field. "ValId." indicates the accuracy (URE%) on the validation dataset by the student network, and "Test" indicates the accuracy on the test dataset.

Referring to FIG. 20, in this second embodiment, the language identification accuracies for the test data at γ=0.1, 0.2 and 0.3 are 4.78, 5.12 and 4.78, respectively. was 86. All of these exceed the best accuracy (5.28 when λ=0.3) in the first embodiment. Not only that, but it also exceeds the accuracy of perturbation (5.17).

As a result, according to the second embodiment, a student neural network is trained to achieve high discrimination accuracy by following a fixed procedure without being affected by chance results such as perturbation. I know you can.

FIG. 21 shows the language discrimination accuracy (URE%) of the student neural network (ITSL: Interactive Teacher-Student Learning) trained by the method according to the second embodiment compared with other methods in tabular form. is shown. Comparisons are baseline (DCNN), according to the first embodiment (FRKD), and with perturbation to the first embodiment (FRKD-P). In these experiments, a teacher model with an utterance length of 4 seconds as an input was used. ISTL set λ=0.3, γ=0.1, and ξ=0.1.

As shown in FIG. 21, the student model (ITSL) according to the second embodiment exhibits accuracy superior to other methods for all utterance lengths. Specifically, the ISTL showed 30.4%, 22.7% and 16.0% improvement over the base model for speech lengths of 2, 1.5 and 1.0 seconds, respectively. ISTL also showed improvements of 9.5%, 6.1%, and 8.7 for FRKD, respectively. Against FRKD-P, ISTL also showed 7.5%, 4.6%, and 7.3% improvement, respectively.

From these results, it can be seen that the language identification model consisting of the student's neural network according to the second embodiment can identify the language with high accuracy even for input with a short utterance length. This shows that the convolution block 228 (see FIGS. 18 and 19) of this model can extract robust features from short utterance length inputs. Therefore, we show that it can be applied not only to language identification, but also to tasks that require some kind of judgment with a short utterance length, such as speaker authentication.

[Realization by computer]
The training apparatuses 250 and 400 according to the embodiments of the present invention and the training method of the language discrimination model by them can be realized by computer hardware and a computer program executed on the computer hardware. 22 shows the appearance of this computer system 630, and FIG. 23 shows the internal configuration of the computer system 630. As shown in FIG.

Referring to FIG. 22, this computer system 630 includes a computer 640 having a memory port 652 and a DVD (Digital Versatile Disk) drive 650, a keyboard 646, a mouse 648 and a monitor 642.

Referring to FIG. 23, computer 640 includes CPU (Central Processing Unit) 656 and GPGPU (General Purpose Image Processing Unit) 657 in addition to memory port 652 and DVD drive 650 . Computer 640 is further coupled to bus 666 which is coupled to CPU 656, GPGPU 657, memory port 652 and DVD drive 650; read only memory (ROM) 658 for storing boot programs and the like; and a random access memory (RAM) 660 for storing working data and the like, and a hard disk 654 . The computer system 630 further includes a network interface (I/F) 644 that provides connection to a network 668 that enables communication with other terminals, and an audio I/F 670 that has the function of inputting and outputting speech signals as audio signals. including.

The computer program for causing the computer system 630 to function as each functional unit of the training device 250 or 400 according to the above-described embodiment to execute each step of the training method for the language discrimination model is stored in the DVD drive 650 or the memory port 652. DVD 662 or removable memory 664 mounted on the hard disk 654 and further transferred to the hard disk 654 . Alternatively, the program may be transmitted to computer 640 over network 668 and stored on hard disk 654 . Alternatively, a source program may be created using the keyboard 646, mouse 648, monitor 642, an editor program, etc., compiled by a compiler, and an object program stored in the hard disk 654. FIG. Programs are loaded into RAM 660 during execution. Programs may be loaded directly into RAM 660 from DVD 662 , from removable memory 664 , or via network 668 .

This program includes a sequence of instructions for causing the computer 640 to function as the training device 250 or 400 according to the above-described embodiment to execute each step of the language discrimination model training method. The CPU 656 and GPGPU 657 are used for a large amount of numerical calculation processing executed in the training of the teacher network 210 and the student network 220 (only the CPU 656 may be used, but using the GPGPU 657 is faster). Some of the basic functions required to make computer 640 perform this operation are the operating system or third party programs running on computer 640, or various dynamically linkable programming toolkits installed on computer 640. or provided by a program library. Therefore, this program itself does not necessarily include all the functions necessary to implement the system, apparatus and method of this embodiment. The program executes the training apparatus described above by dynamically calling, at run time, appropriate functions or programs in a programming toolkit or program library in a controlled manner to achieve the desired result of the instructions. As 250, it need only contain instructions that implement the functions of performing each step of the language discrimination model training method. Of course, the program alone may provide all necessary functions.

In the above embodiment, as shown in FIG. 10, the value u _T (x _T ; Θ _T ) is calculated by the teacher network 210 at the same time as mini-batch processing. However, the invention is not limited to such embodiments. This value can be calculated independently of the student network 220 training process. Therefore, prior to training the student network 220, it may be pre-calculated using each sample. In this case, it is necessary to store this value in a storage device.

The embodiments disclosed this time are merely examples, and the present invention is not limited only to the above-described embodiments. The scope of the present invention is indicated by each claim after taking into account the description of the detailed description of the invention, and all changes within the meaning and range equivalent to the wording described therein include.

40, 100 system 50, 110, 210 teacher network 52, 62 division 54, 64, 68 softmax layer 56 soft label 60, 120, 220 student network 66 first loss function 70 second loss function 72 hard label 74 total loss 112 W _Hint layer 114 source hidden layer 122 W _Guided layer 124 destination hidden layer 126 regressors 200 training methods 212, 222, 522 output vectors 214, 224, 526 feature extractors 216, 226 fully connected layer network 218, 228, 528 convolution blocks 230, 232 speech data samples 250, 400 training device 260 teacher network training data storage unit 262 first language discrimination model storage unit 264 teacher network training unit 266 training data conversion unit 268 student network training data storage unit 270 Second language discrimination model storage unit 272 Student network training unit 280 Teacher network program storage unit 282 Teacher network parameter storage unit 290 Student network program storage unit 292 Student network parameter storage units 336, 338, 366 Training process

Claims (8)

using a first neural network for language discrimination of utterances for a set of a predetermined number of languages, trained on utterance data of a first utterance time, of a second utterance time shorter than the first utterance time; A method for training a language identification model for training a second neural network for language identification of utterances for said set of languages from utterance data, comprising:
each of the utterance data of the second utterance time is any part of the utterance data of the first utterance time;
The first neural network includes a first number of convolutional layer groups arranged to receive the speech data of the first speech time as an input and to propagate the output of each layer, and the first number of convolutional layers. a first classification network that receives the outputs of the layers and outputs language identification information;
The second neural network includes a second number of convolutional layer groups arranged to receive the speech data of the second speech time as an input and propagate the output of each layer, and the second number of convolutional layers. a second classification network for receiving the outputs of the layers and for outputting the language identification;
the layer with the first number of convolutional layers is a transfer source layer from which knowledge is transferred to the second neural network;
the layer with the second number of convolutional layers is a destination layer to which the knowledge is transferred;
The method includes:
preparing the first neural network in operational form;
utterance data of the first utterance time when training the first neural network; utterance data of the second utterance time included in the utterance data of the first utterance time; preparing training data in a machine-readable format including training data consisting of linguistic information indicating the uttered language of the utterance data of the utterance time of and associated with each other;
When the utterance data of the first utterance time of the training data is input to the first number of convolutional layers, the output of the transfer source layer is associated with the utterance data of the first utterance time. at least the output of the transfer destination layer when the utterance data of the second utterance time is input to the second number of convolutional layers, and the linguistic information of the utterance data of the second utterance time and training the second neural network using the language discrimination model. 2. The language discrimination model training method according to claim 1, wherein the number of neurons in said transfer source layer is the same as the number of neurons in said transfer destination layer. 3. The source layer is the topmost layer of the first number of convolutional layers and the destination layer is the topmost layer of the second number of convolutional layers. 3. A method of training a language discrimination model according to item 2. The step of training the second neural network comprises:
Output u _T (x _T ; Θ _T ) of the source layer when the utterance data x _T of the first utterance time of the training data is input to the first number of convolutional layers, where Θ _T represents the parameter set of the source layer;
Output u _S (x _S ; Θ _S of the destination layer when the speech data x _S of the second speech time related to the speech data x _T is input to the second number of convolutional layers ), where Θ _S represents the parameter set of the destination layer;
calculating the following loss function L _FRKD at the destination layer;

where λ is a weighting factor, L _hard (x _s , y) is the output of the second neural network when the speech data x _S is given to the second neural network, and the speech data x _S a loss function defined between the associated linguistic information y;
Using the output of the second neural network when the utterance data _xS is given and the linguistic information y associated with the utterance data _xS , the error backpropagation method is used to run the second neural network. and updating the parameters. using a first neural network for language discrimination of utterances for a set of a predetermined number of languages, trained on utterance data of a first utterance time, of a second utterance time shorter than the first utterance time; An apparatus for training a language identification model for training, from speech data, a second neural network for language identification of utterances for said set of languages, comprising:
each of the utterance data of the second utterance time is any part of the utterance data of the first utterance time;
The first neural network includes a first number of convolutional layer groups arranged to receive the speech data of the first speech time as an input and to propagate the output of each layer, and the first number of convolutional layers. a first classification network that receives the outputs of the layers and outputs language identification information;
The second neural network includes a second number of convolutional layer groups arranged to receive the speech data of the second speech time as an input and propagate the output of each layer, and the second number of convolutional layers. a second classification network for receiving the outputs of the layers and for outputting the language identification ;
the layer with the first number of convolutional layers is a transfer source layer from which knowledge is transferred to the second neural network;
the layer with the second number of convolutional layers is a destination layer to which the knowledge is transferred;
The device comprises:
a model storage device storing the first neural network in operable form;
utterance data of the first utterance time when training the first neural network; utterance data of the second utterance time included in the utterance data of the first utterance time; a training data storage device for storing, in a machine-readable format, training data including language information indicating an utterance language of the utterance data of the utterance time of and training data associated with each other;
When the utterance data of the first utterance time of the training data is input to the first number of convolutional layers, the output of the transfer source layer is associated with the utterance data of the first utterance time. at least the output of the transfer destination layer when the utterance data of the second utterance time is input to the second number of convolutional layers, and the linguistic information of the utterance data of the second utterance time training means for training said second neural network using said language discrimination model training apparatus. A computer program that causes a computer to perform the steps of the training method according to any one of claims 1-4 . A language identification model generated by causing a computer to execute each step of the training method according to any one of claims 1 to 4. A language identification model trained by the language identification model training device according to claim 5 .

Images