JP7274441B2 - LEARNING DEVICE, LEARNING METHOD AND LEARNING PROGRAM - Google Patents

Description

The present invention relates to a learning device, a learning method, and a learning program.

A conventional speech recognition model trains an acoustic model and a language model as separate systems. On the other hand, a technique for learning an end-to-end speech recognition model using a neural network is attracting attention (see Non-Patent Document 1). With this technology, the overall system that takes speech as an input and outputs information specifying a symbol string can be optimized, so speech recognition with higher accuracy than conventional speech recognition is possible.

Also, in model learning, it is generally expected that the more the number of training data is increased, the more the accuracy of the model obtained as a result of learning is improved. For example, in training a speech recognition model, if ideal training data consisting of a pair of clean speech data without noise and its transcript is used, the more the training data, the better the speech recognition performance. Improves accuracy.

J. Chorowski et al., “Attention-Based Models for Speech Recognition”, 2015, Advances in Neural Information Processing Systems 28 (NIPS 2015), pp. 577-585

However, even if training data containing noise or the like is used for learning, it may be difficult to improve the accuracy of speech recognition. For example, in actual speech recognition, much training data contains noise and the like, and it is difficult to prepare a large amount of clean speech data that does not contain noise. In addition, even if the number of training data containing noise, errors, etc. is increased for learning, the accuracy of speech recognition may rather deteriorate.

SUMMARY OF THE INVENTION It is an object of the present invention to improve the accuracy of speech recognition even when training data containing noise and the like is used for learning.

In order to solve the above-described problems and achieve the object, the learning apparatus according to the present invention uses a first neural network to convert the feature amount of an input speech signal for learning into an encoded intermediate feature amount. and a second neural network, a symbol string to be predicted from the intermediate feature amount and a posterior probability based on CTC (Connectionist Temporal Classification) of the symbol string. and a second calculation unit that calculates a predicted symbol string and the posterior probability of the symbol string from the correct symbol string and the intermediate feature value using a third neural network, and the posterior probability based on the CTC When the probability is greater than a predetermined threshold, the first neural network uses the loss function value calculated from the posterior probability calculated by the second calculation unit and the posterior probability based on the CTC, the and an updating unit that updates parameters of the second neural network and the third neural network.

According to the present invention, it is possible to improve the accuracy of speech recognition even if training data containing noise or the like is used for learning.

FIG. 1 is a schematic diagram illustrating the schematic configuration of the learning device of this embodiment. FIG. 2 is a schematic diagram illustrating a schematic configuration of a learning device according to another embodiment. FIG. 3 is a flow chart showing the learning processing procedure. FIG. 4 is a diagram showing an example of a computer that executes a learning program.

An embodiment of the present invention will be described in detail below with reference to the drawings. It should be noted that the present invention is not limited by this embodiment. Moreover, in the description of the drawings, the same parts are denoted by the same reference numerals.

[Configuration of learning device]
FIG. 1 is a schematic diagram illustrating the schematic configuration of the learning device of this embodiment. As illustrated in FIG. 1, a learning device 10 of this embodiment is implemented by a general-purpose computer such as a personal computer, and includes a storage unit 11 and a control unit 12 .

The storage unit 11 is realized by a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. In the storage unit 11, a processing program for operating the learning device 10, data used during execution of the processing program, and the like are stored in advance, or are temporarily stored each time processing is performed.

In this embodiment, the storage unit 11 stores parameters 11a of an end-to-end neural network, which will be described later. These parameters 11a are updated by learning processing, which will be described later.

The control unit 12 is implemented using a CPU (Central Processing Unit) or the like, and executes a processing program stored in a memory. Accordingly, as illustrated in FIG. 1, the control unit 12 includes a data selection unit 12a, an encoder 12b, a first decoder 12c, a second decoder 12d, a data cleansing unit 12e, an update unit 12f, and an end It functions as the determination unit 12g. Note that these functional units may be implemented in different hardware, respectively or partially. Also, the control unit 12 may include other functional units.

The data selector 12a accepts an input of an audio signal for learning. Specifically, the data selection unit 12a selects an audio signal to be used in a learning process, which will be described later, from a set of input training data, and inputs it to the encoder 12b, which will be described later. It should be noted that the processing of the updating unit 12f, which will be described later, may be executed when all the speech signals in the training data are input to the encoder 12b.

The encoder 12b is an example of a conversion unit, and uses a first neural network to convert the feature quantity of the input learning speech signal into an encoded intermediate feature quantity. The encoder 12b is, for example, a Transformer encoder, and converts the logarithmic mel filter bank feature amount X ^fbank , which is the feature amount of the speech signal for each unit time, into a feature amount obtained by contracting the length, etc., using a preprocessing neural network. Accepts X ^sub as input. Then, the encoder 12b converts the feature amount X ^sub into an intermediate feature amount by the first neural network and outputs the intermediate feature amount.

Here, if the total number of layers of the first neural network constituting the encoder 12b is e, the input X _i of the i-th layer (i=0, 1, . . . , e−1), and the output X _i+1 is expressed as , as shown in the following equation (1), each layer i converts the input feature quantity X _i into the intermediate feature quantity X _{i+1 and} outputs it. Also, the e−1-th layer, which is the final layer, outputs the speech feature quantity X _e as an intermediate feature quantity.

Here, PE is a neural network ^that receives ^frame numbers 1, 2, . MHA is a neural network that receives three feature quantity sequences as inputs and outputs a feature quantity sequence having the same dimension and length as the first feature quantity sequence. FF is a neural network that outputs a feature value sequence of the same dimension as the input feature value for each time, which consists of two fully connected layers and a ReLU (Rectified Linear Units) activation layer.

In addition to the above equation (1), the first neural network constituting the encoder 12b is, for example, a two-layer CNN (Convolution Neural Network) and a ReLU activation layer as a preprocessing neural network. may be configured. In that case, if the output length of the CNN is n ^sub and the number of channels is d ^att , each intermediate feature X _i becomes a vector of n ^sub ×d ^att dimensions.

Further, the encoder 12b is not limited to the Transformer encoder, and may be, for example, an RNN (Recurrent Neural Networks) encoder.

The first decoder 12c is an example of a first calculator, and uses a second neural network to obtain a predicted symbol string and a CTC-based posterior probability of the symbol string from the intermediate feature _Xe . calculate. Here, the predicted symbol string is a new symbol string that includes symbols following the correct symbol string given as training data. The first decoder 12c is, for example, a CTC decoder, and uses a first neural network to determine any alignment, which is a symbol string in which symbols corresponding to the time (frame) of the intermediate feature _Xe are arranged. Calculate the posterior probability for

Specifically, the first decoder 12c uses X _e that is the output of the encoder 12b to obtain the CTC-based posterior probability p _ctc (Y|X _e ) as shown in the following equation (2): is calculated and output.

Here, the weight matrix W ^ctc and the bias vector b ^ctc are parameters of the second neural network and are learned in advance.

And the CTC-based posterior probability p _ctc (Y|X _e ) is the posterior probability for any alignment between X _e and Y. Alignment is a sequence in which symbol strings Y corresponding to time t of each input sequence data are arranged. For example, aabcc, abbbc, aaabc, .

C is the output of the first decoder 12c and C[t,π[t]] is the alignment between the output symbol π[t] and the tth frame of _Xe .

Also, the many-to-one mapping function B(π) is a function that removes redundant symbols from the alignment π. For example, if φ is a blank symbol, B(aaφb)=ab. Also, the one-to-many mapping function B ⁻¹ takes the symbol string as input and outputs a set of all the above alignments.

In the second formula of the above formula (2), the posterior probability of each alignment π when X _e is observed is defined as “probability C[t, π[t]] of arranging symbol π[t] at time t. It is calculated as the product of time.

In addition, in the third formula of the above formula (2), the posterior probability of the symbol string Y when X _e is observed is expressed as "the posterior probability of the second formula above for all alignments in which Y appears. It is calculated as the sum total.

The second decoder 12d is an example of a second calculator, and uses a third neural network to predict a symbol string and a posterior to the symbol string from the correct symbol string and the intermediate feature quantity _Xe . Calculate the probability and

For example, the second decoder 12d is a decoder in Transformer. The second decoder 12d converts the speech feature quantity _Xe obtained by the conversion by the encoder 12b and the already predicted symbol string Y[1:u]=Y[1], . . . , Y[u] is input, and the subsequent symbol string Y[2:u+1] is predicted and output as shown in the following equation (3).

Here, Embed is a function that expresses computation by a neural network similar to PE, and outputs a d ^att -dimensional feature amount by inputting a series of symbols Y[1:u] instead of the time (frame) in PE. do.

The total number of layers of the third neural network constituting the second decoder 12d is expressed as d, the input Z _j and the output Z _j+1 of the j-th layer (j=0, 1, . . . , d−1). do. In this case, the second decoder 12d is given Y[1:u] and X _e as shown in the following equation (4), and the posterior probability based on the Transformer, that is, the next symbol is Y The posterior probability p _s2s (Y|X _e ) of [u+1] is calculated and output.

Here, the weight matrix W ^att and the bias vector b ^att are the parameters of the third neural network and are learned in advance.

The learning device 10 learns by regarding the first neural network, the second neural network, and the third neural network as one end-to-end neural network as a whole.

Further, the second decoder 12d is not limited to a Transformer decoder, and may be, for example, an RNN decoder.

Based on the posterior probability calculated by the first decoder 12c, the data cleansing unit 12e selects data to be used for processing by the updating unit 12f, which will be described later. Specifically, when the CTC-based posterior probability is greater than a predetermined threshold, the data cleansing unit 12e causes the updating unit 12f, which will be described later, to perform processing.

For example, the data cleansing unit 12e stores, as an index set I, in the storage unit 11, indexes of data whose CTC-based posterior probabilities are greater than a predetermined threshold.

Note that when the posterior probability based on CTC is equal to or less than a predetermined threshold, the data cleansing unit 12e causes the data selection unit 12a to select another audio signal.

When the CTC-based posterior probability is greater than a predetermined threshold, the updating unit 12f uses the loss function value calculated from the posterior probability calculated by the second decoder 12d and the CTC-based posterior probability to update the first , the second neural network and the third neural network parameters 11a are updated.

Specifically, the updating unit 12f calculates the loss related to the output of the first decoder 12c and the loss related to the output of the second decoder 21d for the audio signal selected by the data cleansing unit 12e, and sums them update the parameters 11a of the first neural network, the second neural network, and the third neural network based on .

Here, the loss related to the output of the first decoder 12c is calculated from the output of each decoder corresponding to the input data of the index included in the index set I shown in the following equation (5). 6) is the CTC loss shown in FIG.

Also, the loss related to the output of the second decoder 12d is calculated from the output of each decoder corresponding to the input data of the index included in the index set I of the above equation (5), as shown in the following equation (7) is the cross-entropy loss shown in .

The update unit 12f uses the weighted sum of the losses of the above equations (6) and (7) as a loss function value, and uses a well-known technique such as error backpropagation learning to obtain the parameter values of the end-to-end neural network. is calculated, and the parameter 11a stored in the storage unit 11 is updated.

In this way, the learning device 10 can perform learning while performing data cleansing to exclude data that should not be used as training data because the CTC-based posterior probability is equal to or less than a predetermined threshold value during learning. It becomes possible.

Note that after the parameter 11a is updated, the learning device 10 receives the input of the speech signal for learning again and predicts the symbol string using the end-to-end neural network.

The termination determination unit 12g terminates updating of the parameter 11a when a predetermined termination condition is satisfied. For example, when the loss function value becomes equal to or less than a predetermined threshold, when the number of times the parameter 11a is updated reaches a predetermined number, or when the amount of update of the parameter 11a becomes equal to or less than a predetermined threshold In at least one of the cases, the update of the parameter 11a is terminated.

Note that in the learning device 10 shown in FIG. 1, the processes of the first decoder 12c and the second decoder 12d are executed in parallel. Here, FIG. 2 is a schematic diagram illustrating a schematic configuration of the learning device 10 of another embodiment. As shown in FIG. 2, the learning device 10 may input only the data selected by the data cleansing unit 12e to the second decoder 12d. In this way, the data cleansing unit 12e may cause the processing of the second decoder 12d described above to be executed only when the posterior probability based on the CTC is greater than a predetermined threshold. In this case, the processing of the second decoder 12d is reduced.

[Learning process]
Next, learning processing by the learning device 10 according to the present embodiment will be described with reference to FIG. FIG. 3 is a flow chart showing the learning processing procedure. The flowchart in FIG. 3 is started, for example, at the timing when the user performs an operation input instructing the start.

First, the encoder 12b receives an audio signal for learning input from the data selector 12a (step S1). Then, the encoder 12b uses the first neural network to convert the feature quantity of the received speech signal into an encoded intermediate feature quantity (step S2).

Also, the first decoder 12c uses the second neural network to calculate the predicted symbol string and the CTC-based posterior probability of the symbol string from the intermediate feature amount (step S3). Also, the second decoder 12d uses a third neural network to calculate a predicted symbol string and the posterior probability of the symbol string from the correct symbol string and the intermediate feature quantity (step S4).

Next, the data cleansing unit 12e checks whether the CTC-based posterior probability is greater than a predetermined threshold, and if it is greater than the predetermined threshold (step S5, Yes), the process proceeds to step S6. On the other hand, when the CTC-based posterior probability is equal to or less than the predetermined threshold (step S5, No), the data cleansing unit 12e returns the process to step S1.

The updating unit 12f updates the parameter 11a of the end-to-end neural network using the loss function value calculated from the posterior probability calculated by the second decoder 12d and the posterior probability based on the CTC (step S6 ).

Then, the termination determination unit 12g confirms whether or not a predetermined termination condition is satisfied (step S7). For example, when the loss function value becomes equal to or less than a predetermined threshold, when the number of times the parameter 11a is updated reaches a predetermined number, or when the amount of update of the parameter 11a becomes equal to or less than a predetermined threshold In at least one of the cases, it is determined that the termination condition is satisfied.

When the termination determination unit 12g determines that the predetermined termination condition is not satisfied (step S7, No), the process returns to step S1 to repeat prediction of the symbol string and update of the parameter 11a. On the other hand, when the termination determination unit 22g determines that the predetermined termination condition is satisfied (step S7, Yes), the series of learning processes is terminated.

As described above, in the learning apparatus 10 of the present embodiment, the encoder 12b uses the first neural network to encode the feature amount of the input speech signal for learning to an intermediate feature amount. Convert to Also, the first decoder 12c uses the second neural network to calculate the predicted symbol string and the CTC-based posterior probability of the symbol string from the intermediate feature amount. Also, the second decoder 12d uses a third neural network to calculate a predicted symbol string and the posterior probability of the symbol string from the correct symbol string and the intermediate feature quantity. Further, when the CTC-based posterior probability is greater than a predetermined threshold, the updating unit 12f uses the loss function value calculated from the posterior probability calculated by the second decoder 12d and the CTC-based posterior probability, Update the parameters of the first neural network, the second neural network and the third neural network.

In this way, the learning device 10 performs data cleansing during learning to exclude data whose posterior probability based on CTC is equal to or less than a predetermined threshold and which may reduce the accuracy of speech recognition when used for learning. be able to. As a result, even if training data containing noise, errors, etc. is used for learning, the accuracy of speech recognition can be improved.

Also, the learning device 10 may perform the processing of the second decoder 12d when the posterior probability based on the CTC is greater than a predetermined threshold. This reduces the processing of the second decoder 12d.

In addition, learning device 10 learns by regarding the first neural network, the second neural network, and the third neural network as one end-to-end neural network as a whole. This optimizes the speech recognition process and enables more accurate speech recognition.

Further, the learning device 10 determines that the loss function value is equal to or less than a predetermined threshold value, the number of updates of the parameter 11a reaches a predetermined number of times, or the update amount of the parameter 11a reaches a predetermined threshold value. In at least one of the cases below, the update of the parameter 11a is terminated. This makes it possible to suppress the processing load of the learning process.

[program]
It is also possible to create a program in which the processing executed by the learning device 10 according to the above embodiment is described in a computer-executable language. As one embodiment, the learning device 10 can be implemented by installing a learning program that executes the above-described speech recognition processing as package software or online software on a desired computer. For example, the information processing device can function as the learning device 10 by causing the information processing device to execute the learning program.

The information processing apparatus referred to here includes a desktop or notebook personal computer. In addition, information processing devices include smart phones, mobile communication terminals such as mobile phones and PHSs (Personal Handyphone Systems), and slate terminals such as PDAs (Personal Digital Assistants). Also, the functions of the learning device 10 may be implemented in a cloud server.

FIG. 4 is a diagram showing an example of a computer that executes a learning program. Computer 1000 includes, for example, memory 1010 , CPU 1020 , hard disk drive interface 1030 , disk drive interface 1040 , serial port interface 1050 , video adapter 1060 and network interface 1070 . These units are connected by a bus 1080 .

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012 . The ROM 1011 stores a boot program such as BIOS (Basic Input Output System). Hard disk drive interface 1030 is connected to hard disk drive 1031 . Disk drive interface 1040 is connected to disk drive 1041 . A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1041, for example. A mouse 1051 and a keyboard 1052 are connected to the serial port interface 1050, for example. For example, a display 1061 is connected to the video adapter 1060 .

Here, the hard disk drive 1031 stores an OS 1091, application programs 1092, program modules 1093 and program data 1094, for example. Each piece of information described in the above embodiment is stored in the hard disk drive 1031 or the memory 1010, for example.

The learning program is also stored in hard disk drive 1031 as, for example, program modules 1093 that describe instructions to be executed by computer 1000 . Specifically, the hard disk drive 1031 stores a program module 1093 that describes each process executed by the learning device 10 described in the above embodiment.

Data used for information processing by the learning program is stored as program data 1094 in the hard disk drive 1031, for example. Then, the CPU 1020 reads out the program module 1093 and the program data 1094 stored in the hard disk drive 1031 to the RAM 1012 as necessary, and executes each procedure described above.

Note that the program module 1093 and program data 1094 related to the learning program are not limited to being stored in the hard disk drive 1031. For example, they may be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1041 or the like. may be Alternatively, program module 1093 and program data 1094 related to the learning program are stored in another computer connected via a network such as LAN or WAN (Wide Area Network), and are read out by CPU 1020 via network interface 1070. may

Although the embodiments to which the invention made by the present inventor is applied have been described above, the present invention is not limited by the descriptions and drawings forming a part of the disclosure of the present invention according to the embodiments. That is, other embodiments, examples, operation techniques, etc. made by those skilled in the art based on this embodiment are all included in the scope of the present invention.

REFERENCE SIGNS LIST 10 learning device 11 storage unit 11a parameter 12 control unit 12a data selection unit 12b encoder 12c first decoder (CTC decoder)
12d second decoder 12e data cleansing unit 12f update unit 12g end determination unit

Claims (6)

a conversion unit that converts the feature quantity of the input speech signal for learning into a coded intermediate feature quantity using the first neural network;
a first calculation unit that calculates a predicted symbol string and a posterior probability based on CTC (Connectionist Temporal Classification) of the symbol string from the intermediate feature amount using a second neural network;
a second calculation unit that calculates a predicted symbol string and the posterior probability of the symbol string from the correct symbol string and the intermediate feature using a third neural network;
When the posterior probability based on the CTC is greater than a predetermined threshold, using the loss function value calculated from the posterior probability calculated by the second calculation unit and the posterior probability based on the CTC, an updating unit that updates the parameters of the neural network of, the second neural network and the third neural network;
A learning device characterized by comprising: 2. The learning apparatus according to claim 1, wherein the process of the second calculator is performed when the posterior probability based on the CTC is greater than a predetermined threshold. 2. The method according to claim 1, wherein learning is performed by considering the first neural network, the second neural network and the third neural network as one end-to-end neural network as a whole. learning device. When the loss function value becomes equal to or less than a predetermined threshold, when the number of times the parameter is updated reaches a predetermined number, or when the amount of update of the parameter becomes equal to or less than a predetermined threshold. 2. The learning apparatus according to claim 1, further comprising an end determination unit for terminating update of said parameters. A learning method executed by a learning device, comprising:
a conversion step of converting the feature quantity of the input learning speech signal into an encoded intermediate feature quantity using the first neural network;
A first calculation step of calculating a predicted symbol string and a posterior probability based on CTC (Connectionist Temporal Classification) of the symbol string from the intermediate feature amount using a second neural network;
a second calculation step of calculating a predicted symbol string and the posterior probability of the symbol string from the correct symbol string and the intermediate feature amount using a third neural network;
When the posterior probability based on the CTC is greater than a predetermined threshold, using the loss function value calculated from the posterior probability calculated by the second calculation step and the posterior probability based on the CTC, an updating step of updating parameters of the neural network of, the second neural network and the third neural network;
A learning method comprising: a transformation step of transforming the feature quantity of the input speech signal for learning into a coded intermediate feature quantity using the first neural network;
a first calculation step of calculating a predicted symbol string and a posterior probability based on CTC (Connectionist Temporal Classification) of the symbol string from the intermediate feature amount using a second neural network;
a second calculation step of calculating a predicted symbol string and the posterior probability of the symbol string from the correct symbol string and the intermediate feature using a third neural network;
When the CTC-based posterior probability is greater than a predetermined threshold, the first an updating step of updating parameters of the neural network of, the second neural network and the third neural network;
A learning program for making a computer execute

Images