JP7359028B2 - Learning devices, learning methods, and learning programs - Google Patents

Description

The present invention relates to a learning device, a speech recognition device, a learning method, and a learning program.

BACKGROUND ART Conventionally, a technique is known in which a model using a neural network (hereinafter referred to as NN as appropriate) is learned by machine learning. For example, a method is known in which a speech recognition model using an end-to-end NN for converting speech data into information (posterior probability) represented by the speech data is learned by machine learning.

This end-to-end NN consists of, for example, an encoder that outputs intermediate features of speech, an attention that determines which part of the intermediate features output from the encoder to focus on (weight), and an attention and a decoder that estimates the character indicated by the voice using the determined weight (see Non-Patent Document 1).

When learning the above-mentioned speech recognition models, clean speech that does not contain noise etc. is often used as training data. However, in real environments, there are many situations where noise etc. are present, and it is necessary to perform speech recognition in situations where noise is present.

Therefore, for example, the technology described in the above-mentioned non-patent document 1 uses both clean speech and noise-containing speech as training data to train a speech recognition model in order to increase robustness against noise. . In this case, first, pair data of audio signals with the same content (text) that includes noise (noisy audio) and one that does not include noise (clean audio) is prepared. Next, the output of the encoder and the output of the decoder's intermediate layer when noisy audio is input become closer to the output of the encoder and the output of the decoder's intermediate layer when the corresponding clean audio is input, respectively. Then, learn the model parameters.

Davis Liang, Zhiheng Huang, and Zachary C Lipton, “Learning Noise-Invariant Representations for Robust Speech Recognition,” in Proceedings of the IEEE Workshop on Spoken Language Technology Workshop (SLT), pp.56-63, 2018.

However, even with models learned using the above techniques, there is a problem in that the recognition accuracy of speech containing noise is not necessarily high. SUMMARY OF THE INVENTION Therefore, it is an object of the present invention to solve the above-mentioned problems and provide a speech recognition means that is highly robust against noise.

In order to solve the above-mentioned problems, the present invention uses, as first teacher data, data in which audio data is associated with correct data of information specifying a symbol string indicated by the audio data, and When converting a symbol string indicated by audio data into identifying information, an encoder that outputs an intermediate feature of the audio data and which element to focus on among the elements constituting the intermediate feature are provided. a first learning unit that performs learning of a speech recognition model, which includes a warning mechanism that outputs a weight indicated by the weight and a value obtained by calculating a weighted sum of the intermediate feature amounts using the weight; Speech recognition after learning by the first learning unit based on the second teacher data in which the noise-containing speech data that is the added speech data is associated with the correct answer data of information specifying the symbol string indicated by the speech data. A first distance that indicates how much of a difference there is in the distribution of weights output from the attention mechanism of the voice recognition model when voice data is input to the model and when the noisy voice data is input to the model. and a second distance indicating how much difference there is between the information output from the decoder of the speech recognition model and the correct data for the speech data when the noisy speech data is input. When learning the speech recognition model after learning by the first learning unit using a distance calculation unit that calculates the distance and the second training data, the difference between the first distance and the second distance is The present invention is characterized by comprising a second learning unit that takes the sum as a loss and updates parameters of an encoder and an attention mechanism of the speech recognition model so that the loss becomes small.

According to the present invention, it is possible to provide a speech recognition means that is highly robust against noise.

FIG. 1 is a diagram showing an example of a speech recognition model using an end-to-end NN. FIG. 2 is a diagram for explaining an overview of the operation of the learning device. FIG. 3 is a diagram for explaining an overview of the operation of the learning device. FIG. 4 is a diagram for explaining an overview of the operation of the learning device. FIG. 5 is a diagram for explaining an overview of the operation of the learning device. FIG. 6 is a diagram showing an example of the configuration of the learning device. FIG. 7 is a flowchart showing an example of the processing procedure of the learning device. FIG. 8 is a diagram showing an example of the configuration of a speech recognition device. FIG. 9 is a flowchart showing an example of the processing procedure of the speech recognition device. FIG. 10 is a diagram showing an example of a computer that executes a learning program.

EMBODIMENT OF THE INVENTION Hereinafter, with reference to drawings, the form (embodiment) for implementing this invention is divided into 1st Embodiment and 2nd Embodiment, and is described. The present invention is not limited to each embodiment. In addition, the case where the model to be learned by the learning device in each embodiment is a speech recognition model using an end-to-end NN including an encoder, attention, and decoder will be described as an example.

[First embodiment]
First, an overview of the operation of the learning device according to the first embodiment will be explained using FIGS. 1 to 5. The learning device includes a speech recognition model (speech recognition unit) using the above-mentioned end-to-end NN. As shown in FIG. 1, this speech recognition model includes an encoder (encoding section), attention (attention mechanism section), and decoder (decoding section).

The encoder converts the input audio data (x ₁ ,...,x _T ) into intermediate feature amounts (h ₂ ,..., h _T ). Attention calculates weight values (α ₀ ,…,α _T ) indicating which element to focus on among the elements constituting the intermediate feature based on the intermediate feature and the hidden state of the decoder. A value (c ₀ ,...,c _L ) calculated by calculating a weighted sum of intermediate features using the weight is output. The decoder estimates information (y _i ) that specifies the next character of the decoder output (y _i-1 ) based on the previous decoder output (y _i-1 ) and the output value c _i from the attention. and output it.

The learning device first pre-trains the above-mentioned speech recognition model using clean speech data (speech data that does not include noise) (FIG. 2(1)). For example, the learning device performs pre-learning of a speech recognition model using, as teacher data, data in which clean speech data is associated with information (correct data) specifying a symbol string indicated by the speech data. As a result, the encoder, attention, and decoder of the speech recognition model are set with parameters for accurately performing speech recognition on clean speech data.

Next, the learning device determines the attention weight for clean voice data based on pair data of clean voice data and voice data with noise (for example, voice data with noise added to the clean voice data). , and the attention weight for the noisy audio data (FIG. 3(2)).

Here, we define the inter-distribution loss of attention weights between clean audio data and noisy audio data ((3) in FIG. 4). For example, the inter-distribution loss L _att between the attention weight (α _i ) of clean audio data and the attention weight (α′ _i ) of noisy audio data is defined by KL divergence. For example, the inter-distribution loss L _att between the attention weight of clean audio data (α _i ) and the attention weight of noisy audio data (α _i ) defined by KL divergence is D _KL (α _i | |α′ _i ).

Next, the learning device learns the attention and the encoder (performs second learning) so that the sum of the character discrimination loss L _char and the attention loss L _att is less than or equal to a predetermined threshold (Fig. 5 (4) ). Note that the above character identification loss L _char indicates the loss between the information (y _i ′) output by the decoder when noisy audio data is input and the correct data.

By the learning device learning the speech recognition model as described above, it is possible to bring the attention weight of noisy speech data closer to the attention weight of clean speech data in the speech recognition model. As a result, the learning device can learn a speech recognition model that is highly robust against noise.

[composition]
Next, a configuration example of the learning device 10 of the first embodiment will be described using FIG. 6. The learning device 10 includes an input section 11, an output section 12, a storage section 13, and a control section 14.

The input unit 11 receives input of data used when the control unit 14 performs various processes. For example, the input unit 11 receives input of teacher data (first teacher data and second teacher data) of a speech recognition model (speech recognition unit 143). The output unit 12 outputs the results of the processing performed by the control unit 14. For example, the output unit 12 outputs the voice recognition result etc. by the voice recognition unit 143.

The storage unit 13 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk, and stores a program that operates the learning device 10 and the execution of the program. The data used inside is stored. For example, the storage unit 13 stores first teacher data and second teacher data. Furthermore, the storage unit 13 stores values of parameters set in the speech recognition unit 143, and the like.

The first teacher data is data in which clean voice data is associated with information (correct data) that specifies a symbol string indicated by the voice data. This first teacher data is used when the first learning section 1471 performs preliminary learning of the speech recognition section 143.

The second teacher data is data in which clean voice data, voice data with noise added to the voice data, and information (correct data) for specifying a symbol string indicated by the voice data are associated.

Note that the clean voice data in the second teacher data may be the same as the clean voice data included in the first teacher data. Further, the noisy audio data may be clean audio data with artificially added noise, or may be audio data recorded in an environment where noise such as noise is generated. This second teacher data is used when the second learning section 1474 performs the second learning of the speech recognition section 143.

The control unit 14 controls the entire learning device 10. The control unit 14 performs, for example, learning of the voice recognition unit 143 and voice recognition using the voice recognition unit 143 after learning.

The control section 14 includes a first data input section 141 , a second data input section 142 , a speech recognition section 143 , and a learning section 147 .

In the case of pre-learning mode, the first data input unit 141 selects voice data that has not yet been selected from the first teacher data, inputs it to the voice recognition unit 143, and causes the voice recognition unit 143 to perform arithmetic processing. let

In the case of the second learning mode, the second data input unit 142 selects a pair of clean audio data and noisy audio data that have not yet been selected from the second teacher data. Then, the second data input unit 142 inputs, for example, the selected pair of clean voice data to the voice recognition unit 143, and causes the voice recognition unit 143 to perform arithmetic processing. The second data input unit 142 also inputs the noisy voice data that is a pair of the clean voice data to the voice recognition unit 143 set with the same parameters, and causes the voice recognition unit 143 to perform calculations. Execute the process.

The speech recognition unit 143 performs speech recognition of input speech data based on a speech recognition model. Specifically, the voice recognition unit 143 converts input voice data into information that specifies a symbol string indicated by the voice data, and outputs the converted information. The speech recognition section 143 includes an encoding section (encoder) 144, an attention mechanism section (attention) 145, and a decoding section (decoder) 146. Parameters used when performing speech recognition are set in each of the encoding section 144, the attention mechanism section 145, and the decoding section 146. Note that the above parameters are updated as appropriate when the speech recognition unit 143 is learning.

The encoding unit 144 converts the input audio data into intermediate feature amounts. The attention mechanism unit 145 inputs the intermediate features and the hidden state S _n of the decoding unit 146, and calculates weights (α ₀ ,...,α _T ) and a value c that is a weighted sum of the intermediate features using the weights. Output _n . The decoding unit 146 receives the previous output y _1:n-1 of the decoder and the output value (c _n ) from the attention mechanism unit 145 as input, and estimates information y _n that specifies the next character. and output it.

The learning section 147 performs learning of the speech recognition section 143. The learning section 147 includes a first learning section 1471, a first distance calculating section 1472, a second distance calculating section 1473 (distance calculating section), and a second learning section 1474. The third distance calculation unit 1475 indicated by a broken line may or may not be equipped, and the case where it is equipped will be described in the second embodiment.

The first learning unit 1471 performs preliminary learning of the speech recognition unit 143 when the learning mode is the preliminary learning mode. For example, the first learning unit 1471 uses the output y _n of the voice recognition unit 143 for the voice data of the first teacher data and information (correct answer) that specifies the symbol string indicated by the voice data until a predetermined condition is met. The parameters of the encoding section 144, the attention mechanism section 145, and the decoding section 146 of the speech recognition section 143 are updated based on the loss (distance) between the speech recognition section 143 and the speech recognition section 143. Further, the parameter values of each part of the speech recognition unit 143 after the above-mentioned pre-learning are stored in the storage unit 13.

The above-mentioned predetermined conditions used by the first learning section 1471 include, for example, that the update of each parameter of the speech recognition section 143 based on the above-mentioned loss has reached a predetermined number of repetitions, and that the loss is below a predetermined threshold. The update amount of the parameter has become less than a predetermined threshold.

Note that if the learning unit 147 determines that the processing by the first learning unit 1471 satisfies the above-described predetermined conditions, it ends the pre-learning mode and switches to the second learning mode.

In the second learning mode, the first distance calculation unit 1472 calculates whether the speech recognition unit 143 after pre-learning has input clean speech data or input noise-containing speech data. A first distance indicating the degree of difference in the distribution of weights output from the attention mechanism unit 145 of the recognition unit 143 is calculated.

For example, the first distance calculation unit 1472 calculates the attention weight output by the attention mechanism unit 145 after pre-learning when clean voice data in the second teacher data is input, and the clean voice data. An inter-distribution distance (first distance) from the attention weight outputted by the attention mechanism unit 145 when paired noise-containing audio data is input is calculated using KL divergence.

In addition, when noisy speech data is input to the speech recognition section 143 after pre-learning, the second distance calculation section 1473 calculates the information output from the decoder of the speech recognition section 143 and the A second distance indicating how much difference there is between the audio data and the correct data (for example, correct characters) in the second teacher data is calculated.

For example, the second distance calculation unit 1473 calculates the distance (second distance) between the information output by the decoding unit 146 and the correct character when noisy voice data in the second teacher data is input. ).

The second learning unit 1474 performs second learning on the speech recognition unit 143 when the learning mode is the second learning mode. For example, the second learning unit 1474 adjusts the first distance calculated by the first distance calculation unit 1472 and the second distance calculated by the second distance calculation unit 1473 until a predetermined condition is satisfied. The parameters of the attention mechanism section 145 and the decoding section 146 of the speech recognition section 143 after pre-learning are updated based on the sum of .

For example, when the second learning unit 1474 uses the second teacher data to update the parameters of the attention mechanism unit 145 and the decoding unit 146 of the speech recognition unit 143 after pre-learning, the second learning unit 1474 uses the first training data described above. The sum of the distance and the second distance is defined as a loss, and the parameters are updated so that the loss becomes smaller.

Note that the predetermined conditions used by the second learning section 1474 are the same conditions as in the case of the first learning section 1471. For example, the update of each parameter of the speech recognition unit 143 has reached a predetermined number of repetitions, the loss has become less than a predetermined threshold, the amount of parameter update has become less than a predetermined threshold, etc. Further, the parameter values of each part of the speech recognition unit 143 after the second learning described above are stored in the storage unit 13.

The learning device 10 may perform the second learning on the voice recognition unit 143 as described above, and then use the learned voice recognition unit 143 to perform voice recognition processing on the input voice data. . For example, the learning device 10 may use the learned speech recognition unit 143 to convert input speech data into information that specifies a symbol string shown in the speech data, and output the information.

[Processing procedure]
The processing procedure of the learning device 10 will be explained using FIG. 7. First, the first data input unit 141 of the learning device 10 inputs the voice data of the first teacher data to the voice recognition unit 143, and the first learning unit 1471 uses the result output from the voice recognition unit 143. Then, preliminary learning of the speech recognition unit 143 is performed (S1). After that, the second data input unit 142 of the learning device 10 inputs the voice data of the second teacher data to the voice recognition unit 143 learned in S1, and the second learning unit 1474 inputs the voice data of the second teacher data A second learning of the speech recognition section 143 is performed using the results output by each section of the speech recognition section 143 (S2).

According to such a learning device 10, the first learning unit 1471 uses the first teacher data to instruct the voice recognition unit 143 to output correct answer data for clean voice data. Set the parameters for each part.

Thereafter, the second learning section 1474 uses the second teacher data to determine whether the above-mentioned The parameters of the warning mechanism section 145 and the encoding section 144 of the speech recognition section 143 are updated so that the sum of the first distance and the second distance becomes smaller.

As a result, the learning device 10 allows the speech recognition unit 143 to output correct data for noise-containing speech, while changing the weight output by the attention mechanism unit 145 for noise-containing speech to clean data. It is possible to approximate the weight output by the attention mechanism unit 145 for the voice. As a result, the learning device 10 can perform learning of the speech recognition unit 143 with high robustness against noise.

[Second embodiment]
The learning device 10 may include a third distance calculation section 1475 (see FIG. 6). An embodiment in this case will be described as a second embodiment. Hereinafter, the same components as those in the first embodiment will be denoted by the same reference numerals, and the description thereof will be omitted. The third distance calculation unit 1475 calculates the difference between the pre-trained voice recognition unit 143 when clean voice data is input and when the voice data containing noise is input. A third distance indicating how much difference there is in the information (intermediate feature amount) output from the encoding unit 144 is calculated.

When the second learning unit 1474 trains the speech recognition unit 143 after the pre-learning, the second learning unit 1474 sets the sum of the first distance, the second distance, and the third distance as a loss, and the loss is The parameters of the encoding section 144 and the attention mechanism section 145 of the speech recognition section 143 are updated so that the size becomes smaller. By doing so, the learning device 10 can perform learning of the speech recognition unit 143 that is more robust against noise.

[Other embodiments]
Note that the speech recognition unit 143 (speech recognition model) trained by the learning device 10 described above may be used for speech recognition by the learning device 10, or may be used for speech recognition by another device.

For example, the speech recognition unit 143 learned by the learning device 10 may be installed in the speech recognition device 100 (see FIG. 8). The speech recognition device 100 includes, for example, an input section 11, an output section 12, a storage section 13, and a control section 14, as shown in FIG. The control unit 14 includes a speech recognition unit 143 trained by the learning device 10. When the input section 11 of the speech recognition device 100 receives input of speech data to be speech recognized (S11 in FIG. 9), speech recognition is performed using the speech recognition section 143 trained by the learning device 10 (S12). The recognition result is output (S13).

[Experimental result]
Note that the results of an experiment using the learning device 10 of the second embodiment described above will be explained below. Note that in this experiment, the data used by the first learning unit 1471 of the learning device 10 for preliminary learning of the speech recognition model is WSJ1si284. This WSJ1si284 is a corpus of audio data read out from the Wall street journal. This corpus has a total number of utterances of 37416 and a length of 80 hours. Furthermore, the data used by the second learning unit 1474 for the second learning of the speech recognition model is WSJ1si84 and tr05_simu of CHiME-4. WSJ1si84 is also a corpus of audio data read out from Wall street journals. This WSJ1si84 has a total number of utterances of 7138 and a length of 15 hours. CHiME-4's tr05_simu is a corpus with noise superimposed on WSJ1si84.

Under the above conditions, the learning device 10 of the second embodiment uses the speech recognition unit 143 after performing the preliminary learning and the second learning to obtain the audio data of CHiME-4 et05_simu and CHiME-4 et05_real. When we measured the false recognition rate, we obtained the following results. Furthermore, CHiME-4 et05_simu is a corpus in which noise is superimposed on the audio data of the Wall street journal. CHiME-4 et05_real is a corpus of audio data read out from Wall street journals in a noisy environment.

For example, as shown below, when speech recognition was performed using the speech recognition model (comparative example) learned by the method described in Non-Patent Document 1, the misrecognition rate of the speech data of CHiME-4 et05_simu was 28.5%. The misrecognition rate of CHiME-4 et05_real voice data was 32.8%. On the other hand, when speech recognition was performed using the speech recognition model learned by the learning device 10 of the second embodiment, the misrecognition rate of the speech data of CHiME-4 et05_simu was 27.8%, and the misrecognition rate of the speech data of CHiME-4 et05_real was 27.8%. The data misrecognition rate was 32.5%.

From the above, the speech recognition model trained by the learning device 10 of the second embodiment is more capable of reducing the false recognition rate for noisy speech data than the method described in Non-Patent Document 1. It could be confirmed. In other words, it was confirmed that the learning device 10 of the second embodiment can create a speech recognition model that is more robust against noise than the method described in Non-Patent Document 1.

[program]
An example of a computer that executes the above program (learning program) will be explained using FIG. 10. As shown in FIG. 10, the computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These parts are connected by a bus 1080.

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). Hard disk drive interface 1030 is connected to hard disk drive 1090. Disk drive interface 1040 is connected to disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100, for example. For example, a mouse 1110 and a keyboard 1120 are connected to the serial port interface 1050. For example, a display 1130 is connected to the video adapter 1060.

Here, as shown in FIG. 10, the hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. The storage unit 13 described in the above embodiment is installed in, for example, the hard disk drive 1090 or the memory 1010.

Then, the CPU 1020 reads out the program module 1093 and program data 1094 stored in the hard disk drive 1090 to the RAM 1012 as necessary, and executes each procedure described above.

Note that the program module 1093 and program data 1094 related to the above-mentioned learning program are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. May be read. Alternatively, the program module 1093 and program data 1094 related to the above program are stored in another computer connected via a network such as a LAN or WAN (Wide Area Network), and read out by the CPU 1020 via the network interface 1070. may be done.

10 Learning device 11 Input section 12 Output section 13 Storage section 14 Control section 100 Speech recognition device 141 First data input section 142 Second data input section 143 Speech recognition section 144 Encoding section 145 Attention mechanism section 146 Decoding section 147 Learning section 1471 First learning section 1472 First distance calculating section 1473 Second distance calculating section 1474 Second learning section 1475 Third distance calculating section

Claims (5)

Converting the audio data into information that specifies the symbol string indicated by the audio data, using data associating audio data with correct data of information specifying the symbol string indicated by the audio data as first teacher data. When doing so, an encoder outputs an intermediate feature of the audio data, a weight indicating which element to focus on among the elements constituting the intermediate feature, and weighting of the intermediate feature using the weight. a first learning unit that performs learning of a speech recognition model comprising a sum calculated value and an attention mechanism that outputs the calculated sum;
Based on the second teacher data that associates the audio data, noisy audio data that is audio data with noise added to the audio data, and correct answer data of information that specifies a symbol string indicated by the audio data, How much does the distribution of weights output from the attention mechanism of the speech recognition model differ between when speech data is input into the speech recognition model trained by the learning section 1 and when the noisy speech data is input? A first distance indicating whether there is a difference, and how much difference there is between the information output from the decoder of the speech recognition model for the noisy speech data and the correct answer data for the speech data. a distance calculation unit that calculates a second distance shown in the figure;
When learning the speech recognition model after learning by the first learning unit using the second teacher data, the sum of the first distance and the second distance is taken as a loss, and the loss is a second learning unit that updates the parameters of the encoder and attention mechanism of the speech recognition model so that the size of the encoder and the attention mechanism become smaller;
a speech recognition unit that uses the speech recognition model learned by the second learning unit to convert input speech data into information that specifies a symbol string shown in the speech data;
A learning device comprising: The voice recognition model after learning by the first learning unit is different when inputting voice data and when inputting noisy voice data, which is voice data with noise added to the voice data. further comprising a third distance calculation unit that calculates a third distance indicating how much difference there is in the information output from the encoder,
The second learning section includes:
When learning the speech recognition model after learning by the first learning unit, the sum of the first distance, the second distance, and the third distance is taken as a loss, and the loss is reduced. The learning device according to claim 1, wherein parameters of an encoder and an attention mechanism of the speech recognition model are updated. The second learning section includes:
When learning the speech recognition model after learning by the first learning unit, updating the parameters of the encoder and attention mechanism of the speech recognition model is repeated until the loss becomes equal to or less than a predetermined threshold. The learning device according to claim 1 or 2, characterized in that the learning device is characterized by: A learning method performed by a learning device, comprising:
Converting the audio data into information that specifies the symbol string indicated by the audio data, using data associating audio data with correct data of information specifying the symbol string indicated by the audio data as first teacher data. When doing so, an encoder outputs an intermediate feature of the audio data, a weight indicating which element to focus on among the elements constituting the intermediate feature, and weighting of the intermediate feature using the weight. a first learning step of learning a speech recognition model comprising a sum calculated value and an attention mechanism that outputs the calculated sum;
Based on the second teacher data that associates the audio data, noisy audio data that is audio data with noise added to the audio data, and correct answer data of information that specifies the symbol string indicated by the audio data, the first How much difference is there in the distribution of weights output from the attention mechanism of the speech recognition model when speech data is input to the speech recognition model after learning in the learning step and when the noisy speech data is input? and the degree of difference between the information output from the decoder of the speech recognition model for the noisy speech data and the correct data for the speech data. a distance calculation step of calculating a second distance shown;
When learning the speech recognition model after learning in the first learning step using the second teacher data, the sum of the first distance and the second distance is taken as a loss, and the loss is a second learning step of updating the parameters of the encoder and attention mechanism of the speech recognition model so as to reduce the size of the encoder and the attention mechanism;
a speech recognition step of converting the input speech data into information specifying a symbol string indicated in the speech data using the speech recognition model learned in the second learning step;
A learning method characterized by including. Converting the audio data into information that specifies the symbol string indicated by the audio data, using data associating audio data with correct data of information specifying the symbol string indicated by the audio data as first teacher data. When doing so, an encoder outputs an intermediate feature of the audio data, a weight indicating which element to focus on among the elements constituting the intermediate feature, and weighting of the intermediate feature using the weight. a first learning step of learning a speech recognition model comprising a sum calculated value and an attention mechanism that outputs the calculated sum;
Based on the second teacher data that associates the audio data, noisy audio data that is audio data with noise added to the audio data, and correct answer data of information that specifies the symbol string indicated by the audio data, the first How much difference is there in the distribution of weights output from the attention mechanism of the speech recognition model when speech data is input to the speech recognition model after learning in the learning step and when the noisy speech data is input? and the degree of difference between the information output from the decoder of the speech recognition model for the noisy speech data and the correct data for the speech data. a distance calculation step of calculating a second distance shown;
When learning the speech recognition model after learning in the first learning step using the second teacher data, the sum of the first distance and the second distance is taken as a loss, and the loss is a second learning step of updating the parameters of the encoder and attention mechanism of the speech recognition model so as to reduce the size of the encoder and the attention mechanism;
a speech recognition step of converting the input speech data into information specifying a symbol string indicated in the speech data using the speech recognition model learned in the second learning step;
A learning program that causes a computer to execute.

Images