CN115019782B - Voice recognition method based on CTC multilayer loss - Google Patents

Abstract

A speech recognition method based on CTC multi-layer loss belongs to the field of pattern recognition and acoustics. The method standardizes the outputs of different layers of the speech recognition network so that the outputs of different layers are as close as possible to the required speech recognition results, thereby improving the performance of speech recognition. The method includes two stages: model training and model testing: in the training stage, the pre-processed training set is input into the constructed multi-layer speech recognition network, the losses of different layers and the weights of different layers are calculated, the weighted sum of the losses of different layers is obtained to obtain the multi-layer loss, the loss is calculated cyclically, and the network parameters are updated until convergence; in the testing stage, the pre-processed test set is input into the trained multi-layer speech recognition network, and the recognition result is output. The present invention only changes the loss function of the CTC speech recognition model training stage, and does not change the structure of the CTC speech recognition model and its speech recognition process, so as to improve the accuracy of speech recognition with the characteristics of low complexity and low overhead.

Description

A speech recognition method based on CTC multi-layer loss

Technical Field

The present invention belongs to the field of pattern recognition and acoustics, and in particular relates to an end-to-end speech recognition technology.

Background technique

Speech recognition is a very important research topic in the field of acoustics. The traditional speech recognition model is a hybrid system using a hidden Markov model, including an acoustic model, a language model, and a pronunciation dictionary, but the model design is complex and difficult. End-to-end speech recognition is a more mainstream speech recognition method today. Compared with traditional methods such as hybrid systems using hidden Markov models, end-to-end speech recognition simplifies the model design, training, and decoding processes. However, this improvement brings more computational costs. Many advanced ASR architectures use an encoding-decoding architecture based on an attention mechanism, which requires a lot of computational costs and a large model size. In addition, the decoder operates in an autoregressive manner and requires sequential calculations, that is, the next token can only be generated after the previous token is completed. The Connectionist Temporal Classification (CTC) model proposed in 2006 does not require a separate decoder, and its design is more compact and faster, making it suitable for use in speech recognition technology. In recent years, scholars have improved the performance of CTC by modifying the model structure and performing pre-training. However, due to the conditional independence assumption of the CTC model, its performance is still weaker than the encoder-decoder model. Overcoming the conditional independence assumption problem of CTC often requires adding an external language model and performing beam search. However, both of them have high complexity and high overhead, and the computational cost is high, making it difficult to improve the performance of the CTC model through low-complexity and low-overhead methods.

Speech recognition technology has many application environments. For example, in voice commands, commands can be issued directly to devices or software through voice, which is suitable for major search scenarios such as video websites and smart hardware; in game entertainment, speech recognition can convert speech into text, which well meets the diverse chat needs of users; it also has important applications in subtitle generation and meeting minutes. With the widespread use of speech recognition technology in production and life, the design of low-complexity, low-cost, and high-performance speech recognition models is particularly important.

Summary of the invention

The present invention is aimed at the situation that the current method for solving the conditional independence hypothesis problem of the CTC model (adding an external language model and performing beam search) has high complexity and high overhead, and proposes a speech recognition method based on CTC multi-layer loss. The method standardizes the outputs of different layers of the speech recognition network so that the outputs of different layers are as close as possible to the required speech recognition results, thereby improving the performance of speech recognition. Standardizing the outputs of different layers has different values for the performance improvement of the CTC multi-layer speech recognition network g, and the values are represented by different weights. In order to avoid the influence of subjectively setting the weights of different layers, the weights are calculated by network training learning. During training, the method obtains the outputs of different layers of the network, calculates the CTC losses of different layers and the weights of different layers, performs weighted summation on the losses of different layers, obtains the final CTC multi-layer loss, and uses the gradient descent method to update the model parameters until the model converges to obtain the final model. Since the method only changes the loss function of the CTC model training stage, and does not change the structure of the CTC model and its test process, there is no additional overhead in the model testing stage. The method is easy to implement and effectively improves the performance of CTC model speech recognition without increasing the complexity and overhead of the CTC model.

The present invention proposes a speech recognition method based on CTC multi-layer loss, which is characterized by comprising a model training phase and a test phase. As shown in FIG1 , the training phase comprises training speech preprocessing, CTC multi-layer speech recognition network construction, probability calculation of different layers, loss calculation of different layers, weight calculation of different layers, CTC multi-layer loss calculation, and updating parameter model convergence. Model testing phase: comprises test speech preprocessing and test speech recognition.

1) Model training phase:

1-1) Training speech preprocessing:

The training set is S = {(x ¹ , y ¹ ), (x ² , y ² ), …, (x ^N , y ^N )}, which means there are N training samples, where the i-th training sample is represented as ( ^xi , ^yi ), x ⁱ is the input sample, which is a voice, and ^yi is the corresponding true label, that is, the text information corresponding to the voice x ⁱ . The voice x ⁱ in the training set is divided into ^Ti frames, and its Mel cepstrum features are solved to obtain the preprocessed training voice.

1-2) Construction of CTC multi-layer speech recognition network:

Construct a CTC multi-layer speech recognition network g, as shown in Figure 2. The network g includes L Transformer layers and 1 softmax layer. The entire network parameter is φ, and the range of L is 20 to 28. Define the input of the network as the preprocessed training speech x ⁱ , and the output of the network is the text information y ⁱ corresponding to the training speech. The network function is

^yi ＝g(φ； ^xi )(i＝1,2,…N)

1-3) Probability calculation at different layers:

The preprocessed training speech ^xi is input into the network g, _xl represents the output of the training speech ^xi after passing through the lth layer of the network g, and the probability P( ^yi | _xl ) that the true label ^yi can be decoded according to _xl is calculated as

This probability is called the different layer probability, where B ^-1 (y ⁱ ) is an alignment set of length T ⁱ that is consistent with y ⁱ , and the contents of the set are all paths that map x _l to y ⁱ , including blank marks, and q is one of the paths.

1-4) Calculation of CTC loss at different layers:

The CTC loss function is defined as the sum of the negative logarithms of the probability P(y ⁱ |x _l ) of decoding to obtain the true label. According to the different layer probabilities obtained in 1-3), the CTC loss of different layers is calculated as

L _{CTC_l} = -ln P(y ⁱ |x _l )(l = 1, 2, ... L)

1-5) Weight calculation of different layers:

The preprocessed training speech xi ^is input into the weight calculation network f to obtain the weights _αl (l＝1,2,…L) of different layers.

in Calculate the parameters of the network f for the weights, And 0≤α _l ≤1. As shown in Figure 3, the weight calculation network f is composed of a convolutional neural network (CNN) network, a pooling layer, a fully connected layer, and a softmax layer, wherein the number of CNN network layers can be set to 4 to 6 layers, and the number of fully connected layers can be set to 3 to 5 layers. The weights of different layers indicate that the standardization of the outputs of different layers has different values for improving the performance of the CTC multi-layer speech recognition network g. The weight is calculated by the network f to avoid subjective weight setting.

1-6) CTC multi-layer loss calculation:

As shown in Figure 2, according to the weights α _l of different layers, the CTC losses of different layers are weighted and summed to obtain the CTC multi-layer loss, that is,

Where L _{MulyiLayer_CTC} is the CTC multi-layer loss.

1-7) Update parameter model convergence:

Using the gradient descent method, minimize L _{MultiLayer_CTC} and update the network parameters φ and Repeat steps 1-3) to 1-6), calculate the CTC multi-layer loss L _{MultiLayer_CTC} , and update the network parameters φ and Until L _{MultiLayer_CTC} is less than the threshold 0.001, the model converges and the training is completed, and the trained CTC multi-layer speech recognition network g and weight calculation network f are obtained.

2) Model testing phase:

2-1) Test speech preprocessing:

For the i-th sample data of the test speech The speech is divided into frames and Mel-frequency cepstrum features are obtained to obtain the preprocessed test speech.

2-2) Test speech recognition:

The preprocessed test speech Input the trained CTC multi-layer speech recognition network g to obtain the speech recognition result.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows the steps of the method of the present invention.

FIG. 2 is the overall architecture of the method of the present invention.

Figure 3 shows the composition of the weight calculation network.

Detailed ways

The present invention proposes a speech recognition method based on CTC multi-layer loss, characterized in that it includes a model training phase and a test phase, as shown in Figure 1, the model training phase includes training speech preprocessing, CTC multi-layer speech recognition network construction, probability calculation of different layers, loss calculation of different layers, CTC multi-layer loss calculation, and updating parameter model convergence. The model testing phase includes test speech preprocessing and test speech recognition. The specific implementation method is described as follows.

1) Model training phase:

1-1) Training speech preprocessing:

The training set is S = {(x ¹ , y ¹ ), (x ² , y ² ), …, (x ^N , y ^N )}, which means there are N training samples, where the i-th training sample is represented as ( ^xi , ^yi ), x ⁱ is the input sample, which is a voice, and ^yi is the corresponding true label, that is, the text information corresponding to the voice x ⁱ . The voice x ⁱ in the training set is divided into ^Ti frames, and its Mel cepstrum features are solved to obtain the preprocessed training voice.

In this embodiment, the aishell-1 dataset is used for training, including a 150-hour training set with a total of 120,098 voices.

1-2) Construction of CTC multi-layer speech recognition network:

Construct a CTC multi-layer speech recognition network g, as shown in Figure 2. The network g includes L Transformer layers and 1 softmax layer. The entire network parameter is φ, and the range of L is 12 to 32. In this embodiment, L is set to 24. Define the input of the network as the preprocessed training speech x ⁱ , and the output of the network is the text information y ⁱ corresponding to the speech. The network function is

^yi ＝g(φ； ^xi )(i＝1,2,…N)

1-3) Probability calculation at different layers:

The preprocessed training speech ^xi is input into the network g, _xl represents the output of the training speech ^xi after passing through the lth layer of the network g, and the probability P( ^yi | _xl ) that the true label ^yi can be decoded according to _xl is calculated as

This probability is called the different layer probability, where B ^-1 (y ⁱ ) is an alignment set of length T ⁱ that is consistent with y ⁱ , and the contents of the set are all paths that map x _l to y ⁱ , including blank marks, and q is one of the paths.

1-4) Calculation of CTC loss at different layers:

The CTC loss function is defined as the sum of the negative logarithms of the probability P(y ⁱ |x _l ) of decoding to obtain the true label. According to the different layer probabilities obtained in 1-3), the CTC loss of different layers is calculated as

L _{CTC_l} = -ln P(y ⁱ |x _l )(l = 1, 2, ... L)

In this embodiment, the CTC loss of each layer of the multi-layer speech recognition network g is calculated.

1-5) Weight calculation of different layers:

The preprocessed training speech xi ^is input into the weight calculation network f to obtain the weights _αl (l＝1,2,…L) of different layers.

in Calculate the parameters of the network f for the weights, And 0≤α _l ≤1. As shown in Figure 3, the weight calculation network f is composed of a convolutional neural network (CNN) network, a pooling layer, a fully connected layer, and a softmax layer, wherein the number of CNN network layers can be set to 4 to 8 layers, and the number of fully connected layers can be set to 3 to 6 layers. The weights of different layers indicate that the standardization of the outputs of different layers has different values for improving the performance of the CTC multi-layer speech recognition network g. The weights are automatically learned by the network f through training to avoid subjective weight setting.

In this embodiment, the weight calculation network f is composed of a 4-layer convolutional neural network (CNN) network, a pooling layer, a 3-layer fully connected layer, and a softmax layer. The weight parameter α _l automatically calculated by the final network f has the characteristics of large weights at high levels and small weights at low levels.

1-6) CTC multi-layer loss calculation:

As shown in Figure 2, according to the weights α _l of different layers, the CTC losses of different layers are weighted and summed to obtain the CTC multi-layer loss, that is,

Where L _{MultiLayer_CTC} is the CTC multi-layer loss.

1-7) Update parameter model convergence:

Using the gradient descent method, minimize L _{MultiLayer_CTC} and update the network parameters φ and Repeat steps 1-3) to 1-6), calculate the CTC multi-layer loss L _{MultiLayer_CTC} , and update the network parameters φ and Until L _{Multilayer_CTC} is less than the threshold 0.001, the training is completed, and the CTC multi-layer speech recognition network g and the weight calculation network f are obtained.

2) Model testing phase:

2-1) Test speech preprocessing:

For the i-th sample data of the test speech Frame the image and obtain the Mel-frequency cepstrum features.

In this embodiment, the aishell-1 dataset is used for testing, including a 5-hour training set with a total of 7176 voices.

2-2) Test speech recognition:

The test sequence Input the trained CTC multi-layer speech recognition network g to obtain the speech recognition result.

The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims (1)

1. A speech recognition method based on CTC multi-layer loss, characterized in that it is divided into a model training phase and a model testing phase; the model training phase includes training speech preprocessing, CTC multi-layer speech recognition network construction, probability calculation of different layers, CTC loss calculation of different layers, weight calculation of different layers, CTC multi-layer loss calculation, and updating parameter model convergence; the model testing phase includes test speech preprocessing and test speech recognition; The specific steps include: 1) Model training phase: 1-1) Training speech preprocessing: The training set is S = {(x ¹ , y ¹ ), (x ² , y ² ), …, (x ^N , y ^N )}, which means there are N training samples, where the i-th training sample is represented by ( ^xi , ^yi ), x ⁱ is the input sample, which is a speech, and ^yi is the corresponding true label, that is, the text information corresponding to the speech x ⁱ ; the speech x ⁱ in the training set is divided into ^Ti frames, and its Mel cepstrum features are solved to obtain the preprocessed training speech; 1-2) Construction of CTC multi-layer speech recognition network: Construct a CTC multi-layer speech recognition network g, which includes L Transformer layers and 1 softmax layer. The entire network parameter is φ, and the range of L is 20 to 28. Define the input of the network as the preprocessed training speech x ⁱ , and the output of the network as the text information y ⁱ corresponding to the training speech. The network function is ^yi = g(φ; ^xi ), where i = 1, 2, ... N; 1-3) Probability calculation at different layers: The preprocessed training speech ^xi is input into the network g, _xl represents the output of the training speech ^xi after passing through the lth layer of the network g, and the probability P( ^yi | _xl ) that the true label ^yi can be decoded according to _xl is calculated as Where l = 1, 2, ... L; This probability is called the probability of different layers, where B ^-1 (y ⁱ ) is an alignment set of length T ⁱ that is consistent with y ⁱ , and the content of the set is all the paths that map x _l to y ⁱ , including blank marks in the path, and q is one of the paths; 1-4) Calculation of CTC loss at different layers: The CTC loss function is defined as the sum of the negative logarithms of the probability P(y ⁱ |x _l ) of decoding to obtain the true label; according to the different layer probabilities obtained in 1-3), the CTC loss of different layers is calculated as L _{CTC_l} = -lnP(y ⁱ |x _l ), where l = 1, 2, ... L; 1-5) Weight calculation of different layers: Input the preprocessed training speech x ⁱ into the weight calculation network f to obtain the weights α _l of different layers, where l = 1, 2, ... L; in Calculate the parameters of the network f for the weights, and 0≤α _l ≤1; the weight calculation network f is composed of a CNN network, a pooling layer, a fully connected layer, and a softmax layer, wherein the number of CNN network layers is set to 4 to 6 layers, and the number of fully connected layers can be set to 3 to 5 layers; the weights of different layers represent that the standardization of the outputs of different layers has different values for improving the performance of the CTC multi-layer speech recognition network g; the weight is calculated by the network f; 1-6) CTC multi-layer loss calculation: According to the weights α _l of different layers, the CTC losses of different layers are weighted and summed to obtain the CTC multi-layer loss, that is, Where L _{MultiLayer_CTC} is the CTC multi-layer loss; 1-7) Update parameter model convergence: Using the gradient descent method, minimize L _{MultiLayer_CTC} and update the network parameters φ and Repeat steps 1-3) to 1-6), calculate the CTC multi-layer loss L _{MultiLayer_CTC} , and update the network parameters φ and Until L _{MultiLayer_CTC} is less than the threshold 0.001, the model converges and the training is completed, and the trained CTC multi-layer speech recognition network g and weight calculation network f are obtained; 2) Model testing phase: 2-1) Test speech preprocessing: For the i-th sample data of the test speech Frame division is performed and Mel-frequency cepstrum features are obtained to obtain the preprocessed test speech; 2-2) Test speech recognition: The preprocessed test speech Input the trained CTC multi-layer speech recognition network g to obtain the speech recognition result.