CN113257248B - Streaming and non-streaming hybrid speech recognition system and streaming speech recognition method - Google Patents

Abstract

The invention provides a streaming and non-streaming mixed speech recognition system, comprising: a streaming encoder, a connected time sequence classification decoder and an attention mechanism decoder; the streaming encoder adopts a Transformer based on a local self-attention mechanism to perform Construction; the linking temporal classification decoder contains a linear mapping layer, which is responsible for mapping the encoding state to the pre-designed vocabulary space, so that the dimension of the encoding state mapping representation is the same as the dimension of the vocabulary space, and then calculates the predicted token through Softmax , used for streaming decoding; the attention mechanism decoder is constructed with Transformer decoder, which is composed of front-end convolutional layer and multi-layer repeated one-way Transformer encoding layer, and the last layer is a linear mapping layer, which makes the encoding state map represented by The dimensions are the same as those of the vocabulary space, and the probability of the final output is calculated.

Description

Streaming and non-streaming hybrid speech recognition system and streaming speech recognition method

technical field

The present application relates to the field of speech recognition, and in particular, to a mixed speech recognition system of streaming and non-streaming.

Background technique

At present, speech recognition technology has been widely used. Speech recognition can be divided into streaming speech recognition systems and non-streaming speech recognition systems according to different application scenarios. In order to reduce delay and real-time rate, streaming speech recognition systems rely on The acoustic context is greatly reduced, which also affects the recognition effect of the model to a certain extent. The non-streaming speech recognition system, which is applied to the occasions where the real-time rate is not required, can use all the acoustic sequences for prediction. Generally, the non-streaming system has a better recognition effect than the streaming recognition system. However, in order to adapt to different task requirements, models are generally trained separately for streaming and non-streaming tasks, and there is no effective solution that can implement a model for two tasks. The present invention proposes a speech recognition system, which integrates streaming and non-streaming models into the same model, realizes one model, two decoding modes, and is suitable for two types of tasks.

At present, there are many solutions for streaming speech recognition and non-streaming speech recognition, but there are not many solutions for unifying the two recognition models into one framework. There are mainly two ideas:

The first idea is Google's idea, which realizes the adaptation of the same encoder for streaming (local context) and non-streaming (global context) through the variable context training of the encoder part. During the model training process, both streaming and non-streaming are trained. When the streaming model is trained, the acoustic context is masked and only the acoustic context is relied upon. When training non-streaming, the masking operation is not used, but the full acoustic context is modeled. In order to eliminate the performance gap between the streaming model and the non-streaming model, the model also uses the idea of knowledge extraction, and uses the non-streaming model to improve the performance of the streaming model. The decoder can use one decoder to implement two decoding modes, and only need to select different encoders for different task requirements.

The second idea is the hybrid model proposed by Alibaba, whose model consists of two encoders (streaming and non-streaming) and two decoders. The system uses different types of encoders to encode the input speech, selects the streaming encoder for streaming tasks, then uses the streaming decoder for preliminary decoding, and uses the non-streaming decoder to re-score the decoding results. . To perform non-streaming decoding is to rely only on the non-streaming encoder and decoder when decoding. This structural model is relatively complex.

The embodiments disclosed in Application Publication No. CN111402891A provide a speech recognition method, apparatus, device and storage medium. The method includes acquiring the speech feature sequence of the current speech signal to be recognized; inputting the speech feature sequence into a pre-trained Deep-FSMN model to obtain an output sequence representing the probability of each phoneme; inputting the output sequence into a pre-trained model. The CTC model obtains the corresponding phoneme sequence; the phoneme sequence is input into the language model, and converted into the final text sequence as the recognition result. In this way, the performance of the model can be improved, the delay of speech recognition can be reduced, the amount of computation can be reduced, and the effect of speech recognition can be improved.

Application publication number CN111968629A claims to protect a Chinese speech recognition method combining Transformer and CNN-DFSMN-CTC. The method includes steps: S1, preprocessing the speech signal, and extracting 80-dimensional log melFbank features; S2, extracting the extracted The 80-dimensional Fbank features are convolved with a CNN convolution network; S3, the features are input into the DFSMN network structure; S4, the CTC loss is used as the loss function of the acoustic model, the Beam search algorithm is used for prediction, and the Adam optimizer is used for optimization; S5, the strong language model Transformer is introduced for iterative training until the optimal model structure is reached; S6, the Transformer is combined with the acoustic model CNN-DFSMN-CTC for adaptation, and verified on multiple data sets, and finally the optimal recognition result is obtained. The invention has higher recognition accuracy and faster decoding speed, and the character error rate reaches 11.8% after verification on multiple data sets, among which the character error rate of 7.8% is best achieved on the Aidatatang data set.

The main problems of the prior art include two aspects:

(1) Model redundancy, large amount of training tasks, and separate training; currently, for streaming recognition and non-streaming recognition tasks, models are usually trained separately, and the models have some duplication of functions and system redundancy problems, and they are separately trained. Training the model increases the task volume of model training.

(2) The model structure is complex; similar to the Alibaba model, the model contains two encoders and two decoders. The combination of different encoders is applied to different tasks, and the model structure is more complex and difficult to train.

SUMMARY OF THE INVENTION

In view of this, the present invention provides a mixed speech recognition system of streaming and non-streaming. Specifically, the present invention is realized by the following technical solutions:

In a first aspect, the present invention provides a streaming and non-streaming hybrid speech recognition system, including: a streaming encoder, a connected timing classification decoder and an attention mechanism decoder; the streaming encoder is for streaming modeling, It is necessary to get rid of the dependence on all acoustic contexts, and use the Transformer based on the local self-attention mechanism to construct and output the encoding state; the connection time series classification decoder contains a linear mapping layer, which is responsible for mapping the encoding state to the pre-designed vocabulary space. , get the encoding state map representation, make the dimension of the encoding state map representation the same as the dimension of the vocabulary space, and then calculate the predicted tag through Softmax, which is mainly used for streaming decoding; the attention mechanism decoder is constructed by using the Transformer decoder, It consists of a front-end convolution layer and a multi-layer repeated one-way Transformer encoding layer, and the last layer is a linear mapping layer, so that the dimension represented by the encoding state map is the same as the dimension of the vocabulary space, and the probability of the final output is calculated;

During the training process of the model, the connection sequence classification decoder calculates the CTC loss function, and the attention mechanism decoder calculates the cross entropy loss function; the weighted summation of the CTC loss function and the cross entropy loss function is used as the model loss of the recognition system. function;

In the streaming reasoning process, the connected time series classification decoder is used as the leading role, and the attention mechanism decoder is assisted. The connected time series classification decoder uses the BeamSearch search algorithm to generate N stream candidate acoustic sequences and the N stream The CTC streaming acoustic scores of the candidate acoustic sequences are calculated by the attention mechanism, and the N streaming candidate acoustic sequences are re-scored by the attention mechanism decoder, and the N streaming candidate acoustic sequences are rearranged according to the scores of each streaming candidate acoustic sequence. Use the stream candidate acoustic sequence with the highest score as the final stream recognition result;

In the non-streaming reasoning process, the attention mechanism decoder is used as the leading, and the time-series classification decoder is used as the auxiliary. The attention mechanism decoder uses the M non-streaming sequences with the highest scores generated by the BeamSearch search algorithm in the decoding process as M pieces of non-flow candidate acoustic sequences, the M pieces of non-flow candidate acoustic sequences are re-scored by the connection timing classification decoder, and the M pieces of non-flow candidate acoustic sequences are rearranged according to the score of each piece of non-flow candidate acoustic sequence, using The non-flow candidate acoustic sequence with the highest score is used as the final non-flow recognition result.

Preferably, the specific form of the model loss function is:

Model loss function=λ*CTC loss function+(1-λ)*cross entropy loss function;

in,

λ: Setting parameter, 0.1≤λ≤0.3.

Preferably, the specific method for the attention mechanism decoder to re-score the N stream candidate acoustic sequences is:

Amplify a sentence start marker at the front of each streaming candidate acoustic sequence as the input streaming candidate sequence;

The attention mechanism decoder uses the N input stream candidate sequences and the coding states corresponding to the N stream candidate sequences as input, and predicts the stream target candidate sequence that contains the end tag and does not contain the start tag. The probabilities at the positions are summed to obtain a streaming attention score, which is used as a score for re-scoring the N streaming candidate acoustic sequences.

Preferably, the re-scoring score further includes: taking the weighted summation of the streaming attention score and the CTC streaming acoustic score as a score for re-scoring the N streaming candidate acoustic sequences.

Preferably, N is a setting parameter, and 10≤N≤100.

Preferably, the specific process that the attention mechanism decoder uses the BeamSearch search algorithm to generate the M non-streaming sequences with the highest scores in the decoding process as the M non-streaming candidate acoustic sequences is as follows:

Start prediction from the start tag, each step needs to input the complete encoding state and the tag predicted in the previous step, and then calculate the score of the predicted tag; repeat this process until the end tag is predicted and stop; then the attention mechanism The M non-streaming sequences with the highest scores generated in the decoding process of the decoder are used as the non-streaming attention scores of the M non-streaming candidate acoustic sequences and the M non-streaming candidate acoustic sequences, and the start markers and end markers are removed.

Preferably, a specific method for re-scoring the M non-streaming candidate acoustic sequences by the concatenated time series classification decoder is: weighted summation of the CTC non-streaming acoustic scores and the non-streaming attention scores to obtain the The concatenated temporal classification decoder re-scores the M non-streaming candidate acoustic sequences.

Preferably, M is a setting parameter, 10≤M≤100.

The present invention also provides a stream speech recognition method, comprising:

(1) Every time the input audio stream reaches a fixed length, it is calculated as an acoustic feature stream and input into the streaming encoder;

(2) After passing through the streaming encoder, the streamed acoustic feature stream is converted into a streaming encoding state and input to the concatenated time series classification decoder;

(3) The BeamSearch search algorithm is used in conjunction with the time series classification decoder to predict the streaming encoding state;

(4) Repeat the above (1)-(3), if the end of the sentence is encountered, the streaming encoding state ends, and finally N pieces of streaming candidate acoustic sequences and the CTC streaming acoustics of the N pieces of streaming candidate acoustic sequences are generated Fraction;

(5) The attention mechanism decoder re-scores the N stream candidate acoustic sequences, and a sentence start marker is amplified at the front of each stream candidate sequence as the input stream candidate sequence; the attention mechanism decoder uses N The input stream candidate sequences and the encoding states corresponding to the N stream candidate sequences are used as input, and the stream target candidate sequences that contain the end marker but not the start marker are predicted, and the probability of each position of the stream target candidate sequence is calculated. and, the streaming attention score is calculated, and the streaming attention score is used as the score for re-scoring the N streaming candidate acoustic sequences;

Or use the weighted summation of the streaming attention score and the CTC streaming acoustic score as the score for re-scoring the N streaming candidate acoustic sequences;

(6) Rearrange the N stream candidate acoustic sequences according to the score of each stream candidate acoustic sequence, and use the stream candidate acoustic sequence with the highest score as the final stream recognition result. The performance of streaming speech recognition is improved by increasing the number N of streaming candidate acoustic sequences. The typical value of N is 10, and the parameter setting range is 10≤N≤100.

The present invention also provides a non-streaming speech recognition method, comprising:

(1) After the audio input ends, extract features from the entire audio segment and input it into the streaming encoder for encoding;

(2) The attention mechanism decoder relies on the entire output of the streaming encoder and the start tag as input, and starts from the start tag to make predictions. Each step needs to input the complete encoding state and the tag predicted in the previous step, and then calculate the prediction marked score;

(3) Repeat step (2) until the end marker is predicted to stop; then the M non-streaming sequences with the highest scores generated during the decoding process of the attention mechanism decoder are used as M non-streaming candidate acoustic sequences and M non-streaming candidates The non-flow attention score of the acoustic sequence, with start and end markers removed;

(4) Re-score all M non-streaming candidate acoustic sequences using the connected time sequence classification decoder, and use the dynamic programming algorithm to calculate the probability of predicting the target non-streaming candidate acoustic sequence under the condition of inputting the complete speech input as the CTC non-streaming acoustic sequence Fraction;

(5) The weighted summation of the CTC non-streaming acoustic score and the non-streaming attention score is obtained to obtain a score for re-scoring the M non-streaming candidate acoustic sequences by the connected time sequence classification decoder, and reordering;

(6) Finally output the branch with the highest score of the M non-flow candidate acoustic sequences for re-score as the identification result.

Compared with the prior art, the above-mentioned technical solutions provided in the embodiments of the present application have the following advantages:

The method provided in the embodiments of the present application,

(1) The model structure is simple. The present invention only includes one stream encoder and two decoders (CTC decoder and Attention) decoder, and its model structure is relatively simple.

(2) The model decoding process is simple and complementary. In the streaming and non-streaming decoding of the model, it is only necessary to reverse the arrangement order of different decoders to obtain a certain performance improvement.

(3) The model training is simple. Compared with the training process of other streaming and non-streaming models, the system itself contains two types of decoders, which can be trained jointly, which mutually improves the convergence speed and training speed of different modules.

Description of drawings

1 is a structural block diagram of a streaming and non-streaming hybrid speech recognition system provided by an embodiment of the present invention;

2 is a schematic diagram of a self-attention mechanism of a streaming encoder Transformer provided by an embodiment of the present invention;

3 is a flowchart of a method for stream speech recognition provided by an embodiment of the present invention;

FIG. 4 is a flowchart of a method for non-streaming speech recognition provided by an embodiment of the present invention.

Detailed ways

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. Where the following description refers to the drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with some aspects of the invention as recited in the appended claims.

As shown in FIG. 1, the streaming and non-streaming hybrid speech recognition system provided by the embodiment of the present application,

Including: a streaming encoder, a connected temporal classification decoder, and an attention mechanism decoder; the streaming encoder is for streaming modeling and needs to get rid of the dependence on all acoustic contexts, and adopts a Transformer based on a local self-attention mechanism. to construct; the connection timing classification decoder includes a linear mapping layer, which is responsible for mapping the encoding state to the pre-designed vocabulary space, so that the dimension of the encoding state mapping representation is the same as the dimension of the vocabulary space, and then through the Softmax calculates the predicted mark, which is mainly used for streaming decoding; the attention mechanism decoder is constructed with a Transformer decoder, which consists of a front-end convolutional layer and a multi-layer repeated one-way Transformer encoding layer, and the last layer is linear The mapping layer makes the dimension of the encoding state mapping representation the same as the dimension of the vocabulary space, and calculates the probability of the final output;

During the training process of the model, the coupled time series classification decoder decoder calculates the CTC loss function, L _CTC ; the attention mechanism decoder calculates the cross entropy loss function, L _CE ; the CTC loss function and the cross entropy are calculated. The weighted summation of the loss function is used as the model loss function L _final of the recognition system;

The specific form of the model loss function is:

Model loss function=λ*CTC loss function+(1-λ)*cross entropy loss function;

in,

λ: set parameter, it is 0.1;

In the streaming reasoning process, the connected time series classification decoder is used as the lead, the attention mechanism decoder is used as the auxiliary, and the connected time series classification decoder uses the BeamSearch search algorithm to generate N pieces of streaming candidate acoustic sequences from the encoding state. and the CTC streaming acoustic scores of the N streaming candidate acoustic sequences, the attention mechanism decoder re-scores the N streaming candidate acoustic sequences, according to the scores of each streaming candidate acoustic sequence The N stream candidate acoustic sequences are rearranged, and the stream candidate acoustic sequence with the highest score is used as the final stream recognition result;

The specific method for the attention mechanism decoder to re-score the N stream candidate acoustic sequences is as follows:

Amplify a sentence start marker at the front of each streaming candidate sequence as the input streaming candidate sequence;

The attention mechanism decoder uses the N input stream candidate sequences and the coding states corresponding to the N stream candidate sequences as input, predicts the stream target candidate sequence that contains the end tag and does not contain the start tag, and predicts the stream target candidate sequence. The probabilities at each position of the sequence are summed, and the streaming attention score is calculated, and the streaming attention score is used as the score for re-scoring the N streaming candidate acoustic sequences;

The re-scoring score further includes: a weighted summation of the streaming attention score and the CTC streaming acoustic score as a score for re-scoring the N streaming candidate acoustic sequences; a typical value of N is 10, the value can be expanded to 50 or 100;

In the non-streaming reasoning process, the attention mechanism decoder is used as the leading role, and the connection time series classification decoder is used as an auxiliary. The attention mechanism decoder uses the BeamSearch search algorithm to generate the highest scoring The M non-streaming candidate acoustic sequences are regarded as M non-streaming candidate acoustic sequences, and the M non-streaming candidate acoustic sequences are re-scored by the connection timing classification decoder, and the M non-streaming candidate acoustic sequences are classified according to the score of each non-streaming candidate acoustic sequence. The flow candidate acoustic sequences are rearranged, and the non-flow candidate acoustic sequence with the highest score is used as the final non-flow identification result;

The specific process that the attention mechanism decoder uses the BeamSearch search algorithm to generate the M non-streaming sequences with the highest scores in the decoding process as the M non-streaming candidate acoustic sequences is as follows:

Start prediction from the start tag, each step needs to input the complete encoding state and the tag predicted in the previous step, and then calculate the score of the predicted tag; repeat this process until the end tag is predicted and stop; then the attention mechanism The M non-streaming sequences with the highest scores generated in the decoding process of the decoder are used as the non-streaming attention scores of the M non-streaming candidate acoustic sequences and the M non-streaming candidate acoustic sequences, and the start marker and the end marker are removed;

The specific method for re-scoring the M non-streaming candidate acoustic sequences by the connection timing classification decoder is: weighted summation of the CTC non-streaming acoustic score and the non-streaming attention score to obtain the connection A score for re-scoring the M non-streaming candidate acoustic sequences by the time-series classification decoder;

The typical value of M is 10, and the value can be extended to 50 or 100;

As shown in Figure 2, the streaming encoder is constructed using the streaming Transformer model structure; it consists of a front-end module and multiple layers of repeated unidirectional Transformer encoding layers. The front-end module contains two layers of convolution, the convolution kernel is 3X3, the stride is set to 2, and the ReLU activation function is used as the connection between the convolutions, and finally the output is the model dimension through the linear mapping layer. The one-way Transformer encoding layer consists of a multi-head one-way self-attention mechanism and a feed-forward network. Each multi-head one-way self-attention mechanism and feed-forward network uses residual connections and post-layer normalization to help the model converge. . The relative position encoding is introduced in the one-way self-attention mechanism. The calculation process of the encoder is as follows:

Convolution front-end module:

where ReLU represents the activation function, Conv2D represents the 2D convolutional layer, x represents the input speech feature, O _conv1 represents the output of the first layer convolution, and O _front represents the output of the convolution front-end module.

Multi-head one-way self-attention mechanism:

Among them, Wi ^Q , ^{Wi W} , _Wi V _and ^WO represent the parameter ^matrix that can be learned, and Q, K , and _V represent the query matrix, key matrix and value matrix respectively. For the self-attention mechanism, Q =K=V, if this is the first layer of self-attention mechanism, then Q=K=V=O _front , if it is not the first layer, this Q=K=V is equal to the output of the previous layer of feedforward neural network. A represents the relative position encoding matrix that can be learned. The implementation of relative position encoding in TransformerXL can also be used instead. d _k represents the dimension of the last dimension of the key-value matrix. Each attention head head _i is an independent attention mechanism, which concatenates H attention outputs and obtains the final output O _SLF through linear mapping

Calculation of feedforward neural network:

Where Linear represents linear mapping, GLU represents activation function;

The attention mechanism decoder is constructed with a Transformer-based decoder, which includes a front-end word embedding representation module, a sine and cosine position encoding, multiple layers of repeated Transformer decoding layers and a final linear mapping. Self-attention mechanism and encoder-decoder attention mechanism and feed-forward network composition. Each multi-head masked self-attention mechanism and encoder-decoder attention mechanism and feed-forward network use residual connections and post-layer normalization to help the model converge. The calculation process is as follows:

where P _e represents the sine-cosine positional encoding, and O _e represents the word embedding representation with positional encoding.

Masked Self-Attention Mechanism:

Among them, ^{Wi Q} , Wi W _, Wi _V and ^WO represent the parameter ^matrix that can be learned, and Q, K , and _V represent the query matrix, the key matrix and the value matrix respectively. For the self-attention mechanism, Q= K=V, if this is the first layer of self-attention mechanism, then Q=K=V=O _e , otherwise, it is equal to the output of the previous layer of feed-forward neural network. In the calculation process of the masked self-attention mechanism, the information of the future frames of each vector needs to be masked to force the model to learn the temporal dependencies between languages.

Encoding and decoding attention mechanism:

Among them, ^{Wi Q} , Wi W _, Wi _V and ^WO represent the parameter ^matrix that can be learned, and Q, K , and _V represent the query matrix, the key matrix and the value matrix respectively. For the self-attention mechanism, Q= K=V= _Oslf ;

Feedforward network calculation:

where Linear represents a linear map and GLU represents an activation function.

As shown in Figure 3, a streaming speech recognition method includes:

(1) Every time the input audio stream reaches a fixed length, it is calculated as an acoustic feature stream and input into the streaming encoder;

(2) After passing through the streaming encoder, the streamed acoustic feature stream is converted into a streaming encoding state and input to the concatenated time series classification decoder;

(3) The BeamSearch search algorithm is used in conjunction with the time series classification decoder to predict the streaming encoding state;

(4) Repeat the above (1)-(3), if the end of the sentence is encountered, the streaming encoding state ends, and finally N pieces of streaming candidate acoustic sequences and the CTC streaming acoustics of the N pieces of streaming candidate acoustic sequences are generated Fraction;

(5) The attention mechanism decoder re-scores the N stream candidate acoustic sequences, and amplifies a sentence start marker at the front of each stream candidate sequence as the input stream candidate sequence; the attention mechanism decoder Using the N input stream candidate sequences and the coding states corresponding to the N stream candidate sequences as input, predicting the stream target candidate sequence that contains the end marker but does not contain the start marker, the probability of each position of the stream target candidate sequence is calculated. performing a summation to obtain a streaming attention score, which is used as a score for re-scoring the N streaming candidate acoustic sequences;

Or weighted summation of the streaming attention score and the CTC streaming acoustic score is used as a score for re-scoring the N streaming candidate acoustic sequences;

(6) Rearrange the N stream candidate acoustic sequences according to the score of each stream candidate acoustic sequence, and use the stream candidate acoustic sequence with the highest score as the final stream recognition result. The performance of streaming speech recognition is improved by increasing the number N of streaming candidate acoustic sequences. The typical value of N is 10, and the parameter setting range is 10≤N≤100.

(7) Steps (4) and (5) are performed segment by segment. Each time a fixed number of words are identified, the current streaming candidate acoustic sequence is reordered, and the streaming candidate acoustic sequence can be reordered depending on the sorting result. A pruning operation is performed to improve the subsequent decoding efficiency.

As shown in Figure 4, a non-streaming speech recognition method includes:

(1) After the audio input ends, extract features from the entire audio segment and input it into the streaming encoder for encoding;

(2) The attention mechanism decoder relies on the entire output of the streaming encoder and the start tag as input, and starts from the start tag to make predictions. Each step needs to input the complete encoding state and the tag predicted in the previous step, and then calculate the prediction marked score;

(3) Step (2) is repeated until the end mark is predicted to stop; then the M non-streaming sequences with the highest scores generated in the decoding process of the attention mechanism decoder are used as M non-streaming candidate acoustic sequences and M non-streaming candidate acoustic sequences. The non-flow attention score of the flow candidate acoustic sequence, excluding the start and end tags;

(4) Re-score all M non-streaming candidate acoustic sequences using the connected time sequence classification decoder, and use the dynamic programming algorithm to calculate the probability of predicting the target non-streaming candidate acoustic sequence under the condition of inputting the complete speech input as the CTC non-streaming acoustic sequence Fraction;

(5) weighted summation of the CTC non-streaming acoustic score and the non-streaming attention score to obtain the score for re-scoring the M non-streaming candidate acoustic sequences by the connected timing classification decoder, and reordering;

(6) Finally output the branch with the highest score of the M non-flow candidate acoustic sequences for re-score as the identification result.

It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, such information should not be limited by these terms. These terms are only used to distinguish the same type of information from each other. For example, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information, without departing from the scope of the present invention. Depending on the context, the word "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determining."

While this specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or what may be claimed, but rather are used primarily to describe features of specific embodiments of particular inventions. Certain features that are described in this specification in multiple embodiments can also be implemented in combination in a single embodiment. On the other hand, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may function as described above in certain combinations and even be originally claimed as such, one or more features from a claimed combination may in some cases be removed from the combination and the claimed A protected combination may point to a subcombination or a variation of a subcombination.

Similarly, although operations are depicted in the figures in a particular order, this should not be construed as requiring the operations to be performed in the particular order shown or sequentially, or that all illustrated operations be performed, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product , or packaged into multiple software products.

Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Furthermore, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the present invention. within the scope of protection.

Claims (9)

1. A hybrid speech recognition system for streaming and non-streaming, comprising: a stream encoder, a concatenated sequential classification decoder, and an attention mechanism decoder; the stream type encoder is constructed by adopting a Transformer based on a local self-attention mechanism, and outputs an encoding state; the joint time sequence classification decoder comprises a linear mapping layer which is responsible for mapping the coding state to a pre-designed word list space to obtain a coding state mapping representation, the dimension of the coding state mapping representation is the same as that of the word list space, and then the predicted mark is calculated through Softmax and used for stream decoding; the attention mechanism decoder is constructed by adopting a Transformer decoder and consists of a front-end convolution layer and a plurality of repeated unidirectional Transformer coding layers, the last layer is a linear mapping layer, the dimensionality represented by the coding state mapping is the same as the dimensionality of the vocabulary space, and the final output probability is calculated; training a flow type and non-flow type mixed voice recognition system, wherein in the training process, the connection time sequence classification decoder calculates a CTC loss function, and the attention mechanism decoder calculates a cross entropy loss function; performing weighted summation on the CTC loss function and the cross entropy loss function to serve as a model loss function of the identification system;

in the flow type reasoning process, the joint time sequence classification decoder is used as a main part, the attention mechanism decoder is used as an auxiliary part, the joint time sequence classification decoder adopts a BeamSearch search algorithm to generate N flow type candidate acoustic sequences and CTC flow type acoustic scores of the N flow type candidate acoustic sequences from a coding state, the attention mechanism decoder performs scoring again on the N flow type candidate acoustic sequences, the N flow type candidate acoustic sequences are rearranged according to the score of each flow type candidate acoustic sequence, and the flow type candidate acoustic sequence with the highest score is used as a final flow type recognition result;

in the non-streaming reasoning process, the attention mechanism decoder is used as a main guide, the connection time sequence classification decoder is used as an auxiliary guide, the attention mechanism decoder adopts M non-stream sequences with the highest scores generated in the decoding process by a BeamSearch search algorithm as M non-stream candidate acoustic sequences, the connection time sequence classification decoder performs re-scoring on the M non-stream candidate acoustic sequences, the M non-stream candidate acoustic sequences are rearranged according to the scores of the non-stream candidate acoustic sequences, and the non-stream candidate acoustic sequence with the highest score is used as a final non-stream identification result.

2. A mixed speech recognition system for streaming and non-streaming according to claim 1, wherein the model loss function is of the specific form:

model loss function = λ × CTC loss function + (1- λ) × cross entropy loss function;

wherein,

λ: setting parameters, wherein lambda is more than or equal to 0.1 and less than or equal to 0.3.

3. The mixed speech recognition system of claim 1, wherein the attention mechanism decoder re-scores the N streaming candidate acoustic sequences by:

amplifying a sentence starting mark at the front end of each streaming candidate acoustic sequence to serve as an input streaming candidate sequence;

the attention mechanism decoder adopts N input streaming candidate sequences and coding states corresponding to the N streaming candidate sequences as input, predicts a streaming target candidate sequence which contains an end mark and does not contain a start mark, sums the probability of each position of the streaming target candidate sequence, and calculates a streaming attention score which is used for reprinting the scores of the N streaming candidate acoustic sequences.

4. The mixed speech recognition system of claim 3, wherein the reprint score further comprises: weighted summing the streaming attention score and the CTC streaming acoustic score as a reprint score for the N streaming candidate acoustic sequences.

5. The system of claim 4, wherein N is a setting parameter, 10 ≦ N ≦ 100.

6. The mixed speech recognition system of claim 1, wherein the attention mechanism decoder adopts the BeamSearch algorithm to generate M non-stream sequences with the highest scores in the decoding process as M non-stream candidate acoustic sequences by:

predicting from a start mark, wherein each step needs to input a complete coding state and a mark obtained by the previous step of prediction, and then calculating the score of the prediction mark; this process is repeated until the end marker is predicted to stop; and then, taking the M non-stream sequences with the highest scores generated in the decoding process of the attention mechanism decoder as the non-stream attention scores of the M non-stream candidate acoustic sequences and the M non-stream candidate acoustic sequences, and removing the start mark and the end mark.

7. The system of claim 6, wherein the specific method for re-scoring the M non-stream candidate acoustic sequences by the concatenated sequential classification decoder is: and weighting and summing the CTC non-stream acoustic scores and the non-stream attention scores to obtain the scores of the M non-stream candidate acoustic sequences rewrited by the joint time sequence classification decoder.

8. The system of claim 7, wherein M is a setting parameter, 10 ≦ M ≦ 100.

9. A method for hybrid speech recognition between streaming and non-streaming, comprising:

the method comprises a streaming and non-streaming mixed voice recognition method, and specifically comprises the following steps:

the stream type voice recognition method comprises the following steps:

(1) calculating an acoustic feature stream to be input into a streaming coder every time an input audio stream reaches a fixed length;

(2) the stream type acoustic characteristic stream is converted into a stream type coding state after passing through a stream type coder and is input into a joint time sequence classification decoder;

(3) the connection time sequence classification decoder adopts a BeamSearch search algorithm to predict the flow type coding state;

(4) repeating the above (1) - (3), if a sentence is over, ending the streaming coding state, and finally generating N streaming candidate acoustic sequences and CTC streaming acoustic scores of the N streaming candidate acoustic sequences;

(5) the attention mechanism decoder performs repeated scoring on the N streaming candidate acoustic sequences, and a sentence starting mark is amplified at the front end of each streaming candidate sequence to serve as an input streaming candidate sequence; the attention mechanism decoder adopts N input streaming candidate sequences and coding states corresponding to the N streaming candidate sequences as input, predicts a streaming target candidate sequence which contains an end mark and does not contain a start mark, sums the probability of each position of the streaming target candidate sequence, and calculates a streaming attention score which is used for reprinting the scores of the N streaming candidate acoustic sequences;

or performing a weighted summation of the streaming attention score and the CTC streaming acoustic score as a reprinting score for the N streaming candidate acoustic sequences;

(6) rearranging the N streaming candidate acoustic sequences according to the score of each streaming candidate acoustic sequence, taking the streaming candidate acoustic sequence with the highest score as a final streaming recognition result, and improving the performance of streaming voice recognition by increasing the number N of the streaming candidate acoustic sequences, wherein the typical value of N is 10, and the parameter setting range is as follows: n is more than or equal to 10 and less than or equal to 100,

the non-streaming mixed voice recognition method comprises the following steps: (1) after the audio input is finished, extracting characteristics of the whole audio, and inputting the characteristics into a stream coder for coding;

(2) the power mechanism decoder relies on all the output and the initial mark of the stream encoder as input, the prediction is carried out from the initial mark, each step needs to input the complete coding state and the mark obtained by the previous step of prediction, and then the fraction of the prediction mark is calculated;

(3) repeating step (2) until the end marker is predicted to stop; then, taking the M non-stream sequences with the highest scores generated in the decoding process of the attention mechanism decoder as the non-stream attention scores of the M non-stream candidate acoustic sequences and the M non-stream candidate acoustic sequences, and removing the start mark and the end mark;

(4) using a connection time sequence classification decoder to reprint scores of all M non-stream candidate acoustic sequences, and using a dynamic programming algorithm to calculate the probability of predicting a target non-stream candidate acoustic sequence under the condition of inputting complete voice input to be used as a CTC non-stream acoustic score;

(5) weighting and summing CTC non-stream acoustic scores and the non-stream attention scores to obtain a joint time sequence classification decoder which performs scoring again on the M non-stream candidate acoustic sequences and reorders the scores;

(6) and finally outputting the M non-flow candidate acoustic sequences to perform scoring with the highest scoring again as a recognition result.

Images