CN118553234A - Speech recognition model training, testing, speech recognition method and device - Google Patents

Abstract

The present disclosure provides a method and device for training a speech recognition model, which relates to the field of artificial intelligence technology, specifically to the technical fields of speech recognition, deep learning, large models, etc., and can be applied to scenarios such as artificial intelligence content generation. The specific implementation scheme is: obtaining a speech sample set, the speech sample set includes at least one speech sample, and the speech sample includes: an audio feature sequence and an initial word unit sequence; obtaining an initial speech recognition model, the speech recognition model is used to characterize the correspondence between the audio feature sequence and the predicted word unit sequence; using the predicted word unit representing the language to replace the language word unit in the initial word unit sequence in the speech sample set, to obtain a training sample set, the predicted word unit is a predicted word unit in the predicted word unit sequence obtained by inputting the speech sample selected from the speech sample set into the speech recognition model; based on the training sample set, the speech recognition model is trained to obtain a trained speech recognition model.

Description

Speech recognition model training, testing, speech recognition method and device

Technical Field

The present disclosure relates to the field of artificial intelligence technology, specifically to technical fields such as speech recognition, deep learning, and large models, and can be applied to scenarios such as artificial intelligence content generation, and in particular to a speech recognition model training method and device, a speech recognition model testing method and device, a speech recognition method and device, an electronic device, a computer-readable storage medium, and a computer program product.

Background Art

At present, speech recognition technology has become increasingly mature, especially for widely used languages such as English and Chinese, speech recognition technology is comparable to that of professional staff. However, some languages are limited by objective factors such as the number of training samples, resulting in the weak speech recognition ability of existing models.

Summary of the invention

The present disclosure provides a speech recognition model training method and device, a speech recognition model testing method and device, a speech recognition method and device, an electronic device, a computer-readable storage medium, and a computer program product.

According to a first aspect, a method for training a speech recognition model is provided, the method comprising: obtaining a speech sample set, the speech sample set comprising at least one speech sample, the speech sample comprising: an audio feature sequence and an initial word unit sequence; obtaining an initial speech recognition model, the speech recognition model being used to characterize the correspondence between the audio feature sequence and the prediction word unit sequence; replacing the language word units in the initial word unit sequence in the speech sample set with prediction word units representing the language to obtain a training sample set, the prediction word units being prediction word units in the prediction word unit sequence obtained by inputting speech samples selected from the speech sample set into the speech recognition model; training the speech recognition model based on the training sample set to obtain a trained speech recognition model.

According to a second aspect, a speech recognition model testing method is provided, the method comprising: obtaining a test set and a trained speech recognition model, the test set comprising at least one test sample, the test sample comprising: an audio feature sequence and a test word unit sequence; the trained speech recognition model is trained using the speech recognition model training method as described in any implementation of the first aspect, the speech recognition model comprising: an encoder, a decoder and a keyword module; selecting a test sample from the test set; inputting the audio feature sequence in the test sample into the encoder to obtain an audio intermediate feature; inputting the audio intermediate feature and the current word unit sequence into the decoder to obtain a predicted word unit output by the speech recognition model; in response to the predicted word unit being a non-terminal symbol, updating the current word unit sequence based on the predicted word unit, continuing to input the audio intermediate feature and the current word unit sequence into the decoder until the predicted word unit output by the speech recognition model is a terminator, and obtaining all predicted word units corresponding to the test sample; based on all predicted word units corresponding to the test sample and the test word unit sequence in the test sample, detecting whether the speech recognition model has passed the test.

According to a third aspect, a speech recognition method is provided, the method comprising: obtaining a speech to be recognized; processing the speech to be recognized to obtain audio feature data; inputting the audio feature data into a speech recognition model to obtain a predicted word unit sequence of the speech to be recognized, the speech recognition model being a trained speech recognition model obtained by using the speech recognition model training method described in any implementation method of the first aspect; obtaining text data of the speech to be recognized based on the predicted word unit sequence.

According to a fourth aspect, a speech recognition model training device is provided, which includes: a sample acquisition unit, configured to acquire a speech sample set, the speech sample set includes at least one speech sample, the speech sample includes: an audio feature sequence and an initial word unit sequence; a model acquisition unit, configured to acquire an initial speech recognition model, the speech recognition model is used to characterize the correspondence between the audio feature sequence and the prediction word unit sequence; a replacement unit, configured to replace the language word unit in the initial word unit sequence in the speech sample set with the prediction word unit representing the language, so as to obtain a training sample set, the prediction word unit is a prediction word unit in the prediction word unit sequence obtained by inputting the speech sample selected from the speech sample set into the speech recognition model; a training unit, configured to train the speech recognition model based on the training sample set, so as to obtain a trained speech recognition model.

According to the fifth aspect, a speech recognition model testing device is provided, the device comprising: an information acquisition unit, configured to acquire a test set and a trained speech recognition model, the test set comprising at least one test sample, the test sample comprising: an audio feature sequence and a test word unit sequence; the trained speech recognition model is trained by using the speech recognition model training device as described in any implementation of the fourth aspect, the speech recognition model comprising: an encoder, a decoder and a keyword module; a selection unit, configured to select a test sample from the test set; an input unit, configured to input the audio feature sequence in the test sample into the encoder to obtain an audio intermediate feature; a obtaining unit, configured to input the audio intermediate feature and the current word unit sequence into the decoder to obtain a predicted word unit output by the speech recognition model; an updating unit, configured to update the current word unit sequence based on the predicted word unit in response to the predicted word unit being a non-terminal symbol, and continue to input the audio intermediate feature and the current word unit sequence into the decoder until the predicted word unit output by the speech recognition model is a terminator, thereby obtaining all predicted word units corresponding to the test sample; a testing unit, configured to detect whether the speech recognition model has passed the test based on all predicted word units corresponding to the test sample and the test word unit sequence in the test sample.

According to the sixth aspect, a speech recognition device is provided, which includes: a speech acquisition unit, configured to acquire speech to be recognized; a processing unit, configured to process the speech to be recognized and obtain audio feature data; a recognition unit, configured to input the audio feature data into a speech recognition model to obtain a predicted word unit sequence of the speech to be recognized, and the speech recognition model is a trained speech recognition model obtained by the speech recognition model training device described in any implementation method of the fourth aspect; a conversion unit, configured to obtain text data of the speech to be recognized based on the predicted word unit sequence.

According to the seventh aspect, an electronic device is provided, which includes: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation of the first aspect, the second aspect, or the third aspect.

According to an eighth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect, the second aspect or the third aspect.

According to a ninth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the method described in any one of the implementations of the first aspect, the second aspect or the third aspect.

The speech recognition model training method and device provided by the embodiments of the present disclosure first obtain a speech sample set, the speech sample set includes at least one speech sample, and the speech sample includes: an audio feature sequence and an initial word unit sequence; secondly, obtain an initial speech recognition model, the speech recognition model is used to characterize the correspondence between the audio feature sequence and the prediction word unit sequence; thirdly, replace the language word unit in the initial word unit sequence in the speech sample set with the prediction word unit representing the language, and obtain a training sample set, the prediction word unit is the prediction word unit in the prediction word unit sequence obtained by inputting the speech sample selected from the speech sample set into the speech recognition model; finally, train the speech recognition model based on the training sample set to obtain a trained speech recognition model. Therefore, before training the speech recognition model, replacing the language word unit in the initial word unit sequence in the speech sample set with the prediction word unit representing the language can enable the model to predict the audio feature sequence in its familiar word unit environment, thereby improving the model convergence speed and the model training effect.

It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

FIG1 is a flow chart of an embodiment of a method for training a speech recognition model according to the present disclosure;

FIG2 is a schematic diagram of a structure of a keyword module training process disclosed herein;

FIG3 is a flow chart of an embodiment of a method for testing a speech recognition model according to the present disclosure;

FIG4 is a schematic diagram of a structure of a speech recognition model testing process disclosed herein;

FIG5 is a flow chart of an embodiment of a speech recognition method according to the present disclosure;

FIG6 is a schematic diagram of the structure of an embodiment of a speech recognition model training device according to the present disclosure;

FIG7 is a schematic diagram of the structure of an embodiment of a speech recognition model testing device according to the present disclosure;

FIG8 is a schematic diagram of the structure of an embodiment of a speech recognition device according to the present disclosure;

FIG9 is a block diagram of an electronic device used to implement the speech recognition model training method or the speech recognition model testing method or the speech recognition method of the embodiment of the present disclosure.

DETAILED DESCRIPTION

The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

Traditional speech recognition technology mainly realizes speech-to-text conversion based on acoustic models and language models. The role of the acoustic model is to build the relationship between speech and text. Common acoustic models include hidden Markov models and Gaussian mixture models. The role of the language model is to build the relationship between words in the text. The commonly used language model is the N-gram model. Because traditional speech recognition technology is limited by manually designed speech features and non-end-to-end recognition solutions, its recognition ability cannot be comparable to that of humans.

Speech recognition technology based on deep learning can automatically obtain useful feature representations from raw speech data and use complex network model structures to learn complex features and structures in speech signals. This makes speech recognition technology more accurate and robust when dealing with non-standard voices, accents, dialects, and voices of different speaking speeds.

With the widespread use of Transformer and other model structures in the field of speech recognition and the rapid development of computer hardware, speech recognition technology has made a qualitative leap. For widely used languages such as English and Chinese, speech recognition technology is already comparable to that of professionals; however, for languages that are not widely used, the recognition ability of language recognition models is still relatively weak.

Based on this, the present disclosure proposes a speech recognition model training method. FIG1 shows a process 100 according to an embodiment of the speech recognition model training method of the present disclosure. The speech recognition model training method includes the following steps:

Step 101: Obtain a speech sample set.

In this embodiment, the speech sample set is a collection of speech samples of the language to be recognized. For example, the language to be recognized is a language with few samples. By using a speech recognition model to recognize the speech of this language and obtain the text of this language, the dissemination, learning and research of this language can be improved.

In this embodiment, the execution subject on which the speech recognition model training method runs can obtain the speech sample set in a variety of ways. For example, the execution subject can obtain the speech sample set stored in the database server through a wired connection or a wireless connection. For another example, the user can obtain the speech sample set collected by the terminal by communicating with the terminal.

Here, the speech sample set may include at least one speech sample, and the speech sample includes: an audio feature sequence and an initial word unit sequence; wherein the audio feature sequence is a sequence obtained after feature extraction of the audio, such as Mel features, Fbank (FilterBank) features, etc.

In this embodiment, the initial word unit sequence, also called token sequence, is obtained by dividing the text corresponding to the audio feature sequence into language units such as words, punctuation marks, numbers or pure letters, and the initial unit sequence contains word units of the language representing the text corresponding to the audio feature sequence.

In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of the voice sample sets involved are carried out after authorization and comply with relevant laws and regulations.

Step 102, obtaining an initial speech recognition model.

In this embodiment, the speech recognition model is used to characterize the correspondence between the audio feature sequence and the predicted word unit sequence. Specifically, the speech recognition model can be a Whisper model, which is a standard transformer model. The Whisper model takes a fixed length (30 seconds) of audio as input, first inputs the Fbank features of the extracted audio into the encoder of the Whisper model to generate the intermediate features of the audio, and then uses the decoder of the Whisper model to decode the intermediate features and output the predicted word unit sequence predicted by the model.

Optionally, the speech recognition model can also be an MMS (Massively Multilingual Speech) model. The MMS model can recognize more than 4,000 spoken languages. Compared with the Whisper model, the MMS model has a larger number of model parameters, more pre-training data, and more powerful model functions. However, the MMS model has a huge number of parameters and the deployment cost is too high.

Step 103, using the predicted word unit representing the language to replace the language word unit in the initial word unit sequence in the speech sample set to obtain a training sample set.

In this embodiment, the predicted word unit is a predicted word unit in a predicted word unit sequence obtained by inputting a speech sample selected from a speech sample set into a speech recognition model.

In this embodiment, a speech sample is selected from the speech sample, and the audio feature sequence of the speech sample is input into the speech recognition model, so that a predicted word unit sequence output by the speech recognition model can be obtained, wherein one predicted word unit in the predicted word unit sequence is a word unit representing a language, and the predicted word unit representing the language in the predicted word unit sequence replaces the language word unit in the initial word unit sequence in the speech sample set, so that the speech recognition model can be more familiar with the training sample set.

In this embodiment, the predicted word unit representing the language replaces the language word unit in the initial word unit sequence to obtain a new word unit sequence, and a training sample set including the audio feature sequence and the new word unit sequence can be obtained.

In this embodiment, the training samples include: an audio feature sequence X = (x ₁ , x ₂ , …x _k ) and its corresponding token sequence Label = (l ₁ , l ₂ , …l _m ). For recognition tasks of languages with few samples, Label records the text information of the language corresponding to the current audio. The speech recognition model takes the audio sequence X as input and outputs the predicted token sequence Output = (o ₁ , o ₂ , …o _m ). During the training process, the cross entropy loss function is generally used to measure the difference between Label and Output, so that the Output is as consistent as possible with the Label. During the prediction process, the text of the language predicted by the model can be obtained through the output of the speech recognition model.

The speech recognition model can be divided into a non-autoregressive model and a non-autoregressive model according to the way the Output is formed. The non-autoregressive model takes the audio sequence X as the output and decodes the Output in parallel; the autoregressive model takes the audio sequence X as the output and decodes the Output in serial. The non-autoregressive model decodes in parallel and has a fast decoding speed, but it is easy to ignore the contextual relationship between the audio sequences, and the model effect is poor; the autoregressive model decodes in serial and has a slow decoding speed, but it can construct the before and after relationship between the audio sequences, and the model effect is better. Since there are fewer samples of this type of language and the training difficulty is slightly greater, in order to ensure the effect of the speech recognition model, the present disclosure adopts the decoding method of the autoregressive model.

In this embodiment, the predicted word unit sequence output by the speech recognition model is an array, such as [223, 2124, 2323, 45], where each number represents a different word, that is, the Output above. In order to improve the training effect of the speech recognition model, during the training of the speech recognition model, the Output can be obtained all at once (non-autoregressive), and when the speech recognition model is tested, the predicted word units in the Output can be obtained one by one (autoregressive).

Step 104: train the speech recognition model based on the training sample set to obtain a trained speech recognition model.

In this embodiment, the execution subject can select training samples from the training sample set obtained in step 103, and perform the training steps from step A to step B below to complete an iterative training of the speech recognition model. Among them, the selection method and the number of training samples selected from the training sample set are not limited in this application, and the number of iterative training of the speech recognition model is not limited.

Step A: Based on the training samples input into the speech recognition model, the network loss value of the speech recognition model is calculated.

In this embodiment, during each iterative training of the speech recognition model, training samples are selected from the training sample set, and the selected training samples are input into the speech recognition model. Based on the predicted word unit sequence output by the speech recognition model, the loss value of the speech recognition model is calculated.

In this embodiment, the loss function of the speech recognition model for calculating the loss value of the speech recognition model can adopt a cross-entropy loss function, which can measure the degree of difference between two different probability distributions in the same random variable, and is represented as the difference between the true probability distribution and the predicted probability distribution in machine learning. The smaller the value of the cross-entropy loss function, the better the prediction effect of the speech recognition model.

Optionally, the loss function of the speech recognition model can also adopt the mean square error function, which is the expectation of the square of the difference between the predicted word unit sequence of the speech recognition model and the true value (the new word unit sequence in the training sample). During the iterative training process of the speech recognition model, the gradient descent algorithm can be used to minimize the loss function of the speech recognition model, thereby iteratively optimizing the network parameters of the speech recognition model.

The original meaning of gradient is a vector, which means that the directional derivative of a loss function at that point reaches the maximum value along that direction, that is, the loss function changes fastest along that direction at that point and the rate of change is the largest. In deep learning, the main task of neural networks is to find the optimal network parameters (weights and biases) during learning. The optimal network parameters are also the parameters when the loss function is minimized.

Step B. Based on the loss value of the speech recognition model, the speech recognition model is trained to obtain a trained speech recognition model.

In this embodiment, the trained speech recognition model is obtained by inputting the selected speech samples into the speech recognition model for iterative training and adjusting the parameters of the speech recognition model.

In this embodiment, the loss value of the speech recognition model can be used to detect whether the speech recognition model meets the training completion condition. After the speech recognition model meets the training completion condition, a trained speech recognition model is obtained.

In this embodiment, the training completion condition includes: the loss value of the speech recognition model is less than a first loss value threshold. The first loss threshold can be determined based on specific training requirements, for example, the first loss threshold is 0.01.

Optionally, in this embodiment, in response to the speech recognition model not meeting the training completion conditions, the relevant parameters in the speech recognition model are adjusted so that the loss value of the speech recognition model converges, and based on the adjusted speech recognition model, the above-mentioned training steps A-B are continued.

In this embodiment, when the speech recognition model does not meet the training completion conditions, adjusting the relevant parameters of the speech recognition model helps to help the loss value of the speech recognition model converge.

In this embodiment, for the Whisper model that recognizes languages with few samples, since the token sequence (word unit sequence) received by the Whisper model must contain the token (word unit) of the language, it is determined which type of language the model performs speech recognition tasks. However, since the original Whisper model has not been trained for data of this type of language, its token sequence lacks the token of this type of language. For this reason, the speech recognition model has been trained many times, and attempts have been made to replace the token of this type of language with tokens of other languages such as Chinese and English; or to assign a new language token to this type of language, but the effect of the model after training has not met expectations. The Whisper model is used to perform a language category test on the audio of this type of language, and it is found that the Whisper model predicts most of the audio of this type of language as another type of language (predicted word unit representing speech), and replacing the token of this type of language with the token of another type of language can achieve good training results.

In this embodiment, the above-mentioned loss value based on the speech recognition model is used to train the speech recognition model to obtain the trained speech recognition model, including: in response to the loss value of the speech recognition model being less than the loss threshold, determining that the speech recognition model meets the training completion condition, and using the feature recognition subnetwork and classification subnetwork in the speech recognition model that meets the training completion condition as the speech recognition model.

The speech recognition model training method provided by the embodiment of the present disclosure first obtains a speech sample set, the speech sample set includes at least one speech sample, and the speech sample includes: an audio feature sequence and an initial word unit sequence; secondly, obtains an initial speech recognition model, the speech recognition model is used to characterize the correspondence between the audio feature sequence and the prediction word unit sequence; thirdly, replaces the language word unit in the initial word unit sequence in the speech sample set with the prediction word unit representing the language, and obtains a training sample set, the prediction word unit is the prediction word unit in the prediction word unit sequence obtained by inputting the speech sample selected from the speech sample set into the speech recognition model; finally, based on the training sample set, trains the speech recognition model to obtain a trained speech recognition model. Therefore, before training the speech recognition model, replacing the language word unit in the initial word unit sequence in the speech sample set with the prediction word unit representing the language can enable the model to predict the audio feature sequence in its familiar word unit environment, thereby improving the model convergence speed and the model training effect.

In some optional implementations of the present disclosure, the above-mentioned speech recognition model includes: an initial recognition sub-model, and a keyword module connected to the initial recognition sub-model; based on the training sample set, the speech recognition model is trained to obtain a trained speech recognition model including: based on the training sample set, the initial recognition sub-model is trained to obtain a trained recognition sub-model; based on the training sample set and the trained recognition sub-model, a keyword module is trained to obtain a trained keyword module; the trained recognition sub-model and the trained keyword module are used as the trained speech recognition model.

As shown in FIG2 , the speech recognition model includes: a recognition sub-model and a keyword module. The training samples in the training sample set are input into the recognition sub-model, and the recognition sub-model is trained multiple times to obtain a trained recognition sub-model.

The method for training a speech recognition model provided by this optional implementation first trains an initial recognition sub-model to obtain a trained recognition sub-model; then, based on a training sample set and the trained recognition sub-model, a keyword model is trained to obtain a trained keyword module, and the initial recognition sub-model and the keyword module are trained step by step, thereby improving the reliability of the trained speech recognition model.

In some optional implementations of the present disclosure, the above-mentioned training of the keyword module based on the training sample set and the trained recognition sub-model to obtain the trained keyword module includes: fixing the parameters of the trained recognition sub-model; inputting the training samples in the training sample set into the trained recognition sub-model to obtain a first probability distribution output by the trained recognition sub-model and a second probability distribution output by the keyword module; calculating the loss value of the initial word unit in the training sample based on the first probability distribution and the second probability distribution; and obtaining the trained keyword module based on the loss value.

In this optional implementation, as shown in FIG2 , the recognition sub-model is a Whisper model. In order to improve the recognition ability of the model for specific keywords (bias words), the keyword module of the present disclosure can adopt a TCPGen (Tree-constrained pointer generator) module. The TCPGen module can help the Whisper model (the trained recognition sub-model at this time) improve its recognition ability for keywords. First, a keyword sequence (biasing list) that needs to be recognized is generated. Then, the TCPGen module generates a keyword prefix tree based on the keyword sequence, and generates a probability distribution P ^ptr (y _i ) and a generation probability P ^gen (y _i ) for all tokens by traversing the tree. Specifically, the TCPGen module first obtains the token sequence corresponding to the keyword by traversing the keyword prefix tree, and then obtains the embedding corresponding to each token sequence, and finally uses the fully connected layer and the softmax function to generate the probability distribution P ^ptr (y _i ) and the generation probability P ^gen (y _i ). Finally, the two will be weighted and summed with the token probability distribution ^Pmdl (y _i ) generated by the Whisper model (although the parameters of the Whisper model are fixed, the Whisper model can also perform forward prediction) to obtain the final token probability distribution P(y _i ), specifically, as shown in formula (1):

P(y _i )=P ^mdl (y _i )(1-P ^gen (y _i ))+P ^ptr (y _i )P ^gen (y _i ) (1)

In this optional implementation, the keyword model is trained on the basis of the completion of the recognition sub-model training, which can ensure the training effect of the keyword model. Specifically, the Whisper model needs to use the technology of shifting the word unit sequence right by one position during training, but through training, it is found that the technology of shifting the word unit sequence right by one position should not be used when training the TCPGen module alone, otherwise the training loss will oscillate, resulting in the inability to train the model. This is because the TCPGen module does not directly predict the token sequence, but makes corrections on the prediction results of the original Whisper model. For this reason, after the Whisper model training is completed, the TCPGen module is trained without using the technology of shifting the word unit sequence right by one position, which can improve the training effect of the TCPGen module.

The method for training a keyword module provided in this embodiment first fixes the parameters of the trained recognition sub-model; then, based on the training sample set, calculates the first probability score output by the trained recognition sub-model and the second probability distribution output by the keyword module; based on the first probability distribution and the second probability distribution, calculates the loss value of the initial word unit in the training sample, and based on the loss value, obtains the trained keyword module, thereby improving the reliability of keyword module training.

In some optional implementations of the present disclosure, the above-mentioned training of the initial recognition sub-model based on the training sample set to obtain the trained recognition sub-model includes: selecting training samples from the training sample set to obtain selected samples; shifting the new word units in the new word unit sequence in the selected samples right by one position to obtain a training word unit sequence; training the initial recognition sub-model based on the audio feature sequence and the training word unit sequence in the selected samples to obtain the trained recognition sub-model.

In this optional implementation, the new word unit sequence is a word unit sequence obtained by replacing the language word units in the initial word unit sequence in the speech sample set with predicted word units representing the language, and the new word units are word units in the new word unit sequence.

In this optional implementation, the recognition sub-model can be a Whisper model, and when training the whisper model, in order to adapt to the characteristics of the Whisper autoregressive model, the word unit sequence is shifted right by one position (shift_tokens_right) during the training process, that is, the model needs to predict the next word in the token sequence. To achieve this, the token sequence input to the model needs to be shifted right by one unit. However, through training, it was found that the word unit sequence shifted right by one position should not be used when training TCPGen alone, otherwise the training loss will oscillate, resulting in the inability to train the model. This is because the TCPGen module does not directly predict the token sequence, but makes corrections based on the prediction results of the original Whisper model.

In this optional implementation, the new word unit in the new word unit sequence in the selected sample is shifted right by one position in order to enable the initial recognition sub-model to effectively predict the next Token.

The method for training an initial recognition sub-model provided in this embodiment selects a training sample from a training sample set, shifts the new word unit in the new word unit sequence in the selected sample right by one position to obtain a training word unit sequence, and trains the initial recognition sub-model based on the audio feature sequence and the training word unit sequence in the selected sample, thereby improving the reliability of the trained recognition sub-model and improving the training effect of the speech recognition model.

In some optional implementations of the present disclosure, the above-mentioned acquisition of a speech sample set includes: acquiring an initial data set including at least one initial speech; preprocessing the initial data set to obtain a processed data set; performing data enhancement on the processed data set to obtain a speech data set; and obtaining a speech sample set based on the speech data set.

In this optional implementation, the initial speech is speech data of a language, and the initial speech may be unprocessed speech data. The speech sample set is obtained by processing the initial speech in the initial data set. For example, if the initial speech is speech of a rare language, the obtained speech sample set is a sample set of the rare language, and the speech recognition model can be trained by the sample set.

In this optional implementation, data preprocessing is the process of preparing raw data and making it suitable for a machine learning model. Data preprocessing can make the preprocessed processed data set suitable for training a machine learning model. The above-mentioned preprocessing of the initial data set to obtain the processed data set includes: removing unrecognizable information such as punctuation marks in the initial data.

In this optional implementation, based on the problem that the sample information amount of the language corresponding to the initial data set is small, data enhancement is performed on the processed data set, thereby increasing the amount of speech data in the speech data set.

In this optional implementation, the above-mentioned voice sample set based on the voice data set includes: extracting the voice features of each voice data in the voice data set to obtain the voice features of each voice data; limiting each voice feature by a fixed length to obtain the audio features of each voice feature, determining the initial word unit of each voice data, and obtaining a voice sample set including at least one voice sample. Among them, the voice feature can be a Mel feature or an Fbank (Filter Banks) feature, and the extraction of the voice feature can adopt mature means, which will not be repeated here.

The method for obtaining a speech sample set provided by this optional implementation first preprocesses an initial data set to obtain a processed data set; then performs data enhancement on the processed data set to obtain a speech data set; based on the speech data set, a speech sample set is obtained, thereby improving the reliability of obtaining the speech sample set.

In some optional implementations of the present disclosure, the above-mentioned preprocessing of the initial data set to obtain a processed data set includes at least one of the following: sampling all initial data in the initial data set at the same sampling rate; deleting repeated and unrecognizable initial data in the initial data set.

Optionally, the preprocessing of the initial data set to obtain the processed data set may further include: processing all initial data in the initial data set so that all the initial data are located in the same audio channel.

Optionally, the above-mentioned preprocessing of the initial data set to obtain the processed data set may also include: using speech synthesis technology to generate new initial data, adding the new initial data to the processed data set, and increasing the size of the data set by supplementing the synthesized audio of the language as a training sample due to the small number of training samples for the language with few samples. Through training, it is found that the synthesized data set should not be too large, otherwise it may have side effects on the training effect, mainly because the quality of the synthesized data is difficult to guarantee. For this reason, the amount of new initial data can be kept at about 30% of the total amount of the speech sample set.

This optional implementation provides a method for preprocessing an initial data set, including sampling the initial data set at the same sampling rate and deleting repeated and unrecognizable initial data in the initial data set, thereby improving the diversity of speech preprocessing.

In some optional implementations of the present disclosure, the above-mentioned data enhancement of the processed data set to obtain a speech data set includes at least one of the following: modifying the speaking speed of each processed data in the processed data set to obtain an increased data set, and adding the increased data set to the processed data set; adding reverberation to each processed data in the processed data set to obtain an reverberation data set, and adding an reverberation data set to the processed data set; adding noise to each processed data in the processed data set to obtain a noise data set, and adding a noise data set to the processed data set; modifying the audio spectrum graph of each processed data in the processed data set to obtain a spectrum data set, and adding a spectrum data set to the processed data set.

In this optional implementation, the speaking speed of the processed data (audio) in the processed data set is modified, the number of training sets is increased, and the data is made more diverse; reverberation is randomly added to the processed data in the processed data set to achieve the purpose of simulating a variety of sound scenes; noise is randomly added to the processed data in the processed data set to increase data complexity; Specaugment technology (a technology used for speech recognition enhancement) can be used to modify the audio spectrum graph of the processed data in the processed data set to improve the quality of audio features.

The method for data enhancement of processed data provided by this optional implementation includes: modifying the speaking speed of processed data in the processed data, adding reverberation to the processed data, adding noise to the processed data, modifying at least one or more of the audio spectrum graphs of the processed data, and improving the multi-sample nature of the speech data set.

Furthermore, based on the speech recognition model training method provided in the above-mentioned embodiment, the present disclosure also provides an embodiment of a speech recognition model testing method. The speech recognition model testing method of the present disclosure combines artificial intelligence fields such as speech recognition and deep learning.

3 , a process 300 of an embodiment of a method for testing a speech recognition model according to the present disclosure is shown. The method for testing a speech recognition model provided in this embodiment includes the following steps:

Step 301, obtaining a test set and a trained speech recognition model.

In this embodiment, the test set includes at least one test sample, and the test sample includes: an audio feature sequence and a test word unit sequence.

In this embodiment, the test sample is a sample with the same data structure as the training sample, that is, the test sample includes: an audio feature sequence and a new word unit sequence, the new word unit sequence is a sequence in which the predicted word units representing the language replace the language word units in the initial word unit sequence, and the new word unit sequence includes at least one new word unit, but the test sample has not been used as a training sample to train the speech recognition model.

In this embodiment, the speech recognition model can be a trained speech recognition model obtained by training using the method described in the embodiment of Figure 1 above. The specific training process can be found in the relevant description of the embodiment of Figure 3, which will not be repeated here.

In this embodiment, the speech recognition model includes: an encoder, a decoder and a keyword module, wherein the encoder and the decoder constitute a trained recognition sub-model.

Step 302: Select test samples from the test set.

In this embodiment, the execution entity may select a test sample from the test set in step 301.

In this embodiment, the selection method and number of test samples from the test set are not limited in this application, and the number of iterative training of the speech recognition model is not limited.

Step 303: Input the audio feature sequence in the test sample into the encoder to obtain the audio intermediate feature.

In this embodiment, as shown in Figure 4, the audio feature sequence can be the audio Mel features of the extracted audio, and the audio Mel features are input into the encoder of the trained recognition sub-model to obtain the audio intermediate features generated by the encoder; when the decoder predicts for the first time, the generated starting token sequence (not shown in Figure 4) is used as the current word unit sequence. As the test time goes by, after multiple predictions, the current word unit sequence is replaced by the current word unit sequence updated based on the predicted word unit.

Step 304, input the audio intermediate features and the current word unit sequence into the decoder to obtain the predicted word unit output by the speech recognition model.

In this embodiment, when the decoder makes the first prediction, the intermediate audio features and the starting token sequence are sent to the decoder for decoding to obtain the predicted token output by the decoder. It should be noted that among all the predicted word units output by the decoder, one predicted word unit can be a predicted word unit representing the language.

Step 305, in response to the predicted word unit being a non-terminal symbol, the current word unit sequence is updated based on the predicted word unit, and the audio intermediate features and the current word unit sequence continue to be input into the decoder until the predicted word unit output by the speech recognition model is a terminator, thereby obtaining all predicted word units corresponding to the test sample.

As shown in Figure 4, EOT represents the end symbol. When the predicted word unit is not an end symbol, that is, Not EOT, the current word unit sequence is updated based on the predicted word unit of the decoder. Specifically, updating the current word unit sequence can be adding the predicted token (predicted word unit) to the end of the current token sequence. It should be noted that when the decoder predicts for the first time, the predicted word unit output by the decoder is added after the starting token sequence as the result of updating the current word unit sequence. The result is used as the current word unit sequence, and the next prediction of the decoder is performed. As shown in Figure 4, when the predicted word unit is an end symbol, the prediction of the audio feature sequence is terminated.

In this embodiment, after the decoder performs multiple predictions, if the predicted word unit output by the decoder is EOT, the decoder ends the prediction and combines the predicted word units output by the decoder from the first output to before outputting EOT to obtain all predicted word units.

Step 306: Based on all the predicted word units corresponding to the test sample and the test word unit sequence in the test sample, detect whether the speech recognition model has passed the test.

In this embodiment, all predicted word units can be arranged together according to the prediction order of the decoder to obtain a predicted word unit sequence, and the predicted word unit sequence can be compared with the test word unit sequence in the test sample; in response to the predicted word unit sequence being the same as the test word unit sequence, it is determined that the speech recognition model has passed the test, and the speech recognition model can be directly applied to the prediction of the text of the language; in response to the predicted word unit sequence being different from the test word unit sequence, it is determined that the speech recognition model has failed the test, and the speech recognition model cannot be directly applied to the prediction of the text of the language.

The speech recognition model testing method provided by the embodiment of the present disclosure obtains a test set and a speech recognition model, selects a test sample from the test set, inputs the audio feature sequence in the test sample into the encoder, and always uses the update unit sequence to replace it when the speech recognition model does not output the end symbol. As a result, the speech recognition model will always predict the audio intermediate features of the test sample to obtain the predicted word unit sequence of the audio intermediate features, and by detecting whether the speech recognition model is qualified by the predicted word unit sequence, the reliability of the speech recognition model test is improved.

In some optional implementations of the present disclosure, detecting whether a speech recognition model has passed the test based on all predicted word units corresponding to the test sample and the test word unit sequence in the test sample includes: sorting all predicted word units corresponding to the test sample to obtain a predicted word unit sequence; calculating the word error rate of the test sample based on the predicted word unit sequence and the test word unit sequence in the test sample; and determining that the speech recognition model is qualified in response to the word error rate being less than an error threshold.

In this optional implementation, the predicted word unit sequence is compared with the test word unit sequence, the number of word units in the predicted word unit sequence that are different from the test word unit sequence is determined, and the ratio of the number of word units to the total number of test word units in the test word unit sequence is calculated to obtain the word error rate.

In this optional implementation, the error threshold can be determined based on the test requirements. For example, the error threshold is 20%, that is, less than 20% of the predicted word units in the predicted word unit sequence are different from the test word units in the test word unit sequence, and the speech recognition model is determined to be qualified.

This optional implementation provides a method for detecting whether a speech recognition model has passed a test. By calculating the word error rate of a test sample through the predicted word unit sequence and the test word unit sequence corresponding to the test sample, it provides a reliable implementation method for testing the speech recognition model and improves the reliability of the speech recognition model test.

Furthermore, based on the speech recognition model training method provided in the above embodiment, the present disclosure also provides an embodiment of a speech recognition method. The speech recognition method of the present disclosure combines artificial intelligence fields such as speech recognition and deep learning.

5 , a process 500 according to an embodiment of the speech recognition method of the present disclosure is shown. The speech recognition method provided in this embodiment includes the following steps:

Step 501: Acquire speech to be recognized.

In this embodiment, the execution subject of the speech recognition method can obtain the speech to be recognized in a variety of ways. For example, the execution subject can obtain the speech to be recognized stored in the database server through a wired connection or a wireless connection. For another example, the execution subject can also receive the speech to be recognized collected by a terminal or other device in real time.

In this embodiment, the speech to be recognized is speech information that needs to be converted from speech to text. The speech to be recognized may be speech in a scarce language, for example, the speech to be recognized is speech in a language with few samples.

Step 502: Process the speech to be recognized to obtain audio feature data.

In this embodiment, the execution entity may process the to-be-recognized speech obtained in step 501 to obtain audio feature data.

In this embodiment, the audio feature data is data obtained after the feature (e.g., Mel feature or Fbank feature) of the speech to be recognized is performed. Specifically, the above step 502 includes: fixing the length of the speech to be recognized (e.g., 30 seconds), extracting the Mel or fbank features of the speech to be recognized under the length, and obtaining the audio feature data of fixed length.

In this embodiment, the audio feature data has the same data structure as the audio feature sequence.

Step 503: input the audio feature data into the speech recognition model to obtain a predicted word unit sequence of the speech to be recognized.

In this embodiment, the speech recognition model can be a trained speech recognition model obtained by training using the method described in the embodiment of Figure 1 above. The specific training process can refer to the relevant description of the embodiment of Figure 1, which will not be repeated here.

Step 504: obtaining text data of the speech to be recognized based on the predicted word unit sequence.

In this embodiment, restoring the single sequence to text data is a mature technical means, and converting the predicted word unit sequence into text data of the speech to be recognized is also a mature means, which will not be described in detail here.

The speech recognition method provided by the present disclosure can be applied to social, learning, work and other scenarios of languages with few samples. First, it can be applied to areas where this type of language is the main language to help people interact with smart devices more conveniently. Second, the model can be used in the field of education to assist students in learning the pronunciation and listening of this type of language. In addition, in the business field, it can be used for customer service and market research to meet the needs of users of this type of language.

The speech recognition method provided by the embodiment of the present disclosure obtains the speech to be recognized, processes the speech to be recognized, obtains audio feature data, inputs the audio feature data into the speech recognition model generated by the speech recognition model training method, obtains the predicted word unit sequence of the audio feature data, and obtains the text data of the speech to be recognized by predicting the word unit sequence. Thus, the speech recognition model is used to generate the speech recognition result, which improves the reliability and accuracy of speech recognition.

Further referring to FIG. 6 , as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a speech recognition model training device. The device embodiment corresponds to the method embodiment shown in FIG. 1 , and the device can be specifically applied to various electronic devices.

As shown in FIG6 , the speech recognition model training device 600 provided in this embodiment includes: a sample acquisition unit 601, a model acquisition unit 602, a replacement unit 603, and a training unit 604. The sample acquisition unit 601 can be configured to acquire a speech sample set, the speech sample set includes at least one speech sample, and the speech sample includes: an audio feature sequence and an initial word unit sequence. The model acquisition unit 602 can be configured to acquire an initial speech recognition model, and the speech recognition model is used to characterize the correspondence between the audio feature sequence and the prediction word unit sequence. The replacement unit 603 can be configured to replace the language word unit in the initial word unit sequence in the speech sample set with the prediction word unit characterizing the language, and obtain the training sample set, and the prediction word unit is the prediction word unit in the prediction word unit sequence obtained by inputting the speech sample selected from the speech sample set into the speech recognition model. The training unit 604 can be configured to train the speech recognition model based on the training sample set to obtain the trained speech recognition model.

In this embodiment, the specific processing of the sample acquisition unit 601, the model acquisition unit 602, the replacement unit 603, and the training unit 604 in the speech recognition model training device 600 and the technical effects brought about by them can be respectively referred to the relevant descriptions of step 101, step 102, step 103, and step 104 in the corresponding embodiment of Figure 1, and will not be repeated here.

In some optional implementations of the present embodiment, the above-mentioned speech recognition model includes: an initial recognition sub-model, and a keyword module connected to the initial recognition sub-model; the above-mentioned training unit 604 is configured to: train the initial recognition sub-model based on the training sample set to obtain a trained recognition sub-model; train the keyword module based on the training sample set and the trained recognition sub-model to obtain a trained keyword module; and use the trained recognition sub-model and the trained keyword module as the trained speech recognition model.

In some optional implementations of the present embodiment, the training unit 604 is further configured to: fix the parameters of the trained recognition sub-model; input the training samples in the training sample set into the trained recognition sub-model to obtain a first probability distribution output by the trained recognition sub-model and a second probability distribution output by the keyword module; calculate the loss value of the initial word unit in the training sample based on the first probability distribution and the second probability distribution; and obtain the trained keyword module based on the loss value.

In some optional implementations of the present embodiment, the training unit 604 is further configured to: select a training sample from a training sample set to obtain a selected sample; shift the new word unit in the new word unit sequence in the selected sample right by one position to obtain a training word unit sequence; train an initial recognition sub-model based on the audio feature sequence and the training word unit sequence in the selected sample to obtain a trained recognition sub-model.

In some optional implementations of this embodiment, the sample acquisition unit 601 is configured to: acquire an initial data set including at least one initial speech; preprocess the initial data set to obtain a processed data set; perform data enhancement on the processed data set to obtain a speech data set; and obtain a speech sample set based on the speech data set.

In some optional implementations of this embodiment, the sample acquisition unit 601 is further configured to implement at least one of the following: sampling all initial data in the initial data set at the same sampling rate; deleting repeated and unrecognizable initial data in the initial data set.

In some optional implementations of this embodiment, the sample acquisition unit 601 is further configured to implement at least one of the following: modifying the speech rate of each processed data in the processed data set to obtain an added data set, and adding the added data set to the processed data set; adding reverberation to each processed data in the processed data set to obtain an reverberation data set, and adding an reverberation data set to the processed data set; adding noise to each processed data in the processed data set to obtain a noise data set, and adding a noise data set to the processed data set; modifying the audio spectrum graph of each processed data in the processed data set to obtain a spectrum data set, and adding a spectrum data set to the processed data set.

The speech recognition model training device provided by the embodiment of the present disclosure, first, the sample acquisition unit 601 acquires a speech sample set, the speech sample set includes at least one speech sample, and the speech sample includes: an audio feature sequence and an initial word unit sequence; secondly, the model acquisition unit 602 acquires an initial speech recognition model, and the speech recognition model is used to characterize the correspondence between the audio feature sequence and the prediction word unit sequence; thirdly, the replacement unit 603 replaces the language word unit in the initial word unit sequence in the speech sample set with the prediction word unit representing the language, and obtains a training sample set, and the prediction word unit is a prediction word unit in the prediction word unit sequence obtained by inputting the speech sample selected from the speech sample set into the speech recognition model; finally, the training unit 604 trains the speech recognition model based on the training sample set to obtain a trained speech recognition model. Therefore, before training the speech recognition model, replacing the language word unit in the initial word unit sequence in the speech sample set with the prediction word unit representing the language can enable the model to predict the audio feature sequence in its familiar word unit environment, thereby improving the model convergence speed and the model training effect.

Further referring to FIG. 7 , as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a speech recognition model testing device, which corresponds to the method embodiment shown in FIG. 3 , and can be specifically applied to various electronic devices.

As shown in FIG7 , the speech recognition model testing device 700 provided in this embodiment includes: an information acquisition unit 701, a selection unit 702, an input unit 703, a obtaining unit 704, an updating unit 705, and a testing unit 706. The information acquisition unit 701 can be configured to obtain a test set and a trained speech recognition model, wherein the test set includes at least one test sample, and the test sample includes: an audio feature sequence and a test word unit sequence; the trained speech recognition model is obtained by training using the speech recognition model training device, and the speech recognition model includes: an encoder, a decoder, and a keyword module. The selection unit 702 can be configured to select a test sample from the test set. The input unit 703 can be configured to input the audio feature sequence in the test sample into the encoder to obtain an audio intermediate feature. The obtaining unit 704 can be configured to input the audio intermediate feature and the current word unit sequence into the decoder to obtain a predicted word unit output by the speech recognition model. The updating unit 705 can be configured to update the current word unit sequence based on the predicted word unit in response to the predicted word unit being a non-terminal symbol, and continue to input the audio intermediate features and the current word unit sequence into the decoder until the predicted word unit output by the speech recognition model is a terminal symbol, thereby obtaining all the predicted word units corresponding to the test sample. The testing unit 706 can be configured to detect whether the speech recognition model has passed the test based on all the predicted word units corresponding to the test sample and the test word unit sequence in the test sample.

In this embodiment, in the speech recognition device 700, the specific processing of the information acquisition unit 701, the selection unit 702, the input unit 703, the obtaining unit 704, the updating unit 705, and the testing unit 706 and the technical effects brought about by them can be respectively referred to the relevant descriptions of step 301, step 302, step 304, step 305, and step 306 in the corresponding embodiment of Figure 3, and will not be repeated here.

In some optional implementations of the present embodiment, the above-mentioned test unit 706 is further configured to: sort all predicted word units corresponding to the test sample to obtain a predicted word unit sequence; calculate the word error rate of the test sample based on the predicted word unit sequence and the test word unit sequence in the test sample; and determine that the speech recognition model is qualified in response to the word error rate being less than an error threshold.

Further referring to FIG. 8 , as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a speech recognition device, which corresponds to the method embodiment shown in FIG. 5 , and can be specifically applied to various electronic devices.

As shown in FIG8 , the speech recognition device 800 provided in this embodiment includes: a speech acquisition unit 801, a processing unit 802, a recognition unit 803, and a conversion unit 804. Among them, the above-mentioned speech acquisition unit 801 can be configured to acquire the speech to be recognized. The above-mentioned processing unit 802 can be configured to process the speech to be recognized and obtain audio feature data. The above-mentioned recognition unit 803 can be configured to input the audio feature data into the speech recognition model to obtain the predicted word unit sequence of the speech to be recognized, and the speech recognition model is the trained speech recognition model obtained by the speech recognition model training device of the above-mentioned embodiment. The above-mentioned conversion unit 804 can be configured to obtain the text data of the speech to be recognized based on the predicted word unit sequence.

In this embodiment, the specific processing of the speech acquisition unit 801, the processing unit 802, the recognition unit 803, and the conversion unit 804 in the speech recognition device 800 and the technical effects brought about by them can be respectively referred to the relevant descriptions of step 501, step 502, step 503, and step 504 in the corresponding embodiment of Figure 5, and will not be repeated here.

In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

FIG9 shows a schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and/or required herein.

As shown in Figure 9, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input/output (I/O) interface 905 is also connected to the bus 904.

A number of components in the device 900 are connected to the I/O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

The computing unit 901 may be a variety of general and/or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a speech recognition model training method or a speech recognition model testing method or a speech recognition method. For example, in some embodiments, the speech recognition model training method or the speech recognition model testing method or the speech recognition method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and/or installed on the device 900 via the ROM 902 and/or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the speech recognition model training method or the speech recognition model testing method or the speech recognition method described above may be executed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute a speech recognition model training method or a speech recognition model testing method or a speech recognition method in any other suitable manner (for example, by means of firmware).

Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and/or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable speech recognition model training device or a speech recognition model testing device or a speech recognition device, so that the program code, when executed by the processor or controller, enables the functions/operations specified in the flow chart and/or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims (23)

1. A method of training a speech recognition model, the method comprising:

Obtaining a set of speech samples, the set of speech samples comprising at least one speech sample, the speech sample comprising: an audio feature sequence and an initial word unit sequence;

acquiring an initial voice recognition model, wherein the voice recognition model is used for representing the corresponding relation between an audio characteristic sequence and a predicted word unit sequence;

Replacing language word units in the initial word unit sequence in the voice sample set by using the predicted word units of the characterization language to obtain a training sample set, wherein the predicted word units are predicted word units in the predicted word unit sequence obtained by inputting voice samples selected from the voice sample set into the voice recognition model;

And training the voice recognition model based on the training sample set to obtain a trained voice recognition model.

2. The method of claim 1, wherein the speech recognition model comprises: the system comprises an initial recognition sub-model and a keyword module connected with the initial recognition sub-model; training the voice recognition model based on the training sample set, wherein the obtaining the trained voice recognition model comprises the following steps:

training the initial recognition sub-model based on the training sample set to obtain a trained recognition sub-model;

Training the keyword module based on the training sample set and the trained recognition sub-model to obtain a trained keyword module;

and taking the trained recognition sub-model and the trained keyword module as a trained voice recognition model.

3. The method of claim 2, wherein the training the keyword module based on the training sample set, the trained recognition sub-model, the obtaining the trained keyword module comprises:

Fixing parameters of the trained recognition sub-model;

Inputting training samples in a training sample set into the trained recognition sub-model to obtain a first probability distribution output by the trained recognition sub-model and a second probability distribution output by the keyword module;

calculating a loss value of an initial word unit in a training sample based on the first probability distribution and the second probability distribution;

And obtaining a trained keyword module based on the loss value.

4. The method of claim 2, wherein the training the initial recognition sub-model based on the training sample set, resulting in a trained recognition sub-model comprises:

selecting a training sample from the training sample set to obtain a selected sample;

Right shifting a new word unit in the new word unit sequence in the selected sample by one bit to obtain a training word unit sequence;

and training the initial recognition sub-model based on the audio feature sequence and the training word unit sequence in the selected sample to obtain a trained recognition sub-model.

5. The method of claim 1, wherein the acquiring a set of speech samples comprises:

acquiring an initial data set comprising at least one initial voice;

preprocessing the initial data set to obtain a processed data set;

Performing data enhancement on the processed data set to obtain a voice data set;

and obtaining a voice sample set based on the voice data set.

6. The method of claim 5, wherein the preprocessing the initial dataset to obtain a processed dataset comprises at least one of:

sampling all initial data in the initial data set at the same sampling rate;

Duplicate and unrecognizable initial data in the initial data set is deleted.

7. The method of claim 5, wherein the data enhancing the processed data set to obtain a speech data set comprises at least one of:

Modifying the speech speed of each processing data in the processing data set to obtain an increasing data set, and increasing the increasing data set in the processing data set;

Adding reverberation to each processing data in the processing data set to obtain a reverberation data set, and adding the reverberation data set in the processing data set;

adding noise to each processing data in the processing data set to obtain a noise data set, and adding the noise data set in the processing data set;

And modifying an audio frequency spectrogram of each processing data in the processing data set to obtain a frequency spectrum data set, and adding the frequency spectrum data set in the processing data set.

8. A speech recognition model testing method, the method comprising:

Obtaining a test set and a trained speech recognition model, the test set comprising at least one test sample, the test sample comprising: an audio feature sequence and a test word unit sequence; the trained speech recognition model is obtained by training the speech recognition model training method according to any one of claims 1 to 7, and the speech recognition model comprises: an encoder, a decoder, and a keyword module;

Selecting a test sample from the test set;

inputting the audio feature sequence in the test sample into the encoder to obtain audio intermediate features;

Inputting the audio intermediate characteristics and the current word unit sequence into the decoder to obtain a predicted word unit output by the voice recognition model;

Responding to the predicted word unit as a non-ending symbol, updating a current word unit sequence based on the predicted word unit, and continuously inputting the audio intermediate feature and the current word unit sequence into the decoder until the predicted word unit output by the voice recognition model is the ending symbol, so as to obtain all the predicted word units corresponding to the test sample;

And detecting whether the voice recognition model is qualified or not based on all the predicted word units corresponding to the test sample and the test word unit sequences in the test sample.

9. The method of claim 8, wherein the detecting whether the speech recognition model is qualified based on all the predicted word units corresponding to the test sample and the sequence of the test word units in the test sample comprises:

sequencing all the predicted word units corresponding to the test sample to obtain a predicted word unit sequence;

calculating the word error rate of the test sample based on the predicted word unit sequence and the test word unit sequence in the test sample;

And determining that the speech recognition model is qualified in response to the word error rate being less than an error threshold.

10. A method of speech recognition, the method comprising:

Acquiring voice to be recognized;

processing the voice to be recognized to obtain audio characteristic data;

Inputting the audio feature data into a voice recognition model to obtain a predicted word unit sequence of the voice to be recognized, wherein the voice recognition model is a trained voice recognition model obtained by adopting the voice recognition model training method according to any one of claims 1-7;

And obtaining text data of the voice to be recognized based on the predicted word unit sequence.

11. A speech recognition model training apparatus, the apparatus comprising:

A sample acquisition unit configured to acquire a set of speech samples, the set of speech samples including at least one speech sample, the speech sample comprising: an audio feature sequence and an initial word unit sequence;

the model acquisition unit is configured to acquire an initial voice recognition model, wherein the voice recognition model is used for representing the corresponding relation between the audio feature sequence and the predicted word unit sequence;

The replacing unit is configured to replace language word units in the initial word unit sequence in the voice sample set by using the predicted word units of the characterization language to obtain a training sample set, wherein the predicted word units are predicted word units in the predicted word unit sequence obtained by inputting voice samples selected from the voice sample set into the voice recognition model;

and the training unit is configured to train the voice recognition model based on the training sample set to obtain a trained voice recognition model.

12. The apparatus of claim 11, wherein the speech recognition model comprises: the system comprises an initial recognition sub-model and a keyword module connected with the initial recognition sub-model; the training unit is configured to: training the initial recognition sub-model based on the training sample set to obtain a trained recognition sub-model; training the keyword module based on the training sample set and the trained recognition sub-model to obtain a trained keyword module; and taking the trained recognition sub-model and the trained keyword module as a trained voice recognition model.

13. The apparatus of claim 12, wherein the training unit is further configured to: fixing parameters of the trained recognition sub-model; inputting training samples in a training sample set into the trained recognition sub-model to obtain a first probability distribution output by the trained recognition sub-model and a second probability distribution output by the keyword module; calculating a loss value of an initial word unit in a training sample based on the first probability distribution and the second probability distribution; and obtaining a trained keyword module based on the loss value.

14. The apparatus of claim 12, wherein the training unit is further configured to: selecting a training sample from the training sample set to obtain a selected sample; right shifting a new word unit in the new word unit sequence in the selected sample by one bit to obtain a training word unit sequence; and training the initial recognition sub-model based on the audio feature sequence and the training word unit sequence in the selected sample to obtain a trained recognition sub-model.

15. The apparatus of claim 11, wherein the sample acquisition unit is configured to: acquiring an initial data set comprising at least one initial voice; preprocessing the initial data set to obtain a processed data set; performing data enhancement on the processed data set to obtain a voice data set; and obtaining a voice sample set based on the voice data set.

16. The apparatus of claim 15, wherein the sample acquisition unit is further configured to at least one of:

sampling all initial data in the initial data set at the same sampling rate;

Duplicate and unrecognizable initial data in the initial data set is deleted.

17. The apparatus of claim 15, wherein the sample acquisition unit is further configured to at least one of:

Modifying the speech speed of each processing data in the processing data set to obtain an increasing data set, and increasing the increasing data set in the processing data set;

Adding reverberation to each processing data in the processing data set to obtain a reverberation data set, and adding the reverberation data set in the processing data set;

adding noise to each processing data in the processing data set to obtain a noise data set, and adding the noise data set in the processing data set;

And modifying an audio frequency spectrogram of each processing data in the processing data set to obtain a frequency spectrum data set, and adding the frequency spectrum data set in the processing data set.

18. A speech recognition model testing apparatus, the apparatus comprising:

An information acquisition unit configured to acquire a test set and a trained speech recognition model, the test set including at least one test sample, the test sample including: an audio feature sequence and a test word unit sequence; the trained speech recognition model is obtained by training the speech recognition model training device according to any one of claims 11 to 17, and the speech recognition model comprises: an encoder, a decoder, and a keyword module;

a selecting unit configured to select a test sample from the test set;

an input unit configured to input the audio feature sequence in the test sample to the encoder to obtain audio intermediate features;

The obtaining unit is configured to input the audio intermediate characteristics and the current word unit sequence into the decoder to obtain a predicted word unit output by the voice recognition model;

the updating unit is configured to respond to the predicted word unit as a non-ending symbol, update the current word unit sequence based on the predicted word unit, and continuously input the audio intermediate feature and the current word unit sequence into the decoder until the predicted word unit output by the voice recognition model is the ending symbol, so as to obtain all the predicted word units corresponding to the test sample;

And the test unit is configured to detect whether the voice recognition model is qualified or not based on all the prediction word units corresponding to the test sample and the test word unit sequences in the test sample.

19. The apparatus of claim 18, wherein the test unit is further configured to: sequencing all the predicted word units corresponding to the test sample to obtain a predicted word unit sequence; calculating the word error rate of the test sample based on the predicted word unit sequence and the test word unit sequence in the test sample; and determining that the speech recognition model is qualified in response to the word error rate being less than an error threshold.

20. A speech recognition device, the device comprising:

A voice acquisition unit configured to acquire a voice to be recognized;

the processing unit is configured to process the voice to be recognized to obtain audio characteristic data;

A recognition unit configured to input the audio feature data into a speech recognition model to obtain the predicted word unit sequence of the speech to be recognized, wherein the speech recognition model is a trained speech recognition model obtained by using the speech recognition model training device according to any one of claims 11 to 17;

and the conversion unit is configured to obtain text data of the voice to be recognized based on the predicted word unit sequence.

21. An electronic device, comprising:

at least one processor; and

A memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-10.

22. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-10.

23. A computer program product comprising a computer program which, when executed by a processor, implements the method of any of claims 1-10.