WO2022121257A1

WO2022121257A1 - Model training method and apparatus, speech recognition method and apparatus, device, and storage medium

Info

Publication number: WO2022121257A1
Application number: PCT/CN2021/097411
Authority: WO
Inventors: 罗剑; 王健宗; 程宁
Original assignee: 平安科技（深圳）有限公司
Priority date: 2020-12-11
Filing date: 2021-05-31
Publication date: 2022-06-16
Also published as: CN112435656A

Abstract

A model training method and apparatus, a speech recognition method and apparatus, a device, and a storage medium. The method comprises: performing iterative training on a first preset speech recognition model according to a plurality of first training samples to obtain a first speech recognition model (S102); fusing the first speech recognition model with a preset language model to obtain a second speech recognition model (S103); inputting second speech sequences in a plurality of second training samples into the second speech recognition model to obtain second text and a fusion score corresponding to each second speech sequence (S104); selecting target speech sequences from the plurality of second speech sequences according to the fusion score of each second speech sequence (S105); and performing iterative training on a second preset speech recognition model according to each target speech sequence, the second text corresponding to the target speech sequence, and the plurality of first training samples, to obtain a target speech recognition model (S106). The method allows for improvement of the speech recognition model training efficiency.

Description

Model training method, speech recognition method, apparatus, equipment and storage medium

This application claims the priority of the Chinese patent application with the application number 202011453446.1 and the invention titled "Model Training Method, Speech Recognition Method, Apparatus, Equipment and Storage Medium" filed with the China Patent Office on December 11, 2020, and its entire contents Incorporated herein by reference.

technical field

The present application relates to the technical field of model construction in artificial intelligence, and in particular, to a model training method, a speech recognition method, an apparatus, a device and a storage medium.

Background technique

Automatic Speech Recognition is a technology that converts speech into text. As an important technology in the field of artificial intelligence, speech recognition is used in various industries related to the Internet, communication, smart home, etc., and speech recognition models are usually used for automatic speech recognition. In order to train a speech recognition model, a large amount of speech data and text data corresponding to the speech data need to be prepared. In the prior art, the text data sample is obtained in the following manner: organize a large number of people to listen to the voice data and write down the correct text data. However, the inventor realized that with the advancement of algorithms and computer capabilities, the speech recognition model allows more and more speech data and corresponding text data to be trained to improve the accuracy of the speech recognition model, which makes the labor cost a resource investment It is time-consuming, expensive and inefficient.

SUMMARY OF THE INVENTION

The main purpose of the present application is to provide a model training method, speech recognition method, apparatus, equipment and storage medium, aiming at improving the training effect and training efficiency of the speech recognition model.

In a first aspect, the present application provides a model training method, including:

Obtain a plurality of first training samples and a plurality of second training samples, the first training samples include a first speech sequence and a first text corresponding to the marked first speech sequence, and the second training samples include a second speech sequence;

performing iterative training on a first preset speech recognition model according to the plurality of first training samples to obtain a first speech recognition model;

Integrating the first speech recognition model with the preset language model to obtain a second speech recognition model;

Inputting a plurality of the second speech sequences into the second speech recognition model to obtain a second text and a fusion score corresponding to each of the second speech sequences;

According to the fusion score of each of the second speech sequences, filter out the target speech sequence from the plurality of second speech sequences;

Perform iterative training on the second preset speech recognition model according to each of the target speech sequences, the second text corresponding to each of the target speech sequences, and a plurality of the first training samples to obtain a target speech recognition model.

In a second aspect, the present application also provides a speech recognition method, comprising:

Get the speech sequence to be recognized;

Perform speech recognition on the speech sequence by using the target speech recognition model to obtain text information corresponding to the speech sequence;

The target speech recognition model is obtained by training according to the above-mentioned model training method.

In a third aspect, the present application also provides a model training device, the model training device comprising:

an acquisition module, configured to acquire a plurality of first training samples and a plurality of second training samples, the first training samples include a first speech sequence and a first text corresponding to the marked first speech sequence, the second the training sample includes a second speech sequence;

a first training module, configured to iteratively train a first preset speech recognition model according to the plurality of first training samples to obtain a first speech recognition model;

a fusion module, configured to fuse the first speech recognition model with the preset language model to obtain a second speech recognition model;

an input module, configured to input a plurality of the second speech sequences into the second speech recognition model, to obtain a second text and a fusion score corresponding to each of the second speech sequences;

a screening module, configured to screen out a target speech sequence from the plurality of second speech sequences according to the fusion score of each of the second speech sequences;

A second training module, configured to iterate the second preset speech recognition model according to each of the target speech sequences, the second text corresponding to each of the target speech sequences, and a plurality of the first training samples Training to get the target speech recognition model.

In a fourth aspect, the present application also provides a computer device, the computer device comprising a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein the computer program is executed by the When executed by the processor, the steps of the above-mentioned model training method or speech recognition method are implemented.

In a fifth aspect, the present application further provides a computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the above-mentioned model training method or speech Identify the steps of the method.

The present application provides a model training method, a speech recognition method, an apparatus, a device and a storage medium. The present application obtains a plurality of first training samples and a plurality of second training samples, wherein the first training samples include a first speech sequence and annotated The first text corresponding to the first voice sequence, the second training sample includes the second voice sequence, and then according to the plurality of first training samples, the first preset voice recognition model is iteratively trained to obtain a first voice recognition model, and the first voice recognition model is obtained. A speech recognition model is fused with a preset language model to obtain a second speech recognition model, and then multiple second speech sequences are input into the second speech recognition model to obtain the second text corresponding to each second speech sequence and the fusion score , according to the fusion score of each second speech sequence, screen out the target speech sequence from multiple second speech sequences, and according to each target speech sequence, the second text corresponding to each target speech sequence and a plurality of first training samples , performing iterative training on the second preset speech recognition model to obtain the target speech recognition model. This application trains the "teacher-noisy student" self-training learning model through multiple labeled first training samples and multiple unlabeled second training samples, which can greatly improve the training effect of the speech recognition model and reduce the need for The number of labeled training samples is required, and the training efficiency of the speech recognition model is improved.

Description of drawings

In order to explain the technical solutions of the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. For those of ordinary skill, other drawings can also be obtained from these drawings without any creative effort.

1 is a schematic flowchart of steps of a model training method provided by an embodiment of the present application;

Fig. 2 is the sub-step flow schematic diagram of the model training method in Fig. 1;

3 is a schematic diagram of a scenario for implementing the model training method provided by this embodiment;

4 is a schematic flowchart of steps of a speech recognition method provided by an embodiment of the present application;

FIG. 5 is a schematic block diagram of a model training apparatus provided by an embodiment of the present application;

Fig. 6 is a schematic block diagram of sub-modules of the model training device in Fig. 5;

FIG. 7 is a schematic block diagram of a speech recognition apparatus provided by an embodiment of the present application;

FIG. 8 is a schematic block diagram of the structure of a computer device according to an embodiment of the present application.

The realization, functional characteristics and advantages of the purpose of the present application will be further described with reference to the accompanying drawings in conjunction with the embodiments.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

The flowcharts shown in the figures are for illustration only, and do not necessarily include all contents and operations/steps, nor do they have to be performed in the order described. For example, some operations/steps can also be decomposed, combined or partially combined, so the actual execution order may be changed according to the actual situation. In addition, although the functional modules are divided in the schematic diagram of the device, in some cases, the modules may be divided differently from the schematic diagram of the device.

Embodiments of the present application provide a model training method, a speech recognition method, an apparatus, a device, and a storage medium. Wherein, the model training method can be applied to a terminal device or a server, and the terminal device can be an electronic device such as a mobile phone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant and a wearable device; the server can be a single server , or a server cluster consisting of multiple servers. The following takes the model training method applied to the server as an example for explanation.

Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and features in the embodiments may be combined with each other without conflict.

Please refer to FIG. 1 , which is a schematic flowchart of steps of a model training method provided by an embodiment of the present application.

As shown in FIG. 1 , the model training method includes steps S101 to S106.

Step S101: Acquire a plurality of first training samples and a plurality of second training samples, where the first training samples include a first speech sequence and a first text corresponding to the marked first speech sequence, and the second training samples include a second speech sequence.

The first training sample includes the first speech sequence and the first text corresponding to the first speech sequence, where the first text is a label of the corresponding first speech sequence, and the second training sample includes the second speech sequence. It should be noted that the first voice sequence and the second voice sequence are audio data, and the first text corresponding to the first voice sequence is the text content recognized by the voice of the first voice sequence. For example, the first speech sequence is a song, and the corresponding first text is lyrics.

The Noisy Student Training (NST) model is a semi-supervised learning model composed of "teacher" and "student". The teacher model (the first preset speech recognition model) is used to learn the marked first training sample, Predict the unlabeled second training sample, obtain the labeled second training sample and the second text corresponding to the second training sample, and then let the student model (the second preset speech recognition model) for the labeled first training sample. The training sample, the labeled second training sample, and the second text corresponding to the second training sample are trained, and the above process is iterated. Through "teacher-noise student" self-training learning, the training effect of the speech recognition model can be greatly improved.

In one embodiment, the total audio length of the plurality of first training samples is higher than the first preset time threshold, and the total audio length of the plurality of second training samples is higher than the second preset time threshold, which can ensure that the data obtained in subsequent training can be guaranteed. The accuracy of the speech recognition model for speech recognition.

Further, the second preset time threshold is higher than the first preset time threshold. In practical applications, the first preset time threshold and the second preset time threshold may be set according to actual application scenarios, for example, the first preset time threshold is 100h, and the second preset time threshold is 500h, which will not be repeated here. .

It should be noted that, in order to further ensure the privacy and security of related information such as the above-mentioned multiple first training samples and multiple second training samples, the above-mentioned related information such as the first training samples and the second training samples can also be stored in one area. In the nodes of the block chain, the technical solution of this application can also be applied to add other data files stored on the block chain. The block chain in the present invention refers to computers such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm, etc. New application model of technology. Blockchain, essentially a decentralized database, is a series of data blocks associated with cryptographic methods. Each data block contains a batch of network transaction information to verify its Validity of information (anti-counterfeiting) and generation of the next block. The blockchain can include the underlying platform of the blockchain, the platform product service layer, and the application service layer.

Step S102: Perform iterative training on a first preset speech recognition model according to a plurality of first training samples to obtain a first speech recognition model.

The first preset speech recognition model is a "teacher" model, and a plurality of first training samples are input into the first preset speech recognition model to obtain a speech recognition result corresponding to each first training sample, and according to each first training sample, the corresponding speech recognition results are obtained. The speech recognition results corresponding to the training samples and the corresponding first text respectively adjust the parameters of the first preset speech recognition model until a first speech recognition model whose performance meets the preset training conditions is obtained.

For example, if the performance is recognition accuracy, the preset training condition may be that the recognition accuracy is higher than the preset accuracy threshold. It should be noted that both the preset training conditions and the preset accuracy threshold can be set according to actual application scenarios. For example, the preset accuracy threshold is 0.9, which is not specifically limited here.

The first preset speech recognition model is, for example, a LAS (Listen, Attend and Spell) model, and the LAS model includes a Listen (listening) layer, an Attend (attention) layer and a Spell (spelling) layer. The first preset speech recognition model extracts the speech signal features of the input first training sample, encodes by inputting the speech signal features into the Listen layer, and then uses the Attention layer to pay attention to different parts of the input at different times (Attend), and finally uses The Spell layer is decoded to obtain the speech recognition result of the first training sample.

In one embodiment, data enhancement is performed on a plurality of first training samples; according to the plurality of first training samples after data enhancement, the first preset speech recognition model is iteratively trained until the first preset speech recognition model is Convergence to obtain the first speech recognition model. It should be noted that the number of samples of the first training sample can be increased through data augmentation, for example, by normalizing the channel length, superimposing clean audio and noisy audio signals to synthesize noisy audio or speed perturbation of the original audio etc. to achieve data enhancement. For the specific implementation process of iterative training of the first preset speech recognition model, reference may be made to the foregoing embodiments. The convergence of the first preset speech recognition model may be that the performance meets the preset training conditions, and the number of iterations is greater than the preset number of iterations and/or the iteration duration. is greater than the preset iteration duration, etc., which is not specifically limited in this embodiment. Add some noise to the first preset speech recognition model through data enhancement, so that the subsequent second preset speech recognition model (student model) can be forced to work harder to learn the output of the first preset speech recognition model (teacher model). The speech recognition result improves the training effect of the target speech recognition model.

Further, the SpecAugment is used to perform data enhancement on the plurality of first training samples, and the robustness of the first speech recognition model is increased by adding some noise to the first preset speech recognition model. Specifically, each first speech sequence is converted into a spectrogram, and time deformation, frequency masking and/or time masking are performed on the plurality of spectrograms by SpecAugment. Before the iterative training of the first preset speech recognition model, the spectrogram of the first speech sequence can be enhanced through SpecAugment, which can increase the training speed of the first speech recognition model for the first training sample, thereby improving the performance of the target speech recognition model. training efficiency.

It should be noted that the time deformation of multiple spectrograms through SpecAugment refers to the mel spectrogram of the π time jump, taking time as the x-axis and frequency as the y-axis, and constructing a random horizontal line through the center of the mel spectrogram. , then the time period (W,W-π) is deformed to the left or right; frequency masking multiple spectrograms means that on the frequency axis of successive mel spectrograms, [f ₀ ,f ₀ +f ] mask, where f is a uniform parameter from 0 to F; frequency masking multiple spectrograms refers to masking [t ₀ ,t ₀ +t] on the time axis of successive mel spectrograms membrane, where t is a uniform distribution from 0 to T. Use SpecAugment to perform data enhancement on multiple first training samples. As the iterative training progresses, the robustness and performance of the first preset speech recognition model will be improved. At this time, the intensity of Specaugment can be increased to give the first preset speech The input of the recognition model brings more noise, thereby improving the training effect of the first speech recognition model.

In one embodiment, noise is added to the first preset speech recognition model; according to a plurality of first training samples, the first preset speech recognition model with noise added is iteratively trained until the first preset speech recognition model with noise added The model converges, and a first speech recognition model is obtained.

Exemplarily, Dropout is used to add noise to the first preset speech recognition model, that is, some hidden neurons of the neural network are randomly disabled in each training of the first preset speech recognition model, and the output of the hidden neurons is 0, and do not update the weights either. For example, if dropout ratio=p is set, then each hidden neuron does not work with probability p. Therefore, in noise student training, adding noise to the first preset speech recognition model through Dropout can make the second speech recognition model (student model) is forced to work harder to learn the speech recognition result output by the first preset speech recognition model (teacher model), so as to improve the training effect of the target speech recognition model.

In one embodiment, data enhancement is performed on a plurality of first training samples, and noise is added to the first preset speech recognition model; according to the plurality of first training samples after data enhancement, the first preset noise added The speech recognition model is iteratively trained until the first preset speech recognition model with added noise converges, and a first speech recognition model is obtained. By performing data enhancement on the first training sample and adding noise to the first preset speech recognition model, the parameters of the iteratively completed first speech recognition model can be made more accurate, thereby improving the training effect of the subsequent target speech recognition model.

Step S103 , fuse the first speech recognition model with the preset language model to obtain a second speech recognition model.

The preset language model is a pre-trained language model (Language Model), and the preset language model is, for example, a statistical language model, a feedforward neural network language model, a recurrent neural network language model, and the like. By fusing the first speech recognition model with the preset language model, the obtained second speech recognition model has better performance, which is beneficial to improve the training effect of the target speech recognition model and make the speech recognition accuracy of the target speech recognition model more accurate. high.

In one embodiment, the data volume of the training samples of the language model is much larger than the data volume of the first training samples of the first speech recognition model, and the fusion of the first speech recognition model and the preset language model can help the second speech recognition model. The model performs semantic information modeling, and the fusion methods include Voting (voting method), Averaging (averaging method), Bagging (guided aggregation method) algorithm, Boosting (boosting method), etc., which are not specifically limited in this embodiment.

Step S104 , inputting multiple second speech sequences into the second speech recognition model to obtain the second text and fusion score corresponding to each second speech sequence.

A plurality of second training samples are input into the second speech recognition model, and a speech recognition result corresponding to each second speech sequence is obtained, and the speech recognition result includes the second text corresponding to the second speech sequence and the fusion score. Predict multiple second voice sequences through the second voice recognition model, and output the second text and fusion score corresponding to each second voice sequence, so as to screen out the second voice sequences that meet the preset conditions from the multiple second voice sequences speech sequence.

Exemplarily, the second speech recognition model is a LAS (Listen, Attend and Spell) model, including a Listen (listen) layer, an Attend (attention) layer and a Spell (spelling) layer. The second speech sequence is the sound signal feature vector x of length T. After the sound signal feature vector x of length T is input into the second speech recognition model, the content related to the sound signal is retained through the Listen layer, and the content that is not related to the sound signal is removed. Noise, the Listen layer is for example a bidirectional LSTM network, and the output feature vector h=BiLSTM(x) of length T; in the Attend layer, the scaled attention mechanism can be used to obtain the hidden layer state of the RNN network in the Attend layer at the current moment S _t , calculate the context vector (Context Vector) at the current moment according to the feature vector h output by the Listen layer and the hidden layer state S _t , that is, the context vector C _t = Attention(S _t , h); in the Spell layer, use the RNN network As a decoder, determine the hidden layer state at the last moment, the output vector and context vector of the Spell layer, and calculate the hidden layer state at the current moment S _t =RNN(S _t-1 ,Y _t-1 ,C _t-1 ), The output vector at the current moment is then passed through the softmax network to output the character distribution probability corresponding to the second speech sequence (the distribution probability of the second text) Y _t =CharacterDistribution(S _t ). Since the second speech recognition model is obtained by the fusion of the trained first speech recognition model LAS and the language model LM, the weighted summation of the character distribution probabilities of the LAS model and the LM model can obtain the fusion corresponding to the second speech sequence. Fraction. For example, the fusion score S=log p(Y _t =k)=log p _LAS (Y _t =k)+βlog p _LM (Y _t =k), where β refers to the hyperparameter that needs to be up-regulated for the second speech sequence, k refers to the second text with the largest character distribution probability at time t, log p _LAS (Y _t =k) refers to the character distribution probability of the second speech sequence output by the LAS model, log p _LM (Y _t =k) is Refers to the character distribution probability of the second speech sequence output by the LM model.

Step S105 , according to the fusion score of each second speech sequence, screen out the target speech sequence from the plurality of second speech sequences.

For the second text corresponding to the second speech sequence output by the second speech recognition model, it is necessary to filter out the target speech sequence that meets the preset conditions. According to the fusion score of each second speech sequence, it is possible to select the target speech sequence from multiple second speech sequences. The target speech sequence is screened out, and the target speech sequence can be used as training data for a high-quality "student" model (the second preset speech recognition model), thereby improving the training effect of the second preset speech recognition model.

In one embodiment, as shown in FIG. 2 , step S105 includes: sub-steps S1051 to S1052 .

Sub-step S1051: Filter the multiple second voice sequences according to the preset score threshold and the fusion score of each second voice sequence to obtain multiple candidate voice sequences.

In one embodiment, the preset score threshold can be flexibly set by the user, the second speech sequence whose fusion score is greater than or equal to the preset score threshold is reserved, and the second speech sequence whose fusion score is less than the preset score threshold is screened out to obtain a plurality of speech sequences. candidate speech sequences. It should be noted that the accuracy rate of the second text corresponding to the second voice sequence with a high fusion score is higher, so retaining the second voice sequence with a higher accuracy rate of the second text is conducive to screening out high-quality second voices. sequence.

In one embodiment, since the sentence lengths of the second speech sequences are different, the speech recognition result of the second speech recognition model will be affected, resulting in inconsistent accuracy of the second text corresponding to each second speech sequence and the fusion score. Therefore, by regularizing the fusion score of the second speech sequence, and then comparing the regularized fusion score with the preset score threshold, the second speech sequence whose fusion score is less than the preset score threshold is filtered out, and a high-quality speech sequence is obtained. Multiple candidate speech sequences.

Among them, the regularization formula is:

l is the character length of the second speech sequence, μ and β are the parameters obtained by performing linear regression on (li _i , S _i ) of multiple second speech sequences, and σ is the calculation

standard deviation of . In some embodiments, the preset score threshold may decrease as the iteration time increases, and the preset score threshold becomes smaller and smaller in the iterative training, so that more and more candidate speech sequences can be used for training the target speech recognition model sample.

Sub-step S1052, according to the probability distribution information of the plurality of first training samples, screen out the target speech sequence from the plurality of candidate speech sequences.

In one embodiment, selecting a target speech sequence from a plurality of candidate speech sequences according to the probability distribution information of the plurality of first training samples includes: generating a plurality of speech sequence sets according to the plurality of candidate speech sequences, wherein each the speech sequence sets include at least one candidate speech sequence; determine the probability distribution information of each speech sequence set; according to the probability distribution information of the plurality of first training samples and the probability distribution information of each speech sequence set, from the plurality of speech sequence sets Select the target speech sequence set. Wherein, the target speech sequence set includes at least one target speech sequence. It should be noted that the distribution of the filtered candidate speech sequences is quite different. If the filtered candidate speech sequences are directly used as the training samples of the second preset speech recognition model, the second preset speech recognition model will be affected. The performance of the speech recognition model, therefore, a target speech sequence set that is similar to the probability distribution information of multiple first training samples is found from multiple speech sequence sets, and at least one target speech sequence in the target speech sequence set is used as the second preset. The training samples of the speech recognition model can improve the performance of the second preset speech recognition model, which can improve the training effect of the target speech recognition model.

Wherein, multiple batches (Batch) are randomly selected from multiple candidate speech sequences to generate multiple speech sequence sets, and each batch includes at least one candidate speech sequence. Each candidate speech sequence carries attribute information, and the attribute information carried by multiple candidate speech sequences can form the probability distribution information of a speech sequence set. The probability distribution information is determined according to the specific service set. For example, the probability distribution information is the length of the audio. , the ratio of male and female speakers, the age of the speaker, the surrounding environment, etc., and compare the probability distribution information corresponding to each speech sequence set with the probability distribution information of multiple first training samples to find out the corresponding probability distribution information of multiple first training samples. The probability distribution information of the approximate target speech sequence set.

In one embodiment, the K-L divergence of each voice sequence set is calculated according to the probability distribution information of a plurality of first training samples and the probability distribution information of each voice sequence set; according to the K-L divergence of each voice sequence set, Select a target speech sequence set from multiple speech sequence sets. Among them, the probability distribution information of the speech sequence set with the lower K-L divergence is closer to the probability distribution information of multiple first training samples, and the speech sequence set with the lowest K-L divergence is selected as the target speech sequence set. Include at least one target speech sequence. By calculating the K-L divergence (K-L Divergence) of each speech sequence set, the target speech sequence set that is similar to the probability distribution information of the multiple first training samples can be accurately found.

The formula for calculating the K-L divergence is as follows:

Wherein, f(M(U)) is the speech sequence set, P(i) is the probability distribution information of a plurality of first training samples, and Q(i) is the probability distribution information of the speech sequence set.

Step S106: Perform iterative training on the second preset speech recognition model according to each target speech sequence, the second text corresponding to each target speech sequence, and a plurality of first training samples to obtain a target speech recognition model.

After acquiring multiple target speech sequences, input multiple first training samples into the speech recognition student model, output the first speech recognition result, and according to the difference between the first speech recognition result and the first text corresponding to each first training sample. The parameters of the second preset speech recognition model are adjusted; when it is determined that the adjusted second preset speech recognition model meets the preset performance conditions, the model training is stopped, and the trained initial speech recognition model is obtained; Multiple target speech sequences are used to train the initial speech recognition model to obtain the target speech recognition model. It should be noted that training the "teacher-noise student" self-training learning model through multiple labeled first training samples and multiple unlabeled second training samples can greatly improve the training effect of the speech recognition model. The requirement for the number of labeled training samples is reduced, and the training efficiency of the speech recognition model is improved.

The preset performance condition is determined according to the speech recognition accuracy and speech recognition speed of the speech recognition student model. In practical applications, the preset performance conditions may also be set according to actual application scenarios. The second preset speech recognition model is initialized by using a plurality of first training samples to ensure the convergence of the training data. The initial speech recognition model is trained through multiple target speech sequences, and a target speech recognition model with better training effect is obtained, and the target speech recognition model has a high accuracy of speech recognition.

In one embodiment, the second preset speech recognition model is, for example, a LAS (Listen, Attend and Spell) model, and the LAS model includes a Listen (listen) layer, an Attend (attention) layer and a Spell (spelling) layer.

In one embodiment, multiple third training samples are generated according to each target speech sequence and the second text corresponding to each target speech sequence; training samples are obtained according to multiple third training samples and multiple first training samples Set; through the training sample set, iterative training is performed on the second preset speech recognition model until the preset condition is reached, and the target speech recognition model is obtained. The preset conditions may be that the performance meets the preset training conditions, the number of iterations is greater than the preset number of iterations, and/or the iteration duration is greater than the preset iteration duration, etc., which are not specifically limited in this embodiment of the present application.

Please refer to FIG. 3 , which is a schematic diagram of a scenario for implementing the model training method provided by this embodiment.

As shown in FIG. 3 , a plurality of first training samples and a plurality of second training samples are obtained, the first training samples include the first speech sequence and the first text corresponding to the marked first speech sequence, and the second training samples include the second speech sequence, and then input a plurality of first training samples into the first preset speech recognition model 10 to iteratively train the first preset speech recognition model 10 to obtain the first speech recognition model 20, and the preset language model 30 Integrate with the first speech recognition model 20 to obtain a second speech recognition model 40, and then input the second speech sequences in the plurality of second training samples into the second speech recognition model 40 to obtain each second speech sequence corresponding to According to the fusion score of each second speech sequence, the target speech sequence is screened from multiple second speech sequences, and each The text and the plurality of first training samples are input into the second preset speech recognition model 50 to iteratively train the second preset speech recognition model 50 to obtain the target speech recognition model 60 .

In the model training method provided by the above-mentioned embodiment, a plurality of first training samples and a plurality of second training samples are obtained, wherein the first training samples include the first speech sequence and the first text corresponding to the marked first speech sequence, and the second training samples are obtained. The sample includes a second speech sequence, and then according to a plurality of first training samples, the first preset speech recognition model is iteratively trained to obtain a first speech recognition model, and the first speech recognition model and the preset language model are fused to obtain The second speech recognition model, and then multiple second speech sequences are input into the second speech recognition model to obtain the second text corresponding to each second speech sequence and the fusion score. According to the fusion score of each second speech sequence, from The target speech sequence is selected from the plurality of second speech sequences, and the second preset speech recognition model is iteratively trained according to each target speech sequence, the second text corresponding to each target speech sequence, and a plurality of first training samples, Get the target speech recognition model. This application trains the "teacher-noisy student" self-training learning model through multiple labeled first training samples and multiple unlabeled second training samples, which can greatly improve the training effect of the speech recognition model and reduce the need for The number of labeled training samples is required, and the training efficiency of the speech recognition model is improved.

Please refer to FIG. 4 , which is a schematic flowchart of steps of a speech recognition method provided by an embodiment of the present application.

As shown in FIG. 4 , the model training method includes steps S201 to S202.

Step S201, acquiring a speech sequence to be recognized.

For example, the speech sequence to be recognized is a piece of speech data sent by a user in a social application.

Step S202 , perform speech recognition on the speech sequence by using the target speech recognition model to obtain text information corresponding to the speech sequence.

The target speech recognition model is obtained by training according to the model training method described in the foregoing embodiment. For example, user A receives a voice sequence sent by user B through the social application of the terminal device, performs voice recognition on the voice sequence through the target voice recognition model, and obtains the text information "Hello" (speech recognition result).

The speech recognition method provided by the above-mentioned embodiment, by acquiring the speech sequence to be recognized, and performing speech recognition on the speech sequence through the target speech recognition model described in the foregoing embodiment, the text information corresponding to the speech sequence is obtained, because the target speech recognition model " The teacher-noise-student" self-training learning model can effectively improve the accuracy of speech recognition.

Please refer to FIG. 5 , which is a schematic block diagram of a model training apparatus provided by an embodiment of the present application.

As shown in FIG. 5 , the model training apparatus 300 includes: an acquisition module 301 , a first training module 302 , a fusion module 303 , an input module 304 , a screening module 305 and a second training module 306 .

The acquisition module 301 is configured to acquire a plurality of first training samples and a plurality of second training samples, the first training samples include the first speech sequence and the first text corresponding to the marked first speech sequence, and the second training samples include the second speech sequence;

A first training module 302, configured to perform iterative training on a first preset speech recognition model according to a plurality of first training samples to obtain a first speech recognition model;

A fusion module 303, configured to fuse the first speech recognition model with the preset language model to obtain a second speech recognition model;

The input module 304 is used for inputting a plurality of second speech sequences into the second speech recognition model to obtain the second text and fusion score corresponding to each second speech sequence;

The screening module 305 is used for screening out the target speech sequence from the plurality of second speech sequences according to the fusion score of each second speech sequence;

The second training module 306 is configured to iteratively train the second preset speech recognition model according to each target speech sequence, the second text corresponding to each target speech sequence, and a plurality of first training samples to obtain the target speech recognition model .

In one embodiment, as shown in Figure 6, the screening module 305 includes:

Filtering submodule 3051, configured to filter a plurality of the second speech sequences according to a preset score threshold and a fusion score of each of the second speech sequences to obtain a plurality of candidate speech sequences;

The screening sub-module 3052 is configured to screen out a target speech sequence from the plurality of candidate speech sequences according to the probability distribution information of the plurality of first training samples.

In one embodiment, the screening sub-module 3052 is also used to:

generating a plurality of speech sequence sets according to the plurality of candidate speech sequences, wherein each of the speech sequence sets includes at least one of the candidate speech sequences;

determining the probability distribution information of each said set of speech sequences;

According to the probability distribution information of the plurality of first training samples and the probability distribution information of each of the speech sequence sets, a target speech sequence set is selected from the plurality of the speech sequence sets.

In one embodiment, the screening sub-module 3052 is also used to:

Calculate the K-L divergence of each of the speech sequence sets according to the probability distribution information of the plurality of first training samples and the probability distribution information of each of the speech sequence sets;

According to the K-L divergence of each of the speech sequence sets, a target speech sequence set is selected from a plurality of the speech sequence sets.

In one embodiment, the first training module 302 is further used to:

performing data enhancement on a plurality of the first training samples;

The first preset speech recognition model is iteratively trained according to the plurality of first training samples after data enhancement, until the first preset speech recognition model converges, and a first speech recognition model is obtained.

In one embodiment, the second training module 306 is also used to:

generating a plurality of third training samples according to each of the target speech sequences and the second text corresponding to each of the target speech sequences;

obtaining a training sample set according to the plurality of third training samples and the plurality of first training samples;

Through the training sample set, the second preset speech recognition model is iteratively trained until a preset condition is reached, and the target speech recognition model is obtained.

Please refer to FIG. 7 , which is a schematic block diagram of a speech recognition apparatus provided by an embodiment of the present application.

As shown in FIG. 7 , the speech recognition device 400 includes:

The acquiring module 401 is used for acquiring the speech sequence to be recognized.

The recognition module 402 is configured to perform speech recognition on the speech sequence through a target speech recognition model to obtain text information corresponding to the speech sequence.

Wherein, the target speech recognition model is obtained by training according to the model training method described in the foregoing embodiment.

It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of the description, for the specific working process of each module and unit of the speech recognition device described above, reference may be made to the corresponding speech recognition method embodiments described above. The process is not repeated here.

The apparatuses provided in the above embodiments may be implemented in the form of a computer program, and the computer program may be executed on the computer device as shown in FIG. 8 .

Please refer to FIG. 8 , FIG. 8 is a schematic block diagram of the structure of a computer device according to an embodiment of the present application. The computer device can be a server or a terminal device.

As shown in FIG. 8 , the computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a storage medium and an internal memory, and the storage medium may be non-volatile or volatile.

The storage medium may store an operating system and a computer program. The computer program includes program instructions that, when executed, can cause the processor to execute any model training method or speech recognition method.

The processor is used to provide computing and control capabilities to support the operation of the entire computer equipment.

The internal memory provides an environment for running the computer program in the storage medium. When the computer program is executed by the processor, the processor can cause the processor to execute any model training method or speech recognition method.

The network interface is used for network communication, such as sending assigned tasks. Those skilled in the art can understand that the structure shown in FIG. 8 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied. Include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

It should be understood that the processor may be a central processing unit (Central Processing Unit, CPU), and the processor may also be other general-purpose processors, digital signal processors (Digital Signal Processors, DSP), application specific integrated circuits (Application Specific Integrated circuits) Circuit, ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Wherein, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor or the like.

Wherein, in one embodiment, the processor is configured to run a computer program stored in the memory to implement the following steps:

In one embodiment, when the processor selects the target speech sequence from the plurality of second speech sequences according to the fusion score of each of the second speech sequences, the processor is configured to:

Filtering a plurality of the second speech sequences according to a preset score threshold and a fusion score of each of the second speech sequences to obtain a plurality of candidate speech sequences;

According to the probability distribution information of the plurality of first training samples, a target speech sequence is selected from the plurality of candidate speech sequences.

In one embodiment, when the processor selects the target speech sequence from the plurality of candidate speech sequences according to the probability distribution information of the plurality of first training samples, the processor is configured to:

In one embodiment, the processor selects the target from the plurality of speech sequence sets according to the probability distribution information of the plurality of first training samples and the probability distribution information of each of the speech sequence sets. When a voice sequence set is used, it is used to implement:

In one embodiment, when implementing the iterative training of the first preset speech recognition model according to the plurality of first training samples to obtain the first speech recognition model, the processor is used to implement:

performing data enhancement on a plurality of the first training samples;

In one embodiment, the processor is performing the processing of the second text according to each of the target speech sequences, the second text corresponding to each of the target speech sequences, and a plurality of the first training samples. The preset speech recognition model is iteratively trained, and when the target speech recognition model is obtained, it is used to achieve:

In one embodiment, the processor is configured to run a computer program stored in a memory to implement the following steps:

Get the speech sequence to be recognized;

The target speech recognition model is obtained by training according to the model training method described in the foregoing embodiment.

It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of the description, for the specific working process of the computer equipment described above, reference may be made to the corresponding process in the foregoing embodiment of the model training method or the speech recognition method. This will not be repeated here.

Embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, wherein when the computer program is executed by a processor, the following steps are implemented:

It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of the description, for the specific working process of the computer-readable storage medium described above, reference may be made to the corresponding models in the foregoing model training method or speech recognition method embodiment. The process is not repeated here.

The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as a hard disk or a memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) ) card, Flash Card, etc.

It should be understood that the terms used in the specification of the present application herein are for the purpose of describing particular embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural unless the context clearly dictates otherwise.

It will also be understood that, as used in this specification and the appended claims, the term "and/or" refers to and including any and all possible combinations of one or more of the associated listed items. It should be noted that, herein, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or system comprising a series of elements includes not only those elements, It also includes other elements not expressly listed or inherent to such a process, method, article or system. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article or system that includes the element.

The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments. The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalents within the technical scope disclosed in the present application. Modifications or substitutions shall be covered by the protection scope of this application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

A model training method, which includes:

Obtain a plurality of first training samples and a plurality of second training samples, the first training samples include a first speech sequence and a first text corresponding to the marked first speech sequence, and the second training samples include a second speech sequence;

performing iterative training on a first preset speech recognition model according to the plurality of first training samples to obtain a first speech recognition model;

Integrating the first speech recognition model with the preset language model to obtain a second speech recognition model;

Inputting a plurality of the second speech sequences into the second speech recognition model to obtain a second text and a fusion score corresponding to each of the second speech sequences;

According to the fusion score of each of the second speech sequences, filter out the target speech sequence from the plurality of second speech sequences;

Perform iterative training on the second preset speech recognition model according to each of the target speech sequences, the second text corresponding to each of the target speech sequences, and a plurality of the first training samples to obtain a target speech recognition model.
The model training method according to claim 1, wherein, according to the fusion score of each of the second speech sequences, selecting a target speech sequence from the plurality of second speech sequences comprises:

Filtering a plurality of the second speech sequences according to a preset score threshold and a fusion score of each of the second speech sequences to obtain a plurality of candidate speech sequences;

According to the probability distribution information of the plurality of first training samples, a target speech sequence is selected from the plurality of candidate speech sequences.
The model training method according to claim 2, wherein, according to the probability distribution information of the plurality of first training samples, selecting the target speech sequence from the plurality of candidate speech sequences comprises:

generating a plurality of speech sequence sets according to the plurality of candidate speech sequences, wherein each of the speech sequence sets includes at least one of the candidate speech sequences;

determining the probability distribution information of each said set of speech sequences;

According to the probability distribution information of the plurality of first training samples and the probability distribution information of each of the speech sequence sets, a target speech sequence set is selected from the plurality of the speech sequence sets.
The model training method according to claim 3, wherein the selection is performed from a plurality of the speech sequence sets according to the probability distribution information of the plurality of first training samples and the probability distribution information of each of the speech sequence sets Target speech sequence set, including:

Calculate the K-L divergence of each of the speech sequence sets according to the probability distribution information of the plurality of first training samples and the probability distribution information of each of the speech sequence sets;

According to the K-L divergence of each of the speech sequence sets, a target speech sequence set is selected from a plurality of the speech sequence sets.
The model training method according to any one of claims 1-4, wherein the iterative training is performed on the first preset speech recognition model according to the plurality of first training samples to obtain the first speech recognition model, include:

performing data enhancement on a plurality of the first training samples;

The first preset speech recognition model is iteratively trained according to the plurality of first training samples after data enhancement, until the first preset speech recognition model converges, and a first speech recognition model is obtained.
The model training method according to any one of claims 1-4, wherein the training method based on each of the target speech sequences, the second text corresponding to each of the target speech sequences, and a plurality of the first training sample, perform iterative training on the second preset speech recognition model to obtain a target speech recognition model, including:

generating a plurality of third training samples according to each of the target speech sequences and the second text corresponding to each of the target speech sequences;

obtaining a training sample set according to the plurality of third training samples and the plurality of first training samples;

Through the training sample set, the second preset speech recognition model is iteratively trained until a preset condition is reached, and the target speech recognition model is obtained.
A speech recognition method, comprising:

Get the speech sequence to be recognized;

Perform speech recognition on the speech sequence by using the target speech recognition model to obtain text information corresponding to the speech sequence;

The target speech recognition model is obtained by training according to the model training method according to any one of claims 1 to 6.
A model training device, wherein the model training device comprises:

an acquisition module, configured to acquire a plurality of first training samples and a plurality of second training samples, the first training samples include a first speech sequence and a first text corresponding to the marked first speech sequence, the second the training sample includes a second speech sequence;

a first training module, configured to iteratively train a first preset speech recognition model according to the plurality of first training samples to obtain a first speech recognition model;

a fusion module, configured to fuse the first speech recognition model with the preset language model to obtain a second speech recognition model;

an input module, configured to input a plurality of the second speech sequences into the second speech recognition model, to obtain a second text and a fusion score corresponding to each of the second speech sequences;

a screening module, configured to screen out a target speech sequence from the plurality of second speech sequences according to the fusion score of each of the second speech sequences;

A second training module, configured to iterate the second preset speech recognition model according to each of the target speech sequences, the second text corresponding to each of the target speech sequences, and a plurality of the first training samples Training to get the target speech recognition model.
A computer device, wherein the computer device includes a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein the computer program, when executed by the processor, implements The following steps:

Obtain a plurality of first training samples and a plurality of second training samples, the first training samples include a first speech sequence and a first text corresponding to the marked first speech sequence, and the second training samples include a second speech sequence;

performing iterative training on a first preset speech recognition model according to the plurality of first training samples to obtain a first speech recognition model;

Integrating the first speech recognition model with the preset language model to obtain a second speech recognition model;

Inputting a plurality of the second speech sequences into the second speech recognition model to obtain a second text and a fusion score corresponding to each of the second speech sequences;

According to the fusion score of each of the second speech sequences, filter out the target speech sequence from the plurality of second speech sequences;

Perform iterative training on the second preset speech recognition model according to each of the target speech sequences, the second text corresponding to each of the target speech sequences, and a plurality of the first training samples to obtain a target speech recognition model.
The computer device according to claim 9, wherein, when the processor implements the filtering of the target speech sequence from the plurality of second speech sequences according to the fusion score of each of the second speech sequences, Used to implement:

Filtering a plurality of the second speech sequences according to a preset score threshold and a fusion score of each of the second speech sequences to obtain a plurality of candidate speech sequences;

According to the probability distribution information of the plurality of first training samples, a target speech sequence is selected from the plurality of candidate speech sequences.
The computer device according to claim 10, wherein, when the processor selects the target speech sequence from the plurality of candidate speech sequences according to the probability distribution information of the plurality of first training samples, Used to implement:

generating a plurality of speech sequence sets according to the plurality of candidate speech sequences, wherein each of the speech sequence sets includes at least one of the candidate speech sequences;

determining the probability distribution information of each said set of speech sequences;

According to the probability distribution information of the plurality of first training samples and the probability distribution information of each of the speech sequence sets, a target speech sequence set is selected from the plurality of the speech sequence sets.
The computer device of claim 11, wherein the processor implements the probability distribution information according to the plurality of first training samples and the probability distribution information of each of the speech sequence sets, from a plurality of the When selecting the target speech sequence set from the described speech sequence set, it is used to realize:

Calculate the K-L divergence of each of the speech sequence sets according to the probability distribution information of the plurality of first training samples and the probability distribution information of each of the speech sequence sets;

According to the K-L divergence of each said voice sequence set, a target voice sequence set is selected from a plurality of said voice sequence sets.
The computer device according to any one of claims 9-12, wherein the processor performs iterative training on the first preset speech recognition model according to the plurality of first training samples to obtain the first A speech recognition model is used to implement:

performing data enhancement on a plurality of the first training samples;

The first preset speech recognition model is iteratively trained according to the plurality of first training samples after data enhancement, until the first preset speech recognition model converges, and a first speech recognition model is obtained.
The computer device according to any one of claims 9-12, wherein, when the computer program is executed by the processor, the following steps are further implemented:

Get the speech sequence to be recognized;

Perform speech recognition on the speech sequence by using the target speech recognition model to obtain text information corresponding to the speech sequence.
A computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the following steps are implemented:

Obtain a plurality of first training samples and a plurality of second training samples, the first training samples include a first speech sequence and a first text corresponding to the marked first speech sequence, and the second training samples include a second speech sequence;

performing iterative training on a first preset speech recognition model according to the plurality of first training samples to obtain a first speech recognition model;

Integrating the first speech recognition model with the preset language model to obtain a second speech recognition model;

Inputting a plurality of the second speech sequences into the second speech recognition model to obtain a second text and a fusion score corresponding to each of the second speech sequences;

According to the fusion score of each of the second speech sequences, filter out the target speech sequence from the plurality of second speech sequences;

Perform iterative training on the second preset speech recognition model according to each of the target speech sequences, the second text corresponding to each of the target speech sequences, and a plurality of the first training samples to obtain a target speech recognition model.
16. The computer-readable storage medium of claim 15, wherein, when the processor implements the filtering of the target speech from the plurality of second speech sequences according to the fusion score of each of the second speech sequences sequence, used to implement:

Filtering a plurality of the second speech sequences according to a preset score threshold and a fusion score of each of the second speech sequences to obtain a plurality of candidate speech sequences;

According to the probability distribution information of the plurality of first training samples, a target speech sequence is selected from the plurality of candidate speech sequences.
The computer-readable storage medium according to claim 16, wherein, when the processor implements the selection of the target speech from the plurality of candidate speech sequences according to the probability distribution information of the plurality of first training samples sequence, used to implement:

generating a plurality of speech sequence sets according to the plurality of candidate speech sequences, wherein each of the speech sequence sets includes at least one of the candidate speech sequences;

determining the probability distribution information of each said set of speech sequences;

According to the probability distribution information of the plurality of first training samples and the probability distribution information of each of the speech sequence sets, a target speech sequence set is selected from the plurality of the speech sequence sets.
The computer-readable storage medium of claim 17, wherein the processor implements the probability distribution information according to the plurality of first training samples and the probability distribution information for each of the speech sequence sets, from When selecting a target voice sequence set from a plurality of said voice sequence sets, it is used to realize:

Calculate the K-L divergence of each of the speech sequence sets according to the probability distribution information of the plurality of first training samples and the probability distribution information of each of the speech sequence sets;

According to the K-L divergence of each of the speech sequence sets, a target speech sequence set is selected from a plurality of the speech sequence sets.
The computer-readable storage medium according to any one of claims 15-18, wherein, when the processor implements the iterative training of the first preset speech recognition model according to the plurality of first training samples , when the first speech recognition model is obtained, it is used to realize:

performing data enhancement on a plurality of the first training samples;

The first preset speech recognition model is iteratively trained according to the plurality of first training samples after data enhancement, until the first preset speech recognition model converges, and a first speech recognition model is obtained.
The computer-readable storage medium of any one of claims 15-18, wherein, when the computer program is executed by the processor, the following steps are further implemented:

Get the speech sequence to be recognized;

Perform speech recognition on the speech sequence by using the target speech recognition model to obtain text information corresponding to the speech sequence.