WO2024251169A1 - 语音识别方法、设备和存储介质 - Google Patents

语音识别方法、设备和存储介质 Download PDF

Info

Publication number
WO2024251169A1
WO2024251169A1 PCT/CN2024/097609 CN2024097609W WO2024251169A1 WO 2024251169 A1 WO2024251169 A1 WO 2024251169A1 CN 2024097609 W CN2024097609 W CN 2024097609W WO 2024251169 A1 WO2024251169 A1 WO 2024251169A1
Authority
WO
WIPO (PCT)
Prior art keywords
speaker
vector representation
decoder
word
speech recognition
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2024/097609
Other languages
English (en)
French (fr)
Inventor
史莫晗
杜志浩
陈谦
俞帆
张仕良
李泱泽
张结
戴礼荣
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba China Co Ltd
Original Assignee
Alibaba China Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba China Co Ltd filed Critical Alibaba China Co Ltd
Publication of WO2024251169A1 publication Critical patent/WO2024251169A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/08—Speech classification or search
    • G10L15/16—Speech classification or search using artificial neural networks
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/08—Speech classification or search
    • G10L15/18—Speech classification or search using natural language modelling
    • G10L15/183—Speech classification or search using natural language modelling using context dependencies, e.g. language models
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/20—Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/26—Speech to text systems

Definitions

  • the present invention relates to the field of artificial intelligence technology, and in particular to a speech recognition method, device and storage medium.
  • An end-to-end speaker-dependent speech recognition model can be used to recognize the speech of multiple speakers.
  • the existing end-to-end speaker-dependent speech recognition model cannot obtain speech recognition results with high accuracy.
  • the embodiments of the present invention provide a speech recognition method, a device and a storage medium, which are used to improve the accuracy of speech recognition results of multiple speakers.
  • an embodiment of the present invention provides a speech recognition method, wherein a speech recognition model includes a speech recognition encoder, a speaker encoder, a speech recognition decoder, a speaker decoder, and a text encoder including an attention layer, and the method includes:
  • the first vector representation, the first n-1 words and a weighted sum of the correlation coefficients to the portrait feature vectors of the multiple speakers are input into the speech recognition decoder to obtain the nth word.
  • an embodiment of the present invention provides a speech recognition device, wherein the speech recognition model includes a speech recognition encoder, a speaker encoder, a speech recognition decoder, a speaker decoder, and a text encoder including an attention layer, and the device includes:
  • An acquisition module is configured to acquire speech signals of multiple speakers and portrait feature vectors of the multiple speakers, wherein the speech signals include the speech of the multiple speakers;
  • a first encoding module configured to obtain a first vector representation corresponding to the speech signal through the speech recognition encoder, and to obtain a second vector representation corresponding to the speech signal through the speaker encoder, wherein the first vector representation is configured to perform speech recognition, and the second vector representation is configured to perform speaker recognition;
  • a second encoding module is configured to encode the first n-1 words output by the speech recognition decoder through the text encoder to obtain a third vector representation corresponding to the n-1th word;
  • a first decoding module is configured to input the first vector representation, the second vector representation and the third vector representation into the speaker decoder to obtain a speaker vector representation corresponding to the nth word; and determine a speaker corresponding to the nth word according to correlation coefficients between the speaker vector representation corresponding to the nth word and the portrait feature vectors of the plurality of speakers;
  • the second decoding module is configured to input the first vector representation, the first n-1 words and the weighted sum of the correlation coefficients to the portrait feature vectors of the multiple speakers into the speech recognition decoder to obtain the nth word.
  • an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the speech recognition method as described in the first aspect.
  • an embodiment of the present invention provides a non-temporary machine-readable storage medium having executable code stored thereon.
  • the executable code When executed by a processor of an electronic device, the processor can at least implement the speech recognition method described in the first aspect.
  • an embodiment of the present invention provides a speech recognition method, the method comprising:
  • a request triggered by a terminal device by calling a speech recognition service wherein the request includes speech signals of multiple speakers and portrait feature vectors of the multiple speakers, and the speech signals include the speech of the multiple speakers;
  • the speech recognition model includes a speech recognition encoder, a speaker encoder, a speech recognition decoder, a speaker decoder, and a text encoder including an attention layer;
  • the speech recognition output information includes text sequences corresponding to each of the multiple speakers.
  • the speech recognition model includes a speech recognition encoder, a speaker encoder, a speech recognition decoder, a speaker decoder and a text encoder including an attention layer.
  • the speech signal can be first encoded by the speech recognition encoder in the speech recognition model to obtain a first vector representation set for speech recognition, and the speech signal can be encoded by the speaker encoder to obtain a second vector representation set for speaker recognition. Then, the first n-1 words output by the speech recognition decoder are encoded by the text encoder to obtain a third vector representation corresponding to the n-1th word.
  • the first vector representation, the second vector representation and the third vector representation are input into the speaker decoder to obtain the speaker vector representation corresponding to the nth word, and the speaker corresponding to the nth word is determined according to the correlation coefficients between the speaker vector representation corresponding to the nth word and the portrait feature vectors of multiple speakers.
  • the weighted sum of the first vector representation, the first n-1 words and the correlation coefficient to the portrait feature vectors of multiple speakers is input into the speech recognition decoder to obtain the nth word.
  • a text encoder including an attention layer is added to the speech recognition model.
  • the first n-1 words output by the speech recognition decoder are encoded by the text encoder, so that the context information of the entire sentence can be better aggregated to obtain a third vector representation containing context information corresponding to the n-1th word. That is, a vector representation that is more accurately set for speaker recognition can be obtained, thereby improving the accuracy of the speaker recognition results corresponding to each word.
  • FIG1 is a flow chart of a speech recognition method provided by an embodiment of the present invention.
  • FIG2 is a schematic diagram of the structure of a speech recognition model provided by an embodiment of the present invention.
  • FIG. 3 is a schematic diagram of the structure of a speech recognition encoder and a speech recognition decoder provided in an embodiment of the present invention
  • FIG4 is a flow chart of determining a speaker vector representation corresponding to an n-th word through a speaker decoder
  • FIG5 is a flow chart of another speech recognition method provided by an embodiment of the present invention.
  • FIG6 is a flow chart of obtaining the nth word through a speech recognition decoder
  • FIG7 is a schematic diagram of an application of a speech recognition method provided by an embodiment of the present invention.
  • FIG8 is a flow chart of another speech recognition method provided by an embodiment of the present invention.
  • FIG9 is a schematic diagram of a speech recognition process in a cloud service mode provided by an embodiment of the present invention.
  • FIG10 is a schematic diagram of the structure of a speech recognition device provided by an embodiment of the present invention.
  • FIG. 11 is a schematic diagram of the structure of an electronic device provided in this embodiment.
  • the user information including but not limited to user device information, user personal information, etc.
  • data including but not limited to data used for analysis, stored data, displayed data, etc.
  • the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
  • SA-ASR Speaker-Attributed Automatic Speech Recognition
  • Serialized Output Training When transcribing speech (i.e., converting speech into text), special delimiters are used to connect the texts spoken by different speakers, and the transcribed texts are arranged in the order in which the different speakers start speaking to generate a text sequence, and the text sequence is output.
  • a speech segment includes two speakers, speaker A said "How is the weather today", and speaker B said "The weather is good today”. Then when performing serialized output training on the speech segment, the output will be: How is the weather today ⁇ sc>The weather is good today, and each word or sentence is associated with the corresponding speaker label.
  • Speaker Profile refers to a set of feature vectors, which contains the feature vectors of the profiles of multiple speakers.
  • the feature vectors of the profiles of each speaker can be extracted from a speech signal of each speaker.
  • End-to-End Different from modular processing of the target task, the target task is processed by a complete model, that is, the input is the original data and the output is the final processing result.
  • the overall objective function can be directly optimized during learning and training.
  • the existing end-to-end speaker-dependent speech recognition model mainly includes speech recognition encoder, speaker encoder
  • the model lacks a context information perception module.
  • the context information corresponding to each word is not fully considered, so that the predicted speaker corresponding to each word is inaccurate, thereby affecting the final speech recognition result.
  • an embodiment of the present invention provides a new speech recognition scheme.
  • a text encoder including an attention layer is added to the speech recognition model.
  • the text encoder can encode the first n-1 words that have been output by the speech recognition decoder to integrate the context information of the sentence, so that the speaker vector representation corresponding to the nth word is more accurate, so that the speaker encoder can more accurately predict the speaker corresponding to each word.
  • FIG. 1 is a flow chart of a speech recognition method provided by an embodiment of the present invention. As shown in FIG. 1 , the method includes the following steps:
  • the speech recognition solution provided by the embodiment of the present invention can use a pre-trained speech recognition model to process the speech signals of multiple speakers to be recognized, so as to obtain each word corresponding to the speech signal and the speaker corresponding to each word. Since the training process of the speech recognition model is similar to the process of using the speech recognition model to recognize the speech signals of multiple speakers to be recognized, the process of using the speech recognition model is described here.
  • the structure of the speech recognition model can be referred to as shown in FIG2.
  • the speech recognition model mainly includes a speech recognition encoder, a speaker encoder, a speech recognition decoder, a speaker decoder, and a text encoder including an attention layer.
  • the speech recognition encoder is mainly configured to convert speech signals of multiple speakers into a first vector representation configured to perform speech recognition.
  • the speaker encoder is mainly configured to convert speech signals of multiple speakers into a second vector representation configured to perform speaker recognition.
  • the speech recognition decoder is mainly configured to complete speech recognition. Speech-to-text recognition is used to identify all the words spoken by each speaker in the speech signal, and finally output the speech recognition results corresponding to each speaker.
  • the speaker decoder is mainly configured to determine the correspondence between each word and the speaker.
  • the text encoder is mainly configured to integrate the context information of the text, and determine the speaker vector representation corresponding to each word according to the context information corresponding to each word, so that the speaker vector representation corresponding to each word is more accurate.
  • the specific structure of the speech recognition encoder and the speech recognition decoder can refer to that shown in FIG3.
  • the speech recognition encoder can include multiple cascaded encoders, and each encoder can include two sublayers: an attention layer and a feedforward neural network layer.
  • the speech recognition decoder can include multiple cascaded decoders, and each decoder includes an attention layer and a feedforward neural network layer.
  • the structure of the speech encoder is similar to that of the speech recognition encoder, and reference can be made to the specific structure shown in FIG3.
  • the speaker encoder can include multiple cascaded encoders, and each encoder includes an attention layer and a feedforward neural network layer.
  • the structure of the speech decoder is similar to that of the speech recognition decoder, and the speaker decoder includes multiple cascaded decoders, and each decoder includes an attention layer and a feedforward neural network layer.
  • the text encoder includes at least one cascaded encoder, and each encoder includes an attention layer and a feedforward neural network layer.
  • the number of encoders included in the speech recognition encoder and the speaker encoder can be set according to actual needs, and is not limited here.
  • the number of decoders included in the speech recognition decoder and the speaker decoder can be set according to actual needs.
  • the attention layer in the decoder here can include a self-attention layer and a source-target attention layer.
  • the speech recognition model mainly includes a speech recognition encoder, a speaker encoder, a speech recognition decoder, a speaker decoder and a text encoder including an attention layer. Then, when the speech recognition model is used to perform speech recognition on the speech signals of multiple speakers, the specific execution process of each encoder and decoder in the speech recognition model is as follows: First, the speech signals of multiple speakers and the portrait feature vectors of multiple speakers are obtained. Among them, the speech signal contains the speech of multiple speakers.
  • the speaker's portrait feature vector refers to the speaker's feature vector, and the portrait feature vector corresponding to the speaker can be extracted from a segment of the speaker's speech signal.
  • the portrait features of the multiple speakers may be extracted in advance to obtain the portrait feature vectors of each speaker.
  • the speech signals of each speaker may be encoded by an encoder having the same structure as the speaker encoder to obtain the portrait feature vectors of each speaker.
  • the speech signal may be processed to extract effective speech information and obtain an acoustic feature sequence corresponding to the speech signal.
  • the specific implementation process may include: segmenting the acquired speech signal containing multiple speakers, cutting the speech signal into frames, then extracting acoustic features from each frame of the speech signal, and generating an acoustic feature sequence corresponding to the speech signal based on the acoustic features corresponding to each frame of the speech signal.
  • the acoustic feature sequence corresponding to the speech signal is input into the speech recognition encoder to obtain the first vector representation corresponding to the speech signal through the speech recognition encoder.
  • the first vector representation is set to perform speech recognition, that is, the first vector representation can characterize the speech features of the speech signal.
  • the corresponding acoustic feature sequence is input into the speaker encoder to obtain the second vector representation corresponding to the speech signal through the speaker encoder.
  • the second vector representation is set to perform speaker recognition, that is, the second vector representation can be set to represent the timbre and other features of the speaker of the speech signal.
  • the speech recognition encoder can convert the acoustic feature sequence corresponding to the speech signal into a vector representation that can be used for speech recognition, so that the speech recognition process can be performed directly based on the first vector representation to obtain each word corresponding to the speech signal.
  • the speaker encoder can convert the acoustic feature sequence corresponding to the speech signal into a vector representation for speaker recognition, so that the speaker recognition process corresponding to each word can be performed directly based on the second vector representation.
  • prediction is performed word by word. For example, after the speech recognition decoder determines the first word corresponding to the speech signal, the first word regresses to the input end of the speech recognition decoder and is set to determine the second word corresponding to the speech signal. The words corresponding to the speech signal are determined in sequence until the end symbol is encountered. Since the specific implementation process is basically the same when predicting each word in the speech signal, the prediction process of the nth word is used as an example here, assuming that the speech recognition decoder has output the first n-1 words. It should be noted that the first recognized word is predicted based on the set starting character or empty character as the input of the speech recognition decoder.
  • the text encoder When determining the nth word, we must first obtain the third vector representation containing context information corresponding to the n-1th word, that is, determine the nth word in combination with the information of the first n-1 words, so that the determination of the nth word is more accurate. Since the text encoder is mainly configured to integrate the context information corresponding to each word to obtain a vector representation containing context information, after obtaining the first n-1 words output by the speech recognition decoder, the text encoder encodes the first n-1 words output by the speech recognition decoder to obtain the third vector representation corresponding to the n-1th word.
  • the text encoder determines the third vector representation corresponding to the n-1th word, it first obtains the first n-1 words that the speech recognition encoder finally outputs after various processing, and encodes the first n-1 words that have been determined, so as to obtain the encoding result corresponding to the n-1th word from the encoding result as the third vector representation corresponding to the n-1th word.
  • the third vector representation corresponding to the 4th word is to be obtained, the first 3 words that the speech recognition encoder has output are first obtained, and the first 3 words are encoded to obtain the third vector representation corresponding to the 4th word.
  • the specific implementation process of encoding the first n-1 words that have been output by the speech recognition decoder through the text encoder to obtain the third vector representation corresponding to the n-1th word may include: obtaining the weighted vector representations of the attention coefficients corresponding to the first n-1 words obtained after the self-attention layer in the speech recognition decoder performs attention calculations on the first n-1 words respectively; and inputting the weighted vector representations of the attention coefficients corresponding to the first n-1 words into the text encoder to obtain the third vector representation corresponding to the n-1th word output by the text encoder.
  • what is input into the text editor is the weighted encoded vectors corresponding to the first n-1 words output by the self-attention layer in the speech recognition decoder.
  • the speech recognition decoder may include at least one decoder, each of which may include Including an attention layer, and having several different attention layers, such as a self-attention layer and a source-target attention layer.
  • the self-attention layer in the above-mentioned speech recognition decoder can be the first layer of the speech recognition decoder, that is, the self-attention layer contained in the first decoder. At this time, it is necessary to collect the results of the self-attention layer in the first layer of the speech recognition encoder for the first n-1 words, respectively, to form a sequence, and send it to the text encoder.
  • the text encoder will output the third vector representation after the fusion context information corresponding to each word of the first n-1 words.
  • the third vector representation corresponding to the n-1th word is taken out from the third vector representation corresponding to the first n-1 words.
  • the reason why the text encoder can fuse text context information is because the text encoder includes an attention layer, such as an attention layer that adopts a self-attention mechanism or other attention mechanisms. In this article, the self-attention layer is taken as an example.
  • the self-attention layer in the first layer performs attention calculation on the first n-1 words respectively, which can be expressed as the following formula: in, Represents the weighted vector representation of the attention coefficient corresponding to the n-1th word output by the first layer of the speech recognition decoder, represents the embedding vector representation corresponding to the n-1th word in the first layer of the input speech recognition decoder, represents the multi-head attention computation performed by the self-attention layer in the first layer of the speech recognition decoder, The embedding vector representations corresponding to the first n-1 words in the first layer of the input speech recognition decoder are represented.
  • the embedding vector representation is set to represent the vector representation after the embedding coding process, and the embedding vector representation can be obtained by embedding coding or embedding coding and position coding the first n-1 words respectively to obtain the embedding vector representations corresponding to the first n-1 words.
  • the formula can be used. Get the embedding vector representation corresponding to the first n-1 words.
  • PosEnc represents position encoding
  • Embed represents embedding encoding
  • y [1:n-1] represents the first n-1 words output by the speech recognition decoder.
  • the text editor After obtaining the weighted vector representations of the attention coefficients corresponding to the first n-1 characters, the text editor processes the weighted vector representations of the attention coefficients corresponding to the first n-1 characters to obtain a third vector representation corresponding to the n-1th character.
  • the text editor may process the weighted vector representations of the attention coefficients corresponding to the first n-1 characters according to the formula Calculate the third vector representation corresponding to the n-1th word. represents the third vector representation corresponding to the n-1th word, Context-Enc represents the context text encoding, Represents the weighted vector of attention coefficients corresponding to the first n-1 words.
  • the first vector representation, the second vector representation and the third vector representation are input into the speaker decoder to obtain the speaker vector representation corresponding to the nth word.
  • the speaker vector representation can be used to characterize the characteristics of the speaker.
  • the speech recognition features, speaker recognition features, and context information features of the speech signal are combined, so that the speaker vector representation corresponding to the nth word determined is of higher quality, thereby improving the accuracy of predicting the speaker corresponding to each word. This makes the transcription results corresponding to the speech signal more accurate.
  • the speaker corresponding to the nth word is then determined based on the correlation coefficients between the speaker vector representation corresponding to the nth word and the portrait feature vectors of multiple speakers.
  • the correlation coefficient is set to characterize the correlation between the speaker vector representation corresponding to each word and the portrait feature vectors of multiple speakers, that is, the correlation coefficient is set to represent the correlation between the nth word and each speaker contained in the speech signal.
  • the speaker corresponding to the nth word when determining the speaker corresponding to the nth word, first obtain the correlation coefficients between the speaker vector representation corresponding to the nth word and the portrait feature vectors of multiple speakers, and then determine the speaker corresponding to the nth word based on the correlation coefficients between the speaker vector representation corresponding to the nth word and the portrait feature vectors of multiple speakers.
  • the speaker vector corresponding to the nth word is represented by qn
  • the portrait feature vector of the first speaker is d1
  • the portrait feature vector of the second speaker is d2
  • the portrait feature vector of the third speaker is d3
  • the correlation coefficient ⁇ n,1 between the speaker vector representation qn corresponding to the nth word and d1 is obtained
  • the correlation coefficient ⁇ n,2 between the speaker vector representation qn corresponding to the nth word and d2 is obtained
  • the correlation coefficient ⁇ n,3 between the speaker vector representation qn corresponding to the nth word and d3 is obtained, and then the speaker corresponding to the nth word is determined based on these three correlation coefficients.
  • the speaker corresponding to the maximum correlation coefficient can be selected from multiple correlation coefficients to be determined as the speaker corresponding to the nth word.
  • the weighted sum of the multiple correlation coefficients for the portrait feature vectors of multiple speakers is calculated, where the weighted sum can be regarded as the weighted speaker vector representation corresponding to the nth word.
  • the formula Calculate the weighted sum of multiple correlation coefficients for multiple speaker portrait feature vectors in represents the weighted sum
  • K represents the total number of speakers contained in the speech signal
  • ⁇ n ,k represents the correlation coefficient between the speaker vector corresponding to the nth word and the portrait feature vector of the kth speaker
  • dk represents the portrait feature vector of the Kth speaker.
  • the weighted sum of the first vector representation, the first n-1 words, and the correlation coefficients for the portrait feature vectors of multiple speakers is input into the speech recognition decoder to obtain the nth word.
  • the weighted sum is the weighted speaker vector representation corresponding to the nth word.
  • the weighted speaker vector representation corresponding to the nth word is introduced when determining the nth word.
  • the nth word obtained in this way includes not only the specific content of the word but also the speaker corresponding to the word. For example, the specific content of the output nth word is "I", and the corresponding speaker is "Xiao Li".
  • each word in the speech signal and the speaker corresponding to each word can be determined in turn, and finally the recognition result in SOT format corresponding to each speaker can be determined.
  • two speakers are identified in a speech signal, and the speaker corresponding to the recognized text "How is the weather today” is speaker a, and the speaker corresponding to the recognized text "The weather is good today” is speaker b, and the output content is: How is the weather today ⁇ sc>The weather is good today.
  • a text encoder including an attention layer is added to the speech recognition model.
  • the text encoder encodes the first n-1 words output by the speech recognition decoder, so that the context information of the entire sentence can be better aggregated to obtain a third vector representation containing context information corresponding to the n-1th word. That is, based on the first vector representation, the second vector representation and the third vector representation, a speaker vector representation that is more accurately set for speaker recognition can be obtained, thereby improving the accuracy of speaker recognition results corresponding to each word.
  • a speaker decoder includes multiple cascaded decoders.
  • the processing of the encoder corresponding to each layer is exemplarily described in conjunction with FIG. 4 .
  • FIG4 is a flow chart of determining a speaker vector representation corresponding to an n-th word through a speaker decoder; as shown in FIG4 , the speaker decoder includes a plurality of cascaded decoders, each of which includes an attention layer and a feedforward neural network layer, and the method includes the following steps:
  • the speaker decoder in the embodiment of the present invention includes multiple decoders, each of which is set at a different level. Each decoder performs decoding processing layer by layer according to the level, and the output of the decoder in the previous layer is used as the input of the decoder in the next layer.
  • each decoder includes an attention layer and a feedforward neural network layer, and the decoding processing is performed by two sublayers of the attention layer and the feedforward neural network layer respectively. First, the vector representation input to the decoder is transmitted to the attention layer for processing, and the processing result is input to the feedforward neural network layer.
  • the feedforward neural network layer processes the processing result output by the attention layer, and the processed result is input to the attention layer in the decoder of the next layer, and the attention layer in the decoder of the next layer processes it, that is, each decoder in the speaker decoder is processed in sequence according to this method.
  • the processing process in each decoder may include: first, the first vector representation, the second vector representation and the third vector representation are input into the speaker decoder, so as to obtain the speaker vector representation weighted by the attention coefficient corresponding to the n-1th word through the attention layer in the first layer decoder. Then, the speaker vector representation weighted by the attention coefficient corresponding to the n-1th word is input into the feedforward neural network layer in the first layer decoder, so as to obtain the speaker vector representation corresponding to the n-1th word input into the second layer decoder through the feedforward neural network layer in the first layer decoder. In other words, the output result of the feedforward neural network layer in the first layer decoder (the speaker vector representation corresponding to the n-1th word) is used as the input of the attention layer in the second layer decoder.
  • the attention layer in the first layer decoder can be calculated according to the formula Calculate the speaker vector representation weighted by the attention coefficient corresponding to the n-1th word.
  • the feedforward neural network layer in the first decoder can be calculated according to the formula Calculate the speaker vector representation corresponding to the n-1th word.
  • the speaker vector representation corresponding to the n-1th word input to the second layer decoder represents the speaker vector representation weighted by the attention coefficient corresponding to the n-1th word output by the attention layer of the first decoder, Represents the first layer of a feed-forward neural network in the speaker encoder.
  • the attention layers in the second layer decoder and the decoders above the second layer include a self-attention layer and a source-target attention layer. Then, after obtaining the speaker vector representation corresponding to the n-1th word input to the second layer decoder through the feedforward neural network layer in the first layer decoder, it is output to the self-attention layer in the second layer decoder, and the self-attention layer in the second layer decoder performs self-attention calculation on the speaker vector representation corresponding to the n-1th word and the speaker vector representation corresponding to the first n-1 words, and obtains the vector representation corresponding to the n-1th word corresponding to the self-attention layer in the second layer decoder after the self-attention coefficient weighted, and inputs the vector representation corresponding to the n-1th word after the self-attention coefficient weighted to the source-target attention layer in the second layer decoder.
  • the source-target attention layer in the second layer decoder performs attention calculation on the vector representation weighted by the self-attention coefficient corresponding to the n-1th word and the second vector representation to obtain the vector representation weighted by the attention coefficient corresponding to the n-1th word corresponding to the source-target attention layer in the second layer decoder, and inputs the vector representation weighted by the attention coefficient corresponding to the n-1th word to the feedforward neural network layer in the second layer decoder.
  • the feedforward neural network layer in the second decoder processes the vector representation weighted by the attention coefficient corresponding to the n-1th word to obtain the speaker vector representation corresponding to the n-1th word input to the third encoder.
  • the decoders above the second layer perform calculations and processing respectively according to the above method until the speaker vector representation corresponding to the n-1th word output by the last layer decoder in the speaker decoder is determined, and the speaker vector representation corresponding to the n-1th word is determined based on the speaker vector representation corresponding to the n-1th word output by the last layer decoder and the speaker vector representation weighted by the attention coefficient corresponding to the n-1th word output by the attention layer in the first layer decoder.
  • the detailed processing process is referred to the above description and will not be repeated here.
  • the self-attention layers in the second layer decoder and the second layer and above decoders can be calculated according to the formula Calculate the weighted vector representation of the self-attention coefficient corresponding to the n-1th word of the source-target attention layer input to the decoder of this layer.
  • l represents the number of layers corresponding to the encoder, and l>1, represents the n-1th output of the self-attention layer in the lth encoder
  • the weighted vector representation of the self-attention coefficient corresponding to the word represents the speaker vector representation corresponding to the n-1th word output by the feedforward neural network layer in the l-1th layer encoder, represents the multi-head attention of the self-attention layer in the l-th encoder layer, Represents the speaker vector representation corresponding to the first n-1 words output by the feedforward neural network layer in the l-1th encoder layer.
  • the source-target attention layer in the second layer decoder and the decoders above the second layer can be calculated according to the formula Calculate the weighted vector representation of the attention coefficient corresponding to the n-1th word of the feedforward neural network layer input to the decoder of this layer.
  • l represents the number of layers corresponding to the encoder, and l>1, represents the weighted vector representation of the attention coefficient corresponding to the n-1th word output by the source-target attention layer in the l-th encoder (i.e., the weighted vector representation of the attention coefficient corresponding to the n-1th word input to the feedforward neural network layer in the l-th encoder), It represents the weighted vector representation of the self-attention coefficient corresponding to the n-1th word output by the self-attention layer in the lth layer encoder, represents the multi-head attention of the source-target attention layer in the l-th encoder layer, and H spk represents the second vector representation.
  • the feedforward neural network layers in the second layer decoder and the decoders above the second layer can be calculated according to the formula Calculate the speaker vector representation corresponding to the n-1th word input to the next layer of encoder.
  • l represents the number of layers corresponding to the encoder, and l>1
  • l represents the speaker vector representation corresponding to the n-1th word output by the feedforward neural network in the l-th layer encoder (i.e., the speaker vector representation corresponding to the n-1th word input to the l+1-th layer encoder)
  • the output of the feedforward neural network layer in each layer decoder is determined in sequence until the speaker vector representation corresponding to the n-1th word output by the last layer decoder in the speaker decoder is determined. Finally, the speaker decoder skips the speaker vector representation corresponding to the n-1th word output by the last layer decoder and the speaker vector representation weighted by the attention coefficient corresponding to the n-1th word output by the attention layer in the first layer decoder, and predicts the speaker vector representation corresponding to the nth word based on the skip-connection result.
  • the speaker vector representation corresponding to the n-1th word output by the last layer decoder in the speaker decoder is The speaker vector weighted by the attention coefficient corresponding to the n-1th word output by the attention layer in the first layer decoder is expressed as According to the formula Determine the speaker vector representation q n corresponding to the nth word, where W q represents the weight set in the speaker encoder.
  • the speaker decoder includes a multi-layer decoder
  • the first layer decoder processes the first vector representation, the second vector representation and the third vector representation containing context information to obtain the first layer decoder.
  • the attention layer in the decoder outputs the speaker vector representation weighted by the attention coefficient corresponding to the n-1th word, that is, the first layer decoder combines the context information when processing, so that the speaker representation weighted by the attention coefficient corresponding to the n-1th word is more accurate.
  • the feedforward neural network layer in the first layer decoder processes the speaker vector weighted by the attention coefficient corresponding to the n-1th word obtained after processing in combination with the context information, so that the speaker vector representation corresponding to the n-1th word input to the second layer decoder has a higher quality and can better reflect the speaker characteristics corresponding to the n-1th word. Then, the speaker vector representation corresponding to each word with higher quality can be obtained through the processing of multiple layers of decoders in turn, thereby making the speaker results predicted by each word more accurate.
  • the speaker corresponding to each character can be further determined based on the speaker vector representation corresponding to each character.
  • the speaker corresponding to the nth character is determined based on the correlation coefficient between the speaker vector representation corresponding to the nth character and the portrait feature vectors of multiple speakers.
  • the correlation coefficient between the speaker vector representation corresponding to the nth character and the portrait feature vectors of multiple speakers is first determined.
  • the specific process of determining the correlation coefficient between the speaker vector representation corresponding to the nth character and the portrait feature vectors of multiple speakers is exemplified in conjunction with Figure 5.
  • FIG5 is a flow chart of another speech recognition method provided by an embodiment of the present invention. As shown in FIG5 , in order to improve the accuracy of the speech recognition result of the method, based on the above embodiment, the method may further include the following steps:
  • a correlation coefficient between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker is determined by jointly scoring using a set similarity algorithm and a set scoring model including an attention module.
  • the correlation coefficient can be set to represent the posterior probability of the nth word corresponding to each speaker.
  • the first similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker is determined according to the set similarity algorithm. That is to say, according to the set similarity algorithm, the first similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of each speaker can be determined respectively.
  • the set similarity algorithm can be a cosine similarity algorithm, for example, according to the formula The first similarity between the speaker vector corresponding to the nth word and the portrait feature vectors of each speaker can be determined.
  • the speaker vector representation corresponding to the nth word is respectively calculated with the portrait feature vector of each speaker to obtain the probability that the nth word corresponds to each speaker.
  • the determination of the first similarity only considers the speaker vector representation corresponding to the nth word, and does not consider the speaker vector representation corresponding to the first n words. Therefore, the determination of the first similarity is independent of other speakers.
  • the speaker vector representations of other speakers are fully combined. Specifically, by adding a scoring model containing an attention module to the speech recognition model, the second similarity between the speaker vector representation corresponding to each of the first n words and the portrait feature vector of any speaker is obtained through the scoring model containing the attention module.
  • the specific implementation process of determining the second similarity may be: obtaining the speaker vector representation corresponding to each of the first n words, and splicing the speaker vector representation corresponding to each of the first n words with the portrait feature vector of any speaker. Inputting the multiple spliced vector representations into the set scoring model including the attention module to obtain the second similarity between the speaker vector representation corresponding to each of the first n words and the portrait feature vector of any speaker.
  • the formula Determine the second similarity between the speaker vector representation corresponding to each of the first n words and the portrait feature vector of each speaker represents the second similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of the kth speaker
  • tanh is the hyperbolic tangent function
  • CD-scorer represents the set scoring model including the attention module.
  • q [1:n] represents the speaker vector representation corresponding to the first n words
  • d k represents the portrait feature vector of the kth speaker.
  • the second similarity between the speaker vector representation corresponding to the nth word and the portrait feature vectors of each speaker can be generated according to this formula.
  • the implementation method for determining the second similarity can be: obtain the speaker vector representation corresponding to each of the first n words, and concatenate the speaker vector representation corresponding to each of the first n words with the kth speaker feature vector to obtain a first vector sequence with a length of n. And input the first vector sequence into the scoring model (for example, using the structure of the transfomer model). In the structure), a second vector sequence is obtained.
  • the second vector sequence is a sequence of values between [-1, 1] obtained after tanh transformation, and the second similarity between the speaker vector corresponding to the last determined nth word and the portrait feature vector of the kth speaker is taken out. Tanh is an example of a mapping function, which is not limited to this.
  • the correlation coefficient between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker is determined according to the first similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker and the second similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker.
  • the first similarity and the second similarity are calculated for each speaker, the first similarity and the second similarity between the nth word and each speaker are added to obtain a first sum value, and the first sum value is divided by the sum of the first similarity and the second similarity between the nth word and each speaker to obtain the correlation coefficient between the speaker vector representation corresponding to the nth word and the portrait feature vector of each speaker.
  • the formula Determine the correlation coefficient between the speaker vector representation corresponding to the nth word and the portrait feature vector of each speaker.
  • ⁇ n,k represents the correlation coefficient between the speaker vector representation corresponding to the nth word and the portrait feature vector of the kth speaker
  • exp represents an exponential function with e as the base.
  • K is the total number of speakers included in the speech signal
  • j is the jth speaker.
  • k in the above can be any one of the K speakers, then the correlation coefficient between the speaker vector representation corresponding to the nth word and the portrait feature vector of each speaker can be obtained according to the above formula.
  • the speaker corresponding to the nth word can be determined based on the correlation coefficients between the speaker vector representation corresponding to the nth word and the portrait feature vectors of the multiple speakers.
  • a first similarity between the speaker vector representation corresponding to the nth word and a portrait feature vector of any speaker is determined according to a set similarity algorithm, and a second similarity between the speaker vector representation corresponding to each of the first n words and a portrait feature vector of any speaker is obtained through a set scoring model including an attention module, and then a correlation coefficient between the speaker vector representation corresponding to the nth word and a portrait feature vector of any speaker is determined according to the first similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker and the second similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker, that is, the correlation coefficient determined by combining the set similarity algorithm and the set scoring model including the attention module is more accurate, so that the speech recognition model has better performance, thereby further improving the accuracy of the speech recognition result.
  • the above embodiment introduces the method of determining the speaker vector representation corresponding to the nth word and the speaker vector representation corresponding to the plurality of speakers.
  • the specific implementation method of the correlation coefficient between the portrait feature vectors is to calculate the weighted sum of the correlation coefficient to the portrait feature vectors of the multiple speakers after determining the correlation coefficients between the speaker vector representation corresponding to the nth word and the portrait feature vectors of the multiple speakers. Specifically, the product values of the multiple correlation coefficients and the portrait feature vectors of the respective speakers are obtained, and then the multiple product values are summed to obtain the weighted speaker feature vector corresponding to the nth word.
  • a speech recognition decoder includes multiple cascaded decoders.
  • the processing process of the encoder corresponding to each layer is exemplarily illustrated in conjunction with Figure 6.
  • FIG6 is a flow chart of obtaining the nth word through a speech recognition decoder; as shown in FIG6 , the speech recognition decoder includes a plurality of cascaded decoders, each of which includes an attention layer and a feedforward neural network layer, and the method includes the following steps:
  • the weighted vector representation of the attention coefficient corresponding to the n-1th word and the weighted sum are input into the feedforward neural network layer in the first layer decoder to obtain the vector representation corresponding to the n-1th word input into the second layer decoder.
  • the speech recognition decoder in the embodiment of the present invention includes a multi-layer decoder, and each layer of the decoder performs decoding processing layer by layer according to the level, and the output of the upper layer of the decoder is used as the input of the next layer of the decoder.
  • each decoder includes an attention layer and a feedforward neural network layer, and the two sublayers process the vector representation input to the encoder respectively.
  • the function of the speech decoder is to predict each word one by one in an autoregressive manner.
  • the nth word when performing the nth iteration, it is set to predict the nth word corresponding to the input speech signal, and the nth word is predicted according to the weighted sum of the weighted vector representation of the attention coefficient corresponding to the first n-1 words of the input, the n-1 word, and the portrait feature vectors of multiple speakers by the correlation coefficient.
  • the predicted nth word is obtained according to the probability of the nth word corresponding to each word in the dictionary V. Assuming that there are 4950 words in the dictionary, determine the probability on of the nth word corresponding to each word in the dictionary V, that is, on is 4950 probabilities, and select the word with the largest probability among the 4950 probabilities obtained as the prediction result of the nth word.
  • embedding encoding is performed on the first n-1 words respectively to obtain the embedding vector representation corresponding to each of the first n-1 words.
  • embedding processing can be performed on the first n-1 words respectively to obtain the processed vector representation
  • position encoding processing can be performed on the processed vector representation to obtain the embedding vector representation corresponding to each of the first n-1 words.
  • PosEnc represents the position encoding
  • Embed represents the embedding function
  • y [1:n-1] represents the first n-1 words that the speech recognition decoder has output.
  • the embedded vector representations and the first vector representation corresponding to the first n-1 characters are input into the attention layer in the first decoder.
  • the attention layer in the first decoder processes the embedded vector representations and the first vector representation corresponding to the first n-1 characters to obtain the vector representation weighted by the attention coefficient corresponding to the n-1th character corresponding to the attention layer in the first decoder, and inputs it into the feedforward neural network layer in the first decoder.
  • the attention layer in each decoder in the speech recognition encoder may include a self-attention layer and a source-target attention layer. Then, after the speaker encoder receives the embedding vector representation and the first vector representation corresponding to the first n-1 words, the embedding vector representation corresponding to the first n-1 words is input into the self-attention layer in the first layer decoder to obtain the vector representation weighted by the self-attention coefficient corresponding to the n-1th word output by the self-attention layer in the first layer decoder, and the vector representation is input into the source-target attention layer in the first layer decoder.
  • the source-target attention layer in the first layer decoder calculates the vector representation weighted by the self-attention coefficient corresponding to the n-1th word and the first vector representation to obtain the vector representation weighted by the attention coefficient corresponding to the n-1th word output by the source-target attention layer. Then, the weighted sum of the vector representation weighted by the attention coefficient corresponding to the n-1th word and the correlation coefficient to the portrait feature vectors of multiple speakers is input into the feedforward neural network layer in the first layer decoder to obtain the vector representation corresponding to the n-1th word input into the second layer decoder. According to the above method, the processing operations of the second layer and the decoders above the second layer are completed respectively until the vector representation corresponding to the n-1th word output by the last layer decoder in the speech recognition decoder is obtained.
  • the self-attention layer in the first layer and above decoders can be calculated according to the formula Calculate the weighted vector representation of the self-attention coefficient corresponding to the n-1th word of the source-target attention layer input to the decoder of this layer.
  • l represents the number of layers corresponding to the encoder
  • l represents the weighted vector representation of the self-attention coefficient corresponding to the n-1th word output by the self-attention layer in the l-th encoder
  • the weighted vector representation of the self-attention coefficient corresponding to the n-1th word input to the source-target attention layer in the l-th encoder represents the vector representation of the n-1th word output by the feedforward neural network layer in the l-1th encoder layer
  • the source-target attention layer in the first and higher decoders can be calculated according to the formula Calculate the weighted vector representation of the attention coefficient corresponding to the n-1th word of the feedforward neural network layer input to the decoder of this layer.
  • l represents the layer corresponding to the encoder Number, and represents the weighted vector representation of the attention coefficient corresponding to the n-1th word output by the source-target attention layer in the l-th encoder (i.e., the weighted vector representation of the attention coefficient corresponding to the n-1th word input to the feedforward neural network layer in the l-th encoder), It represents the weighted vector representation of the self-attention coefficient corresponding to the n-1th word output by the self-attention layer in the lth layer encoder, denotes the multi-head attention of the source-target attention layer in the l-th encoder layer, and Hasr denotes the first vector representation.
  • the feedforward neural network layer in the first layer decoder can be calculated according to the formula Calculate the vector representation corresponding to the n-1th word input to the second layer encoder.
  • l represents the number of layers corresponding to the encoder
  • W spk is the weight corresponding to the speaker encoder
  • the formula Calculate the vector representation corresponding to the n-1th word input to the next layer of encoder (that is, the output of the feedforward neural network layer is used as the input of the next layer of encoder).
  • l represents the number of layers corresponding to the encoder, and l>1.
  • the nth word is determined based on the vector representation corresponding to the n-1th word output by the last layer of the decoder in the speech recognition decoder. Specifically, the vector representation corresponding to the n-1th word output by the last layer of the decoder is processed using the softmax operation to obtain the posterior probability o n corresponding to the nth word, and then the nth word is predicted based on the posterior probability o n corresponding to the nth word.
  • the vector representation corresponding to the n-1th word output by the last layer of the decoder is According to the formula Determine the posterior probability o n corresponding to the nth word, where softmax is the activation function, and W o and b o are the weights corresponding to the speech recognition encoder.
  • the first n-1 words that have been output are embedded and encoded.
  • the embedded vector representation of the n-1th word is taken out as the query in the self-attention layer, and the embedded vector representations corresponding to the n-1 words are used as key and value for self-attention processing to obtain the result of the n-1th word paying attention to the first n-1 words, that is, the encoding vector weighted by the self-attention coefficient corresponding to the n-1th word, and then added to the embedded vector representation of the n-1th word before weighting.
  • the addition result is the weighted vector representation of the self-attention coefficient corresponding to the n-1th word, and it is used as one of the inputs of the source-target attention layer.
  • the other two inputs are two first vector representations, and finally the weighted vector representation of the attention coefficient corresponding to the n-1th word output by the source-target attention layer in the first layer is obtained.
  • the weighted vector representation of the attention coefficient corresponding to the n-1th word is input into the feedforward neural network layer in the first layer encoder to obtain the first layer encoder.
  • the output of the encoder is the input of the second layer encoder.
  • the output is calculated with This item is not added when the feedforward neural network layer in other layers of encoder is processed.
  • the feedforward neural network layer in the first layer of encoder introduces The purpose is to introduce the weighted speech vector representation of the nth word corresponding to each speaker in the process of speech recognition, so that the recognized nth word does not only include the specific content corresponding to the word but also includes the speaker corresponding to the word.
  • the speech recognition encoder can directly output the recognition results in SOT format, such as a text sequence of one speaker ⁇ sc> another speaker's text sequence.
  • the speech decoder predicts each word one by one in an autoregressive manner.
  • each layer of decoders in the speech encoder decodes the weighted sum of the portrait feature vectors of multiple speakers of the first n-1 words of the input, the weighted vector representation of the attention coefficient corresponding to the n-1th word, and the correlation coefficient layer by layer, and the output of the previous layer of decoders is used as the input of the next layer of decoders.
  • the nth word is determined according to the vector representation corresponding to the n-1th word output by the last layer of decoders in the speech recognition decoder.
  • the nth word obtained in this way does not contain the content corresponding to the word and the speaker corresponding to the word, so that the text content in the SOT format can be directly generated based on the predicted words in the future.
  • the context information is comprehensively considered when determining each word, so that the predicted words can be more accurate.
  • the speech recognition model includes a speech recognition encoder, a speaker encoder, a speech recognition decoder, a speaker decoder, a text encoder including an attention layer, and a joint scorer.
  • the joint scorer includes a speaker-independent scorer and a speaker-dependent scorer.
  • the speaker-independent scorer can determine the first similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker according to the set similarity algorithm.
  • the speaker-dependent scorer includes a set scoring model including an attention module, so as to obtain the second similarity between the speaker vector representation corresponding to each of the first n words and the portrait feature vector of any speaker through the set scoring model including the attention module.
  • each word corresponding to the speech signal is predicted one by one in an iterative manner.
  • the specific recognition prediction process of each word is basically the same, and the recognition prediction process of the nth word is used as an example here.
  • the speech signals of multiple speakers and the portrait feature vectors of multiple speakers are obtained, wherein the speech signals contain the speech of multiple speakers.
  • the speech signals of multiple speakers are input into the speech recognition encoder and the speaker encoder, and the first vector representation corresponding to the speech signal is obtained through the speech recognition encoder, and the second vector representation corresponding to the speech signal is obtained through the speaker encoder.
  • the first vector representation is set to perform speech recognition
  • the second vector representation is set to perform speaker recognition.
  • the first n-1 words that have been output by the speech recognition decoder are obtained, and the first n-1 words that have been output by the speech recognition decoder are encoded by the text encoder to obtain the third vector representation corresponding to the n-1th word.
  • the first vector representation, the second vector representation, and the third vector representation are input into the speaker decoder to obtain the speaker vector representation corresponding to the nth word.
  • the speaker vector representation corresponding to the nth word is spliced with the portrait feature vector of any speaker, and the spliced vector representation is input into the speaker-independent scorer.
  • the first similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker is determined according to the set similarity algorithm.
  • the speaker vector representation corresponding to each of the first n words is spliced with the portrait feature vector of any speaker, and the multiple spliced vector representations are input into the scoring model including the attention module set in the speaker-dependent scorer to obtain the second similarity between the speaker vector representation corresponding to each of the first n words and the portrait feature vector of any speaker.
  • the correlation coefficient between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker is determined. Then, the weighted sum of the correlation coefficient to the portrait feature vectors of multiple speakers is calculated.
  • the weighted sum of the first vector representation, the first n-1 words, and the correlation coefficients for the portrait feature vectors of multiple speakers is input into the speech recognition decoder to obtain the posterior probability corresponding to the nth word. Then, the nth word is predicted based on the posterior probability corresponding to the nth word. Finally, a text sequence in SOT format is generated based on each word output by the speech recognizer.
  • a two-stage decoding process is performed.
  • the first stage decoding is to predict each word corresponding to the speech signal one by one according to the method in each of the above-mentioned embodiments.
  • the second stage decoding process is performed.
  • the second stage decoding is to obtain the prediction results of all N words after the first stage decoding process, and use the prediction results of these N words as the input of the text encoder, and then calculate the correlation coefficients between the speaker vector representation corresponding to an nth word and the portrait feature vectors of multiple speakers one by one, that is, to obtain the probability distribution of the speaker corresponding to each word in the N words, and according to the probability distribution of the speaker corresponding to each word in the N words, to re-obtain the speaker corresponding to each word, so that the result of the N words finally obtained can be made more accurate.
  • Its specific implementation process can be shown in Figure 8.
  • FIG8 is a flow chart of another speech recognition method provided by an embodiment of the present invention. As shown in FIG8 , in order to improve the accuracy of the speech recognition result of the method so that the speech recognition method can better combine the global context information, based on the above embodiment, the method may further include the following steps:
  • the complete text sequence output by the first stage speech recognition decoder is first determined. Then the complete text sequence is input into the text encoder, so that the text encoder encodes the text sequence to obtain the fourth vector representation corresponding to each word in the text sequence. Then, the first vector representation, the second vector representation and the fourth vector representation are input into the speaker decoder to obtain the speaker vector representation corresponding to each word. According to the correlation coefficient between the speaker vector representation corresponding to each word and the portrait feature vectors of multiple speakers, the speaker corresponding to each word is re-determined.
  • the first stage decoding is to predict the nth word based on the first n-1 words, and only the context information corresponding to the word can be obtained, but no context information.
  • the second stage decoding is to predict again based on all N words after all N words have been predicted once, so that the context information and context information can be seen for each word, which can make the prediction results of the speaker corresponding to each word obtained by recalculating more accurate.
  • the speech recognition method provided by the embodiment of the present invention can be executed in the cloud, and several computing nodes (cloud servers) can be deployed in the cloud, and each computing node has computing, storage and other processing resources.
  • cloud multiple computing nodes can be organized to provide a certain service.
  • a computing node can also provide one or more services.
  • the cloud can provide the service by providing a service interface to the outside, and the user calls the service interface to use the corresponding service.
  • the cloud can provide a service interface for speech recognition service.
  • the user calls the service interface through the terminal device to trigger a speech recognition service request to the cloud.
  • the request includes speech signals of multiple speakers and portrait feature vectors of multiple speakers.
  • the speech signal contains the speech of multiple speakers.
  • the cloud determines the computing node that responds to the request and uses the processing resources in the computing node to perform the following steps:
  • the speech recognition model includes a speech recognition encoder, a speaker encoder, a speech recognition decoder, a speaker decoder, and a text encoder including an attention layer;
  • the speech recognition output information is sent to the terminal device, wherein the speech recognition output information includes text sequences corresponding to multiple speakers.
  • the user can call the speech recognition service through the terminal device E1 illustrated in Figure 9 to perform speech recognition on the target speech signal to obtain a text sequence prediction corresponding to the target speech signal.
  • the service interface for the user to call the service includes a software development kit (Software Development Kit, referred to as SDK), an application programming interface (Application Programming Interface, referred to as API), and the like.
  • Figure 9 illustrates the situation of the API interface.
  • the speech recognition service is provided by the service cluster E2 and the service cluster E2 includes at least one computing node.
  • the service cluster E2 executes the steps in the aforementioned embodiment to obtain a text sequence corresponding to the target speech signal, including multiple speakers respectively, and feeds it back to the terminal device E1.
  • FIG 10 is a structural schematic diagram of a speech recognition device provided in an embodiment of the present invention.
  • the speech recognition model includes a speech recognition encoder, a speaker encoder, a speech recognition decoder, a speaker decoder and a text encoder including an attention layer.
  • the device includes: an acquisition module 11, a first encoding module 12, a second encoding module 13, a first decoding module 14, and a second decoding module 15.
  • the acquisition module 11 is configured to acquire speech signals of multiple speakers and portrait feature vectors of the multiple speakers, wherein the speech signals include the speech of the multiple speakers.
  • the first encoding module 12 is configured to obtain a first vector representation corresponding to the speech signal through the speech recognition encoder, and to obtain a second vector representation corresponding to the speech signal through the speaker encoder, wherein the first vector representation is configured to perform speech recognition, and the second vector representation is configured to perform speaker recognition.
  • the second encoding module 13 is configured to encode the first n-1 words output by the speech recognition decoder through the text encoder to obtain a third vector representation corresponding to the n-1th word.
  • the first decoding module 14 is configured to input the first vector representation, the second vector representation and the third vector representation into the speaker decoder to obtain the speaker vector representation corresponding to the nth word; and determine the speaker corresponding to the nth word according to the correlation coefficients between the speaker vector representation corresponding to the nth word and the portrait feature vectors of the multiple speakers.
  • the second decoding module 15 is configured to input the first vector representation, the first n-1 words and the weighted sum of the correlation coefficients to the portrait feature vectors of the multiple speakers into the speech recognition decoder to obtain the nth word.
  • the first decoding module 14 is further configured to: determine, for the portrait feature vector of any speaker among the multiple speakers, a first similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker according to a set similarity algorithm; obtain the speaker vector representation corresponding to each of the first n words; and respectively compare the speaker vector representation corresponding to each of the first n words with the portrait feature vector of any speaker.
  • the invention relates to a method for obtaining a first speaker vector corresponding to each of the first n words and a portrait feature vector of any speaker, wherein the first speaker vector corresponding to the first n words is concatenated; the plurality of concatenated vector representations are input into a set scoring model including an attention module to obtain a second similarity between the speaker vector representation corresponding to each of the first n words and the portrait feature vector of any speaker; and the correlation coefficient between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker is determined according to the first similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker and the second similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker.
  • the second encoding module 13 is also configured to: obtain the weighted vector representations of the attention coefficients corresponding to the first n-1 words respectively obtained after the self-attention layer in the speech recognition decoder performs attention calculations on the first n-1 words respectively; input the weighted vector representations of the attention coefficients corresponding to the first n-1 words respectively into the text encoder to obtain a third vector representation corresponding to the n-1th word output by the text encoder.
  • the text encoder includes at least one cascaded encoder, each of which includes an attention layer and a feedforward neural network layer.
  • the speech recognition decoder includes a plurality of cascaded decoders, each of which includes an attention layer and a feedforward neural network layer; based on this, the second decoding module 15 is also configured to: perform embedding coding on the first n-1 characters respectively to obtain the embedding vector representations corresponding to the first n-1 characters respectively; input the embedding vector representations corresponding to the first n-1 characters respectively and the first vector representation into the first layer decoder of the speech recognition decoder to obtain the weighted vector representation of the attention coefficient corresponding to the n-1th character output by the attention layer in the first layer decoder; input the weighted vector representation of the attention coefficient corresponding to the n-1th character and the weighted sum into the feedforward neural network layer in the first layer decoder to obtain the vector representation corresponding to the n-1th character input to the second layer decoder; determine the nth character according to the vector representation corresponding to the n-1th character output by the last layer decoder in the speech recognition
  • the speaker decoder includes a plurality of cascaded decoders, each of which includes an attention layer and a feedforward neural network layer; based on this, the first decoding module 14 is further configured to: input the first vector representation, the second vector representation and the third vector representation into a first layer decoder in the speaker decoder to obtain a speaker vector representation weighted by the attention coefficient corresponding to the n-1th word output by the attention layer in the first layer decoder; input the speaker vector representation weighted by the attention coefficient corresponding to the n-1th word into the feedforward neural network layer in the first layer decoder to obtain a speaker vector representation corresponding to the n-1th word input to the second layer decoder; determine the speaker vector representation corresponding to the n-1th word output by the last layer decoder in the speaker decoder; and determine the speaker vector representation corresponding to the n-1th word according to the speaker vector representation corresponding to the n-1th word output by the last layer decoder and the speaker vector representation weighted by
  • the second decoding module 15 is further configured to: determine the complete text sequence output by the speech recognition decoder; encode the text sequence through the text encoder to obtain the text sequence; A fourth vector representation corresponding to each word in the column; inputting the first vector representation, the second vector representation and the fourth vector representation into the speaker decoder to obtain the speaker vector representation corresponding to each word; and re-determining the speaker corresponding to each word according to the correlation coefficients between the speaker vector representation corresponding to each word and the portrait feature vectors of the multiple speakers.
  • the device shown in FIG. 10 can execute the steps in the speech recognition method in the aforementioned embodiment.
  • the device shown in FIG. 10 can execute the steps in the speech recognition method in the aforementioned embodiment.
  • the embodiment of the present invention also provides an electronic device, as shown in FIG11 , which may include: a processor 21, a memory 22, and a communication interface 23.
  • the memory 22 stores executable codes, and when the executable codes are executed by the processor 21, the processor 21 implements the speech recognition method in the above embodiment.
  • an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored.
  • the processor can at least implement the speech recognition method provided in the aforementioned embodiment.
  • each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software.
  • the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
  • the embodiments of the present disclosure provide a speech recognition method, device and storage medium.
  • the specific implementation scheme is to add a text encoder including an attention layer to the speech recognition model.
  • the text encoder encodes the first n-1 words that have been output by the speech recognition decoder, so as to better aggregate the context information of the entire sentence to obtain a third vector representation containing context information corresponding to the n-1th word, that is, in this way, a vector representation that is more accurately set for speaker recognition can be obtained, thereby improving the accuracy of the speaker recognition results corresponding to each word.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)

Abstract

本公开提供一种语音识别方法、设备和存储介质，方法包括：通过语音识别编码器获取语音信号对应的第一向量表示，通过说话者编码器获取语音信号对应的第二向量表示；通过文本编码器对语音识别解码器已经输出的前n-1个字进行编码处理，得到第n-1个字所对应的第三向量表示；将第一向量表示、第二向量表示和第三向量表示输入说话者解码器中，获得第n个字对应的说话者向量表示；根据第n个字对应的说话者向量表示分别与多个说话者的画像特征向量之间的相关性系数，确定第n个字对应的说话者；将第一向量表示、前n-1个字以及相关性系数对多个说话者的画像特征向量的加权和，输入语音识别解码器，获得第n个字。

Description

语音识别方法、设备和存储介质
交叉援引
本公开要求于2023年06月08日提交中国专利局、优先权号为202310680032.X、发明名称为“语音识别方法、设备和存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本公开中。
技术领域
本发明涉及人工智能技术领域，尤其涉及一种语音识别方法、设备和存储介质。
背景技术
在很多场景中都会遇到多个说话者的语音识别问题，例如，在会议场景下，说话的人数可能为两人或者多人，需要同时对多个说话者的语音进行识别，以确定每个说话者说了什么内容。当前语音识别技术已经可以以较高精度识别一个人所讲的内容，但是当说话的人数为两人或者多人时，语音识别率就会极大的降低。
可以采用端到端说话人相关语音识别模型对多个说话者的语音进行识别。但是，现有的端到端说话人相关语音识别模型并不能获得准确性高的语音识别结果。
发明内容
本发明实施例提供一种语音识别方法、设备和存储介质，用以提高多个说话者的语音识别结果的准确性。
第一方面，本发明实施例提供一种语音识别方法，语音识别模型中包括语音识别编码器、说话者编码器、语音识别解码器、说话者解码器以及包含注意力层的文本编码器，所述方法包括：
获取多个说话者的语音信号和所述多个说话者的画像特征向量，所述语音信号中包含所述多个说话者的语音；
通过所述语音识别编码器获取所述语音信号对应的第一向量表示，通过所述说话者编码器获取所述语音信号对应的第二向量表示，所述第一向量表示被设置为进行语音识别，所述第二向量表示被设置为进行说话者识别；
通过所述文本编码器对所述语音识别解码器已经输出的前n-1个字进行编码处理，以得到第n-1个字所对应的第三向量表示；
将所述第一向量表示、所述第二向量表示和所述第三向量表示输入所述说话者解码器中，以获得第n个字对应的说话者向量表示；
根据所述第n个字对应的说话者向量表示分别与所述多个说话者的画像特征向量之间的相关性系数，确定所述第n个字对应的说话者；
将所述第一向量表示、所述前n-1个字以及所述相关性系数对所述多个说话者的画像特征向量的加权和，输入所述语音识别解码器，以获得所述第n个字。
第二方面，本发明实施例提供一种语音识别装置，语音识别模型中包括语音识别编码器、说话者编码器、语音识别解码器、说话者解码器以及包含注意力层的文本编码器，所述装置包括：
获取模块，被设置为获取多个说话者的语音信号和所述多个说话者的画像特征向量，所述语音信号中包含所述多个说话者的语音；
第一编码模块，被设置为通过所述语音识别编码器获取所述语音信号对应的第一向量表示，通过所述说话者编码器获取所述语音信号对应的第二向量表示，所述第一向量表示被设置为进行语音识别，所述第二向量表示被设置为进行说话者识别；
第二编码模块，被设置为通过所述文本编码器对所述语音识别解码器已经输出的前n-1个字进行编码处理，以得到第n-1个字所对应的第三向量表示；
第一解码模块，被设置为将所述第一向量表示、所述第二向量表示和所述第三向量表示输入所述说话者解码器中，以获得第n个字对应的说话者向量表示；根据所述第n个字对应的说话者向量表示分别与所述多个说话者的画像特征向量之间的相关性系数，确定所述第n个字对应的说话者；
第二解码模块，被设置为将所述第一向量表示、所述前n-1个字以及所述相关性系数对所述多个说话者的画像特征向量的加权和，输入所述语音识别解码器，以获得所述第n个字。
第三方面，本发明实施例提供一种电子设备，包括：存储器、处理器、通信接口；其中，所述存储器上存储有可执行代码，当所述可执行代码被所述处理器执行时，使所述处理器执行如第一方面所述的语音识别方法。
第四方面，本发明实施例提供了一种非暂时性机器可读存储介质，所述非暂时性机器可读存储介质上存储有可执行代码，当所述可执行代码被电子设备的处理器执行时，使所述处理器至少可以实现如第一方面所述的语音识别方法。
第五方面，本发明实施例提供一种语音识别方法，该方法包括：
接收终端设备通过调用语音识别服务触发的请求，所述请求中包括多个说话者的语音信号和所述多个说话者的画像特征向量，所述语音信号中包含所述多个说话者的语音；
根据所述模型训练服务对应的计算资源执行如下步骤：
获取语音识别模型，所述语音识别模型中包括语音识别编码器、说话者编码器、语音识别解码器、说话者解码器以及包含注意力层的文本编码器；
通过所述语音识别编码器获取所述语音信号对应的第一向量表示，通过所述说话者编码器获取所述语音信号对应的第二向量表示，所述第一向量表示被设置为进行语音识别，所述第二向量表示被设置为进行说话者识别；
通过所述文本编码器对所述语音识别解码器已经输出的前n-1个字进行编码处理，以得到第n-1个字所对应的第三向量表示；
将所述第一向量表示、所述第二向量表示和所述第三向量表示输入所述说话者解码器中，以获得第n个字对应的说话者向量表示；
根据所述第n个字对应的说话者向量表示分别与所述多个说话者的画像特征向量之间的相关性系数，确定所述第n个字对应的说话者；
将所述第一向量表示、所述前n-1个字以及所述相关性系数对所述多个说话者的画像特征向量的加权和，输入所述语音识别解码器，以获得所述第n个字；
向所述终端设备发送语音识别输出信息，所述语音识别输出信息中包括所述多个说话者各自对应的文字序列。
在上述本发明实施例提供的语音识别方案中，语音识别模型中包括语音识别编码器、说话者编码器、语音识别解码器、说话者解码器以及包含注意力层的文本编码器。在对多个说话者的语音信号进行语音识别时，可以先通过语音识别模型中的语音识别编码器对该语音信号进行编码处理，以获取被设置为进行语音识别的第一向量表示，并通过说话者编码器对该语音信号进行编码处理，以获取被设置为进行说话者识别的第二向量表示。而后，通过文本编码器对语音识别解码器已经输出的前n-1个字进行编码处理，以得到第n-1个字所对应的第三向量表示。之后，将第一向量表示、第二向量表示以及第三向量表示输入说话者解码器中，以获得第n个字对应的说话者向量表示，并根据第n个字对应的说话者向量表示分别与多个说话者的画像特征向量之间的相关性系数，确定第n个字对应的说话者。最后，将第一向量表示、前n-1个字以及相关性系数对多个说话者的画像特征向量的加权和，输入语音识别解码器中，以获得第n个字。
在上述方案中，在语音识别模型中添加包含注意力层的文本编码器，通过文本编码器对语音识别解码器已经输出的前n-1个字进行编码处理，可以更好地聚合整个句子的上下文信息，以获得第n-1个字所对应的包含上下文信息的第三向量表示，即这样可以获得更准确地被设置为进行说话者识别的向量表示，从而提高了各个字对应的说话者识别结果的准确性。
附图说明
为了更清楚地说明本发明实施例中的技术方案，下面将对实施例描述中所需要使用的附图作一简单地介绍，显而易见地，下面描述中的附图是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本发明实施例提供的一种语音识别方法的流程图；
图2为本发明实施例提供的一种语音识别模型的结构示意图；
图3为本发明实施例提供的语音识别编码器和语音识别解码器的结构示意图；
图4为一种通过说话者解码器确定第n个字对应的说话者向量表示的流程图；
图5为本发明实施例提供的另一种语音识别方法的流程图；
图6为一种通过语音识别解码器获得第n个字的流程图；
图7为本发明实施例提供的一种语音识别方法的应用示意图；
图8为本发明实施例提供的另一种语音识别方法的流程图；
图9为本发明实施例提供的一种云端服务模式下语音识别过程的示意图；
图10为本发明实施例提供的一种语音识别装置的结构示意图；
图11为本实施例提供的一种电子设备的结构示意图。
具体实施方式
在下面的描述中阐述了很多具体细节以便于充分理解本公开。但是本公开能够以很多不同于在此描述的其它方式来实施，本领域技术人员可以在不违背本公开内涵的情况下做类似推广，因此本公开不受下面公开的具体实施的限制。
为使本发明实施例的目的、技术方案和优点更加清楚，下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。另外，下述各方法实施例中的步骤时序为一种举例，而非严格限定。
需要说明的是，本发明实施例中所涉及的用户信息(包括但不限于用户设备信息、用户个人信息等)和数据(包括但不限于用于分析的数据、存储的数据、展示的数据等)，均为经用户授权或者经过各方充分授权的信息和数据，并且相关数据的收集、使用和处理需要遵守相关国家和地区的相关法律法规和标准，并提供有相应的操作入口，供用户选择授权或者拒绝。
下面先对本发明实施例中涉及到的一些概念进行简单介绍。
说话人相关语音识别(Speaker-Attributed Automatic Speech Recognition，简称SA-ASR)：是一种用于解决“谁说了什么内容”的语音识别任务。
序列化输出训练(Serialized Output Training，简称SOT)：在对语音进行转录(即语音转为文字)时，采用特殊的分隔符号将不同说话者所讲的文字连接起来，并且按照不同说话者开始说话的时间顺序对转录文字进行排列，以生成文字序列，并输出文字序列。例如，在一段语音中包括两个说话者，说话者A说了“今天天气怎么样”，说话者B说了“今天天气不错”。那么在对该段语音进行序列化输出训练时，将输出：今天天气怎么样<sc>今天天气不错，并且每个字或者说每句话关联对应的说话者标签。
说话者画像(Speaker Profile)：是指一组特征向量，其包含有多个说话者的画像特征向量。可以从各个说话者的一段语音信号中提取出各个说话者的画像特征向量。
端到端(End to End，简称E2E)：区别于对目标任务进行模块化处理，而由一个完整的模型对目标任务进行处理，即输入是原始数据，输出是最后的处理结果。在学习训练时可以直接优化整体的目标函数。
现有的端到端说话人相关语音识别模型中主要包括语音识别编码器、说话者编码 器、语音识别解码器、说话者解码器，该模型中缺乏对上下文信息感知模块，在通过这些编码器和解码器对多个说话者的语音进行识别时，没有充分考虑各个字所对应的上下文信息，使得预测出的各个字对应的说话者的结果不准确，从而影响了最终的语音识别结果，为了解决多说话者的语音识别结果不准确的问题，本发明实施例提供了一种新的语音识别方案。在该语音识别方案中，在语音识别模型中添加包含注意力层的文本编码器，在确定第n个字所对应的说话者向量表示时，通过该文本编码器可以对语音识别解码器已经输出的前n-1个字进行编码处理，以整合句子的上下文信息，使得第n个字所对应的说话者向量表示更精准，从而使得说话者编码器可以更准确地预测出各个字对应的说话者。
下面结合附图，对本发明的一些实施方式作详细说明。在各实施例之间不冲突的情况下，下述的实施例及实施例中的特征可以相互组合。
图1为本发明实施例提供的一种语音识别方法的流程图，如图1所示，该方法包括如下步骤：
101、获取多个说话者的语音信号和多个说话者的画像特征向量，语音信号中包含多个说话者的语音。
102、通过语音识别编码器获取语音信号对应的第一向量表示，通过说话者编码器获取语音信号对应的第二向量表示，第一向量表示被设置为进行语音识别，第二向量表示被设置为进行说话者识别。
103、通过文本编码器对语音识别解码器已经输出的前n-1个字进行编码处理，以得到第n-1个字所对应的第三向量表示。
104、将第一向量表示、第二向量表示和第三向量表示输入说话者解码器中，以获得第n个字对应的说话者向量表示。
105、根据第n个字对应的说话者向量表示分别与多个说话者的画像特征向量之间的相关性系数，确定第n个字对应的说话者。
106、将第一向量表示、前n-1个字以及相关性系数对多个说话者的画像特征向量的加权和，输入语音识别解码器，以获得第n个字。
本发明实施例提供的语音识别方案可以利用预先训练好的语音识别模型对待识别的多个说话者的语音信号进行处理，以获得与该语音信号相对应的各个字以及各个字所对应的说话者。由于语音识别模型的训练过程与使用语音识别模型对待识别的多个说话者的语音信号进行识别的过程有相似之处，这里描述使用语音识别模型的过程。
其中，语音识别模型的结构可以参考附图2所示。语音识别模型中主要包括语音识别编码器、说话者编码器、语音识别解码器、说话者解码器以及包含注意力层的文本编码器。语音识别编码器主要被设置为将多个说话者的语音信号转换为被设置为进行语音识别的第一向量表示。说话者编码器主要被设置为将多个说话者的语音信号转换为被设置为进行说话者识别的第二向量表示。语音识别解码器主要被设置为完成语 音到文字的识别，以识别出语音信号中各个说话者所说的全部文字，最终输出各个说话者对应的语音识别结果。说话者解码器主要被设置为确定各个文字与说话者的对应关系。文本编码器主要被设置为整合文字的上下文信息，根据各个字所对应的上下文信息确定各个字所对应的说话者向量表示，以使得获得的各个字对应的说话者向量表示更加准确。
在一可选实施例中，语音识别编码器和语音识别解码器的具体结构可以参考附图3所示。语音识别编码器中可以包括级联的多个编码器，每个编码器中可以包括两个子层：注意力层和前馈神经网络层。语音识别解码器中可以包括级联的多个解码器，每个解码器中包括注意力层和前馈神经网络层。同样地，说话编码器与语音识别编码器具体相似的结构，可以参考图3所示的具体结构。其中，说话者编码器中可以包括级联的多个编码器，并且每个编码器中包括注意力层和前馈神经网络层。同样地，说话解码器与语音识别解码器具体相似的结构，说话者解码器中包括级联的多个解码器，每个解码器中包括注意力层和前馈神经网络层。文本编码器中包括级联的至少一个编码器，每个编码器中包括注意力层和前馈神经网络层。其中，语音识别编码器、说话者编码器中所包含的编码器的数量可以根据实际需求进行设定，在此不做限定。同样地，语音识别解码器、说话者解码器中所包含的解码器的数量可以根据实际需求进行设定。另外，这里的解码器中的注意力层可以包括自注意力层和源-目标注意力层。
由上述描述可以获知语音识别模型主要包括语音识别编码器、说话者编码器、语音识别解码器、说话者解码器以及包含注意力层的文本编码器。那么，利用语音识别模型对多个说话者的语音信号进行语音识别时，语音识别模型中的各个编码器和解码器的具体执行过程如下：首先，获取多个说话者的语音信号和多个说话者的画像特征向量。其中，语音信号中包含多个说话者的语音。说话者的画像特征向量是指说话者的特征向量，可以从该说话者的一段语音信号中提取出该说话者对应的画像特征向量。
在对多个说话者的语音信号进行处理之前，可以预先对多个说话者的画像特征进行提取，以获得各个说话者的画像特征向量。具体地，可以通过一个与说话者编码器结构相同的编码器对各个说话者的语音信号进行编码处理，以获得各个说话者的画像特征向量。
在获取到包含多个说话者的语音信号之后，可以对该语音信号进行处理，以提取有效的语音信息，获得该语音信号对应的声学特征序列。其具体实现过程可以包括：对获取到的包含多个说话者的语音信号进行分割处理，将语音信号切成一帧一帧，然后对每帧语音信号提取声学特征，并根据各帧语音信号对应的声学特征，生成该语音信号对应的声学特征序列。
接着，将语音信号对应的声学特征序列输入至语音识别编码器中，以通过语音识别编码器获取该语音信号对应的第一向量表示。其中，第一向量表示被设置为进行语音识别，也就是说该第一向量表示可以表征该语音信号的语音特征。并将语音信号对 应的声学特征序列输入至说话者编码器中，以通过说话者编码器获取该语音信号对应的第二向量表示。其中，第二向量表示被设置为进行说话者识别，也就是说该第二向量表示可以被设置为表征该语音信号的说话者的音色等特征。
在本实施例中，语音识别编码器可以将语音信号对应的声学特征序列转换为可以进行语音识别的向量表示，以便于后续直接基于第一向量表示进行语音识别处理，以获得该语音信号所对应的各个字。说话者编码器可以将语音信号对应的声学特征序列转换为进行说话者识别的向量表示，以便于后续直接基于第二向量表示进行各个字所对应的说话者识别处理。
对语音信号进行识别转录处理时，采用逐个字逐个字的形式进行预测，例如语音识别解码器确定出该语音信号对应的第1个字之后，第1个字自回归到语音识别解码器的输入端，被设置为确定该语音信号对应的第2个字，依次确定该语音信号对应的各个字，直到遇到结束符为止。由于对语音信号中的各个字进行预测时，其具体实现过程基本一致，那么这里以第n个字的预测过程进行举例说明，其中，假设语音识别解码器已经输出了前n-1个字。需要说明的是，识别出的第1个字是基于设定的起始字符或者说空字符作为语音识别解码器的输入来预测出的。
在确定第n个字时，首先要获得第n-1个字所对应的包含上下文信息的第三向量表示，即结合前n-1个字的信息，确定第n个字，这样确定出第n个字更准确。由于文本编码器主要是被设置为整合各个字所对应的上下文信息，以获得包含上下文信息的向量表示，那么在获取到语音识别解码器输出的前n-1个字之后，通过文本编码器对语音识别解码器已经输出的这前n-1个字进行编码处理，以得到第n-1个字所对应的第三向量表示。
需要注意的是：文本编码器在确定第n-1个字所对应的第三向量表示时，首先获取语音识别编码器经过各种处理之后最终输出的前n-1个字，并对这已经确定出的前n-1个字进行编码处理，以从编码结果中获得第n-1个字所对应的编码结果作为该第n-1个字对应的第三向量表示。例如，当要获得第4个字所对应的第三向量表示时，首先获取语音识别编码器已经输出的前3个字，对这前3个字进行编码处理，获得第4个字对应的第三向量表示。
在一可选实施例中，通过文本编码器对语音识别解码器已经输出的前n-1个字进行编码处理，以得到第n-1个字所对应的第三向量表示的具体实现过程可以包括：获取语音识别解码器中的自注意力层分别对前n-1个字进行注意力计算后得到的前n-1个字分别对应的注意力系数加权后的向量表示；并将前n-1个字分别对应的注意力系数加权后的向量表示输入文本编码器中，以获得文本编码器输出的第n-1个字所对应的第三向量表示。也就是说，输入到文本编辑器的是语音识别解码器中自注意力层输出的前n-1个字各自对应的加权后编码向量。
如上文所述，语音识别解码器中可以包括至少一个解码器，每个解码器中都可以 包括注意力层，而且具有可以包括几种不同的注意力层，比如自注意力层和源-目标注意力层。为了简化计算复杂度以及保留每个字的浅层次的语义信息，可选地，上述语音识别解码器中的自注意力层可以是语音识别解码器的第一层即第一个解码器中包含的自注意力层。此时，就是要收集语音识别编码器的第一层中自注意力层针对前n-1个字分别输出的结果，形成一个序列，送入文本编码器中。实际上，文本编码器会输出前n-1个字每个字对应的融合上下文信息后的第三向量表示，这里只是从中这前n-1个字对应的第三向量表示中取出第n-1个字对应的第三向量表示。之所以文本编码器可以进行文字上下文信息的融合是因为文本编码器中包括注意力层，比如可以是采用自注意力机制或其他注意力机制的注意力层，本文中以自注意力层为例。
可选地，针对语音识别解码器来说，第一层中的自注意力层分别对前n-1个字进行注意力计算可以表示为如下公式：其中，表示语音识别解码器的第一层输出的第n-1个字对应的注意力系数加权后的向量表示，表示输入语音识别解码器的第一层中的第n-1个字对应的嵌入向量表示，表示语音识别解码器的第一层中自注意力层所执行的多头注意力计算，表示输入语音识别解码器的第一层中的前n-1个字分别对应的嵌入向量表示。其中，嵌入向量表示被设置为表示嵌入编码处理后的向量表示，并且该嵌入向量表示可以通过对前n-1个字分别进行嵌入编码或者嵌入编码和位置编码，以得到前n-1个字各自对应的嵌入向量表示。具体的，可以采用公式获得前n-1个字各自对应的嵌入向量表示。其中，PosEnc表示位置编码，Embed表示嵌入编码，y[1:n-1]表示语音识别解码器输出的前n-1个字。
在获得前n-1个字分别对应的注意力系数加权后的向量表示之后，文本编辑器对前n-1个字分别对应的注意力系数加权后的向量表示进行处理，以获得第n-1个字所对应的第三向量表示。在一可选实施例中，文本编辑器可以根据公式计算第n-1个字所对应的第三向量表示。其中，表示第n-1个字所对应的第三向量表示，Context-Enc表示上下文文本编码，表示前n-1个字分别对应的注意力系数加权后的向量。
在获得第三向量表示之后，将第一向量表示、第二向量表示和第三向量表示输入说话者解码器中，以获得第n个字对应的说话者向量表示。其中，说话者向量表示可以用于表征说话者的特征。在确定的第n个字对应的说话者向量表示时结合了语音信号的语音识别特征、说话者识别特征、以及上下文信息特征，这样确定出的第n个字对应的说话者向量表示质量更高，从而可以提高各个字对应的预测说话者的准确性， 使得语音信号所对应的转录结果更准确。
通过说话者解码器确定出第n个字对应的说话者向量表示之后，接着根据第n个字对应的说话者向量表示分别与多个说话者的画像特征向量之间的相关性系数，确定第n个字对应的说话者。其中，相关性系数被设置为表征各个字对应的说话者向量表示与多个说话者的画像特征向量之间的相关性，也就是说该相关性系数被设置为表示第n个字与语音信号中包含的各个说话者之间的相关性。那么，在确定第n个字对应的说话者时，首先获取第n个字对应的说话者向量表示分别与多个说话者的画像特征向量之间的相关性系数，然后根据第n个字对应的说话者向量表示分别与多个说话者的画像特征向量之间的相关性系数，确定第n个字对应的说话者。
例如，假设第一段语音信号中包含3个说话者，第n个字对应的说话向量表示为qn，第一个说话者的画像特征向量为d1，第二个说话者的画像特征向量为d2，第三个说话者的画像特征向量为d3，获取第n个字对应的说话者向量表示qn与d1之间的相关性系数βn,1，获取第n个字对应的说话者向量表示qn与d2之间的相关性系数βn,2，获取第n个字对应的说话者向量表示qn与d3之间的相关性系数βn,3,然后根据这三个相关性系数，确定第n个字对应的说话者。在一可选实施例中，可以从多个相关性系数中选取最大相关性系数对应的说话者确定为第n个字对应的说话者。
然后，计算多个相关性系数对多个说话者的画像特征向量的加权和，其中加权和可以看作为加权后的第n个字对应的说话者向量表示。可选地，可以根据公式计算多个相关性系数对多个说话者的画像特征向量的加权和其中，表示加权和，K表示语音信号中包含的说话者总数，βn,k表示第n个字所对应的说话者向量与第k个说话者的画像特征向量之间的相关系数，dk表示第K个说话者的画像特征向量。
最后，将第一向量表示、前n-1个字以及相关性系数对多个说话者的画像特征向量的加权和，输入语音识别解码器，以获得第n个字。其中，加权和为第n个字对应的加权后的说话者向量表示，在确定第n个字时引入第n个字对应的加权后的说话者向量表示，这样获得的第n个字不包括该字的具体内容还包括该字所对应的说话者。例如输出的第n个字的具体内容是“我”，对应的说话者是“小李”。
依据上述方法可以依次确定语音信号中各个字以及每个字对应的说话者，最终可以确定出每个说话者对应的SOT格式的识别结果。例如，在一段语音信号中识别出包括两个说话者，且识别出的文字“今天天气怎么样”所对应的说话者为说话者a,识别出的文字“今天天气不错”所对应的说话者为说话b，输出的内容为：今天天气怎么样<sc>今天天气不错。
在本发明实施例中，在语音识别模型中添加包含注意力层的文本编码器，通过文本编码器对语音识别解码器已经输出的前n-1个字进行编码处理，可以更好地聚合整个句子的上下文信息，以获得第n-1个字所对应的包含上下文信息的第三向量表示， 即这样基于第一向量表示、第二向量表示以及第三向量表示，可以获得更准确地被设置为进行说话者识别的说话者向量表示，从而提高了各个字对应的说话者识别结果的准确性。
在实际应用中，一个说话者解码器中包括级联的多个解码器。为了便于理解语音识别模型中的说话者解码器中各个层所对应的编码器的具体处理过程，结合图4对各个层所对应的编码器的处理过程进行示例性说明。
图4为一种通过说话者解码器确定第n个字对应的说话者向量表示的流程图；如图4所示，说话者解码器中包括级联的多个解码器，每个解码器中包括注意力层和前馈神经网络层，该方法包括如下步骤：
401、将第一向量表示、第二向量表示和第三向量表示输入说话者解码器中的第一层解码器中，以获得第一层解码器中的注意力层输出的第n-1个字对应的注意力系数加权后的说话者向量表示。
402、将第n-1个字对应的注意力系数加权后的说话者向量表示输入第一层解码器中的前馈神经网络层，以获得输入到第二层解码器的第n-1个字对应的说话者向量表示。
403、确定说话者解码器中最后一层解码器输出的第n-1个字对应的说话者向量表示。
404、根据最后一层解码器输出的第n-1个字对应的说话者向量表示和第一层解码器中的注意力层输出的第n-1个字对应的注意力系数加权后的说话者向量表示，确定第n个字对应的说话者向量表示。
本发明实施例中的说话者解码器中包括多个解码器，各个解码器分别设置于不同的层级，各个解码器按照层级逐层进行解码处理，并且上一层解码器的输出作为下一层解码器的输入。另外，各个解码器均包括注意力层和前馈神经网络层，分别由注意力层和前馈神经网络层两个子层进行解码处理，首先将输入到解码器中的向量表示传输至注意力层中进行处理，并将处理结果输入至前馈神经网络层，前馈神经网络层对注意力层输出的处理结果进行处理，并将处理后的结果输入至下一层的解码器中的注意力层中，由下一层的解码器中的注意力层进行处理，即说话者解码器中的各个解码器均按照该方法依次进行处理。
具体的，各个解码器中的处理过程可以包括：首先将第一向量表示、第二向量表示和第三向量表示输入说话者解码器中，以通过第一层解码器中的注意力层，获得第n-1个字对应的注意力系数加权后的说话者向量表示。然后，将第n-1个字对应的注意力系数加权后的说话者向量表示输入至第一层解码器中的前馈神经网络层，以通过第一层解码器中的前馈神经网络层获得输入到第二层解码器的第n-1个字对应的说话者向量表示。也就是说，第一层解码器中的前馈神经网络层输出结果(第n-1个字对应的说话者向量表示)作为第二层解码器中的注意力层的输入。
在一可选实施例中，具体的，第一层解码器中的注意力层可以根据公式计算第n-1个字对应的注意力系数加权后的说话者向量表示。其中，表示第一层解码器注意力层输出的第n-1个字对应的注意力系数加权后的说话者向量表示，表示第一层编码器的注意力层的多头注意力，Hasr表示第一向量表示，Hspk表示第二向量表示。在计算出第n-1个字对应的注意力系数加权后的说话者向量表示之后，第一层解码器中的前馈神经网络层可以根据公式计算第n-1个字对应的说话者向量表示。其中，表示输入到第二层解码器的第n-1个字对应的说话者向量表示，表示第一层解码器注意力层输出的第n-1个字对应的注意力系数加权后的说话者向量表示，表示说话者编码器中的第一层前馈神经网络。
在一可选实施例中，第二层解码器以及第二层以上的解码器中的注意力层包括自注意力层和源-目标注意力层。那么，在通过第一层解码器中的前馈神经网络层获得输入到第二层解码器的第n-1个字对应的说话者向量表示之后，将其输出至第二层解码器中的自注意力层，第二层解码器中的自注意力层对第n-1个字对应的说话者向量表示、前n-1个字对应的说话者向量表示进行自注意力计算，获得第二层解码器中的自注意力层对应的第n-1个字对应的自注意力系数加权后的向量表示，并将第n-1个字对应的自注意力系数加权后的向量表示输入至第二层解码器中的源-目标注意力层。第二层解码器中的源-目标注意力层对第n-1个字对应的自注意力系数加权后的向量表示和第二向量表示进行注意力计算，以获得第二层解码器中的源-目标注意力层对应的第n-1个字对应的注意力系数加权后的向量表示，并将第n-1个字对应的注意力系数加权后的向量表示输入至第二层解码器中的前馈神经网络层。第二层解码器中的前馈神经网络层对第n-1个字对应的注意力系数加权后的向量表示进行处理，以获得输入至第三层编码器的第n-1个字对应的说话者向量表示。
第二层以上解码器按照上述方法分别进行计算处理，直至确定出说话者解码器中最后一层解码器输出的所述第n-1个字对应的说话者向量表示，并根据最后一层解码器输出的第n-1个字对应的说话者向量表示和第一层解码器中的注意力层输出的第n-1个字对应的注意力系数加权后的说话者向量表示，确定第n个字对应的说话者向量表示，其详细处理过程参考上述描述在此不再赘述。
在一可选实施例中，第二层解码器以及第二层以上的解码器中的自注意力层可以根据公式计算输入至该层解码器中的源-目标注意力层的第n-1个字对应的自注意力系数加权后的向量表示。其中，l表示编码器所对应的层数，且l>1,表示第l层编码器中的自注意力层输出的第n-1 个字对应的自注意力系数加权后的向量表示(即输入至第l层编码器中的源-目标注意力层的第n-1个字对应的自注意力系数加权后的向量表示)，表示第l-1层编码器中的前馈神经网络层输出的第n-1个字对应的说话者向量表示，表示第l层编码器中的自注意力层的多头注意力，表示第l-1层编码器中的前馈神经网络层输出的前n-1个字对应的说话者向量表示。
在一可选实施例中，第二层解码器以及第二层以上的解码器中的源-目标注意力层可以根据公式计算输入至该层解码器中的前馈神经网络层的第n-1个字对应的注意力系数加权后的向量表示。其中，l表示编码器所对应的层数，且l>1,表示第l层编码器中的源-目标注意力层输出的第n-1个字对应的注意力系数加权后的向量表示(即输入至第l层编码器中的前馈神经网络层的第n-1个字对应的注意力系数加权后的向量表示)，表示第l层编码器中的自注意力层输出的第n-1个字对应的自注意力系数加权后的向量表示，表示第l层编码器中的源-目标注意力层的多头注意力，Hspk表示第二向量表示。
在一可选实施例中，第二层解码器以及第二层以上的解码器中的前馈神经网络层可以根据公式计算输入至下一层编码器的第n-1个字对应的说话者向量表示。其中，l表示编码器所对应的层数，且l>1,表示第l层编码器中的前馈神经网络输出的第n-1个字对应的说话者向量表示(即输入至第l+1层编码器中的第n-1个字对应的说话者向量表示)，表示第l层编码器中的源-目标注意力层输出的第n-1个字对应的注意力系数加权后的向量表示，表示第l层编码器中的前馈神经网络。
依次确定各个层解码器中的前馈神经网络层输出，直到确定出说话者解码器中最后一层解码器输出的第n-1个字对应的说话者向量表示。最后，说话者解码器对最后一层解码器输出的第n-1个字对应的说话者向量表示和第一层解码器中的注意力层输出的第n-1个字对应的注意力系数加权后的说话者向量表示进行跳接处理(skip-connect)，根据跳接结果来预测第n个字对应的说话者向量表示。具体的，在一可选实施例中，假设说话者解码器中最后一层解码器输出的第n-1个字对应的说话者向量表示为第一层解码器中的注意力层输出的第n-1个字对应的注意力系数加权后的说话者向量表示为根据公式确定第n个字对应的说话者向量表示qn。其中，Wq表示说话者编码器中设定的权重。
本发明实施例，说话者解码器中包含多层解码器，第一层解码器通过对第一向量表示、第二向量表示和包含上下文信息的第三向量表示进行处理，以获得第一层解码 器中的注意力层输出的第n-1个字对应的注意力系数加权后的说话者向量表示，即在第一层解码器在进行处理时结合了上下文信息，使得获得第n-1个字对应的注意力系数加权后的说话者表示更准确。然后，第一层解码器中的前馈神经网络层对结合上下文信息处理后获得的第n-1个字对应的注意力系数加权后的说话者向量进行处理，这样输入到第二层解码器的第n-1个字对应的说话者向量表示具有较高的质量，可以更好地反映第n-1个字对应的说话者特征，再依次通过多层解码器的处理可以获得的更高质量的各个字对应的说话者向量表示，从而使得各个字所预测的说话者结果的准确性。
上述实施例介绍了确定各个字对应的说话者向量表示的实施方式，然而在实际应用中，在确定出了各个字对应的说话者向量表示之后，可以进一步根据各个字对应的说话者向量表示确定各个字对应的说话者。具体的，在一可选实施例中，根据第n个字对应的说话者向量表示分别与多个说话者的画像特征向量之间的相关性系数，确定第n个字对应的说话者。那么在确定第n个字对应的说话者时，首先确定第n个字对应的说话者向量表示分别与多个说话者的画像特征向量之间的相关性系数。在下述实施例中，结合附图5对确定第n个字对应的说话者向量表示分别与多个说话者的画像特征向量之间的相关性系数的具体过程进行示例性说明。
图5为本发明实施例提供的另一种语音识别方法的流程图；如图5所示，为了提高该方法语音识别结果的准确性，在上述实施例的基础上，该方法还可以包括如下步骤：
501、针对多个说话者中任一说话者的画像特征向量，根据设定的相似度算法，确定第n个字对应的说话者向量表示与任一说话者的画像特征向量之间的第一相似度。
502、获取前n个字各自对应的说话者向量表示。
503、分别将前n个字各自对应的说话者向量表示与任一说话者的画像特征向量进行拼接。
504、将多个拼接后的向量表示输入设定的包含注意力模块的打分模型，以获得前n个字各自对应的说话者向量表示分别与任一说话者的画像特征向量之间的第二相似度。
505、根据第n个字对应的说话者向量表示与任一说话者的画像特征向量之间的第一相似度以及第n个字对应的说话者向量表示与任一说话者的画像特征向量之间的第二相似度，确定第n个字对应的说话者向量表示与任一说话者的画像特征向量之间的相关性系数。
在本发明实施例中，采用设定的相似度算法和设定的包含注意力模块的打分模型联合打分的方式确定第n个字对应的说话者向量表示与任一说话者的画像特征向量之间的相关性系数。其中，相关性系数可以被设置为表示第n个字对应各个说话者的后验概率。
具体的，针对多个说话者中任一说话者的画像特征向量，根据设定的相似度算法，确定第n个字对应的说话者向量表示与任一说话者的画像特征向量之间的第一相似度。也就是说，根据设定的相似度算法，可以分别确定出第n个字对应的说话者向量表示与各个说话者的画像特征向量之间的第一相似度。其中，可选地，设定的相似度算法可以为余弦相似度算法，例如，根据公式可以确定第n个字对应的说话者向量表示与各个说话者的画像特征向量之间的第一相似度。其中，表示第n个字对应的说话者向量表示与第k个说话者的画像特征向量之间的第一相似度，cos表示余弦函数，qn表示第n个字对应的说话者向量表示，dk表示第k个说话者的画像特征向量，k是语音信号中包含的K个说话者中的第k个说话者。也就是说，第n个字对应的说话者向量表示分别与各个说话者的画像特征向量进行余弦相似度分数计算，得到第n个字分别对应于各个说话者的概率。
由上述描述可知：第一相似度的确定只考虑了第n个字对应的说话者向量表示，未考虑前n个字对应的说话者向量表示，因此，第一相似度的确定是其它说话者无关的。
为了提高确定出的各个字对应各个说话者的后验概率的准确度，在确定第n个字对应的说话者向量表示与任一说话者的画像特征向量之间的相关性系数时，充分结合了其他说话者的说话者向量表示。具体的，通过在语音识别模型中添加设定的包含注意力模块的打分模型，以通过设定的包含注意力模块的打分模型，获得前n个字各自对应的说话者向量表示分别与任一说话者的画像特征向量之间的第二相似度。
其中，确定第二相似度的具体实现过程可以为：获取前n个字各自对应的说话者向量表示，分别将前n个字各自对应的说话者向量表示与任一说话者的画像特征向量进行拼接。将多个拼接后的向量表示输入设定的包含注意力模块的打分模型，以获得前n个字各自对应的说话者向量表示分别与任一说话者的画像特征向量之间的第二相似度。
具体的，在一可选实施例中，可以根据公式确定前n个字各自对应的说话者向量表示分别与各个说话者的画像特征向量之间的第二相似度。其中，表示第n个字对应的说话者向量表示与第k个说话者的画像特征向量之间的第二相似度，tanh为双曲正切函数，CD-scorer表示设定的包含注意力模块的打分模型。q[1:n]表示前n个字对应的说话者向量表示，dk表示第k个说话者的画像特征向量。可以按照该公式分别生成第n个字对应的说话者向量表示与各个说话者的画像特征向量之间的第二相似度。由该公式可知，确定第二相似度的实现方式可以为：获取前n个字中每个字对应的说话者向量表示，将前n个字中每个字对应的说话者向量表示分别与第k个说话人特征向量拼接，得到第一向量序列，长度为n。并将第一向量序列输入至打分模型(比如采用transfomer模型的结 构)中，获得第二向量序列。其中第二向量序列是经过tanh变换后得到的取值在[-1,1]的一个序列，取出其中最后一个确定为第n个字对应的说话者向量表示与第k个说话者的画像特征向量之间的第二相似度。Tanh为一种映射函数的举例，不以此为限。
在确定出第一相似度和第二相似度之后，根据第n个字对应的说话者向量表示与任一说话者的画像特征向量之间的第一相似度以及第n个字对应的说话者向量表示与任一说话者的画像特征向量之间的第二相似度，确定第n个字对应的说话者向量表示与任一说话者的画像特征向量之间的相关性系数。也就是说，对于第n个字来说，针对每个说话者均计算第一相似度和第二相似度，将第n个字与每个说话者之间的第一相似度和第二相似度相加，获得第一和值，并将第一和值除以第n个字与各个说话者之间第一相似度和第二相似度总和，获得第n个字对应的说话者向量表示与每个说话者的画像特征向量之间的相关性系数。
具体的，在一可选实施例中，可以根据公式确定第n个字对应的说话者向量表示与各个说话者的画像特征向量之间的相关性系数。其中，βn,k表示第n个字对应的说话者向量表示与第k个说话者的画像特征向量之间的相关性系数，exp表示以e为底的指数函数，表示第n个字对应的说话者向量表示与第k个说话者的画像特征向量之间的第一相似度，表示第n个字对应的说话者向量表示与第k个说话者的画像特征向量之间的第二相似度，K为语音信号中所包含的说话者总数，j为第j个说话者。另外，上述中的k可以为K个说话者中的任意一个，那么根据上述公式可以获得第n个字对应的说话者向量表示与各个说话者的画像特征向量之间的相关性系数。
在确定出第n个字对应的说话者向量表示分别与所述多个说话者的画像特征向量之间的相关性系数之后，可以根据第n个字对应的说话者向量表示分别与多个说话者的画像特征向量之间的相关性系数，确定第n个字对应的说话者。
本发明实施例中，通过根据设定的相似度算法，确定第n个字对应的说话者向量表示与任一说话者的画像特征向量之间的第一相似度，通过设定的包含注意力模块的打分模型，获得前n个字各自对应的说话者向量表示分别与任一说话者的画像特征向量之间的第二相似度，再根据第n个字对应的说话者向量表示与任一说话者的画像特征向量之间的第一相似度以及第n个字对应的说话者向量表示与任一说话者的画像特征向量之间的第二相似度，确定第n个字对应的说话者向量表示与任一说话者的画像特征向量之间的相关性系数，即采用设定的相似度算法和设定的包含注意力模块的打分模型结合的方式，确定出的相关性系数更准确，使得语音识别模型具有更好的性能，从而可以进一步提高语音识别结果的准确性。
上述实施例介绍了确定第n个字对应的说话者向量表示分别与所述多个说话者的 画像特征向量之间的相关性系数的具体实现方式，在确定出第n个字对应的说话者向量表示分别与多个说话者的画像特征向量之间的相关性系数之后，计算相关性系数对多个说话者的画像特征向量的加权和。具体的，获取多个相关性系数分别与各个说话者的画像特征向量的乘积值，然后对多个乘积值进行求和运算，以获得第n个字对应的加权后的说话人特征向量。
在获取到相关性系数对多个说话者的画像特征向量的加权和之后，将第一向量表示、前n-1个字以及相关性系数对多个说话者的画像特征向量的加权和，输入语音识别解码器，以获得第n个字。在实际应用中，一个语音识别解码器中包括级联的多个解码器。为了便于理解语音识别模型中的语音识别解码器中各个层所对应的编码器对第一向量表示、前n-1个字以及相关性系数对多个说话者的画像特征向量的加权和的具体处理过程，结合图6对各个层所对应的编码器的处理过程进行示例性说明。
图6为一种通过语音识别解码器获得第n个字的流程图；如图6所示，语音识别解码器中包括级联的多个解码器，每个解码器中包括注意力层和前馈神经网络层，该方法包括如下步骤：
601、对前n-1个字分别进行嵌入编码，得到前n-1个字各自对应的嵌入向量表示。
602、将前n-1个字各自对应的嵌入向量表示和第一向量表示输入语音识别解码器的第一层解码器中，以获得第一层解码器中的注意力层输出的第n-1个字对应的注意力系数加权后的向量表示。
603、将第n-1个字对应的注意力系数加权后的向量表示和加权和输入第一层解码器中的前馈神经网络层，以获得输入到第二层解码器的第n-1个字对应的向量表示。
604、根据语音识别解码器中最后一层解码器输出的第n-1个字对应的向量表示，确定第n个字。
本发明实施例中的语音识别解码器中包括多层解码器，各层解码器按照层级逐层进行解码处理，并且上一层解码器的输出作为下一层解码器的输入。其中，每个解码器中包括注意力层和前馈神经网络层，由两个子层分别对输入至编码器中的向量表示进行处理。语音解码器的作用是采用自回归的方式逐个预测各个字。例如，在进行第n次迭代时，被设置为预测输入语音信号对应的第n个字，根据输入的前n-1个字、第n-1个字对应的注意力系数加权后的向量表示以及相关性系数对多个说话者的画像特征向量的加权和，预测第n个字。其中，预测出的第n个字是根据第n个字对应字典V中各个字的概率得到的。假设字典中一个4950个字，确定第n个字对应于字典V中各个字的概率on，即on为4950个概率，在获得的4950个概率中选取其中概率最大的字作为第n个字的预测结果。
具体的，对前n-1个字分别进行嵌入编码，得到前n-1个字各自对应的嵌入向量表示。可选地，可以先对前n-1个字分别进行嵌入处理，获得处理后的向量表示，对该处理后的向量表示进行位置编码处理，获得前n-1个字各自对应的嵌入向量表示。例 如，根据公式获得前n-1个字各自对应的嵌入向量表示。其中，PosEnc表示位置编码，Embed表示嵌入函数，y[1:n-1]表示语音识别解码器已经输出的前n-1个字。
在得到前n-1个字各自对应的嵌入向量表示之后，将前n-1个字各自对应的嵌入向量表示和第一向量表示输入至第一层解码器中的注意力层，第一层解码器中的注意力层对前n-1个字各自对应的嵌入向量表示和第一向量表示进行处理，以获得第一层解码器中的注意力层对应的第n-1个字对应的注意力系数加权后的向量表示，并将其输入至第一层解码器中的前馈神经网络层。
在一可选实施例中，语音识别编码器中的各个解码器中的注意力层可以包括自注意力层和源-目标注意力层。那么，当说话者编码器接收到前n-1个字各自对应的嵌入向量表示和第一向量表示之后，将前n-1个字各自对应的嵌入向量表示输入至第一层解码器中的自注意力层，以获得第一层解码器中的自注意力层输出的第n-1个字对应的自注意力系数加权后的向量表示，并将该向量表示输入至第一层解码器中的源-目标注意力层中。第一层解码器中的源-目标注意力层对第n-1个字对应的自注意力系数加权后的向量表示和第一向量表示进行计算，获得源-目标注意力层输出的第n-1个字对应的注意力系数加权后的向量表示。接着，将第n-1个字对应的注意力系数加权后的向量表示和相关性系数对多个说话者的画像特征向量的加权和输入第一层解码器中的前馈神经网络层，以获得输入到第二层解码器的第n-1个字对应的向量表示。按照上述方法分别完成第二层以及第二层以上解码器的处理操作，直至获得语音识别解码器中最后一层解码器输出的第n-1个字对应的向量表示。
具体的，在一可选实施例中，第一层及第一层以上的解码器中的自注意力层可以根据公式计算输入至该层解码器中的源-目标注意力层的第n-1个字对应的自注意力系数加权后的向量表示。其中，l表示编码器所对应的层数，表示第l层编码器中的自注意力层输出的第n-1个字对应的自注意力系数加权后的向量表示(即输入至第l层编码器中的源-目标注意力层的第n-1个字对应的自注意力系数加权后的向量表示)，表示第l-1层编码器中的前馈神经网络层输出的第n-1个字对应的向量表示，表示第l层编码器中的自注意力层的多头注意力，表示第l-1层编码器中的前馈神经网络层输出的前n-1个字对应的向量表示(输入至第l层编码器中的前n-1个字对应的向量表示)。
在一可选实施例中，第一层及第一层以上的解码器中的源-目标注意力层可以根据公式计算输入至该层解码器中的前馈神经网络层的第n-1个字对应的注意力系数加权后的向量表示。其中，l表示编码器所对应的层 数，且表示第l层编码器中的源-目标注意力层输出的第n-1个字对应的注意力系数加权后的向量表示(即输入至第l层编码器中的前馈神经网络层的第n-1个字对应的注意力系数加权后的向量表示)，表示第l层编码器中的自注意力层输出的第n-1个字对应的自注意力系数加权后的向量表示，表示第l层编码器中的源-目标注意力层的多头注意力，Hasr表示第一向量表示。
在一可选实施例中，第一层解码器中的前馈神经网络层可以根据公式计算输入至第二层编码器的第n-1个字对应的向量表示。其中，l表示编码器所对应的层数，且l＝1,表示第l层编码器中的前馈神经网络输出的第n-1个字对应的向量表示(即输入至第l+1层编码器中的第n-1个字对应的向量表示)，表示第l层编码器中的源-目标注意力层输出的第n-1个字对应的注意力系数加权后的向量表示，表示第l层编码器中的前馈神经网络，Wspk为说话者编码器对应的权重，为相关性系数对多个说话者的画像特征向量的加权和。对于第二层及第二层以上解码器中的前馈神经网络层可以根据公式计算输入至下一层编码器的第n-1个字对应的向量表示(即前馈神经网络层的输出作为下一层编码器的输入)。其中，l表示编码器所对应的层数，且l>1。
最后，根据语音识别解码器中最后一层解码器输出的第n-1个字对应的向量表示，确定第n个字。具体的，利用softmax操作对其最后一层解码器输出的第n-1个字对应的向量表示进行处理，获得第n个字对应的后验概率on，再根据第n个字对应的后验概率on，预测第n个字。例如，假设最后一层解码器输出的第n-1个字对应的向量表示为根据公式确定第n个字对应的后验概率on，其中softmax为激活函数，Wo、bo分别为语音识别编码器对应的权重。
由上述描述可知：对于语音识别解码器中的第1层解码器，对已经输出的前n-1个字进行嵌入编码处理，在得到输入至第1层解码器的n-1个字分别对应的嵌入向量表示之后，取出第n-1个字的嵌入向量表示作为自注意力层中的query，这n-1个字分别对应的嵌入向量表示作为key、value，进行自注意力处理，得到第n-1个字对这前n-1个字进行注意力加权后的结果，即第n-1个字对应的自注意力系数加权后的编码向量，再与加权前的第n-1个字的嵌入向量表示相加，获得的相加结果即为第n-1个字对应的自注意力系数加权后的向量表示，并将其作为源-目标注意力层的其中一个输入，另外两个输入为两个第一向量表示，最终得到第1层中源-目标注意力层输出的第n-1个字对应的注意力系数加权后的向量表示。之后，将第n-1个字对应的注意力系数加权后的向量表示输入到第一层编码器中的前馈神经网络层中，以便得到第1层编 码器的输出结果亦即为第2层编码器的输入。
需要注意的是，针对第1层编码器中的前馈神经网络层，再计算其输出的时候会加上这一项，其他层编码器中的前馈神经网络层进行处理时不加此项。其中，在第一层编码器中的前馈神经网络层进行计算时引入的是为了在语音识别第n个字的过程中引入第n个字对应于各个说话者的加权后的说话向量表示，从而识别出的第n个字不包括该字对应的具体内容是什么还包括该字对应的说话者是什么。这样语音识别编码器最后就可以直接输出SOT格式的识别结果，例如一个说话者的文字序列<sc>另一个说话者的文字序列。
综上，在本发明实施例中，语音解码器采用自回归的方式逐个预测各个字，在对各个字进行预测处理时，语音编码器中的各层解码器按照层级逐层对输入的前n-1个字、第n-1个字对应的注意力系数加权后的向量表示以及相关性系数对多个说话者的画像特征向量的加权和进行解码处理，并且上一层解码器的输出作为下一层解码器的输入，最后根据语音识别解码器中最后一层解码器输出的第n-1个字对应的向量表示，确定第n个字，这样获得的第n个字不包含该字对应的内容和该字对应的说话者，以便后续直接基于预测出的各个字生成SOT格式的文字内容，而且确定出的各个字时综合考虑了上下文信息，可以使得预测出的各个字更准确。
近年来，随着越来越多的研究集中在更真实场景下的语音识别处理，例如多人会议自动记录、多方人机交互、音/视频自动标注等。具体应用时，参考附图7所示。语音识别模型中包括语音识别编码器、说话者编码器、语音识别解码器、说话者解码器、包含注意力层的文本编码器、以及联合打分器。其中，联合打分器包括说话者无关打分器和说话者有关打分器，说话者无关打分器可以根据设定的相似度算法，确定第n个字对应的说话者向量表示与任一说话者的画像特征向量之间的第一相似度，说话者有关打分器中包括设定的包含注意力模块的打分模型，以通过设定的包含注意力模块的打分模型，获得前n个字各自对应的说话者向量表示分别与任一说话者的画像特征向量之间的第二相似度。
具体的，在进行语音识别时，按照迭代的方式逐个预测语音信号对应的各个字。各个字的具体识别预测过程基本一致，这里以第n个字的识别预测过程进行示例性说明。首先获取多个说话者的语音信号和多个说话者的画像特征向量，其中，语音信号中包含多个说话者的语音。接着，将多个说话者的语音信号输入至语音识别编码器和说话者编码器中，通过语音识别编码器获取语音信号对应的第一向量表示，通过说话者编码器获取语音信号对应的第二向量表示。其中，第一向量表示被设置为进行语音识别，第二向量表示被设置为进行说话者识别。获取语音识别解码器已经输出的前n-1个字，通过文字编码器对语音识别解码器已经输出的前n-1个字进行编码处理，以得到第n-1个字所对应的第三向量表示。将第一向量表示、第二向量表示和第三向量表示输入说话者解码器中，以获得第n个字对应的说话者向量表示。
然后，分别将第n个字对应的说话者向量表示与任一说话者的画像特征向量进行拼接，将拼接后的向量表示输入至说话者无关打分器中，针对多个说话者中任一说话者的画像特征向量，根据设定的相似度算法，确定第n个字对应的说话者向量表示与任一说话者的画像特征向量之间的第一相似度。接着，分别将前n个字各自对应的说话者向量表示与任一说话者的画像特征向量进行拼接，并将多个拼接后的向量表示输入至说话者相关打分器中设定的包含注意力模块的打分模型，以获得前n个字各自对应的说话者向量表示分别与任一说话者的画像特征向量之间的第二相似度。根据第n个字对应的说话者向量表示与任一说话者的画像特征向量之间的第一相似度以及第n个字对应的说话者向量表示与任一说话者的画像特征向量之间的第二相似度，确定第n个字对应的说话者向量表示与任一说话者的画像特征向量之间的相关性系数。而后，计算该相关性系数对多个说话者的画像特征向量的加权和。
接着，将第一向量表示、前n-1个字以及相关性系数对多个说话者的画像特征向量的加权和，输入语音识别解码器，获得第n个字对应的后验概率。再根据第n个字对应的后验概率，预测第n个字。最后，根据语音识别器输出的各个字，生成SOT格式的文字序列。
其中，本发明实施例中涉及到的具体的实施过程可以参考上述实施例中内容，在此不再赘述。
为了使得上述实施例中文本编码器、说话者无关打分器、以及说话者相关打分器可以更好地获得语音信号对应地全局上下文信息，进行双阶段解码处理。第一阶段解码就是按照上述各个实施例中的方法逐个预测语音信号对应的各个字。当获得语音信号对应的所有N个字之后，进行第二阶段解码处理，第二阶段解码是通过第一阶段解码处理后得到所有N个字的预测结果，将这N个字的预测结果作为文本编码器的输入，再逐个计算出一个第n个字对应的说话者向量表示分别与多个说话者的画像特征向量之间的相关性系数，即获得N个字中每个字对应的说话者的概率分布，根据N个字中每个字对应的说话者的概率分布，以重新获得各个字对应的说话者，这样可以使得最后获得的N个字的结果更加准确。其具体的实现过程可以参考附图8所示。
图8为本发明实施例提供的另一种语音识别方法的流程图；如图8所示，为了提高该方法语音识别结果的准确性，使得该语音识别方法可以更好地结合全局的上下文信息，在上述实施例的基础上，该方法还可以包括如下步骤：
801、确定语音识别解码器输出的完整的文字序列。
802、通过文本编码器对文字序列进行编码处理，以得到文字序列中各个字对应的第四向量表示。
803、将第一向量表示、第二向量表示和第四向量表示输入说话者解码器中，以获得各个字对应的说话者向量表示。
804、根据各个字对应的说话者向量表示分别与多个说话者的画像特征向量之间的 相关性系数，重新确定各个字对应的说话者。
在本实施例中，在进行第二阶段解码时，首先确定第一阶段语音识别解码器输出的完整的文字序列。然后将完整的文字序列输入至文本编码器，以通过文本编码器对文字序列进行编码处理，以得到文字序列中各个字对应的第四向量表示。而后，将第一向量表示、第二向量表示和第四向量表示输入说话者解码器中，以获得各个字对应的说话者向量表示。根据各个字对应的说话者向量表示分别与多个说话者的画像特征向量之间的相关性系数，重新确定各个字对应的说话者。
由上述描述可知：第一阶段解码是基于前n-1个字预测第n个字，只能获取到该字对应的上文信息，没有下文信息。第二阶段解码是在所有N个字预测完一遍之后，基于所有N个字再进行预测时，这样针对其中每个字就可以看到上文信息和下文信息了，可以使得再计算一遍获得的各个字对应的说话者的预测结果会更准确。
其中，本发明实施例中涉及到的具体的实施过程可以参考上述各个实施例中的内容，在此不再赘述。
本发明实施例提供的语音识别方法可以在云端来执行，在云端可以部署有若干计算节点(云服务器)，每个计算节点中都具有计算、存储等处理资源。在云端，可以组织由多个计算节点来提供某种服务，当然，一个计算节点也可以提供一种或多种服务。云端提供该服务的方式可以是对外提供服务接口，用户调用该服务接口以使用相应的服务。
针对本发明实施例提供的方案，云端可以提供有语音识别服务的服务接口，用户通过终端设备调用该服务接口，以向云端触发语音识别服务请求，该请求中包括多个说话者的语音信号和多个说话者的画像特征向量，语音信号中包含多个说话者的语音，云端确定响应该请求的计算节点，利用该计算节点中的处理资源执行如下步骤：
获取语音识别模型，语音识别模型中包括语音识别编码器、说话者编码器、语音识别解码器、说话者解码器以及包含注意力层的文本编码器；
通过语音识别编码器获取语音信号对应的第一向量表示，通过说话者编码器获取语音信号对应的第二向量表示，第一向量表示被设置为进行语音识别，第二向量表示被设置为进行说话者识别；
通过文本编码器对语音识别解码器已经输出的前n-1个字进行编码处理，以得到第n-1个字所对应的第三向量表示；
将第一向量表示、第二向量表示和第三向量表示输入说话者解码器中，以获得第n个字对应的说话者向量表示；
根据第n个字对应的说话者向量表示分别与多个说话者的画像特征向量之间的相关性系数，确定第n个字对应的说话者；
将第一向量表示、前n-1个字以及相关性系数对多个说话者的画像特征向量的加权和，输入语音识别解码器，以获得第n个字；
向终端设备发送语音识别输出信息，语音识别输出信息中包括多个说话者各自对应的文字序列。
上述执行过程可以参考前述其他实施例中的相关说明，在此不赘述。
为便于理解，结合图9来示例性说明。用户可以通过图9中示意的终端设备E1调用语音识别服务，以对目标语音信号进行语音识别，以获得目标语音信号对应的文本序列预测。用户调用该服务的服务接口包括软件开发工具包(Software Development Kit，简称SDK)、应用程序接口(Application Programming Interface，简称API)等形式。图9中示意的是API接口的情形。在云端，如图中所示，假设由服务集群E2提供语音识别服务，服务集群E2中包括至少一个计算节点。服务集群E2收到该请求后，执行前述实施例中的步骤，以得到目标语音信号对应的包括多个说话者各自对应的文字序列，并反馈给终端设备E1。
以下将详细描述本发明的一个或多个实施例的语音识别装置。本领域技术人员可以理解，这些装置均可使用市售的硬件组件通过本方案所教导的步骤进行配置来构成。
图10为本发明实施例提供的一种语音识别装置的结构示意图，语音识别模型中包括语音识别编码器、说话者编码器、语音识别解码器、说话者解码器以及包含注意力层的文本编码器，如图10所示，该装置包括：获取模块11、第一编码模块12、第二编码模块13、第一解码模块14、第二解码模块15。
获取模块11，被设置为获取多个说话者的语音信号和所述多个说话者的画像特征向量，所述语音信号中包含所述多个说话者的语音。
第一编码模块12，被设置为通过所述语音识别编码器获取所述语音信号对应的第一向量表示，通过所述说话者编码器获取所述语音信号对应的第二向量表示，所述第一向量表示被设置为进行语音识别，所述第二向量表示被设置为进行说话者识别。
第二编码模块13，被设置为通过所述文本编码器对所述语音识别解码器已经输出的前n-1个字进行编码处理，以得到第n-1个字所对应的第三向量表示。
第一解码模块14，被设置为将所述第一向量表示、所述第二向量表示和所述第三向量表示输入所述说话者解码器中，以获得第n个字对应的说话者向量表示；根据所述第n个字对应的说话者向量表示分别与所述多个说话者的画像特征向量之间的相关性系数，确定所述第n个字对应的说话者。
第二解码模块15，被设置为将所述第一向量表示、所述前n-1个字以及所述相关性系数对所述多个说话者的画像特征向量的加权和，输入所述语音识别解码器，以获得所述第n个字。
可选地，所述第一解码模块14还被设置为：针对所述多个说话者中任一说话者的画像特征向量，根据设定的相似度算法，确定所述第n个字对应的说话者向量表示与所述任一说话者的画像特征向量之间的第一相似度；获取前n个字各自对应的说话者向量表示；分别将所述前n个字各自对应的说话者向量表示与所述任一说话者的画像 特征向量进行拼接；将多个拼接后的向量表示输入设定的包含注意力模块的打分模型，以获得所述前n个字各自对应的说话者向量表示分别与所述任一说话者的画像特征向量之间的第二相似度；根据所述第n个字对应的说话者向量表示与所述任一说话者的画像特征向量之间的第一相似度以及所述第n个字对应的说话者向量表示与所述任一说话者的画像特征向量之间的第二相似度，确定所述第n个字对应的说话者向量表示与所述任一说话者的画像特征向量之间的相关性系数。
可选地，所述第二编码模块13还被设置为：获取所述语音识别解码器中的自注意力层分别对所述前n-1个字进行注意力计算后得到的所述前n-1个字分别对应的注意力系数加权后的向量表示；将所述前n-1个字分别对应的注意力系数加权后的向量表示输入所述文本编码器中，以获得所述文本编码器输出的第n-1个字所对应的第三向量表示。
可选地，所述文本编码器中包括级联的至少一个编码器，每个所述编码器中包括注意力层和前馈神经网络层。
可选地，所述语音识别解码器中包括级联的多个解码器，每个解码器中包括注意力层和前馈神经网络层；基于此，所述第二解码模块15还被设置为：对所述前n-1个字分别进行嵌入编码，得到所述前n-1个字各自对应的嵌入向量表示；将所述前n-1个字各自对应的嵌入向量表示和所述第一向量表示输入所述语音识别解码器的第一层解码器中，以获得所述第一层解码器中的注意力层输出的所述第n-1个字对应的注意力系数加权后的向量表示；将所述第n-1个字对应的注意力系数加权后的向量表示和所述加权和输入所述第一层解码器中的前馈神经网络层，以获得输入到第二层解码器的所述第n-1个字对应的向量表示；根据所述语音识别解码器中最后一层解码器输出的所述第n-1个字对应的向量表示，确定所述第n个字。
可选地，所述说话者解码器中包括级联的多个解码器，每个解码器中包括注意力层和前馈神经网络层；基于此，所述第一解码模块14还被设置为：将所述第一向量表示、所述第二向量表示和所述第三向量表示输入所述说话者解码器中的第一层解码器中，以获得所述第一层解码器中的注意力层输出的所述第n-1个字对应的注意力系数加权后的说话者向量表示；将所述第n-1个字对应的注意力系数加权后的说话者向量表示输入所述第一层解码器中的前馈神经网络层，以获得输入到第二层解码器的所述第n-1个字对应的说话者向量表示；确定所述说话者解码器中最后一层解码器输出的所述第n-1个字对应的说话者向量表示；根据所述最后一层解码器输出的所述第n-1个字对应的说话者向量表示和所述第一层解码器中的注意力层输出的所述第n-1个字对应的注意力系数加权后的说话者向量表示，确定所述第n个字对应的说话者向量表示。
可选地，所述第二解码模块15还被设置为：确定所述语音识别解码器输出的完整的文字序列；通过所述文本编码器对所述文字序列进行编码处理，以得到所述文字序 列中各个字对应的第四向量表示；将所述第一向量表示、所述第二向量表示和所述第四向量表示输入所述说话者解码器中，以获得所述各个字对应的说话者向量表示；根据所述各个字对应的说话者向量表示分别与所述多个说话者的画像特征向量之间的相关性系数，重新确定所述各个字对应的说话者。
图10所示装置可以执行前述实施例中语音识别方法中的步骤，详细的执行过程和技术效果参见前述实施例中的描述，在此不再赘述。
本发明实施例还提供一电子设备，如图11所示，该电子设备可以包括：处理器21、存储器22、通信接口23。其中，存储器22上存储有可执行代码，当所述可执行代码被处理器21执行时，使处理器21实现如前述实施例中语音识别方法。
另外，本发明实施例提供了一种非暂时性机器可读存储介质，所述非暂时性机器可读存储介质上存储有可执行代码，当所述可执行代码被电子设备的处理器执行时，使所述处理器至少可以实现如前述实施例中提供的语音识别方法。
以上所描述的装置实施例是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。本领域普通技术人员在不付出创造性的劳动的情况下，即可以理解并实施。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到各实施方式可借助加必需的通用硬件平台的方式来实现，当然也可以通过硬件和软件结合的方式来实现。基于这样的理解，上述技术方案本质上或者说对现有技术做出贡献的部分可以以计算机产品的形式体现出来，本发明可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
最后应说明的是：以上实施例用以说明本发明的技术方案，而非对其限制；尽管参照前述实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的精神和范围。
工业实用性
本公开实施例提供了一种语音识别方法、设备和存储介质，具体实施方案为在语音识别模型中添加包含注意力层的文本编码器，通过文本编码器对语音识别解码器已经输出的前n-1个字进行编码处理，可以更好地聚合整个句子的上下文信息，以获得第n-1个字所对应的包含上下文信息的第三向量表示，即这样可以获得更准确地被设置为进行说话者识别的向量表示，从而提高了各个字对应的说话者识别结果的准确性。

Claims (10)

  1. 一种语音识别方法，其特征在于，语音识别模型中包括语音识别编码器、说话者编码器、语音识别解码器、说话者解码器以及包含注意力层的文本编码器；所述方法包括：
    获取多个说话者的语音信号和所述多个说话者的画像特征向量，所述语音信号中包含所述多个说话者的语音；
    通过所述语音识别编码器获取所述语音信号对应的第一向量表示，通过所述说话者编码器获取所述语音信号对应的第二向量表示，所述第一向量表示被设置为进行语音识别，所述第二向量表示被设置为进行说话者识别；
    通过所述文本编码器对所述语音识别解码器已经输出的前n-1个字进行编码处理，以得到第n-1个字所对应的第三向量表示；
    将所述第一向量表示、所述第二向量表示和所述第三向量表示输入所述说话者解码器中，以获得第n个字对应的说话者向量表示；
    根据所述第n个字对应的说话者向量表示分别与所述多个说话者的画像特征向量之间的相关性系数，确定所述第n个字对应的说话者；
    将所述第一向量表示、所述前n-1个字以及所述相关性系数对所述多个说话者的画像特征向量的加权和，输入所述语音识别解码器，以获得所述第n个字。
  2. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    针对所述多个说话者中任一说话者的画像特征向量，根据设定的相似度算法，确定所述第n个字对应的说话者向量表示与所述任一说话者的画像特征向量之间的第一相似度；
    获取前n个字各自对应的说话者向量表示；
    分别将所述前n个字各自对应的说话者向量表示与所述任一说话者的画像特征向量进行拼接；
    将多个拼接后的向量表示输入设定的包含注意力模块的打分模型，以获得所述前n个字各自对应的说话者向量表示分别与所述任一说话者的画像特征向量之间的第二相似度；
    根据所述第n个字对应的说话者向量表示与所述任一说话者的画像特征向量之间的第一相似度以及所述第n个字对应的说话者向量表示与所述任一说话者的画像特征向量之间的第二相似度，确定所述第n个字对应的说话者向量表示与所述任一说话者的画像特征向量之间的相关性系数。
  3. 根据权利要求1所述的方法，其特征在于，所述通过所述文本编码器对所述语音识别解码器已经输出的前n-1个字进行编码处理，以得到第n-1个字所对应的第三向量表示，包括：
    获取所述语音识别解码器中的自注意力层分别对所述前n-1个字进行注意力计算后得到的所述前n-1个字分别对应的注意力系数加权后的向量表示；
    将所述前n-1个字分别对应的注意力系数加权后的向量表示输入所述文本编码器中，以获得所述文本编码器输出的第n-1个字所对应的第三向量表示。。
  4. 根据权利要求3所述的方法，其特征在于，所述文本编码器中包括级联的至少一个编码器，每个所述编码器中包括注意力层和前馈神经网络层。
  5. 根据权利要求1所述的方法，其特征在于，所述语音识别解码器中包括级联的多个解码器，每个解码器中包括注意力层和前馈神经网络层；
    所述将所述第一向量表示、所述前n-1个字以及所述相关性系数对所述多个说话者的画像特征向量的加权和，输入所述语音识别解码器，以获得所述第n个字，包括：
    对所述前n-1个字分别进行嵌入编码，得到所述前n-1个字各自对应的嵌入向量表示；
    将所述前n-1个字各自对应的嵌入向量表示和所述第一向量表示输入所述语音识别解码器的第一层解码器中，以获得所述第一层解码器中的注意力层输出的所述第n-1个字对应的注意力系数加权后的向量表示；
    将所述第n-1个字对应的注意力系数加权后的向量表示和所述加权和输入所述第一层解码器中的前馈神经网络层，以获得输入到第二层解码器的所述第n-1个字对应的向量表示；
    根据所述语音识别解码器中最后一层解码器输出的所述第n-1个字对应的向量表示，确定所述第n个字。
  6. 根据权利要求1所述的方法，其特征在于，所述说话者解码器中包括级联的多个解码器，每个解码器中包括注意力层和前馈神经网络层；
    所述将所述第一向量表示、所述第二向量表示和所述第三向量表示输入所述说话者解码器中，以获得第n个字对应的说话者向量表示，包括：
    将所述第一向量表示、所述第二向量表示和所述第三向量表示输入所述说话者解码器中的第一层解码器中，以获得所述第一层解码器中的注意力层输出的所述第n-1个字对应的注意力系数加权后的说话者向量表示；
    将所述第n-1个字对应的注意力系数加权后的说话者向量表示输入所述第一层解码器中的前馈神经网络层，以获得输入到第二层解码器的所述第n-1个字对应的说话者向量表示；
    确定所述说话者解码器中最后一层解码器输出的所述第n-1个字对应的说话者向量表示；
    根据所述最后一层解码器输出的所述第n-1个字对应的说话者向量表示和所述第一层解码器中的注意力层输出的所述第n-1个字对应的注意力系数加权后的说话者向量表示，确定所述第n个字对应的说话者向量表示。
  7. 根据权利要求1至6中任一项所述的方法，其特征在于，所述方法还包括；
    确定所述语音识别解码器输出的完整的文字序列；
    通过所述文本编码器对所述文字序列进行编码处理，以得到所述文字序列中各个字对应的第四向量表示；
    将所述第一向量表示、所述第二向量表示和所述第四向量表示输入所述说话者解码器中，以获得所述各个字对应的说话者向量表示；
    根据所述各个字对应的说话者向量表示分别与所述多个说话者的画像特征向量之间的相关性系数，重新确定所述各个字对应的说话者。
  8. 一种语音识别方法，其特征在于，所述方法包括：
    接收终端设备通过调用语音识别服务触发的请求，所述请求中包括多个说话者的语音信号和所述多个说话者的画像特征向量，所述语音信号中包含所述多个说话者的语音；
    根据所述模型训练服务对应的计算资源执行如下步骤：
    获取语音识别模型，所述语音识别模型中包括语音识别编码器、说话者编码器、语音识别解码器、说话者解码器以及包含注意力层的文本编码器；
    通过所述语音识别编码器获取所述语音信号对应的第一向量表示，通过所述说话者编码器获取所述语音信号对应的第二向量表示，所述第一向量表示被设置为进行语音识别，所述第二向量表示被设置为进行说话者识别；
    通过所述文本编码器对所述语音识别解码器已经输出的前n-1个字进行编码处理，以得到第n-1个字所对应的第三向量表示；
    将所述第一向量表示、所述第二向量表示和所述第三向量表示输入所述说话者解码器中，以获得第n个字对应的说话者向量表示；
    根据所述第n个字对应的说话者向量表示分别与所述多个说话者的画像特征向量之间的相关性系数，确定所述第n个字对应的说话者；
    将所述第一向量表示、所述前n-1个字以及所述相关性系数对所述多个说话者的画像特征向量的加权和，输入所述语音识别解码器，以获得所述第n个字；
    向所述终端设备发送语音识别输出信息，所述语音识别输出信息中包括所述多个说话者各自对应的文字序列。
  9. 一种电子设备，其特征在于，包括：存储器、处理器、通信接口；其中，所述存储器上存储有可执行代码，当所述可执行代码被所述处理器执行时，使所述处理器执行如权利要求1至7中任一项所述的语音识别方法。
  10. 一种非暂时性机器可读存储介质，其特征在于，所述非暂时性机器可读存储介质上存储有可执行代码，当所述可执行代码被电子设备的处理器执行时，使所述处理器执行如权利要求1至7中任一项所述的语音识别方法。
PCT/CN2024/097609 2023-06-08 2024-06-05 语音识别方法、设备和存储介质 Ceased WO2024251169A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202310680032.X 2023-06-08
CN202310680032.XA CN116825095A (zh) 2023-06-08 2023-06-08 语音识别方法、设备和存储介质

Publications (1)

Publication Number Publication Date
WO2024251169A1 true WO2024251169A1 (zh) 2024-12-12

Family

ID=88123382

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/097609 Ceased WO2024251169A1 (zh) 2023-06-08 2024-06-05 语音识别方法、设备和存储介质

Country Status (2)

Country Link
CN (1) CN116825095A (zh)
WO (1) WO2024251169A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116825095A (zh) * 2023-06-08 2023-09-29 阿里巴巴（中国）有限公司 语音识别方法、设备和存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109215662A (zh) * 2018-09-18 2019-01-15 平安科技（深圳）有限公司 端对端语音识别方法、电子装置及计算机可读存储介质
US20200043467A1 (en) * 2018-07-31 2020-02-06 Tencent Technology (Shenzhen) Company Limited Monaural multi-talker speech recognition with attention mechanism and gated convolutional networks
CN111009237A (zh) * 2019-12-12 2020-04-14 北京达佳互联信息技术有限公司 语音识别方法、装置、电子设备及存储介质
CN111899727A (zh) * 2020-07-15 2020-11-06 苏州思必驰信息科技有限公司 用于多说话人的语音识别模型的训练方法及系统
CN113889089A (zh) * 2021-09-29 2022-01-04 北京百度网讯科技有限公司 语音识别模型的获取方法、装置、电子设备以及存储介质
CN116825095A (zh) * 2023-06-08 2023-09-29 阿里巴巴（中国）有限公司 语音识别方法、设备和存储介质

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115713939B (zh) * 2023-01-06 2023-04-21 阿里巴巴达摩院(杭州)科技有限公司 语音识别方法、装置及电子设备

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200043467A1 (en) * 2018-07-31 2020-02-06 Tencent Technology (Shenzhen) Company Limited Monaural multi-talker speech recognition with attention mechanism and gated convolutional networks
CN109215662A (zh) * 2018-09-18 2019-01-15 平安科技（深圳）有限公司 端对端语音识别方法、电子装置及计算机可读存储介质
CN111009237A (zh) * 2019-12-12 2020-04-14 北京达佳互联信息技术有限公司 语音识别方法、装置、电子设备及存储介质
CN111899727A (zh) * 2020-07-15 2020-11-06 苏州思必驰信息科技有限公司 用于多说话人的语音识别模型的训练方法及系统
CN113889089A (zh) * 2021-09-29 2022-01-04 北京百度网讯科技有限公司 语音识别模型的获取方法、装置、电子设备以及存储介质
CN116825095A (zh) * 2023-06-08 2023-09-29 阿里巴巴（中国）有限公司 语音识别方法、设备和存储介质

Also Published As

Publication number Publication date
CN116825095A (zh) 2023-09-29

Similar Documents

Publication Publication Date Title
CN109785824B (zh) 一种语音翻译模型的训练方法及装置
CN106683677B (zh) 语音识别方法及装置
CN109979432B (zh) 一种方言翻译方法及装置
CN114097026A (zh) 语音识别的上下文偏置
US11721324B2 (en) Providing high quality speech recognition
EP4273855B1 (en) Speech recognition method and apparatus, and storage medium
KR20230086737A (ko) 단순화된 스트리밍 및 비스트리밍 스피치 인식을 위한 캐스케이드 인코더들
CN111081230A (zh) 语音识别方法和设备
CN112489651B (zh) 语音识别方法和电子设备、存储装置
JP7329393B2 (ja) 音声信号処理装置、音声信号処理方法、音声信号処理プログラム、学習装置、学習方法及び学習プログラム
CN112951211B (zh) 一种语音唤醒方法及装置
US11380315B2 (en) Characterizing accuracy of ensemble models for automatic speech recognition by determining a predetermined number of multiple ASR engines based on their historical performance
WO2024088262A1 (zh) 语音识别模型的数据处理系统及方法、语音识别方法
CN119547136A (zh) 用于罕见词语音辨识的场境感知神经置信度估计
CN117935787B (zh) 一种数据筛选标注方法、装置、电子设备和存储介质
EP3939034B1 (en) Characterizing accuracy of ensemble models for automatic speech recognition
CN115273862B (zh) 语音处理的方法、装置、电子设备和介质
CN113470617A (zh) 语音识别方法以及电子设备、存储装置
Thukroo et al. Spoken language identification system for Kashmiri and related languages using mel-spectrograms and deep learning approach
CN114495914B (zh) 语音识别方法、语音识别模型的训练方法及相关装置
KR20240068723A (ko) Rnn-T로 구현된 자동 음성 인식 시스템에서 음향과 텍스트 표현의 융합
WO2025183894A1 (en) Speculative asr decoding to reduce overall latency of speech applications
WO2024251169A1 (zh) 语音识别方法、设备和存储介质
CN117711376A (zh) 语种识别方法、系统、设备及存储介质
CN113793599B (zh) 语音识别模型的训练方法和语音识别方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24818701

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE