TWI725111B - Voice-based role separation method and device - Google Patents

Abstract

This application discloses a voice-based role separation method, which includes: extracting feature vectors frame by frame from a voice signal to obtain feature vector sequences; assigning role labels to the feature vectors; and training a deep neural network DNN model using feature vectors with role labels According to the DNN model and the hidden Markov model HMM obtained by using feature vector training, determine the character sequence corresponding to the feature vector sequence, and output the role separation result; wherein, the DNN model is used to output the corresponding each according to the input feature vector The probability of the role, HMM is used to describe the jump relationship between the roles. This application also provides a voice-based role separation device. The above-mentioned method provided in this application uses a DNN model with powerful feature extraction capabilities to model the speaker's character, and has a stronger characterization ability than the traditional GMM, and the characterization of the character is more precise and accurate, so it can obtain More accurate role separation results.

Description

Voice-based role separation method and device

This application relates to the field of speech recognition, and specifically relates to a voice-based role separation method. This application also relates to a voice-based role separation device.

Voice is the most natural way for humans to communicate, and voice recognition technology is a technology that allows machines to convert voice signals into corresponding texts or commands through the process of recognition and understanding. Speech recognition is an interdisciplinary subject. The fields involved include: signal processing, pattern recognition, probability theory and information theory, sound mechanism and hearing mechanism, artificial intelligence, and so on.

In practical applications, in order to be able to analyze the speech signal more accurately, it is not only necessary to perform speech recognition, but also to identify the speaker of each segment of speech, so naturally there is a need to separate speech according to roles. In many scenarios such as daily life, conferences, and telephone conversations, there are dialogue voices. By separating the roles of dialogue voices, it can be determined which part of the voice is spoken by one person and which part of the voice is spoken by the other person. After the dialogue voice is separated according to roles, combined with speaker recognition and voice recognition, a broader application space will be generated. For example, the dialogue voice of the customer service center is separated according to roles, and then voice recognition can determine what the customer service said. , What the customer said, so that the corresponding customer service quality inspection can be carried out or the potential demand of the customer can be explored.

In the prior art, GMM (Gaussian Mixture Model) and HMM (Hidden Markov Model) are usually used to separate the roles of dialogue speech, that is, GMM modeling is used for each role, and for different roles The jump is modeled by HMM. Since GMM modeling technology was proposed earlier, and its ability to fit arbitrary functions depends on the number of mixed Gaussian functions, its ability to characterize characters has certain limitations, resulting in a relatively low accuracy of role separation. , Unable to meet the needs of the application.

The embodiments of the present application provide a voice-based role separation method and device to solve the problem of relatively low accuracy of the existing role separation technology based on GMM and HMM.

This application provides a voice-based role separation method, including: extracting feature vectors frame by frame from a voice signal to obtain a feature vector sequence; assigning role labels to the feature vectors; training a deep neural network DNN model by using the feature vectors with role labels; According to the DNN model and the hidden Markov model HMM obtained by the feature vector training, determine the character sequence corresponding to the feature vector sequence, and output the character separation result; wherein, the DNN model is used to output the corresponding character according to the input feature vector HMM is used to describe the jump relationship between characters.

Optionally, after the step of extracting feature vectors frame by frame from the speech signal and before the step of assigning role labels to the feature vectors, the following operations are performed: by identifying and excluding audio frames that do not contain voice content, The voice signal is divided into voice segments; the assigning role labels to feature vectors includes: assigning role labels to feature vectors in each voice segment; determining the role sequence corresponding to the feature vector sequence includes: determining where each voice segment is The character sequence corresponding to the included feature vector sequence.

Optionally, the allocating role labels to the feature vectors in each voice segment includes: assigning role tags to the feature vectors in each voice segment by establishing Gaussian mixture models GMM and HMM; wherein the GMM is used for each role Outputting the probability that the feature vector corresponds to the character according to the input feature vector; said determining the character sequence corresponding to the feature vector sequence contained in each speech segment according to the DNN model and the HMM trained using the feature vector includes: According to the DNN model and the HMM used to assign role labels to the feature vectors in each voice segment, determine the role sequence corresponding to the feature vector sequence contained in each voice segment.

Optionally, by establishing the Gaussian mixture model GMM and HMM, assigning role labels to the feature vectors in each voice segment includes: selecting a corresponding number of voice segments according to a preset initial number of roles, and separately for each voice segment Specify different roles; use the feature vector in the voice segment of the specified role to train GMM and HMM for each role; decode the GMM and HMM obtained from the training, and obtain the probability value ranking of the feature vector sequence contained in each voice segment The leading role sequence; determine whether the probability value corresponding to the role sequence is greater than a preset threshold; if so, assign role labels to the feature vectors in each voice segment according to the role sequence.

Optionally, when the result of judging whether the probability value corresponding to the role sequence is greater than a preset threshold is No, perform the following operation: according to the role sequence, specify a corresponding role for each voice segment; For the feature vectors and corresponding roles in each voice segment, train the GMM and HMM for each role; turn to the step of decoding according to the GMM and HMM obtained by training.

Optionally, the specifying a corresponding role for each voice segment according to the role sequence includes: for each voice segment, specifying the mode of the role corresponding to each feature vector as the role of the voice segment.

Optionally, the training of the GMM and HMM for each role according to the feature vector in each voice segment and the corresponding role includes: training the GMM in an incremental manner on the basis of the model obtained in the previous training, and HMM.

Optionally, when the result of judging whether the probability value corresponding to the character sequence is greater than a preset threshold is No, perform the following operation: judging whether the number of times of training GMM and HMM under the current number of characters is less than the preset training The upper limit of the number of times; if yes, perform the step of assigning a corresponding role to each voice segment according to the role sequence; if not, perform the following operation: adjust the number of roles, select the corresponding number of voice segments and separate for each voice segment Specify different roles; and go to the step of using the feature vector in the voice segment of the specified role to train GMM and HMM for each role.

Optionally, when the result of judging whether the number of times of training GMM and HMM under the current number of characters is less than the preset upper limit of the number of training times is no, the following operation is performed: judging whether the current number of characters meets the preset requirements; if so, Turn to the step of assigning role labels to the feature vectors in each voice segment according to the role sequence, and if not, execute the step of adjusting the number of roles.

Optionally, the preset initial number of characters is 2, and the adjusting the number of characters includes: adding 1 to the current number of characters.

Optionally, extracting the feature vector frame by frame from the voice signal to obtain the feature vector sequence includes: framing the voice signal according to a preset frame length to obtain multiple audio frames; extracting the feature vector of each audio frame, Obtain the feature vector sequence.

Optionally, the extracting the feature vector of each audio frame includes: extracting the MFCC feature, the PLP feature, or the LPC feature.

Optionally, the recognizing and removing audio frames that do not contain voice content includes: using VAD technology to recognize the audio frames that do not contain voice content, and performing a corresponding removal operation.

Optionally, after VAD technology is used to perform the recognition and removal operations, and the voice signal is divided into voice segments, the following VAD smoothing operation is performed: a voice segment whose duration is less than a preset threshold and adjacent voices Segment merge.

Optionally, the training of the deep neural network DNN model by using the feature vector with the role label includes: training the DNN model by using a backpropagation algorithm.

Optionally, the determining the character sequence corresponding to the feature vector sequence according to the DNN model and the hidden Markov model HMM trained by using feature vectors includes: performing a decoding operation according to the DNN model and the HMM to obtain and output the feature The probability value of the vector sequence is sorted by the first character sequence, and the character sequence is taken as the character sequence corresponding to the feature vector sequence.

Optionally, the output of the character separation result includes: outputting the start and end time information of the audio frame to which the corresponding feature vector belongs for each character according to the character sequence corresponding to the feature vector sequence.

Optionally, the selecting a corresponding number of voice segments includes: selecting the number of voice segments whose duration meets a preset requirement.

Correspondingly, the present application also provides a voice-based role separation device, including: a feature extraction unit for extracting feature vectors from the voice signal frame by frame to obtain a feature vector sequence; a label assignment unit for assigning role labels to the feature vectors The DNN model training unit is used to train the DNN model using feature vectors with role labels, wherein the DNN model is used to output the probability corresponding to each role according to the input feature vector; the role determination unit is used to train the DNN model according to the DNN model Using the HMM trained with the feature vector, determine the character sequence corresponding to the feature vector sequence and output the character separation result, wherein the HMM is used to describe the jump relationship between the characters.

Optionally, the device further includes: a voice segment segmentation unit, configured to identify and eliminate audio frames that do not contain voice content after the feature extraction unit extracts the feature vector and before triggering the label distribution unit to work , Divide the voice signal into voice segments; the label assignment unit is specifically configured to assign role labels to the feature vectors in each voice segment; the role determination unit is specifically configured to, according to the DNN model and the utilization feature The HMM obtained by vector training determines the character sequence corresponding to the feature vector sequence contained in each speech segment and outputs the character separation result.

Optionally, the label allocation unit is specifically configured to allocate role labels to feature vectors in each voice segment by establishing GMM and HMM, where the GMM is used to output the feature for each role according to the input feature vector The probability that the vector corresponds to the role; the role determination unit is specifically configured to determine the features contained in each voice segment according to the DNN model and the HMM used to assign role labels to the feature vectors in each voice segment The role sequence corresponding to the vector sequence.

Optionally, the label assignment unit includes: an initial role specifying subunit, which is used to select a corresponding number of voice segments according to a preset initial number of roles, and to specify different roles for each voice segment; an initial model training subunit, It is used to train the GMM and HMM for each role by using the feature vector in the voice segment of the specified role; the decoding subunit is used to decode the GMM and HMM obtained by training to obtain the feature vector sequence contained in each voice segment. The probability value of the character sequence is ranked higher; the probability judgment subunit is used to judge whether the probability value corresponding to the role sequence is greater than a preset threshold; the label allocation subunit is used when the output of the probability judgment subunit is yes At this time, according to the role sequence, a role label is assigned to the feature vector in each speech segment.

Optionally, the label assignment unit further includes: a voice segment role designation subunit, configured to specify a corresponding role for each voice segment according to the role sequence when the output of the probability judgment subunit is no ; The model update training subunit is used to train the GMM and HMM for each role according to the feature vector in each voice segment and the corresponding role, and trigger the decoding subunit to work.

Optionally, the voice segment-by-segment role specification subunit is specifically configured to specify, for each voice segment, the mode of the role corresponding to each feature vector as the role of the voice segment.

Optionally, the model update training subunit is specifically configured to train the GMM and HMM in an incremental manner on the basis of the model obtained in the previous training.

Optionally, the label allocation unit further includes: a training frequency judgment subunit, configured to judge whether the number of times of training GMM and HMM under the current number of characters is less than a preset number when the output of the probability judgment subunit is no The upper limit of the number of times of training, and when the judgment result is yes, trigger the work of the character designation subunits per voice segment.

The number of roles adjustment subunit is used to adjust the number of roles when the output of the number of training judgement subunits is no, select a corresponding number of voice segments and assign different roles for each voice segment, and trigger the initial model training Subunit work.

Optionally, the label assigning unit further includes: a character quantity judging subunit, which is used to judge whether the current character quantity meets a preset requirement when the output of the training times judging subunit is no, and if so, trigger the The label allocation subunit works, otherwise the role quantity adjustment subunit is triggered to work.

Optionally, the feature extraction unit includes: a framing subunit, configured to perform framing processing on the voice signal according to a preset frame length to obtain multiple audio frames; a feature extraction execution subunit, used to extract each audio frame To obtain the feature vector sequence.

Optionally, the feature extraction execution subunit is specifically configured to extract MFCC features, PLP features, or LPC features of each audio frame to obtain the feature vector sequence.

Optionally, the voice segment segmentation unit is specifically configured to recognize and eliminate the audio frames that do not contain voice content by using VAD technology, and segment the voice signal into voice segments.

Optionally, the device further includes: a VAD smoothing unit, configured to merge a voice segment with a duration less than a preset threshold with an adjacent voice segment after the voice segment segmentation unit uses the VAD technology to segment the voice segment.

Optionally, the DNN model training unit is specifically configured to train the DNN model by using a backpropagation algorithm.

Optionally, the role determination unit is specifically configured to perform a decoding operation according to the DNN model and the HMM, obtain the role sequence with the highest ranking of the probability values of outputting the feature vector sequence, and use the role sequence as the related The character sequence corresponding to the feature vector sequence.

Optionally, the character determination unit outputs the character separation result in the following manner: according to the character sequence corresponding to the feature vector sequence, for each character, output the start and end time information of the audio frame to which the corresponding feature vector belongs.

Optionally, the initial role designation subunit or the role number adjustment subunit specifically selects a corresponding number of voice segments in the following manner: selecting the number of voice segments whose duration meets a preset requirement.

Compared with the prior art, this application has the following advantages:

The voice-based role separation method provided by this application first extracts a feature vector sequence frame by frame from a voice signal, and then trains a DNN model on the basis of assigning role labels to the feature vector, and trains the DNN model according to the DNN model and using the feature vector. In the HMM, the character sequence corresponding to the feature vector sequence is determined, and the result of the character separation is obtained. The above-mentioned method provided in this application uses a DNN model with powerful feature extraction capabilities to model the speaker's character, and has a stronger characterization ability than the traditional GMM, and the characterization of the character is more precise and accurate, so it can obtain More accurate role separation results.

601‧‧‧Feature Extraction Unit

602‧‧‧label distribution unit

603‧‧‧DNN model training unit

604‧‧‧Role Judgment Unit

Fig. 1 is a flowchart of an embodiment of a voice-based role separation method of the present application; Fig. 2 is a processing flowchart of extracting a feature vector sequence from a voice signal provided by an embodiment of the present application; Fig. 3 is a flowchart provided by an embodiment of the present application The process flow chart of using GMM and HMM to assign role labels to the feature vectors in each voice segment; FIG. 4 is a schematic diagram of the voice segment division provided by an embodiment of the present application; FIG. 5 is the topology of the DNN network provided by an embodiment of the present application Structural schematic diagram; FIG. 6 is a schematic diagram of an embodiment of a voice-based role separation device of the present application.

In the following description, many specific details are set forth in order to fully understand this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar promotion without violating the connotation of this application. Therefore, this application is not limited by the specific implementation disclosed below.

In this application, a voice-based role separation method and a voice-based role separation device are respectively provided, which are described in detail in the following embodiments. For ease of understanding, before describing the embodiments, a brief description of the technical background, technical solutions, and writing methods of the embodiments of the present application will be given first.

Existing role separation technologies applied in the speech field usually use GMM (Gaussian mixture model) to model characters, and HMM (Hidden Markov Model) to model jumps between characters.

The HMM is a statistical model used to describe a Markov process with hidden unknown parameters. Hidden Markov model is a kind of Markov chain. Its state (called hidden state) cannot be directly observed, but it is probabilistically related to the observable observation vector. Therefore, HMM is a double random process that includes two parts: Markov chain with state transition probability (usually described by transition matrix A), and a random process describing the output relationship between the hidden state and the observation vector (usually described by confusion matrix B, where each element of the hidden state corresponds to the observation The output probability of the vector, also known as the transmitter rate). An HMM with N states can be represented by the triple parameter λ={π,A,B}, where π is the initial probability of each state.

The GMM can be simply understood as the superposition of multiple Gaussian density functions. Its core idea is to use the combination of multiple Gaussian distribution probability density functions to describe the distribution of feature vectors in the probability space. This model can be used to smoothly approximate any shape The density distribution. Its parameters include: the mixing weight of each Gaussian distribution, the mean vector, and the covariance matrix.

In the existing voice-based role separation applications, GMM is usually used to model each role. The state in the HMM is each role. The observation vector is the feature vector extracted from the voice signal frame by frame. Each state outputs the emission of the feature vector. The probability is determined by GMM (the confusion matrix can be obtained according to GMM), and the role separation process is the process of using GMM and HMM to determine the role sequence corresponding to the feature vector sequence.

Since the function fitting function of GMM is limited by the number of Gaussian density functions used, its own expressive ability has certain limitations, resulting in a relatively low accuracy of the existing GMM and HMM for role separation. In response to this problem, the technical solution of this application uses a deep neural network (DNN) to determine the transmitter rate of each state of the HMM on the basis of pre-assigning role labels to the feature vector of each speech frame, and determines and features according to DNN and HMM For the character sequence corresponding to the vector sequence, because DNN has the powerful ability to combine low-level features to form more abstract high-level features, it can achieve more accurate character descriptions, and therefore can obtain more accurate character separation results.

The technical solution of the present application first assigns role labels to the feature vectors extracted from the speech signal. At this time, the assigned role labels are usually not very accurate, but can provide a reference for the subsequent implementation of the supervised learning process, and train on this basis The obtained DNN model can describe the characters more accurately, so that the result of the character separation is more accurate. In the specific implementation of the technical solutions of the present application, statistical algorithms or classifiers can be used to implement the function of assigning role tags. In the following embodiments provided in the present application, GMM and HMM are used as features. The implementation of vector assignment role tags.

Hereinafter, the embodiments of the present application will be described in detail. Please refer to FIG. 1, which is a flowchart of an embodiment of a voice-based role separation method of this application. The method includes the following steps:

Step 101: Extract feature vectors frame by frame from the speech signal to obtain a feature vector sequence.

The voice signal to be separated is usually a time-domain signal. In this step, a feature vector sequence that can characterize the voice signal is obtained through two processing processes of framing and feature vector extraction. This will be further described with reference to FIG. 2 below.

Step 101-1: Perform framing processing on the voice signal according to the preset frame length to obtain multiple audio frames.

In specific implementation, the frame length can be preset according to requirements, for example, it can be set to 10ms, or 15ms, etc., and then the time domain speech signal is segmented frame by frame according to the frame length, thereby dividing the speech signal into multiple Audio frame. According to different splitting strategies adopted, adjacent audio frames may not overlap or overlap.

Step 101-2: Extract the feature vector of each audio frame to obtain the feature vector sequence.

After the speech signal in the time domain is divided into multiple audio frames, the feature vector that can characterize the speech signal can be extracted frame by frame. Due to the relatively weak description ability of the speech signal in the time domain, the Fourier transform can usually be performed for each audio frame, and then the frequency domain features can be extracted as the feature vector of the audio frame. For example, the MFCC (Mel Frequency Cepstrum Coefficient-Meer) can be extracted. Frequency cepstral coefficient) feature, PLP (Perceptual Linear Predictive-perceptual linear prediction) feature, or LPC (Linear Predictive Coding-linear predictive coding) feature, etc.

The following takes the extraction of the MFCC feature of a certain audio frame as an example to further describe the feature vector extraction process. First, the time-domain signal of the audio frame is obtained through FFT (Fast Fourier Transformation) to obtain the corresponding spectrum information, the spectrum information is passed through the Mel filter bank to obtain the Mel spectrum, and the cepstrum analysis is performed on the Mel spectrum. The core generally uses DCT (Discrete Cosine Transform) for inverse transformation, and then takes preset N coefficients (for example, N=12 or 38) to obtain the feature vector of the audio frame: MFCC feature. Each audio frame is processed in the above manner to obtain a series of feature vectors that characterize the speech signal, that is, the feature vector sequence described in this application.

Step 102: Assign a role label to the feature vector.

In this embodiment, role labels are assigned to the feature vectors in the feature vector sequence by establishing GMM and HMM. Considering that in addition to the voice signal corresponding to each role, a voice signal may also include parts without voice content, for example: silent parts caused by listening, thinking, etc. Since these parts do not contain character information, in order to improve the accuracy of character separation, such audio frames can be identified and eliminated from the speech signal in advance.

Based on the above considerations, this embodiment first removes audio frames that do not contain voice content and divides the voice segments before assigning role labels to feature vectors, and then assigns role labels to feature vectors in each voice segment on this basis. The assignment of role labels includes: initial division of roles, repeated calculations and training of GMM and HMM based on the initial division, if the trained model does not meet the preset requirements, adjust the number of roles and then retrain GMM and HMM until the training is obtained If the model meets the preset requirements, the feature vector in each speech segment is assigned role labels according to the model. The above processing process will be described in detail below with reference to FIG. 3.

Step 102-1: Divide the voice signal into voice segments by identifying and excluding audio frames that do not contain voice content.

The prior art usually adopts an acoustic segmentation method, that is, according to an existing model, separating, for example, "music segment", "speech segment", "silent segment", etc. from the speech signal. In this way, the acoustic models corresponding to various audio segments need to be trained in advance, such as the acoustic model corresponding to the “music segment”. Based on the acoustic model, the audio segment corresponding to the acoustic model can be separated from the speech signal.

Preferably, the technical solution of the present application can use VAD (Voice Activity Detection) technology to identify parts that do not contain voice content. In this way, compared to the technology that uses the acoustic segmentation method, there is no need to train different audio signals in advance. The acoustic model corresponding to the segment is more adaptable. For example, it is possible to identify whether the audio frame is a silent frame by calculating the energy characteristics of the audio frame, the zero-crossing rate, etc. For the presence of environmental noise and relatively strong conditions, the above-mentioned multiple methods can be used for identification, or the establishment of a noise model can be used to identify .

After recognizing audio frames that do not contain voice content, on the one hand, this part of the audio frames can be removed from the voice signal to improve the accuracy of role separation; on the other hand, by identifying audio frames that do not contain voice content, it is equivalent to The starting point and ending point of each valid voice (including voice content) are recognized, so the voice segment can be divided on this basis.

Please refer to FIG. 4, which is a schematic diagram of the voice segment division provided by this embodiment. In this figure, the audio frames between _{time t 2} and t _{3 and} between t ₄ and t _{5 are detected through VAD technology.} It is a mute frame, this step can remove this part of the mute frame from the voice signal, and accordingly divide 3 voice segments: voice segment 1 (seg1) located between _{t 1} and t _{2 and between t 3} and t ₄ speech segment 2 (seg2), and located between the feature vector t and t _{5. 6} speech segment 3 (seg3), each voice segment including a number of audio frames, each audio frame has a corresponding. On the basis of dividing the voice segment, you can roughly assign roles, which is convenient to provide a reasonable starting point for subsequent training.

Preferably, after using the VAD technology to perform the above-mentioned processing, the VAD smoothing operation can also be performed. This is mainly due to the actual vocalization of human beings. The duration of the real speech segment will not be too short. If the VAD operation is performed, the duration of some speech segments obtained is less than the preset threshold (for example, the length of the speech segment). It is 30ms, and the preset threshold is 100ms), then such a voice segment can be merged with an adjacent voice segment to form a longer voice segment. The voice segment division obtained after VAD smoothing is closer to the real situation, which helps to improve the accuracy of role separation.

In this step, the voice signal is divided into several voice segments through the VAD technology, and the task of the subsequent steps 102-2 to 102-11 is to use GMM and HMM to assign role labels to the feature vectors in each voice segment.

Step 102-2: Select a corresponding number of voice segments according to the preset initial number of roles, and specify different roles for each voice segment.

In this step, you can randomly select the same number of voice segments as the initial characters from the already divided voice segments. Considering that the selected voice segment will be used for initial training of GMM and HMM, if the duration is relatively short, it can be used for training. If the data is too long, the possibility of including more than one character will increase. Both of these situations are not conducive to the initial training. Therefore, this embodiment provides a better implementation method, that is, when selecting according to the initial number of characters Long voice segments that meet the preset requirements, and assign different roles for each voice segment.

In this embodiment, the preset initial number of characters is 2, and the preset requirement for selecting a voice segment is: the duration is between 2s and 4s, so this step selects 2 voice segments that meet the above requirements from the divided voice segments Voice segments, and assign different roles for each voice segment. Still taking the voice segment division shown in Figure 4 as an example, seg1 and seg2 each meet the above-mentioned duration requirements, so you can select two voice segments, seg1 and seg2, and assign role 1 (s1) for seg1 and seg2 2(s2).

Step 102-3: Use the feature vector in the voice segment of the designated role to train GMM and HMM for each role.

This step trains the GMM for each role and the HMM that describes the jump relationship between the roles according to the feature vector contained in the voice segment of the specified role. This step is the initial training performed under a specific number of roles. Still taking the voice segment division shown in Figure 4 as an example, under the initial number of characters, the feature vector contained in seg1 is used to train the GMM (gmm1) of character 1, and the feature vector contained in seg2 is used to train the GMM of character 2 ( gmm2), if the GMM and HMM trained under this number of roles do not meet the requirements, you can adjust the number of roles and repeat to this step, and perform the corresponding initial training according to the adjusted number of roles.

The process of training GMM and HMM for each role is the process of learning various parameters related to HMM on the basis of a given observation sequence (ie: the sequence of feature vectors contained in each speech segment, that is, training samples). Each parameter includes: HMM transition matrix A, GMM mean vector corresponding to each role, covariance matrix and other parameters. In the specific implementation, the Baum-Welch algorithm can be used for training. First estimate the initial value of each parameter based on the training sample. According to the training sample and the initial value of each parameter, estimate the posterior probability of _{being in a certain state s j at time t} (s _j ), and then update the various parameters of the HMM according to the calculated posterior probability, and then estimate the posterior probability γt(s _j ) again according to the training sample and the updated parameters..., repeat the calculation In the above process, until a set of HMM parameters is found to maximize the probability of outputting the observation sequence. After the parameters that meet the above requirements are obtained, the initial training of GMM and HMM under a specific number of roles is completed.

Step 102-4: Decode according to the GMM and HMM obtained by training, and obtain the character sequence with the highest probability value of the feature vector sequence contained in each speech segment.

In step 102-1, the speech signal has been divided into several speech segments, and each audio frame in each speech segment has a corresponding feature vector, which together form the feature vector sequence described in this step. In this step, on the basis of the given feature vector sequence and the trained GMM and HMM, the HMM state sequence to which the feature vector sequence may be subordinate is found, that is, the role sequence.

The function completed in this step is the usual HMM decoding process. According to the feature vector sequence, search and output the character sequence with the highest probability value of the feature vector sequence. As a preferred embodiment, usually the maximum probability value corresponding The role sequence of, that is, the role sequence that is most likely to output the feature vector sequence, is also called the best hidden state sequence.

In a specific implementation, an exhaustive search method can be used to calculate the probability value of each possible character sequence outputting the feature vector sequence, and select the maximum value from it. In order to improve the calculation efficiency, as a preferred embodiment, the Viterbi algorithm can be used to reduce the complexity of the calculation by using the invariance of the HMM transfer probability in time, and the output of the feature vector sequence is obtained in the search After the maximum probability value, go back according to the information recorded during the search process to obtain the corresponding character sequence.

Step 102-5: Determine whether the probability value corresponding to the role sequence is greater than a preset threshold, if yes, execute step 102-6, otherwise go to step 102-7 for execution.

If the probability value corresponding to the character sequence obtained through the decoding process in step 102-4 is greater than the preset threshold, such as 0.5, it can usually be considered that the current GMM and HMM are stable, and step 102-6 can be performed to identify the characteristics of each voice segment The vector assigns role labels (the subsequent step 104 can use the stabilized HMM to determine the role sequence corresponding to the feature vector sequence), otherwise, go to step 102-7 to determine whether to continue the repeated operation training.

Step 102-6: Assign role labels to the feature vectors in each speech segment according to the role sequence.

Since the current GMM and HMM are already stable, it is possible to assign a role label to the feature vector in each voice segment according to the role sequence obtained through decoding in step 102-4. In specific implementation, since each role in the role sequence has a one-to-one correspondence with each feature vector in each speech segment, a role label can be assigned to each feature vector according to the one-to-one correspondence. At this point, the feature vectors in each speech segment have their own role tags. After step 102 is executed, step 103 can be continued.

Step 102-7: Determine whether the number of times of training GMM and HMM under the current number of characters is less than the preset upper limit of the number of times of training; if yes, execute step 102-8, otherwise go to step 102-10 for execution.

Performing to this step indicates that the GMM and HMM obtained by the current training are not stable yet, and iterative computing training needs to be continued. Considering that the current number of characters used in the training process is inconsistent with the actual number of characters (the number of real characters involved in the voice signal), GMM and HMM may not be able to meet the requirements even after multiple iterations of training. The probability value corresponding to the acquired character sequence never satisfies the condition greater than the preset threshold). In order to avoid meaningless loop repetitive calculation process, the upper limit of training times for training GMM and HMM under each number of characters can be preset. If it is determined in this step that the number of training times under the current number of characters is less than the upper limit, proceed to step 102-8 to assign characters to each voice segment to continue the repeated calculation training, otherwise it means that the number of characters currently used may be inconsistent with the actual situation , So you can go to step 102-10 to determine whether you need to adjust the number of characters.

Step 102-8: Assign a corresponding role to each voice segment according to the role sequence.

In step 102-4, the character sequence has been obtained through decoding. Since each character in the character sequence has a one-to-one correspondence with the feature vector in each voice segment, it is possible to know the role corresponding to each feature vector in each voice segment . In this step, for each voice segment in the voice signal, a role is assigned to the voice segment by calculating the mode of the role corresponding to each feature vector. For example: a voice segment contains 10 audio frames, that is, contains 10 feature vectors, of which 8 feature vectors correspond to character 1 (s1), and 2 feature vectors correspond to character 2 (s2), then each of the voice segments The mode of the character corresponding to the feature vector is character 1 (s1), so character 1 (s1) is designated as the character of the speech segment.

Step 102-9: According to the feature vector in each voice segment and the corresponding role, train the GMM and HMM for each role, and go to step 102-4 to continue execution.

On the basis of specifying roles for each voice segment in step 102-8, GMM and HMM for each role can be trained. Still taking the voice segment division shown in Figure 4 as an example, if step 102-8 specifies seg1 and seg3 as role 1 (s1) and seg2 as role 2 (s2), then the feature vectors contained in seg1 and seg3 can be used Train the GMM (gmm1) of character 1, and the feature vector contained in seg2 is used to train the GMM (gmm2) of character 2. For the training methods of GMM and HMM, please refer to the relevant text of step 102-3, which will not be repeated here.

In specific implementation, this technical solution is usually an iterative computing training process. In order to improve training efficiency, this step can train new GMM and HMM incrementally on the basis of the GMM and HMM obtained in the previous training. On the basis of the parameters obtained in the training, the current sample data is used to continue to adjust the various parameters, which can increase the training speed.

After completing the above training process and obtaining the new GMM and HMM, you can go to step 102-4 for execution, and perform decoding according to the new model and perform subsequent operations.

Step 102-10: Determine whether the current number of characters meets the preset requirements; if yes, go to step 102-6 to execute, otherwise continue to execute step 102-11.

Performing this step usually means that the GMM and HMM trained under the current number of roles are not stable, and the number of training times has equaled or exceeded the preset upper limit of training times. In this case, it can be judged whether the current number of characters meets the expected number. If the requirements are met, it means that you can stop the role separation process and go to step 102-6 to assign role tags; otherwise, continue to step 102-11 to adjust the number of roles.

Step 102-11: Adjust the number of roles, select a corresponding number of voice segments and assign different roles for each voice segment; and go to step 102-3 to continue execution.

For example, if the current number of characters is 2, the preset requirement for the number of characters is "the number of characters is equal to 4", and step 102-10 determines that the current number of characters does not meet the preset requirement. In this case, you can perform this step to determine the number of characters. Adjust, for example: add 1 to the current number of characters, that is, update the current number of characters to 3.

According to the adjusted number of roles, a corresponding number of voice segments are selected from each voice segment included in the voice signal, and a different role is assigned to each selected voice segment. For the time length requirements of the selected speech segment, please refer to the relevant text in step 102-2, which will not be repeated here.

Still taking the voice segment division shown in Figure 4 as an example, if the current number of roles increases from 2 to 3, and seg1, seg2, and seg3 all meet the duration requirements for selecting the voice segment, then these 3 voice segments can be selected in this step, And specify role 1 (s1) for seg1, role 2 (s2) for seg2, and role 3 (s3) for seg3.

After completing the above operations of adjusting the number of characters and selecting voice segments, you can go to step 102-3 to initially train GMM and HMM for the adjusted number of characters.

Step 103: Train the DNN model by using the feature vector with the role label.

At this point, role labels have been assigned to the feature vectors in each voice segment. On this basis, this step uses the feature vectors with role labels as samples to train the DNN model. The DNN model is used to output the corresponding output according to the input feature vector. The probability of each character. In order to facilitate understanding, a brief description of DNN is given first.

DNN (Deep Neural Networks-Deep Neural Networks) usually refers to an input layer, 3 or more hidden layers (it can also include 7, 9, or even more hidden layers), and an output layer Neural network. Each hidden layer can extract certain features, and use the output of this layer as the input of the next layer. By extracting features layer by layer, the low-level features are formed into more abstract high-level features, which can realize the recognition of objects or types.

Please refer to Figure 5, which is a schematic diagram of the topological structure of the DNN network. The DNN network in the figure has a total of n layers, each layer has multiple neurons, and the different layers are fully connected; each layer has its own activation function f (Sigmoid function for example). The input is the feature vector v, the transition matrix from the i-th layer to the i+1-th layer is w _i(i+1) , the bias vector of the _{i+1-th layer is b (i+1)} , and the output of the i-th layer is out _i , the input of _i+1 is in i+1, the calculation process is: in _i+1 =out _i * wi _(i+1) + b _(i+1) out _i+1 = f(in _{i+ 1} )

It can be seen that the parameters of the DNN model include the transfer matrix w between layers and the bias vector b of each layer. The main task of training the DNN model is to determine the above parameters. In practical applications, BP (Back-propagation) algorithm is usually used for training. The training process is a supervised learning process: the input signal is a labeled feature vector, which is propagated forward hierarchically and reaches the output layer Then back propagation layer by layer, and adjust the parameters of each layer through the gradient descent method so that the actual output of the network is constantly close to the expected output. For a DNN network with thousands of neurons in each layer, the number of parameters may be millions or more. The DNN model obtained by completing the above training process usually has very powerful feature extraction capabilities and recognition capabilities.

In this embodiment, the DNN model is used to output the probability corresponding to each role according to the input feature vector. Therefore, the output layer of the DNN model can use a classifier (such as Softmax) as the startup function. In step 102, the pre-assigned role label is completed. After processing, if the number of roles involved in the role label is n, then the output layer of the DNN model can include n nodes, corresponding to n roles, and for each node of the input feature vector, the probability value of the feature vector corresponding to the role is output .

In this step, the feature vector with role label is used as a sample to perform supervised training on the constructed DNN model. In specific implementation, the above-mentioned BP algorithm can be directly used for training. Considering that purely using BP algorithm for training may fall into a local minimum, the resulting model cannot meet the needs of the application. Therefore, this embodiment adopts the prediction method. Training (pre-training) combined with BP algorithm for DNN model training.

Pre-training usually uses unsupervised greedy layer-by-layer training algorithm. First, unsupervised training of the network with one hidden layer is used, and then the trained parameters are retained, the number of network layers is increased by 1, and then the network with two hidden layers is trained Network...and so on, until the network with the largest hidden layer. After the layer-by-layer training is completed, the parameter values learned in the unsupervised training process are used as the initial values, and then the traditional BP algorithm is used for supervised training, and finally the DNN model is obtained.

Since the initial distribution obtained by pre-training is closer to the final convergence value than the random initial parameters used by the pure BP algorithm, it is equivalent to a good starting point for the subsequent supervised training process, so the DNN model obtained by training usually does not It will fall into a local minimum, and a higher recognition rate can be obtained.

Step 104: Determine the role sequence corresponding to the feature vector sequence according to the DNN model and the HMM obtained by the feature vector training, and output the role separation result.

Since the DNN model is used to output the probability corresponding to each role according to the input feature vector, at the same time, according to the distribution of the role label of the feature vector sequence, the prior probability corresponding to each role can be obtained, and the priori probability of each feature vector The probability is usually fixed. Therefore, according to Bayes' theorem, according to the output of the DNN model and the above-mentioned prior probability, the probability of each character outputting the corresponding feature vector can be known, that is, the DNN model trained in step 103 can be used to determine the states of the HMM Transmitter rate.

The HMM may be obtained by training with a feature vector sequence on the basis of using the above-mentioned DNN model to determine the HMM transmitter rate. Considering that the description of the jump relationship between the roles by the HMM used when assigning the feature vector to the feature vector in step 102 is basically stable, no additional training can be performed. Therefore, this embodiment directly uses the HMM and uses the training to obtain The DNN model replaces GMM, that is, the DNN model determines the transmitter rate of each state of the HMM.

In this embodiment, the voice segment is segmented in step 102-1. This step determines the character sequence corresponding to the feature vector sequence contained in each voice segment according to the DNN model and the HMM used when pre-assigning role tags. .

The process of determining the character sequence according to the feature vector sequence is a commonly described decoding problem. The decoding operation can be performed according to the DNN model and HMM to obtain the character with the highest probability value (for example, the highest probability value) outputting the feature vector sequence. Sequence, and use the character sequence as a character sequence corresponding to the feature vector sequence. For specific instructions, please refer to the relevant text in step 102-4, which will not be repeated here.

After the character sequence corresponding to the feature vector sequence contained in each speech segment is obtained through the decoding process, the corresponding character separation result can be output. Since each character in the character sequence has a one-to-one correspondence with the feature vector, and the audio frame corresponding to each feature vector has its own time start and end points, this step can output the audio information of the corresponding feature vector for each character The start and end time information of the frame.

So far, through step 101 to step 104, the specific implementation of the voice-based role separation method provided in the present application has been described in detail. It should be noted that in this embodiment, in the process of pre-assigning role labels to the feature vector in step 102, a top-down method is adopted to gradually increase the number of roles. In other implementations, a bottom-up method of gradually reducing the number of roles can also be used: initially each segmented voice segment can be assigned to a different role, and then the GMM and HMM for each role can be trained, if The probability value of the GMM and HMM obtained through iterative calculation training after performing the decoding operation is always not greater than the preset threshold. Then when adjusting the number of characters, you can evaluate the similarity between the GMMs of each character (for example, calculating KL Divergence), merge the speech segments corresponding to the GMM whose similarity meets the preset requirements, and reduce the number of characters accordingly, and repeat the repeated operation to perform the above process until the probability value obtained by the HMM through decoding is greater than the preset threshold or the number of characters meets the preset threshold. If required, stop the iterative calculation process, and assign role labels to the feature vectors in each voice segment according to the role sequence obtained by decoding.

In summary, the voice-based role separation method provided by this application uses a DNN model with powerful feature extraction capabilities to model roles, which has a more powerful characterization ability than traditional GMM, and the characterization of roles is more refined. , Accurate, so you can get more accurate role separation results. The technical solution of the present application can not only be applied to scenarios where the role separation of conversational voices such as call center and conference voices is performed, but also can be applied to other scenarios where roles in the voice signal need to be separated, as long as the voice signal contains For two or more roles, the technical solution of this application can be adopted and corresponding beneficial effects can be obtained.

In the foregoing embodiment, a voice-based role separation method is provided. Correspondingly, this application also provides a voice-based role separation device. Please refer to FIG. 6, which is a schematic diagram of an embodiment of a voice-based role separation apparatus according to this application. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the part of the description of the method embodiment. The device embodiments described below are merely illustrative.

A voice-based role separation device in this embodiment includes: a feature extraction unit 601, configured to extract feature vectors from a voice signal frame by frame to obtain a feature vector sequence; a label allocation unit 602, configured to assign role labels to feature vectors; The DNN model training unit 603 is used to train a DNN model using feature vectors with role labels, where the DNN model is used to output the probability corresponding to each role according to the input feature vectors; the role determination unit 604 is used to train DNN models based on the DNN The model and the HMM trained by the feature vector determine the character sequence corresponding to the feature vector sequence and output the character separation result, wherein the HMM is used to describe the jump relationship between the characters.

Optionally, the device further includes: a voice segment segmentation unit, configured to identify and eliminate audio frames that do not contain voice content after the feature extraction unit extracts the feature vector and before triggering the label distribution unit to work , Divide the voice signal into voice segments; the label assignment unit is specifically configured to assign role labels to the feature vectors in each voice segment; the role determination unit is specifically configured to, according to the DNN model and the utilization feature The HMM obtained by vector training determines the character sequence corresponding to the feature vector sequence contained in each speech segment and outputs the character separation result.

Optionally, the label allocating unit is specifically configured to pre-allocate role labels for the feature vectors in each voice segment by establishing GMM and HMM, wherein the GMM is used to output the feature vector for each role according to the input feature vector. The feature vector corresponds to the probability of the character; the role determination unit is specifically configured to determine the content contained in each voice segment according to the DNN model and the HMM used to assign role labels to the feature vectors in each voice segment The character sequence corresponding to the feature vector sequence.

Optionally, the label assignment unit includes: an initial role specifying subunit, which is used to select a corresponding number of voice segments according to a preset initial number of roles, and to specify different roles for each voice segment; an initial model training subunit, It is used to train the GMM and HMM for each role by using the feature vector in the voice segment of the specified role; the decoding subunit is used to decode the GMM and HMM obtained by training to obtain the feature vector sequence contained in each voice segment. The probability value of the character sequence is ranked higher; the probability judgment subunit is used to judge whether the probability value corresponding to the role sequence is greater than a preset threshold; the label allocation subunit is used when the output of the probability judgment subunit is yes At this time, according to the role sequence, a role label is assigned to the feature vector in each speech segment.

Optionally, the label assignment unit further includes: a voice segment role specification subunit, configured to specify a corresponding role for each voice segment according to the role sequence when the output of the probability judgment subunit is no ; The model update training subunit is used to train the GMM and HMM for each role according to the feature vector in each voice segment and the corresponding role, and trigger the decoding subunit to work.

Optionally, the voice segment-by-segment role specification subunit is specifically configured to specify, for each voice segment, the mode of the role corresponding to each feature vector as the role of the voice segment.

Optionally, the model update training subunit is specifically configured to train the GMM and HMM in an incremental manner on the basis of the model obtained in the previous training.

Optionally, the label allocation unit further includes: a training frequency judgment subunit, configured to judge whether the number of times of training GMM and HMM under the current number of characters is less than a preset number when the output of the probability judgment subunit is no The upper limit of the number of times of training, and when the judgment result is yes, trigger the work of the character designation subunits per voice segment.

The number of roles adjustment subunit is used to adjust the number of roles when the output of the number of training judgement subunits is no, select a corresponding number of voice segments and assign different roles for each voice segment, and trigger the initial model training Subunit work.

Optionally, the label assignment unit further includes: a character quantity judging subunit, which is used to judge whether the current character quantity meets a preset requirement when the output of the training times judging subunit is no, and if so, trigger the The label allocation subunit works, otherwise the role quantity adjustment subunit is triggered to work.

Optionally, the feature extraction unit includes: a framing subunit, configured to perform framing processing on the voice signal according to a preset frame length to obtain multiple audio frames; a feature extraction execution subunit, used to extract each audio frame To obtain the feature vector sequence.

Optionally, the feature extraction execution subunit is specifically configured to extract MFCC features, PLP features, or LPC features of each audio frame to obtain the feature vector sequence.

Optionally, the voice segment segmentation unit is specifically configured to recognize and eliminate the audio frames that do not contain voice content by using VAD technology, and segment the voice signal into voice segments.

Optionally, the device further includes: a VAD smoothing unit, configured to merge a voice segment with a duration less than a preset threshold with an adjacent voice segment after the voice segment segmentation unit uses the VAD technology to segment the voice segment.

Optionally, the DNN model training unit is specifically configured to train the DNN model by using a backpropagation algorithm.

Optionally, the role determination unit is specifically configured to perform a decoding operation according to the DNN model and the HMM, obtain the role sequence with the highest ranking of the probability values of outputting the feature vector sequence, and use the role sequence as the related The character sequence corresponding to the feature vector sequence.

Optionally, the character determination unit outputs the character separation result in the following manner: according to the character sequence corresponding to the feature vector sequence, for each character, output the start and end time information of the audio frame to which the corresponding feature vector belongs.

Optionally, the initial role designation subunit or the role number adjustment subunit specifically selects a corresponding number of voice segments in the following manner: selecting the number of voice segments whose duration meets a preset requirement.

Although this application is disclosed as above in preferred embodiments, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, this application The scope of protection shall be subject to the scope defined in the patent scope of this application.

In a typical configuration, the computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include non-permanent memory in computer-readable media, random access memory (RAM) and/or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

1. Computer-readable media includes permanent and non-permanent, removable and non-removable media, and information storage can be realized by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), and other types of random access memory (RAM) , Read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital multi-function Optical discs (DVD) or other optical storage, magnetic cassettes, magnetic tape storage or other magnetic storage devices, or any other non-transmission media, can be used to store information that can be accessed by computing devices. According to the definition in this article, computer-readable media does not include non-transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

2. Those skilled in the art should understand that the embodiments of this application can be provided as methods, systems or computer program products. Therefore, this application may adopt the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware. Moreover, this application can be in the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program codes. .

Claims (33)

A voice-based role separation method, which is characterized by: extracting feature vectors from the voice signal frame by frame to obtain a feature vector sequence; assigning role labels to the feature vectors; training a deep neural network DNN model using the feature vectors with role labels ; According to the DNN model and the hidden Markov model HMM obtained by using feature vector training, determine the character sequence corresponding to the feature vector sequence, and output the role separation result; wherein, the DNN model is used to output the corresponding character for each role according to the input feature vector Probability, HMM is used to describe the jump relationship between roles, wherein, according to the DNN model and the hidden Markov model HMM trained by using feature vectors, determining the character sequence corresponding to the feature vector sequence includes: according to the DNN model and the HMM Perform a decoding operation to obtain the character sequence with the greatest probability of outputting the feature vector sequence, and use the character sequence as the character sequence corresponding to the feature vector sequence. The voice-based role separation method according to the first item of the scope of patent application, wherein after the step of extracting feature vectors from the voice signal frame by frame and before the step of assigning role labels to the feature vectors, the following operations are performed : Divide the voice signal into voice segments by recognizing and removing audio frames that do not contain voice content; the assignment of role tags to feature vectors includes: The feature vector assigns role labels; the determining the role sequence corresponding to the feature vector sequence includes: determining the role sequence corresponding to the feature vector sequence contained in each speech segment. The voice-based role separation method according to item 2 of the scope of patent application, wherein the assigning role labels to the feature vectors in each voice segment includes: establishing Gaussian mixture models GMM and HMM to assign the features in each voice segment The vector assigns role labels; where the GMM is used to output the probability that the feature vector corresponds to the role for each role according to the input feature vector; the DNN model and the HMM trained with the feature vector are used to determine each speech segment The role sequence corresponding to the included feature vector sequence includes: determining the role sequence corresponding to the feature vector sequence contained in each voice segment according to the DNN model and the HMM used for assigning role labels to the feature vector in each voice segment. According to the voice-based role separation method described in item 3 of the scope of patent application, wherein the establishment of Gaussian mixture models GMM and HMM to assign role labels to feature vectors in each voice segment includes: according to a preset initial role Select the corresponding number of voice segments, and specify different roles for each voice segment; use the feature vectors in each voice segment corresponding to each specified role to train the GMM and HMM for each role; according to the training GMM and HMM decodes and obtains the character sequence with the highest probability value of the feature vector sequence contained in each speech segment; It is determined whether the probability value corresponding to the role sequence is greater than a preset threshold; if so, a role label is assigned to the feature vector in each voice segment according to the role sequence. The voice-based role separation method according to item 4 of the scope of patent application, wherein, when the result of judging whether the probability value corresponding to the role sequence is greater than the preset threshold is No, the following operation is performed: according to the role sequence , Specify the corresponding role for each voice segment; train GMM and HMM for each role according to the feature vector in each voice segment and the corresponding role; go to the step of decoding according to the GMM and HMM obtained by training . The voice-based role separation method according to item 5 of the scope of patent application, wherein, according to the role sequence, assigning a corresponding role to each voice segment includes: for each voice segment, corresponding each feature vector therein The mode of the character of is designated as the character of the speech segment. The voice-based role separation method according to item 5 of the scope of patent application, wherein the training of GMM and HMM for each role according to the feature vector in each voice segment and the corresponding role includes: Based on the trained model, the GMM and HMM are trained in an incremental manner. The voice-based role separation method according to item 5 of the scope of patent application, wherein, when the result of judging whether the probability value corresponding to the role sequence is greater than a preset threshold is no, the following operations are performed: Determine whether the number of times of training GMM and HMM under the current number of roles is less than the preset upper limit of the number of times of training; if yes, execute the step of assigning a corresponding role to each voice segment according to the role sequence; if not, execute the following operations: Adjust the number of roles, select a corresponding number of voice segments and specify different roles for each voice segment respectively; and turn to the step of using the feature vector in the voice segment of the specified role to train GMM and HMM for each role. The voice-based role separation method according to item 8 of the scope of patent application, characterized in that, when the result of judging whether the number of training GMM and HMM under the current number of roles is less than the preset upper limit of the number of training times is no, Perform the following operations: determine whether the current number of roles meets the preset requirements; if so, go to the step of assigning role labels to the feature vectors in each voice segment according to the role sequence; if not, execute the adjustment of the number of roles step. The voice-based role separation method according to item 8 of the scope of patent application is characterized in that the preset initial number of roles is 2, and the adjustment of the number of roles includes: adding 1 to the current number of roles. The voice-based role separation method according to item 1 of the scope of patent application, wherein the extracting feature vectors from the voice signal frame by frame to obtain the feature vector sequence includes: framing the voice signal according to a preset frame length , Get multiple audio frames; The feature vector of each audio frame is extracted to obtain the feature vector sequence. The voice-based role separation method according to item 11 of the scope of patent application, wherein said extracting the feature vector of each audio frame includes: extracting MFCC feature, PLP feature, or LPC feature. The voice-based role separation method according to item 2 of the scope of patent application, wherein the recognizing and removing audio frames that do not contain voice content includes: using VAD technology to identify the audio frames that do not contain voice content, and execute the corresponding Eliminate operation. According to the voice-based role separation method described in item 13 of the scope of patent application, the following VAD smoothing operation is performed after VAD technology is used to perform the recognition and removal operations, and the voice signal is divided into voice segments: The voice segment whose duration is less than the preset threshold is merged with the adjacent voice segment. The voice-based role separation method according to the first item of the scope of patent application, wherein the training of the deep neural network DNN model by using the feature vector with the role label includes: training the DNN model by using a backpropagation algorithm. The voice-based role separation method according to item 1 of the scope of patent application, wherein the output of the role separation result includes: outputting the audio frame of the corresponding feature vector for each role according to the role sequence corresponding to the feature vector sequence Start and end time information. The voice-based role separation method according to item 4 or 8 of the scope of patent application, wherein the selecting a corresponding number of voice segments includes: selecting the number of voice segments whose duration meets a preset requirement. A voice-based role separation device, which is characterized by comprising: a feature extraction unit for extracting feature vectors from the voice signal frame by frame to obtain a feature vector sequence; a label assignment unit for assigning role labels to feature vectors; DNN model The training unit is used to train the DNN model using the feature vector with the role label, where the DNN model is used to output the probability corresponding to each role according to the input feature vector; the role determination unit is used to train based on the DNN model and using the feature vector The obtained HMM determines the character sequence corresponding to the feature vector sequence and outputs the character separation result. The HMM is used to describe the jump relationship between the characters. The character determination unit is specifically used to perform a decoding operation according to the DNN model and the HMM, Obtain the character sequence with the largest probability value of outputting the feature vector sequence, and use the character sequence as the character sequence corresponding to the feature vector sequence. According to the 18th item of the scope of patent application, the voice-based role separation device further includes: a voice segment segmentation unit, which is configured to pass through after the feature extraction unit extracts the feature vector and before triggering the label assignment unit to work Recognize and eliminate audio frames that do not contain voice content, and divide the voice signal into voice segments; the label assignment unit is specifically used to assign role labels to the feature vectors in each voice segment; the role determination unit is specifically used to, according to The DNN model and utilization The HMM obtained by feature vector training determines the character sequence corresponding to the feature vector sequence contained in each speech segment and outputs the character separation result. The voice-based role separation device according to item 19 of the scope of patent application, wherein the label allocation unit is specifically configured to allocate role labels to the feature vectors in each voice segment by establishing GMM and HMM, wherein the GMM is used for For each role, output the probability that the feature vector corresponds to the role according to the input feature vector; the role determination unit is specifically used to assign role labels to the feature vectors in each voice segment according to the DNN model and the HMM used, Determine the character sequence corresponding to the feature vector sequence contained in each speech segment. The voice-based role separation device according to item 20 of the scope of patent application, wherein the label allocation unit includes: an initial role designation subunit for selecting a corresponding number of voice segments according to a preset initial number of roles, and for each Each voice segment specifies different roles; the initial model training subunit is used to train the GMM and HMM for each role using the feature vector in the voice segment of the specified role; the decoding subunit is used to train the GMM and HMM obtained according to the training Perform decoding to obtain the character sequence with the highest probability value of the feature vector sequence contained in each speech segment; the probability judgment subunit is used to judge whether the probability value corresponding to the character sequence is greater than the preset threshold; the label allocation subunit, Used to assign roles to the feature vectors in each voice segment according to the sequence of roles when the output of the probability judgment subunit is yes label. The voice-based role separation device according to item 21 of the scope of patent application, wherein the label assignment unit further includes: a voice segment role designation subunit for determining whether the output of the subunit is negative according to the probability The role sequence specifies the corresponding role for each voice segment; the model update training subunit is used to train the GMM and HMM for each role according to the feature vector in each voice segment and the corresponding role, and trigger the decoder Unit work. According to the voice-based role separation device described in item 22 of the scope of patent application, the voice segment-by-speech role specifying subunit is specifically used to specify the mode of the role corresponding to each feature vector for each voice segment as The role of the voice segment. According to the voice-based role separation device according to item 22 of the scope of patent application, the model update training subunit is specifically used to train the GMM and HMM in an incremental manner on the basis of the model obtained in the previous training. The voice-based role separation device according to item 22 of the scope of patent application, wherein the label assignment unit further includes: a training frequency judging subunit for judging whether the output of the probability judging subunit is negative. Whether the number of times of training GMM and HMM is less than the preset upper limit of the number of training times, and when the judgment result is yes, trigger the work of the role-designated subunit of each voice segment; unit When the output of is No, adjust the number of roles, select the corresponding number of voice segments and assign different roles for each voice segment, and trigger the initial model training subunit to work. The voice-based role separation device according to item 25 of the scope of patent application, wherein the label assignment unit further includes: a character quantity judging subunit for judging the current role when the output of the training times judging subunit is no Whether the quantity meets the preset requirements, if it does, trigger the label allocation subunit to work, otherwise trigger the role quantity adjustment subunit to work. The voice-based role separation device according to item 18 of the scope of patent application, wherein the feature extraction unit includes: a framing subunit for performing framing processing on the voice signal according to a preset frame length to obtain multiple audio signals Frame: The feature extraction execution subunit is used to extract the feature vector of each audio frame to obtain the feature vector sequence. The voice-based role separation device according to item 27 of the scope of patent application, wherein the feature extraction execution subunit is specifically configured to extract MFCC features, PLP features, or LPC features of each audio frame to obtain the feature vector sequence. The voice-based role separation device according to item 19 of the scope of patent application, wherein the voice segment segmentation unit is specifically used to recognize and eliminate the audio frame that does not contain voice content by using VAD technology, and cut the voice signal Divided into voice segments. According to the voice-based angle described in item 29 of the scope of patent application The color separation device further includes: a VAD smoothing unit, which is used to merge a voice segment with a duration less than a preset threshold with an adjacent voice segment after the voice segment segmentation unit adopts the VAD technology to segment the voice segment. According to the voice-based role separation device described in item 18 of the scope of patent application, the DNN model training unit is specifically used to train the DNN model by using a backpropagation algorithm. The voice-based role separation device according to item 18 of the scope of patent application, wherein the role determination unit outputs the role separation result in the following manner: according to the role sequence corresponding to the feature vector sequence, output the corresponding feature vector for each role The start and end time information of the audio frame to which it belongs. According to the voice-based role separation device described in item 21 or 25 of the scope of patent application, the initial role designation subunit or the role number adjustment subunit specifically selects the corresponding number of voice segments in the following manner: Set the required number of voice segments.

Images