CN112767957B - Method for obtaining prediction model, prediction method of voice waveform and related device - Google Patents
Method for obtaining prediction model, prediction method of voice waveform and related device Download PDFInfo
- Publication number
- CN112767957B CN112767957B CN202011627633.7A CN202011627633A CN112767957B CN 112767957 B CN112767957 B CN 112767957B CN 202011627633 A CN202011627633 A CN 202011627633A CN 112767957 B CN112767957 B CN 112767957B
- Authority
- CN
- China
- Prior art keywords
- waveform
- sample
- subsequences
- predicted
- point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000013528 artificial neural network Methods 0.000 claims description 18
- 238000009826 distribution Methods 0.000 claims description 15
- 238000013139 quantization Methods 0.000 claims description 7
- 125000004122 cyclic group Chemical group 0.000 claims description 5
- 230000006870 function Effects 0.000 claims description 5
- 238000005070 sampling Methods 0.000 claims description 2
- 230000001934 delay Effects 0.000 claims 1
- 238000004364 calculation method Methods 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 11
- 230000000306 recurrent effect Effects 0.000 description 5
- 241000288105 Grus Species 0.000 description 2
- 230000003111 delayed effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/04—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using predictive techniques
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/08—Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/27—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique
- G10L25/30—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
The application discloses a method for obtaining a prediction model, a prediction method of a voice waveform, electronic equipment and a computer readable storage medium. The waveform values of the current waveform points in the plurality of sample subsequences are simultaneously input into the prediction model, so that the predicted waveform values of the next waveform points in the plurality of sample subsequences can be simultaneously obtained. Therefore, the application can reduce the calculation amount of the predicted generated voice waveform and improve the efficiency of generating the voice waveform, thereby realizing the purpose of generating the voice waveform in real time and being not easy to generate the jamming when generating the voice waveform in real time.
Description
Technical Field
The present application relates to the field of speech signal processing technology, and in particular, to a method for obtaining a prediction model, a method for predicting a speech waveform, an electronic device, and a computer-readable storage medium.
Background
With the development of deep learning, a large number of neural networks are used for prediction of speech waveforms, such as WaveNet, sampleRNN, waveRNN, parallelWaveNet, waveGlow. However, when the neural network is used for predicting and generating the voice waveform, the calculated amount is large, so that the efficiency of generating the voice waveform is low, the voice waveform is difficult to generate in real time, the clamping phenomenon is easy to occur in practical application, and the user experience is reduced.
Disclosure of Invention
The application mainly solves the technical problem of providing a method for obtaining a prediction model, a method for predicting a voice waveform, electronic equipment and a computer readable storage medium, and can improve the efficiency of generating the voice waveform.
In order to solve the technical problems, the application adopts a technical scheme that:
there is provided a method of obtaining a predictive model comprising:
dividing a sample voice waveform into a plurality of sample subsequences, and performing time delay processing on the plurality of sample subsequences;
Constructing a prediction model; the method comprises the steps of obtaining a predicted waveform value of a next waveform point in a plurality of sample subsequences based on a prediction model by using sample acoustic parameters and waveform values of current waveform points in the plurality of sample subsequences; wherein the sample acoustic parameters are extracted from the sample speech waveform;
training the predictive model using the sample acoustic parameters and the number of sample subsequences after delay processing.
In order to solve the technical problems, the application adopts another technical scheme that:
Provided is a method for predicting a speech waveform, including:
acquiring text acoustic parameters based on text information, and setting a plurality of initialization waveform values;
Inputting the text acoustic parameters and the initialized waveform values into a prediction model, and obtaining a plurality of prediction subsequences by using the prediction model; wherein the predictor sequence corresponds to the initialization waveform value one by one;
and obtaining predicted voice waveforms corresponding to the text information according to the plurality of predicted subsequences.
In order to solve the technical problems, the application adopts another technical scheme that:
There is provided an electronic device comprising a memory and a processor coupled to each other, the memory storing program instructions, the processor being capable of executing the program instructions to implement a method of obtaining a predictive model as described in the above technical solution and/or a method of predicting a speech waveform as described in the above technical solution.
In order to solve the technical problems, the application adopts another technical scheme that:
There is provided a computer readable storage medium having stored thereon program instructions executable by a processor to implement a method of obtaining a predictive model as described in the above technical solutions and/or a method of predicting a speech waveform as described in the above technical solutions.
The beneficial effects of the application are as follows: compared with the prior art, the method for obtaining the prediction model provided by the application is characterized in that the sample voice waveform is firstly divided into a plurality of sample subsequences, the sample subsequences are subjected to delay processing, then an initial prediction model is constructed, and the initial prediction model is trained according to the plurality of sample subsequences and acoustic parameters of the sample voice waveform, so that the prediction model is obtained. The waveform values of the current waveform points in the plurality of sample subsequences are simultaneously input into the prediction model, so that the predicted waveform values of the next waveform points in the plurality of sample subsequences can be simultaneously obtained. Therefore, the application can reduce the calculation amount of the predicted generated voice waveform and improve the efficiency of generating the voice waveform, thereby realizing the purpose of generating the voice waveform in real time and being not easy to generate the jamming when generating the voice waveform in real time.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. Wherein:
FIG. 1 is a flow chart of an embodiment of a method for obtaining a predictive model according to the present application;
FIG. 2 is a flow chart of an embodiment of the step S11 in FIG. 1;
FIG. 3 is an exemplary diagram of one embodiment of a number of sample subsequences;
FIG. 4 is a diagram illustrating an embodiment of performing a time delay process on the sub-sequence of samples in FIG. 3;
FIG. 5 is a flowchart illustrating the step S13 of FIG. 1 according to an embodiment;
FIG. 6 is a flowchart of the step S12 of FIG. 1;
FIG. 7 is a schematic diagram of a prediction model according to an embodiment;
FIG. 8 is a flowchart illustrating an embodiment of a method for predicting a speech waveform according to the present application;
FIG. 9 is a flowchart illustrating the step S53 in FIG. 8;
FIG. 10 is a flowchart illustrating the step S52 of FIG. 8;
FIG. 11 is a flowchart illustrating the step S72 in FIG. 10;
FIG. 12 is a schematic diagram of an electronic device according to an embodiment of the present application;
fig. 13 is a schematic structural diagram of an embodiment of a computer readable storage medium according to the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, are intended to fall within the scope of the present application.
Referring to fig. 1, fig. 1 is a flowchart illustrating an embodiment of a method for obtaining a prediction model according to the present application, which includes the following steps.
Step S11, dividing the sample voice waveform into a plurality of sample subsequences, and performing time delay processing on the plurality of sample subsequences.
In this embodiment, a sample speech waveform is first prepared as training data of a prediction model, specifically, recording data of a speaker is first collected, and the recording data is preprocessed (e.g., denoising, energy management, etc.) to obtain the sample speech waveform. Wherein each waveform point in the sample speech waveform is characterized by an original amplitude value, and the original amplitude value generally varies within [ -1,1 ]. Therefore, it is preferable to perform a pre-emphasis process on the sample speech waveform before dividing the sample speech waveform into a number of sample sub-sequences, and to quantize the original amplitude value of each waveform point in the sample speech waveform into the waveform value of each waveform point. The quantization process can bring quantization noise to the sample voice waveform, pre-emphasis processing can compensate the quantization noise, and particularly u-law 8-bit quantization can be adopted during quantization, and each waveform point in the sample voice waveform is represented by 256 integer waveform values within [0, 255 ].
Referring to fig. 2, fig. 2 is a flow chart of an embodiment of step S11 in fig. 1, and the method specifically includes dividing a sample speech waveform into a plurality of sample subsequences, and performing time delay processing on the plurality of sample subsequences. The number of waveform points of the sample subsequence is a first value, and is denoted as B, that is, there are B sample subsequences, the number of waveform points included in the sample voice waveform is an integer multiple of the first value, and the multiple is a second value, and is denoted as N, that is, the sample voice waveform includes b×n waveform points, and each sample subsequence includes N waveform points.
Step S21, dividing the sample voice waveform into a plurality of sample subsequences which all comprise waveform points with second values, wherein the waveform points corresponding to the same serial number in the first to last sample subsequences are waveform points with first values which are continuously arranged in the sample voice waveform, and any two adjacent waveform points in the plurality of sample subsequences correspond to two waveform points with first values which are spaced apart from the original serial number in the sample voice waveform.
Referring specifically to fig. 3, fig. 3 is an exemplary diagram of an embodiment of a plurality of sample sub-sequences, where numerals 1,2, …,36 are original sequence numbers representing 36 (b=4, n=9) waveform points in a sample speech waveform, 9 waveform points in the 4 th row in the diagram form a first sample sub-sequence, respectively corresponding to the waveform points with the original sequence number 1,5,9,13,17,21,25,29,33 in the sample speech waveform, where any two adjacent waveform points correspond to two waveform points with the original sequence number interval of 4 in the sample speech waveform, and the second, third, and fourth sample sub-sequences corresponding to the 3, 2, and 1 rows are analogized sequentially. Finally, the first waveform point (the waveform point with the sequence number of 1) in each of the first to fourth sample subsequences is 4 waveform points with the original sequence numbers of 1,2,3 and 4 in the sample voice waveform, the second waveform point is 4 waveform points with the original sequence numbers of 5,6,7 and 8 in the sample voice waveform, and so on. Wherein the values of B and N can be reasonably set according to actual conditions. Of course, the number of waveform points in practical application is far greater than that shown in fig. 3.
In summary, the above-mentioned process of dividing the sample speech waveform into a plurality of sample subsequences is to place waveform points with the original sequence number b×n+b in the sample speech waveform in the corresponding sample subsequences, where B is B integers from 1 to B, and N is N integers from 0 to N-1. For example, b=1, then N waveform points in the sample speech waveform corresponding to N integer original sequence numbers between 0 and N-1 corresponding to b×n+1 are placed in the first sample subsequence.
Step S22, sequentially delaying the first to last sub-sequence samples by a preset multiple of a preset duration, wherein the preset multiple corresponding to the first to last sub-sequence samples is an integer between 0 and a difference between the first value and 1.
After the sample speech waveform is divided into a plurality of sample subsequences according to the manner described in step S21, waveform points corresponding to the same sequence number in the first to last subsequence samples are a plurality of waveform points continuously arranged in the sample speech waveform, and have strong correlation with each other, so that the method is not suitable for training data of a subsequent prediction model, and time delay processing is required for the plurality of subsequence samples.
Still taking the four sample subsequences shown in fig. 3 as an example, the delay procedure is described. Referring to fig. 4, fig. 4 is an exemplary diagram of an embodiment of performing time delay processing on the sample subsequence in fig. 3, where a first value b=4, and the first to fourth subsequence samples are sequentially delayed by 0 times, 1 times, 2 times, and 3 times of the preset time length d, that is, sequentially delayed by 0,2, 4, and 6 time lengths corresponding to the waveform points, as shown in fig. 4, where the position vacated after the time delay may be supplemented by 0.
After the time delay, taking the 7 th waveform point (25,18,11,4) and the 8 th small point (29,22,15,8) of each of the four sample subsequences in fig. 4 as an example, the 8 th waveform point is predicted by using the 7 th waveform point, when the 8 th waveform point (22,15,8) of each of the second, third and fourth sample subsequences is predicted, the previous and subsequent information can be seen, and taking the original serial number 22 as an example, the first 7 waveform points of each of the four sample subsequences comprise the original serial number larger than 22 and the original serial number smaller than 22, so that the predicted waveform value of the waveform point of the original serial number 22 can be predicted more accurately.
In addition, after the time delay, the waveform value of the waveform point corresponding to the same serial number in the plurality of sample subsequences may be expressed as (x t,xt-D,xt-2D,…,xt-(B-1)D), the waveform value of the waveform point corresponding to the next serial number may be expressed as (x t+B,xt-D+B,xt-2D+B,…,xt-(B-1)D+B), where t is the original serial number of the waveform point corresponding to the first sample subsequence in the sample voice waveform, and t- (B-1) x D is the original serial number of the waveform point corresponding to the last sample subsequence in the sample voice waveform. The waveform values at waveform points corresponding to serial number 7 in the four sample subsequences in fig. 4 can be expressed as (x 25,x18,x11,x7) where t=25, b=4, d=7.
S12, constructing a prediction model; the method comprises the steps of obtaining a predicted waveform value of a next waveform point in a plurality of sample subsequences based on a prediction model by utilizing sample acoustic parameters and waveform values of current waveform points in the plurality of sample subsequences; wherein the sample acoustic parameters are extracted from the sample speech waveform.
After the sample speech waveform is obtained, sample acoustic parameters such as mel spectrum, cepstrum fundamental frequency and the like can be extracted from the sample speech waveform, the sample acoustic parameters and waveform values of current waveform points in a plurality of sample subsequences are input into a prediction model, and a predicted waveform value of a next waveform point in the plurality of sample subsequences is obtained based on the prediction model. That is, the predicted waveform value of the next waveform point is predicted by using the waveform value of the current waveform point, and the predicted waveform value of each waveform point is obtained. The waveform value of the next waveform point in the plurality of sample subsequences is acquired simultaneously, i.e. the plurality of sample subsequences share the model parameters.
Step S13, training a prediction model by using the sample acoustic parameters and a plurality of sample subsequences after time delay processing.
Specifically, referring to fig. 5, fig. 5 is a flowchart illustrating an embodiment of step S13 in fig. 1, and the prediction model may be trained by the following steps.
Step S31, sequentially taking the first waveform point to the last waveform point in the plurality of sample subsequences as a current waveform point, and obtaining the predicted waveform value of the next waveform point in the plurality of sample subsequences based on a prediction model by utilizing the sample acoustic parameters and the waveform values of the current waveform point in the plurality of sample subsequences.
As described above, the prediction model may obtain the predicted waveform value of the next waveform point by point according to the waveform value of the current waveform point, and obtain the predicted waveform value of the next waveform point in the plurality of sample subsequences at the same time, so that the first waveform point to the last waveform point in the plurality of sample subsequences may be sequentially used as the current waveform point, and the predicted waveform value of the next waveform point may be obtained point by using the prediction model. For example, the 7 th waveform point corresponding to each of the plurality of sample subsequences after the time delay is input into a prediction model, the prediction waveform value of the 8 th waveform point is obtained by using the prediction model, the 8 th waveform point corresponding to each of the plurality of sample subsequences after the time delay is input into the prediction model, and the prediction waveform value of the 9 th waveform point is obtained by using the prediction model.
Step S32, updating the prediction model by using a loss function between the predicted waveform value and the actual waveform value of the corresponding waveform point to obtain a trained prediction model.
After the prediction model is constructed, training is needed to obtain optimal model parameters, so that the prediction model is convenient for the subsequent prediction of the voice waveform. After the predicted waveform value of the next waveform point in the plurality of sample subsequences is obtained, since the actual waveform value of the next waveform point in the plurality of sample subsequences is known, a loss function between the predicted waveform value and the actual waveform value of the corresponding waveform point may be calculated first, and model parameters of the prediction model may be updated based on a Back Propagation algorithm (Back Propagation) by using the cross entropy loss function, so as to obtain a trained prediction model.
In the embodiment, the waveform values of the current waveform points in the plurality of sample subsequences are simultaneously input into the prediction model, so that the predicted waveform values of the next waveform points in the plurality of sample subsequences can be simultaneously obtained. Compared with a prediction model obtained by directly training a sample voice waveform in the prior art, when the prediction model predicts the voice waveform in the embodiment, the calculated amount is only 1/B of the original calculated amount, wherein B is the number of sample subsequences. Therefore, the application can reduce the calculation amount of the predicted generated voice waveform and improve the efficiency of generating the voice waveform, thereby realizing the purpose of generating the voice waveform in real time and being not easy to generate the jamming when generating the voice waveform in real time.
In some embodiments, referring to fig. 6, fig. 6 is a flowchart illustrating an embodiment of step S12 in fig. 1, where the prediction model may be constructed by the following steps.
Step S41, sequentially connecting a plurality of convolution layers and a plurality of first full connection layers in series to form a condition network, sequentially connecting a plurality of circulating neural network layers in series, and respectively connecting a last circulating neural network layer and a plurality of second full connection layers in series to form a point-level network; wherein the second full-connection layer corresponds to the sample subsequences one-to-one.
Referring to fig. 7, fig. 7 is a schematic structural diagram of an embodiment of a prediction model, where the prediction model is formed by a condition network 710 and a point level network 720, so that a plurality of convolution layers 711 and a plurality of first full connection layers 712 (Fully Connected Layer, FC) are first required to be sequentially connected in series to form the condition network 710, a plurality of recurrent neural network layers 721 (Gated Recurrent Units, GRU) are sequentially connected in series, and a last recurrent neural network layer GRU and a plurality of second full connection layers 722 are respectively connected in series to form the point level network 720. Fig. 7 schematically illustrates a case where two layers of kernel size=3 of one-dimensional convolution (Conv 1 d) and two layers of first full-connection layers FC form a conditional network, and 2 layers of GRUs and 4 layers of second full-connection layers FC form a point-level network, where the second full-connection layers FC are in one-to-one correspondence with the sample subsequences, that is, the number of second full-connection layers FC is the first value B.
Step S42, connecting the condition network and the point level network in series to form a prediction model; the first convolution layer takes sample acoustic parameters as input, the first circulating neural network layer takes output of the last first full-connection layer and waveform values of current waveform points of a plurality of sample subsequences as input, the rest circulating neural network layers take output of the last first full-connection layer and output of the last circulating neural network layer as input, and a plurality of second full-connection layers are output layers of a prediction model.
With continued reference to fig. 7, after the condition network and the point level network are formed respectively, the two are connected in series to form a prediction model. The condition network takes the sample acoustic parameters (Acoustic Features, AF) as input, and outputs corresponding hidden layer variables f, specifically, the sample acoustic parameters are input into a first convolution layer, and the last first full-connection layer outputs the hidden layer variables f. The hidden layer variable f is input into each cyclic neural network layer GRU to enhance the control capability of acoustic parameters, meanwhile, the first GRU also inputs the waveform values of the current waveform points of a plurality of sample subsequences, such as (x t,xt-D,xt-2D,…,xt-(B-1)D), and other GRUs also input the output of the last GRU. Each second full connection layer takes the output of the last GRU as input and as output layer of the prediction model. The waveform value of the current waveform point in fig. 7 is characterized as (x t,xt-7,xt-14,xt-21), i.e., b=4, d=7.
In addition, each second full-connection layer outputs a probability distribution of predicted waveform values of the next waveform point in the corresponding sample subsequence, for example, after u-law 8 bits are quantized, all possible values of the predicted waveform values are 256 integers within [0, 255], then the second full-connection layer outputs a probability distribution formed by the probabilities of the 256 values, and the corresponding actual waveform value is known, then the probability is 100%, and the probabilities of other values are 0, so that the cross entropy loss function is calculated based on the probability distribution. That is, each value in the output (x t+4,xt-3,xt-10,xt-17) of the second fully-connected layer in fig. 7 represents a probability distribution.
The first layer of GRU adopts a larger hidden layer node number (for example 512), can memorize historical waveform point information, namely, forecast waveform value of the next waveform point, can utilize all the later input historical waveform point information, can improve the accuracy of forecast, and the second layer of GRU adopts a smaller hidden layer node number (for example 16) to abstract the hidden layer information of the first layer of GRU to a lower dimension. In other embodiments, a recurrent neural network layer GRU may also be employed.
Furthermore, the last first fully-connected layer is output with a first time domain resolution; the plurality of second full connection layers are output in a second time domain resolution, and the first time domain resolution is smaller than the second time domain resolution. That is, the point-level network performs a plurality of calculations within a period of time between outputting the present hidden layer information and the next hidden layer information by the conditional network, during which time the hidden layer information of the present time is inputted to the point-level network. That is, before the next output of the last first full-connection layer, the current output of the last first full-connection layer is input to the plurality of recurrent neural network layers.
In the embodiment, the convolutional layer, the full-connection layer and the cyclic neural network are utilized to construct a prediction model, and the waveform values of the current waveform points in the plurality of sample subsequences are simultaneously input into the prediction model, so that the predicted waveform values of the next waveform points in the plurality of sample subsequences can be simultaneously obtained. Therefore, the application can reduce the calculation amount of the predicted generated voice waveform and improve the efficiency of generating the voice waveform, thereby realizing the purpose of generating the voice waveform in real time and being not easy to generate the jamming when generating the voice waveform in real time.
Based on the same inventive concept, the present application also provides a method for predicting a speech waveform, referring to fig. 8, fig. 8 is a flowchart of an embodiment of the method for predicting a speech waveform according to the present application, where the method for predicting a speech waveform includes the following steps.
Step S51, acquiring text acoustic parameters based on the text information, and setting a plurality of initialization waveform values.
When it is required To convert Text information into Speech information, it is required To convert the Text information into a phoneme sequence by using a Text analysis model in a Text To Speech system (TTS), and then convert the phoneme sequence into Text acoustic parameters by using an acoustic model in the TTS, so as To input the Text acoustic parameters into a prediction model for obtaining Speech waveforms. Further, besides the acquired text acoustic parameters, a plurality of initialization waveform values are required to be set for subsequent prediction.
Step S52, inputting the text acoustic parameters and the initialized waveform values into a prediction model, and obtaining a plurality of prediction subsequences by using the prediction model; wherein the predicted subsequences are in one-to-one correspondence with the initialization waveform values.
In some embodiments, before inputting the text acoustic parameter and the initialized waveform value into the prediction model, the prediction model may be obtained by using the technical scheme described in any embodiment corresponding to the method for obtaining the prediction model, and then the method is applied to the method for predicting the speech waveform. The specific method for obtaining the prediction model may refer to any of the above embodiments, and will not be described herein.
Inputting the acquired text acoustic parameters and the initialized waveform values into a prediction model, and obtaining a plurality of prediction subsequences by utilizing autoregressive of the prediction model; wherein the predicted subsequences are in one-to-one correspondence with the initialization waveform values. That is, each initialization waveform value can correspondingly predict one prediction subsequence, and the prediction model can obtain a plurality of prediction subsequences at the same time.
Step S53, obtaining the predicted voice waveform corresponding to the text information according to the plurality of predicted subsequences.
After a plurality of prediction subsequences are obtained by prediction using the prediction model, the prediction subsequences need to be integrated into an entire predicted voice waveform corresponding to the text information so as to be subjected to subsequent processing. Referring to fig. 9, fig. 9 is a flowchart illustrating an embodiment of step S53 in fig. 8, where a predicted speech waveform corresponding to text information may be obtained by the following steps.
In step S61, all waveform points respectively included in the plurality of prediction subsequences are aligned in the time domain.
With continued reference to fig. 3 and fig. 4, according to the embodiment of the method for obtaining the prediction model, the plurality of prediction subsequences obtained by using the prediction model are time-delayed and are not aligned in the time domain, so that all waveform points respectively included in the plurality of prediction subsequences need to be aligned in the time domain, that is, the reverse process of the time-delayed processing is equivalent, that is, the process is changed from fig. 4 to fig. 3. After alignment, waveform points corresponding to the same serial number in the plurality of predicted subsequences are waveform points continuously arranged in the predicted voice waveform.
Step S62, sequentially taking the first waveform point to the last waveform point of the plurality of aligned predicted sub-sequences as the current waveform point, and sequentially arranging the predicted waveform values of the current waveform points in the plurality of predicted sub-sequences according to the serial numbers of the predicted sub-sequences to obtain a predicted voice waveform.
After all waveform points included in the plurality of predicted subsequences are aligned in the time domain, waveform points corresponding to the same sequence number in the plurality of predicted subsequences are waveform points continuously arranged in the predicted voice waveform, so that the first waveform point to the last waveform point of the plurality of aligned predicted subsequences can be sequentially used as current waveform points, and predicted waveform values of the current waveform points in the plurality of predicted subsequences are sequentially arranged according to the sequence numbers of the predicted subsequences to obtain the predicted voice waveform.
For example, the initial waveform value set in step S51 is (128,128,128,128), i.e. 4 predicted sub-sequences can be obtained, and the 1 st waveform point of each of the first to fourth predicted sub-sequences becomes the 1 st to 4 th waveform points in the predicted sub-sequences.
After obtaining the predicted sub-sequence, if the waveform value of each waveform point is represented by a quantized value and has been pre-emphasized before, the method further comprises the steps of:
and inversely quantizing the predicted waveform value of each waveform point in the predicted voice waveform into the amplitude value of each waveform point, and performing inverse pre-emphasis processing on the predicted voice waveform after the inverse quantization.
The present embodiment can simultaneously obtain a plurality of prediction subsequences using a prediction model, and integrate the plurality of prediction subsequences into a predicted speech waveform corresponding to text information. Compared with a prediction model obtained by directly training a sample voice waveform in the prior art, when the prediction model predicts the voice waveform in the embodiment, the calculated amount is only 1/B of the original calculated amount, wherein B is the number of the prediction subsequences. Therefore, the application can reduce the calculated amount of predicting and generating the predicted voice waveform and improve the efficiency of generating the predicted voice waveform, thereby realizing the purpose of generating the voice waveform in real time and being not easy to generate the jamming when generating the voice waveform in real time.
In some embodiments, referring to fig. 10, fig. 10 is a flowchart illustrating an embodiment of step S52 in fig. 8, where a plurality of prediction subsequences may be obtained by using a prediction model as follows.
In step S71, the plurality of initialization waveform values are respectively used as the waveform values of the first waveform point in the plurality of prediction subsequences.
As described above, the initialized waveform values are in one-to-one correspondence with the predicted sub-sequences, so that before prediction, the initialized waveform values are used as the waveform values of the first waveform point in the predicted sub-sequences, respectively, so as to start the prediction process in combination with the text acoustic parameters.
Step S72, repeatedly executing the steps of inputting the waveform value of the previous waveform point in the plurality of prediction subsequences and the text acoustic parameters into the prediction model, and obtaining the predicted waveform value of the current waveform point in the plurality of prediction subsequences by using the prediction model so as to obtain the predicted waveform value of each waveform point in the plurality of prediction subsequences.
The prediction model acquires the predicted waveform values point by point, and the prediction model can acquire a plurality of predicted waveform values at the same time, so that the steps of inputting the waveform value of the previous waveform point in the plurality of predicted subsequences and the text acoustic parameters into the prediction model, and acquiring the predicted waveform value of the current waveform point in the plurality of predicted subsequences by using the prediction model are required to be repeatedly executed, so that the predicted waveform value of each waveform point in the plurality of predicted subsequences is acquired by autoregressive, and the plurality of predicted subsequences are acquired. Wherein the initializing waveform value is used when the predicted waveform value of the second waveform point is obtained, and the predicted waveform value of the second waveform point is used when the predicted waveform value of the third waveform point is obtained
Specifically, referring to fig. 11, fig. 11 is a flowchart illustrating an embodiment of step S72 in fig. 10, a predicted waveform value of a current waveform point in a plurality of predicted sub-sequences may be obtained by the following steps.
Step S81, inputting the waveform value of the previous waveform point in the plurality of prediction subsequences and the text acoustic parameters into the prediction model, and obtaining a plurality of probability distributions of the current waveform point output by the prediction model, wherein the plurality of probability distributions comprise probabilities corresponding to all the prediction waveform values.
In this embodiment, the probability distribution of all predicted waveform values is output by the prediction model, so after the waveform values of the previous waveform points in the plurality of prediction subsequences and the text acoustic parameters are input into the prediction model, the probability distribution of all possible predicted waveform values of the current waveform point is obtained, for example, after u-law 8 bits are quantized, the possible values of all predicted waveform values are 256 integers within [0, 255], and then the probability distribution is the probability of 256 values, and the sum of all the probabilities is 1.
Step S82, randomly sampling a plurality of probability distributions to obtain respective corresponding predicted waveform values.
After the probability distribution of each of a plurality of current waveform points corresponding to the initialized waveform values one by one is obtained, each probability distribution is randomly sampled to obtain a corresponding predicted waveform value. That is, the predicted waveform value corresponding to one probability value is randomly selected from 256 probability values, and finally the predicted waveform values the same as the number of the initialized waveform values are obtained at the same time.
According to the method and the device for generating the predicted voice waveform, the plurality of predicted subsequences can be obtained simultaneously by using the prediction model, so that the calculated amount of predicting the generated predicted voice waveform can be reduced, the efficiency of generating the predicted voice waveform is improved, the purpose of generating the voice waveform in real time can be achieved, and the voice waveform is not easy to be blocked when being generated in real time.
In addition, referring to fig. 12, fig. 12 is a schematic structural diagram of an embodiment of an electronic device according to the present application, where the electronic device includes a memory 1210 and a processor 1220 that are coupled to each other, and the memory 1210 stores program instructions, and the processor 1220 can execute the program instructions to implement a method for obtaining a prediction model according to the above embodiment and/or a method for predicting a speech waveform according to the above embodiment. The specific reference may be made to any of the above embodiments, and further details are not described.
In addition, the present application further provides a computer readable storage medium, please refer to fig. 13, fig. 13 is a schematic structural diagram of an embodiment of the computer readable storage medium of the present application, and the storage medium 1300 has stored thereon program instructions 1310, where the program instructions 1310 can be executed by a processor to implement a method for obtaining a prediction model according to the above embodiment, and/or a method for predicting a speech waveform according to the above embodiment. The specific reference may be made to any of the above embodiments, and further details are not described.
The foregoing description is only of embodiments of the present application, and is not intended to limit the scope of the application, and all equivalent structures or equivalent processes using the descriptions and the drawings of the present application or directly or indirectly applied to other related technical fields are included in the scope of the present application.
Claims (13)
1. A method of obtaining a predictive model, comprising:
dividing a sample voice waveform into a plurality of sample subsequences, and performing time delay processing on the plurality of sample subsequences;
Constructing a prediction model; the prediction model takes a sample acoustic parameter and waveform values of current waveform points of the plurality of sample subsequences as inputs, and takes predicted waveform values of next waveform points of the plurality of sample subsequences as outputs; wherein the sample acoustic parameters are extracted from the sample speech waveform;
training the predictive model using the sample acoustic parameters and the number of sample subsequences after delay processing.
2. The method of claim 1, wherein the step of constructing a predictive model comprises:
sequentially connecting a plurality of convolution layers and a plurality of first full connection layers in series to form a conditional network, sequentially connecting a plurality of circulating neural network layers in series, and respectively connecting the last circulating neural network layer and a plurality of second full connection layers in series to form a point-level network; wherein the second full-connection layer corresponds to the sample subsequences one-to-one;
Connecting the condition network and the point level network in series to form the prediction model; the first convolutional layer takes the sample acoustic parameters as input, the first cyclic neural network layer takes the output of the last first full-connection layer and waveform values of current waveform points of the plurality of sample subsequences as input, the rest of the cyclic neural network layers take the output of the last first full-connection layer and the output of the last cyclic neural network layer as input, and the plurality of second full-connection layers are output layers of the prediction model.
3. The method of claim 2, wherein a last one of the first fully-connected layers is output at a first time-domain resolution, the number of second fully-connected layers is output at a second time-domain resolution, and the first time-domain resolution is less than the second time-domain resolution; before the next output of the last first full-connection layer, the last first full-connection layer outputs the current output of the last first full-connection layer.
4. The method of claim 1, wherein the number of waveform points of the sub-sequence of samples is a first number, the sample speech waveform includes a number of waveform points that is an integer multiple of the first number and a multiple is a second number, the steps of dividing the sample speech waveform into a number of sub-sequences of samples and performing a time delay process on the number of sub-sequences of samples comprise:
Dividing the sample voice waveform into a plurality of sample subsequences which all comprise waveform points of the second numerical value, wherein the waveform points corresponding to the same serial number in the first to last sample subsequences are waveform points of a first numerical value continuously arranged in the sample voice waveform, and any two adjacent waveform points in the plurality of sample subsequences correspond to two waveform points with original serial numbers spaced by the first numerical value in the sample voice waveform;
And sequentially delaying the first to last subsequence samples by a preset multiple of a preset duration, wherein the preset multiple respectively corresponding to the first to last subsequence samples is an integer between 0 and the difference between the first value and 1.
5. The method of claim 1, wherein the step of training the predictive model using the sample acoustic parameters and the number of sample subsequences after delay processing comprises:
Sequentially taking the first waveform point to the last waveform point in the plurality of sample subsequences as a current waveform point, and obtaining a predicted waveform value of a next waveform point in the plurality of sample subsequences based on the prediction model by utilizing the sample acoustic parameters and the waveform values of the current waveform point in the plurality of sample subsequences;
And updating the prediction model by using a loss function between the predicted waveform value and the actual waveform value of the corresponding waveform point so as to obtain the trained prediction model.
6. The method of claim 1, wherein prior to the step of dividing the sample speech waveform into a number of sample subsequences, further comprising:
and pre-emphasis processing is carried out on the sample voice waveform, and the original amplitude value of each waveform point in the sample voice waveform is quantized into the waveform value of each waveform point.
7. A method for predicting a speech waveform, comprising:
acquiring text acoustic parameters based on text information, and setting a plurality of initialization waveform values;
Inputting the text acoustic parameters and the initialized waveform values into a prediction model, and obtaining a plurality of prediction subsequences by using the prediction model; wherein the predictor sequence corresponds to the initialization waveform value one by one; the prediction model is obtained by training a plurality of sample subsequences after processing by using sample acoustic parameters and time delays, the plurality of sample subsequences are obtained by dividing sample speech waveforms, the prediction model takes waveform values of current waveform points of the sample acoustic parameters and the plurality of sample subsequences as input, and predicted waveform values of next waveform points of the plurality of sample subsequences as output; wherein the sample acoustic parameters are extracted from the sample speech waveform;
and obtaining predicted voice waveforms corresponding to the text information according to the plurality of predicted subsequences.
8. The method of claim 7, wherein the step of inputting the text acoustic parameters and the initialization waveform values into a predictive model and obtaining a number of predicted subsequences using the predictive model comprises:
Respectively taking the plurality of initialization waveform values as waveform values of a first waveform point in the plurality of prediction subsequences;
And repeatedly executing the steps of inputting the waveform value of the previous waveform point in the plurality of prediction subsequences and the text acoustic parameter into the prediction model, and obtaining the predicted waveform value of the current waveform point in the plurality of prediction subsequences by utilizing the prediction model so as to obtain the predicted waveform value of each waveform point in the plurality of prediction subsequences.
9. The method of claim 8, wherein the step of inputting the waveform value of the previous waveform point in the plurality of predicted sub-sequences and the text acoustic parameter into the prediction model and obtaining the predicted waveform value of the current waveform point in the plurality of predicted sub-sequences using the prediction model comprises:
inputting the waveform value of the previous waveform point in the plurality of prediction subsequences and the text acoustic parameters into the prediction model, and obtaining a plurality of probability distributions of the current waveform point output by the prediction model, wherein the plurality of probability distributions comprise probabilities corresponding to all the predicted waveform values;
And randomly sampling the probability distributions to obtain a plurality of corresponding predicted waveform values.
10. The method of claim 8, wherein the step of obtaining predicted speech waveforms corresponding to the text information from the plurality of predicted sub-sequences comprises:
aligning all waveform points respectively included by the plurality of prediction subsequences in a time domain;
Sequentially taking the first waveform point to the last waveform point of the plurality of the predicted subsequences after alignment as a current waveform point, and sequentially arranging predicted waveform values of the current waveform points in the plurality of the predicted subsequences according to sequence numbers of the predicted subsequences to obtain the predicted voice waveform.
11. The method of claim 8, further comprising, after the step of obtaining predicted speech waveforms corresponding to the text information from the plurality of predicted subsequences:
And inversely quantizing the predicted waveform value of each waveform point in the predicted voice waveform into the amplitude value of each waveform point, and performing inverse pre-emphasis processing on the predicted voice waveform after inverse quantization.
12. An electronic device comprising a memory and a processor coupled to each other, the memory storing program instructions that are executable by the processor to implement the method of obtaining a predictive model according to any one of claims 1-6 and/or the method of predicting a speech waveform according to any one of claims 7-11.
13. A computer readable storage medium having stored thereon program instructions executable by a processor to implement the method of obtaining a predictive model according to any of claims 1-6 and/or the method of predicting a speech waveform according to any of claims 7-11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011627633.7A CN112767957B (en) | 2020-12-31 | 2020-12-31 | Method for obtaining prediction model, prediction method of voice waveform and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011627633.7A CN112767957B (en) | 2020-12-31 | 2020-12-31 | Method for obtaining prediction model, prediction method of voice waveform and related device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112767957A CN112767957A (en) | 2021-05-07 |
| CN112767957B true CN112767957B (en) | 2024-05-31 |
Family
ID=75699304
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011627633.7A Active CN112767957B (en) | 2020-12-31 | 2020-12-31 | Method for obtaining prediction model, prediction method of voice waveform and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112767957B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004233774A (en) * | 2003-01-31 | 2004-08-19 | Nippon Telegr & Teleph Corp <Ntt> | Speech synthesizing method, speech synthesizing device and speech synthesizing program |
| JP2007318691A (en) * | 2006-05-29 | 2007-12-06 | Nippon Telegr & Teleph Corp <Ntt> | Apparatus and method for determining linear prediction model degree, program thereof and recording medium |
| CN104112444A (en) * | 2014-07-28 | 2014-10-22 | 中国科学院自动化研究所 | Text message based waveform concatenation speech synthesis method |
| CN106782577A (en) * | 2016-11-11 | 2017-05-31 | 陕西师范大学 | A kind of voice signal coding and decoding methods based on Chaotic time series forecasting model |
| CN107945786A (en) * | 2017-11-27 | 2018-04-20 | 北京百度网讯科技有限公司 | Phoneme synthesizing method and device |
| CN108573694A (en) * | 2018-02-01 | 2018-09-25 | 北京百度网讯科技有限公司 | Language material expansion and speech synthesis system construction method based on artificial intelligence and device |
| CN108806665A (en) * | 2018-09-12 | 2018-11-13 | 百度在线网络技术（北京）有限公司 | Phoneme synthesizing method and device |
| CN109036371A (en) * | 2018-07-19 | 2018-12-18 | 北京光年无限科技有限公司 | Audio data generation method and system for speech synthesis |
| CN109767755A (en) * | 2019-03-01 | 2019-05-17 | 广州多益网络股份有限公司 | A kind of phoneme synthesizing method and system |
| CN110033755A (en) * | 2019-04-23 | 2019-07-19 | 平安科技（深圳）有限公司 | Phoneme synthesizing method, device, computer equipment and storage medium |
| CN110634476A (en) * | 2019-10-09 | 2019-12-31 | 深圳大学 | Method and system for rapidly building robust acoustic model |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11462209B2 (en) * | 2018-05-18 | 2022-10-04 | Baidu Usa Llc | Spectrogram to waveform synthesis using convolutional networks |
-
2020
- 2020-12-31 CN CN202011627633.7A patent/CN112767957B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004233774A (en) * | 2003-01-31 | 2004-08-19 | Nippon Telegr & Teleph Corp <Ntt> | Speech synthesizing method, speech synthesizing device and speech synthesizing program |
| JP2007318691A (en) * | 2006-05-29 | 2007-12-06 | Nippon Telegr & Teleph Corp <Ntt> | Apparatus and method for determining linear prediction model degree, program thereof and recording medium |
| CN104112444A (en) * | 2014-07-28 | 2014-10-22 | 中国科学院自动化研究所 | Text message based waveform concatenation speech synthesis method |
| CN106782577A (en) * | 2016-11-11 | 2017-05-31 | 陕西师范大学 | A kind of voice signal coding and decoding methods based on Chaotic time series forecasting model |
| CN107945786A (en) * | 2017-11-27 | 2018-04-20 | 北京百度网讯科技有限公司 | Phoneme synthesizing method and device |
| CN108573694A (en) * | 2018-02-01 | 2018-09-25 | 北京百度网讯科技有限公司 | Language material expansion and speech synthesis system construction method based on artificial intelligence and device |
| CN109036371A (en) * | 2018-07-19 | 2018-12-18 | 北京光年无限科技有限公司 | Audio data generation method and system for speech synthesis |
| CN108806665A (en) * | 2018-09-12 | 2018-11-13 | 百度在线网络技术（北京）有限公司 | Phoneme synthesizing method and device |
| CN109767755A (en) * | 2019-03-01 | 2019-05-17 | 广州多益网络股份有限公司 | A kind of phoneme synthesizing method and system |
| CN110033755A (en) * | 2019-04-23 | 2019-07-19 | 平安科技（深圳）有限公司 | Phoneme synthesizing method, device, computer equipment and storage medium |
| CN110634476A (en) * | 2019-10-09 | 2019-12-31 | 深圳大学 | Method and system for rapidly building robust acoustic model |
Non-Patent Citations (2)
| Title |
|---|
| Low Bit-Rate Voice Compression Based on Frequency Domain Interpolative Techniques;Udaya Bhaskar等;《IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING》;20060331;第14卷(第2期);全文 * |
| 基于线性预测下的语音信号合成;贺艳平;《西北民族大学学报（自然科学版）》;第31卷(第4期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112767957A (en) | 2021-05-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110491416B (en) | Telephone voice emotion analysis and identification method based on LSTM and SAE | |
| CN110136731B (en) | Cavity causal convolution generation confrontation network end-to-end bone conduction voice blind enhancement method | |
| KR102353284B1 (en) | Generate audio using neural networks | |
| JP2016212273A (en) | Recurrent type neural network learning method, computer program for the same, and voice recognition device | |
| CN110767210A (en) | Method and device for generating personalized voice | |
| CN112767959B (en) | Voice enhancement method, device, equipment and medium | |
| CN113450761B (en) | Parallel voice synthesis method and device based on variation self-encoder | |
| CN112434891A (en) | Method for predicting solar irradiance time sequence based on WCNN-ALSTM | |
| KR100693284B1 (en) | Apparatus for speech recognition | |
| CN113782044B (en) | Voice enhancement method and device | |
| CN112767957B (en) | Method for obtaining prediction model, prediction method of voice waveform and related device | |
| CN113129863B (en) | Voice duration prediction method, device, equipment and readable storage medium | |
| JP2019159464A (en) | Device, method and program utilizing language model | |
| CN111653261A (en) | Speech synthesis method, speech synthesis device, readable storage medium and electronic equipment | |
| JP7103390B2 (en) | Acoustic signal generation method, acoustic signal generator and program | |
| CN112863539B (en) | High-sampling-rate voice waveform generation method, device, equipment and storage medium | |
| WO2021062705A1 (en) | Single-sound channel robustness speech keyword real-time detection method | |
| Yingle et al. | Speaker gender identification based on combining linear and nonlinear features | |
| de Oliveira et al. | Artificial neural networks for dictionary selection in adaptive greedy decomposition algorithms with reduced complexity | |
| JP2734828B2 (en) | Probability calculation device and probability calculation method | |
| Otsuka et al. | An on-line algorithm of guitar performance transcription using non-negative matrix factorization | |
| WO2022168162A1 (en) | Prior learning method, prior learning device, and prior learning program | |
| Prozorov et al. | Markov Model of Quantized Speech Signal | |
| WO2020032177A1 (en) | Method and device for generating frequency component vector of time-series data | |
| JP3898676B2 (en) | Voice recognition device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230523 Address after: 230026 No. 96, Jinzhai Road, Hefei, Anhui Applicant after: University of Science and Technology of China Applicant after: IFLYTEK Co.,Ltd. Address before: 230088 666 Wangjiang West Road, Hefei hi tech Development Zone, Anhui Applicant before: IFLYTEK Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |