KR102198598B1 - Method for generating synthesized speech signal, neural vocoder, and training method thereof - Google Patents

Abstract

Obtaining a plurality of acoustic parameters including spectrum-related parameters and excitation-related parameters, estimating an excitation signal based on the plurality of acoustic parameters, and at least one of spectrum-related parameters for the estimated excitation signal A method of generating a speech signal using a neural vocoder that generates a target speech signal by applying a linear synthesis filter based on is provided.

Description

Synthetic voice signal generation method, neural vocoder and training method of neural vocoder {METHOD FOR GENERATING SYNTHESIZED SPEECH SIGNAL, NEURAL VOCODER, AND TRAINING METHOD THEREOF}

The following description relates to a method of generating a synthesized speech signal using a neural vocoder, and a training method of a neural vocoder and a neural vocoder.

Speech synthesis technology is a technology that creates a synthesized sound similar to a human voice based on input data. For example, text to speech (TTS) is provided by converting input text into human speech.

This synthesized speech is generated by a vocoder that generates a speech signal based on the input acoustic parameter. In recent years, with the development of artificial intelligence and deep learning technologies, neural vocoders using artificial neural networks have been proposed in generating synthetic speech. The neural vocoder can be trained speaker-independently or speaker-dependently using speech data from speakers, and can generate a synthesized speech signal for input acoustic parameters using the result of the training.

However, since the speech signal has a dynamic characteristic, it is difficult for an artificial neural network (eg, CNN) to fully capture the dynamic characteristic of the speech signal. In particular, spectral distortion is likely to occur in a high frequency region of a speech signal, which may lead to a deterioration in quality of a synthesized speech signal.

Accordingly, there is a need for a neural vocoder system that can reduce spectral distortion in a high frequency region and improve the quality of a synthesized speech signal, while also simplifying the process of training speech data.

Korean Patent Laid-Open Publication No. 10-2018-0113325 (published on October 16, 2018) describes the voice synthesizer's voice so that the voice synthesizer can modulate the voice of the synthesized tone according to the intention of the developer or user in synthesizing the voice waveform. A speech synthesis apparatus and method that provides a function of encoding a model, converting a speech model code, and decoding a speech model to synthesize a modulated speech waveform are described.

The information described above is for illustrative purposes only, may include content that does not form part of the prior art, and may not include what the prior art may present to a person skilled in the art.

Obtaining a plurality of acoustic parameters including spectrum-related parameters and excitation-related parameters, estimating an excitation signal based on the plurality of acoustic parameters, and linear synthesis based on at least one of spectrum-related parameters for the estimated excitation signal A method of generating a speech signal using a neural vocoder that generates a target speech signal by applying a filter can be provided.

In one aspect, in a method for generating a speech signal performed by a neural vocoder implemented by a computer, spectral parameters and excitation related parameters classified according to periodicity of excitation are Obtaining a plurality of acoustic parameters including, estimating an excitation signal based on the plurality of acoustic parameters, and linear synthesis based on at least one of the spectrum related parameters for the estimated excitation signal A method of generating a speech signal is provided comprising the step of generating a target speech signal by applying a filter.

The excitation related parameters may include a first excitation parameter indicating excitation below a predetermined cutoff frequency and a second excitation parameter indicating excitation exceeding the cutoff frequency.

The first excitation parameter may indicate a harmonic spectrum of the excitation, and the second excitation parameter may indicate other parts of the excitation.

The spectrum-related parameters include a frequency parameter indicating a pitch of a voice signal, an energy parameter indicating an energy of a voice signal, a parameter indicating whether the voice signal is voiced or unvoiced, and a line spectral frequency of the voice signal. Frequency; may include a parameter indicating LSF).

The generating of the target speech signal includes converting a parameter representing the LSF to Linear Predictive Coding (LPC), and applying the linear synthesis filter based on the converted LPC to the estimated excitation signal. It may include the step of.

The plurality of acoustic parameters may be generated by an acoustic model based on an input text or an input speech signal.

The neural vocoder is trained based on the voice signal input for training, and the training is an excitation signal from the input voice signal by applying a linear prediction analysis filter to the input voice signal. Separating and modeling a probability distribution of the separated excitation signal, and the step of estimating the excitation signal includes, using the modeled probability distribution of the excitation signal, for the plurality of acoustic parameters We can estimate the excitation signal.

The step of separating the excitation signal may include converting a parameter representing the LSF of the input speech signal into Linear Predictive Coding (LPC), and converting the input speech signal to the input speech signal. It may include applying the linear prediction analysis filter based on LPC.

The separated excitation signal may be a residual component of the input speech signal.

In another aspect, in the training method of a neural vocoder implemented by a computer, receiving a speech signal, including spectrum-related parameters from the input speech signal and excitation-related parameters classified according to periodicity of excitation Extracting a plurality of acoustic parameters, separating an excitation signal from the input speech signal by applying a linear prediction analysis filter based on at least one of the spectrum related parameters to the input speech signal, and the separated excitation A method of training a neural vocoder, comprising modeling a probability distribution of a signal is provided.

The step of separating the excitation signal includes converting a parameter representing the LSF of the input voice signal among the spectrum-related parameters to LPC, and the converted LPC of the input voice signal with respect to the input voice signal. It may include the step of applying the linear prediction analysis filter.

The excitation related parameters may include a first excitation parameter indicating excitation below a predetermined cutoff frequency and a second excitation parameter indicating excitation exceeding the cutoff frequency.

In another aspect, in a neural vocoder, a parameter acquisition unit for acquiring a plurality of acoustic parameters including spectral parameters and excitation related parameters classified according to periodicity of excitation, the plurality of An excitation signal estimator that estimates an excitation signal based on the acoustic parameters of and generates a target speech signal by applying a linear synthesis filter based on at least one of the spectrum related parameters to the estimated excitation signal. There is provided a neural vocoder including a voice signal generator.

The speech signal generation unit includes a conversion unit for converting a parameter representing the LSF of the speech signal among the spectrum-related parameters into Linear Predictive Coding (LPC), and based on the converted LPC for the estimated excitation signal. The above linear synthesis filter can be applied

The neural vocoder is trained based on the input speech signal for training, and the neural vocoder excites from the input speech signal by applying a linear prediction analysis filter to the input speech signal. An excitation signal separation unit for separating a signal and a modeling unit for modeling a probability distribution of the separated excitation signal, the excitation signal estimating unit, using a probability distribution of the modeled excitation signal, the plurality of sounds It is possible to estimate the excitation signal for the parameters.

The excitation signal separation unit includes a conversion unit for converting a parameter representing the LSF of the input speech signal into Linear Predictive Coding (LPC), and a converted LPC of the input speech signal with respect to the input speech signal. The linear prediction analysis filter based on may be applied.

The neural vocoder performs estimation on the excitation signal as a target, and a target speech signal is generated by applying a linear prediction filter to the estimated excitation signal, thereby improving the quality of the generated target speech signal. Spectral distortion in the high frequency region of the signal can be reduced.

1 shows a method of generating a synthesized speech signal based on an input text or speech signal according to an embodiment.
2 is a block diagram showing the structure of a neural vocoder system according to an embodiment.
3 is a block diagram showing the structure of a processor of a neural vocoder system according to an embodiment.
4 is a flowchart illustrating a method of generating a voice signal according to an exemplary embodiment.
5 is a flowchart illustrating a method of training a neural vocoder according to an embodiment.
6 illustrates a method for generating a synthesized speech of a target speaker by constructing a speaker adaptive model according to an embodiment.
7 is a block diagram showing the structure of a processor of a neural vocoder according to an embodiment.
8 is a flowchart illustrating a training method of a neural vocoder for constructing a speaker adaptive model according to an embodiment.
9 illustrates a voice signal and an excitation signal, and a relationship thereof according to an example.
10A to 10C illustrate statistical parametric speech synthesis (SPSS) systems for generating synthesized speech signals using different types of vocoders, respectively.
11 and 13 illustrate a method of training a neural vocoder by separating an excitation signal from a speech signal input for training according to an embodiment.
12 and 14 illustrate a method of generating a synthesized speech signal by estimating an excitation signal from acoustic parameters generated by an acoustic model based on input text, according to an exemplary embodiment.
15 is an acoustic parameter of a negative log-likelihood (NLL) obtained in a training process/synthetic voice signal generation process according to an example, and differences according to whether or not parameters classified according to periodicity are used It is a graph showing
16 schematically shows a speaker-dependent feature and a speaker-independent feature of a voice signal with respect to voice signals from a plurality of speakers according to an example.
FIG. 17 illustrates a synthesized speech of a target speaker using a source model constructed by training speech data sets from a plurality of speakers and a speaker adaptive model constructed by training speech data sets from a target speaker according to an example. How to create.
18 and 19 show results of comparing and evaluating the quality of a synthesized speech signal generated according to whether or not a speaker adaptation algorithm is applied according to an example.
20 shows a result of MOS (Mean Opinion Score) (MOS) evaluation between an ExcitNet vocoder and another vocoder according to an example.
FIG. 21 shows a performance change of a neural vocoder for constructing a speaker adaptive model in a case where the F0 scaling factor is different according to an example.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 shows a method of generating a synthesized speech signal based on an input text or speech signal according to an embodiment.

The voice signal may represent voice, and in the detailed description below, terms of'voice signal' and'voice' may be used interchangeably for convenience of description.

The acoustic model 110 may generate acoustic parameter(s) from a text or speech signal input to generate a synthesized speech signal. The acoustic model 110 may be designed using a deep learning-based statistical parametric speech synthesis (SPSS) system. The acoustic model 110 may consist of multiple feedforward and long-term memory layers trained to represent a nonlinear mapping function between language input and acoustic output parameters. The acoustic model 110 may be, for example, a DNN TTS module. The acoustic parameter may be a feature used to generate a synthesized speech signal, or may be a parameter used to configure a feature.

The vocoder 120 may generate a synthesized speech signal by converting the acoustic parameter generated by the acoustic model 110 into a speech signal. The vocoder 120 may be a neural vocoder. The neural vocoder may be trained using a deep learning model. The neural vocoder may be, for example, WaveNet, SampleRNN, or WaveRNN. Further, the neural vocoder may be a generic generative model that is not limited thereto.

A “neural vocoder” may be used to indicate a device that includes a model trained for generation of a (synthetic) speech signal (eg, WaveNet, SampleRNN, WaveRNN or generic model) and various filters.

The vocoder 120 may estimate an excitation signal of a speech signal based on acoustic parameters obtained from the acoustic model 110. That is, the excitation signal of the voice signal may be a target of the vocoder 120.

The excitation signal is a component representing the vibration of the speech among speech signals, and can be distinguished from a component (spectral component) representing a change in the speech signal that changes according to the shape of a talker's mouth. The change in the excitation signal can be limited only by the movement of the speaker's vocal cord. The excitation signal may be a residual signal of the speech signal.

As the excitation signal estimated by the vocoder 120 is applied with a linear prediction filter generated based on an acoustic parameter representing a spectral component of the speech signal, a target speech signal (i.e., a synthesized speech signal) is generated. I can.

The vocoder 120 targets an excitation signal, not a speech signal, and a target speech signal is generated by applying a linear prediction filter to the estimated excitation signal, thereby improving the quality of the generated target speech signal. , It is possible to reduce spectral distortion in a high frequency region of an audio signal.

A more specific method of generating a target speech signal through estimating an excitation signal and a more specific method of training a neural vocoder to estimate an excitation signal will be described in more detail with reference to FIGS. 2 to 5 to be described later.

2 is a block diagram showing the structure of a neural vocoder system according to an embodiment.

A more detailed configuration of the neural vocoder system 200 will be described with reference to FIG. 2. The illustrated neural vocoder system 200 may represent a computer (computer system) including a neural vocoder.

The neural vocoder system 200 may be implemented as a fixed terminal implemented as a computer system or a mobile terminal. For example, the neural vocoder system 200 includes an AI speaker, a smart phone, a mobile phone, a navigation system, a computer, a notebook computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a tablet PC, and a game console. It may be implemented as a (game console), a wearable device, an internet of things (IoT) device, a virtual reality (VR) device, an augmented reality (AR) device, or the like. Alternatively, the neural vocoder system 200 may be implemented as a server or other computing device that communicates with the aforementioned terminal through a network.

The neural vocoder system 200 may include a memory 210, a processor 220, a communication module 230, and an input/output interface 240. The memory 210 is a non-transitory computer-readable recording medium, and is a non-destructive large capacity such as random access memory (RAM), read only memory (ROM), disk drive, solid state drive (SSD), flash memory, etc. It may include a permanent mass storage device. Here, a non-destructive mass storage device such as a ROM, SSD, flash memory, disk drive, etc. may be included in the neural vocoder system 200 as a separate permanent storage device separate from the memory 210. In addition, the memory 210 includes an operating system and at least one program code (for example, a browser installed and driven in the neural vocoder system 200 or a code for an application installed in the neural vocoder system 200 to provide a specific service, etc.) ) Can be saved. These software components may be loaded from a computer-readable recording medium separate from the memory 210. Such a separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, disk, tape, DVD/CD-ROM drive, and memory card. In another embodiment, software components may be loaded into the memory 210 through the communication module 230 rather than a computer-readable recording medium. For example, at least one program may be loaded into the memory 210 based on a computer program installed by files provided through a file distribution system (eg, an external server) for distributing the installation files of developers or applications. I can.

The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to the processor 220 by the memory 210 or the communication module 230. For example, the processor 220 may be configured to execute a command received according to a program code stored in a recording device such as the memory 210.

The communication module 230 may provide a function for the neural vocoder system 200 to communicate with other electronic devices or servers through a network. The communication module 230 may be a hardware module such as a network interface card, a network interface chip, and a networking interface port of the neural vocoder system 200, or a software module such as a network device driver or a networking program.

The input/output interface 240 may be a means for an interface with an input/output device (not shown). For example, the input device may include a device such as a keyboard, a mouse, a microphone, and a camera, and the output device may include a device such as a display, a speaker, and a haptic feedback device. As another example, the input/output interface 240 may be a means for interfacing with a device in which input and output functions are integrated into one, such as a touch screen. The input/output device may be a configuration of the neural vocoder system 200. When the neural vocoder system 200 is implemented as a server, the neural vocoder system 200 may not include an input/output device and an input/output interface 240.

In addition, in another embodiment, the neural vocoder system 200 may include more components than the illustrated components. However, most of the prior art components need not be clearly shown, which has been omitted.

Referring to FIG. 3, a method of generating a target speech signal by estimating an excitation signal based on more detailed configurations of the processor 220, and a method of training a neural vocoder to estimate an excitation signal will be described. .

In the above, descriptions of the technical features described above with reference to FIG. 1 may be applied to FIG. 2 as they are, and thus, overlapping descriptions will be omitted.

3 is a block diagram showing the structure of a processor of a neural vocoder system according to an embodiment.

Each of the components 310 to 340 of the processor 220 to be described later may be implemented as one or more software modules and/or hardware modules. Depending on the embodiment, components of the processor 220 may be selectively included or excluded from the processor 220. In addition, according to embodiments, components of the processor 220 may be separated or merged to express the function of the processor 220.

Components of the processor 220 may be expressions of different functions of the processor 220 performed by the processor 220 according to an instruction provided by a program code stored in the neural vocoder system 200. .

The parameter acquisition unit 310 of the processor 220 may acquire a plurality of acoustic parameters including spectral parameters and excitation related parameters classified according to periodicity of excitation. The plurality of acoustic parameters acquired by the parameter acquisition unit 310 may be generated by an acoustic model based on a text input from a user or a voice signal input from a speaker.

The excitation signal estimating unit 320 of the processor 220 may estimate an excitation signal based on a plurality of acoustic parameters. The excitation signal estimation unit 320 (neural vocoder) may be trained based on a voice signal input for training. The excitation signal estimator 320 may estimate an excitation signal for a plurality of acoustic parameters by using a probability distribution of an excitation signal modeled through training.

The processor 220 may include a component 340 for performing training of a neural vocoder. The excitation signal separation unit 342 of the processor 220 may separate the excitation signal from the speech signal input for training by applying a linear prediction analysis filter to the speech signal input for training. . The excitation signal separation unit 342 includes a conversion unit 343 for converting a parameter representing a line spectral frequency (LSF) of a speech signal input for training into Linear Predictive Coding (LPC). can do. The linear prediction analysis filter is based on a parameter indicating LSF and may be generated based on the transformed LPC. The modeling unit 344 of the processor 220 may model a probability distribution of the separated excitation signal.

The speech signal generation unit 330 of the processor 220 applies a linear (prediction) synthesis filter based on at least one of spectrum-related parameters to the excitation signal estimated by the excitation signal estimation unit 320 to generate the target speech signal. Can be generated. The target voice signal may be a voiced voice signal.

The speech signal generation unit 330 may include a conversion unit 332 that converts a parameter representing the LSF of the speech signal among the acquired spectrum related parameters into Linear Predictive Coding (LPC). The linear prediction synthesis filter is based on a parameter representing the LSF of a speech signal among acquired spectrum related parameters, and may be generated based on the converted LPC. In other words, the speech signal generation unit 330 may generate a target speech signal by applying a linear prediction synthesis filter based on the converted LPC to the estimated excitation signal.

A more specific method of generating the target speech signal by estimating the excitation signal will be described in more detail with reference to the flowchart of FIG. 4 to be described later, and a more specific method of training a neural vocoder to estimate the excitation signal will be described later. It will be described in more detail with reference to the flow chart in 5.

In the above, descriptions of the technical features described above with reference to FIGS. 1 and 2 may be applied to FIG. 3 as they are, and thus redundant descriptions are omitted.

4 is a flowchart illustrating a method of generating a voice signal according to an exemplary embodiment.

In step 410, the parameter acquisition unit 310 may acquire a plurality of acoustic parameters including spectrum-related parameters and excitation-related parameters classified according to periodicity of excitation. The plurality of acoustic parameters acquired by the parameter acquisition unit 310 may be generated by an acoustic model based on a text input from a user or a voice signal input from a speaker. That is, the parameter obtaining unit 310 may receive the plurality of acoustic parameters from the acoustic model.

Spectral related parameters may be parameters representing spectral components of a spectral component constituting a speech signal. The excitation related parameters may be parameters representing a component corresponding to a residual signal (excitation signal) excluding a spectral component in the speech signal. The signal of the spectral component may represent a part of the speech signal that changes according to the shape of the speaker's mouth. The excitation signal may represent a portion of a voice signal representing vibration of a voice among voice signals. The change in the excitation signal can be limited only by the movement of the speaker's vocal cords.

Spectral related parameters include, for example, a frequency parameter (F0) representing the pitch of the speech signal, an energy parameter representing the energy of the speech signal (for example, a parameter representing the gain), the voiced or unvoiced speech of the speech signal. ) And a parameter indicating a line spectral frequency (LSF) of a voice signal.

Here, the related parameters may include parameters classified according to the periodicity of the excitation. Here, the related parameters may be, for example, TFTE (Time-Frequency Trajectory Excitation) parameters. TFTE can represent the spectral shape of excitation along the frequency axis and the evolution of this shape along the time axis. The excitation-related parameters are a first excitation parameter (SEW (Slowly Evolving Waveform) parameter) representing a component that changes more slowly in the time-frequency axis of the excitation signal, and a second excitation parameter representing a component that changes more rapidly in the time-frequency axis of the excitation signal. It may include an excitation parameter (Rapidly Evolving Waveform (REW) parameter).

The first excitation parameter may indicate excitation below a predetermined cutoff frequency, and the second excitation parameter may indicate excitation exceeding the cutoff frequency. The first excitation parameter may represent a harmonic spectrum of excitation, and the second excitation parameter may represent other parts of the excitation. For example, the first excitation parameter (SEW parameter) corresponding to the harmonic excitation spectrum can be obtained by low-pass filtering each frequency component of the TFTE along the time domain axis (with a predetermined cut-off frequency). The residual noise spectrum exceeding the predetermined cut-off frequency can be obtained through subtracting SEW from TFTE as the second excitation parameter (REW parameter). By using the first excitation parameter (SEW parameter) and the second excitation parameter, the periodicity of excitation can be more effectively expressed. The first excitation parameter and the second excitation parameter may correspond to an ITFTE (Improved Time-Frequency Trajectory Excitation) parameter.

In step 420, the excitation signal estimating unit 320 may estimate an excitation signal based on a plurality of acoustic parameters. That is, the excitation signal estimating unit 320 may estimate the excitation signal by inputting spectrum related parameters and excitation related parameters. The estimated excitation signal may be a time sequence of the excitation signal.

The excitation signal estimating unit 320 is trained based on the voice signal input for training, and the excitation signal estimating unit 320 uses a probability distribution of the excitation signal modeled according to the training to obtain a plurality of acoustic parameters. Can estimate the excitation signal for A training method of a neural vocoder including the excitation signal estimation unit 320 will be described in more detail with reference to FIG. 5 to be described later.

The excitation signal estimating unit 320 may be implemented as, for example, WaveNet, SampleRNN, or WaveRNN. In addition, the excitation signal estimating unit 320 may be implemented as a general generative model that is not limited thereto.

In step 430, the speech signal generation unit 330 applies a linear (prediction) synthesis filter based on at least one of spectrum-related parameters to the excitation signal estimated by the excitation signal estimator 320 Can be created. The target voice signal may be a voiced voice signal. Step 430 will be described in more detail with reference to steps 432 and 434.

In step 432, the conversion unit 332 may convert a parameter representing the LSF of the speech signal among the acquired spectrum-related parameters into Linear Predictive Coding (LPC). The linear prediction synthesis filter is based on a parameter representing the LSF of the speech signal among the acquired spectrum related parameters and may be generated based on the transformed LPC.

In step 434, the speech signal generator 330 may generate a target speech signal by applying a linear prediction synthesis filter based on the LPC transformed in step 432 to the estimated excitation signal.

The target speech signal generated by the steps 410 to 430 has superior quality compared to the speech signal generated by directly estimating the speech signal without estimating the excitation signal as a target. In particular, in the high frequency region of the speech signal The spectral distortion of can be reduced.

In the above, descriptions of the technical features described above with reference to FIGS. 1 to 3 may be applied to FIG. 4 as they are, and thus redundant descriptions are omitted.

5 is a flowchart illustrating a method of training a neural vocoder according to an embodiment.

A method of modeling a probability distribution of an excitation signal capable of estimating an excitation signal based on acquired acoustic parameters will be described in detail with reference to FIG. 5.

In step 510, the neural vocoder system 200 may receive a speech signal for training. The voice signal for training may be directly input to the neural vocoder system 200 from a speaker, or may be input to the neural vocoder system 200 by transmitting data including the voice signal from an electronic device that has received the voice signal.

In step 520, the neural vocoder system 200 may extract a plurality of acoustic parameters including spectrum-related parameters and excitation-related parameters classified according to periodicity of the excitation from the input speech signal. The neural vocoder system 200 may extract a plurality of acoustic parameters from a speech signal through speech analysis. For example, the neural vocoder system 200 may extract a plurality of acoustic parameters from a speech signal by using a parametric vocoder existing inside or outside the neural vocoder system 200.

Spectral related parameters include, for example, a frequency parameter (F0) representing the pitch of the speech signal, an energy parameter representing the energy of the speech signal (for example, a parameter representing the gain), the voiced or unvoiced speech of the speech signal. ) And a parameter indicating a line spectral frequency (LSF) of a voice signal. Here, the related parameters may include parameters classified according to the periodicity of the excitation. Here, the related parameters may be, for example, TFTE (Time-Frequency Trajectory Excitation) parameters. TFTE can represent the spectral shape of excitation along the frequency axis and the evolution of this shape along the time axis. The excitation-related parameters may include a SEW parameter representing a component that changes more slowly in the time-frequency axis of the excitation signal and a REW parameter representing a component that changes more rapidly in the time-frequency axis of the excitation signal. The SEW parameter may indicate excitation below a predetermined cutoff frequency, and the REW parameter may indicate excitation above the cutoff frequency. The SEW parameter may indicate a harmonic spectrum of the excitation, and the REW parameter may indicate other parts of the excitation. For example, the SEW parameter corresponding to the harmonic excitation spectrum can be obtained by low-pass filtering each frequency component of the TFTE along the time domain axis (with a predetermined cut-off frequency). The residual noise spectrum exceeding a predetermined cut-off frequency can be obtained through subtracting SEW from TFTE as the REW parameter.

The above-described steps 510 and 520 may be performed by the processor 220 of the neural vocoder system 200 like steps 530 and 540 to be described later.

In step 530, the excitation signal separation unit 342 applies an excitation signal from the input speech signal by applying a linear prediction analysis filter based on at least one of spectrum related parameters to the input speech signal. Can be separated. The linear prediction analysis filter may be a filter that separates a spectral formant structure from a speech signal. The separated excitation signal may be a residual component (ie, a residual signal) of an input speech signal. The excitation signal is one of various types of excitation models, such as pulse or noise (PoN), band aperiodic (BAP), glottal excitation, and time-frequency trajectory excitation (TFTE) models, to reduce the amount of information. It may be approximated by at least one.

The method of separating the excitation signal from the speech signal will be described in more detail with reference to steps 532 and 534.

In step 532, the conversion unit 343 of the excitation signal separation unit 342 may convert a parameter representing the LSF of the input speech signal among spectrum related parameters into LPC. The linear prediction analysis filter is based on a parameter representing the LSF of the speech signal among the acquired spectrum related parameters, and may be generated based on the converted LPC.

In step 534, the excitation signal separation unit 342 may separate the excitation signal from the speech signal by applying the LPC-based linear prediction analysis filter to the input speech signal.

In step 540, the modeling unit 344 may model a probability distribution of the separated excitation signal. The modeling unit 344 may be implemented as, for example, WaveNet, SampleRNN, or WaveRNN. In addition, the modeling unit 344 may be implemented as a generic generative model that is not limited thereto.

The excitation signal estimating unit 320 may perform estimation of the excitation signal of the above-described step 420 by using the probability distribution of the excitation signal modeled by the modeling unit 344.

The neural vocoder of the embodiment described above with reference to FIGS. 1 to 4 may be referred to as an ExcitNet vocoder in that it trains an excitation signal and generates a synthesized speech signal by estimating the excitation signal.

Since the change in the excitation signal can be limited only by the movement of the speaker's vocal cords, the process of training the excitation signal can be much simpler (compared to training the speech signal). In addition, since the ITFTE parameter is used as a conditional feature that effectively represents the degree of periodicity of the excitation signal, the accuracy of modeling the probability distribution of the excitation signal can be greatly improved.

In the above, descriptions of the technical features described above with reference to FIGS. 1 to 4 may be applied to FIG. 5 as they are, and thus redundant descriptions are omitted.

Below, referring to Figs. 6 to 8, a speaker adaptive model capable of generating a synthesized speech of a target speaker of high quality with only a small amount (ie, a short time) of speech data from the target speaker is constructed, and the target Describes how to generate the speaker's synthesized voice.

6 illustrates a method for generating a synthesized speech of a target speaker by constructing a speaker adaptive model according to an embodiment.

In the detailed description to be described later, the voice data set may represent a voice signal or data including a voice signal. For example, the audio data set may represent audio signals recorded for a predetermined time from a speaker.

The source model 610 may be an acoustic model trained on speech data sets from a plurality of speakers. The source model 610 may be a speaker-independently trained acoustic model for a plurality of speakers. For example, the source model 610 may be a speaker-independently trained acoustic model using a 1-hour speech data set from each of 10 speakers. The source model 610 may be designed using a deep learning-based statistical parametric speech synthesis (SPSS) system. The acoustic model 110 may be, for example, a DNN TTS module.

The source model 610, which is trained speaker-independently through speech data sets from a plurality of speakers, may be used as an initializer of the speaker adaptive model 620. In other words, weights from the source model 610 may be set as initial values in the training of the speaker adaptive model 620 on the speech data set from the target speaker. The weight values from the source model 610 may correspond to the acoustic parameters described above, for example.

The speaker adaptive model 620 may be implemented using a neural vocoder. The neural vocoder may be trained using a deep learning model. The neural vocoder may be, for example, WaveNet, SampleRNN, ExcitNet or WaveRNN. Further, the neural vocoder may be a generic generative model that is not limited thereto.

The speaker adaptation model 620 may be trained independently for a specific speaker by applying a speaker adaptation algorithm. For example, the speaker adaptive model 620 may be trained in a speaker-dependent manner for a specific target speaker (eg, a celebrity, a celebrity, etc.). The speaker adaptive model 620 may generate the updated weight value(s) by training the speech data set from the target speaker.

The speaker adaptive model 620 trains the speech data set from the target speaker using weight values from the source model 610, which is not random, but is trained independently of the speaker, as an initial value, so that a relatively small size ( That is, it is possible to generate a synthesized speech (synthetic speech signal) of a high quality target speaker simply by training a speech data set of a short time). For example, the speaker adaptive model 620 can generate a synthesized speech of a target speaker of high quality just by training a speech data set of the target speaker within 10 minutes.

According to the embodiment, for a celeb where it is difficult to secure a voice data set of several hours to tens of hours or more, it is only necessary to secure a voice data set of about 10 minutes and use it as training data. It is possible to build a speaker adaptive model 620 capable of generating.

In the above, descriptions of the technical features described above with reference to FIGS. 1 to 5 may be applied to FIG. 6 as they are, and thus redundant descriptions will be omitted.

7 is a block diagram showing the structure of a processor of a neural vocoder according to an embodiment.

The processor 220 described with reference to FIG. 7 may correspond to the processor 220 described above with reference to FIG. 3. Each of the components 710 to 720 of the processor 220 to be described later may be implemented as one or more software modules and/or hardware modules. Depending on the embodiment, components of the processor 220 may be selectively included or excluded from the processor 220. In addition, according to embodiments, components of the processor 220 may be separated or merged to express the function of the processor 220. The components 710 to 720 may be expressions of different functions of the processor 220 performed by the processor 220 according to an instruction provided by a program code stored in the neural vocoder system 200. .

The processor 220 may include a speaker adaptive model building unit 720. The speaker adaptive model building unit 720 may set weights from the source model 610, which is independently trained by the speaker, for speech data sets from a plurality of speakers, as initial values, and the set initial values For, it is possible to build a speaker adaptive model 620 that generates updated weight values by training a set of speech data from the target speaker. The updated weight values generated through the speaker adaptive model 620 may be used to generate a synthesized speech corresponding to the target speaker.

The processor 220 may further include a source model building unit 710. The source model building unit 710 may build a source model 610 for independently training speech data sets from a plurality of speakers. The constructed source model 610 can be operated as an initializer of a model for training a speech data set from a target speaker.

The source model building unit 710 is not included in the processor 220 and may be implemented in a device separate from the neural vocoder system 200. The speaker adaptive model construction unit 720 acquires a weight value from the source model 610 built by the source model construction unit 710 implemented in such a separate device, and constructs the speaker adaptive model 620 It is possible to train the target speaker's speech data set to perform.

In the above, descriptions of the technical features described above with reference to FIGS. 1 to 6 may be applied to FIG. 7 as they are, and thus redundant descriptions will be omitted.

8 is a flowchart illustrating a training method of a neural vocoder for constructing a speaker adaptive model according to an embodiment.

In step 810, the source model building unit 710 may build a source model 610 for independently training speech data sets from a plurality of speakers. The plurality of speakers may be any users who provide a set of speech data to train the source model 610.

In step 820, the speaker adaptive model building unit 720 may obtain weight values from the source model 610. The weight values from the source model 610 may represent values representing global characteristics that are not classified for each speaker included in speech data sets from a plurality of speakers. The global characteristic may represent, for example, a formant characteristic or an amplitude-frequency characteristic (pattern) for a specific pronunciation (eg,'ah' or'ee'). That is to say, the source model 610 may train such speaker-independent global characteristics of speech using speech data sets from multiple speakers.

In step 830, the speaker adaptive model construction unit 720 may set weight values obtained from the source model 610 as initial values. In other words, the source model 610 may be used as an initializer of the speaker adaptive model 620 constructed by the speaker adaptive model building unit 720.

In step 840, the speaker adaptive model building unit 720 may generate updated weight values by training a speech data set from a target speaker on the acquired initial value. That is to say, the speaker adaptive model building unit 720 trains the speech data set from the target speaker on the initial value from the source model 610, so that the speaker adapted for the target speaker (that is, dependent on the target speaker) An adaptive model 620 can be built.

The speaker adaptive model building unit 720 adjusts the weight values from the source model 610 to reflect the unique characteristics of the target speaker included in the speech data set from the target speaker, thereby generating updated weight values. have. For example, the speaker adaptive model construction unit 720 trains a speech data set from a target speaker to determine a value representing a global characteristic that is not classified for each speaker from the source model 610, which is a unique characteristic of the target speaker. By fine-tuning to include, updated weight values may be generated.

The generated updated weight values may be used to generate a synthesized speech signal corresponding to the target speaker. The synthesized speech signal corresponding to the target speaker may be, for example, a synthesized speech of a celebrity corresponding to the target speaker.

The size of each of the speech data sets from a plurality of speakers for training the source model 610 (i.e., the length of the recorded speech signal, e.g., 1 hour or more) is the size of the speech data set from the target speaker ( That is, it may be longer than the length of the recorded voice signal, eg, 10 minutes).

Through a fine-tuning mechanism of the adaptation process as described in step 830, the target speaker's unique characteristics may be captured from the target speaker's speech data set. Therefore, through the method of the described embodiment, even if the voice data set for training from the target speaker is insufficient, vocoding performance can be improved.

The training method of the neural vocoder described above with reference to FIGS. 6 to 8 may be combined with the training method of the neural vocoder of the embodiment described above with reference to FIGS. 1 to 4 and the method of generating a synthesized speech signal. For example, the above-described ExcitNet vocoder can be combined with the embodiments described above with reference to FIGS. 6 to 8.

For example, a neural vocoder trained by performing steps 810 to 840 may correspond to the neural vocoder system 200 described above with reference to FIGS. 1 to 4. The target speech signal generated in step 430 may be a synthesized speech signal corresponding to the target speaker trained by the speaker adaptive model 620.

By combining the training method of the neural vocoder described above with reference to FIGS. 6 to 8 and the technical characteristics of the ExcitNet model described above with reference to FIGS. 1 to 4, the quality of the synthesized speech corresponding to the target speaker can be improved.

In the above, descriptions of the technical features described above with reference to FIGS. 1 to 7 may be applied to FIG. 8 as they are, and thus redundant descriptions will be omitted.

9 illustrates a voice signal and an excitation signal, and a relationship thereof according to an example.

As shown, assuming that the speech signal is S(n) and the excitation signal included in S(n) is e(n), the relationship between S(n) and e(n) is Equation 1 below. Can be expressed together.

h(n) may represent a linear prediction synthesis filter. h(n) may represent the remaining components (ie, spectral components) excluding the e(n) component of S(n). h(n) may be generated based on a parameter representing the LSF of S(n).

Target by applying a linear prediction synthesis filter (ie, h(n)) to the excitation signal (ie, e(n)) estimated by step 420 described above with reference to FIG. 4 according to the relationship of Equation 1 An audio signal S(n) may be generated. A specific example of the linear prediction synthesis filter will be described in more detail with reference to FIG. 14.

The relationship of Equation 1 may be similarly applied to the separation of the excitation signal (ie, e(n)) of the step 530 described above with reference to FIG. 5. That is to say, the excitation signal e(n) can be separated from the speech signal S(n) by applying a linear prediction analysis filter to the speech signal S(n) input for training. A specific example of the linear prediction analysis filter will be described in more detail with reference to FIG. 13.

In the above, descriptions of the technical features described above with reference to FIGS. 1 to 8 may be applied to FIG. 9 as they are, and thus redundant descriptions are omitted.

10A to 10C illustrate statistical parametric speech synthesis (SPSS) systems for generating synthesized speech signals using different types of vocoders, respectively.

10A is a diagram for speech synthesis including an acoustic model 1010 and an LPC synthesis module 1020 that generates a speech signal by synthesizing an acoustic feature (sound parameter) from the acoustic model 1010 with Linear Predictive Coding (LPC). Represents the framework. The LPC synthesis module 1020 is an LPC vocoder, and may correspond to, for example, the linear prediction synthesis filter described above.

10B shows a framework for speech synthesis including a WaveNet vocoder 1022 as a neural vocoder that estimates a speech signal based on an acoustic model 1010 and an acoustic feature (sound parameter) from the acoustic model 1010.

FIG. 10C may show a framework for speech synthesis using the ExcitNet vocoder 1024, as described above with reference to FIGS. 1 to 5. The structure shown in FIG. 10C may be a combination of the LPC coder 1020 of FIG. 10A and the WaveNet vocoder 1022 of FIG. 10B.

In the structure of FIG. 10C, the ExcitNet vocoder 1022 can estimate the excitation signal based on the acoustic features (sound parameters) from the acoustic model 1010. The estimated excitation signal may be converted as a target speech signal according to LPC (Linear Predictive Coding) synthesis through the linear prediction synthesis filter 1030.

A more detailed example of the structure of FIG. 10C will be described in more detail with reference to FIGS. 12 and 14.

In the above, descriptions of the technical features described above with reference to FIGS. 1 to 9 may be applied to FIGS. 10A to 10C as they are, so a duplicate description will be omitted.

11 and 13 illustrate a method of training a neural vocoder by separating an excitation signal from a speech signal input for training according to an embodiment.

As illustrated in FIG. 11, for a speech signal input for training, the parametric vocoder 1110 may extract acoustic parameters. For the input speech signal, the excitation signal may be separated from the input speech signal by applying the linear prediction analysis filter 1140 generated based on spectrum-related parameters among the extracted acoustic parameters.

The WaveNet vocoder 1130 may configure (1120) and receive the extracted acoustic parameters as auxiliary features. The auxiliary feature may include the spectral related parameters and excitation related parameters described above. The WaveNet vocoder 1130 may model a probability distribution of the excitation signal based on the auxiliary feature and the separated excitation signal. The WaveNet vocoder 1130 may be implemented as an ExcitNet vocoder or a neural vocoder of another generic generative model.

The structure of FIG. 11 will be described in more detail with reference to FIG. 13. In the speech signal input for training, acoustic features (sound parameters) may be extracted according to the speech analysis 1310. Among the acoustic parameters, a parameter representing the LSF may be converted to LPC (1320). The linear prediction analysis filter 1340 may be implemented based on the transformed LPC. The excitation signal may be separated from the input speech signal by applying the linear prediction analysis filter 1340 to the input speech signal. The separated excitation signal may be input to an ExcitNet model (ie, an ExcitNet vocoder) 1350. Meanwhile, acoustic parameters may be configured as auxiliary features (1330), and auxiliary features may be input to the ExcitNet model (1350). The ExcitNet model 1350 includes input auxiliary features (ie, acoustic features). Parameters) and the separated excitation signal, the probability distribution of the excitation signal may be modeled In the illustrated example, e _n may correspond to the separated excitation signal.

12 and 14 illustrate a method of generating a synthesized speech signal by estimating an excitation signal from acoustic parameters generated by an acoustic model based on input text, according to an exemplary embodiment.

As illustrated in FIG. 12, the acoustic model 1150 may generate acoustic parameters based on the received language parameters. The WaveNet vocoder 1170 may configure (1160) receive acoustic parameters as auxiliary features. The auxiliary feature may include the spectral related parameters and excitation related parameters described above. The WaveNet vocoder 1170 may estimate the excitation signal based on the acoustic parameters. The WaveNet vocoder 1170 may also be implemented as an ExcitNet vocoder or a neural vocoder of another generic generative model. A target synthesized speech may be generated by applying the linear prediction synthesis filter 1180 generated based on spectrum related parameters among the extracted acoustic parameters to the estimated excitation signal.

은 생성된 타겟 음성 신호에 대응할 수 있고,

은 추정된 여기 신호에 대응할 수 있다. The structure of FIG. 12 will be described in more detail with reference to FIG. 14. Language features (corresponding to the language parameters described above) may be extracted by performing text analysis 1410 on the input text for generation of a synthetic speech signal. In extracting the language feature, as shown, a duration model 1420 for estimating a phoneme duration may be further used. The acoustic model 1430 may generate acoustic features (sound parameters) from the extracted language features. Among the acoustic parameters, a parameter indicating the LSF may be converted to LPC (1440). A linear prediction synthesis filter 1470 may be implemented based on the transformed LPC. Acoustic parameters can be configured as auxiliary features (1450), and the auxiliary features can be input into an ExcitNet model (ie, ExcitNet vocoder) 1460. The ExcitNet model 1460 may estimate the excitation signal based on the input auxiliary features (ie, acoustic parameters). A target speech signal may be generated by applying the linear prediction synthesis filter 1470 based on the transformed LPC to the estimated excitation signal. In the illustrated example,

May correspond to the generated target speech signal,

May correspond to the estimated excitation signal.

As described above, descriptions of the technical features described above with reference to FIGS. 1 to 10C may be applied to FIGS. 11 to 14 as they are, so a duplicate description will be omitted.

15 is an acoustic parameter of a negative log-likelihood (NLL) obtained in a training process/synthetic voice signal generation process according to an example, and the difference according to whether or not parameters classified according to periodicity are used. It is a graph showing.

In the training process, the lower the NLL, the higher the modeling accuracy. In the illustrated graph, it can be seen that the NLL in the case of using the ITFTE parameter such as the SEW parameter and the REW parameter described above is lower than in the case of not.

In addition, in the validation process, the lower the NLL, the better the quality of the generated synthesized speech. In the illustrated graph, it can be seen that the NLL in the case of using the ITFTE parameter such as the SEW parameter and the REW parameter is lower than in the case of not.

That is to say, through the graph shown, it is possible to greatly reduce the error in modeling the probability distribution of the excitation signal through the use of the ITFTE parameter in the training of the neural vocoder, and in the estimation of the excitation signal for the generation of synthetic speech It can be seen that by using the parameters, errors in the generation of a synthesized speech signal can be greatly reduced.

In the above, descriptions of the technical features described above with reference to FIGS. 1 to 14 may be applied to FIG. 15 as they are, and thus redundant descriptions will be omitted.

16 schematically shows a speaker-dependent feature and a speaker-independent feature of a voice signal with respect to voice signals from a plurality of speakers according to an example. FIG. 17 illustrates a synthesized speech of a target speaker using a source model constructed by training speech data sets from a plurality of speakers and a speaker adaptive model constructed by training speech data sets from a target speaker according to an example. Shows how to create.

As shown in FIG. 16, the speaker-independent feature may be a common feature in the voices of speakers (speakers 1 to 3). In other words, speaker-independent characteristics may represent global characteristics that are not classified for each speaker. Speaker-dependent characteristics can represent unique characteristics for each speaker.

As shown in FIG. 17, a source model 610 may be constructed by speaker-independently training speech data sets from a plurality of speakers, and based on the weight values from the source model 610, By training the speech data set, a speaker adaptive model 620 that is dependent on the target speaker can be built. The weight value from the source model 610 may be finely adjusted to reflect the unique characteristics of the target speaker as the speech data set from the target speaker is trained in the speaker adaptive model 620. As shown, the source model 610 and the speaker adaptive model 620 can be implemented using the ExcitNet model. As shown, according to an embodiment, a speaker adaptation algorithm may be applied to a neural vocoder. Although not shown, a speaker adaptation algorithm may be similarly applied to an acoustic model (eg, DNN TTS) corresponding to the source model 610.

In the above, descriptions of the technical features described above with reference to FIGS. 1 to 15 may be applied to FIGS. 16 and 17 as they are, and thus redundant descriptions will be omitted.

18 and 19 show results of comparing and evaluating the quality of a synthesized speech signal generated according to whether or not a speaker adaptation algorithm is applied according to an example.

The scores of FIGS. 18 and 19 represent the average of scores evaluated by the evaluators listening to the voice signal. RAW can correspond to the original audio signal.

Referring to FIG. 18, it can be seen that the quality of the synthesized speech signal is highly evaluated when the speaker adaptation algorithm is applied in both the WaveNet model and the ExcitNet model. In other words, it can be seen that the case where the speaker adaptive model 620 is constructed to generate a synthesized speech signal (with SA) as described above with reference to FIGS. 6 to 8 exhibits superior performance compared to the case where it is not.

19 shows a result of comparing and evaluating the quality of a more detailed synthesized speech signal. 19 will be described in more detail below.

As described above, descriptions of the technical features described above with reference to FIGS. 1 to 17 may be applied to FIGS. 18 and 19 as they are, so a duplicate description will be omitted.

In the following, the above-described ExcitNet model will be described in more detail with reference to FIGS. 1 to 5, and comparative test results with other models will be further described.

The ExcitNet model (ExcitNet vocoder) may be a WaveNet-based neural excitation model for a statistical parametric speech synthesis (SPSS) system. The WaveNet-based neural vocoding system greatly improves the recognition quality of the synthesized speech signal, but sometimes it outputs noise because it cannot capture the complex time-varying characteristics of the speech signal. ExcitNet-based neural vocoding system can train by separating residual components (i.e., excitation signals) using an adaptive inverse filter that separates spectral components from speech signals (e.g., within the WaveNet framework), and In generating a signal, an excitation signal can be estimated as a target. Through this method, since the spectral component of the speech signal can be better expressed by the deep learning framework and the residual component is efficiently generated by the WaveNet framework, the quality of the synthesized speech signal can be improved.

The experimental results below also show that the ExcitNet-based neural vocoding system (speaker-dependent and speaker-independently trained) outperforms the existing linear prediction vocoder and WaveNet vocoder.

For testing, two speech coppers, which are phonetically rich in prosody, were used to train an acoustic model and a speaker-dependent (SD) ExcitNet vocoder. Each corpus was recorded by a professional Korean woman (KRF) and a Korean man (KRM). The speech signal was sampled at 24 kHz, and each sample was quantized to 16 bits. Table 1 below shows the number of utterances in each set. Voice copper recorded by 5 Korean women and 5 Korean men was used to train the speaker independent (SI) ExcitNet vocoder. A total of 6,422 (10 hours) and 1,080 (1.7 hours) utterances were used for training and validation, respectively. Voice samples recorded by the same KRF and KRM speakers not included in the SI data set were used for testing.

Speaker training Verification exam KRF 3,826(7h) 270(30min) 270(30min) KRM 2,294(7h) 160(30min) 160(30min)

Tables 2 and 3 below show the distortion between the original voice and the generated voice as a result of an objective test in LSD (Log-Spectral Distance) (dB) and F0 RMSE (Root Mean Square Error) (Hz), respectively. WN denotes WaveNet vocoder, WN-NS denotes applying the noise-shaping method to WaveNet vocoder, and ExcitNet denotes ExcitNet vocoder. The smallest error appears in bold. Tables 2 and 3 may be results measured for voiced sounds.

Speaker system SD SI 1h 3h 5h 7h KRF WN 4.21 4.19 4.18 4.13 4.18 WN-NS 4.15 4.12 4.07 4.01 4.06 ExcitNet 4.11 4.09 4.05 3.99 4.04 KRM WN 3.73 3.72 3.69 3.67 3.70 WN-NS 3.54 3.46 3.41 3.41 3.46 ExcitNet 3.53 3.46 3.41 3.40 3.45

Speaker system SD SI 1h 3h 5h 7h KRF WN 32.61 31.69 31.56 31.49 32.30 WN-NS 31.96 31.75 31.52 31.38 32.23 ExcitNet 31.44 31.43 31.37 31.29 31.88 KRM WN 12.60 12.39 12.05 12.05 13.96 WN-NS 12.32 12.16 11.97 11.97 13.34 ExcitNet 12.72 12.10 11.93 11.93 12.96

As shown in Tables 2 and 3, it can be seen that in most cases of SD and SI, the ExcitNet vocoder shows the least distortion between the original voice and the generated voice. Table 4 below shows the measured LSD (dB) for unvoiced sounds and transition regions.

Speaker system SD SI 1h 3h 5h 7h KRF WN 4.19 4.16 4.15 4.09 4.18 WN-NS 4.26 4.18 4.12 4.03 4.06 ExcitNet 4.15 4.10 4.06 3.98 4.04 KRM WN 3.95 3.96 3.92 3.92 3.70 WN-NS 4.41 3.95 3.88 3.88 3.46 ExcitNet 3.91 3.83 3.76 3.76 3.45

As shown in Table 4, it can be seen that in all cases of SD and SI, the ExcitNet vocoder has the least distortion between the original voice and the generated voice. Tables 5 and 6 below show the results (%) of the preference test as the result of the subjective test. The parts that showed high preference from listeners are indicated in bold in the table. Compared to the rest, it can be seen that the ExcitNet vocoder has remarkably superior recognition quality of synthesized speech. The evaluators were 12 Korean-speaking listeners, and 20 randomly selected utterances were tested.

KRF WN WN-NS ExcitNet Neutral p-value SD 6.8 64.1 - 29.1 <10 ^-30 7.3 - 83.6 9.1 <10 ^-49 - 12.7 58.2 29.1 <10 ^-17 SI 12.7 66.8 - 20.5 <10 ^-22 8.6 - 73.6 27.7 <10 ^-35 - 19.5 38.6 41.8 <10 ^-3

KRF WN WN-NS ExcitNet Neutral p-value SD 11.8 60.5 - 27.7 <10 ^-19 17.3 - 77.7 5.0 <10 ^-24 - 16.4 73.6 10.0 <10 ^-22 SI 27.3 48.6 - 24.1 <10 ^-3 13.6 - 75.5 10.9 <10 ^-27 - 17.3 63.6 19.1 <10 ^-15

20 shows a result of MOS (Mean Opinion Score) (MOS) evaluation between an ExcitNet vocoder and another vocoder according to an example. An evaluation of the result of analysis/synthesis (A/S) in the case of extracting an acoustic feature from the recorded voice and evaluation of the result in the SPSS in the case of generating an acoustic feature from an acoustic model were performed.

In terms of S/A, the SI-ExcitNet vocoder showed similar performance to the ITFTE vocoder, but far superior to the WORLD system. In all cases, the SD-ExcitNet vocoder showed the highest recognition quality (representing 4.35 and 4.47 MOS for KRF and KRM speakers, respectively). Because it is difficult to express high-pitched female voices, the MOS results for KRF speakers were worse than those for KRM speakers in SI vocoders (WORLD, ITFTE and SI-ExcitNet). On the other hand, SD-ExcitNet's results for KRF speakers are similar to those for KRM speakers, indicating that speaker-specific characteristics must be modeled in order to effectively express high-pitched voices. In terms of SPSS, both SD and SI-ExcitNet vocoders showed much better recognition quality than parametric ITFTE vocoders. Although the acoustic model produced overly flat speech parameters, the ExcitNet vocoder was able to mitigate the smoothing effect by directly estimating the time domain excitation signal. As a result, the SPSS system using the SD-ExcitNet vocoder achieved 3.78 and 3.85 MOS for KRF and KRM speakers, respectively. SI-ExcitNet vocoder achieved 2.91 and 2.89 MOS for KRF and KRM speakers, respectively.

In the following, a neural vocoder that constructs the speaker adaptive model 620 described above will be described in more detail with reference to FIGS. 6 to 8, and a comparison test result with other models will be further described. The neural vocoder of the embodiment can construct a high-quality speech synthesis system even when training data from a target speaker is insufficient, such as a speech data set of only 10 minutes.

The neural vocoder of the embodiment is a speaker-independently trained source model capable of extracting universal characteristics for a plurality of speakers in order to solve the problem of lack of information related to the target speaker caused by limited training data for the target speaker ( 610) is used. The weight value from the source model 610 is used to initialize training of the speaker adaptive model 620, and may be finely adjusted to indicate the unique characteristics of the target speaker. According to this adaptation process, since the deep neural network can capture the characteristics of the target speaker, the discontinuity problem occurring in the speaker-independent model can be reduced. Experimental results to be described later also show that the neural vocoder of the embodiment significantly improves the recognition quality of synthesized speech compared to the conventional method.

SD represents a speaker-dependent trained model (without the weight from the source model 610 as an initial value), SI represents a speaker-independently trained model, and SA is described above with reference to FIGS. 6 to 8 A model trained in speaker adaptive mode as described above (that is, a model trained in a speaker-dependent manner using the weight from the source model 610 as an initial value) is shown.

In the SD and SA models, voice copper recorded by a Korean female speaker was used. The speech signal was sampled at 24 kHz, and each sample was quantized to 16 bits. A total of 90 (10 minutes), 40 (5 minutes), and 130 (15 minutes) utterances were used for training, validation and testing. To train the SI model, voice data recorded by 5 Korean male speakers and 5 Korean female speakers that were not included in the SD and SA model training were used. For this, 6,422 (10 hours) and 1,080 (1.7 hours) utterances were used for training and verification, respectively. Test sets of SD and SA models were also used to evaluate SI models.

Tables 7 and 8 below show the distortion between the original voice and the generated voice as a result of an objective test in LSD (Log-Spectral Distance) (dB) and F0 RMSE (Root Mean Square Error) (Hz), respectively. Table 7 shows the evaluation (A/S) of the results of analysis/synthesis when acoustic features extracted from recorded speech are directly used to construct auxiliary features. Table 8 shows the evaluation of the results in SPSS. The smallest error appears in bold.

system WaveNet ExcitNet LSD F0 RMSE LSD F0 RMSE SD 3.65 38.27 2.10 19.83 SI 2.00 10.64 1.32 10.48 SA 1.79 10.70 1.12 9.66

system WaveNet ExcitNet LSD F0 RMSE LSD F0 RMSE SD 4.78 48.75 4.50 39.14 SI 4.51 35.53 4.42 36.28 SA 4.45 35.45 4.36 35.47

In Tables 7 and 8, it can be seen that in both the WaveNet vocoder and the ExcitNet vocoder, the case of SA shows the least distortion between the original voice and the generated voice. FIG. 21 shows an F0 scaling factor according to an example. It represents the performance change of the neural vocoder constructing the speaker adaptive model in the case of different from.

In order to verify the effectiveness of the training method applying the SA of the embodiment, the performance change of the neural vocoder when F0 is manually changed was investigated. It has been found that the SI model is effective in generating pitch-modified synthetic speech. Since the SA model also uses the SI model, it can be expected to show higher performance than the SD approach.

In the test, the F0 trajectory was generated by the SPSS framework and then multiplied by a scaling factor (0:6, 0:8, 1:0 and 1:2) to correct the auxiliary feature vector. The speech signal was synthesized using a neural vocode system.

21 shows the results of the F0 RMSE (Hz) test for different F0 scaling factors. 21, it can be seen that the SA model includes a much smaller modification error compared to the existing SD model. Compared with the SI model, it can be seen that the SA-ExcitNet vocoder maintains the same quality even though all weights are optimized for the characteristics of the target speaker.

In addition, it can be seen that the ExcitNet vocoder shows much better performance than the WaveNet vocoder. Since the ExcitNet vocoder trains the vocal cord movement change (excitation signal change), it can be seen that it can reconstruct the F0-modified voice segment more flexibly than the WaveNet-based approach.

19 is a subjective test result, and shows MOS evaluation results between vocoders of SD, SI and SA. An evaluation of the result of analysis/synthesis (A/S) in the case of extracting an acoustic feature from the recorded voice and evaluation of the result in the SPSS in the case of generating an acoustic feature from an acoustic model were performed.

In the A/S result, the SD-WaveNet vocoder was not able to train the target speaker's characteristics with limited training data, resulting in the worst result. The SI-WaveNet vocoder showed similar performance to the ITFTE vocoder, and it was found to outperform WORLD system. It was confirmed that the use of SA in all WaveNet vocoders shows excellent performance. The results for the ExcitNet vocoder showed a similar trend to that of the WaveNet vocoder, but the ExcitNet vocoder improved the modeling accuracy of the remaining signals by separating the formant component of the voice signal through the LP inverse filter. appear. As a result, the SA-ExcitNet vocoder achieved 4.40 MOS in A/S results.

In the results of SPSS, both SI-WaveNet vocoder and SI-ExcitNet vocoder were found to provide better recognition quality than parametric ITFTE vocoder. As a result, it was confirmed that the SA training model of the embodiment significantly improved the quality of the synthesized speech signal compared to the existing speaker-dependent method and speaker-independent method. As in the A/S result, the ExcitNet vocoder showed better performance than the WaveNet vocoder in the SPSS result. Although the acoustic model produced overly flat speech parameters, the ExcitNet vocoder was able to mitigate the smoothing effect by directly estimating the time domain excitation signal. As a result, the SPSS system with the SA-ExcitNet vocoder achieved 3.77 MOS.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments are a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable gate array (PLU). It may be implemented using one or more general purpose computers or special purpose computers, such as a logic unit), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be embodyed in any type of machine, component, physical device, computer storage medium or device to be interpreted by the processing device or to provide instructions or data to the processing device. have. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. In this case, the medium may be one that continuously stores a program executable by a computer, or temporarily stores a program for execution or download. In addition, the medium may be a variety of recording means or storage means in a form in which a single or several pieces of hardware are combined, but is not limited to a medium directly connected to a computer system, but may be distributed on a network. Examples of media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magnetic-optical media such as floptical disks, and And ROM, RAM, flash memory, and the like may be configured to store program instructions. In addition, examples of other media include an app store that distributes applications, a site that supplies or distributes various software, and a recording medium or a storage medium managed by a server.

As described above, although the embodiments have been described by the limited embodiments and drawings, various modifications and variations are possible from the above description by those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and claims and equivalents fall within the scope of the claims to be described later.

Claims (17)

In the method of generating a speech signal performed by a neural vocoder implemented by a computer,
Obtaining a plurality of acoustic parameters including spectral parameters and excitation related parameters classified according to periodicity of excitation;
Estimating an excitation signal based on the plurality of acoustic parameters; And
Generating a target speech signal by applying a linear synthesis filter based on at least one of the spectrum-related parameters to the estimated excitation signal
Including,
The neural vocoder is trained based on the voice signal input for training,
The above training,
Separating an excitation signal from the input speech signal by applying a linear prediction analysis filter to the input speech signal; And
Modeling the probability distribution of the separated excitation signal
Including,
The step of estimating the excitation signal,
Using the modeled probability distribution of the excitation signal, an excitation signal for the plurality of acoustic parameters is estimated. The method of claim 1,
The excitation-related parameters include a first excitation parameter indicating excitation below a predetermined cutoff frequency and a second excitation parameter indicating excitation above the cutoff frequency. The method of claim 2,
The first excitation parameter represents a harmonic spectrum of the excitation, and the second excitation parameter represents the other part of the excitation. The method of claim 1,
The spectrum related parameters are,
A frequency parameter representing the pitch of the speech signal, an energy parameter representing the energy of the speech signal, a parameter representing whether the speech signal is voiced or unvoiced, and the line spectral frequency (LSF) of the speech signal. A method of generating a speech signal, including parameters. The method of claim 4,
Generating the target speech signal,
Converting the parameter representing the LSF into Linear Predictive Coding (LPC); And
Applying the linear synthesis filter based on the transformed LPC to the estimated excitation signal
Including a voice signal generation method. The method of claim 1,
The plurality of acoustic parameters are generated by an acoustic model based on an input text or an input speech signal. delete The method of claim 1,
The step of separating the excitation signal,
Converting a parameter representing the LSF of the input speech signal into Linear Predictive Coding (LPC); And
Applying the linear prediction analysis filter based on the converted LPC of the input speech signal to the input speech signal
Including a voice signal generation method. The method of claim 1,
The separated excitation signal is a residual component of the input speech signal. In the training method of a neural vocoder implemented by a computer,
Receiving a voice signal for training the neural vocoder;
Extracting a plurality of acoustic parameters including spectrum-related parameters and excitation-related parameters classified according to periodicity of excitation from the input speech signal;
Separating an excitation signal from the input speech signal by applying a linear prediction analysis filter based on at least one of the spectrum-related parameters to the input speech signal; And
Modeling the probability distribution of the separated excitation signal
Including,
The probability distribution of the modeled excitation signal is used to estimate the excitation signal for other spectral related parameters and other acoustic parameters including other excitation related parameters obtained. The method of claim 10,
The step of separating the excitation signal,
Converting a parameter representing the LSF of the input voice signal from among the spectrum related parameters to LPC; And
Applying the linear prediction analysis filter based on the converted LPC of the input speech signal to the input speech signal
Including, training method of a neural vocoder. The method of claim 10,
The excitation-related parameters include a first excitation parameter indicating excitation below a predetermined cutoff frequency and a second excitation parameter indicating excitation exceeding the cutoff frequency. A non-transitory computer-readable recording medium in which a program for executing the method of any one of claims 1 to 6 and 8 to 12 is recorded on a computer. In the neural vocoder,
A parameter acquisition unit that acquires a plurality of acoustic parameters including spectral parameters and excitation related parameters classified according to periodicity of excitation;
An excitation signal estimating unit estimating an excitation signal based on the plurality of acoustic parameters; And
A speech signal generator that generates a target speech signal by applying a linear synthesis filter based on at least one of the spectrum-related parameters to the estimated excitation signal
Including,
The neural vocoder is trained based on the voice signal input for training,
An excitation signal separation unit for separating an excitation signal from the input speech signal by applying a linear prediction analysis filter to the input speech signal; And
Modeling unit for modeling the probability distribution of the separated excitation signal
Including more,
The excitation signal estimating unit estimates an excitation signal for the plurality of acoustic parameters by using a probability distribution of the modeled excitation signal. The method of claim 14,
The speech signal generation unit includes a conversion unit for converting a parameter representing the LSF of the speech signal among the spectrum related parameters into Linear Predictive Coding (LPC),
A neural vocoder for applying the linear synthesis filter based on the transformed LPC to the estimated excitation signal. delete The method of claim 14,
The excitation signal separation unit includes a conversion unit for converting a parameter representing the LSF of the input speech signal into Linear Predictive Coding (LPC),
A neural vocoder for applying the linear prediction analysis filter based on a transformed LPC of the input speech signal to the input speech signal.

Images