KR102624194B1 - Non autoregressive speech synthesis system and method using speech component separation - Google Patents

Abstract

A non-autoregressive speech synthesis system and method using speech component separation is disclosed. A speech synthesis method performed by a speech synthesis system according to an embodiment is a non-autoregressive speech consisting of a first network for separating speech components from speech and a second network for estimation of speech components from text. Receiving speech input into the synthesis model; and estimating the final spoken voice through speech synthesis of the spoken voice using the configured non-autoregressive-based speech synthesis model, wherein the non-autoregressive-based speech synthesis model is unsupervised in the first network. It may be configured to separate voice components from the spoken voice through learning, generate the separated voice components from the text in the second network, and estimate the final spoken voice through voice synthesis in a non-autoregressive manner.

Description

Non-autoregressive speech synthesis system and method using speech component separation {NON AUTOREGRESSIVE SPEECH SYNTHESIS SYSTEM AND METHOD USING SPEECH COMPONENT SEPARATION}

The explanation below is about voice synthesis technology.

Text-to-speech (TTS), a mapping process that converts character sequences into speech (or sound functions), is an essential technology for designing human-computer interaction systems. Recently, neural TTS models have achieved significant success in terms of quality and naturalness of synthesized speech based on an autoregressive end-to-end framework. Autoregressive methods have contributed to the rapid development of neural TTS models, but the training and inference speeds are low, and a single misprediction of a speech frame can affect the inference of all subsequent frames.

In recent years, non-autoregressive TTS models have focused on using parallel architecture to solve autoregressive TTS model problems while achieving similar performance to autoregressive TTS models.

However, in the conventional technology, the estimated value of prosody, a component of speech, is used as a target in the speech synthesis model, so the occurrence of errors due to estimation errors cannot be avoided. In addition, it is possible to convert timbres through voice component separation technology through unsupervised learning, but a voice synthesis system using this has not been built.

It is possible to provide a non-autoregressive-based speech synthesis method and system that separates speech components from spoken speech through unsupervised learning without explicit prosodic labels and generates the separated speech components from text to estimate the final spoken speech. there is.

The speech synthesis method performed by the speech synthesis system inputs the spoken voice into a non-autoregressive speech synthesis model consisting of a first network for separation of speech components and a second network for estimation of speech components from text. Receiving stage; and estimating the final spoken voice through speech synthesis of the spoken voice using the configured non-autoregressive-based speech synthesis model, wherein the non-autoregressive-based speech synthesis model is unsupervised in the first network. It may be configured to separate voice components from the spoken voice through learning, generate the separated voice components from the text in the second network, and estimate the final spoken voice through voice synthesis in a non-autoregressive manner.

The non-autoregressive-based speech synthesis model is a prosody component and content including a speaker, rhythm, and pitch from the spoken voice through unsupervised learning in the first network. separate components, estimate components for the separated content from the text in the second network, and generate, at a decoder, a plurality of spectrograms based on the parallel structure of the first network and the second network. It may be configured.

The first network may include a first content encoder, a rhythm encoder, and a pitch encoder for separating the voice components.

Latent vectors representing content codes, rhythm codes, and pitch codes are extracted from the first network through a first content encoder, rhythm encoder, and pitch encoder included in the first network, and the extracted latent vectors are It can be used as input data for the decoder to estimate the spectrogram of the final spoken voice.

The second network may include a second content encoder and a first content encoder learned from the first network.

In the second network, text is converted into linguistic features including content information through the second content encoder, and each of the converted linguistic features is analyzed through a duration predictor and a length regulator. A scalar value corresponding to the duration of the text token is estimated, and the length of the linguistic feature corresponding to each text token can be adjusted based on the estimated scalar value.

As the linguistic features converted from the second network are input to the first content encoder learned from the first network, content information may be extracted through random resampling.

The content encoder learned in the first network extracts a content code, and the spectrogram of the final spoken voice can be estimated through the extracted content code through the second network.

The decoder can be trained to reconstruct the final utterance spectrogram.

The duration predictor can be learned without explicit labels through a length-level loss for the duration of each text token.

The loss of the length level may be induced so that the sum of scalar values output from the period predictor becomes the length of the final utterance spectrogram.

It may include a computer program stored in a non-transitory computer-readable recording medium to execute the speech synthesis method on the speech synthesis system.

The speech synthesis system includes a speech input unit that receives spoken speech into a non-autoregressive speech synthesis model consisting of a first network for separation of speech components and a second network for estimation of speech components from text; and a speech estimation unit that estimates the final spoken voice through speech synthesis of the spoken voice using the constructed non-autoregressive speech synthesis model, wherein the non-autoregressive based speech synthesis model is used as a busy signal in the first network. It may be configured to separate voice components from the spoken voice through learning, generate the separated voice components from the text in the second network, and estimate the final spoken voice through voice synthesis in a non-autoregressive manner.

Through non-autoregressive speech synthesis technology based on speech component separation technology through unsupervised learning, content elements and prosodic elements are separated from spoken speech without separate explicit labels, and separated content elements and separated prosodic elements are separated. By utilizing it, the training pipeline can be simplified and high expressivity and tunability can be achieved while maintaining high-quality speech synthesis performance.

Figure 1 is a diagram for explaining the structure of a non-autoregressive-based speech synthesis model, according to one embodiment.
Figure 2 is a diagram for explaining the detailed structure of a non-autoregressive-based speech synthesis model, according to one embodiment.
Figure 3 is a block diagram for explaining the configuration of a voice synthesis system according to an embodiment.
Figure 4 is a flowchart for explaining a voice synthesis method in a voice synthesis system according to an embodiment.
Figure 5 is a diagram showing the results of a mel spectrogram generated using content information extracted from voice and content, respectively, in one embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

Figure 1 is a diagram for explaining the structure of a non-autoregressive-based speech synthesis model, according to one embodiment.

In a speech synthesis model, it is a difficult task to directly estimate speech in which various information is intertwined in a parallel manner from text. In an embodiment, a system and method will be described in which a technology for separating content including text information and prosody information corresponding to intonation, speaker information, rhythm, etc. from the spoken voice is applied to a non-autoregressive voice synthesis technology. The non-autoregressive-based speech synthesis model 100 according to the embodiment is composed of a network for decomposing speech components and a network for generating decomposed speech components from text by synthesizing speech from speech components. It enables more effective learning than directly generating voices.

The speech synthesis system includes a first network (SCD; Speech Component Decomposition network) 110 for separation of speech components and a second network (SCG; Speech Component Generation network) 120 for estimation of speech components from text. It is possible to construct a non-autoregressive-based speech synthesis model (100) including.

First, the first network 110 will be described. Voice can be typically separated into the components of content, speaker (e.g., tone), rhythm, and pitch, and other information (e.g., energy, Background noise) are generally not distinct factors that determine the quality of voice synthesis, so we will focus on using the four voice components (content, speaker, rhythm, and pitch).

First network 110 may separate speech component(s) from spoken speech. The first network 110 can separate speech components from spoken speech without an explicit prosody label through an information bottleneck. The first network 110 may be composed of a first content encoder 132, a rhythm encoder 131, and a pitch encoder 134 for separating each voice component. At this time, since the speaker information has an explicit label, a separate speaker encoder is not configured. Each encoder can extract latent vectors for speech components from which other components have been removed through a carefully designed information bottleneck. The extracted latent vectors represent a content code, a rhythm code, a pitch code, and a speaker code, respectively, and will be used as input data for speech speech estimation in the decoder 140. You can.

Equation 1:

Equation 1 represents an equation for extracting the content code, rhythm code, and pitch code through the first content encoder 132, the rhythm encoder 131, and the pitch encoder 134. here, , and represents the content code, rhythm code, and pitch code, , and means the first content encoder 132, rhythm encoder 131, and pitch encoder 134. A stands for random sampling operation, S stands for speech spectrogram, and P stands for normalized pitch contour. The decoder 140 can estimate the speech spectrogram by receiving the decomposition code and speaker ID extracted from the encoder as input. In other words, for the rhythm encoder 131, a speech spectrogram may be input, and for the first content encoder 132 and the pitch encoder 134, randomly sampled speech spectrograms and pitch may be input.

The spoken voice S may be input to the content encoder 132 and the rhythm encoder 131, and the normalized pitch contour extracted from the spoken voice may be input to the pitch encoder 134. A random sampling process A may be performed on the inputs of the content encoder 132 and the pitch encoder 134 to intentionally damage rhythm information.

Voice segmentation can perform various voice conversions depending on the reference voice. The decoder 140 for voice segmentation can be expressed as follows.

Equation 2:

here, is the predicted speech spectrogram, D is the decoder, and U is the speaker ID. However, these models can be designed for speech conversion tasks. Equation 2 represents the process of estimating the spectrogram of the final spoken voice in the decoder (D) 140 by inputting voice components.

The second network 120 will now be described. In input text-speech pairs for training a non-autoregressive-based speech synthesis model, the text input contains only content information, so directly estimating speech in which various prosodic information is intertwined is limited. Additionally, most rhymes do not have explicit labels, making them more difficult. The speech synthesis system according to the embodiment can estimate the final spoken voice through a process of estimating decomposed speech components from text through unsupervised learning in the second network 120. The second network 120 may estimate speech components from text. The second network 120 may include a second content encoder 135 and a content encoder learned in the first network 110. The second content encoder 135 converts the text into linguistic features including content information, and encodes each text through a duration predictor and a length regulator for the converted linguistic features. A scalar value corresponding to the duration of a text token can be estimated, and the length of the linguistic feature corresponding to each text token can be adjusted using the estimated scalar value. The length of the linguistic feature converted through this process becomes the same as the length of the spectrogram of the speech voice to be estimated, and rhythm information is added to the content information. The converted linguistic features are input to the first content encoder 136 learned in the first network 110, and content information can be extracted by performing non-positional resampling. At this time, the first content encoder 136 learned in the first network 110 is copied to the second network 120 and the parameters are fixed. In the first network 110 and the second network 120, the content encoders 135 and 136 have different voice components included in the input, but the purpose of the encoder is to separate content information and residual information. The same result, the content code, can be extracted and used to estimate the final speech voice.

The objective function of the non-autoregressive-based speech synthesis model 100, which includes a first network 110 and a second network 120 for estimating speech components from text, will be described. The non-autoregressive-based speech synthesis model 100 estimates the spectrogram of the spoken voice through the content code extracted through the second network 120, and can be expressed as Equation 3 below.

Equation 3:

During training, the decoder is trained to reconstruct the final utterance (target) spectrogram, and the reconstruction loss can be expressed as Equation 4.

Equation 4:

In the non-autoregressive-based speech synthesis model 100, the duration predictor can be learned without an explicit label through length-level loss for the duration of each text token. The length-level loss can be expressed as Equation 5, and the sum of the scalar values output from the period predictor becomes the length of the final utterance spectrogram, leading to each scalar value becoming a period.

Equation 5:

Here, L represents the length of the final utterance (target) spectrogram, N represents the number of text tokens, and d _n represents the length of the nth token.

Therefore, the final loss can be expressed as Equation 6.

Equation 6:

Figure 2 is a diagram for explaining the detailed structure of a non-autoregressive-based speech synthesis model, according to one embodiment.

In the non-autoregressive synthetic model 100, the shadow block represents a parameter freeze block. For the pitch encoder, rhythm encoder and content encoder, the number of convolutional layers n ₁ is 3, 1 and 1 with dimensions 256, 128 and 512. The dimensions of group normalization are 16, 8, and 32. The number n ₂ of BLSTM (bidirectional LSTM) layers is 1, 1, 2 with dimensions 32, 1, and 8. The downsampling coefficient of all encoders is the same at 8.

The non-autoregressive-based speech synthesis model 100 may be composed of four learnable encoders and one decoder. All learnable encoders used in first network 110 and second network 120 share a similar structure consisting of a convolutional layer followed by a group normalization layer and a BLSTM layer. Except for the rhythm encoder, a random resampling operation is added to intentionally corrupt the rhythm information at the input stage. All outputs that pass through the convolutional layers are fed into the BLSTM layer stack and feature dimensionality is reduced. after that, Except, the three encoders can use an additional down-sampling operation to generate codes for the final speech components with reduced time dimension. In the case of , it can be directly fed to the period predictor without going through a down-sampling operation to estimate the period corresponding to each phoneme token. The period predictor consists of two one-dimensional convolutional layers, a regular layer and a linear layer. The output representation of is upsampled by the length adjuster based on the predicted period and copied from the first network 110. Is creates . The decoder 140 may first receive the speaker identification label and speech component code and restore the original sampling rate. All inputs can then be concatenated to produce the final output, the final utterance spectrogram, through a stack of BLSTM layers.

FIG. 3 is a block diagram for explaining the configuration of a voice synthesis system according to an embodiment, and FIG. 4 is a flowchart for explaining a voice synthesis method in the voice synthesis system according to an embodiment.

The processor of the voice synthesis system 300 may include a voice input unit 310 and a voice estimation unit 320. These processor components may be expressions of different functions performed by the processor according to control instructions provided by program codes stored in the speech synthesis system. The processor and its components can control the voice synthesis system to perform steps 410 to 420 included in the voice synthesis method of FIG. 4. At this time, the processor and its components may be implemented to execute instructions according to the code of an operating system included in the memory and the code of at least one program.

The processor may load the program code stored in the file of the program for the voice synthesis method into memory. For example, when a program is executed in a voice synthesis system, the processor can control the voice synthesis system to load program code from the program file into memory under the control of the operating system. At this time, each of the voice input unit 310 and the voice estimation unit 320 is a different functional expression of the processor for executing the subsequent steps 410 to 420 by executing the command of the corresponding portion of the program code loaded in the memory. You can.

In step 410, the voice input unit 310 receives the spoken voice as input to a non-autoregressive-based voice synthesis model consisting of a first network for separation of voice components and a second network for estimation of voice components from text. You can. For example, the voice input unit 310 can receive the entire spoken voice, and can receive the spoken voice of a preset section among all the spoken voices. Additionally, the spoken voice input to the voice input unit 310 may be composed of Korean or various foreign languages other than Korean.

In step 420, the voice estimation unit 320 may estimate the final spoken voice through voice synthesis of the spoken voice using the constructed non-autoregressive-based voice synthesis model. The voice estimation unit 320 may obtain a final speech spectrogram for the final speech voice.

Figure 5 is a diagram showing the results of a mel spectrogram generated using content information extracted from voice and content, respectively, in one embodiment.

In an embodiment, an experiment may be performed using the VCTK data set consisting of approximately 44 hours of speech collected from 109 speakers. For the experiment, the data set is divided into a training set of 40 hours and an evaluation set of 4 hours each. Training sets and evaluation sets can be divided into two types: visible sets and invisible sets. Visible data includes different utterances from the same speaker, and unseen data can only be included in the evaluation set.

The utterance spectrogram generated in all experiments can be converted to a waveform using a vocoder. Experiments can evaluate models in two aspects: synthesis quality and speed. Speech synthesis quality can be assessed using a subjective listening test involving 12 participants and expressed as a mean opinion score (MOS) from 1 to 5 with a 95% confidence interval. Two different speaker types, including visible and invisible speakers, can be set up for evaluation during training.

Table 1:

Table 1 shows the quality of synthesized speech. Table 1 shows the results of measuring the MOS score to evaluate the sound quality of the non-autoregressive speech synthesis model and baseline model proposed in the example. To compare the naturalness of the non-autoregressive-based speech synthesis model proposed in the embodiment, it can be compared with the ground truth mel-spectrogram (GT mel) and synthesized speech of FastSpeech2, the shortest parallel TTS model. Additionally, the content components generated in the first network may be compared with the synthesized speech of the SpeechSplit model to determine whether the content components separated in the second network are well estimated. The MOS score indicates that the non-autoregressive speech synthesis model proposed in the example has slightly better synthesis quality than FastSpeech2. In addition, the synthesis quality is similar to SpeechSplit, a conventional non-autoregressive speech synthesis model, indicating that content information decomposed into speech was successfully estimated from text through the first network. In other words, it can be confirmed that the non-autoregressive speech synthesis model proposed in the example is superior to FastSpeech2, which is a non-autoregressive speech synthesis model.

Table 2 shows the results showing the degree of improvement in synthesis speed compared to Tacotron2, a conventional autoregressive method.

Table 2:

To measure the improvement in inference efficiency, the time required to generate a mel spectrogram of speech can be compared for the proposed non-autoregressive speech synthesis model and Tacotron2, a parallel TTS model. As shown in Table 2, it can be seen that the non-autoregressive-based speech synthesis model proposed in the example shows slightly better performance than tacotron2 in terms of inference speed.

Figure 5 is a spectrogram of the final spoken voice estimated through speech component codes extracted from the first and second networks, and it can be seen that the spectrograms generated by the two networks are similar. The two Mel spectrograms are similar, indicating that the speech components of the first and second networks can replace each other.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), etc. , may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (13)

In a speech synthesis method performed by a speech synthesis system,
Receiving the spoken voice as input to a non-autoregressive-based speech synthesis model consisting of a first network for separating speech components and a second network for estimating speech components from text; and
Step of estimating the final spoken voice through speech synthesis of the spoken voice using the configured non-autoregressive speech synthesis model.
Including,
The non-autoregressive based speech synthesis model is,
The first network separates speech components from the spoken voice through unsupervised learning, the second network generates the separated speech components from text, and the final spoken voice is estimated through speech synthesis in a non-autoregressive manner. Configured to do so, the first network separates components for prosody and content, including the speaker, rhythm, and pitch, from the spoken voice through unsupervised learning, and Speech synthesis, characterized in that it is configured to estimate components for the separated content from the text in a second network and generate a plurality of spectrograms based on the parallel structure of the first network and the second network in a decoder. method. delete According to paragraph 1,
The first network is,
A voice synthesis method comprising a first content encoder, a rhythm encoder, and a pitch encoder for separating the voice components. According to paragraph 3,
In the first network, a content code, a speaker code, a rhythm code, and a pitch code are generated through a first content encoder, a rhythm encoder, and a pitch encoder included in the first network. A latent vector representing is extracted, and the extracted latent vector is used as input data of a decoder to estimate the spectrogram of the final spoken voice. According to paragraph 1,
The second network is,
A speech synthesis method comprising a second content encoder and a first content encoder learned from the first network. According to clause 5,
In the second network, text is converted into linguistic features including content information through the second content encoder, and each of the converted linguistic features is analyzed through a duration predictor and a length regulator. A voice characterized in that a scalar value corresponding to the duration of a text token is estimated, and the length of the linguistic feature corresponding to each text token is adjusted based on the estimated scalar value. Synthesis method. In a speech synthesis method performed by a speech synthesis system,
Receiving the spoken voice as input to a non-autoregressive-based speech synthesis model consisting of a first network for separating speech components and a second network for estimating speech components from text; and
Step of estimating the final spoken voice through speech synthesis of the spoken voice using the configured non-autoregressive speech synthesis model.
Including,
The non-autoregressive based speech synthesis model is,
The first network separates speech components from the spoken voice through unsupervised learning, the second network generates the separated speech components from text, and the final spoken voice is estimated through speech synthesis in a non-autoregressive manner. It is structured so that
The second network is,
Comprising a second content encoder and a first content encoder learned in the first network, converted from text in the second network into linguistic features including content information through the second content encoder, and the converted language For enemy features, a scalar value corresponding to the duration of each text token is estimated through a duration predictor and a length regulator, and based on the estimated scalar value. The length of the linguistic feature corresponding to each text token is adjusted, and random resampling is performed as the converted linguistic feature in the second network is input to the first content encoder learned in the first network. A voice synthesis method characterized in that content information is extracted through. In clause 7,
The content encoder learned in the first network extracts a content code,
A voice synthesis method characterized in that the spectrogram of the spoken voice is estimated through the extracted content code through the second network. According to paragraph 1,
The decoder is,
A voice synthesis method characterized by learning to reconstruct the spectrogram of the final spoken voice. According to clause 6 or 7,
The period predictor is,
A speech synthesis method characterized by learning without explicit labels through length-level loss for the period of each text token. According to clause 10,
The loss of the length level is,
A speech synthesis method, characterized in that the sum of the scalar values output from the period predictor is derived to be the length of the final speech spectrogram. A computer program stored in a non-transitory computer-readable recording medium for executing the speech synthesis method of any one of claims 1, 3 to 9 on the speech synthesis system. In the voice synthesis system,
A voice input unit that receives the spoken voice as input to a non-autoregressive-based voice synthesis model consisting of a first network for separating voice components from voice and a second network for estimation of voice components from text; and
A voice estimation unit that estimates the final spoken voice through voice synthesis of the spoken voice using the non-autoregressive voice synthesis model constructed above.
Including,
The non-autoregressive based speech synthesis model is,
The first network separates speech components from the spoken voice through unsupervised learning, the second network generates the separated speech components from text, and the final spoken voice is estimated through speech synthesis in a non-autoregressive manner. Configured to do so, the first network separates components for prosody and content, including the speaker, rhythm, and pitch, from the spoken voice through unsupervised learning, and Speech synthesis, characterized in that it is configured to estimate components for the separated content from the text in a second network and generate a plurality of spectrograms based on the parallel structure of the first network and the second network in a decoder. system.

Images