KR102532253B1 - A method and a TTS system for calculating a decoder score of an attention alignment corresponded to a spectrogram - Google Patents

Abstract

A method of calculating a score of attention alignment corresponding to a spectrogram according to an aspect includes m values among values in an s step constituting a second axis on which the attention alignment is expressed. extracting a value; and calculating an s score in the s th step using the extracted m values, wherein m and s each include a natural number of 1 or more, and the second axis represents the spectrogram Corresponds to the time step of an encoder included in the synthesizer to generate, and the spectrogram corresponds to a verbal utterance of a sequence of characters in a specific natural language.

Description

A method and a speech synthesis system for calculating a decoder score of an attention alignment corresponding to a spectrogram {A method and a TTS system for calculating a decoder score of an attention alignment corresponded to a spectrogram}

A method for calculating a decoder score of attention alignment corresponding to a spectrogram and a speech synthesis system.

Recently, with the development of artificial intelligence technology, an interface using a voice signal is becoming common. Accordingly, research on a speech synthesis technology capable of uttering a synthesized voice according to a given situation is being actively conducted.

Speech synthesis technology is applied to many fields, such as virtual assistants, audio books, automatic interpretation and translation, and virtual voice actors, in conjunction with artificial intelligence-based voice recognition technology.

Conventional speech synthesis methods include various methods such as Unit Selection Synthesis (USS) and HMM-based Speech Synthesis (HTS). The USS method is a method of cutting and storing voice data in phoneme units and searching for and attaching sound pieces suitable for speech during speech synthesis. way to reconstruct it. However, the conventional voice synthesis method described above has many limitations in synthesizing a natural voice reflecting a speaker's speech style or emotional expression.

Accordingly, recently, a speech synthesis method for synthesizing speech from text based on an artificial neural network is attracting attention.

An object of the present invention is to provide a method for calculating a decoder score of attention alignment corresponding to a spectrogram and a speech synthesis system. In addition, it is an object of the present invention to provide an artificial intelligence-based voice synthesis technology capable of realizing a natural voice as if a real speaker is speaking. In addition, it is to provide a high-efficiency artificial intelligence-based voice synthesis technology using a small amount of learning data.

The technical problem to be solved is not limited to the technical problems as described above, and other technical problems can be inferred.

A method of calculating a score of attention alignment corresponding to a spectrogram according to an aspect includes m values among values in an s step constituting a second axis on which the attention alignment is expressed. extracting a value; and calculating an s score in the s th step using the extracted m values, wherein m and s each include a natural number of 1 or more, and the second axis represents the spectrogram Corresponds to the time step of an encoder included in the synthesizer to generate, and the spectrogram corresponds to a verbal utterance of a sequence of characters in a specific natural language.

In the method described above, in the extracting step, the top m values are extracted from among the values in the s th step, and m includes a natural number of 2 or more.

In the above method, the last value of s means a step corresponding to the end of the spectrogram.

The method described above further includes evaluating the quality of the spectrogram by comparing the calculated s score with a predetermined value.

In the above method, the steps constituting the second axis each correspond to a single phoneme constituting the input text.

A computer-readable recording medium according to another aspect includes a program for executing the above-described method on a computer.

A speech synthesis system according to another aspect includes a speaker encoder; synthesizer; and a vocoder, wherein the synthesizer extracts m values from among values at an s step constituting a second axis on which an attention alignment corresponding to the spectrogram is expressed, and the extraction An encoder included in a synthesizer that calculates a s score in the s th step using the m values, wherein m and s each include a natural number of 1 or more, and the second axis generates the spectrogram ( encoder, and the spectrogram corresponds to a verbal utterance of a sequence of characters in a particular natural language.

In the system described above, the combiner extracts top m values from among the values in the s th step, where m includes a natural number of 2 or more.

In the above system, the last value of s means a step corresponding to the end of the spectrogram.

In the system described above, the synthesizer evaluates the quality of the spectrogram by comparing the calculated s-th score with a predetermined value.

In the above system, the steps constituting the second axis each correspond to a single phoneme constituting the input text.

As the speech synthesis system calculates attention alignment scores (encoder score, decoder score, and final score), the quality of the Mel spectrogram corresponding to the attention alignment may be determined. Accordingly, the speech synthesis system may select a mel spectrogram of the highest quality from among a plurality of mel spectrograms. Accordingly, the speech synthesis system can output synthesized speech of the highest quality.

1 is a diagram schematically illustrating the operation of a speech synthesis system.
2 is a diagram illustrating an embodiment of a speech synthesis system.
3 is a diagram illustrating an embodiment of outputting a Mel spectrogram through a synthesizer.
4A and 4B are diagrams illustrating an example of a Mel spectrogram and attention alignment.
5A and 5B are diagrams for explaining the quality of attention alignment.
6 is a diagram for describing coordinate axes representing attention alignment and quality of attention alignment.
7 is a diagram for explaining an example in which a synthesizer calculates an encoder score.
8 is a diagram for explaining an example in which a synthesizer calculates a decoder score.
9 is a diagram for explaining an example of extracting a part having a valid meaning from attention alignment.
10A to 10C are diagrams for explaining the relationship between the quality of attention alignment and an encoder score and a decoder score.
11 is a flowchart illustrating an example of a method of calculating a decoder score of attention alignment.

The terms used in the present embodiments have been selected from general terms that are currently widely used as much as possible while considering the functions in the present embodiments, but these may vary depending on the intention of a person skilled in the art or a precedent, the emergence of new technologies, etc. there is. In addition, in a specific case, there are also terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the relevant part. Therefore, the term used in the present embodiments should be defined based on the meaning of the term and the general content of the present embodiment, not a simple name of the term.

Since the present embodiments can have various changes and various forms, some embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the present embodiments to a specific disclosure, and should be understood to include all changes, equivalents, or substitutes included in the spirit and scope of the present embodiments. Terms used in this specification are only used for description of the embodiments, and are not intended to limit the embodiments.

Terms used in the present embodiments have the same meaning as commonly understood by a person of ordinary skill in the art to which the present embodiments belong, unless otherwise defined. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present embodiments, in an ideal or excessively formal meaning. should not be interpreted.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The detailed description of the present invention which follows refers to the accompanying drawings which illustrate, by way of illustration, specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable any person skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different from each other but are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented from one embodiment to another without departing from the spirit and scope of the present invention. It should also be understood that the location or arrangement of individual components within each embodiment may be changed without departing from the spirit and scope of the present invention. Therefore, the detailed description to be described later is not performed in a limiting sense, and the scope of the present invention should be taken as encompassing the scope claimed by the claims and all scopes equivalent thereto. Like reference numbers in the drawings indicate the same or similar elements throughout the various aspects.

Meanwhile, technical features individually described in one drawing in this specification may be implemented individually or simultaneously.

In this specification, “~ unit” may be a hardware component such as a processor or a circuit, and/or a software component executed by the hardware component such as a processor.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily practice the present invention.

1 is a diagram schematically illustrating the operation of a speech synthesis system.

A speech synthesis device is a device that artificially converts text into human voice.

For example, the voice synthesizing system 100 of FIG. 1 may be an artificial neural network based voice synthesizing system. An artificial neural network refers to a model in general that has problem-solving ability by changing synaptic coupling strength through learning of artificial neurons that form a network through synaptic coupling.

The voice synthesis system 100 may be implemented in various types of devices such as a personal computer (PC), a server device, a mobile device, and an embedded device, and as a specific example, a smartphone or a tablet that performs voice synthesis using an artificial neural network. Devices, Augmented Reality (AR) devices, Internet of Things (IoT) devices, self-driving cars, robotics, medical devices, e-book readers, and navigation devices may be used, but are not limited thereto.

Furthermore, the voice synthesis system 100 may correspond to a dedicated hardware accelerator (HW accelerator) mounted in the above device. Alternatively, the speech synthesis system 100 may be a hardware accelerator such as a neural processing unit (NPU), a tensor processing unit (TPU), or a neural engine, which are dedicated modules for driving an artificial neural network, but is not limited thereto.

Referring to FIG. 1 , the speech synthesis system 100 may receive text input and specific speaker information. For example, the speech synthesis system 100 may receive “Have a good day!” as a text input as shown in FIG. 1 and may receive “speaker 1” as a speaker information input.

“Speaker 1” may correspond to a voice signal or a voice sample representing a preset speech characteristic of speaker 1. For example, speaker information may be received from an external device through a communication unit included in the voice synthesis system 100 . Alternatively, speaker information may be input from a user through the user interface of the speech synthesis system 100 and may be selected from among various types of speaker information pre-stored in the database of the speech synthesis system 100, but is not limited thereto. no.

The speech synthesis system 100 may output speech based on the received text input and specific speaker information. For example, the speech synthesis system 100 may “Have a good day!” and “Speaker 1” are received as an input, and a voice for “Have a good day!” reflecting the speech characteristics of Speaker 1 may be output. The speech characteristics of speaker 1 may include at least one of various factors such as voice, prosody, pitch, and emotion of speaker 1. That is, the output voice may be a voice as if speaker 1 naturally pronounces “Have a good day!” A detailed operation of the voice synthesis system 100 will be described later with reference to FIGS. 2 to 4 .

2 is a diagram illustrating an embodiment of a speech synthesis system. The voice synthesis system 200 of FIG. 2 may be the same as the voice synthesis system 100 of FIG. 1 .

Referring to FIG. 2 , a speech synthesis system 200 may include a speaker encoder 210 , a synthesizer 220 and a vocoder 230 . Meanwhile, in the voice synthesizing system 200 shown in FIG. 2, only components related to one embodiment are shown. Accordingly, it is obvious to those skilled in the art that the voice synthesis system 200 may further include other general-purpose components in addition to the components shown in FIG. 2 .

The voice synthesis system 200 of FIG. 2 may output speech by receiving speaker information and text as inputs.

For example, the speaker encoder 210 of the speech synthesis system 200 may receive speaker information as an input and generate a speaker embedding vector. Speaker information may correspond to a speaker's voice signal or voice sample. The speaker encoder 210 may receive the speaker's voice signal or voice sample, extract the speaker's speech characteristics, and represent the speech characteristics as an embedding vector.

The speaker's speech characteristics may include at least one of various factors such as speech speed, pause period, pitch, timbre, prosody, intonation, or emotion. That is, the speaker encoder 210 may represent discontinuous data values included in speaker information as vectors composed of consecutive numbers. For example, the speaker encoder 210 includes a pre-net, a CBHG module, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), and a BRDNN ( A speaker embedding vector may be generated based on at least one or a combination of two or more of various artificial neural network models such as Bidirectional Recurrent Deep Neural Network).

For example, the synthesizer 220 of the speech synthesis system 200 may output speech data by receiving text and an embedding vector representing speech characteristics of a speaker as inputs.

For example, synthesizer 220 may include a text encoder (not shown) and a decoder (not shown). Meanwhile, it is apparent to those skilled in the art that the synthesizer 220 may further include other general-purpose components in addition to the above-described components.

An embedding vector representing speech characteristics of a speaker may be generated from the speaker encoder 210 as described above, and a text encoder (not shown) or a decoder (not shown) of the synthesizer 220 may generate the speaker's character from the speaker encoder 210. Embedding vectors representing speech characteristics may be received.

A text encoder (not shown) of synthesizer 220 may receive text as an input and generate a text embedding vector. Text may contain sequences of characters in a particular natural language. For example, a sequence of characters may include alphabetic characters, numbers, punctuation marks, or other special characters.

A text encoder (not shown) may divide input text into consonant units, character units, or phoneme units, and input the separated text into an artificial neural network model. For example, a text encoder (not shown) may generate a text embedding vector based on at least one or a combination of two or more of various artificial neural network models such as pre-net, CBHG module, DNN, CNN, RNN, LSTM, and BRDNN. .

Alternatively, the text encoder (not shown) may divide the input text into a plurality of short texts and generate a plurality of text embedding vectors for each of the short texts.

A decoder (not shown) of the synthesizer 220 may receive the speaker embedding vector and the text embedding vector from the speaker encoder 210 as inputs. Alternatively, the decoder (not shown) of the synthesizer 220 may receive a speaker embedding vector from the speaker encoder 210 as an input and a text embedding vector from a text encoder (not shown) as an input.

A decoder (not shown) may generate voice data corresponding to the input text by inputting the speaker embedding vector and the text embedding vector to the artificial neural network model. That is, a decoder (not shown) may generate voice data for input text in which speech characteristics of a speaker are reflected. For example, the voice data may correspond to a spectrogram or a mel-spectrogram corresponding to input text, but is not limited thereto. In other words, a spectrogram or mel spectrogram corresponds to a verbal utterance of a sequence of characters composed of a particular natural language.

A spectrogram is a graph that visualizes the spectrum of a voice signal. The x-axis of the spectrogram represents time, and the y-axis represents frequency, and the value of each frequency per time can be expressed in color according to the size of the value. The spectogram may be a result of performing short-time Fourier transform (STFT) on a continuously given audio signal.

STFT is a method of dividing a speech signal into sections of a certain length and applying a Fourier transform to each section. At this time, since the result of performing the STFT on the voice signal is a complex value, it is possible to generate a spectrogram including only magnitude information by taking an absolute value of the complex value to lose phase information.

Meanwhile, the mel spectrogram is obtained by re-adjusting the frequency interval of the spectrogram to a mel scale. The human auditory organ is more sensitive in the low frequency band than the high frequency band, and the Mel scale reflects this characteristic and expresses the relationship between the physical frequency and the frequency perceived by the actual person. The Mel spectrogram may be generated by applying a filter bank based on the Mel scale to the spectrogram.

Meanwhile, although not shown in FIG. 2 , the synthesizer 220 may further include an attention module for generating an attention alignment. The attention module is a module that learns which output of a specific time-step of a decoder (not shown) is most related to an output of all time-steps of a text encoder (not shown). A higher quality spectrogram or MEL spectrogram can be output by using the attention module.

3 is a diagram illustrating an embodiment of outputting a Mel spectrogram through a synthesizer. The synthesizer 300 of FIG. 3 may be the same as the synthesizer 220 of FIG. 2 .

Referring to FIG. 3 , the synthesizer 300 may receive a list including input texts and speaker embedding vectors corresponding thereto. For example, the synthesizer 300 inputs the input text 'first sentence', the corresponding speaker embedding vector embed_voice1, the input text 'second sentence' and the corresponding speaker embedding vector embed_voice2, and the corresponding input text 'third sentence' and a list 310 including embed_voice3, which is a speaker embedding vector corresponding thereto, may be received as an input.

The combiner 300 may generate as many mel spectrograms 310 as the number of input texts included in the received list 310 . Referring to FIG. 3 , it can be seen that Mel spectrograms corresponding to each input text of 'first sentence', 'second sentence', and 'third sentence' are generated.

Alternatively, the synthesizer 300 may generate as many mel spectrograms 320 and attention alignment as the number of input texts. Although not shown in FIG. 3 , for example, attention alignment corresponding to input texts of 'first sentence', 'second sentence', and 'third sentence' may be additionally generated. Alternatively, the synthesizer 300 may generate a plurality of MEL spectrograms and a plurality of attention alignments for each of the input texts.

Referring back to FIG. 2 , the vocoder 230 of the voice synthesis system 200 may generate voice data output from the synthesizer 220 into actual voice. Voice data output as described above may be a spectrogram or a mel spectrogram.

For example, the vocoder 230 may generate an actual voice signal from voice data output from the synthesizer 220 using Inverse Short-Time Fourier Transform (ISTFT). However, since the spectrogram or MEL spectrogram does not contain phase information, the ISTFT alone cannot perfectly restore the actual speech signal.

Accordingly, the vocoder 230 may generate voice data output from the synthesizer 220 as an actual voice signal using, for example, a Griffin-Lim algorithm. The Griffin-Rim algorithm is an algorithm for estimating phase information from magnitude information of a spectrogram or a Mel spectrogram.

Alternatively, the vocoder 230 may generate an actual voice signal from voice data output from the synthesizer 220 based on, for example, a neural vocoder.

A neural vocoder is an artificial neural network model that generates a voice signal by receiving a spectrogram or a mel spectrogram as an input. The neural vocoder can learn a relationship between a spectrogram or a MEL spectrogram and a voice signal through a large amount of data, and through this, it can generate a real voice signal of high quality.

The neural vocoder may correspond to a vocoder based on an artificial neural network model such as WaveNet, Parallel WaveNet, WaveRNN, WaveGlow, or MelGAN, but is not limited thereto.

For example, a WaveNet vocoder is composed of several dilated causal convolution layers and is an autoregressive model using sequential features between voice samples. For example, the WaveRNN vocoder is an autoregressive model in which several dilated causal convolution layers of WaveNet are replaced with GRU (Gated Recurrent Unit).

For example, the WaveGlow vocoder can learn to produce a simple distribution such as a Gaussian distribution from a voice dataset (x) using an invertible transform function. After learning, the WaveGlow vocoder can output a voice signal from Gaussian distribution samples using the inverse function of the conversion function.

As described above with reference to FIGS. 2 and 3 , the synthesizers 220 and 300 may generate attention alignment. Specifically, the attention alignment may be generated corresponding to the spectrogram (or MEL spectrogram). For example, when the synthesizers 220 and 300 generate a total of x spectrograms (or MEL spectrograms), attention alignment may be generated corresponding to each of the x spectrograms. Accordingly, the quality of the corresponding spectrogram (or MEL spectrogram) may be determined through the attention alignment.

The synthesizers 220 and 300 according to an embodiment may generate a plurality of spectrograms (or mel spectrograms) with respect to a single input pair composed of an input text and a speaker embedding vector. Also, the synthesizers 220 and 300 may calculate an attention alignment score corresponding to each of a plurality of spectrograms (or MEL spectrograms). Accordingly, the synthesizers 220 and 300 may select one of a plurality of spectrograms (or MEL spectrograms) based on the calculated score. Here, the selected spectrogram (or mel spectrogram) may represent the highest quality synthesized speech for a single input pair.

Hereinafter, examples in which the synthesizers 220 and 300 calculate an attention alignment score will be described with reference to FIGS. 4 to 11 . Hereinafter, although the synthesizers 220 and 300 are described as calculating the attention alignment score, the module that calculates the attention alignment score may not be the synthesizer 220 and 300. For example, the attention alignment score may be calculated by a separate module included in the speech synthesis system 200 or another module separate from the speech synthesis system 200 .

In addition, in the following, a spectrogram and a mel spectrogram are described as interchangeable terms. In other words, although described as a spectrogram below, it may be replaced with a Mel spectrogram. In addition, although it is described as a Mel spectrogram in the following, it may be replaced with a spectrogram.

4A and 4B are diagrams illustrating an example of a Mel spectrogram and attention alignment.

4A shows an example of a Mel spectrogram generated by the synthesizers 220 and 300 according to a certain input pair (input text and speaker embedding vector). In addition, FIG. 4B shows attention alignment corresponding to the MEL spectrogram of FIG. 4A.

For example, if the amount of data is not large or sufficient learning is not performed, the synthesizers 220 and 300 may not be able to generate high-quality Mel spectrograms. Attention alignment can be interpreted as a history of every moment that the synthesizers 220 and 300 focus on when generating the MEL spectrogram.

For example, if the line representing the attention alignment is thick and the noise is small, it can be interpreted that the synthesizers 220 and 300 confidently performed inference at every moment of generating the Mel spectrogram. That is, in the case of the above example, it can be determined that the synthesizers 220 and 300 have generated a high-quality Mel spectrogram. Therefore, the quality of attention alignment (for example, the degree of darkness of the attention alignment, the degree of clarity of the attention alignment, etc.) can be used as a very important indicator in estimating the inference quality of the synthesizers 220 and 300. there is.

5A and 5B are diagrams for explaining the quality of attention alignment.

5A to 5D show attention alignments corresponding to the same input pair (input text and speaker embedding vector).

Referring to FIG. 5A , an example in which the middle portion 510 of the attention alignment is not generated is illustrated. That is, according to the attention alignment of FIG. 5A , the quality of the corresponding MEL spectrogram may be interpreted as being low.

Referring to FIG. 5B , compared to the attention alignment of FIG. 5A , attention alignment of relatively better quality is shown. That is, the Mel spectrogram corresponding to the attention alignment of FIG. 5B may be interpreted as having higher quality than the Mel spectrogram corresponding to the attention alignment of FIG. 5A. However, even in the case of the attention alignment of FIG. 5B, since an unclear portion is included in the middle portion 520, it can be interpreted that the quality of the Mel spectrogram is not very high.

When the attention alignments shown in FIGS. 5A and 5B are generated, it should be determined that the quality of the corresponding MEL spectrogram is not high. The synthesizers 220 and 300 according to an embodiment determine the quality of the attention alignment based on the score of the attention alignment. That is, the synthesizers 220 and 300 may determine the quality of the MEL spectrogram according to the attention alignment score.

For example, the synthesizers 220 and 300 may calculate an encoder score and a decoder score of attention alignment. Then, the synthesizers 220 and 300 may calculate a total score of attention alignment by combining the encoder score and the decoder score.

The quality of attention alignment may be determined by any one of an encoder score, a decoder score, and a total score. Accordingly, the synthesizers 220 and 300 may calculate any one of an encoder score, a decoder score, and a total score as needed.

6 is a diagram for describing coordinate axes representing attention alignment and quality of attention alignment.

Referring to FIG. 6 , attention alignment is shown in two-dimensional coordinates. At this time, the horizontal axis of the two-dimensional coordinates means a decoder time step, and the vertical axis means an encoder time step. That is, the two-dimensional coordinates representing the attention alignment mean which part should be focused when the synthesizers 220 and 300 generate the MEL spectrogram.

The decoder time step means the time invested by the synthesizers 220 and 300 to utter each phoneme included in the input text. The decoder time steps are arranged in time intervals corresponding to a single hop size, and a single hop size means 1/80 second.

Encoder time steps correspond to phonemes included in the input text. For example, if the input text is 'first sentence', the encoder time steps are 'f', 'i', 'r', 's', 't', '', 's', 'e', ' It may consist of n', 't', 'e', 'n', 'c', and 'e'.

Referring to FIG. 6 , each of the points constituting the attention alignment is represented by a specific color. Here, the color may be matched with a specific value corresponding thereto. For example, each of the colors constituting the attention alignment is a value representing a probability distribution, and may be a value between 0 and 1.

7 is a diagram for explaining an example in which a synthesizer calculates an encoder score.

Referring to FIG. 7 , values 710 corresponding to '50' of the decoder time step in the attention alignment are displayed. Since Attention Alignment records and transposes each softmax result value, the sum of all values corresponding to a single step constituting the decoder time step is 1. That is, the sum of all the values 710 of FIG. 7 becomes 1.

On the other hand, referring to the upper a number of values 720 among the values 710, it is determined which phoneme the synthesizers 220 and 300 focus on to generate the mel spectrogram at the time point corresponding to '50' of the decoder time step. can judge Accordingly, the synthesizers 220 and 300 calculate encoder scores for each of the steps constituting the decoder time step to determine whether the mel spectrogram appropriately represents the input text (ie, the quality of the mel spectrogram). can

For example, the synthesizers 220 and 300 may calculate an encoder score based on Equation 1 below.

는 어텐션 얼라인먼트(align_decoder)에서 디코더 타임 스텝을 기준으로 s 번째 스텝의 i번째 상위 값을 나타낸다(s 및 i는 1 이상의 자연수).In Equation 1,

represents the i-th upper value of the s-th step based on the decoder time step in the attention alignment (align _decoder ) (s and i are natural numbers greater than or equal to 1).

That is, the synthesizers 220 and 300 extract n values from among the values in the s-th step of the decoder type step (n is a natural number greater than or equal to 2). Here, the n values may mean top n values in the s th step.

)를 연산한다. 예를 들어, 합성기(220, 300)는 추출된 n 개의 값들을 더하여 제s 스코어(

)를 연산할 수 있다.Then, the synthesizers 220 and 300 use the extracted n values to obtain the s score in the s th step (

) is computed. For example, the synthesizers 220 and 300 add the extracted n values to the s score (

) can be computed.

In this way, the synthesizers 220 and 300 respectively compute encoder scores from the step corresponding to the beginning of the spectrogram to the step corresponding to the end in the decoder time step. Then, the synthesizers 220 and 300 may evaluate the quality of the mel spectrogram by comparing the calculated encoder score with a predetermined value. An example in which the synthesizers 220 and 300 evaluate the quality of the MEL spectrogram based on the encoder score will be described later with reference to FIG. 10 .

8 is a diagram for explaining an example in which a synthesizer calculates a decoder score.

Referring to FIG. 8 , values 810 corresponding to '10' of the encoder time step in the attention alignment are displayed. Also, among the values 810, top b values 820 are displayed.

As described above with reference to FIG. 7, the encoder score is calculated with values at each step constituting the decoder time step. On the other hand, the decoder score is calculated with the values at each step constituting the decoder time step. Encoder scores and decoder scores have different purposes. Specifically, the encoder score is an index for determining whether the attention module has well determined the phoneme to be focused on every hour. On the other hand, the decoder score is an indicator for determining whether the attention module has focused well on specific phonemes constituting the input text without missing time allocation.

는 어텐션 얼라인먼트(align_encoder)에서 인코더 타임 스텝을 기준으로 s번째 스텝의 i번째 상위 값을 나타낸다(s 및 i는 1 이상의 자연수).In Equation 2,

represents the i-th upper value of the s-th step based on the encoder time step in the attention alignment (align _encoder ) (s and i are natural numbers greater than or equal to 1).

That is, the synthesizers 220 and 300 extract m values from values in the s th step of the encoder type step (m is a natural number greater than or equal to 2). Here, m values may mean top m values in the s th step.

)를 연산한다. 예를 들어, 합성기(220, 300)는 추출된 m 개의 값들을 더하여 제s 스코어(

)를 연산할 수 있다.Then, the synthesizers 220 and 300 use the extracted m values to score the s in the s step (

) is computed. For example, the synthesizers 220 and 300 add the extracted m values to the s score (

) can be computed.

In this way, the synthesizers 220 and 300 respectively compute decoder scores from the step corresponding to the beginning of the spectrogram to the step corresponding to the end in the encoder time step. Then, the synthesizers 220 and 300 may evaluate the quality of the mel spectrogram by comparing the calculated decoder score with a predetermined value. An example in which the synthesizers 220 and 300 evaluate the quality of the mel spectrogram based on the decoder score will be described later with reference to FIG. 10 .

That is, the decoder score is calculated as a value obtained by adding the upper m number of encoder time steps in the attention alignment. This may be an indicator of how much energy the speech synthesis system uses to speak each phoneme constituting the input text.

9 is a diagram for explaining an example of extracting a part having a valid meaning from attention alignment.

The length of the decoder time step is equal to the length of the mel spectrum. Accordingly, a part having a valid meaning in the attention alignment corresponds to the length of the Mel spectrum.

Meanwhile, the encoder time step is the length of phonemes constituting the input text. Therefore, the part having a valid meaning in the attention alignment corresponds to the length corresponding to the result of decomposing the text into phonemes.

10A to 10C are diagrams for explaining the relationship between the quality of attention alignment and an encoder score and a decoder score.

FIG. 10A means attention alignment, FIG. 10B means attention alignment encoder score of FIG. 10A, and FIG. 10C means attention alignment decoder score of FIG. 10A.

Referring to FIG. 10A , it can be seen that the quality of the attention alignment is low in the first portion 1010 and the second portion 1020 . Specifically, the first part (1010) appears to be unfocused on a specific phoneme included in the input text, and the second part (1020) appears to be unfocused on any phoneme at a specific point in time when the MEL spectrogram is generated. appear.

Meanwhile, referring to FIG. 10B , it can be seen that the encoder score 1030 corresponding to the second part 1020 is calculated low. Also, referring to FIG. 10C , it can be seen that the decoder score 1040 corresponding to the first part 1010 is calculated low. That is, the synthesizers 220 and 300 may evaluate the quality of the mel spectrogram by comparing the encoder score or the decoder score with a predetermined value (threshold value).

Meanwhile, the synthesizers 220 and 300 may evaluate the quality of the mel spectrogram by combining the encoder score and the decoder score.

)를 연산할 수 있다.For example, the synthesizers 220 and 300 modify the encoder score of Equation 1 according to Equation 3 below, so that the final encoder score (

) can be computed.

은 멜 스펙트로그램의 길이(frame length)이고, s는 디코더 타임 스텝을 의미한다. 수학식 3을 구성하는 다른 변수들은 수학식 1에서 설명한 바와 동일하다.In Equation 3,

is the length of the mel spectrogram (frame length), and s means the decoder time step. Other variables constituting Equation 3 are the same as those described in Equation 1.

)를 연산할 수 있다.In addition, the synthesizers 220 and 300 modify the encoder score of Equation 2 according to Equation 4 below, so that the final decoder score (

) can be computed.

은 집합 x를 구성하는 값들 중에서 y번째로 작은 값(즉, 하위 y 번째 값)을 의미하고,

은 인코더 타임 스텝을 의미한다.

은 디코더 스코어의 길이를 의미하여, 하위

번째의 값까지 모두 더한 값이 된다. In Equation 4,

Means the y-th smallest value (ie, the lower y-th value) among the values constituting the set x,

denotes the encoder time step.

means the length of the decoder score,

It is the sum of all values up to the second value.

)를 연산할 수 있다.Then, the synthesizers 220 and 300 score the final score according to Equation 5 below (

) can be computed.

In Equation 5, 0.1 means a weight, and the value of the weight can be changed as needed.

As described above with reference to FIGS. 4 to 10 , as the synthesizers 220 and 300 calculate the attention alignment scores (encoder score, decoder score, and final score), the quality of the Mel spectrogram corresponding to the attention alignment increases. can be judged Accordingly, the speech synthesis system 100 or 200 may select a mel spectrogram of the highest quality from among a plurality of mel spectrograms. Accordingly, the voice synthesizing systems 100 and 200 can output synthesized voices of the highest quality.

11 is a flowchart illustrating an example of a method of calculating a decoder score of attention alignment.

Referring to FIG. 11 , a method for calculating a decoder score is composed of steps processed time-sequentially in the voice synthesis systems 100 and 200 or synthesizers 220 and 300 shown in FIGS. 1 to 3 . Therefore, even if the content is omitted below, the information described above regarding the speech synthesis system 100, 200 or synthesizer 220, 300 shown in FIGS. can be applied.

In step 1110, the synthesizers 220 and 300 extract m values from among the values in the s-th step constituting the first axis on which the alignment is expressed. Here, m and s each represent a natural number of 1 or greater. Also, the last value of s means a step corresponding to the end of the spectrogram. The first axis is an encoder time step, and the encoder time step means a time step of a decoder included in the synthesizers 220 and 300 that generate the spectrogram. Also, a spectrogram corresponds to a verbal utterance of a sequence of characters composed of a particular natural language.

In step 1120, the synthesizers 220 and 300 calculate the s score in the s th step using the extracted m values. For example, the synthesizers 220 and 300 may extract top m values from among the values in the s th step, where m is a natural number of 2 or greater.

Also, although not shown in FIG. 11 , the synthesizers 220 and 300 may evaluate the quality of the spectrogram by comparing the s score with a predetermined value.

The above description of this specification is for illustrative purposes, and those skilled in the art will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. You will be able to. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present embodiment is indicated by the appended claims rather than the detailed description above, and should be construed as including all changes or modifications derived from the meaning and scope of the claims and equivalent concepts thereof.

200: speech synthesis system
210: speaker encoder
220: synthesizer
230: Vocoder

Claims (5)

A method performed by a synthesizer included in a speech synthesis system that calculates a score of attention alignment corresponding to a spectrogram to output a synthesized speech of the highest quality, comprising:
calculating an encoder score;
calculating a decoder score;
calculating a final score corresponding to the attention alignment based on the encoder score and the decoder score; and
Evaluating the quality of the spectrogram by comparing the final score with a predetermined value; including,
Calculating the encoder score,
extracting n values from values in an s step constituting a first axis on which the attention alignment is expressed; and
Determining an s-th encoder score in the s-th step using the extracted n values;
Calculating the decoder score,
extracting m values from values in an s step constituting a second axis in which the attention alignment is expressed; and
Computing the s-th decoder score in the s-th step using the extracted m values;
Each of n, m, and s includes a natural number of 1 or more,
The first axis corresponds to a time step of a decoder included in a synthesizer generating the spectrogram,
The second axis corresponds to a time step of an encoder included in a synthesizer generating the spectrogram,
The spectrogram is generated by a speaker embedding vector and a text embedding vector, and corresponds to a verbal utterance of a sequence of characters composed of a specific natural language. According to claim 1,
In the step of extracting the n values,
Extracting the top n values among the values in the s th step;
The step of extracting the m values,
Extracting the top m values among the values in the s th step;
Wherein n and m comprise a natural number greater than or equal to 2. According to claim 1,
The last value of s means a step corresponding to the end of the spectrogram. delete According to claim 1,
The steps constituting the second axis each correspond to a single phoneme constituting the input text.

Images