KR102358692B1 - Method and tts system for changing the speed and the pitch of the speech - Google Patents

Abstract

Provided is a method for synthesizing a voice that changes in speed and pitch. The method of the present invention performs a short-time Fourier transform on a first voice signal based on a first hop length and a first window length to generate a spectrogram, and generates voice signals of sections having a second window length at a second hop length interval from the spectrogram. Here, a ratio of the first hop length to the second hop length is equal to a value of a double speed rate, and a ratio of the first window length to the second window length may be set to be equal to a value of a pitch change rate to generate the second voice signal for which the speed and the pitch are changed. Therefore, the present invention is capable of providing a high-quality voice.

Description

Method and speech synthesis system for changing the speed and pitch of speech

The present disclosure relates to a method and a speech synthesis system for changing the speed and pitch of speech.

Recently, with the development of artificial intelligence technology, interfaces using voice signals are becoming common. Accordingly, research on a speech synthesis technology capable of uttering a synthesized voice according to a given situation is being actively conducted.

Speech synthesis technology is applied to many fields such as virtual assistants, audiobooks, automatic interpretation and translation, and virtual voice actors by combining speech recognition technology based on artificial intelligence.

As a conventional speech synthesis method, there are various methods such as unit selection synthesis (USS) and statistic-based parameter synthesis (HMM-based speech synthesis, HTS). The USS method cuts out speech data into phoneme units and stores them, and finds sound pieces suitable for utterance during speech synthesis. The HTS method extracts parameters corresponding to speech characteristics to create a statistical model, and then converts text into speech based on the statistical model. as a way to reconstruct it. However, the conventional voice synthesis method described above has many limitations in synthesizing a natural voice reflecting the speaker's speech style or emotional expression.

Accordingly, recently, a speech synthesis method for synthesizing speech from text based on an artificial neural network is attracting attention.

The goal is to provide an artificial intelligence-based speech synthesis technology that can implement natural speech as if a real speaker is speaking.

An object of the present invention is to provide an artificial intelligence-based speech synthesis technology that can freely change the speed and pitch of a speech signal synthesized from text.

The technical problem to be solved is not limited to the technical problems as described above, and other technical problems may be inferred.

As a technical means for achieving the above technical problem, a first aspect of the present disclosure is to set sections having a first window length based on a first hop length in a first voice signal. step; generating a spectrogram by performing a short-time Fourier transform on the sections; determining a multiplication rate and a pitch change rate for changing the speed and pitch of the first voice signal; generating speech signals of sections having a second window length based on a second hop length from the spectrogram; and generating a second voice signal whose speed and pitch are changed based on the voice signals of the sections, wherein a ratio of the first hop length to the second hop length is equal to a value of the speed ratio, and the second It may provide a method, wherein the ratio of the first window length to the window length is set to be equal to the value of the pitch change rate.

Also, the value of the second hop length may correspond to a preset value, and the first hop length may be set to be equal to a value obtained by multiplying the second hop length by the multiplication rate.

Also, the value of the first window length may correspond to a preset value, and the second window length may be set to be equal to a value obtained by dividing the first window length by the pitch change rate.

In addition, the generating of the speech signals of sections having the second window length may include estimating phase information by repeatedly performing a short-time Fourier transform and an inverse short-time Fourier transform on the spectrogram. ; and generating voice signals of sections having a second window length based on the second hop length based on the phase information.

The present invention can provide high-quality speech by freely changing the speed and pitch of a speech signal synthesized from text.

1 is a diagram schematically illustrating an operation of a speech synthesis system.
2 is a diagram illustrating an embodiment of a speech synthesis system.
3 is a diagram illustrating an embodiment of a synthesizer of a voice synthesis system.
4 is a diagram illustrating an embodiment of outputting a Mel spectrogram through a synthesizer.
5 is a diagram illustrating an embodiment of a speech synthesis system.
6 is a diagram illustrating an embodiment of performing a Short-time Fourier Transform on an input voice signal.
7 is a diagram illustrating an embodiment of changing a speed in a voice post-processing unit.
8 is a diagram illustrating an embodiment of changing a pitch in a voice post-processing unit.
9 is a flowchart illustrating an embodiment of a method for changing the speed and pitch of a voice signal.

The terms used in the present embodiments are selected as currently widely used general terms as possible while considering the functions in the present embodiments, but may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, etc. have. In addition, in certain cases, there are also terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the relevant part. Therefore, the terms used in the present embodiments should be defined based on the meaning of the term and the contents throughout the present embodiments, rather than the simple name of the term.

Since the present embodiments may have various changes and may have various forms, some embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the present embodiments to a specific disclosed form, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present embodiments. The terms used herein are used only for description of the embodiments, and are not intended to limit the present embodiments.

Unless otherwise defined, terms used in the present embodiments have the same meanings as commonly understood by those of ordinary skill in the art to which the present embodiments belong. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present embodiments, have an ideal or excessively formal meaning. should not be interpreted.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0010] Reference is made to the accompanying drawings, which show by way of illustration specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein may be implemented with changes from one embodiment to another without departing from the spirit and scope of the present invention. In addition, it should be understood that the location or arrangement of individual components within each embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the following detailed description is not to be taken in a limiting sense, and the scope of the present invention should be taken as encompassing the scope of the claims and all equivalents thereto. In the drawings, like reference numerals refer to the same or similar elements throughout the various aspects.

On the other hand, in the present specification, technical features that are individually described within one drawing may be implemented individually or may be implemented at the same time.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings in order to enable those of ordinary skill in the art to easily practice the present invention.

1 is a diagram schematically illustrating an operation of a speech synthesis system.

A speech synthesis system is a system that converts text into human speech.

For example, the speech synthesis system 100 of FIG. 1 may be an artificial neural network-based speech synthesis system. An artificial neural network refers to an overall model that has problem-solving ability by changing the strength of synaptic bonding through learning in which artificial neurons that form a network through synaptic bonding.

The voice synthesis system 100 may be implemented in various types of devices such as a personal computer (PC), a server device, a mobile device, and an embedded device, and as a specific example, a smartphone or tablet that performs voice synthesis using an artificial neural network. It may correspond to a device, an Augmented Reality (AR) device, an Internet of Things (IoT) device, an autonomous vehicle, robotics, a medical device, an e-book terminal, and a navigation device, but is not limited thereto.

Furthermore, the speech synthesis system 100 may correspond to a dedicated hardware accelerator mounted on the above device. Alternatively, the speech synthesis system 100 may be a hardware accelerator such as a neural processing unit (NPU), a tensor processing unit (TPU), or a neural engine, which is a dedicated module for driving an artificial neural network, but is not limited thereto.

Referring to FIG. 1 , the speech synthesis system 100 may receive a text input and specific speaker information. For example, the speech synthesis system 100 may receive “Have a good day!” as shown in FIG. 1 as a text input, and may receive “speaker 1” as a speaker information input.

“Speaker 1” may correspond to a voice signal or a voice sample indicating a preset speech characteristic of speaker 1. For example, the speaker information may be received from an external device through a communication unit included in the speech synthesis system 100 . Alternatively, the speaker information may be input from a user through the user interface of the voice synthesis system 100 and may be selected from among various speaker information pre-stored in the database of the voice synthesis system 100, but is limited thereto. no.

The speech synthesis system 100 may output a speech based on a text input received as an input and specific speaker information. For example, the speech synthesis system 100 may say “Have a good day!” and “Speaker 1” may be received as inputs, and a voice for “Have a good day!” in which the speech characteristics of speaker 1 are reflected may be output. The speech characteristics of the speaker 1 may include at least one of various factors such as the speaker 1's voice, a rhyme, a pitch, and an emotion. That is, the output voice may be a voice as if the speaker 1 naturally pronounces “Have a good day!”. A detailed operation of the speech synthesis system 100 will be described later with reference to FIGS. 2 to 4 .

2 is a diagram illustrating an embodiment of a speech synthesis system. The speech synthesis system 200 of FIG. 2 may be the same as the speech synthesis system 100 of FIG. 1 .

Referring to FIG. 2 , the speech synthesis system 200 may include a speaker encoder 210 , a synthesizer 220 , and a vocoder 230 . Meanwhile, in the speech synthesis system 200 illustrated in FIG. 2 , only components related to an embodiment are illustrated. Accordingly, it is apparent to those skilled in the art that the speech synthesis system 200 may include other general-purpose components in addition to the components shown in FIG. 2 .

The speech synthesis system 200 of FIG. 2 may receive speaker information and text as inputs and output a speech.

For example, the speaker encoder 210 of the speech synthesis system 200 may receive speaker information as an input and generate a speaker embedding vector. The speaker information may correspond to a speaker's voice signal or a voice sample. The speaker encoder 210 may receive the speaker's voice signal or voice sample, extract the speaker's utterance characteristics, and represent this as an embedding vector.

The speaker's speech characteristics may include at least one of various factors such as speech speed, pause period, pitch, tone, rhyme, intonation, or emotion. That is, the speaker encoder 210 may represent the discontinuous data values included in the speaker information as a vector composed of continuous numbers. For example, the speaker encoder 210 includes a pre-net, a CBHG module, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a BRDNN ( A speaker embedding vector may be generated based on at least one or a combination of two or more of various artificial neural network models, such as a Bidirectional Recurrent Deep Neural Network).

For example, the synthesizer 220 of the speech synthesis system 200 may receive text and an embedding vector representing the speaker's utterance characteristics as inputs, and output a spectrogram.

3 is a diagram illustrating an embodiment of a synthesizer of a voice synthesis system. The synthesis unit 300 of FIG. 3 may be the same as the synthesis unit 220 of FIG. 2 .

Referring to FIG. 3 , the synthesizer 300 of the speech synthesis system 200 may include a text encoder and a decoder. Meanwhile, it is apparent to those skilled in the art that the synthesis unit 300 may further include other general-purpose components in addition to the components illustrated in FIG. 3 .

The embedding vector representing the speech characteristic of the speaker may be generated from the speaker encoder 210 as described above, and the encoder or decoder of the synthesizing unit 300 receives the embedding vector representing the speech characteristic of the speaker from the speaker encoder 210 . can do.

The text encoder of the synthesizing unit 300 may generate a text embedding vector by receiving text as an input. The text may include a sequence of characters in a particular natural language. For example, the sequence of characters may include alphabetic characters, numbers, punctuation marks, or other special characters.

The text encoder may divide the inputted text into a unit of a alphabet, a unit of a letter, or a unit of a phoneme, and may input the separated text into the artificial neural network model. For example, the text encoder may generate a text embedding vector based on at least one or a combination of two or more of various artificial neural network models such as pre-net, CBHG module, DNN, CNN, RNN, LSTM, and BRDNN.

Alternatively, the text encoder may divide the input text into a plurality of short texts and generate a plurality of text embedding vectors for each of the short texts.

The decoder of the synthesizer 300 may receive a speaker embedding vector and a text embedding vector from the speaker encoder 210 as inputs. Alternatively, the decoder of the synthesizing unit 300 may receive a speaker embedding vector from the speaker encoder 210 as an input and receive a text embedding vector from the text encoder as an input.

The decoder may generate a spectrogram corresponding to the input text by inputting the speaker embedding vector and the text embedding vector to the artificial neural network model. That is, the decoder may generate a spectrogram of the input text in which the speaker's speech characteristics are reflected. For example, the spectrogram may correspond to a mel-spectrogram, but is not limited thereto.

A spectrogram is a graph that visualizes the spectrum of a voice signal. The x-axis of the spectrogram represents time and the y-axis represents frequency, and the value of each time frequency can be expressed as a color according to the size of the value. The spectogram may be a result of performing short-time Fourier transform (STFT) on a continuously given speech signal.

STFT is a method in which a voice signal is divided into sections of a certain length and a Fourier transform is applied to each section. At this time, since the result of performing STFT on the speech signal is a complex value, it is possible to take an absolute value to the complex value to lose phase information and to generate a spectrogram including only magnitude information.

On the other hand, the Mel spectrogram is a readjustment of the frequency interval of the spectrogram to the Mel scale. The human auditory organ is more sensitive in the low frequency band than in the high frequency band, and the Mel Scale expresses the relationship between the physical frequency and the frequency perceived by humans by reflecting these characteristics. The Mel spectrogram may be generated by applying a filter bank based on the Mel scale to the spectrogram.

Meanwhile, although not shown in FIG. 3 , the synthesizing unit 300 may further include an attention module for generating the attention alignment. The attention module is a module that learns which output of a specific time-step of the decoder is most related to the output of all the time-steps of the encoder. A higher quality spectrogram or Mel spectrogram can be output by using the attention module.

4 is a diagram illustrating an embodiment of outputting a Mel spectrogram through a synthesizing unit. The synthesis unit 400 of FIG. 4 may be the same as the synthesis unit 300 of FIG. 3 .

Referring to FIG. 4 , the combining unit 400 may receive a list including input texts and speaker embedding vectors corresponding thereto. For example, the synthesizing unit 400 inputs the input text 'first sentence', the speaker embedding vectors corresponding thereto, embed_voice1, and the input text 'second sentence', and the corresponding speaker embedding vectors embed_voice2, 'third sentence'. A list 410 including text and a speaker embedding vector corresponding thereto, embed_voice3, may be received as an input.

The synthesizer 400 may generate as many Mel spectrograms 420 as the number of input texts included in the received list 410 . Referring to FIG. 4 , it can be seen that Mel spectrograms corresponding to input texts of 'first sentence', 'second sentence' and 'third sentence' are generated.

Alternatively, the synthesizing unit 400 may generate as many Mel spectrograms 420 and attention alignment as the number of input texts together. Although not shown in FIG. 4 , for example, attention alignment corresponding to each input text of 'first sentence', 'second sentence' and 'third sentence' may be additionally generated. Alternatively, the synthesizer 400 may generate a plurality of Mel spectrograms and a plurality of attention alignments for each of the input texts.

Returning to FIG. 2 again, the vocoder 230 of the speech synthesis system 200 may generate the spectrogram output from the synthesizer 220 as an actual speech. The spectrogram output as described above may be a Mel spectrogram.

In an embodiment, the vocoder 230 may generate the spectrogram output from the synthesizer 220 as an actual voice signal using Inverse Short-Time Fourier Transform (ISFT). Since the spectrogram or Mel spectrogram does not include phase information, when generating a voice signal using ISFT, the phase information of the spectrogram or Mel spectrogram is not considered.

In another embodiment, the vocoder 230 may generate the spectrogram output from the synthesizer 220 as an actual voice signal using a Griffin-Lim algorithm. The Griffin-Rim algorithm is an algorithm for estimating phase information from magnitude information of a spectrogram or Mel spectrogram.

Alternatively, the vocoder 230 may generate the spectrogram output from the synthesizer 220 as an actual voice signal, for example, based on a neural vocoder.

A neural vocoder is an artificial neural network model that generates a voice signal by receiving a spectrogram or a Mel spectrogram as an input. A neural vocoder can learn the relationship between a spectrogram or a Mel spectrogram and a voice signal from a large amount of data, and can generate a high-quality real voice signal through this.

A neural vocoder may correspond to a vocoder based on an artificial neural network model such as, but not limited to, WaveNet, Parallel WaveNet, WaveRNN, WaveGlow, or MelGAN.

For example, the WaveNet vocoder consists of several dilated causal convolution layers and is an autoregressive model that uses sequential features between speech samples. WaveRNN vocoder is an autoregressive model that replaces the dilated causal convolution layer of multiple layers of WaveNet with Gated Recurrent Unit (GRU). The WaveGlow vocoder can learn to obtain a simple distribution such as a Gaussian distribution from the spectrogram dataset (x) using an invertible transform function. After learning, the WaveGlow vocoder can output a voice signal from a Gaussian distribution sample by using the inverse function of the transform function.

5 is a diagram illustrating an embodiment of a speech synthesis system. The speech synthesis system 500 of FIG. 5 may be the same as the speech synthesis system 100 of FIG. 1 or the speech synthesis system 200 of FIG. 2 .

Referring to FIG. 5 , the speech synthesis system 500 may include a speaker encoder 510 , a synthesizer 520 , a vocoder 530 , and a voice post-processor 540 . Meanwhile, in the speech synthesis system 500 illustrated in FIG. 5 , only components related to an embodiment are illustrated. Accordingly, it is apparent to those skilled in the art that the speech synthesis system 500 may further include other general-purpose components in addition to the components shown in FIG. 5 .

The speaker encoder 510 , the synthesizer 520 , and the vocoder 530 of FIG. 5 may be the same as the speaker encoder 210 , the synthesizer 220 , and the vocoder 230 of FIG. 2 , respectively. Accordingly, descriptions of the speaker encoder 510 , the synthesizer 520 , and the vocoder 530 of FIG. 5 will be omitted.

As described above, the synthesizer 520 may generate a spectrogram or a Mel spectrogram by inputting text and a speaker embedding vector received from the speaker encoder 510 as inputs. Also, the vocoder 530 may generate an actual speech by receiving a spectrogram or a Mel spectrogram as an input.

The voice post-processing unit 540 of FIG. 5 receives the voice generated by the vocoder 530 as an input and performs post-processing operations such as noise removal, audio stretching, or pitch shifting. can The voice post-processing unit 540 may generate a corrected voice to be finally heard by the user by performing post-processing on the input voice.

For example, the voice post-processing unit 540 may correspond to a phase vocoder, but is not limited thereto. The phase vocoder corresponds to a vocoder that can adjust the frequency domain and time domain of a voice using phase information.

The phase vocoder may perform short-time Fourier transform (STFT) on the input voice signal to convert a time domain voice signal into a time-frequency domain voice signal. As described above with reference to FIG. 2 , the STFT divides the voice signal into several sections of a short time and performs a Fourier transform on each section, so that frequency characteristics that change with time can be confirmed.

Alternatively, since the converted time-frequency domain voice signal has a complex value, the phase vocoder may generate a spectrogram including only magnitude information by taking an absolute value from the complex value. Alternatively, the phase vocoder may generate a mel spectrogram by re-adjusting a frequency interval of the spectrogram to a mel scale.

The phase vocoder may perform post-processing operations such as noise removal, audio stretching, or pitch change using the converted time-frequency domain speech signal or spectrogram.

6 is a diagram illustrating an embodiment in which a short-time Fourier transform is performed on a voice signal.

Referring to FIG. 6 , the speech post-processing unit 540 may divide a time domain speech signal into sections having a constant window length, and perform Fourier transform on each section. Accordingly. For each section, a converted time-frequency domain voice signal or spectrogram may be generated. 6 shows that a spectrogram is generated by performing a Fourier transform on each section. In this case, the spectrogram may correspond to a Mel spectrogram. Meanwhile, the window length may be set to the same value as a Fast Fourier Transform size (FFT size) indicating the number of samples to be subjected to Fourier transform, but is not limited thereto.

Meanwhile, in consideration of a trade-off relationship between frequency resolution and temporal resolution, a hop length may be set so that sections having a constant window length overlap.

For example, when the value of the sampling rate is 24000 and the window length is 0.05 seconds, the Fourier transform may be performed using 1200 samples for each section. Also, when the hop length is 0.125 seconds, the length between sections having the first window length may correspond to 0.125 seconds.

After the phase vocoder performs post-processing on the generated spectrogram as described above, the final voice may be output by using an inverse short-time Fourier transform. Alternatively, the phase vocoder may output the final voice by using the Griffin-Lim algorithm after performing post-processing on the spectrogram generated as described above.

7 is a diagram illustrating an embodiment of changing a speed in a voice post-processing unit.

The above-described voice processing unit 540 of FIG. 5 may change the speed of the voice generated by the vocoder 530, which is also referred to as audio stretching. Audio stretching is to change the speed or playback time of a voice signal without affecting the pitch of the voice signal.

The voice processing unit 540 may generate a spectrogram from the voice generated by the vocoder 530 and change the speed of the voice in the process of restoring the generated spectogram back to the voice.

Referring to FIG. 7 , the voice processing unit 540 may perform a short-time Fourier transform on the first voice signal 710 generated by the vocoder 530 . The spectrogram 720 of FIG. 7 may correspond to a spectrogram generated by performing a short-time Fourier transform on the first voice signal 710 generated by the vocoder 530 . For example, the spectrogram 720 of FIG. 7 may correspond to a Mel spectrogram.

For example, the voice post-processing unit 540 sets sections having a first window length based on a first hop length in the first voice signal 710 generated by the vocoder 530 . can The first hop length may correspond to a length between sections having the first window length.

For example, referring to FIG. 7 , when the sampling rate value is 24000, the first window length may be 0.05 seconds, and the first hop length may be 0.025 seconds. In this case, the speech post-processing unit 540 may perform Fourier transform using 1200 samples for each section.

The speech post-processing unit 540 may generate a time-frequency domain speech signal by performing a short-time Fourier transform on the divided sections as described above, and a spectrogram 720 based on the time-frequency domain speech signal. ) can be created. Specifically, since the speech signal in the time-frequency domain has a complex value, it is possible to generate the spectrogram 720 including only magnitude information by taking the absolute value of the complex value to lose phase information. In this case, the spectrogram 720 may correspond to a Mel spectrogram.

Meanwhile, the voice post-processing unit 540 may determine a speed ratio to change the speed of the first voice signal 710 generated by the vocoder 530 . For example, in order to generate a voice that is twice as fast as the speed of the first voice signal 710 generated by the vocoder 530 , the double speed may be determined to be 2 .

The speech post-processing unit 540 may generate speech signals of sections having the second window length based on the second hop length from the spectrogram 720 . For example, the speech post-processing unit 540 may estimate phase information by repeatedly performing a short-time Fourier transform and an inverse short-time Fourier transform on the spectrogram 720 , and the estimated phase information Voice signals of the sections may be generated based on . The voice post-processing unit 540 may generate the second voice signal 730 of which the speed is changed based on the voice signals of the sections.

In order to change the speed of the first voice signal 710 , the voice post-processing unit 540 may set the ratio of the first hop length to the second hop length to be the same as the speed ratio. For example, the second hop length may correspond to a preset value, and the first hop length may be set equal to a value obtained by multiplying the second hop length by a multiplication rate. Alternatively, the first hop length may correspond to a preset value, and the second hop length may be set equal to a value obtained by dividing the first hop length by the multiplication rate. Meanwhile, the first window length and the second window length may be equal to each other, but is not limited thereto.

For example, referring to FIG. 7 , the second window length may be equal to 0.05 seconds as the first window length, and the second hop length may be 0.0125 seconds. 7 illustrates a process of generating a voice that is twice as fast as the speed of the first voice signal 710, it can be seen that the ratio of the first hop length to the second hop length is set to be equal to 2.

The voice post-processing unit 540 may generate the second voice signal 730 in which the speed and pitch are changed based on the voice signals of sections having the second window length based on the second hop length. The corrected voice signal may correspond to a voice signal in which the speed of the first voice signal 710 is changed by a double speed. Referring to FIG. 7 , it can be seen that the second voice signal 730 that is twice as fast as the speed of the first voice signal 710 is generated.

8 is a diagram illustrating an embodiment of changing a pitch in a voice post-processing unit.

The above-described voice processing unit 540 of FIG. 5 may change the pitch of the voice generated by the vocoder 530, which is also referred to as pitch shifting. Pitch shifting is to change the pitch of a voice signal without affecting the speed or reproduction time of the voice signal.

The voice processing unit 540 may generate a spectrogram from the voice generated by the vocoder 530 and change the pitch of the voice in the process of restoring the generated spectogram back to the voice.

As described above with reference to FIG. 7 , the voice processing unit 540 may generate a spectrogram or a Mel spectrogram by performing a short-time Fourier transform on the first voice signal generated by the vocoder 530 .

For example, the voice post-processing unit 540 may set sections having the first window length based on the first hop length in the vocoder 530 .

For example, when the value of the sampling rate is 24000, the first window length may be 0.05 seconds, and the first hop length may be 0.0125 seconds.

The speech post-processing unit 540 may generate a time-frequency domain speech signal by performing a short-time Fourier transform on the divided sections, and generates a spectrogram or a Mel spectrogram based on the time-frequency domain speech signal. can do.

Meanwhile, the voice post-processing unit 540 may determine a pitch change rate to change the pitch of the first voice signal. For example, in order to generate a voice having a pitch 1.25 times higher than that of the first voice signal, the pitch change rate may be determined to be 1.25.

The speech post-processing unit 540 may generate speech signals of sections having the second window length based on the second hop length from the spectrogram. For example, the speech post-processing unit 540 may estimate phase information by repeatedly performing short-time Fourier transform and inverse short-time Fourier transform on the spectrogram, and generate speech signals of the sections based on the estimated phase information. can The voice post-processing unit 540 may generate a second voice signal whose pitch is changed based on the voice signals of the sections.

The voice post-processing unit 540 may set the ratio of the first window length to the second window length to be equal to the pitch change rate in order to change the pitch of the first voice signal. For example, the first window length may correspond to a preset value, and the second window length may be set equal to a value obtained by dividing the first window length by the pitch change rate. Alternatively, the second window length may correspond to a preset value, and the first window length may be set equal to a value obtained by multiplying the second window length by a pitch change rate. Meanwhile, the first hop length and the second hop length may be the same, but is not limited thereto.

For example, in FIG. 8B , the first voice signal and the first voice signal generated by the vocoder 530 have a sampling rate of 24000, a first window length of 0.05 seconds, and a first hop length of 0.0125 seconds. A spectrogram generated by performing a short-time Fourier transform is shown. The generated spectrogram can have 601 frequency arrays up to 12000hz at intervals of 20hz.

On the other hand, when the pitch change rate is 1.25, the frequency components of 9600hz or higher become 12000hz after the pitch change, so frequency components of 9600hz or higher may be lost. Accordingly, pitch change can be performed only for frequency components of 9600 Hz or less, and 481 frequency arrays can be arranged up to 9600 Hz at an interval of 20 Hz. That is, when generating a voice signal whose pitch is corrected by 1.25 times from the spectrogram, the second window length can be set to 0.04 seconds, which is a value obtained by dividing the value of the first window length by the pitch change rate of 1.25 in order to use only 481 frequency arrays. have. (c) of FIG. 8 (c) generates speech signals of sections having a second window length of 0.04 seconds based on the second hop length of 0.0125 seconds with respect to the spectrogram of FIG. 8 (b), and based on the speech signals of the sections may represent the second voice signal whose pitch is changed by 1.25 times.

Alternatively, when the pitch change rate is 0.75, in order to increase the size of the 601 frequency arrays to 801, the remaining 200 frequency components may be zero padded. Accordingly, the second window length may be set as a value obtained by dividing the value of the first window length by the pitch change rate of 0.75. FIG. 8(a) may represent a second voice signal in which the pitch of the voice signal of FIG. 8(b) is changed by 0.75 times.

As described above, in order to correct the speed of the first voice signal generated by the vocoder 530, the ratio of the first hop length to the second hop length may be set to be equal to the value of the speed ratio. In addition, in order to correct the pitch of the first voice signal generated by the vocoder 530, the ratio of the first window length to the second window length may be set to be equal to the value of the pitch change rate.

In summary, the speed and pitch of the first voice signal generated by the vocoder 530 may be simultaneously corrected. For example, when the ratio of the first hop length to the second hop length is equal to the value of the multiplication rate, and the ratio of the first window length to the second window length is set to be equal to the value of the pitch change rate, the first The speed of the first voice signal may be changed by the double speed, and the pitch of the first voice signal may be changed by the pitch change rate.

9 is a flowchart illustrating an embodiment of a method for changing the speed and pitch of a voice signal.

Referring to FIG. 9 , in operation 810 , the voice post-processing unit may set sections having a first window length based on a first hop length in the first voice signal.

In operation 820, the speech post-processing unit may generate a spectrogram by performing a short-time Fourier transform on the sections.

For example, the speech post-processing unit may generate a time-frequency domain speech signal by performing a Fourier transform for each section. The speech post-processing unit may take an absolute value from a speech signal in the time-frequency domain, lose phase information, and generate a spectrogram including only magnitude information.

In operation 830, the voice post-processing unit may determine a multiplication rate and a pitch change rate for changing the speed and pitch of the first voice signal. For example, in order to generate a voice that is twice as fast as the speed of the first voice signal generated by the vocoder, the double speed may be determined to be 2. Alternatively, in order to generate a voice having a pitch 1.25 times higher than that of the first voice signal generated by the vocoder, the pitch change rate may be determined to be 1.25.

In operation 840, the speech post-processing unit may generate speech signals of sections having the second window length based on the second hop length from the spectrogram.

For example, the speech post-processing unit may estimate phase information by repeatedly performing short-time Fourier transform and inverse short-time Fourier transform on the spectrogram. For example, the voice post-processing unit may use a Griffin-Lim algorithm, but is not limited thereto. The speech post-processing unit may generate speech signals of sections having a second window length based on the second hop length based on the estimated phase information.

In operation 840, the voice post-processing unit may generate a second voice signal having a changed speed and pitch based on the voice signals of the sections.

For example, the voice post-processing unit may add all the voice signals of the sections to finally generate the second voice signal in which the speed and the pitch of the first voice signal are changed by the double speed and the pitch change rate, respectively.

Various embodiments of the present disclosure may be implemented as software (eg, a program) including one or more instructions stored in a storage medium readable by a machine. For example, the processor of the device may call at least one of the one or more instructions stored from the storage medium and execute it. This makes it possible for the device to be operated to perform at least one function according to the called at least one command. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

In this specification, “unit” may be a hardware component such as a processor or circuit, and/or a software component executed by a hardware component such as a processor.

The description of the present specification described above is for illustration, and those of ordinary skill in the art to which the content of this specification belongs will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be able Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present embodiment is indicated by the claims to be described later rather than the above detailed description, and it should be construed to include all changes or modifications derived from the meaning and scope of the claims and their equivalents.

Claims (4)

setting, by a voice post-processing unit, sections having a first window length based on a first hop length in a first voice signal;
generating a spectrogram by performing a Short-time Fourier Transform on the sections;
determining a multiplier rate and a pitch change rate for changing the speed and pitch of the first voice signal;
generating speech signals of sections having a second window length based on a second hop length from the spectrogram; and
generating a second voice signal having a changed speed and pitch based on the voice signals of the sections;
The ratio of the first hop length to the second hop length is set to be equal to the value of the multiplication rate, and the ratio of the first window length to the second window length is set to be equal to the value of the pitch change rate,
The value of the second hop length corresponds to a preset value, and the first hop length is set to be equal to a value obtained by multiplying the second hop length by the multiplication rate;
The value of the first window length corresponds to a preset value, and the second window length is set to be equal to a value obtained by dividing the first window length by the pitch change rate,
The generating of voice signals of sections having the second window length comprises:
estimating phase information by repeatedly performing, by the speech post-processing unit, a short-time Fourier transform and an inverse short-time Fourier transform on the spectrogram; and
and generating voice signals of sections having a second window length based on the second hop length based on the phase information. delete delete delete

Images