KR20240014251A - Method and tts system for changing the speed and the pitch of the speech - Google Patents

Abstract

A method for synthesizing voices with changed speed and pitch is provided.
The method of the present invention generates a spectrogram by performing a short-time Fourier transform on the first voice signal based on the first hop length and the first window length, and has a second window length at a second hop length interval from the spectrogram. Voice signals of sections can be generated. At this time, the ratio of the first hop length to the second hop length is set to be equal to the value of the double speed rate, and the ratio of the first window length to the second window length is set to be equal to the value of the pitch change rate so that the speed and pitch are A changed second voice signal can be generated.

Description

Method and voice synthesis system for changing the speed and pitch of voice {METHOD AND TTS SYSTEM FOR CHANGING THE SPEED AND THE PITCH OF THE SPEECH}

This disclosure relates to a method for changing the speed and pitch of speech and a speech synthesis system.

With the recent development of artificial intelligence technology, interfaces that utilize voice signals are becoming common. Accordingly, research is being actively conducted on speech synthesis technology that allows speech synthesis to be performed according to a given situation.

Voice synthesis technology, combined with artificial intelligence-based voice recognition technology, is being applied to many fields such as virtual assistants, audiobooks, automatic interpretation and translation, and virtual voice actors.

Conventional speech synthesis methods include various methods such as unit selection synthesis (USS) and statistical parameter synthesis (HMM-based Speech Synthesis, HTS). The USS method is a method of cutting and storing speech data into phoneme units and finding and concatenating sound fragments suitable for speech during speech synthesis. The HTS method extracts parameters corresponding to speech characteristics, creates a statistical model, and converts the text into speech based on the statistical model. This is a way to reconstruct it. However, the conventional voice synthesis method described above has many limitations in synthesizing natural voices that reflect the speaker's speaking style or emotional expression.

Accordingly, recently, a speech synthesis method for synthesizing speech from text based on an artificial neural network has been attracting attention.

The goal is to provide artificial intelligence-based voice synthesis technology that can produce natural voices that sound like actual speakers are speaking.

The goal is to provide artificial intelligence-based speech synthesis technology that can freely change the speed and pitch of speech signals synthesized from text.

The technical challenges to be solved are not limited to the technical challenges described above, and other technical challenges can be inferred.

As a technical means for achieving the above-described technical problem, the first aspect of the present disclosure is to set sections having a first window length based on the first hop length in the first voice signal. step; generating a spectrogram by performing short-time Fourier transform on the sections; determining a speed and pitch change rate for changing the speed and pitch of the first voice signal; generating voice signals in sections having a second window length based on the second hop length from the spectrogram; and generating a second voice signal with changed speed and pitch based on the voice signals of the sections, wherein the ratio of the first hop length to the second hop length is equal to the value of the double speed rate, and the second A method may be provided in which the ratio of the first window length to the window length is set to be equal to the value of the pitch change rate.

Additionally, the value of the second hop length may correspond to a preset value, and the first hop length may be set to be equal to the value obtained by multiplying the second hop length by the speed ratio.

Additionally, the value of the first window length may be set to a preset value, and the second window length may be set to be equal to the value obtained by dividing the first window length by the pitch change rate.

In addition, generating voice signals in sections having the second window length includes estimating phase information by repeatedly performing short-time Fourier transform and inverse short-time Fourier transform on the spectrogram. ; and generating voice signals of sections having a second window length based on the second hop length based on the phase information.

The present invention can provide high-quality speech by freely changing the speed and pitch of the speech signal synthesized from text.

1 is a diagram schematically showing the operation of a voice synthesis system.
Figure 2 is a diagram showing an embodiment of a voice synthesis system.
Figure 3 is a diagram showing an embodiment of a synthesis unit of a voice synthesis system.
Figure 4 is a diagram showing an embodiment of outputting a Mel spectrogram through a synthesis unit.
Figure 5 is a diagram showing an embodiment of a voice synthesis system.
Figure 6 is a diagram showing an embodiment of performing short-time Fourier Transform on an input voice signal.
Figure 7 is a diagram showing an embodiment of changing the speed in the voice post-processing unit.
Figure 8 is a diagram showing an embodiment of changing the pitch in a voice post-processing unit.
Figure 9 is a flowchart showing an embodiment of a method for changing the speed and pitch of a voice signal.

The terms used in the present embodiments were selected as widely used general terms as possible while considering the functions in the present embodiments, but this may vary depending on the intention or precedent of a technician working in the field, the emergence of new technology, etc. there is. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the relevant section. Therefore, the terms used in the present embodiments should be defined based on the meaning of the term and the overall content of the present embodiments, rather than simply the name of the term.

Since these embodiments can be subject to various changes and have various forms, some embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the present embodiments to a specific disclosure form, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present embodiments. The terms used in this specification are merely used to describe the embodiments and are not intended to limit the embodiments.

Unless otherwise defined, the terms used in the present embodiments have the same meaning as generally understood by those skilled in the art to which the present embodiments belong. Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in the present embodiments, they should not be used in an ideal or excessively formal sense. It should not be interpreted.

The detailed description of the present invention described below refers to the accompanying drawings, which show by way of example specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different from one another but are not necessarily mutually exclusive. For example, specific shapes, structures and characteristics described herein may be implemented with changes from one embodiment to another without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the detailed description described below is not to be taken in a limiting sense, and the scope of the present invention should be taken to encompass the scope claimed by the claims and all equivalents thereof. Like reference numbers in the drawings indicate identical or similar elements throughout various aspects.

Meanwhile, technical features described individually in one drawing in this specification may be implemented individually or simultaneously.

Hereinafter, several embodiments of the present invention will be described in detail with reference to the attached drawings in order to enable those skilled in the art to easily practice the present invention.

1 is a diagram schematically showing the operation of a voice synthesis system.

A speech synthesis system is a system that converts text into human voice.

For example, the voice synthesis system 100 of FIG. 1 may be a voice synthesis system based on an artificial neural network. Artificial neural network refers to an overall model in which artificial neurons, which form a network by combining synapses, change the strength of synaptic connections through learning and have problem-solving capabilities.

The voice synthesis system 100 can be implemented with various types of devices such as personal computers (PCs), server devices, mobile devices, and embedded devices. Specific examples include smartphones and tablets that perform voice synthesis using an artificial neural network. This may apply to devices, AR (Augmented Reality) devices, IoT (Internet of Things) devices, self-driving cars, robotics, medical devices, e-readers, and navigation, but is not limited thereto.

Furthermore, the voice synthesis system 100 may correspond to a dedicated hardware accelerator (HW accelerator) mounted on the above device. Alternatively, the voice synthesis system 100 may be a hardware accelerator such as a neural processing unit (NPU), a tensor processing unit (TPU), or a neural engine, which are dedicated modules for driving an artificial neural network, but is not limited thereto.

Referring to FIG. 1, the speech synthesis system 100 can receive text input and specific speaker information. For example, the voice synthesis system 100 may receive “Have a good day!” as shown in FIG. 1 as a text input, and “Speaker 1” as speaker information input.

“Speaker 1” may correspond to a voice signal or voice sample representing the preset speech characteristics of Speaker 1. For example, speaker information may be received from an external device through a communication unit included in the voice synthesis system 100. Alternatively, speaker information may be input from the user through the user interface of the voice synthesis system 100, and may be selected as one of various speaker information pre-stored in the database of the voice synthesis system 100, but is not limited thereto. no.

The speech synthesis system 100 can output speech based on text input received as input and specific speaker information. For example, speech synthesis system 100 may say “Have a good day!” and “Speaker 1” can be received as input, and a voice for “Have a good day!” reflecting the speech characteristics of Speaker 1 can be output. Speaker 1's speech characteristics may include at least one of various elements such as Speaker 1's voice, prosody, pitch, and emotion. In other words, the voice output may be a voice that seems to be Speaker 1 naturally pronouncing “Have a good day!” The specific operation of the voice synthesis system 100 will be described later with reference to FIGS. 2 to 4.

Figure 2 is a diagram showing an embodiment of a voice synthesis system. The voice synthesis system 200 of FIG. 2 may be the same as the voice synthesis system 100 of FIG. 1 .

Referring to FIG. 2, the voice synthesis system 200 may include a speaker encoder 210, a synthesis unit 220, and a vocoder 230. Meanwhile, in the voice synthesis system 200 shown in FIG. 2, only components related to one embodiment are shown. Accordingly, it is obvious to those skilled in the art that the voice synthesis system 200 may further include other general-purpose components in addition to the components shown in FIG. 2.

The speech synthesis system 200 of FIG. 2 can receive speaker information and text as input and output speech.

For example, the speaker encoder 210 of the speech synthesis system 200 may receive speaker information as input and generate a speaker embedding vector. Speaker information may correspond to the speaker's voice signal or voice sample. The speaker encoder 210 can receive the speaker's voice signal or voice sample, extract the speaker's speech characteristics, and represent this as an embedding vector.

A speaker's speech characteristics may include at least one of various factors such as speech rate, pause, pitch, timbre, prosody, intonation, or emotion. That is, the speaker encoder 210 can represent discontinuous data values included in speaker information as a vector composed of continuous numbers. For example, the speaker encoder 210 includes a pre-net, a CBHG module, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), and a BRDNN ( A speaker embedding vector can be generated based on at least one or a combination of two or more of various artificial neural network models such as Bidirectional Recurrent Deep Neural Network).

For example, the synthesis unit 220 of the speech synthesis system 200 may receive text and an embedding vector representing the speaker's speech characteristics as input and output a spectrogram.

Figure 3 is a diagram showing an embodiment of a synthesis unit of a voice synthesis system. The synthesis unit 300 of FIG. 3 may be the same as the synthesis unit 220 of FIG. 2 .

Referring to FIG. 3, the synthesis unit 300 of the speech synthesis system 200 may include a text encoder and decoder. Meanwhile, it is obvious to those skilled in the art that the synthesis unit 300 may further include other general-purpose components in addition to the components shown in FIG. 3.

An embedding vector representing the speaker's speech characteristics may be generated from the speaker encoder 210 as described above, and the encoder or decoder of the synthesis unit 300 receives the embedding vector representing the speaker's speech characteristics from the speaker encoder 210. can do.

The text encoder of the synthesis unit 300 may receive text as input and generate a text embedding vector. Text may contain sequences of characters in a particular natural language. For example, a sequence of characters may include alphabetic characters, numbers, punctuation marks, or other special characters.

The text encoder can separate the input text into alphabet units, letter units, or phoneme units, and input the separated text into an artificial neural network model. For example, the text encoder may generate a text embedding vector based on at least one or a combination of two or more of various artificial neural network models such as pre-net, CBHG module, DNN, CNN, RNN, LSTM, and BRDNN.

Alternatively, the text encoder may separate the input text into a plurality of short texts and generate a plurality of text embedding vectors for each of the short texts.

The decoder of the synthesis unit 300 may receive a speaker embedding vector and a text embedding vector as input from the speaker encoder 210. Alternatively, the decoder of the synthesis unit 300 may receive a speaker embedding vector as an input from the speaker encoder 210 and a text embedding vector as an input from the text encoder.

The decoder can input the speaker embedding vector and the text embedding vector into the artificial neural network model to generate a spectrogram corresponding to the input text. In other words, the decoder can generate a spectrogram for the input text that reflects the speaker's speech characteristics. For example, the spectrogram may correspond to a mel-spectrogram (mel-spectrogram), but is not limited thereto.

A spectrogram is a visualization of the spectrum of a voice signal and expressed as a graph. The x-axis of the spectrogram represents time and the y-axis represents frequency, and the value of each time frequency can be expressed in color according to the size of the value. A spectogram may be the result of performing STFT (Short-time Fourier transform) on continuously given voice signals.

STFT is a method of dividing a voice signal into sections of a certain length and applying Fourier transform to each section. At this time, since the result of performing STFT on the voice signal is a complex value, the absolute value of the complex value is taken, phase information is lost, and a spectrogram containing only magnitude information can be generated.

Meanwhile, the Mel spectrogram is a re-adjustment of the frequency interval of the spectrogram into Mel Scale. The human hearing system is more sensitive to low frequencies than to high frequencies, and the Mel scale reflects this characteristic and expresses the relationship between physical frequencies and frequencies perceived by humans. A Mel spectrogram can be generated by applying a filter bank based on the Mel scale to the spectrogram.

Meanwhile, although not shown in FIG. 3, the synthesis unit 300 may further include an attention module for generating attention alignment. The attention module is a module that learns which output of a specific time step of the decoder is most related to the output of all time steps of the encoder. Using the attention module, a higher quality spectrogram or mel spectrogram can be output.

Figure 4 is a diagram showing an embodiment of outputting a Mel spectrogram through a synthesis unit. The synthesis unit 400 of FIG. 4 may be the same as the synthesis unit 300 of FIG. 3 .

Referring to FIG. 4, the synthesis unit 400 may receive a list including input texts and speaker embedding vectors corresponding to them. For example, the synthesis unit 400 inputs the input text 'first sentence', the corresponding speaker embedding vector, embed_voice1, the input text 'second sentence', the corresponding speaker embedding vector, embed_voice2, and the 'third sentence' input. A list 410 containing text and the corresponding speaker embedding vector, embed_voice3, can be received as input.

The synthesis unit 400 may generate a Mel spectrogram 420 equal to the number of input texts included in the received list 410. Referring to FIG. 4, it can be seen that Mel spectrograms corresponding to each input text of 'first sentence', 'second sentence', and 'third sentence' were generated.

Alternatively, the synthesis unit 400 may generate as many mel spectrograms 420 and attention alignment as the number of input texts. Although not shown in FIG. 4, for example, attention alignment corresponding to each input text of 'first sentence', 'second sentence', and 'third sentence' may be additionally generated. Alternatively, the synthesis unit 400 may generate a plurality of mel spectrograms and a plurality of attention alignments for each of the input texts.

Returning to FIG. 2, the vocoder 230 of the voice synthesis system 200 can generate the spectrogram output from the synthesis unit 220 as actual speech. As described above, the output spectrogram may be a Mel spectrogram.

In one embodiment, the vocoder 230 may generate the spectrogram output from the synthesis unit 220 as an actual voice signal using an inverse short-time Fourier transform (ISFT). Since the spectrogram or Mel spectrogram does not contain phase information, when generating a voice signal using ISFT, the phase information of the spectrogram or Mel spectrogram is not considered.

In another embodiment, the vocoder 230 may generate the spectrogram output from the synthesis unit 220 as an actual voice signal using the Griffin-Lim algorithm. The Griffin-Rim algorithm is an algorithm that estimates phase information from the size information of a spectrogram or mel spectrogram.

Alternatively, the vocoder 230 may generate the spectrogram output from the synthesis unit 220 as an actual voice signal based on, for example, a neural vocoder.

A neural vocoder is an artificial neural network model that receives a spectrogram or mel spectrogram as input and generates a voice signal. A neural vocoder can learn the relationship between a spectrogram or mel spectrogram and a voice signal through a large amount of data, and through this, can generate high-quality real voice signals.

A neural vocoder may correspond to a vocoder based on an artificial neural network model such as WaveNet, Parallel WaveNet, WaveRNN, WaveGlow, or MelGAN, but is not limited thereto.

For example, the WaveNet vocoder is composed of several dilated causal convolution layers and is an autoregressive model that uses sequential features between speech samples. The WaveRNN vocoder is an autoregressive model that replaces WaveNet's multiple layers of dilated causal convolution layers with a GRU (Gated Recurrent Unit). WaveGlow vocoder can learn to produce a simple distribution such as Gaussian distribution from the spectrogram dataset (x) using an invertible transformation function. After learning, the WaveGlow vocoder can output voice signals from samples of Gaussian distribution using the inverse function of the conversion function.

Figure 5 is a diagram showing an embodiment of a voice synthesis system. The voice synthesis system 500 of FIG. 5 may be the same as the voice synthesis system 100 of FIG. 1 or the voice synthesis system 200 of FIG. 2.

Referring to FIG. 5 , the voice synthesis system 500 may include a speaker encoder 510, a synthesis unit 520, a vocoder 530, and a voice post-processing unit 540. Meanwhile, only components related to one embodiment are shown in the voice synthesis system 500 shown in FIG. 5. Accordingly, it is obvious to those skilled in the art that the voice synthesis system 500 may further include other general-purpose components in addition to the components shown in FIG. 5.

Each of the speaker encoder 510, synthesis unit 520, and vocoder 530 of FIG. 5 may be the same as the speaker encoder 210, synthesis unit 220, and vocoder 230 of FIG. 2 described above. Therefore, descriptions of the speaker encoder 510, synthesis unit 520, and vocoder 530 of FIG. 5 are omitted.

As described above, the synthesis unit 520 can generate a spectrogram or mel spectrogram using text and the speaker embedding vector received from the speaker encoder 510 as input. Additionally, the vocoder 530 can generate actual speech by using a spectrogram or mel spectrogram as input.

The voice post-processing unit 540 in FIG. 5 receives the voice generated by the vocoder 530 as an input and performs post-processing tasks such as noise removal, audio stretching, or pitch shifting. You can. The voice post-processing unit 540 may perform post-processing on the input voice to generate a corrected voice that will be finally heard by the user.

For example, the voice post-processing unit 540 may correspond to a phase vocoder, but is not limited thereto. A phase vocoder corresponds to a vocoder that can adjust the frequency domain and time domain of a voice using phase information.

The phase vocoder can convert a voice signal in the time domain into a voice signal in the time-frequency domain by performing a short-time Fourier transform (STFT) on the input voice signal. As described above in FIG. 2, STFT divides the voice signal into several short-time sections and performs Fourier transform on each section, so that frequency characteristics that change with time can be confirmed.

Alternatively, since the converted voice signal in the time-frequency domain has a complex value, the phase vocoder can generate a spectrogram containing only size information by taking the absolute value of the complex value. Alternatively, the phase vocoder may generate a Mel spectrogram by readjusting the frequency interval of the spectrogram to the Mel scale.

The phase vocoder can perform post-processing tasks such as noise removal, audio stretching, or pitch change using the converted time-frequency domain voice signal or spectrogram.

Figure 6 is a diagram showing an embodiment of performing short-time Fourier transform on a voice signal.

Referring to FIG. 6, the voice post-processing unit 540 may divide the voice signal in the time domain into sections with a certain window length and perform Fourier transform on each section. Accordingly. For each section, a converted time-frequency domain voice signal or spectrogram can be generated. Figure 6 shows that a spectrogram is generated by performing Fourier transform on each section. At this time, the spectrogram may correspond to a mel spectrogram. Meanwhile, the window length may be set to the same value as the FFT size (Fast Fourier Transform size), which means the number of samples to perform Fourier transformation, but is not limited to this.

Meanwhile, considering the trade-off relationship between frequency resolution and time resolution, the hop length can be set so that sections with a constant window length overlap.

For example, if the sampling rate is 24000 and the window length is 0.05 seconds, Fourier transform can be performed using 1200 samples for each section. Additionally, when the hop length is 0.125 seconds, the length between sections having the first window length may correspond to 0.125 seconds.

The phase vocoder can perform post-processing on the spectrogram generated as described above and then output the final voice using an inverse short-time Fourier transform. Alternatively, the phase vocoder may perform post-processing on the generated spectrogram as described above and then output the final voice using the Griffin-Lim algorithm.

Figure 7 is a diagram showing an embodiment of changing the speed in the voice post-processing unit.

The voice processing unit 540 of FIG. 5 described above can change the speed of the voice generated by the vocoder 530, which is also referred to as audio stretching. Audio stretching is changing the speed or playback time of an audio signal without affecting the pitch of the audio signal.

The voice processing unit 540 may generate a spectrogram from the voice generated by the vocoder 530 and change the speed of the voice in the process of restoring the generated spectrogram back to voice.

* Referring to FIG. 7, the voice processor 540 may perform a short-time Fourier transform on the first voice signal 710 generated by the vocoder 530. The spectrogram 720 of FIG. 7 may correspond to a spectrogram generated by performing a short-time Fourier transform on the first voice signal 710 generated by the vocoder 530. For example, the spectrogram 720 of FIG. 7 may correspond to a Mel spectrogram.

For example, the voice post-processing unit 540 sets sections having a first window length based on the first hop length in the first voice signal 710 generated by the vocoder 530. You can. The first hop length may correspond to the length between sections having the first window length.

For example, referring to FIG. 7, when the value of the sampling rate is 24000, the first window length may be 0.05 seconds, and the first hop length may be 0.025 seconds. In this case, the voice post-processing unit 540 may perform Fourier transform using 1200 samples for each section.

The voice post-processing unit 540 can generate a voice signal in the time-frequency domain by performing a short-time Fourier transform on the divided sections as described above, and a spectrogram (720) based on the voice signal in the time-frequency domain. ) can be created. Specifically, since the voice signal in the time-frequency domain has a complex value, the phase information can be lost by taking the absolute value of the complex value and a spectrogram 720 containing only size information can be generated. At this time, the spectrogram 720 may correspond to a Mel spectrogram.

Meanwhile, the voice post-processing unit 540 may determine a speed ratio to change the speed of the first voice signal 710 generated by the vocoder 530. For example, in order to generate a voice that is twice as fast as the speed of the first voice signal 710 generated by the vocoder 530, the speed ratio may be set to 2.

The voice post-processing unit 540 may generate voice signals of sections having a second window length based on the second hop length from the spectrogram 720. For example, the voice post-processing unit 540 may estimate phase information by repeatedly performing short-time Fourier transform and inverse short-time Fourier transform on the spectrogram 720, and the estimated phase information. Based on , voice signals of the above sections can be generated. The voice post-processing unit 540 may generate a second voice signal 730 whose speed is changed based on the voice signals of the above sections.

In order to change the speed of the first voice signal 710, the voice post-processing unit 540 may set the ratio of the first hop length to the second hop length to be equal to the double speed rate. For example, the second hop length may correspond to a preset value, and the first hop length may be set to be equal to the second hop length multiplied by the double speed rate. Alternatively, the first hop length may correspond to a preset value, and the second hop length may be set to be equal to the first hop length divided by the double speed rate. Meanwhile, the first window length and the second window length may be the same, but are not limited thereto.

For example, referring to FIG. 7, the second window length may be 0.05 seconds, the same as the first window length, and the second hop length may be 0.0125 seconds. Figure 7 shows the process of generating a voice that is twice as fast as the speed of the first voice signal 710, and it can be seen that the ratio of the first hop length to the second hop length is set to be equal to 2.

The voice post-processing unit 540 may generate a second voice signal 730 whose speed and pitch are changed based on voice signals in sections having a second window length based on the second hop length. The corrected voice signal may correspond to a voice signal in which the speed of the first voice signal 710 is changed by the double speed rate. Referring to FIG. 7, it can be seen that the second voice signal 730 is generated at twice the speed of the first voice signal 710.

Figure 8 is a diagram showing an embodiment of changing the pitch in a voice post-processing unit.

The voice processing unit 540 of FIG. 5 described above can change the pitch of the voice generated by the vocoder 530, which is also referred to as pitch shifting. Pitch shifting is changing the pitch of a voice signal without affecting the speed or reproduction time of the voice signal.

The voice processing unit 540 may generate a spectrogram from the voice generated by the vocoder 530 and change the pitch of the voice in the process of restoring the generated spectrogram back to voice.

As described above in FIG. 7 , the voice processor 540 may perform a short-time Fourier transform on the first voice signal generated by the vocoder 530 to generate a spectrogram or Mel spectrogram.

For example, the voice post-processing unit 540 may set sections having a first window length based on the first hop length in the vocoder 530.

For example, if the value of the sampling rate is 24000, the first window length may be 0.05 seconds, and the first hop length may be 0.0125 seconds.

The voice post-processing unit 540 can generate a voice signal in the time-frequency domain by performing a short-time Fourier transform on the divided sections, and generate a spectrogram or Mel spectrogram based on the voice signal in the time-frequency domain. can do.

Meanwhile, the voice post-processing unit 540 may determine a pitch change rate to change the pitch of the first voice signal. For example, to generate a voice with a pitch 1.25 times higher than the pitch of the first voice signal, the pitch change rate may be determined to be 1.25.

The voice post-processing unit 540 may generate voice signals in sections having a second window length based on the second hop length from the spectrogram. For example, the voice post-processing unit 540 may estimate phase information by repeatedly performing short-time Fourier transform and inverse short-time Fourier transform on the spectrogram, and generate voice signals of the sections based on the estimated phase information. You can. The voice post-processing unit 540 may generate a second voice signal with the pitch changed based on the voice signals of the above sections.

In order to change the pitch of the first voice signal, the voice post-processing unit 540 may set the ratio of the first window length to the second window length to be equal to the pitch change rate. For example, the first window length may correspond to a preset value, and the second window length may be set to be equal to the first window length divided by the pitch change rate. Alternatively, the second window length may correspond to a preset value, and the first window length may be set to be equal to the second window length multiplied by the pitch change rate. Meanwhile, the first hop length and the second hop length may be the same, but are not limited thereto.

For example, in (b) of FIG. 8, the first voice signal generated by the vocoder 530 and the first voice signal have a sampling rate of 24000, a first window length of 0.05 seconds, and a first hop length of 0.0125 seconds. Shows a spectrogram generated by performing a short-time Fourier transform. The generated spectrogram can have 601 frequency arrays up to 12000hz at 20hz intervals.

Meanwhile, when the pitch change rate is 1.25, the frequency components above 9600hz become 12000hz after the pitch change, so the frequency components above 9600hz may be lost. Therefore, pitch change can be performed only for frequency components below 9600hz, and there can be 481 frequency arrays up to 9600hz at 20hz intervals. That is, when generating a voice signal whose pitch has been corrected by 1.25 times from the spectrogram, in order to use only 481 frequency arrays, the second window length can be set to 0.04 seconds, which is the value of the first window length divided by the pitch change rate of 1.25. there is. FIG. 8(c) generates voice signals in sections with a second window length of 0.04 seconds based on a second hop length of 0.0125 seconds with respect to the spectrogram in FIG. 8(b), and generates voice signals in sections with a second window length of 0.04 seconds. This can represent a second voice signal whose pitch has been changed by 1.25 times.

Alternatively, when the pitch change rate is 0.75, the remaining 200 frequency components can be zero padded to increase the size of the 601 frequency array to 801. Therefore, the second window length can be set as the value of the first window length divided by the pitch change rate of 0.75. Figure 8 (a) may represent a second voice signal in which the pitch of the voice signal in Figure 8 (b) is changed by 0.75 times.

As described above, in order to correct the speed of the first voice signal generated by the vocoder 530, the ratio of the first hop length to the second hop length can be set to be equal to the value of the double speed rate. Additionally, in order to correct the pitch of the first voice signal generated by the vocoder 530, the ratio of the first window length to the second window length can be set to be equal to the value of the pitch change rate.

In combination with this, the speed and pitch of the first voice signal generated by the vocoder 530 can be corrected simultaneously. For example, when the ratio of the first hop length to the second hop length is set to be equal to the value of the double speed rate, and the ratio of the first window length to the second window length is set to be equal to the value of the pitch change rate, 1 The speed of the voice signal can be changed by the double speed rate, and the pitch of the first voice signal can be changed by the pitch change rate.

Figure 9 is a flowchart showing an embodiment of a method for changing the speed and pitch of a voice signal.

Referring to FIG. 9, in step 810, the voice post-processing unit may set sections having a first window length based on the first hop length in the first voice signal.

In step 820, the speech post-processing unit may generate a spectrogram by performing short-time Fourier transform on the sections.

For example, the voice post-processing unit may perform Fourier transform on each section to generate a voice signal in the time-frequency domain. The speech post-processing unit can take the absolute value of the speech signal in the time-frequency domain, cancel out the phase information, and generate a spectrogram containing only size information.

In step 830, the voice post-processing unit may determine a speed and pitch change rate for changing the speed and pitch of the first voice signal. For example, in order to generate a voice that is twice as fast as the speed of the first voice signal generated by the vocoder, the speed ratio may be set to 2. Alternatively, in order to generate a voice with a pitch 1.25 times higher than the pitch of the first voice signal generated by the vocoder, the pitch change rate may be determined to be 1.25.

In step 840, the voice post-processing unit may generate voice signals of sections having a second window length based on the second hop length from the spectrogram.

For example, the voice post-processing unit may estimate phase information by repeatedly performing short-time Fourier transform and inverse short-time Fourier transform on the spectrogram. For example, the voice post-processing unit may use the Griffin-Lim algorithm, but is not limited thereto. The voice post-processing unit may generate voice signals of sections having a second window length based on the estimated phase information and the second hop length.

In step 840, the voice post-processing unit may generate a second voice signal whose speed and pitch are changed based on the voice signals of the sections.

For example, the voice post-processing unit may add up all the voice signals of the sections and finally generate a second voice signal in which the speed and pitch of the first voice signal are changed by the speed rate and pitch change rate, respectively.

Various embodiments of the present disclosure may be implemented as software (eg, a program) including one or more instructions stored in a storage medium that can be read by a machine. For example, the processor of the device may call at least one instruction among one or more instructions stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves). This term refers to cases where data is stored semi-permanently in the storage medium. There is no distinction between temporary storage cases.

In this specification, a “part” may be a hardware component, such as a processor or circuit, and/or a software component executed by the hardware component, such as a processor.

The description of the present specification described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. You will be able to. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of this embodiment is indicated by the patent claims described later rather than the detailed description above, and should be interpreted to include all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts.

Claims (4)

Setting sections having a first window length based on a first hop length in the first voice signal;
Generating a spectrogram by performing short-time Fourier Transform on the sections;
determining a speed and pitch change rate for changing the speed and pitch of the first voice signal;
generating voice signals in sections having a second window length based on the second hop length from the spectrogram; and
Generating a second voice signal with changed speed and pitch based on the voice signals of the sections,
The ratio of the first hop length to the second hop length is set to be equal to a value of the double speed rate, and the ratio of the first window length to the second window length is set to be equal to a value of the pitch change rate. According to claim 1,
The method wherein the value of the second hop length corresponds to a preset value, and the first hop length is set to be equal to a value obtained by multiplying the second hop length by the speed ratio. According to claim 1,
The method wherein the value of the first window length corresponds to a preset value, and the second window length is set to be equal to the first window length divided by the pitch change rate. According to claim 1,
The step of generating voice signals of sections having the second window length includes:
estimating phase information by repeatedly performing short-time Fourier transform and inverse short-time Fourier transform on the spectrogram; and
A method comprising generating voice signals of sections having a second window length based on the second hop length based on the phase information.