KR102480360B1 - Apparatus, method and computer program for generating synthesized sound source using learning through image - Google Patents

Abstract

An apparatus for generating a synthesized sound source using learning through images includes an input unit for receiving input data including a sample sound source in which a user's voice is recorded and text for generating a synthesized sound source, and the input sample sound source is transmitted to the sample sound source. A first image generating unit that converts a frequency spectrogram into a frequency spectrogram including frequency characteristics of the converted frequency spectrogram and generates a first image based on the converted frequency spectrogram, and corresponds to the text from the generated first image using a learning model and a second image inference unit for inferring a second image to generate a synthesized sound source for generating a synthesized sound source based on the inferred second image and the sample sound source.

Description

Apparatus, method and computer program for generating synthesized sound sources using learning through images

The present invention relates to an apparatus, method, and computer program for generating a synthesized sound source using learning through images.

Speech synthesis technology refers to a technology in which a machine automatically analyzes a human voice and synthesizes it with text for generating a synthesized sound source to artificially create speech sounds. Voice synthesis technology is representative of voices used in subways and buses.

Regarding such voice synthesis technology, Korea Patent Registration No. 10-1880378, which is a prior art, discloses a voice synthesis method and apparatus.

Conventionally, in order to generate a synthesized sound source based on the user's voice, a voice file in which the user's voice is recorded is converted into a Mel spectrogram, and then the magnitude of the Mel spectrogram and the input text are machine learning. It was put into the input of and learned. The MEL spectrogram is data obtained by analyzing frequency characteristics of voice, and includes voice height information.

Thereafter, the Mel spectrogram was inferred and a synthesized sound source file (wav) was generated using a vocoder such as a griffin-lim algorithm or a wavenet, for example.

However, the conventional method has a problem of overfitting and low accuracy in inferring the Mel spectrogram by learning by inputting the size of the Mel spectrogram and the input text as input to the machine learning.

A sample sound source in which the user's voice is recorded is converted into a first image through a frequency spectrogram, and a second image corresponding to the text is inferred from the converted image, so that a synthesized sound source can be generated using learning through images. It is intended to provide a synthetic sound source device, method, and computer program for

An object of the present invention is to provide a synthesized sound source device, method, and computer program capable of generating a synthesized sound source based on the inferred second image and sample sound source.

An object of the present invention is to provide a synthesized sound source device, method, and computer program that generate a synthesized sound source based on the user's voice by reflecting the user's voice characteristics on the synthesized sound source.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problems described above, and other technical problems may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention is an input unit for receiving input data including text for generating a sample sound source in which a user's voice is recorded and a synthesized sound source, and the input sample sound source A first image generating unit that converts to a frequency spectrogram including frequency characteristics of the sample sound source and generates a first image based on the converted frequency spectrogram, and the generated first image using a learning model An apparatus for generating a synthesized sound source comprising a second image inference unit for inferring a second image corresponding to the text from and a synthesized sound source generator for generating a synthesized sound source based on the inferred second image and the sample sound source. there is.

Another embodiment of the present invention is a step of receiving input data including text for generating a sample sound source in which a user's voice is recorded and a synthesized sound source, the input sample sound source including frequency characteristics of the sample sound source Converting to a frequency spectrogram, generating a first image based on the converted frequency spectrogram, inferring a second image corresponding to the text from the generated first image using a learning model, and A method for generating a synthesized sound source may be provided, including generating a synthesized sound source based on the inferred second image and the sample sound source.

Another embodiment of the present invention, when the computer program is executed by a computing device, receives input data including text for generating a sample sound source in which a user's voice is recorded and a synthesized sound source, and generates the input sample sound source. Converting to a frequency spectrogram including frequency characteristics of the sample sound source, generating a first image based on the converted frequency spectrogram, and corresponding to the text from the generated first image using a learning model A computer program stored in a medium including a sequence of instructions for inferring a second image and generating a synthesized sound source based on the inferred second image and the sample sound source may be provided.

The above-described means for solving the problems is only illustrative and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, a sample sound source in which a user's voice is recorded is converted into a first image based on a frequency spectrogram, and a second image is generated based on the converted first image and an input test. By reasoning, it is possible to provide a synthesized sound source generation device, method, and computer program capable of generating a synthesized sound source through learning through images.

Apparatus and method for generating a synthesized sound source capable of preventing an overfitting problem that may occur during learning and improving accuracy in an image inference process necessary for generating a synthesized sound source by performing learning through images and a computer program.

An apparatus, method, and computer program for generating a synthesized sound source capable of generating a synthesized sound source based on a user's tone by extracting voice characteristics of a user and inserting the extracted voice characteristics into a synthesized sound source may be provided.

1 is a block diagram of an apparatus for generating a synthesized sound source according to an embodiment of the present invention.
2A and 2B are exemplary diagrams illustrating frequency spectrograms converted from sample sound sources according to an embodiment of the present invention.
3 is a diagram showing a 4-dimensional array for extracting RGB data from a first image according to an embodiment of the present invention.
4 is a flowchart of a method for generating a synthesized sound source by using learning through images in an apparatus for generating a synthesized sound source according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, this means that it may further include other components, not excluding other components, unless otherwise stated, and one or more other characteristics. However, it should be understood that it does not preclude the possibility of existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

In this specification, a "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Further, one unit may be realized using two or more hardware, and two or more units may be realized by one hardware.

In this specification, some of the operations or functions described as being performed by a terminal or device may be performed instead by a server connected to the terminal or device. Likewise, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the corresponding server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an apparatus for generating a synthesized sound source according to an embodiment of the present invention. Referring to FIG. 1 , the synthetic sound generator 100 includes an input unit 110, a first image generator 120, an RGB extraction unit 130, a second image inference unit 140, and a tone feature extraction unit 150. ) and a synthesized sound generator 160.

The input unit 110 may receive input data including text for generating a sample sound source recorded with a specific voice and a synthesized sound source.

The first image generating unit 120 may convert the input sample sound source into a frequency spectrogram including frequency characteristics of the sample sound source. Here, the frequency spectrogram may be a Mel spectrogram. A process of converting a sample sound source into a frequency spectrogram will be described in detail with reference to FIGS. 2A and 2B.

2A and 2B are exemplary diagrams illustrating frequency spectrograms converted from sample sound sources according to an embodiment of the present invention.

The first image generating unit 120 may convert the input sample sound source into time series data and then convert the time series data into a frequency spectrogram. Here, the type of time series data may be set to a float in which the number of significant digits is 6 to 7 digits and the byte is 4 digits among approximate numeric data formats used for floating point number data.

The first image generating unit 120 extracts the magnitude of a frequency from the time-series data by using, for example, Short-Time Fourier Transform (STFT), and converts the time-magnitude domain into a time-magnitude domain. An array of frequency spectrograms may be generated by transforming into a time-frequency domain.

Here, the STFT means that an input signal is converted into a corresponding spectrum, and a Fourier Transform (FT) that converts a function in the time domain into a function in the frequency domain is applied to an actual recorded sound.

At this time, the parameters used are, for example, as follows.

[parameter]

sample_rate=22050Hz, Channel=80

Frame_shift_ms=12.5ms

Frame_length_ms=50ms

Hop_size=275(frmae_shift_ms*sample_rate)

win_size=1100(frame_length_ms*sample_rate)

Referring to FIG. 2A, the first image generating unit 120 generates a two-dimensional array (200) with time on the x-axis and frame on the y-axis based on the shape data of the MEL spectrogram (eg, configured as (frame, channel)). ) can be created. For example, when the shape of the MEL spectrogram is (2000, 80), the first image generator 120 may generate the first image through the 2D array 200 .

Referring to FIG. 2B , the first image generating unit 120 generated through this process may generate a first image represented by a time axis 210 and a frequency axis 220 .

3 is a diagram showing a 4-dimensional array for extracting RGB data from a first image according to an embodiment of the present invention.

Referring to FIGS. 2B and 3 , the RGB extraction unit 130 may extract RGB data from the first image.

For example, the RGB extractor 130 may extract a 4-dimensional array 300 having an x-axis as time and a y-axis as a frame from the 2-dimensional array 200 . Here, each cell of the array may include R, G, and B values.

The RGB extraction unit 130 may extract R, G, and B data from the 4-dimensional array 300 .

The second image inference unit 140 may infer a second image corresponding to text from the first image by using a learning model. Here, the second image may be an image representing a MEL spectrogram.

For example, when the text is "hello", the second image reasoning unit 140 encodes "hello" as a number such as '5131...' and inputs it to the learning model along with the first image, thereby generating a second image. images can be inferred. At this time, the second image inference unit 140 may extract order characteristics between phonemes for the text input as numbers, and infer a second image to correspond to the phonemes of the text from the first image.

To this end, the second image inference unit 140 inputs the first image and various texts (eg, hello, nice to meet you, I love you, etc.) in advance and trains the learning model to generate images corresponding to each text. there is.

For example, the second image inference unit 140 may infer the second image by inputting the extracted RGB data to the learning model.

The second image inference unit 140 may derive an image shape based on the extracted RGB data and the shape of the frequency spectrogram, and rearrange the derived image shape to deduce the second image.

Returning to FIG. 1 again, the timbre feature extractor 150 may extract the user's voice feature from the input sample sound source. Here, the characteristics of the user's voice may include a pitch, an aperiodic spectrum, a harmonic spectrum, and the like. This is to insert the synthesized sound source into the second image since there is no frequency information about the user's voice when generating the synthesized sound source from the second image.

For example, the timbre feature extractor 150 converts the pitch (F0), which is the fundamental frequency of a periodic waveform (typically, the pitch in a voice signal exists between 40 and 400 Hz) into ACF (Autocorrelation) and AMDF (Average Magnitude). Difference Function), Ceptrum, etc.

For another example, the timbre feature extractor 150 performs inverse filtering on the sample sound source, extracts a residual frame, calculates a Discrete Fourier Transform (DFT), extracts a periodic spectrum, and extracts a zero-phase An aperiodic spectrum may be extracted based on a periodic frame synthesized based on zero-phase and a periodic exitation generated based on Pitch Synchronous Overlap and Add (PSOLA).

As another example, the timbre feature extractor 150 may extract the harmonic spectrum using an equation of H(k) = k*F0.

The synthesized sound source generator 160 may generate a synthesized sound source based on the inferred second image and the sample sound source. In this case, the synthesized sound source generator 160 may generate a synthesized sound source by further reflecting the extracted characteristics of the user's voice.

Through this process, it is possible to generate a synthesized sound source to represent the user's voice of good quality.

The apparatus 100 for generating a synthesized sound source may be executed by a computer program stored in a medium including a sequence of instructions for generating a synthesized sound source using learning through images. When the computer program is executed by a computing device, the computer program receives input data including text for generating a sample sound source in which a user's voice is recorded and a synthesized sound source, and converts the input sample sound source to a frequency including frequency characteristics of the sample sound source. Converting to a spectrogram, generating a first image based on the converted frequency spectrogram, inferring a second image corresponding to text from the first image generated using a learning model, and inferring the inferred second image and sample It may include a sequence of instructions to generate a synthesized sound source based on the sound source.

4 is a flowchart of a method for generating a synthesized sound source by using learning through images in an apparatus for generating a synthesized sound source according to an embodiment of the present invention. A method of generating a synthesized sound source by using learning through images in the apparatus 100 for generating a synthesized sound source shown in FIG. It includes the steps processed by Therefore, even if the content is omitted below, it is also applied to the method of generating a synthesized sound source using learning through images in the apparatus 100 for generating a synthesized sound source according to the embodiment shown in FIGS. 1 to 3 .

In step S410, the synthesized sound source generating apparatus 100 may receive input data including a sample sound source in which the user's voice is recorded and text for generating the synthesized sound source.

In step S420, the synthetic sound source generating apparatus 100 may convert the input sample sound source into a frequency spectrogram including frequency characteristics of the sample sound source.

In step S430, the apparatus 100 for generating a synthesized sound source may generate a first image based on the converted frequency spectrogram.

In step S440, the apparatus 100 for generating a synthesized sound source may infer a second image corresponding to the text from the generated first image using the learning model.

In step S450, the apparatus 100 for generating a synthesized sound source may generate a synthesized sound source based on the inferred second image and the sample sound source.

In the foregoing description, steps S410 to S450 may be further divided into additional steps or combined into fewer steps, depending on an embodiment of the present invention. Also, some steps may be omitted as needed, and the order of steps may be switched.

The method for generating a synthesized sound source by using learning through images in the synthetic sound source generating apparatus described with reference to FIGS. 1 to 4 is a recording medium including a computer program stored in a medium executed by a computer or instructions executable by a computer. It can also be implemented in the form of. In addition, the method for generating a synthesized sound source by using image-based learning in the apparatus for generating a synthesized sound source described with reference to FIGS. 1 to 4 may be implemented in the form of a computer program stored in a medium executed by a computer.

Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention. do.

100: synthetic sound source generating device
110: input unit
120: first image generator
130: RGB extraction unit
140: second image reasoning unit
150: tone feature extraction unit
160: synthetic sound generator

Claims (15)

An apparatus for generating a synthesized sound source using learning through images,
an input unit for receiving input data including text for generating a sample sound source in which a user's voice is recorded and a synthesized sound source;
a first image generating unit that converts the input sample sound source into a frequency spectrogram including frequency characteristics of the sample sound source, and generates a first image based on the converted frequency spectrogram;
a second image inference unit inferring a second image corresponding to the text from the generated first image using a learning model; and
A synthesized sound source generator for generating a synthesized sound source based on the inferred second image and the sample sound source
including,
Information on the text and the first image are input to the learning model, and the second image is inferred from the learning model;
Further comprising an RGB extraction unit for extracting RGB data from the first image,
wherein the second image inference unit infers the second image by inputting the extracted RGB data to the learning model.
According to claim 1,
A timbre feature extraction unit extracting characteristics of the user's voice from the input sample sound source,
The synthesized sound source generating unit generates the synthesized sound source by further reflecting the extracted characteristics of the user's voice.
According to claim 2,
The feature of the user's voice includes at least one of a pitch, an aperiodic spectrum, and a harmonic spectrum.
According to claim 1,
Wherein the frequency spectrogram is a Mel spectrogram.
delete According to claim 1,
Wherein the second image inference unit derives an image shape based on the extracted RGB data and the shape of the frequency spectrogram, and infers the second image by rearranging the derived image shape. .
According to claim 1,
wherein the second image reasoning unit learns the learning model to generate a plurality of images by inputting the first image and a plurality of texts.
A method for generating a synthesized sound source using learning through images in a synthesized sound source generating device,
Receiving input data including text for generating a sample sound source in which a user's voice is recorded and a synthesized sound source;
converting the input sample sound source into a frequency spectrogram including frequency characteristics of the sample sound source;
generating a first image based on the converted frequency spectrogram;
inferring a second image corresponding to the input text from the generated first image using a learning model; and
Generating a synthesized sound source based on the inferred second image and the sample sound source;
Information on the text and the first image are input to the learning model, and the second image is inferred from the learning model;
extracting RGB data from the first image; and
Further comprising the step of inferring the second image by inputting the extracted RGB data to the learning model.
According to claim 8,
extracting characteristics of the user's voice from the input sample sound source; and
Further comprising the step of generating the synthesized sound source by further reflecting the characteristics of the extracted user's voice.
According to claim 9,
The feature of the user's voice includes at least one of a pitch, an aperiodic spectrum, and a harmonic spectrum.
According to claim 8,
The frequency spectrogram is a Mel spectrogram, a method for generating a synthesized sound source.
delete According to claim 11,
deriving an image shape based on the extracted RGB data and the shape of the frequency spectrogram; and
Further comprising inferring the second image by rearranging the derived image shape.
According to claim 8,
Further comprising the step of training the learning model to generate a plurality of images by inputting the first image and a plurality of texts.
A computer program stored in a medium containing a sequence of instructions for generating a synthesized sound source using learning through images,
When the computer program is executed by a computing device,
Receive input data including text for generating a sample sound source and synthesized sound source in which the user's voice is recorded;
converting the input sample sound source into a frequency spectrogram including frequency characteristics of the sample sound source, and generating a first image based on the converted frequency spectrogram;
Inferring a second image corresponding to the input text from the generated first image using a learning model;
generating a synthesized sound source based on the inferred second image and the sample sound source;
A sequence of instructions for inferring the second image from the learning model by inputting information about the text and the first image to the learning model;
Extracting RGB data from the first image;
The command to infer the second image is,
A computer program stored in a medium including instructions for inferring the second image by inputting the extracted RGB data to the learning model.

Images