KR20210125677A - Device, method and computer program of generating voice for training data - Google Patents

Abstract

A device for generating a voice for learning data comprises: an image conversion part that converts a voice into a first spectrogram; a simulation part that uses a pre-learned deep learning model to generate a plurality of simulation border images similar to a border of the first spectrogram, and generates a plurality of simulation color images similar to a color of the first spectrogram; and a voice generating part that combines the plurality of simulation border images and the plurality of simulation color images to generate a plurality of second spectrograms, and converts the plurality of second spectrograms into voices to generate a plurality of voices for learning data. Therefore, the present invention is capable of reducing the cost of collecting voices for learning data in machine learning.

Description

Voice generating apparatus, method and program for training data

The present invention relates to an apparatus, method and program for generating speech for training data in machine learning.

In order to use voice for machine learning, a certain amount of voice for training data must be secured. In other words, in order to improve the performance of speech machine learning, various types and vast amounts of speech are required.

In particular, since a single voice, such as a human scream, has a very large variation in characteristics such as height, intensity, and pattern for each person, machine learning needs to be learned through a larger amount of voice.

However, astronomical costs of hundreds of millions of units are consumed to collect voices such as human screams.

Korean Patent Publication No. 10-2019-0106902 (published on September 18, 2019)

The present invention is to solve the problems of the prior art described above, and to provide a voice generating apparatus for learning data, a method and a program capable of reducing the cost of collecting the voice for the learning data of machine learning.

However, the technical problems to be achieved by the present embodiment are not limited to the technical problems described above, and other technical problems may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention, an image conversion unit for converting a voice into a first spectrogram; a copying unit for generating a plurality of simulated border images similar to the border of the first spectrogram by using the pre-learned deep learning model, and generating a plurality of simulated color images similar to the color of the first spectrogram; and generating a plurality of second spectrograms by combining the plurality of simulated border images and the plurality of simulated color images, and converting the plurality of second spectrograms into voices to generate a plurality of voices for learning data. It is possible to provide a voice generating apparatus for learning data, which includes a unit.

Another embodiment of the present invention, an image conversion step of converting speech into a first spectrogram; A simulation step of generating a plurality of simulated border images similar to the border of the first spectrogram using the pre-learned deep learning model; A simulation step of generating a plurality of simulated color images similar to the color of the first spectrogram by using the pre-learned deep learning model; generating a plurality of second spectrograms by combining the plurality of simulated border images and the plurality of simulated color images; It is possible to provide a method for generating a voice for learning data, comprising the step of converting the plurality of second spectrograms into voices to generate a plurality of voices for the training data.

Another embodiment of the present invention, when a computer program is executed by a computing device, converts a voice into a first spectrogram, and uses a pre-trained deep learning model to simulate a plurality of similar to the border of the first spectrogram A border image is generated, a plurality of simulated color images similar to the color of the first spectrogram are generated and simulated, and a plurality of second spectrograms are generated by combining the plurality of simulated border images and the plurality of simulated color images. and a sequence of instructions for converting the plurality of second spectrograms into speech to generate speech for a plurality of learning data, the computer program stored in the medium may be provided.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, it is possible to reduce the cost of collecting speech for machine learning training data by artificially synthesizing a vast amount of speech from the minimum speech for training data.

1 is a block diagram of an apparatus for generating a voice for learning data according to an embodiment of the present invention.
2 is an exemplary diagram for explaining a voice according to an embodiment of the present invention.
3 is a diagram exemplarily illustrating a divided border and a color image according to an embodiment of the present invention.
Figure 4 is an exemplary view for explaining the edge simulating part according to an embodiment of the present invention.
5 is an exemplary view for explaining a color simulating unit according to an embodiment of the present invention.
6 is an exemplary view for explaining a voice generator according to an embodiment of the present invention.
7 is a flowchart of a method of generating a voice for training data according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . Also, when a part "includes" a component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware.

Some of the operations or functions described as being performed by the terminal or device in the present specification may be instead performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an apparatus for generating a voice for learning data according to an embodiment of the present invention. FIG. 1 exemplarily shows minimal components that can be controlled by the apparatus 100 for generating speech for training data.

Referring to FIG. 1 , the voice generating apparatus 100 for training data may include a database 110 , an image converting unit 120 , a copying unit 130 , a voice generating unit 140 , and a learning unit 150 . can The imitation part 130 may include an edge imitation part 132 and a color imitation part 134 .

The apparatus 100 for generating a voice for training data may generate a plurality of artificial voices for training data having similar characteristics based on the collected voice.

The database 110 may store and manage voices collected through various paths. In addition, the database 110 may store and manage the generated artificial voice for learning data. For example, in the present application, the voice may be a single voice, such as a scream of a person or a sound of a door closing, but is not necessarily limited thereto.

Also, the database 110 may collect only a minimum number of voices for generating artificial voices for training data.

The database 110 may separate only a single voice, which is a target voice, from among the collected voices, and manage the corresponding data by dividing the data into a target voice and a non-target voice.

The database 110 may use a non-target voice, that is, a non-target voice, as a kind of 'False Data' in order to train the neural network model through the collected voice for the target voice.

In order to supplement the amount of data, the database 110 may manage the separated target voice by dividing it into learning and imitation. The database 110 may provide an effect of bootstrap aggregating (Bagging: bootstrap aggregating) data for the target voice by frequently switching the divided target voices for learning and imitation.

For example, if it is assumed that human screams A, B, C, and D are stored in the database 110 and a deep learning model is trained using them, the database 110 sets A and B to C for learning. and D can be used as imitations. At this time, the voice apparatus 100 for training data trains A and B on the deep learning model and if the generated simulation result is similar to C or D, the simulation result may be evaluated as excellent. In addition, the database 110 may use A and C for learning and B and D for imitation, and using this, it is possible to learn a deep learning model and compare and determine the generated simulation result. In this way, the database 110 can exhibit the effect of learning a lot of heterogeneous data with little data by repeatedly learning the deep learning model by modifying the same type of data that is stored and managed.

The image converter 120 may convert the voice into the first spectrogram. For example, the image conversion unit 120 may convert a single voice, such as a scream of a person, a sound of a wooden door squeaking, and a sound of an iron door closing with a 'thump', into the first spectrogram.

A spectrogram may be combined with features of a waveform and a spectrum. A spectrogram image is three-dimensional data composed of time on the X-axis, amplitude on the Y-axis, and frequency on the Z-axis. In the spectrogram image, a difference in amplitude according to a change in a time axis and a frequency axis may be displayed as a difference in print density and display color.

The image converter 120 may divide the first spectrogram into a border image expressing the border of the first spectrogram and a color image expressing the color of the first spectrogram.

Referring to FIG. 3 , the image converter 120 converts the first spectrogram (eg, 200 in FIG. 2 ) to a border image 310 expressing the border of the first spectrogram and the color of the first spectrogram. It can be divided into color images 320 .

The image converting unit 120 divides the first spectrogram into borders and colors, and inputs the divided first spectrogram to the deep learning model in the replicating unit 130 to artificially generate a simulated image.

The replica unit 130 may generate a plurality of simulated border images similar to the border of the first spectrogram by using the pre-learned deep learning model, and may generate a plurality of simulated color images similar to the color of the first spectrogram. . The imitation part 130 may include an edge imitation part 132 and a color imitation part 134 .

The voice generator 140 generates a plurality of second spectrograms by combining a plurality of simulated border images and a plurality of simulated color images, and converts the plurality of second spectrograms into voices to generate a plurality of voices for learning data. can do.

The learning unit 150 learns the first deep learning model to generate a plurality of simulated border images by inputting a plurality of border images 310 into the first deep learning model, and applies the plurality of color images to the second deep learning model. The second deep learning model can be trained to generate a plurality of simulated color images by input.

Referring briefly to FIG. 2 , a voice may be expressed as an image as shown in FIG. 2 . In particular, a single voice, such as a scream of a person, has a characteristic that a sound is suddenly generated at a specific time. Such a one-shot voice may include a sharp cut surface (refer to reference numeral 200) perpendicular to the time axis (x-axis). Also, since the single speech is single, the right side of the image (refer to reference numeral 200) ends within a finite range.

Even if a single voice belongs to the same category as a human scream, the shape or color may be different when the waveform image is visually checked. In such a single voice, it is very difficult for a person to recognize or judge information about the voice by directly viewing the waveform image. Therefore, it is difficult to artificially generate a voice such as a scream.

The replicating unit 130 may replicate the first spectrogram image by using the pre-learned deep learning model. Here, the deep learning model may be a generative adversarial network (GAN). For example, the deep learning model may be a Deep Convolutional Generative Adversarial Network (DCGAN).

The replica unit 130 may generate a plurality of replica images similar to the borders and colors of the first spectrogram based on the generative adversarial neural network. A generative adversarial neural network is a neural network in which competitive learning is applied to a deep neural network.

A generative adversarial neural network is a model that learns to generate a similar sample using a patch unit sample randomly extracted from a real image and learns to determine the authenticity of the generated similar sample.

Such a generative adversarial neural network includes a generator that generates a similar sample from a real image and a discriminator that determines the authenticity of the similar sample, and each of the generator and the discriminator can influence each other while learning.

Looking at the structure of the constructor, the constructor can be formed by stacking a fully connected convolutional layer with 64 channels, a batch normalization layer, and a Rectified Linear Unit (ReLU) layer 8 times. At this time, a depth convolutional layer is used in the structure of the generator to reduce the total number of parameters in the structure of the adversarial generative neural network. Here, since the depth convolution layer has a feature of combining the result values of each channel into one, it is possible to reduce the number of parameters while maintaining the characteristics of each layer.

Looking at the structure of the discriminator, the discriminator can be formed in a structure in which a fully connected convolutional layer, a batch normalization layer, and a Leaky ReLU layer are stacked 5 times.

On the other hand, deep convolutional generative adversarial neural networks are deep learning models with excellent ability to learn the boundaries of objects. In addition, deep convolutional generative adversarial neural networks can generate similar images in large numbers by adjusting dummy parameter values.

As such, the deep convolutional generative adversarial neural network can be utilized to learn patterns of image boundaries and colors that satisfy specific conditions. In addition, the deep convolutional generative adversarial neural network can be trained to artificially generate similar images by analyzing the patterns of borders and colors of images through learning.

The deep convolutional generative adversarial neural network can generate continuous artificial images by adjusting the numerical parameters when both learning of the border pattern and the color pattern are completed.

Such a deep convolutional generative adversarial neural network can learn the spectrogram of speech and synthesize an artificial image similar to the spectrogram. When the synthesized artificial image is converted back into speech, speech that is very similar to real speech and has various characteristics can be generated.

For example, a deep convolutional generative adversarial neural network modulates input parameters to detect the sound of an intermediate door from blunt to light closing, from spectrograms for 'squeaking' wooden doors and 'thumping' iron doors. You can create spectrograms for various sounds.

The replica unit 130 may include a frame replica unit 132 for generating a plurality of replica frame images by inputting an edge image to the first deep learning model.

The frame replicating unit 132 may artificially generate a plurality of replica frame images by inputting the frame image divided by the image conversion unit 120 into the first deep learning model.

The replica unit 130 may include a color replica unit 134 for generating a plurality of replica color images by inputting a color image to the second deep learning model.

The color simulating unit 134 may artificially generate a plurality of simulated color images by inputting the color image divided by the image converting unit 120 into the second deep learning model.

Figure 4 is an exemplary view for explaining the edge simulating part according to an embodiment of the present invention.

First, the learning unit 150 analyzes what type of boundary line pattern is shown in the first spectrogram in which the target voice is converted based on the target voice of the database 110, and the first deep learning to simulate it. The model 410 may be trained.

For example, the learning unit 150 inputs the border image 420 into the first deep learning model 410, and the first deep learning model 410 to generate a plurality of simulated border images similar to the border image 420. ) can be learned.

Thereafter, the border replicating unit 132 may artificially generate and output a plurality of simulated border images 430 similar to the border image 420 based on the learned first deep learning model 410 .

For example, the border replica 132 inputs a border image 420 featuring three peaks to the trained first deep learning model 410, similar to the border image 420 with three peaks. It is possible to generate and output a plurality of simulated border images 430 featuring three peaks.

5 is an exemplary view for explaining a color simulating unit according to an embodiment of the present invention.

First, the learning unit 150 analyzes what type of energy pattern the target voice shows in the converted first spectrogram based on the target voice of the database 110, and the second deep learning to simulate it. The model 510 may be trained.

For example, the learning unit 150 inputs the color image 520 into the second deep learning model 510 to generate a plurality of simulated color images similar to the color image 520 . ) can be learned.

Thereafter, the color simulation unit 134 may artificially generate and output a plurality of simulated color images 530 similar to the color image 520 based on the learned second deep learning model 510 .

6 is an exemplary view for explaining a voice generator according to an embodiment of the present invention.

Referring to FIG. 6 , the voice generator 140 combines a plurality of simulated border images 610 and a plurality of simulated color images 620 generated based on the first spectrogram to obtain a plurality of second spectrograms 630 . ) and may generate a plurality of voices 640 for training data based on the plurality of second spectrograms 630 .

For example, the voice generating unit 140 may integrate the results 610 and 620 generated through the edge replicating unit 132 and the color replicating unit 134 . Specifically, the voice generator 140 generates a second spectrogram 630 by parallelly integrating a plurality of simulated border images 610 and a plurality of simulated color images 620 generated based on the first spectrogram. can create

The voice generator 140 may generate a plurality of voices 640 for training data by unifying the second spectrogram 630 and performing an inverse operation on the unified second spectrogram 630 to make the voices.

For example, the voice generating unit 140 may inversely code the second spectrogram generated by integrating the results of the edge replicating unit 132 and the color replicating unit 134 into a sound waveform and outputting it as a voice file. Specifically, the voice generating unit 140 creates an artificial wave by combining the results of the generated border replicating unit 132 and the color replicating unit 134, and combining and restoring the corresponding artificial waves in parallel to form one voice file. can be created and output the corresponding audio file.

7 is a flowchart of a method of generating a voice for training data according to an embodiment of the present invention. The method for generating a voice for training data shown in FIG. 7 includes steps processed in time series according to the embodiments shown in FIGS. 1 to 6 . Therefore, even if omitted below, it is also applied to the method of generating the voice for the training data in the voice generating server for the training data according to the embodiment shown in FIGS. 1 to 6 .

In step S710, the apparatus for generating speech for training data may convert speech into a first spectrogram.

In step S720, the apparatus for generating speech for training data may generate a plurality of simulated border images similar to the borders of the first spectrogram by using the pre-learned deep learning model.

In step S730, the apparatus for generating speech for training data may generate a plurality of simulated color images similar to the color of the first spectrogram by using the pre-learned deep learning model.

In step S740, the apparatus for generating speech for training data may generate a plurality of second spectrograms by combining a plurality of simulated border images and a plurality of simulated color images.

In operation S750, the apparatus for generating speech for training data may generate a plurality of speech for training data by converting the plurality of second spectrograms into speech.

In the above description, steps S710 to S750 may be further divided into additional steps or combined into fewer steps according to an embodiment of the present invention. In addition, some steps may be omitted if necessary, and the order between the steps may be switched.

The method for generating a voice for learning data in the voice generating server for learning data described through FIGS. 1 to 6 is in the form of a recording medium including a computer program stored in a medium executed by a computer or instructions executable by a computer. can also be implemented. In addition, the method for generating a voice for learning data in the voice generating server for learning data described with reference to FIGS. 1 to 6 may be implemented in the form of a computer program stored in a medium executed by a computer.

Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The description of the present invention described above is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

100: speech generating device for training data
110: database
120: image conversion unit
130: copycat
140: voice generator
150: study department

Claims (13)

An apparatus for generating speech for machine learning training data, the apparatus comprising:
an image converter for converting voice into a first spectrogram;
a copying unit for generating a plurality of simulated border images similar to the border of the first spectrogram by using the pre-learned deep learning model, and generating a plurality of simulated color images similar to the color of the first spectrogram; and
A voice generator for generating a plurality of second spectrograms by combining the plurality of simulated border images and the plurality of simulated color images, and converting the plurality of second spectrograms into voices to generate a plurality of voices for learning data
A voice generating device for training data that includes.
The method of claim 1,
The image converting unit divides the first spectrogram into an edge image expressing an edge of the first spectrogram and a color image expressing a color of the first spectrogram.
3. The method of claim 2,
The imitation unit,
A voice generating apparatus for learning data, comprising a frame replica for generating the plurality of simulated frame images by inputting the frame image to the first deep learning model.
4. The method of claim 3,
The imitation unit,
A color simulating unit for generating the plurality of simulated color images by inputting the color image to a second deep learning model
Which will further include a voice generating device for training data.
5. The method of claim 4,
By inputting a plurality of the border images into the first deep learning model, the first deep learning model is trained to generate the plurality of simulated border images, and a plurality of the color images are input to the second deep learning model, and the A learning unit that trains the second deep learning model to generate a plurality of simulated color images
Which will further include a voice generating device for training data.
The method of claim 1,
The deep learning model is a generative adversarial network (GAN), a voice generating device for training data.
A method for generating speech for machine learning training data, the method comprising:
an image conversion step of converting speech into a first spectrogram;
A simulation step of generating a plurality of simulated border images similar to the border of the first spectrogram using the pre-learned deep learning model;
A simulation step of generating a plurality of simulated color images similar to the color of the first spectrogram by using the pre-learned deep learning model;
generating a plurality of second spectrograms by combining the plurality of simulated border images and the plurality of simulated color images;
converting the plurality of second spectrograms into voices to generate a plurality of voices for learning data
A method for generating speech for training data, comprising:
8. The method of claim 7,
In the image conversion step, the first spectrogram is divided into an edge image expressing an edge of the first spectrogram and a color image expressing a color of the first spectrogram.
9. The method of claim 8,
The simulation step is
A method for generating speech for training data, including a frame simulating step of generating the plurality of simulated border images by inputting the border image to the first deep learning model.
10. The method of claim 9,
The simulation step is
The method for generating voice for training data further comprising a color simulating step of generating the plurality of simulated color images by inputting the color image to a second deep learning model.
11. The method of claim 10,
By inputting a plurality of the border images into the first deep learning model, the first deep learning model is trained to generate the plurality of simulated border images, and a plurality of the color images are input to the second deep learning model, and the training the second deep learning model to generate a plurality of simulated color images.
The method of generating a voice for training data further comprising a.
8. The method of claim 7,
The deep learning model is a generative adversarial network (GAN), a voice generation method for training data.
A computer program stored on a computer readable medium comprising a sequence of instructions for generating speech for machine learning training data, the computer program comprising:
When the computer program is executed by a computing device,
converting speech into a first spectrogram,
A plurality of simulated border images similar to the border of the first spectrogram are generated using the pre-learned deep learning model, and a plurality of simulated color images similar to the color of the first spectrogram are generated and simulated,
Commands for generating a plurality of second spectrograms by combining the plurality of simulated border images and the plurality of simulated color images, and converting the plurality of second spectrograms into voices to generate a plurality of voices for learning data A computer program stored on a medium comprising a sequence.

Images