KR20210021719A - System and Method for Sound Translation Based on Generative Adversarial Networks - Google Patents

Abstract

Disclosed are an acoustic deformation system and method. In an embodiment of the present invention, the acoustic deformation system and method facilitate transformation, which mimics the target tone, by applying training based on self-attention module and characteristic loss function when transforming between mutually unpaired sounds, and increasing the reality of the deformed sound by maintaining elements such as pitch, length, silence, and vibration. The acoustic deformation system comprises: a first generator; a second generator; a first discriminator; a second discriminator; and a training unit.

Description

System and Method for Sound Translation Based on Generative Adversarial Networks}

The present invention relates to a sound transformation system and method using a generative confrontation network for implementing transformation between sounds that are mutually unpaired.

The contents described below merely provide background information related to the present invention and do not constitute the prior art.

When you want to hear a specific song with a specific singer's voice, but the realization of the situation is not realistically possible, sound translation is a useful method. However, in most cases, the targets for transformation are mutually unpaired, and sound transformation should be attempted in situations where information on lyrics or pitch is insufficient. The fact that it is practically difficult to obtain time-synchronized data from two different singers also makes acoustic deformation difficult.

As a conventional method, for example, in Patent Document 1 or Non-Patent Document 2, a learning model capable of transforming a style or shape between heterogeneous image domains has been proposed. However, even if it is possible to transform heterogeneous images, it takes a lot of time and cost to collect images that match each other. In the case of sound transformation in which timbre change is important, it is practically impossible to obtain time-synchronized data as described above, and thus, there have been few attempts.

As a conventional method, for example, in Non-Patent Document 3, after expressing an unpaired voice in the frequency domain (as if in an image format), using the learning model exemplified in Non-Patent Document 2, Attempted transformation. As a result of the trial, the change in prosody between voices before and after transformation was confirmed through a subjective verification method called MOS (Mean Opinion Score) test. In the case of a pitched sound, in addition to prosody, rich harmonic structures are clearly observed, so the harmonic structure is an important element of the tone expressing the sound. Accordingly, there is a need for an acoustic transformation method in which a block capable of reflecting a more harmonic structure and a cost function in consideration of a cost function so that transformation that simulates the target tone can be easily performed.

Patent Document 1: US Patent No. US 10275473 B2 (Method for learning cross-domain relations based on generative adversarial networks, 2019.04.30, registration)

Non-Patent Document 1: Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), 2014. Non-Patent Document 2: Zhu, J., Park, T., Isola, P., and Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. 2017. Non-Patent Document 3: Kaneko, T. and Kameoka, H. Cyclegan-vc: Non-parallel voice conversion using cycle-consistent adversarial networks. In EUSIPCO, pp. 2100-2104, 2018.

The present disclosure applies a training based on a self-attention module and a characteristic loss function when transforming between mutually unpaired sounds, thereby facilitating the transformation that simulates the target tone, while making the pitch, length, and silence ) And vibration are maintained, thereby providing an acoustic deformation system and method in which the reality of the deformed sound is increased.

According to an embodiment of the present invention, there is provided a sound transformation system for translating between sound data, comprising: a first generator configured to transform a first sound in a first domain into a first transformed sound; A second generator for transforming the second sound of the second domain into a second modified sound; A first discriminator configured to distinguish between the first sound of the first domain and the second modified sound modified by the second generator; A second discriminator configured to distinguish between the second sound in the second domain and the first modified sound transformed by the first generator; And a training unit for training the first generator, the second generator, the first discriminator, and the second discriminator, wherein the training unit trains the first generator and the second discriminator in opposition to each other, and the first It provides an acoustic transformation system, characterized in that training two generators and the first discriminator in opposition to each other.

In addition, according to another embodiment of the present invention, in a sound transformation system for translation between sound data, the first sound of the first domain is simulated by the target sound of the second domain, and the first transformation ( translated) a first generator that transforms into sound; And a second generator for transforming the second sound of the second domain into a second modified sound by simulating the target sound of the first domain, wherein the first generator and the second generator Based on a distance metric between 1 sound and a second modified sound modified by the second generator, and a distance metric between the second sound in the second domain and the first modified sound modified by the first generator, in advance Sound characterized by being trained Provides a transformation system.

Further, according to another embodiment of the present invention, in the learning method of the acoustic variant system implemented on, a computer to perform a sound (sound) transformation (translation) between the data, using a first generator (generator), a first Transforming the first sound of the domain into a first translated sound and transforming the first transformed sound into a second reconstruction sound using a second generator; Transforming a second sound of a second domain into a second modified sound using the second generator, and transforming the second modified sound into a first reconstructed sound using the first generator; Discriminating between the first sound of the first domain and the second modified sound transformed by the second generator using a first discriminator; Using a second distinguisher, the second sound of the second domain and the modified by the first generator Discriminating the first modified sound; And a distance metric between the first sound and the second modified sound, a distance metric between the second sound and the first modified sound, a distance metric between the first reconstructed sound and the second sound, and the second reconstructed sound and the It provides a method of learning a computer-implemented acoustic transformation system, comprising the process of training the first generator and the second generator based on part or all of the distance metric between the first sounds.

Further, according to another embodiment of the present invention, in the acoustic variant method implemented in, a computer to perform a sound (sound) transformation (translation) between the data, using a first generator (generator), the first domain 1 process of transforming the sound into a first translated sound; And transforming the second sound of the second domain into a second modified sound using a second generator, wherein the first generator and the second generator include the first sound and the second sound of the first domain. Characterized in that it is pre-trained based on a distance metric between the second modified sound modified by a generator and a distance metric between the second sound in the second domain and the first modified sound modified by the first generator Provides a sound transformation method.

In addition, according to another embodiment of the present invention, in order to execute each step of the learning method of the acoustic transformation system according to any one of claims 13 to 17, it is stored in a computer-readable, nonvolatile or non-transitory recording medium. Provide computer programs.

In addition, according to another embodiment of the present invention, a computer program stored in a nonvolatile or non-transitory recording medium is provided that can be read by a computer in order to execute each step of the sound modification method according to claim 18.

As described above, according to the present embodiment, when transforming between mutually unpaired sounds, by applying training based on a self-attention module and a characteristic loss function, the transformation that simulates the target tone is facilitated. Factors such as pitch, rhythm, silence, and vibration have an effect of increasing the reality of the modified sound by providing a sound modification system and method that can be maintained . In addition, if the technical device and method of the present embodiment are appropriately modified and used, it is possible to expand the field of application to a one-to-many or many-to-many type of inter-acoustic transformation.

1 is a block diagram of a system for transforming a vocal voice according to an embodiment of the present invention.
2 is a structural diagram of a learning model for a singing voice transformer according to an embodiment of the present invention.
3 is a diagram for explaining a location of a self attention module according to an embodiment of the present invention.
4 is a diagram showing a generator of a vocal voice modifier according to an embodiment of the present invention.
5 is a diagram showing the performance of a vocal voice modifier according to an embodiment of the present invention.
6 is a diagram illustrating a tone change by a vocal voice modifier according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to elements of each drawing, it should be noted that the same elements are assigned the same numerals as possible, even if they are indicated on different drawings. In addition, in describing the embodiments, when it is determined that a detailed description of a related known configuration or function may obscure the subject matter of the embodiments, a detailed description thereof will be omitted.

In addition, terms such as first, second, A, B, (a), (b) may be used to describe the constituent elements of the present embodiments. These terms are for distinguishing the constituent element from other constituent elements, and the nature, order, or order of the constituent element is not limited by the term. Throughout the specification, when a part'includes' or'includes' a certain element, it means that other elements may be further included rather than excluding other elements unless otherwise stated. . In addition, the'... Terms such as'sub' and'module' mean a unit that processes at least one function or operation, which may be implemented by hardware or software, or a combination of hardware and software.

DETAILED DESCRIPTION OF THE INVENTION The detailed description to be disclosed below together with the accompanying drawings is intended to describe exemplary embodiments of the present invention and is not intended to represent the only embodiments in which the present invention may be practiced.

The present invention proposes a new GAN (Generative Adversarial Networks)-based model capable of learning a relationship between domains with respect to domains to which mutually unpaired sounds belong.

Sound includes voice, singing voice, music and natural sound. In an embodiment of the present invention, a description will be given focusing on singing voice translation as an example of sound transformation.

The singing voice transforming device according to an embodiment of the present invention may transform the singing voice from one domain to another domain by using a relationship between learned domains.

1 is a block diagram of a system for transforming a vocal voice according to an embodiment of the present invention.

The singing voice transformation system 100 illustrated in FIG. 1 includes some or all of the input unit 110, the transformation unit 120, and the output unit 130. In this case, the components included in the vocal voice transforming system according to the present embodiment are not necessarily limited thereto. For example, a training unit (not shown) for training a learning model may be additionally provided on the singing voice transformation system, or may be implemented in a form that is interlocked with an external training unit.

The input unit 110 acquires necessary data in the process of performing a vocal-voice transformation and converts it into a form suitable for vocal-voice transformation.

For example, the input unit 110 according to the present embodiment accepts a singing voice belonging to a source domain in a time domain and then transforms it into data in a frequency domain. do. As a transform method, a Fast Fourier Transform (FFT) or a cepstrum transform may be used, but is not limited thereto.

In this embodiment, the input unit 110 expresses the vocal voice as two-dimensional data in the frequency domain using the principle of a spectrogram. First, M-order FFT is performed while overlapping and sliding part of a section to be performed with respect to the singing voice in the time domain to obtain M-order FFT vectors in the frequency domain. Next, N M-order vectors in the frequency domain are combined in the form of a column vector to generate an MxN matrix, which is two-dimensional data.

In the present embodiment, the expression “mutually unpaired” is used for singing voices belonging to two domains. This expression means that the two singing voices belong to different singers, and that the time-synchronization does not coincide with each other.

The transforming unit 120 performs translation between unmatched vocal voices.

The transforming unit 120 according to the present embodiment receives the data in the frequency domain converted by the input unit 110 as an input, and transforms the target tone into data in the mimic frequency domain. The transforming unit 120 uses a learning model based on a neural network previously learned by the training unit to transform the vocal voice. The structure of the learning model and the training process of the learning model will be described later.

The output unit 130 provides the modified vocal voice in an auditory form to a user of the vocal voice transforming system.

The output unit 130 according to the present exemplary embodiment receives the data modified in the frequency domain from the transform unit 120 and converts it into a modified song voice in the time domain through a synthesis process. Finally, the transformed data in the time domain is provided to the user of the vocal voice transforming system in an auditory form.

2 is a structural diagram of a learning model for a singing voice transformer according to an embodiment of the present invention.

The learning model shown in FIG. 2 is a model based on Generative Adversarial Networks (GAN) having a reconstruction path. The GAN-based system used in the learning model is a structure in which two GANs are coupled (coupling), and for specific details on the combination structure of GAN and GAN, refer to Patent Document 1, Non-Patent Document 1, or Non-Patent Document 2. Hope. Hereinafter, concepts used in the learning model according to the present embodiment will be mainly described.

GAN basically includes a generator and a discriminator. The generator performs mapping between the two domains. Depending on the elements constituting the domain, correspondence can be expressed in various ways, such as transformation, translation, and pairing, and in the case of vocal speech, the term transformation will be used.

Hereinafter, in the implementation of the GAN according to the present embodiment, the generator uses a convolutional neural network (CNN), and the distinguisher uses only the decoder part among the components of the CNN.

The role of the generator is to simulate data belonging to the codomain by generating similar data that the discriminator cannot distinguish. The generator is trained so that the data generated by the generator is mistaken and the discriminator outputs true (probability 1). The role of the distinguisher distinguishes between similar data generated by the generator and target data belonging to the conjugate domain. The discriminator is trained to output true (true, probability 1) for target data belonging to the conjugate domain, and to output false (probability 0) for similar data. Therefore, in the basic GAN, a generative adversarial loss function is defined based on the roles of the generator and the distinguisher, and the generator and the distinguisher are identified through unsupervised learning based on the defined loss function. Train.

The structure in which the two GANs are combined is a part separated by a dotted box in FIG. 2, and includes two generators and two distinguishers in order to effectively perform a one-to-one correspondence between two domains A and B. . Hereinafter, the two generators are represented by a first and a second generator, and the two distinctions are represented by a first and a second separator. In this example, domain A contains the singing voice of singer A, and domain B contains the singing voice of singer B. The first generator performs domain A to B and the second generator performs domain B to A transformation. In addition, the second discriminator distinguishes the output of the first generator and the target of domain B, and the first discriminator distinguishes the output of the second generator and the target of domain A.

Hereinafter, terms necessary to express the loss function according to the present embodiment are defined. First, the transformation from domain A to domain B performed by the first generator is denoted by G _AB , and on the contrary, the transformation from domain B to domain A performed by the second generator is denoted by G _BA. In addition, the functions of the first and second classifiers are denoted as D _A and D _{B, respectively.} The singing voices belonging to domains A and B are expressed as first and second singing voices, respectively, and x _{A and x B} are expressed as symbols. Regardless of the input, _{the output of G AB} is marked as starting _with'first ', and similarly, the output of G BA is marked as starting with '2'. In the basic GAN structure, the output of the generator is classified using the expression'transformation'. According to FIG. 2, it is assumed that a plurality of first generators and second generators are used, but this is for convenience in describing the learning model, and in actual implementation, each of the first and second generators is implemented as one. The training procedure of the learning model using each of the first and second generators will be described later with reference to FIG. 4.

To explain the functions of the generator and the distinguisher again using the above notations, first, the first generator generates a first modified song voice by transforming the first song voice belonging to domain A, and the second generator generates a first modified song voice. By transforming the second vocal voice, a second transformed vocal voice is generated. The second distinguisher distinguishes between the first modified vocal voice and the second vocal voice of the domain B, and the first distinguisher distinguishes between the second modified vocal voice and the first vocal voice of the domain A.

Since the main role of the generator is to generate similar data that is deformed so that the distinguisher cannot distinguish it, the loss function is expressed as the probability after the similar data passes through the classifier, and a negative sign is used to express it as a positive value. Use. The generation loss function for the first and second generators is expressed by Equation 1.

는 기대(expectation) 함수이다.here,

Is an expectation function.

Since the main role of the distinguisher is to distinguish between similar data generated by the generator and data belonging to the conjugate domain, the loss function is'probability after conjugated data passes through the classifier' and'probability after similar data passes through the classifier. Is expressed as the sum of the probability subtracted from the total probability 1. The discrimination loss function for the first and second discriminators is expressed by Equation 2.

Equations 1 and 2 are loss functions of a structure in which two GANs are combined. The training unit uses the loss function of Equation 1 and Equation 2 to train the generator and the distinguisher.

As shown in Fig. 2, when the reconstruction path is included, an additional loss is used for training of the generator. The reconstruction path is configured by reusing the first and second generators, and is a path for confirming how well the source vocal voice has been reconstructed after two consecutive transformations. In order to reconstruct the vocal voice, the second generator transforms the first transformed vocal voice to generate a second reconstructed vocal voice, and the first generator transforms the second transformed vocal voice to generate the first reconstructed vocal voice.

The loss function according to the addition of the reconstruction path is defined as a distance metric between the source vocal voice and the reconstructed vocal voice, and the reconstruction loss function for the first and second generators is expressed by Equation 3.

는 거리 메트릭이고, 거리 메트릭은 어떠한 형태의 메트릭(L1, L2, 코사인 유사도 등)이 사용되어도 무방하다. 재구성 경로까지 고려한 제1 및 제2 생성기에 대한 손실함수는 수학식 4로 표현한다. 트레이닝부는 제1 및 제2 생성기를 트레이닝시키기 위하여 수학식 4의 손실함수를 사용한다. here,

Is a distance metric, and any type of metric (L1, L2, cosine similarity, etc.) may be used as the distance metric. The loss function for the first and second generators considering the reconstruction path is expressed by Equation 4. The training unit uses the loss function of Equation 4 to train the first and second generators.

Equations 2 and 4 show satisfactory performance when applied to image pairing as the loss function used in Patent Document 1 or Non-Patent Document 2. However, as in the case of vocal speech transformation, when the tone transformation represented by harmonic structures is possible and maintenance of elements such as lyrics, pitch, breath, and vibration is required, training based on the above equation is insufficient. There is a side.

Therefore, in this embodiment, in order to further improve the transformation process, a feature path and an identity-mapping path may be added, and a self-attention module may be added in addition to this. have. Hereinafter, a portion added for improvement will be mainly described.

First, as shown in FIG. 2, the identity mapping path according to the present embodiment is a path for confirming how well the identity mapping of the source vocal voice is properly mapped, and is configured by reusing the first and second generators. By adding an identity mapping path, it helps to maintain elements such as lyrics, pitch, breath, and vibration of the source vocal voice. For identity mapping, the second generator transforms the first vocal voice to generate a second identity mapping vocal voice, and the first generator transforms the second vocal voice to generate the first identity mapping vocal voice.

The loss function according to the addition of the identity mapping path is defined as a distance metric between the source song speech and the identity mapping song speech, and the reconstruction loss function for the first and second generators is expressed by Equation 5.

Next, as shown in FIG. 2, the characteristic path according to the present embodiment is configured by reusing the first and second generators, and is a path for confirming how well the characteristic of the source vocal voice has been reconstructed. Like the identity mapping path, the addition of a characteristic path helps to maintain elements such as lyrics, pitch, breath, and vibration of the source vocal voice. For feature extraction, the first generator extracts a first song voice characteristic from the second song voice, and extracts a first reconstructed song voice characteristic from the second modified song voice. Then, the second generator extracts a second song-sound characteristic from the first song-speech, and extracts a second reconstructed song-speech characteristic from the first modified song-speech.

The loss function according to the addition of the characteristic path is defined as a distance metric between the source song-speech characteristic and the reconstructed song-speech characteristic, and the characteristic loss function for the first and second generators is expressed by Equation 6.

Here, F is a mapping for extracting features. For feature extraction, any layer of the CNN implementing the generator may be used, but in this embodiment, the last stage of the encoder constituting the CNN is used. The reason for using the last stage is that the output of this stage has been confirmed to best express the global features implied in the singing voice.

Finally, by combining Equations 4 to 6, the loss function for the first and second generators according to the present embodiment is expressed by Equation 7.

The training unit uses the loss function of Equation 7 to train the first and second generators.

The harmonic structure of vocal speech in the frequency domain depends on partial and local peaks located at integer multiples of the fundamental frequency (f0 frequency). In this embodiment, the self-attention module includes partial peaks and peaks inherent in each of the first and second modified vocal vocals, the first and second reconstructed vocal vocals, and the first and second identity mapping vocal vocals. It serves to globally reinforce the harmonic structure of (inter or intra partial peaks). The position where the self attention module is used is the rear end of the generator and the distinguisher, as shown in FIG. 3. Therefore, two-dimensional data, which is the output of the generator or the discriminator, is used as the input of the self attention module. As described above, the two-dimensional data represents information in the frequency domain.

In the self-attention module, a multi-head attention method may be used. Multihead attention is expressed by Equation 8.

Here, Q, K, and V are a query matrix, a key matrix, and a value matrix, respectively, and H is an output matrix of the multihead attention. In multihead attention, a weight corresponding to the attention for each row vector constituting a value matrix is obtained using a query matrix and a key matrix, and then the weight is applied to the row vector. In self attention, we use the same matrix for Q, K, and V. And, in multihead attention, parallel processing is performed by a plurality of heads.

In this embodiment, the input is a two-dimensional Since it is data, there is an effect that the peaks in the frequency domain constituting the two-dimensional data and the harmonic structure between the peaks are emphasized by self-attention. The output of the self attention module becomes the final output of the generator or discriminator. In addition, in the present embodiment, the training unit updates a parameter used for multihead attention based on the loss function indicated in Equations 2 and 7.

4 is a diagram showing a generator of a vocal voice modifier according to an embodiment of the present invention.

With reference to FIG. 4, a learning procedure of a learning model for a singing voice translator according to the present embodiment will be described. Since the training of the first and second classifiers is a process in which the training unit updates the parameters of each classifier in the direction of reducing the loss function expressed in Equation 2, only the training procedure for the first and second generators will be described below.

It is assumed that during the previous training epoch, the training unit has updated the parameters of the first and second generators.

The first generator generates a first modified song voice ( G _AB (x _A )) by _{modifying the first song voice (x A} ) belonging to the domain A. In addition, the first generator generates a first identity mapping song voice ( G _AB (x _B )) and a first song voice characteristic ( F (x _B )) by _{transforming the second song voice (x B) belonging to domain B.} .

Subsequently, the second generator generates a second modified song voice ( G _BA (x _B )) by _{transforming the second song voice (x B) belonging to the domain B.} In addition, the second generator generates a second identity mapping song voice ( G _BA (x _A )) and a first song voice characteristic ( F (x _A )) by _{transforming the first song voice (x A) belonging to domain A.} .

Subsequently, the first generator transforms the second transformed vocal voice ( G _BA (x _B )) to transform the first reconstructed vocal vocal ( G _AB ( G _BA (x _B ))) and the first reconstructed vocal vocal characteristic ( F ( G _BA (x _B ))).

Subsequently, the second generator transforms the first transformed vocal voice ( G _AB (x _A )) to transform the second reconstructed vocal voice ( G _BA ( G _AB (x _A ))) and the second reconstructed vocal voice characteristic ( F ( G _AB (x _A ))).

Subsequently, the training unit calculates the final loss function expressed in Equation 7 based on Equation 1 and Equation 3 to Equation 6 using the generated products of the first and second generators and the first and second distinguishers. do.

Finally, the training unit updates the parameters of the first and second generators in a direction to reduce the calculated final loss function, thereby completing one epoch for training the generators.

The learning procedure as described above is described as being sequentially executed, but is not limited thereto. In other words, since it may be applicable to change and execute the above-described processes or execute one or more processes in parallel, it is not limited to the above-described time-series order.

Next, referring to FIG. 4, a process of transforming a vocal voice according to the present embodiment will be described. In the case of singing voice transformation, as described above, the trained first and second generators are used, but both may be used, or only one may be used when only one is desired to be transformed in one direction. The first generator generates a song voice (G _AB (x _A )) that simulates the target by modifying _{the first song voice (x A) belonging to the domain A.} In addition, the second generator generates a song voice (G _BA (x _B )) that simulates the target by modifying _{the second song voice (x B) belonging to the domain B.}

Hereinafter, a performance evaluation result of the vocal voice modifier according to the present embodiment will be described. The database used for performance evaluation is the vocal voices of famous male and female singers. For convenience, a female singer was assigned to domain A and a male singer's vocal voice was assigned to domain B. The amount of singing voice used was approximately 150 minutes, and this amount was divided and used during learning and evaluation.

In the evaluation environment, first, a CNN is used as a generator, and only the decoder part of the CNN is used as a classifier. The Adam Optimizer is used to train the learning model.

In this embodiment, instead of a subjective method such as a MOS (Mean Opinion Score) test, in order to effectively measure the tone distortion of the modified vocal voice, the similarity to the harmonic components of the same sound (eg, F3, B4) between comparison targets is considered. do. As a metric, the cosine similarity calculated in the frequency domain is used, and the cosine similarity for each note between the source vocal voice, the target vocal voice, and the modified vocal voice is measured.

The evaluation results according to this embodiment are shown in FIG. 5. In FIG. 5, the vertical axis is the cosine similarity calculated in the frequency domain. In FIG. 5, the numbers on the horizontal axis are piano key numbers and correspond to F3 to B4 notes (corresponding to 175 to 494 Hz). It can be seen that the similarity ([+SA+FEAT]A2B-B in FIG. 5) between the first modified vocal voice and the second vocal voice of the presented model is greater than in all other cases. In other cases compared, the similarity (AB) between the first and second vocal voices, the self-attention module and the characteristic loss function L _feat were all excluded, the self-attention module only was excluded, and _{only the L feat} was excluded. Similarity between the first modified song voice and the second song voice by each model (in order, [-SA-FEAT]A2B-B, [+SA-FEAT]A2B-B, [-SA+FEAT]A2B-B), etc. to be.

The results explaining the effect of the vocal speech transformation according to the present embodiment are shown in FIG. 6. Spectrograms of each of the first modified vocal and second vocal vocals are displayed at the top and bottom, with the vertical axis indicating frequency and the horizontal axis indicating time. As intended in this embodiment, it can be observed that the tone deformation (solid box) represented by the harmonic structure is observed, and elements such as vibration (thin solid box) and breath (dotted box) are maintained.

As described above, as an example of the acoustic transformation, the vocal voice modifier according to the present embodiment exhibited excellent performance during vocal-voice deformation.

Therefore, the acoustic modifier according to the present embodiment applies a training based on a self attention module and a characteristic loss function when transforming between mutually unpaired sounds, thereby facilitating the transformation that simulates the target tone. Factors such as pitch, rhythm, silence and vibration have the effect of providing a sustainable acoustic transformation system and method. Accordingly, it is possible to increase the reality of the transformed voice when the sound is transformed.

This embodiment has been described mainly on the transformation between the two sounds, but if the technical device and method of the present embodiment are appropriately modified and used, the transformation between sounds in a one-to-many or many-to-many form It is possible to expand the field of application up to.

In addition, in the present embodiment, sound is treated as data that is not matched with each other, but if the technical device and method of this embodiment are appropriately modified and used, it is possible to expand the category of data that is not matched to include images or images. Do. Preferably, when applied to acoustic data, it exhibits excellent performance as in this embodiment.

Various implementations of the systems and techniques described herein include digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and/or their It can be realized in combination. Various such implementations may include being implemented as one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor) coupled to receive data and instructions from and transmit data and instructions to and from a storage system, at least one input device, and at least one output device. Or a general purpose processor). Computer programs (which are also known as programs, software, software applications or code) contain instructions for a programmable processor and are stored on a "computer-readable medium".

Computer-readable media is any computer program product, apparatus, and/or device (e.g., CD-ROM, ROM, memory card, It represents a nonvolatile or non-transitory recording medium such as a hard disk, magneto-optical disk, and storage device).

Various implementations of the systems and techniques described herein may be implemented by a programmable computer. Here, the computer includes a programmable processor, a data storage system (including volatile memory, nonvolatile memory, or other types of storage systems or combinations thereof), and at least one communication interface. For example, the programmable computer may be one of a server, a network device, a set-top box, an embedded device, a computer expansion module, a personal computer, a laptop, a personal data assistant (PDA), a cloud computing system, or a mobile device.

The above description is merely illustrative of the technical idea of the present embodiment, and those of ordinary skill in the technical field to which the present embodiment pertains will be able to make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are not intended to limit the technical idea of the present embodiment, but to explain the technical idea, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of this embodiment should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present embodiment.

100: singing voice transformation system 110: input unit
120: deformation unit 130: output unit

Claims (20)

In the sound transformation system for translation between sound data,
A first generator for transforming the first sound of the first domain into a first modified sound;
A second generator for transforming the second sound of the second domain into a second modified sound;
A first discriminator configured to distinguish between the first sound of the first domain and the second modified sound modified by the second generator;
A second discriminator configured to distinguish between the second sound in the second domain and the first modified sound transformed by the first generator; And
And a training unit for training the first generator, the second generator, the first discriminator, and the second discriminator,
And the training unit trains the first generator and the second discriminator in opposition to each other, and trains the second generator and the first discriminator in opposition to each other. The method of claim 1,
The training unit,
Cause the first generator to transform the second modified sound into a first reconstructed sound,
Sound modification, characterized in that the second generator transforms the first modified sound into a second reconstructed sound system. The method of claim 2,
The training unit,
GAN-based sound modification, characterized in that training the first generator and the second generator based on a distance metric between the first reconstructed sound and the second sound and a distance metric between the second reconstructed sound and the first sound system. The method of claim 2,
The training unit,
Transformation from the second sound of the second domain executed by the first generator to a first identity-mapping sound; And
Transformation from the first sound of the first domain to a second identity mapping sound executed by the second generator
Acoustic modification, characterized in that performing at least one of the modifications system. The method of claim 4,
The training unit,
Training at least one of the first generator and the second generator based on at least one of a distance metric between the first identity mapping sound and the second sound and a distance metric between the second identity mapping sound and the first sound Acoustic transformation, characterized in that system. The method of claim 4,
The training unit,
A function performed by the first generator, including extraction from the second sound in the second domain to first acoustic characteristics and extraction from the second modified sound to a first reconstructed sound characteristic; And
A function performed by the second generator, including extraction from the first sound to a second sound characteristic in the first domain and extraction from the first modified sound to a second reconstructed sound characteristic
Acoustic modification, characterized in that performing at least one of the functions system. The method of claim 6,
The training unit,
Based on at least one of a distance metric between the first acoustic characteristic and the first reconstructed acoustic characteristic, and a distance metric between the second acoustic characteristic and the second reconstructed acoustic characteristic, at least one of the first and second generators is Acoustic transformation characterized by training system. The method of claim 6,
At least one of the first generator, the second generator, the first discriminator, and the second discriminator comprises a self-attention module. system. According to claim 8
The self attention module,
Acoustic modification, characterized in that updating a parameter used in the self-attention module based on the same loss function as the loss function applied to the first generator, the second generator, the first discriminator, and the second discriminator system. In the sound transformation system for translation between sound data,
A first generator for transforming a first sound in a first domain into a first translated sound by simulating a target sound in a second domain; And
Including a second generator for transforming the second sound of the second domain into a second modified sound by simulating the target sound of the first domain,
The first generator and the second generator are a distance metric between the first sound of the first domain and the second modified sound modified by the second generator, and the second sound of the second domain and the first generator. Sound, characterized in that it is pre-trained based on the distance metric between the first modified sound transformed by Transformation system. The method of claim 10,
At least one of the first generator and the second generator includes a self-attention module, or a first and second distinction used when the first generator and the second generator are trained in advance. At least one of the groups is sound modification, characterized in that it comprises a self-attention module system. The method of claim 10,
A sound modification, characterized in that, using a metric calculated on a frequency domain, a similarity between the first modified sound and the second sound and a similarity between the second modified sound and the first sound are checked. system. In performing the sound (sound) transformation (translation) between the data, the learning method of the acoustic system modification that is implemented on a computer,
Using a first generator, the first sound in the first domain is transformed into a first translated sound, and the first transformed sound is transformed into a second reconstruction sound using a second generator The process of doing;
Transforming a second sound of a second domain into a second modified sound using the second generator, and transforming the second modified sound into a first reconstructed sound using the first generator;
Discriminating between the first sound of the first domain and the second modified sound transformed by the second generator using a first discriminator;
Using a second discriminator, the second sound of the second domain and the transformed by the first generator Discriminating the first modified sound; And
A distance metric between the first sound and the second modified sound, a distance metric between the second sound and the first modified sound, a distance metric between the first reconstructed sound and the second sound, and the second reconstructed sound and the second sound 1 A process of training the first generator and the second generator based on part or all of the distance metric between sounds
Characterized in that it comprises a, learning method of the acoustic transformation system implemented on a computer. The method of claim 13,
The training process,
Transforming the second sound of the second domain into a first identity-mapping sound using the first generator; And
Process of transforming the first sound of the first domain into a second identity mapping sound using the second generator
Including at least one course of,
Update at least one parameter of the first generator and the second generator based on at least one of a distance metric between the first identity mapping sound and the second sound and a distance metric between the second identity mapping sound and the first sound Characterized in that, the learning method of the acoustic transformation system implemented on a computer. The method of claim 14,
The training process,
Extracting first acoustic features from the second sound in the second domain and extracting first reconstructed sound characteristics from the second modified sound using the first generator; And
A process of extracting a second acoustic characteristic from the first sound of the first domain and a second reconstructed sound characteristic from the first modified sound using the second generator
Including at least one course of,
At least one parameter of the first generator and the second generator based on at least one of a distance metric between the first acoustic characteristic and the first reconstructed acoustic characteristic, and a distance metric between the second acoustic characteristic and the second reconstructed acoustic characteristic A learning method of a sound transformation system implemented on a computer, characterized in that to update. The method of claim 15,
The training process,
Using a self-attention module included in at least one of the first generator, the second generator, the first discriminator, and the second discriminator, the first modified sound and the second modified sound, An acoustic transformation system implemented on a computer, characterized in that the harmonic structure inherent in each of the first and second reconstructed sounds, and the first and second identity mapping sounds, is reinforced. Method of learning. The method of claim 16,
The training process,
And updating a parameter of the self-attention module based on the same loss function as the loss function applied to the first generator, the second generator, the first discriminator, and the second discriminator. , A learning method of a sound transformation system implemented on a computer. In the modified acoustic method for performing acoustic (sound) transformation (translation) between the data, implemented on a computer,
Transforming a first sound of a first domain into a first translated sound using a first generator; And
Using a second generator, including the process of transforming the second sound of the second domain into a second modified sound,
The first generator and the second generator are a distance metric between the first sound of the first domain and the second modified sound modified by the second generator, and the second sound of the second domain and the first generator. The sound transforming method, characterized in that the training is performed in advance based on the distance metric between the first transformed sounds transformed by the method. A computer program stored in a nonvolatile or non-transitory recording medium that can be read by a computer to execute each step of the learning method of the acoustic transformation system according to any one of claims 13 to 17. A computer program stored in a nonvolatile or non-transitory recording medium that can be read by a computer to execute each step of the sound modification method according to claim 18.

Images