KR20210098935A - System and Method for Sound Translation Based on Generative Adversarial Networks - Google Patents

Abstract

Disclosed are a system and method for modifying sound. An objective, in the present embodiment, is to provide the system and method for modifying the sound that applies training based on a self-attention module and a characteristic loss function when transforming between mutually unpaired sounds, and increases a reality of the deformed sound by facilitating transformations that mimic a target tone while maintaining the elements such as pitch, beat, silence, and vibration. The system comprises: a first generator; a second generator; a first discriminator; a second discriminator; and a training part.

Description

System and Method for Sound Translation Based on Generative Adversarial Networks

The present invention relates to a system and method for acoustic modification using generative antagonistic networks for implementing transformations between mutually unpaired sounds.

The content described below merely provides background information related to the present invention and does not constitute the prior art.

Sound translation is a useful method when you want to hear a specific song with a specific singer's voice, but the realization of the situation is not realistically feasible. However, in most cases, not only are the objects mutually unpaired, but also sound modification should be attempted in a situation where information on lyrics or pitch is insufficient. The fact that it is practically difficult to obtain time-synchronized data from two different singers also makes acoustic deformation difficult.

As a conventional method, for example, Patent Document 1 or Non-Patent Document 2 proposes a learning model capable of transforming a style or shape between heterogeneous image domains. However, even if transformation between heterogeneous images is possible, collecting images that match each other is time-consuming and expensive. In the case of acoustic deformation in which timbre change is important, it is practically impossible to obtain time-synchronized data as described above, and thus attempts have been rare.

As a conventional method, for example, in Non-Patent Document 3, an unpaired voice is expressed in the frequency domain (as if in the form of an image), and then between heterogeneous voices using the learning model exemplified in Non-Patent Document 2 modification was attempted. As a result of the trial, the change in prosody between voices before and after transformation was confirmed through a subjective verification method called MOS (Mean Opinion Score) test. In the case of a pitched sound, since rich harmonic structures are clearly observed in addition to rhyme, the harmonic structure is an important element of the timbre expressing the sound. Therefore, there is a need for an acoustic modification method that considers a block and a cost function that can reflect the harmonic structure more so that it can be easily deformed to simulate the target tone.

Patent Document 1: US Patent No. US 10275473 B2 (Method for learning cross-domain relations based on generative adversarial networks, 2019.04.30, registered)

Non-Patent Document 1: Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), 2014. Non-Patent Document 2: Zhu, J., Park, T., Isola, P., and Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. 2017. Non-Patent Document 3: Kaneko, T. and Kameoka, H. Cyclegan-vc: Non-parallel voice conversion using cycle-consistent adversarial networks. In EUSIPCO, pp. 2100-2104, 2018.

In the present disclosure, when transforming between mutually unpaired sounds, by applying training based on a self-attention module and a characteristic loss function, it is easy to transform to simulate a target tone while facilitating the transformation of tone, length, and silence (silence). ) and vibration, thereby increasing the realism of the deformed sound.

According to an embodiment of the present invention, in a learning apparatus of a sound modification system for translation between sound data, a first generator for transforming a first sound of a first domain into a first modified sound ); a second generator for transforming a second sound of the second domain into a second modified sound; a first discriminator for discriminating the first sound of the first domain and the second deformed sound deformed by the second generator; a second discriminator for discriminating the second sound of the second domain and the first deformed sound deformed by the first generator; and a training unit configured to train the first generator and the second discriminator in opposition to each other and to train the second generator and the first discriminator to be opposed to each other, wherein the training unit comprises: transformation from the second sound to a first identity-mapping sound in two domains; and a transformation of at least one of the transformation from the first sound to the second identity mapping sound of the first domain executed by the second generator.

Further, according to another embodiment of the present invention, in a sound modification system for translation between sound data, a first transformation ( translated) a first generator that translates into sound; and a second generator configured to transform the second sound of the second domain into a second modified sound by simulating the target sound of the first domain, wherein the first generator generates the second sound of the second domain After transforming into a first identity-mapping sound and using the second generator to transform the first sound of the first domain into a second identity-mapping sound, the first and second generators are , a distance metric between the first sound in the first domain and a second deformed sound deformed by the second generator; a distance metric between the second sound in the second domain and the first deformed sound deformed by the first generator; and a distance metric between the first identity mapping sound and the second sound, and a distance metric between the second identity mapping sound and the first sound, characterized in that the sound is trained in advance. A transformation system is provided.

Further, according to another embodiment of the invention, to perform a sound (sound) transformation (translation) between the data, in a learning method, sound modification system of a computer device to perform, using a first generator (generator), transforming a first sound of a first domain into a first translated sound, and transforming the first transformed sound into a second reconstruction sound using a second generator; transforming a second sound of a second domain into a second deformed sound using the second generator, and transforming the second deformed sound into a first reconstructed sound using the first generator; distinguishing the first sound of the first domain from the second deformed sound deformed by the second generator using a first discriminator; Using a second discriminator, the second sound in the second domain and the second sound modified by the first generator distinguishing the first modified sound; and a distance metric between the first sound and the second modified sound, a distance metric between the second sound and the first modified sound, a distance metric between the first reconstructed sound and the second sound, and the second reconstructed sound and the and training the first generator and the second generator based on part or all of a distance metric between first sounds, wherein the training includes: using the first generator, the first generator of the second domain 2 transforming the sound into a first identity-mapping sound; and using the second generator to transform the first sound of the first domain into a second identity mapping sound. do.

Further, according to another embodiment of the invention, to perform a sound (sound) transformation (translation) between the data, according to sound transformation method of the computer device implementing, using the first generator (generator), the first domain transforming the first sound into a first translated sound; and transforming a second sound of a second domain into a second deformed sound using a second generator, wherein the first generator maps the second sound of the second domain to a first identity-mapping (identity-mapping) method. mapping) sound, and using the second generator to transform the first sound of the first domain into a second identity mapping sound, the first generator and the second generator may a distance metric between the first sound and a second deformed sound deformed by the second generator; a distance metric between the second sound in the second domain and the first deformed sound deformed by the first generator; and a distance metric between the first identity mapping sound and the second sound and a distance metric between the second identity mapping sound and the first sound. .

In addition, according to another embodiment of the present invention, there is provided a computer program stored in a computer-readable recording medium in order to execute each step of the learning method of the acoustic modification system.

In addition, according to another embodiment of the present invention, there is provided a computer program stored in a computer-readable recording medium in order to execute each step of the sound modification method.

As described above, according to the present embodiment, when transforming between mutually unpaired sounds, training based on the self-attention module and the characteristic loss function is applied to facilitate the transformation to simulate the target tone while facilitating the transformation. Elements such as pitch, length, silence, and vibration have the effect of increasing the realism of the modified sound by providing a sustainable acoustic modification system and method . In addition, if the technical apparatus and method of the present embodiment are appropriately modified and used, it is possible to expand the field of application to a one-to-many or many-to-many type of inter-acoustic transformation.

1 is a configuration diagram of a system for modifying a singing voice according to an embodiment of the present invention.
2 is a structural diagram of a learning model for a singing voice modifier according to an embodiment of the present invention.
3 is a view for explaining a position of a self-attention module according to an embodiment of the present invention.
4 is a view showing a generator of a singing voice modifier according to an embodiment of the present invention.
5 is a diagram illustrating the performance of a singing voice modifier according to an embodiment of the present invention.
6 is a diagram illustrating a tone change by a singing voice modifier according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the present embodiments, if it is determined that a detailed description of a related well-known configuration or function may obscure the gist of the present embodiments, the detailed description thereof will be omitted.

Also, in describing the components of the present embodiments, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the components from other components, and the essence, order, or order of the components are not limited by the terms. Throughout the specification, when a part 'includes' or 'includes' a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. . In addition, the '... Terms such as 'unit' and 'module' mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software.

DETAILED DESCRIPTION The detailed description set forth below in conjunction with the appended drawings is intended to describe exemplary embodiments of the present invention and is not intended to represent the only embodiments in which the present invention may be practiced.

The present invention proposes a new Generative Adversarial Networks (GAN)-based model capable of learning a relationship between domains with respect to a domain to which mutually unpaired sounds belong.

The sound includes voice, singing voice, music, natural sound, and the like. In one embodiment of the present invention, as an example of the acoustic transformation, a description will be focused on singing voice translation.

The singing voice modifier according to an embodiment of the present invention may use the learned relationship between domains to translate a singing voice from one domain to another domain.

1 is a configuration diagram of a system for modifying a singing voice according to an embodiment of the present invention.

The singing voice modification system 100 illustrated in FIG. 1 includes some or all of the input unit 110 , the modification unit 120 , and the output unit 130 . In this case, the components included in the singing voice modification system according to the present embodiment are not necessarily limited thereto. For example, a training unit (not shown) for training the learning model may be additionally provided on the singing voice transformation system, or may be implemented in a form linked to an external training unit.

The input unit 110 acquires necessary data in the process of performing the singing voice transformation and converts it into a form suitable for the singing voice transformation.

For example, the input unit 110 according to the present embodiment receives a singing voice belonging to a source domain in a time domain and then converts it into data in a frequency domain. do. As the transformation method, a Fast Fourier Transform (FFT) or a cepstrum transform may be used, but the present invention is not limited thereto.

In the present embodiment, the input unit 110 expresses the singing voice as two-dimensional data in the frequency domain using the principle of a spectrogram. First, with respect to a singing voice in the time domain, an M-order FFT is performed while overlapping and sliding a part of the performed section to obtain M-order FFT vectors in the frequency domain. Next, an MxN matrix, which is two-dimensional data, is generated by combining N M-th vectors in the frequency domain in the form of a column vector.

In the present embodiment, the expression 'mutually unpaired' is used for singing voices belonging to the two domains. This expression means that the two singing voices are from different singers, and the time-synchronization is not consistent with each other.

The transforming unit 120 performs translation between unmatched singing voices.

The transform unit 120 according to the present embodiment receives the frequency domain data converted by the input unit 110 as an input, and transforms the data into frequency domain data that mimics the target tone. The transforming unit 120 uses a neural network-based learning model previously learned by the training unit for singing voice transformation. The structure of the learning model and the training process of the learning model will be described later.

The output unit 130 provides the modified singing voice to the user of the singing voice modification system in an auditory form.

The output unit 130 according to the present embodiment receives the transformed data in the frequency domain from the transforming unit 120 and then converts it into a transformed singing voice in the time domain through a synthesis process. Finally, the transformed data in the time domain is provided to the user of the audible voice modification system in an auditory form.

2 is a structural diagram of a learning model for a singing voice modifier according to an embodiment of the present invention.

The learning model shown in FIG. 2 is a model based on Generative Adversarial Networks (GANs) with a reconstruction path. The GAN-based system used in the learning model is a structure in which two GANs are coupled. hope Hereinafter, concepts used in the learning model according to the present embodiment will be mainly described.

GAN basically includes a generator and a discriminator. The generator performs mapping between the two domains. Depending on the elements constituting the domain, the correspondence may be expressed in various ways such as transformation, translation, pairing, etc. In the case of singing voice, the term transformation will be used.

Hereinafter, in the implementation of the GAN according to the present embodiment, the generator uses a convolutional neural network (CNN), and the discriminator uses only the decoder part among the components of the CNN.

The role of the generator is to simulate data belonging to a codomain by generating similar data to the extent that the discriminator cannot distinguish them. The generator is trained so that it misinterprets the data it generates and outputs true (probability 1) as the discriminator. The role of the discriminator is to distinguish the similar data generated by the generator and the target data belonging to the conjugation domain. The discriminator is trained to output true (true, probability 1) to target data belonging to the conjugation domain and to output false (false, probability 0) to similar data. Therefore, in the basic GAN, a generative adversarial loss function is defined based on the roles of the generator and the discriminator, and the generator and the discriminator are separated through unsupervised learning based on the defined loss function. train

The structure in which two GANs are combined, as a part separated by a dotted box in FIG. 2, includes two generators and two distinguishers in order to effectively perform a one-to-one correspondence between the two domains A and B. . Hereinafter, two generators are referred to as first and second generators, and two differentiators are referred to as first and second distinguishers. In the present embodiment, domain A includes the singer A's singing voice, and domain B includes the singer B's singing voice. The first generator performs the transformation from domain A to B, and the second generator performs the transformation from domain B to A. The second discriminator discriminates the output of the first generator from the target of the domain B, and the first discriminator discriminates the output of the second generator and the target of the domain A.

Hereinafter, terms necessary to express the loss function according to the present embodiment are defined. First, the transformation from the domain A to the domain B performed by the first generator is denoted as G _AB , and the transformation from the domain B to the domain A performed by the second generator is denoted as G _{BA .} Also, the functions of the first and second classifiers are denoted as D _A and D _{B , respectively.} The singing voices belonging to domains A and B are expressed as first and second singing voices, respectively, and are denoted by _{symbols x A} and x _B. Regardless of the input, _{the output of G AB} is indicated as starting with 'first', and similarly, _{the output of G BA} is indicated as starting with 'second'. In the basic GAN structure, the output of the generator is classified using the expression 'transformation'. According to FIG. 2 , a plurality of each of the first and second generators is used, but this is for convenience of description of the learning model, and in actual implementation, each of the first and second generators is implemented as one. A training procedure of the learning model using each of the first and second generators will be described later with reference to FIG. 4 .

To explain the functions of the generator and the discriminator again using the above-mentioned notations, first, the first generator transforms the first sung voice belonging to the domain A to generate a first modified sung voice, and the second generator belongs to the domain B A second modified singing voice is generated by modifying the second singing voice. And a 2nd discriminator distinguishes a 1st modified song voice from the 2nd song voice of the domain B, and a 1st discriminator distinguishes a 2nd modified song voice from the 1st song voice of the domain A.

Since the main role of the generator is to generate similar data that the classifier cannot distinguish, the loss function is expressed as a probability after the similar data has passed through the classifier, and a negative sign is used to express it as a positive value. use The generation loss functions for the first and second generators are expressed by Equation (1).

는 기대(expectation) 함수이다.here,

is an expectation function.

Since the main role of the classifier is to distinguish the similar data generated by the generator and the data belonging to the conjugate domain, the loss function is the 'probability after the conjugated data passes the classifier' and 'the probability after the similar data passes the classifier' is expressed as the sum of the 'probabilities subtracted from the total probability 1'. The discrimination loss function for the first and second discriminators is expressed by Equation (2).

Equations 1 and 2 are loss functions of a structure in which two GANs are combined. The training unit uses the loss functions of Equations 1 and 2 to train the generator and the discriminator.

As shown in Fig. 2, when a reconstruction path is included, an additional loss is used for training the generator. The reconstruction path is constructed by reusing the first and second generators, and is a path for confirming how well the source singing voice is reconstructed after two consecutive transformations. For the reconstruction of the singing voice, the second generator deforms the first modified singing voice to generate a second reconstructed singing voice, and the first generator modifies the second modified singing voice to generate a first reconstructed singing voice.

The loss function according to the addition of the reconstruction path is defined as a distance metric between the source and reconstructed singing voices, and the reconstruction loss functions for the first and second generators are expressed by Equation (3).

는 거리 메트릭이고, 거리 메트릭은 어떠한 형태의 메트릭(L1, L2, 코사인 유사도 등)이 사용되어도 무방하다. 재구성 경로까지 고려한 제1 및 제2 생성기에 대한 손실함수는 수학식 4로 표현한다. 트레이닝부는 제1 및 제2 생성기를 트레이닝시키기 위하여 수학식 4의 손실함수를 사용한다. here,

is a distance metric, and any type of metric (L1, L2, cosine similarity, etc.) may be used as the distance metric. The loss functions for the first and second generators considering the reconstruction path are expressed by Equation (4). The training unit uses the loss function of Equation 4 to train the first and second generators.

Equations 2 and 4 showed satisfactory performance when applied to image pairing as a loss function used in Patent Document 1 or Non-Patent Document 2. However, as in the case of singing voice transformation, when the tonal transformation represented by harmonic structures is possible and maintenance of elements such as lyrics, pitch, breathing and vibration is required, training based on the above-mentioned formula alone is insufficient. there is a side

Therefore, in this embodiment, in order to further improve the transformation process, a feature path and an identity-mapping path are added, and in addition to this, a self-attention module can be added. there is. Hereinafter, the parts added for improvement will be mainly described.

First, as shown in FIG. 2 , the identity mapping path according to the present embodiment is a path for confirming how properly the source singing voice is identity mapped, and is constructed by reusing the first and second generators. By adding an identity mapping path, it helps to maintain elements such as lyrics, pitch, breathing and vibration of the source singing voice. For the identity mapping, the second generator transforms the first singing voice to generate a second identity mapping singing voice, and the first generator modifies the second singing voice to generate a first identity mapping singing voice.

The loss function according to the addition of the identity mapping path is defined as a distance metric between the source singing voice and the identity mapping song voice, and the identity mapping loss function for the first and second generators is expressed by Equation (5).

Next, as shown in FIG. 2 , the characteristic path according to the present embodiment is constructed by reusing the first and second generators, and is a path for confirming how well the characteristics of the source sung voice are reconstructed. Like the identity mapping path, the addition of a characteristic path helps to maintain elements such as lyrics, pitch, breathing and vibration of the source singing voice. For the feature extraction, the first generator extracts a first sung voice characteristic from the second sung voice, and extracts a first reconstructed sung voice characteristic from the second modified sung voice. And a 2nd generator extracts a 2nd song voice characteristic from a 1st song voice, and extracts a 2nd reconstructed song voice characteristic from a 1st modified song voice.

The loss function according to the addition of the characteristic path is defined as a distance metric between the source sung speech characteristic and the reconstructed sung speech characteristic, and the characteristic loss function for the first and second generators is expressed by Equation (6).

Here, F is a mapping for extracting characteristics. In the feature extraction, any layer of CNN implementing the generator may be used, but in this embodiment, the last layer of the encoder constituting the CNN is used. The reason for using the last stage is that it has been confirmed that the output of this stage best expresses the global characteristics contained in the singing voice.

Finally, by combining Equations 4 to 6, the loss functions for the first and second generators according to the present embodiment are expressed by Equation 7.

The training unit uses the loss function of Equation 7 to train the first and second generators.

In the frequency domain, the harmonic structure of a singing voice depends on partial and local peaks located at integer multiples of the fundamental frequency (f0 frequency). In the present embodiment, the self-attention module is configured to configure partial peaks and inter-peaks inherent in each of the first and second modified sung voices, the first and second reconstructed phonic voices, and the first and second identity mapping sung voices, respectively. It serves to globally strengthen the harmonic structure of (inter or intra partial peaks). The position where the self-attention module is used is at the rear end of the generator and the discriminator, as shown in FIG. 3 . Therefore, as an input of the self-attention module, two-dimensional data that is an output of a generator or a discriminator is used. As described above, the two-dimensional data represents information in the frequency domain.

In the self-attention module, a multi-head attention method may be used. The multi-head attention is expressed by Equation (8).

Here, Q, K, and V are a query matrix, a key matrix, and a value matrix, respectively, and H is an output matrix of multi-head attention. In the multi-head attention, a weight corresponding to an attention for each row vector constituting a value matrix is obtained using a query matrix and a key matrix, and then the weight is applied to the row vector. In self-attention, we use the same matrix for Q, K and V. And, in the multi-head attention, parallel processing by a plurality of heads is performed.

In this embodiment, a two-dimensional input is passed from a generator or discriminator. Since it is data, there is an effect that the peaks on the frequency domain constituting the two-dimensional data and the harmonic structure between the peaks are emphasized by the self-attention. The output of the self-attention module becomes the final output of the generator or discriminator. In addition, in the present embodiment, the training unit updates the parameters used for multi-head attention based on the loss function shown in Equations 2 and 7.

4 is a view showing a generator of a singing voice modifier according to an embodiment of the present invention.

A learning procedure of the learning model for a singing voice translator according to the present embodiment will be described with reference to FIG. 4 . Since the training of the first and second classifiers is a process in which the training unit updates the parameters of each classifier in the direction of reducing the loss function expressed in Equation (2), only the training procedure for the first and second generators will be described below.

It is assumed that during the previous training epoch, the training unit has updated the parameters of the first and second generators.

The first generator transforms the first sung voice (x _A ) belonging to the domain A to generate the first modified sung voice ( G _AB (x _A )). In addition, the first generator generates a first identity mapping singing voice ( G _AB (x _B )) and a first phonic voice characteristic ( F (x _B )) by _{transforming the second phonic voice (x B ) belonging to the domain B.} .

Then, the second generator _{transforms the second song voice (x B} ) belonging to the domain B to generate the second modified syllable voice ( G _BA (x _B )). In addition, the second generator generates a second identity mapping sung voice ( G _BA (x _A )) and a first phonic voice characteristic ( F (x _A )) by _{transforming the first sung voice (x A ) belonging to the domain A.} .

Then, the first generator modifies the second modified phonological voice ( G _BA (x _B )) to obtain a first reconstructed singable voice ( G _AB ( G _BA (x _B ))) and a first reconstructed phonic voice characteristic ( F ( G )). _BA (x _B ))).

Then, the second generator modifies the first modified phonological voice ( G _AB (x _A )) to obtain a second reconstructed phonic voice ( G _BA ( G _AB (x _A ))) and a second reconstructed phonic voice characteristic ( F ( G )). _AB (x _A ))).

Next, the training unit calculates the final loss function expressed in Equation 7 based on Equations 1 and 3 to 6 by using the generated results of the first and second generators and the first and second discriminators. do.

Finally, the training unit updates the parameters of the first and second generators in the direction of reducing the calculated final loss function, thereby ending one epoch for training the generators.

The learning procedure as described above is described as being sequentially executed, but is not necessarily limited thereto. In other words, since it may be applicable to change and execute the above-described process or to execute one or more processes in parallel, it is not limited to the time-series order as described above.

Next, a process of modifying a singing voice according to the present embodiment will be described with reference to FIG. 4 . When modifying the singing voice, as described above, the trained first and second generators are used, but both or only one may be used when only deformation in one direction is desired. The first generator transforms the first singing voice (x _A ) belonging to the domain A to generate a singing voice ( G _AB (x _A )) that mimics the target. In addition, the second generator generates a song voice G _BA (x _B ) that mimics the target by modifying _{the second singing voice (x B ) belonging to the domain B.}

Hereinafter, performance evaluation results of the singing voice transducer according to the present embodiment will be described. The database used for performance evaluation is the singing voices of famous male and female singers. For convenience, a female singer is assigned to domain A and a male singer's voice is assigned to domain B. The singing voice used was approximately 150 minutes long, and this amount was divided and used during learning and evaluation.

The evaluation environment first uses CNN as a generator and only the decoder part of CNN as a classifier. The Adam Optimizer is used to train the learning model.

In this embodiment, instead of a subjective method such as the MOS (Mean Opinion Score) test, in order to effectively measure the tonal variation of the deformed singing voice, the similarity of the harmonic components of the same sound (eg, F3, B4) between comparison objects is considered. do. As a metric, cosine similarity calculated in the frequency domain is used, and the cosine similarity for each note between the source sung voice, the target sung voice, and the modified sung voice is measured.

The evaluation result according to this embodiment is shown in FIG. 5 . In FIG. 5 , the vertical axis represents the cosine similarity calculated in the frequency domain. Numbers on the horizontal axis in FIG. 5 are piano key numbers, and correspond to notes F3 to B4 (corresponding to 175 to 494 Hz). It can be seen that the similarity ([+SA+FEAT]A2B-B in FIG. 5) between the first modified singing voice and the second singing voice of the presented model is greater than in all other cases. In other cases compared, the similarity (AB) between the first and second singing voices, _{the model in which both the self-attention module and the characteristic loss function L feat} are excluded, the model in which only the self-attention module is excluded, and L _feat are excluded. The degree of similarity between the first modified singing voice and the second singing voice by each model ([-SA-FEAT]A2B-B, [+SA-FEAT]A2B-B, [-SA+FEAT]A2B-B), etc. am.

The results for explaining the effect of the singing voice modification according to the present embodiment are shown in FIG. 6 . Spectrograms of each of the first modified singing voice and the second singing voice are displayed at the upper and lower ends, wherein the vertical axis indicates frequency and the horizontal axis indicates time. As intended in this embodiment, it can be observed that the tone deformation (solid line box) represented by the harmonic structure can be observed, and it can be observed that elements such as vibration (thin solid line box) and breathing (dashed line box) are maintained.

As described above, as an example of acoustic modification, the singing voice modifier according to the present embodiment showed excellent performance in the singing voice modification.

Therefore, the sound modifier according to the present embodiment applies training based on the self-attention module and the characteristic loss function when transforming between mutually unpaired sounds, while facilitating the transformation to simulate the target tone. Elements such as pitch, length, silence, and vibration have the effect of providing a sustainable acoustic modification system and method. Accordingly, it is possible to increase the reality of the deformed voice when the sound is deformed.

Although the present embodiment has been mainly described for deformation between two sounds, if the technical apparatus and method of the present embodiment are appropriately modified and used, one-to-many or many-to-many form of deformation between sounds It is possible to expand the field of application to

In addition, although sound is handled as data that does not match each other in this embodiment, if the technical apparatus and method of this embodiment are appropriately modified and used, it is possible to expand the category of unmatched data to include images or images do. Preferably, when applied to acoustic data, it exhibits excellent performance as in this embodiment.

Various implementations of the systems and techniques described herein may include digital electronic circuitry, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and/or combination can be realized. These various implementations may include being implemented in one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor) coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device. or may be a general-purpose processor). Computer programs (also known as programs, software, software applications or code) contain instructions for a programmable processor and are stored on a “computer-readable medium”.

A computer-readable medium includes any computer program product, apparatus, and/or device (eg, a CD-ROM, ROM, memory card, a non-volatile or non-transitory recording medium such as a hard disk, a magneto-optical disk, and a storage device).

Various implementations of the systems and techniques described herein may be implemented by a programmable computer. Here, the computer includes a programmable processor, a data storage system (including volatile memory, non-volatile memory, or other types of storage systems or combinations thereof), and at least one communication interface. For example, the programmable computer may be one of a server, a network appliance, a set-top box, an embedded device, a computer expansion module, a personal computer, a laptop, a Personal Data Assistant (PDA), a cloud computing system, or a mobile device.

The above description is merely illustrative of the technical idea of this embodiment, and various modifications and variations will be possible by those skilled in the art to which this embodiment belongs without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain rather than limit the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the equivalent range should be interpreted as being included in the scope of the present embodiment.

100: singing voice transformation system 110: input unit
120: deformation unit 130: output unit

Claims (14)

A learning apparatus for a sound transformation system for translation between sound data, the learning apparatus comprising:
a first generator for transforming a first sound of a first domain into a first modified sound;
a second generator for transforming a second sound of the second domain into a second modified sound;
a first discriminator for discriminating the first sound of the first domain and the second deformed sound deformed by the second generator;
a second discriminator for discriminating the second sound of the second domain and the first deformed sound deformed by the first generator; and
A training unit configured to train the first generator and the second discriminator to be opposed to each other, and to train the second generator and the first discriminator to be opposed to each other.
includes,
The training unit,
transformation from the second sound to a first identity-mapping sound of the second domain executed by the first generator; and
Transformation from the first sound to a second identity mapping sound of the first domain executed by the second generator
A learning apparatus of an acoustic deformation system, characterized in that performing at least one of the deformations. According to claim 1,
The training unit,
cause the first generator to transform the second modified sound into a first reconstructed sound;
The learning apparatus of the acoustic modification system, characterized in that the second generator transforms the first modified sound into a second reconstructed sound. 3. The method of claim 2,
The training unit,
and training the first and second generators based on a distance metric between the first reconstructed sound and the second sound and a distance metric between the second reconstructed sound and the first sound. learning device. 3. The method of claim 2,
The training unit,
Training at least one of the first generator and the second generator based on at least one of a distance metric between the first identity mapping sound and the second sound and a distance metric between the second identity mapping sound and the first sound Learning apparatus of the acoustic deformation system, characterized in that. 3. The method of claim 2,
The training unit,
a function, executed by the first generator, comprising extraction from the second sound to first acoustic features in the second domain and from the second modified sound to a first reconstructed acoustic feature; and
a function executed by the second generator, comprising extraction from the first sound to a second acoustic characteristic of the first domain and from the first modified sound to a second reconstructed acoustic characteristic
A learning apparatus for an acoustic deformation system, characterized in that it performs at least one function of 6. The method of claim 5,
The training unit,
at least one of the first and second generators based on at least one of a distance metric between the first acoustic characteristic and the first reconstructed acoustic characteristic and a distance metric between the second acoustic characteristic and the second reconstructed acoustic characteristic Learning apparatus of the acoustic deformation system, characterized in that training. A sound transformation system for translation between sound data, the system comprising:
a first generator that transforms the first sound of the first domain into a first translated sound by simulating the target sound of the second domain; and
A second generator for transforming the second sound of the second domain into a second modified sound by simulating the target sound of the first domain;
The first generator transforms the second sound of the second domain into a first identity-mapping sound, and uses the second generator to transform the first sound of the first domain into a second identity After transforming into a mapping sound,
The first generator and the second generator,
a distance metric between the first sound in the first domain and a second deformed sound deformed by the second generator;
a distance metric between the second sound in the second domain and the first deformed sound deformed by the first generator; and
at least one of a distance metric between the first identity mapping sound and the second sound and a distance metric between the second identity mapping sound and the first sound
Sound characterized in that it is trained in advance based on transformation system. 8. The method of claim 7,
Acoustic deformation, characterized in that by using a metric calculated in the frequency domain, a degree of similarity between the first modified sound and the second sound and a degree of similarity between the second modified sound and the first sound are confirmed. system. To perform the audio (sound) transformation (translation) between the data, according to which the computer devices to perform learning method of the acoustic variant system,
A first generator is used to transform a first sound of a first domain into a first translated sound, and a second generator is used to transform the first modified sound into a second reconstruction sound. process;
transforming a second sound of a second domain into a second deformed sound using the second generator, and transforming the second deformed sound into a first reconstructed sound using the first generator;
distinguishing the first sound of the first domain from the second deformed sound deformed by the second generator using a first discriminator;
Using a second discriminator, the second sound in the second domain and the second sound modified by the first generator distinguishing the first modified sound; and
a distance metric between the first sound and the second modified sound, a distance metric between the second sound and the first deformed sound, a distance metric between the first reconstructed sound and the second sound, and the second reconstructed sound and the second sound 1 The process of training the first generator and the second generator based on some or all of the distance metric between sounds
including,
The training process is
transforming the second sound of the second domain into a first identity-mapping sound using the first generator; and
A process of transforming the first sound of the first domain into a second identity mapping sound using the second generator
A learning method of the acoustic deformation system, characterized in that it further comprises at least one of the processes. 10. The method of claim 9,
The training process is
At least one parameter of the first generator and the second generator is updated based on at least one of a distance metric between the first identity mapping sound and the second sound and a distance metric between the second identity mapping sound and the first sound A learning method of the acoustic modification system, characterized in that. 11. The method of claim 10,
The training process is
extracting first acoustic features from the second sound of the second domain using the first generator, and extracting first reconstructed acoustic features from the second modified sound; and
A process of extracting a second acoustic characteristic from the first sound of the first domain using the second generator and extracting a second reconstructed acoustic characteristic from the first modified sound
comprising at least one process of
a parameter of at least one of the first and second generators based on at least one of a distance metric between the first acoustic characteristic and the first reconstructed acoustic characteristic and a distance metric between the second acoustic characteristic and the second reconstructed acoustic characteristic A learning method of the acoustic deformation system, characterized in that for updating the. A sound transformation method implemented by a computer device to perform translation between sound data , the method comprising:
transforming a first sound of a first domain into a first translated sound by using a first generator; and
using a second generator to transform a second sound of a second domain into a second modified sound,
The first generator transforms the second sound of the second domain into a first identity-mapping sound, and uses the second generator to transform the first sound of the first domain into a second identity After transforming into a mapping sound,
The first generator and the second generator,
a distance metric between the first sound in the first domain and a second deformed sound deformed by the second generator;
a distance metric between the second sound in the second domain and the first deformed sound deformed by the first generator; and
at least one of a distance metric between the first identity mapping sound and the second sound and a distance metric between the second identity mapping sound and the first sound
Sound deformation method, characterized in that pre-trained based on. A computer program stored in a computer-readable recording medium to execute each step of the learning method of the acoustic modification system according to any one of claims 9 to 11. A computer program stored in a computer-readable recording medium to execute each step of the sound modification method according to claim 12.

Images