KR20210155520A - Method and Apparatus for Synthesizing/Modulating Singing Voice of Multiple Singers - Google Patents

Abstract

Disclosed are a device and method for synthesizing/modulating a singing voice for a plurality of singers. The present embodiment obtains a user request to specify a singer and song, generates tones and singing styles for the singer using a deep neural network-based inference model, and generates a formant for which the tone is adjusted and a pitch skeleton for which the singing style is adjusted. In addition, provided are the device and method for synthesizing/modulating the singing voice that generates the singing voice for the song from the pitch skeleton masked with a formant using a deep neural network-based SR transformation model. Therefore, the present invention is capable of having an effect for which a natural singing voice synthesis/modulation for the plurality of singers is enabled.

Description

Method and Apparatus for Synthesizing/Modulating Singing Voice of Multiple Singers

The present invention relates to an apparatus and method for synthesizing/modulating the voice of a plurality of singers. More specifically, singing voice synthesis that automatically generates the singing voices of a plurality of singers or a combination of the characteristics of each of the two singers from the singer characteristics, lyric information, and pitch information based on a user request using a deep neural network / relates to a modulation device and method.

The content described below merely provides background information related to the present invention and does not constitute the prior art.

Singing Voice Synthesis (SVS) is a method of generating a natural singing voice using sheet music and lyrics information. Compared to text-to-speech (TTS) that converts text into speech, SVS requires a function to adjust pitch information of each syllable. Recently, as the field of application of deep neural networks has been expanded and remarkable achievements have been achieved, deep neural networks are also being applied to the SVS method.

As a deep neural network-based SVS method, a parametric method for predicting a parameter capable of synthesizing a singing voice using a deep neural network exists (see Non-Patent Document 1). Although the parametric method suggested the possibility of implementing SVS using a deep neural network, it has a disadvantage that the performance of the entire SVS is determined according to the performance of a vocoder using parameters. Therefore, as an alternative, an SVS method for generating a linear spectrogram based on an end-to-end deep neural network or implementing a vocoder using a deep neural network has been tried.

Another requirement in the SVS implementation is the synthesis of vocal voices for multiple singers. As shown in Fig. 8(a), the SVS for a single singer generates a formant from lyric information, that is, a text, and a pitch contour or pitch skeleton from the pitch. After that, they are combined in a deep neural network to synthesize singing voices. On the other hand, as an SVS method for a plurality of singers, as shown in FIG. There is a method to do it (refer to Non-Patent Document 2). Although this direct SVS method can be implemented simply, there is a disadvantage that the deep neural network needs to be retrained whenever a new singer is added.

Therefore, as shown in FIG. 8(c), after generating a singer ID characteristic (eg, tone or singing style) based on a singing query, an end-to-end deep neural network is used to identify the singer ID characteristic, lyrics There is a need for a singing voice synthesis and modulation method capable of automatically generating natural singing voices for a plurality of singers from information and pitch information, or a singing voice in which the characteristics of each of two singers are combined.

Non-Patent Document 1: Merlijn Blaauw and Jordi Bonada, “A neural parametric singing synthesizer modeling timbre and expression from natural songs,” Applied Sciences, vol. 7, no. 12, pp. 1313, 2017. Non-Patent Document 2: Pritish Chandna, Merlijn Blaauw, Jordi Bonada, and Emilia Gomez, “Wgansing: A multi-voice singing voice synthesizer based on the wasserstein-gan,” arXiv preprint arXiv:1903.10729, 2019. Non-Patent Document 3: Hideyuki Tachibana, Katsuya Uenoyama, and Shunsuke Aihara, “Efficiently trainable text-to-speech system based on deep convolutional networks with guided attention,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 4784-4788. Non-Patent Document 4: Juheon Lee, Hyeong-Seok Choi, Chang-Bin Jeon, Junghyun Koo, and Kyogu Lee, “Adversarially trained end-to-end korean singing voice synthesis system,” Proc. Interspeech 2019, pp. 2588-2592, 2019.

The present disclosure obtains a user request for designating a singer and a song, generates a tone and a singing style for the singer using an inference model based on a deep neural network, a tone-controlled formant, and a pitch with an adjusted singing style create a skeleton Another object of the present invention is to provide an apparatus and method for synthesizing/modulating a singing voice for generating a singing voice for a song from a pitch skeleton masked with a formant using a deep neural network-based SR transformation model.

According to an embodiment of the present invention, in a singing voice synthesis and modulation method executed by a computing device, a user request for specifying a tone of a first singer, a singing style of a second singer, and a song ) to obtain a first singer spectrogram for a first singer, a second singer spectrogram for a second singer, and text for lyrics of the song, and from MIDI data for the song, the pitch of the song the process of obtaining (pitch); Using an inference model based on a pre-trained deep neural network, a first timbre for the first singer is generated from the first singer spectrogram, and the text and the a first process of generating a first formant mask from the first tone; a second process of generating a second singing style for the second singer from the second singer spectrogram using the inference model, and generating a second pitch skeleton from the pitch and the second singing style; and generating a third inferred spectrogram of low resolution by masking the second pitch skeleton using the first formant mask.

According to another embodiment of the present invention, by obtaining a user request specifying the tone of the first singer, the singing style of the second singer, and the song, the first singer spectrogram for the first singer, the second an input unit for obtaining a second singer spectrogram for a singer and text for lyrics of the song, and obtaining a pitch of the song from MIDI data for the song; Using an inference model based on a pre-trained deep neural network, a first timbre for the first singer is generated from the first singer spectrogram, and the text and the create a first formant mask from the first tone; a modulator for generating a second singing style for the second singer from the second singer spectrogram by using the inference model, and generating a second pitch skeleton from the pitch and the second singing style; and a third masking unit generating a third inferred spectrogram of low resolution by masking the second pitch skeleton using the first formant mask.

According to another embodiment of the present invention, there is provided a computer program stored in a computer-readable recording medium in order to execute each step included in the method for synthesizing and modulating a singing voice.

As described above, according to this embodiment, a user request for designating a singer and a song is obtained, a tone and a singing style for the singer are generated using an inference model based on a deep neural network, and the tone is adjusted formant; And by providing a singing voice synthesis/modulation apparatus and method for generating a pitch skeleton with a controlled singing style, and generating a singing voice for a song from a pitch skeleton masked with a formant using an SR transformation model based on a deep neural network, There is an effect that natural singing voice synthesis/modulation for a plurality of singers is possible.

Also, according to this embodiment, a tone and a singing style for a singer are generated using an inference model based on a deep neural network, a tone-controlled formant, and a tone high skeleton with an adjusted singing style are generated, and a deep neural network-based inference model is generated. By providing a singing voice synthesis/modulation device and method for generating a singing voice for a song from a pitch skeleton masked with a formant using an SR transformation model, a singable voice based on an end-to-end network is provided. There is an effect that synthesis/modulation becomes possible.

In addition, according to this embodiment, by obtaining a user request to specify the tone of the first singer, the singing style of the second singer, and the song, the tone of the first singer and the tone of the second singer are obtained using an inference model based on a deep neural network. By providing a singing voice synthesis/modulation apparatus and method for generating a singing style, and independently generating a formant in which the tone of a first singer is adjusted, and a pitch skeleton in which the singing style of a second singer is adjusted, the tone and singing style It has the effect of enabling the synthesis/modulation of singing voices that independently cross-regulate.

1 is a block diagram of an apparatus for synthesizing and modulating a singing voice according to an embodiment of the present invention.
2 is a block diagram of a singer ID encoder, a formant mask decoder, and a pitch skeleton decoder according to an embodiment of the present invention.
3 is a flowchart of a method for synthesizing and modulating a singing voice according to an embodiment of the present invention.
4 is a block diagram of an apparatus for synthesizing and modulating a singing voice according to another embodiment of the present invention.
5 is a flowchart of a method for synthesizing and modulating a singing voice according to another embodiment of the present invention.
6 is a block diagram of a learning model according to an embodiment of the present invention.
7 is a flowchart of a learning method for a learning model according to an embodiment of the present invention.
8 is a conceptual diagram of a conventional method for synthesizing and modulating a singing voice.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in the description of the present embodiments, if it is determined that a detailed description of a related well-known configuration or function may obscure the gist of the present embodiments, the detailed description thereof will be omitted.

Also, in describing the components of the present embodiments, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the elements from other elements, and the essence, order, or order of the elements are not limited by the terms. Throughout the specification, when a part 'includes' or 'includes' a certain element, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated. . In addition, the '... Terms such as 'unit' and 'module' mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software.

DETAILED DESCRIPTION The detailed description set forth below in conjunction with the appended drawings is intended to describe exemplary embodiments of the present invention and is not intended to represent the only embodiments in which the present invention may be practiced.

This embodiment discloses a device and method for synthesizing/modulating a plurality of singers' singing voices. In more detail, by obtaining a user request to designate a singer and a song, and using an inference model based on a deep neural network to generate a tone and a singing style for the singer, the tone-adjusted formant, and the singing style-adjusted pitch Creates a skeleton independently. In addition, we propose an apparatus and method for synthesizing/modulating a singing voice to generate a singing voice for a song from a pitch skeleton masked with a formant using a deep neural network-based SR transformation model.

A singing voice refers to a song expressed by a human voice.

A Mel spectrogram is a coefficient obtained by extracting characteristics of an audio signal by filtering an audio signal in a frequency domain using a Mel filter bank. In this embodiment, a Mel spectrogram is used to indicate the singer's singing voice and the inferred singing voice. A high-resolution linear spectrogram includes hundreds to thousands of samples, but the number of filters constituting a filter bank expressing a Mel spectrogram is about tens.

MIDI (Musical Instrument Digital Interface) is a common method for expressing music played by an instrument. The MIDI format is a rule for the command sequence used by hardware or software such as a synthesizer or sequencer to reproduce music. MIDI data expresses the pitch, the length of the note, and the velocity (velocity, in MIDI, the velocity of the note is expressed as velocity).

1 is a block diagram of an apparatus for synthesizing and modulating a singing voice according to an embodiment of the present invention.

In an embodiment according to the present invention, the singing voice synthesis and modulation device 100 obtains a user request to designate a target singer and song, and uses an inference model based on a deep neural network to provide a tone for the singer ( timbre) and a singing style, and independently generate a tone-adjusted formant and a pitch skeleton with an adjusted singing style, and a deep neural network-based SR (Super-resolution) transformation model ( transform model) to generate a singing voice for a song from a pitch skeleton masked with a formant. The singing voice synthesis and modulation apparatus 100 includes all or a part of an input unit 102 , a Mel modulation unit 104 , an SR inference unit 106 , and an output unit 108 . Here, components included in the apparatus 100 for synthesizing and modulating a voice according to the present embodiment are not necessarily limited thereto. For example, the apparatus 100 for synthesizing and modulating a singing voice may additionally include a training unit (not shown) for training the inference model and the SR transformation model, or may be implemented in a form that interworks with an external training unit.

1 is an exemplary configuration according to the present embodiment, and various implementations including different components or different connections between components depending on the type of input, the structure and operation of the inference model and the SR transformation model, and the type of output This is possible.

The input unit 102 according to the present embodiment obtains a user request for designating a target singer and a song, obtains a singer Mel spectrogram for the singer, and text for the lyrics of the song, and obtains MIDI for the song Acquire the pitch from the data. Here, the user request designates a target singer from among a plurality of singers and a song to be sung by the target singer from among the plurality of songs.

In addition, MIDI data is MIDI data about the existing singing voice which expresses a designated song. Accordingly, any MIDI data capable of expressing the pitch constituting a song may be used.

The text is expressed as a vector, for example, in the case of Hangeul, the initial consonant, the middle consonant, and the final consonant (onset, nucleus, and coda) including one syllable are expressed as a vector. The pitch is expressed as a vector, and it is assumed to include the start and end of the note, that is, the length of the note. In the Mel spectrogram, each coefficient generated by the Mel filter bank is expressed as a vector.

In this embodiment, in order to appropriately express the characteristics of the singing voice while reducing the complexity of the inference model, a low-resolution Mel spectrogram is used to express the singing voice, but the present invention is not limited thereto. Therefore, any type of low-resolution data capable of expressing the characteristics of a singing voice, such as data on a frequency domain or a time domain generated using another signal processing method, may be used.

In the singing voice synthesis process, the singer Mel spectrogram is used as a seed representing the characteristics of the target singer's voice. The singer Mel spectrogram may be generated from the singing voice of a short section for the target singer. For example, after sampling 12 seconds of singing voice at 22.05 KHz, setting the size of each window and hop to 1,024, and generating an 80-dimensional Mel spectrogram, approximately 256 frames of A Mel spectrogram may be obtained. The singer Mel spectrogram obtained in this way may be repeatedly applied to the singing voice synthesis and modulation apparatus 100 while the singing voice for a specified song is synthesized.

The input unit 102 may generate a singer Mel spectrogram by acquiring the singing voice of a designated singer from a storage device (not shown). Alternatively, the singer Mel spectrogram may be obtained directly from the storage device. The input unit 102 may obtain text representing lyrics of a specified song from a storage device. Also, the input unit 102 may obtain MIDI data for a song from a storage device and extract a pitch included in the MIDI data. Accordingly, the storage device stores the singing voices of the plurality of singers or the singer Mel spectrogram, texts for the plurality of songs, MIDI data, and the like.

The Mel modulator 104 according to the present embodiment generates a low-resolution inferred Mel spectrogram from the singer Mel spectrogram, the text, and the pitch using an inference model based on a deep neural network. More specifically, the inference model generates singer ID (Identity) characteristics for the target singer from the singer Mel spectrogram, that is, the tone and singing style, generates a formant mask from the text and tone, the pitch and A pitch skeleton is generated from a singing style, and a low-resolution inferred mel spectrogram is generated from the pitch skeleton and formant mask. The inference model of the Mel modulator 104 is a singer ID encoder 121, a text encoder 122, a Mel spectrogram encoder 123, a pitch encoder 124, an attention unit 125, and a formant mask decoder. 131 , and includes all or part of the tone high skeleton decoder 132 and the masking unit 133 .

The inference model according to the present embodiment is implemented as a deep learning-based deep neural network based on a plurality of convolution layers, but is not necessarily limited thereto. For example, any deep neural network having a recurrent structure, such as a Recurrent Neural Network (RNN) or Long Short-Term Memory (LSTM), may be used. The inference model may be pre-trained using the learning model. The structure of the learning model and the training process of the learning model will be described later.

2 is a block diagram of a singer ID encoder, a formant mask decoder, and a pitch skeleton decoder according to an embodiment of the present invention.

The singer ID encoder 121 extracts a singing style and tone from the singer Mel spectrogram as global features for the target singer. 2, using two one-dimensional convolutional layers (conv1d), two Rectified Linear Units (ReLU), and an average time pooling layer, the mantissa ID encoder 121 is a mantissa A time-invariant global characteristic in which the change with time is removed from the Mel spectrogram is obtained. Time-invariant global properties can be converted into singer ID embeddings that express tone and singing style based on a dense layer and tile process.

The text encoder 122 extracts text characteristics from the text. The text encoder 122 is implemented based on a convolutional layer, and the extracted text characteristic indicates a characteristic for the pronunciation of lyrics.

The Mel spectrogram encoder 123 is implemented based on a convolutional layer and auto-regressively extracts audio characteristics from an initial condition. As an initial condition, zero may be used, and the inferred Mel spectrogram of the previous time is fed back and used as an input of the Mel spectrogram encoder 123 .

The pitch encoder 124 extracts pitch characteristics from the pitch, and is implemented based on a convolutional layer.

The attention unit 125 concatenates the result of attention between the text characteristic and the audio characteristic to the audio characteristic to generate a synchronized audio characteristic in which the synchronization between the text characteristic and the audio characteristic is matched.

The formant mask decoder 131 generates a formant mask by conditioning and combining the text characteristic with the tone, which is a global characteristic. 2, the formant mask decoder 131 is a convolution layer, an adjustment layer, a Highway Causal Convolution (HWC) layer (see Non-Patent Document 3), and a sigmoid (activation function) as an activation function. sigmoid) function and combined convolutional layer. In the adjustment layer of the formant mask decoder 131, the tone is adjusted and combined during the singer ID embedding, which is a global characteristic, so that a formant mask, which is a characteristic related to pronunciation and tone, can be generated from the text characteristic.

The pitch skeleton decoder 132 generates a pitch skeleton by adjusting and combining a singing style and a pitch characteristic with the synchronized audio characteristic. As shown in FIG. 2 , it includes a convolutional layer, an adjustment layer, a highway non-causal convolution (HWNC) layer (see Non-Patent Document 3), and a convolutional layer combined with a sigmoid function as an activation function. In the adjustment layer of the pitch skeleton decoder 132, the singing style of the singer ID embedding, which is a global characteristic, and the pitch characteristic, which is a local feature, are adjusted and combined, so that the pitch and style-related characteristic from the synchronized audio characteristic is the pitch skeleton. can be created.

The adjustment process performed in the adjustment layer of the formant mask decoder 131 and the pitch skeleton decoder 132 may be expressed by Equation (1).

은 음색이며,

는 영(zero) 입력이다. 또한 음고골격 디코더(132)의 경우, x는 동기 오디오 특성이고,

은 가창 스타일이며,

는 음고 특성이다. 기호 σ(시그모이드) 및 ReLU는 활성함수이고, 기호 ⊙는 구성요소 별(element-wise) 승산을 나타내는 연산자이다. 기호 *는 콘볼루션을 의미하는 연산자이고,

,

및

는 콘볼루션 연산을 위한 가중치이다. Here, in the case of the formant mask decoder 131, z is the output of the adjustment layer, x is the text characteristic,

is the tone,

is a zero input. Also, in the case of the pitch skeleton decoder 132, x is a synchronous audio characteristic,

is a singing style,

is a pitch characteristic. The symbols σ (sigmoid) and ReLU are activation functions, and the symbol ⊙ is an operator representing element-wise multiplication. The symbol * is an operator that means convolution,

,

and

is the weight for the convolution operation.

As described above, the singer ID embedding, which is the output of the singer ID encoder 121, can independently perform a function of adjusting a tone when the combined target is text, and adjusting a singing style in the case of an audio characteristic.

The masking unit 133 generates an inferred Mel spectrogram by masking the pitch skeleton using a formant mask. Here, the masking refers to the process of multiplying the formant mask and the tone high skeleton for each component.

The inferred Mel spectrogram generated by the inference model of the Mel modulator 104 is data of a frequency domain for a singing voice.

The SR inference unit 106 according to the present embodiment generates a high-resolution linear spectrogram by up-sampling the inferred Mel spectrogram using the SR transformation model. The SR reasoning unit 106 may generate singing voice data of improved quality in the frequency domain by applying the SR technique to the inferred Mel spectrogram.

The SR transformation model according to this embodiment is implemented as a deep learning-based deep neural network based on a plurality of convolutional layers. The SR transformation model may be trained in advance using a learning model. The structure of the learning model and the training process of the learning model will be described later.

The output unit 108 according to the present embodiment converts the linear spectrogram to generate a singing voice in an auditory form. The output unit 108 may generate an auditory form of singing voice in the time domain from the linear spectrogram in the frequency domain and provide it to the user.

3 is a flowchart of a method for synthesizing and modulating a singing voice according to an embodiment of the present invention.

The singing voice synthesis and modulation apparatus 100 according to an embodiment of the present invention obtains a user request for designating a singer and a song, and obtains a singer Mel spectrogram for the singer, and text for the lyrics of the song and obtains a pitch from the MIDI data for the song (S300).

Here, the song query designates a target singer from among a plurality of singers and a song to be sung by the target singer from among a plurality of songs. In addition, MIDI data is MIDI data about the existing singing voice which expresses a designated song.

The singing voice synthesis and modulation apparatus 100 may generate singer Mel spectrogram data by acquiring the singing voice of a designated singer from the storage device. Alternatively, the singer Mel spectrogram data may be directly acquired from the storage device. The apparatus 100 for synthesizing and modulating a singing voice may obtain text representing lyrics of a specified song from a storage device. Also, the singing voice synthesis and modulation apparatus 100 may obtain MIDI data for a song from a storage device and extract a pitch included in the MIDI data.

The singing voice synthesis and modulation device 100 generates a low-resolution inferred Mel spectrogram from a singer Mel spectrogram, text, and pitch using a pre-trained deep neural network-based reasoning model (S302). More specifically, the inference model generates singer ID (Identity) characteristics for the target singer from the singer Mel spectrogram, that is, the tone and singing style, generates a formant mask from the text and tone, and the pitch and the pitch from the singing style. A skeleton is created, and a low-resolution inferred Mel spectrogram is generated from the pitch skeleton and the formant mask.

The inference model according to this embodiment is implemented as a deep learning-based deep neural network based on a plurality of convolutional layers. The inference model may be pre-trained using the learning model. The structure of the learning model and the training process of the learning model will be described later.

The singing voice synthesis and modulation device 100 generates a high-resolution linear spectrogram by up-sampling the inferred Mel spectrogram using a pre-trained deep neural network-based SR transformation model (S304).

The SR transformation model according to this embodiment is implemented as a deep learning-based deep neural network based on a plurality of convolutional layers. The SR transformation model may be trained in advance using a learning model. The structure of the learning model and the training process of the learning model will be described later.

The singing voice synthesis and modulation apparatus 100 converts the linear spectrogram to generate an auditory type of singing voice (S306). The apparatus 100 for synthesizing and modulating a singing voice may generate a audible voice in a time domain from a linear spectrogram on a frequency domain and provide it to a user.

Hereinafter, a process ( S302 ) in which the inference model generates an inference Mel spectrogram will be described in detail.

The inference model extracts the singing style and timbre of the singer from the singer Mel spectrogram (S320). The inference model generates timbre and singing style as singer ID embeddings, which are time-invariant global properties with the change over time removed from the singer Mel spectrogram.

The inference model extracts text characteristics from the text (S322). The extracted text characteristics represent the characteristics for the pronunciation of lyrics.

The inference model auto-regressively extracts audio characteristics from the initial conditions (S324). Zero may be used as an initial condition, and the inferred Mel spectrogram of the previous time is fed back and used to extract audio characteristics.

The inference model extracts a pitch characteristic from the pitch (S326).

The inference model concatenates the result of attention between the text feature and the audio feature to the audio feature to match the synchronization between the text feature and the audio feature. A characteristic is created (S328).

The inference model generates a formant mask in which the tone is conditioned from the text characteristics (S330). As the tone is adjusted and combined during embedding of the singer ID, which is a global characteristic, a formant mask, which is a characteristic related to pronunciation and tone, may be generated from the text characteristic.

The inference model is synchronous audio From the characteristics, a tone height skeleton in which the singing style and tone characteristics are adjusted is generated (S332). A singing style and a pitch characteristic, which is a local characteristic, are adjusted and combined during singer ID embedding, which is a global characteristic, so that a pitch skeleton, which is a characteristic related to a pitch and a style, can be generated from the synchronized audio characteristic.

The inference model generates an inference Mel spectrogram by masking the pitch skeleton using a formant mask (S334). Here, the masking refers to the process of multiplying the formant mask and the tone high skeleton for each component.

As described above, according to this embodiment, a tone and a singing style for a singer are generated using an inference model based on a deep neural network, a tone-controlled formant, and a tone high skeleton with an adjusted singing style are generated, By providing a singing voice synthesis/modulation device and method for generating a singing voice for a song from a pitch skeleton masked with a formant using a deep neural network-based SR transformation model, an end-to-end network It has the effect of enabling the synthesis/modulation of singing voices based on the

As described above, the singer ID embedding, which is the output of the singer ID encoder 121, independently performs a function of adjusting the tone when the object to be combined is text, and adjusting the singing style in the case of audio characteristics. Thus, by applying the inference model twice (or using two inference models) to generate singer ID embeddings for each of the two singers, the singer's timbre and other A singer's singing style can independently generate a cross-reflected singing voice.

4 is a block diagram of an apparatus for synthesizing and modulating a singing voice according to another embodiment of the present invention.

In another embodiment of the present invention, the singing voice synthesis and modulation device 100 obtains a user request specifying a tone of a first singer, a singing style of a second singer, and a song, and is based on a deep neural network. generating the tone of the first singer and the singing style of the second singer using the inference model of , to generate a singing voice for a song from a pitch skeleton masked with a formant using a deep neural network-based SR transformation model (inference model). The singing voice synthesis and modulation apparatus 100 includes all or a part of an input unit 102 , a Mel modulation unit 104 , a masking unit 401 , an SR reasoning unit 106 , and an output unit 108 . Here, components included in the apparatus 100 for synthesizing and modulating a voice according to the present embodiment are not necessarily limited thereto. For example, the apparatus 100 for synthesizing and modulating a singing voice may additionally include a training unit (not shown) for training the inference model and the SR transformation model, or may be implemented in a form that interworks with an external training unit.

The input unit 102 obtains a user request specifying the tone of the first singer, the singing style of the second singer, and a song, and obtains a first singer Mel spectrogram for the first singer and a second singer Mel for the second singer The spectrogram and text for the lyrics of the song are obtained, and the pitch is obtained from the MIDI data for the song.

Here, the user request designates a first singer for a tone color, a second singer for a singing style, and a song to be sung by a target singer among a plurality of songs from among a plurality of singers. In addition, MIDI data is MIDI data about the existing singing voice which expresses a designated song.

The Mel modulator 104 generates a singer ID characteristic for the first singer, that is, a first tone and a first singing style, from the first singer Mel spectrogram using an inference model based on a deep neural network, and generates the text and the first tone. A first formant mask is generated from , and a first pitch skeleton is generated from the pitch and the first singing style. Also, the Mel modulator 104 generates a low-resolution first inferred Mel spectrogram by masking the first pitch skeleton using the first formant mask, and automatically operates the inference model to automatically regressively operate the state for the first mantissa. update the value Here, the state value of the first mantissa refers to parameters of the inference model, an input in which calculation is performed, an output of an intermediate layer, a final output, and the like.

In addition, the Mel modulator 404 uses a deep neural network-based reasoning model to generate singer ID characteristics for the second singer from the second singer Mel spectrogram, that is, the second tone and the second singing style, and the text and the second singing style. A second formant mask is generated from the two tones, and a second pitch skeleton is generated from the pitch and the second singing style. In addition, the Mel modulator 104 generates a low-resolution second inferred Mel spectrogram by masking the second pitch skeleton using the second formant mask, and automatically operates the inference model to automatically regressively operate the state for the second mantissa. update the value Here, the state value of the second mantissa refers to parameters of the inference model, an input in which calculation is performed, an output of an intermediate layer, a final output, and the like.

The inference model according to this embodiment is implemented as a deep learning-based deep neural network based on a plurality of convolutional layers. The inference model may be pre-trained using the learning model. The structure of the learning model and the training process of the learning model will be described later.

The masking unit 401 generates a low-resolution third inferred Mel spectrogram by masking the second pitch skeleton using the first formant mask. Here, the masking refers to a process of multiplying the first formant mask and the second pitch skeleton for each component. Accordingly, the generated third inferred Mel spectrogram is a Mel spectrogram for a singing voice in which the tone of the first singer and the singing style of the second singer are combined.

Although FIG. 4 shows a diagram sequentially using one Mel modulation unit, in another embodiment of the present invention, the singing voice synthesis and modulation apparatus 100 uses two Mel modulation units to form the first singer's formant. It is possible to generate the mask and the high-pitched skeleton of the second singer in parallel.

4, when the first singer and the second singer are the same, the singing voice synthesis and modulation device 100, as shown in FIG. Mel spectrogram can be generated.

The SR inference unit 106 generates a high-resolution linear spectrogram by up-sampling the third inferred Mel spectrogram using the SR transformation model.

The SR transformation model according to this embodiment is implemented as a deep learning-based deep neural network based on a plurality of convolutional layers. The SR transformation model may be trained in advance using a learning model. The structure of the learning model and the training process of the learning model will be described later.

The output unit 108 according to the present embodiment converts the linear spectrogram to generate a singing voice in an auditory form. The output unit 108 may generate an auditory form of singing voice in the time domain from the linear spectrogram in the frequency domain and provide it to the user.

5 is a flowchart of a method for synthesizing and modulating a singing voice according to another embodiment of the present invention.

The singing voice synthesis and modulation device 100 obtains a user request specifying the tone of the first singer, the singing style of the second singer, and a song, and provides the first singer Mel spectrogram for the first singer and the second singer. A second singer Mel spectrogram and text for lyrics of a song are acquired, and a pitch is acquired from MIDI data for a song (S500).

Here, the user request designates a first singer for a tone color, a second singer for a singing style, and a song to be sung by a target singer among a plurality of songs from among a plurality of singers. In addition, MIDI data is MIDI data about the existing singing voice which expresses a designated song.

The singing voice synthesis and modulation device 100 generates a singer ID characteristic for the first singer from the first singer Mel spectrogram using a pre-trained deep neural network-based inference model, that is, the first tone and the first singing style. Then, a first formant mask is generated from the text and the first tone, and a first pitch skeleton is generated from the pitch and the first singing style (S502). In addition, the singing voice synthesis and modulation device 100 generates a low-resolution first inferred Mel spectrogram by masking the first pitch skeleton using the first formant mask, and automatically operates the inference model to automatically regressively operate the first singer update the status value for Here, the state value of the first mantissa refers to parameters of the inference model, an input in which calculation is performed, an output of an intermediate layer, a final output, and the like.

The singing voice synthesis and modulation device 100 generates a singer ID characteristic for the second singer from the second singer Mel spectrogram using a pre-trained deep neural network-based inference model, that is, a second tone and a second singing style. Then, a second formant mask is generated from the text and the second tone, and a second pitch skeleton is generated from the pitch and the second singing style (S504). In addition, the singing voice synthesis and modulation device 100 generates a second inferred Mel spectrogram of low resolution by masking the second pitch skeleton using the second formant mask, and automatically operates the inference model to automatically regressively operate the second mantissa. update the status value for Here, the state value of the second mantissa refers to parameters of the inference model, an input in which calculation is performed, an output of an intermediate layer, a final output, and the like.

The inference model according to this embodiment is implemented as a deep learning-based deep neural network based on a plurality of convolutional layers. The inference model may be pre-trained using the learning model. The structure of the learning model and the training process of the learning model will be described later.

The singing voice synthesizing and modulating apparatus 100 generates a low-resolution third inferred Mel spectrogram by masking the second pitch skeleton using the first formant mask (S506). Here, the masking refers to a process of multiplying the first formant mask and the second pitch skeleton for each component. Accordingly, the generated third inferred Mel spectrogram is a Mel spectrogram for a singing voice in which the tone of the first singer and the singing style of the second singer are combined.

When the first singer and the second singer are the same, the apparatus 100 for synthesizing and modulating a singing voice may generate a Mel spectrogram for a singing voice in which the tone and singing style of one singer are combined.

The singing voice synthesis and modulation device 100 generates a high-resolution linear spectrogram by up-sampling the inferred Mel spectrogram using a pre-trained SR transformation model (S508).

The SR inference model according to this embodiment is implemented as a deep learning-based deep neural network based on a plurality of convolutional layers. The SR inference model may be trained in advance using the learning model. The structure of the learning model and the training process of the learning model will be described later.

The singing voice synthesis and modulation apparatus 100 converts the linear spectrogram to generate an auditory type of singing voice (S510). The apparatus 100 for synthesizing and modulating a singing voice may generate a audible voice in a time domain from a linear spectrogram on a frequency domain and provide it to a user.

As described above, according to this embodiment, the tone of the first singer and the tone of the first singer and the tone and the tone of the second singer are obtained using an inference model based on a deep neural network by obtaining a user request specifying a song and a singing style of the second singer. By providing a singing voice synthesis/modulation apparatus and method for generating a singing style of a second singer, and independently generating a formant in which the tone of the first singer is adjusted, and a pitch skeleton in which the singing style of the second singer is adjusted, It has the effect of enabling the synthesis/modulation of singing voices that independently cross-regulate the tone and singing style.

As described above, the apparatus 100 for synthesizing and modulating a singing voice according to the present embodiment includes a deep learning-based learning model, and a training process for an inference model and an SR transformation model using the provided learning model can be performed. This learning model creates a tone and a singing style for a target singer using an inference model based on a deep neural network, and independently generates a tone-adjusted formant and a tone-high skeleton with an adjusted singing style, and Using the SR transformation model, the singing voice for the song is generated from the pitch skeleton masked with the formant, and the pair of inferred Mel spectrogram and linear spectrogram and GT (Ground Truth) using a deep neural network-based discriminator It may be a model trained to discriminate between pairs of inference Mel spectrograms and GT linear spectrograms.

In this embodiment, the combined form of the Mel modulator 104 and the SR inference unit 106 of the singing voice synthesis and modulation device 100 is used as a generator, and includes a generator and a discriminator. An inference model and an SR transformation model may be trained using the Generative Adversarial Networks (GAN)-based learning model 600 . In this embodiment, by adopting the GAN-based learning model 600 , it is possible to train the inference model and the SR transformation model of the singing voice synthesis and modulation device 100 to generate a more realistic singing voice for the target singer.

6 is a block diagram of a learning model according to an embodiment of the present invention.

In the embodiment according to the present invention, training is performed on the inference model and the SR transformation model of the singing voice synthesis and modulation device 100 using the GAN-based learning model 600 . The learning model 600 includes all or part of an input unit 102 , a generator 602 including a Mel modulator 104 , and an SR inference unit 106 , and a discriminator 604 . Here, components included in the learning model 600 according to the present embodiment are not necessarily limited thereto. For example, the learning model 600 may additionally include a training unit (not shown) for training the inference model and the SR transformation model, or may be implemented in a form that is linked with an external training unit.

The input unit 102 obtains a singer Mel spectrogram for each singer, a text for each song lyrics, and an audio Mel spectrogram for each song with respect to a plurality of singers and a plurality of songs for learning, and each song Obtain the pitch from MIDI data for

MIDI data is MIDI data about the existing singing voice which expresses each of a some song. Accordingly, any MIDI data capable of expressing the pitch constituting a song may be used.

In the present embodiment, the singer's singing voice is expressed using a low-resolution Mel spectrogram in order to appropriately express the characteristics of the singing voice while reducing the complexity of the inference model, but the present invention is not limited thereto. Therefore, any type of low-resolution data capable of expressing the characteristics of a singing voice, such as data on a frequency domain or a time domain generated using other signal processing methods, may be used.

The singer Mel spectrogram for learning is used as a seed representing the characteristics of the singer's voice. Singer Mel spectrogram can be generated from singing voice of a short section. For example, if 12 seconds of singing voice is sampled at 22.05 KHz, window and hop sizes are set to 1,024 each, and an 80-dimensional Mel spectrogram is generated, approximately 256 frames of Mel spectrogram are generated. A spectrogram may be obtained. The singer Mel spectrogram for a plurality of singers thus obtained may be randomly applied to the learning model 600 while the singing voice for a specified song is synthesized/modulated.

The audio Mel spectrogram for learning may be obtained from an existing singing voice. Therefore, when the existing singing voice is the singing voice of a singer for learning, some sections of the existing singing voice may be used to generate the singer Mel spectrogram for learning.

The input unit 102 may generate a singer Mel spectrogram by acquiring a singing voice for each of a plurality of singers from a storage device (not shown). Alternatively, the singer Mel spectrogram may be obtained directly from the storage device. The input unit 102 may obtain text representing the lyrics of each of the plurality of songs from the storage device. The input unit 102 may generate an audio Mel spectrum by acquiring pre-existing singing voices for each of a plurality of songs from a storage device. Alternatively, the audio melt spectrum may be obtained directly from the storage device. Also, the input unit 102 may obtain MIDI data for each of a plurality of songs from the storage device and extract a pitch included in the MIDI data.

The Mel modulator 104 generates a low-resolution inferred Mel spectrogram from the singer Mel spectrogram, text, audio Mel spectrogram, and pitch by using the inference model. More specifically, the inference model generates singer ID (Identity) characteristics for the target singer from the singer Mel spectrogram, that is, the tone and singing style, generates a formant mask from the text and tone, the audio Mel spectrogram, and the pitch and a pitch skeleton from the singing style, and an inferred Mel spectrogram from the pitch skeleton and a formant mask. The inference model according to this embodiment is implemented as a deep learning-based deep neural network based on a plurality of convolutional layers.

Meanwhile, the SR inference unit 106 generates a high-resolution linear spectrogram by upsampling the inferred Mel spectrogram using the SR transformation model. The SR transformation model according to this embodiment is implemented as a deep learning-based deep neural network based on a plurality of convolutional layers.

Therefore, the generator 602 of the GAN-based learning model 600 generates an inference Mel spectrogram as an intermediate output and a linear spectrogram as a final output.

Except for the mel spectrogram encoder 123 included in the mel modulator 104, detailed structures and operations of the mel modulator 104 and the SR inference unit 106 are Since it has already been described, further description will be omitted.

Different from the singing voice synthesis and modulation device 100 , in the learning model, an audio Mel spectrogram for an existing singing voice is used for training of the inference model. Accordingly, the Mel spectrogram encoder 123 extracts audio characteristics from the audio Mel spectrogram. Similar to the singing voice synthesis and modulation device 100 , an auto-regressive operation is performed by feeding back an inference Mel spectrogram of a previous time and concatenating it to an audio Mel spectrogram. Accordingly, the Mel modulator 104 using the mantissa Mel spectrogram performs the same operation as modulating the audio Mel spectrogram.

와 GT 추론 멜 스펙트로그램의 확률 분포 p(M)이 유사하다는 가정 하에, 추론 모델 및 SR 변환 모델은 공동으로(jointly) 트레이닝될 수 있다. 따라서, 구별기(604)는 추론 멜 스펙트로그램

및 선형 스펙트로그램

의 페어(pair)와 GT 추론 멜 스펙트로그램 M 및 GT 선형 스펙트로그램 S의 페어 간을 구별할 수 있다. 페어 간 구별을 위하여, 구별기(604)는, 선형 스펙트로그램에 추론 멜 스펙트로그램을 조절하여 가산함으로써 허(fake)출력을 생성하고, GT 선형 스펙트로그램에 GT 추론 멜 스펙트로그램을 조절하여 가산함으로써 진(true)출력을 생성할 수 있다. The discriminator 604 according to the present embodiment discriminates the linear spectrogram that is the final output of the generator 602 and the GT linear spectrogram. Probability Distribution of Inferred Mel Spectrogram

Assuming that the probability distribution p(M) of the GT inference Mel spectrogram is similar to , the inference model and the SR transformation model can be jointly trained. Thus, the discriminator 604 is an inferred Mel spectrogram.

and linear spectrogram

It is possible to distinguish between a pair of , and a pair of GT inferred Mel spectrogram M and GT linear spectrogram S. For pair-to-pair discrimination, the discriminator 604 generates a fake output by adjusting and adding the inferred Mel spectrogram to the linear spectrogram, and adjusts and adds the GT inferred Mel spectrogram to the GT linear spectrogram. Can produce true output.

The discriminator 604 is implemented as a deep learning-based deep neural network based on multiple convolutional layers.

When training the generator 602 and the discriminator 604 , the training unit may use various types of distance metric-based loss in addition to the loss based on the GAN structure.

The training unit according to the present embodiment uses an adversarial loss as shown in Equation 2 (see Non-Patent Document 4).

는 생성기(602) G의 GAN 손실이고,

는 구별기(604) D의 GAN 손실이다. 함수 f는 스칼라 함수로서 예로는 시그모이드 함수가 존재한다.here,

is the GAN loss of the generator 602 G,

is the GAN loss of the discriminator 604 D. The function f is a scalar function, and an example of a sigmoid function exists.

을 이용한다.The training unit loses inference as shown in Equation 3 for training of the inference model.

use the

는 유도 어텐션 손실(guided attention loss)이다(비특허문헌 3 참조). 마지막 항에서

은 시간에 따른 추론 멜 스펙트로그램의 증분(increment)이고,

은 시간에 따른 GT 추론 멜 스펙트로그램의 증분이다. 따라서, 마지막 항은 증분

과

간의 거리 메트릭에 기반하는 손실이다.Here, the first term is the loss based on the distance metric between the inferred Mel spectrogram and the GT inferred Mel spectrogram,

is a guided attention loss (refer to Non-Patent Document 3). in the last paragraph

is the increment of the inferred Mel spectrogram over time,

is the increment of the GT inferred Mel spectrogram over time. Therefore, the last term is incremented

class

It is a loss based on the distance metric between

을 이용한다.The training unit SR loss based on the metric distance between the linear spectrogram and the GT linear spectrogram for training the SR transformation model.

use the

Here, as the distance metric, any one (or a combination thereof) that can express the metric difference between two comparison objects, such as cross entropy, L1, and L2 metrics, can be used.

Combining the above losses, the total loss of the generator 602 and the discriminator 604 of the GAN-based learning model can be expressed by Equations 4 and 5.

는 생성기(602)의 총손실이고,

는 구별기(604)의 총손실이며,

및

는 손실에 관련된 하이퍼파라미터이다.In Equations 4 and 5,

is the total loss of the generator 602,

is the total loss of the discriminator 604,

and

is a hyperparameter related to the loss.

The training unit according to the present embodiment performs training on the generator 602 and the discriminator 604 by updating the parameters of the generator 602 and the discriminator 604 in a direction in which the total loss is reduced.

Also, parameters of the generator 602 and the discriminator 604 may be updated in a direction in which all or part of the loss items included in the total loss are reduced.

In addition, at least one parameter of the generator 602 and the discriminator 604 may be updated in a direction in which all or part of the loss items included in the total loss are reduced.

Training of GAN-based deep learning models is known to be difficult. In particular, it can be difficult to implement stable training in the early stages of learning. Accordingly, the training unit according to the present embodiment may increase the learning efficiency of the learning model 600 by changing the settings for each of the hyperparameters. In the initial stage of training, the training unit may perform training by setting hyperparameters for some items among the total losses expressed in Equations 4 and 5 to zero. For example, only the synthetic loss item may be activated, and the antagonistic loss and SR loss items may be deactivated.

In the later stage when the operation of the inference model among the generators 602 is stable, the training unit sets the hyperparameters that were set to zero to non-zero values, so that the parameters of the generator 602 and the discriminator 604 are calculated using all loss items. can be updated.

7 is a flowchart of a learning method for a learning model according to an embodiment of the present invention.

For a plurality of singers and a plurality of songs, the training unit of the learning model 600 obtains a singer Mel spectrogram for each singer, a text for each song lyrics, and an audio Mel spectrogram for each song, each song A pitch is obtained from the MIDI data for (S700). MIDI data is MIDI data about the existing singing voice expressing each of a some song.

The training unit may generate a singer Mel spectrogram by acquiring a singing voice for each of the plurality of singers from the storage device. Alternatively, the singer Mel spectrogram may be obtained directly from the storage device. The training unit may obtain text representing lyrics of each of the plurality of songs from the storage device. The training unit may generate an audio Mel spectrum by acquiring pre-existing singing voices for each of the plurality of songs from the storage device. Alternatively, the audio melt spectrum may be obtained directly from the storage device. Also, the training unit may obtain MIDI data for each of a plurality of songs from the storage device and extract a pitch included in the MIDI data.

The training unit generates a low-resolution inference Mel spectrogram from the singer Mel spectrogram, text, audio Mel spectrogram, and pitch by using a deep neural network-based reasoning model (S702). More specifically, the inference model generates singer ID (Identity) characteristics for the target singer from the singer Mel spectrogram, that is, the tone and singing style, generates a formant mask from the text and tone, the audio Mel spectrogram, and the pitch and a pitch skeleton from the singing style, and an inferred Mel spectrogram from the pitch skeleton and the formant mask.

The inference model according to this embodiment is implemented as a deep learning-based deep neural network based on a plurality of convolutional layers.

The training unit generates a high-resolution linear spectrogram by up-sampling the inferred Mel spectrogram using the deep neural network-based SR transformation model (S704).

The SR transformation model according to this embodiment is implemented as a deep learning-based deep neural network based on a plurality of convolutional layers.

The training unit distinguishes between a pair of inferred Mel spectrogram and linear spectrogram and a pair of GT inferred Mel spectrogram and GT linear spectrogram using a deep neural network-based discriminator (S706).

The discriminator 604 according to the present embodiment is implemented as a deep learning-based deep neural network based on a plurality of convolutional layers.

The training unit calculates a total loss by using the inference model, the SR transformation model, and the output of the discriminator (S708).

Since the contents of each loss item constituting the total loss have already been described, further detailed description will be omitted.

The training unit updates at least one parameter of the inference model, the SR transformation model, and the discriminator in a direction in which all or part of the loss items included in the total loss are reduced ( S710 ).

Hereinafter, a process ( S702 ) in which the inference model generates an inference Mel spectrogram will be described in detail.

The inference model extracts the singing style and timbre of the singer from the singer Mel spectrogram (S720). The inference model generates timbre and singing style as singer ID embeddings, which are time-invariant global properties with the change over time removed from the singer Mel spectrogram.

The inference model extracts text characteristics from the text (S722). The extracted text characteristics indicate the characteristics of the pronunciation of the lyrics.

The inference model extracts audio characteristics from the audio melt spectrogram (S724). The inference Mel spectrogram of the previous time is fed back and concatenated to the audio Mel spectrogram, whereby an auto-regressive operation is performed. Therefore, using the mantissa Mel spectrogram, the inference model performs the same operation as modulating the audio Mel spectrogram.

The inference model extracts a pitch characteristic from the pitch (S726).

The inference model concatenates the result of attention between the text feature and the audio feature to the audio feature to match the synchronization between the text feature and the audio feature. A characteristic is created (S728).

The inference model generates a tone-conditioned formant mask from the text characteristics (S730). As the tone is adjusted and combined during embedding of the singer ID, which is a global characteristic, a formant mask, which is a characteristic related to pronunciation and tone, may be generated from the text characteristic.

The inference model is synchronous audio From the characteristics, a tone height skeleton in which the singing style and tone characteristics are adjusted is generated (S732). A singing style and a pitch characteristic, which is a local characteristic, are adjusted and combined during singer ID embedding, which is a global characteristic, so that a pitch skeleton, which is a characteristic related to a pitch and a style, can be generated from the synchronized audio characteristic.

The inference model generates an inference Mel spectrogram by masking the pitch skeleton using a formant mask (S734). Here, the masking refers to the process of multiplying the formant mask and the tone high skeleton for each component.

A device (not shown) on which the singing voice synthesis and modulation apparatus 100 according to the present embodiment is mounted may be a programmable computer and includes at least one communication interface capable of being connected to a server (not shown).

The training for the inference model and the SR transformation model as described above can be carried out in the device on which the singable speech synthesis and modulation apparatus 100 is mounted by using the computing power of the device on which the singing voice synthesis and modulation apparatus 100 is mounted. have.

Training for the inference model and the SR transformation model as described above may be performed in the server. The training unit of the server may perform training on the deep learning model having the same structure as the inference model and the SR transformation model, which are components of the apparatus 100 for synthesizing and modulating a singing voice mounted on the device. Using a communication interface connected to the device, the server transmits the parameters of the trained deep learning model to the device, and the apparatus 100 for synthesizing and modulating the singing voice using the received parameters sets the parameters of the inference model and the SR transformation model. can In addition, parameters of the inference model and the SR transformation model may be set at the time of shipment of the device or the time when the apparatus 100 for synthesizing and modulating the audible voice is mounted on the device.

As described above, according to this embodiment, a user request for designating a singer and a song is obtained, a tone and a singing style for the singer are generated using an inference model based on a deep neural network, and the tone is adjusted formant; And by providing a singing voice synthesis/modulation device and method for generating a pitch skeleton with a controlled singing style, and generating a singing voice for a song from a pitch skeleton masked with a formant using an SR transformation model based on a deep neural network, There is an effect that natural singing voice synthesis/modulation for a plurality of singers is possible.

Although it is described that each process is sequentially executed in each flowchart according to the present embodiment, the present invention is not limited thereto. In other words, since it may be applicable to change and execute the processes described in the flowchart or to execute one or more processes in parallel, the flowchart is not limited to a time-series order.

Various implementations of the systems and techniques described herein may be implemented in digital electronic circuitry, integrated circuitry, field programmable gate array (FPGA), application specific integrated circuit (ASIC), computer hardware, firmware, software, and/or combination can be realized. These various implementations may include being implemented in one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor) coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device. or may be a general-purpose processor). Computer programs (also known as programs, software, software applications or code) contain instructions for a programmable processor and are stored on a "computer-readable recording medium".

The computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored. These computer-readable recording media are non-volatile or non-transitory, such as ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, and storage device. media, and may further include transitory media such as carrier waves (eg, transmission over the Internet) and data transmission media. In addition, the computer-readable recording medium is distributed in network-connected computer systems, and computer-readable codes may be stored and executed in a distributed manner.

Various implementations of the systems and techniques described herein may be implemented by a programmable computer. Here, the computer includes a programmable processor, a data storage system (including volatile memory, non-volatile memory, or other types of storage systems or combinations thereof) and at least one communication interface. For example, a programmable computer may be one of a server, a network appliance, a set-top box, an embedded device, a computer expansion module, a personal computer, a laptop, a Personal Data Assistant (PDA), a cloud computing system, or a mobile device.

The above description is merely illustrative of the technical idea of this embodiment, and a person skilled in the art to which this embodiment belongs may make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain rather than limit the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present embodiment.

100: singing voice synthesis and modulation device
102: input unit 104: Mel modulation unit
105: SR reasoning unit 108: output unit
121: singer ID encoder 125: attention
131: formant mask decoder 132: tone high skeleton decoder
400: learning model
602: generator 604: distinguisher

Claims (9)

A method for singing voice synthesis and modulation performed by a computing device, the method comprising:
Obtaining a user request specifying the tone of the first singer, the singing style of the second singer, and a song, the first singer spectrogram for the first singer, the second singer spectrogram for the second singer; and obtaining text for the lyrics of the song, and obtaining a pitch of the song from MIDI data for the song.
Using an inference model based on a pre-trained deep neural network, a first timbre for the first singer is generated from the first singer spectrogram, and the text and the a first process of generating a first formant mask from the first tone;
a second process of generating a second singing style for the second singer from the second singer spectrogram using the inference model, and generating a second pitch skeleton from the pitch and the second singing style; and
The process of generating a low-resolution third inferred spectrogram by masking the second pitch skeleton using the first formant mask
Singing voice synthesis and modulation method comprising a. According to claim 1,
generating a high-resolution linear spectrogram by up-sampling the third inferred spectrogram using a pre-trained deep neural network-based SR (super-resolution) transform model; and
The process of converting the linear spectrogram to generate a singing voice in an auditory form
Singing voice synthesis and modulation method, characterized in that it further comprises. According to claim 1,
The first process and the second process are
The process of extracting a singing style and tone for a singer from the singer spectrogram;
extracting text characteristics from the text;
a process of auto-regressively extracting audio characteristics from initial conditions;
extracting a pitch characteristic from the pitch;
Synchronous audio obtained by concatenating an attention result between the text characteristic and the audio characteristic to the audio characteristic to match the synchronization between the text characteristic and the audio characteristic the process of creating a characteristic;
generating a formant mask in which the tone is conditioned from the text characteristics;
said synchronous audio generating a pitch skeleton in which the singing style and the pitch characteristic are adjusted from the characteristics; and
The process of generating an inference spectrogram by masking the pitch skeleton using the phoneme mask
Singing voice synthesis and modulation method comprising a. 4. The method of claim 3,
The process of generating the formant mask is,
The first result of applying the convolution to each of the text characteristics and the tone, and the summation result of applying a sigmoid activation function, and a first result of applying a rectified linear unit (ReLU) activation function. 2 After calculating the result, the method for synthesizing and modulating a singing voice comprising the step of performing element-wise multiplication between the first result and the second result. 4. The method of claim 3,
The process of generating the eumgo skeleton is,
After calculating the summation result by applying convolution to each of the text characteristic, the synchronous audio characteristic, and the tone, a third result applied to the sigmoid activation function, and a fourth result applied to the ReLU activation function , an adjustment process of performing multiplication for each component between the third result and the fourth result. Obtaining a user request specifying the tone of the first singer, the singing style of the second singer, and a song, the first singer spectrogram for the first singer, the second singer spectrogram for the second singer; and an input unit for obtaining text for the lyrics of the song and obtaining a pitch of the song from MIDI data for the song;
Using an inference model based on a pre-trained deep neural network, a first timbre for the first singer is generated from the first singer spectrogram, and the text and the create a first formant mask from the first tone; a modulator for generating a second singing style for the second singer from the second singer spectrogram by using the inference model, and generating a second pitch skeleton from the pitch and the second singing style; and
A third masking unit generating a third inferred spectrogram of low resolution by masking the second pitch skeleton using the first formant mask
Singing voice synthesis and modulation device comprising a. 7. The method of claim 6,
an SR inference unit for generating a high-resolution linear spectrogram by up-sampling the third inferred spectrogram using a pre-trained deep neural network-based SR (super-resolution) transform model; and
An output unit that converts the linear spectrogram to generate a singing voice in an auditory form
Singing voice synthesis and modulation device, characterized in that it further comprises. 7. The method of claim 6,
The modulator is
a singer ID encoder for extracting a singing style and timbre for a singer from the singer spectrogram;
a text encoder for extracting text characteristics from the text;
a spectrogram encoder for auto-regressively extracting audio characteristics from initial conditions;
a pitch encoder for extracting pitch characteristics from the pitch;
an attention unit configured to concatenate a result of attention between the text characteristic and the audio characteristic to the audio characteristic to generate a synchronized audio characteristic in which synchronization between the text characteristic and the audio characteristic is matched;
a formant mask decoder for generating the conditioned formant mask from the text characteristics;
a pitch skeleton decoder for generating a pitch skeleton in which the singing style and the pitch characteristic are adjusted from the synchronized audio characteristic; and
A masking unit that generates an inference spectrogram by masking the pitch skeleton using the formant mask
Singing voice synthesis and modulation device comprising a. A computer program stored in a computer-readable recording medium in order to execute each process included in the method for synthesizing and modulating a singing voice according to any one of claims 1 to 5.

Images