TWI836255B - Method and apparatus in designing a personalized virtual singer using singing voice conversion - Google Patents

Abstract

A method and an apparatus in designing a personalized virtual singer using singing voice conversion are provided. In the method, a music score file represented by symbols is parsed to extract multiple lyrics and multiple notes. Audio data of each of the notes is loaded from an audio database. A vocoder is used to perform acoustic modeling on the audio data to adjust each audio data and concatenate the adjusted audio data to generate singing voice data. A generator of a voice conversion model generator is used to convert multiple acoustic features in the singing voice data into output features conditioned on target attributes, and the voice conversion model is trained according to multiple types of losses of the voice conversion model to obtain synthetic singing voice data with optimized output features. This method can customize a fixed timbre virtual singer to generate multiple personalized virtual singers.

Description

Method and device for designing personalized virtual singer through singing voice conversion

The present invention relates to a singing voice synthesis method and device, and in particular to a method and device for designing a personalized virtual singer through singing voice conversion.

Singing synthesis is a technology that generates human singing voices. For a long time, people have tried to use different methods to achieve singing synthesis, such as converting voice conversations into singing a song according to the sheet music and lyrics. However, the timbre conversion of singing voices is mostly a one-to-one conversion, so it is limited to a single singer, or singing any specified song with a specified personalized timbre.

In other words, if you want to convert the timbre of a certain B into the timbre of a certain A, you must have said or sung a certain song. It is impossible to sing the original song sung by a certain B with the timbre of a certain A.

The present invention provides a method and device for designing a personalized virtual singer through voice conversion, which enables the virtual singer to sing any song according to the score and lyrics with the voice of a specific person, that is, to convert the voice of a specific person to the song sung by the virtual singer.

The present invention provides a method for designing a personalized virtual singer through singing voice conversion, which is suitable for electronic devices equipped with processors. This method includes the following steps: parse a music score file represented by a symbol to extract multiple lyrics and multiple notes; load the extracted audio data of each note from an audio database; use a vocoder (vocoder) performs acoustic modeling on audio data to adjust each audio data and splice the adjusted audio data to generate singing voice data; and utilize a generator of a speech conversion model to convert multiple acoustic features in the singing voice data into target The audio attributes are the conditional output features, and the speech conversion model is trained according to various losses of the speech conversion model to obtain synthetic singing data with optimized output features.

The invention provides a device for designing a personalized virtual singer through singing voice conversion, which includes a connection device, a storage device and a processor. Among them, the storage device is used to store computer programs. The processor is coupled to the connection device and the storage device, and is configured to load and execute a computer program in the storage device to: parse a music score file represented by a symbol to retrieve a plurality of lyrics and a plurality of notes; from an audio database Loading the captured audio data of each note; performing acoustic modeling on the audio data using a vocoder to adjust each audio data and splicing the adjusted audio data to generate singing voice data; and utilizing a generator of a speech conversion model Convert multiple acoustic features in the singing data into output features conditioned on the target audio attributes, and train the speech conversion model based on various losses of the speech conversion model to obtain synthetic singing data with optimized output features.

In order to make the above features and advantages of the present invention more clearly understood, the following is a detailed description of the embodiments with the accompanying drawings.

10: Design a personalized virtual singer's device through singing voice conversion

12: Connecting device

14:Storage device

16: Processor

30:System

31: Song Sheet

32: lyrics + notes

33: Audio database

34: Pre-recorded note audio data

35:Synthesized vocal data

40: Vocal Synthesizer

41:Vocoder

42: Basic frequency

43: Spectral envelope

44: Non-periodic envelope

45: Pitch adjustment

46:Duration

47: Singing Expression

48: Speech conversion model

400:StarGAN-VC model

S202~S206, S1~S4: steps

FIG. 1 is a block diagram of a device for designing a personalized virtual singer through voice conversion according to an embodiment of the present invention.

FIG2 is a flow chart of a method for designing a personalized virtual singer through voice conversion according to an embodiment of the present invention.

FIG. 3 is a system architecture diagram of a singing voice synthesizer according to an embodiment of the present invention.

Figure 4 is a schematic diagram of the StarGAN-VC model according to an embodiment of the present invention.

Embodiments of the present invention mainly combine a concatenative singing voice synthesizer and a multi-speaker voice conversion model, which include using a concatenative singing voice synthesizer to parse a music data representation (such as MusicXML) file to synthesize a virtual singing voice. Afterwards, the sound conversion model is used to convert the virtual singing voice into different timbres. The embodiment of the present invention implements a splicing method singing voice synthesizer by pre-recording the pronunciations of all Chinese words. For the voice conversion model, the embodiment of the present invention uses a voice convert (VC) model of a deep adversarial network (GAN) to realize voice conversion between speakers, so that the model can be implemented Now generate multiple sounds with distinguishable timbres.

FIG. 1 is a block diagram of a device for designing a personalized virtual singer through singing voice conversion according to an embodiment of the present invention. Please refer to Figure 1. The device 10 (hereinafter referred to as the device 10) for designing a personalized virtual singer through singing voice conversion according to an embodiment of the present invention is, for example, a file server, a database server, an application server, a workstation or a personal computer with computing capabilities. Computer devices such as computers include components such as a connection device 12, a storage device 14, and a processor 16. Their functions are described as follows: The connection device 12 is, for example, any wired or wireless interface device, which can be used to connect and access an audio database located at a remote or local end (ie, stored in the storage device 14) to query and receive audio data. For the wired method, the connection device 12 may be a universal serial bus (USB), RS232, a universal asynchronous receiver/transmitter (UART), an internal integrated circuit (I2C), Interfaces such as serial peripheral interface (SPI), display port, or thunderbolt port, but are not limited to these. For wireless mode, the connection device 12 may support wireless fidelity (Wi-Fi), RFID, Bluetooth, infrared, near-field communication (NFC) or device-to-device (device-to-device). -device, D2D) and other communication protocol devices, but not limited to this. In some embodiments, the connection device 12 may also be a network card that supports Ethernet or wireless network standards such as 802.11g, 802.11n, 802.11ac, etc., but is not limited thereto.

The storage device 14 is, for example, any type of fixed or removable random access memory. Access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), flash memory (Flash memory), hard disk or similar components or a combination of the above components, and used for storage can be processed by A computer program executed by the processor 16. In one embodiment, the storage device 14 can also store an audio database constructed by the device 10 or audio data downloaded from a remote audio database, without limitation.

The processor 16 is, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose microprocessor (Microprocessor), microcontroller (Microcontroller), or digital signal processor (Digital Signal). Processor (DSP), programmable controller, Application Specific Integrated Circuits (ASIC), Programmable Logic Device (PLD) or other similar devices or combinations of these devices, but this implementation Examples are not limited to this. In this embodiment, the processor 16 can load a computer program from the storage device 14 to execute the multi-speaker singing voice synthesis method according to the embodiment of the present invention.

FIG2 is a flow chart of a method for designing a personalized virtual singer through voice conversion according to an embodiment of the present invention. Please refer to FIG1 and FIG2 at the same time. The method of this embodiment is applicable to the device 10 of FIG1. The following is a detailed description of the multi-speaker voice synthesis method of this embodiment in conjunction with the various devices and components of the device 10.

In step S202, the device 10 uses the processor 16 to parse the music score file represented by a symbol to extract a plurality of lyrics and a plurality of notes. In one embodiment, the processor 16 can also parse the singing expression information (legato, vibrato), etc. from the music score file, and there is no limit here. The symbolic representation is, for example, MusicXML, but not Limited to this.

In step S202, the processor 16 uses the connection device 12 to load the captured audio data of each note from the audio database. The audio database is, for example, located in a remote server or stored in a local storage device 14, but is not limited thereto.

In step S204, the processor 16 uses a vocoder to perform acoustic modeling on the audio data to adjust each audio data and splice the adjusted audio data to generate singing data. The vocoder is, for example, a WORLD vocoder, but is not limited thereto.

In step S206, the processor 16 uses a generator of a speech conversion model to convert multiple acoustic features in the singing data into output features conditioned on the target audio attributes, and trains the speech conversion model according to various losses of the speech conversion model. , to obtain synthesized singing data with optimized output characteristics. The voice conversion model is, for example, a model based on the Star Generative adversarial network (StarGAN)-Voice conversion (VC) architecture, and the loss includes, for example, adversarial loss (adversarial loss). , classification loss, cycle-consistency loss, identity-mapping loss, one or a combination thereof, but is not limited to this.

Specifically, FIG. 3 is a system architecture diagram of a singing voice synthesizer according to an embodiment of the present invention. Referring to Figure 3, an embodiment of the present invention discloses a system 30 including a singing voice synthesizer 40. The singing voice synthesizer 40 is, for example, a concatenative singing voice synthesizer, which is based on samples in a pre-recorded audio database 33. splicing. Therefore, for each word, a large amount of pronunciation audio data needs to be pre-recorded to build an audio database33.

In one embodiment, the singing synthesizer 40 is implemented in Chinese, so a large amount of audio data needs to be recorded to cover all Chinese characters. During the singing process, since the pitch of the notes will change, the pitch of the individual words is less important. Only 413 pronunciations need to be recorded to cover all Chinese characters.

The input data of the vocal synthesizer 40 is a sheet music 31 represented by musical notation, such as MusicXML, which is an XML-based file format for representing Western musical notation and is supported by more than 250 notation programs, including some graphical user interface programs such as MuseScore. Since the MusicXML used by the vocal synthesizer of this embodiment is an XML-based file format, it can add other tags to represent singing expression techniques, such as legato, vibrato, etc.

In step S1 of the synthesis process, the system 30 first parses the input MusicXML into lyrics and notes 32 (including pitch, note value and/or note duration). In one embodiment, the system 30 also parses the input MusicXML into singing expression information (legato, vibrato), etc., which is not limited here. For each word in the lyrics, in step S2, the system 30 loads the pre-recorded audio data 34 of these words from the audio database 33. Then, in step S3, the vocal synthesizer 40 uses a vocoder 41 (e.g., WORLD vocoder) to perform acoustic modeling on the audio data 34 of the single word, including parsing the audio signal 34 into a harmonic spectrum envelope 43 and a non-periodic envelope 44 according to the fundamental frequency 42, and in step S4, applying the fundamental frequency characteristics to the pitch frequency of each note to perform pitch adjustment 45, and trimming the audio data 34 to match the duration of the note 46. In addition, singing expression effects 47 such as legato, vibrato, fade-in and fade-out can be added between Chinese characters to make the spliced sound smoother. Finally, all the above data and the adjusted audio data of all words will be sent to the speech conversion model 48 for conversion and splicing. Finally, in step S5, the singing synthesizer 40 outputs the synthesized singing data 35 with the selected personal timbre.

In the speech conversion part, the embodiment of the present invention uses a model based on the StarGAN-VC architecture. StarGAN-VC is a non-parallel many-to-many speech conversion model, a variant of the original adversarial network (GAN) (called StarGAN), which can use a single encoder type generator (generator) The network,simultaneously learns many-to-many mappings, and the,properties output by this generator are controlled by auxiliary,inputs. StarGAN-VC also uses adversarial loss to train the generator to make the output of the generator indistinguishable from real speech and ensure that the mapping between each pair of attribute domains can retain language information ( linguistic information). The advantage of StarGAN-VC is that it does not require any information related to the input audio attributes during testing.

In detail, FIG4 is a schematic diagram of a StarGAN-VC model according to an embodiment of the present invention. Referring to FIG4, the goal of the StarGAN-VC model 400 of the present invention is to obtain a generator G that can learn the mapping relationship between multiple domains/speakers, including converting the input acoustic feature x into the output feature x ' under the condition of the target audio attribute c' : G ( x, c' ) → x'

為聲學特徵序列，其中Q為特徵維度、T為特徵序列長度，c和c'分別是來源和目標說話者對應的域碼(domain code)，其中c

{1,...,N}，N為域/說話者的數目。 Where x

is the acoustic feature sequence, where Q is the feature dimension, T is the feature sequence length, c and c' are the domain codes corresponding to the source and target speakers respectively, and c

{1,..., N }, where N is the number of domains/speakers.

The speech conversion model described can solve the optimization problem based on adversarial loss, classification loss, period consistency loss, and identity mapping loss, which are described below: Adversarial loss is used to describe the difference between converted features and real features. Degree, which is defined as follows:

Where D is the discriminator of the target condition. By maximizing this adversarial loss, the discriminator D can learn the optimal decision boundary between the transformed features and the true features conditioned on the target audio attribute c '. Conversely, the generator G can make the transformed features conditioned on the target audio attribute c ' indistinguishable from the true features by minimizing the adversarial loss.

Classification loss enables the speech conversion model to synthesize acoustic features belonging to the target domain. Among them, classifier C can be trained as real acoustic features:

Among them, the classifier C can classify the true acoustic features into the corresponding target domain c1 ' by minimizing the classification loss.

Furthermore, the generator G can optimize the classifier C :

Here, the target domain c1 ' refers to a specific singer. G(x,c1 ' ) is the audio of acoustic feature x generated by the generator G with the voice of c1 ' as the condition. The probability of this audio being classified as c1 ' by the classifier C is C(c1 ' | G(x,c1 ' )). The classifier C can generate acoustic features that can be classified as the target domain c1 ' by minimizing the classification loss.

Although the above-mentioned adversarial loss and classification loss can respectively make the transformed acoustic features real and classifiable, they cannot guarantee that the transformed acoustic features can retain the input components. To compensate for this defect, the following periodic consistency loss can be used:

The above cycle constraints can force the generator G to find the best source and target pairing that will not damage the components.

To further restrict input retention, the following identity mapping loss can be employed:

In summary, the minimization objective of StarGAN-VC, based on the generator G , discriminator D and classifier C , is as follows: L _D = - L _adversarial

0、λ _{cycleconsistency}

0、λ _{identitymapping}

0，其是規則化參數，分別用以加權分類損失、週期一致性損失、 身份映射損失相對於對抗性損失的重要性。 Among them, λ _{classification}

0.λ _{cycle consistency}

0. λ _{identity mapping}

0, which is the regularization parameter, used to weight the importance of classification loss, periodic consistency loss, and identity mapping loss relative to adversarial loss.

In the synthesis process, the sound synthesizer selects the audio data of the words from the pre-established audio database according to the lyrics in the sheet music, and then uses the vocoder to decompose the audio waveform into three main features: fundamental frequency, spectral envelope, and aperiodic envelope. Then, the fundamental frequency features are used to model the melody and the above envelope as acoustic features of the speech conversion model.

Based on using StarGAN-VC as the speech conversion model, the model contains three parts to be trained, namely the above-mentioned generator G , discriminator D and classifier C. Among them, for the generator G, for example, two-dimensional convolutional neural networks (Convolutional Neural Networks, CNN) can be used. In this model, for example, the input acoustic feature sequence is regarded as a two-dimensional image of one channel. For the discriminator D used to identify true/false, for example, the concept of PatchGAN can be used, which is used to solve the image-to-image problem and try to distinguish whether each N × N patch in the image is true or false. Fake. This discriminator D is, for example, convolutionally performed on the entire image, and all responses are averaged to provide the final output of the discriminator D. In a speech conversion model, the discriminator D , for example, classifies patches or segments of audio rather than the entire audio. Although this increases the difficulty of classifier C , it can effectively improve the performance of the discriminator in speech conversion. Finally, for the domain classifier C , for example, use a gated convolutional neural network (Gated CNN), which uses a gating mechanism and can achieve the same goal as long short term memory (LSTM) at a faster speed. Similar results online. The above network can be trained for 200k steps using the Adam optimizer, where the batch size is 8, the learning rates of the generator G and the discriminator D can be set to 0.0001 respectively, and the momentum term Can be set to 0.5. In addition, the above-mentioned λ _{classification} , λ _{cycleconsistency} , and λ _{identitymapping} may all be set to 10, for example, but are not limited thereto. Those skilled in the art may modify it to other values according to actual needs.

In summary, the method and device for designing a personalized virtual singer through singing voice conversion according to embodiments of the present invention use a unit selection splicing method to pre-record multiple pronunciations of text and design them into virtual singers, which can Sing any song with sheet music and lyrics accurately. In addition, the Star-GAN algorithm is used to train the timbre conversion of multiple speakers together with the timbres of virtual singers, so that each other's timbres can be interchanged. Thereby, for a selected speaker, the singing voice synthesis device of the embodiment of the present invention can combine the speaker's timbre with the singing ability of the virtual singer, and sing any song with the speaker's timbre.

Although the present invention has been disclosed above through embodiments, they are not intended to limit the present invention. Anyone with ordinary knowledge in the technical field may make some modifications and modifications without departing from the spirit and scope of the present invention. Therefore, The protection scope of the present invention shall be determined by the appended patent application scope.

S202~S208: steps

Claims (14)

A method for designing a personalized virtual singer through singing voice conversion, suitable for electronic devices equipped with processors. The method includes the following steps: parsing (parse) a music score file represented by a symbol to extract multiple lyrics and multiple notes ; Load the captured audio data of each note from an audio database; use a vocoder to decompose the waveform of the audio data into basic frequency, spectral envelope and aperiodic envelope, according to the The basic frequency models the spectral envelope and the aperiodic envelope as multiple acoustic features of the speech conversion model, and uses the basic frequency to adjust the pitch of the audio data corresponding to each note. and splicing the adjusted audio data to generate singing voice data; and using a generator of a speech conversion model to convert multiple acoustic features in the singing voice data into output features conditioned on the target audio attributes of the virtual singer, and based on Multiple losses of the speech conversion model are used to train the speech conversion model, and the loss is minimized or maximized according to the type of each of the calculated losses, so as to obtain synthetic singing data with optimized output features, wherein the The above losses include one or a combination of adversarial loss, classification loss, period consistency loss, identity mapping loss. The method described in claim 1 further includes: recording multiple audio data of each of multiple words in a language and recording them in the audio database. The method of claim 1, wherein the step of training the speech conversion model according to multiple losses of the speech conversion model includes: Calculating the adversarial loss used to describe the degree of difference between the converted output features and the acoustic features; and by maximizing the adversarial loss, so that the discriminator of the speech conversion model learns to The target audio attribute is a conditional optimal decision boundary between the converted output feature and the acoustic feature. The method according to claim 1, wherein the step of training the speech conversion model according to multiple losses of the speech conversion model includes: calculating a parameter used to describe the acoustic features synthesized by the speech conversion model belonging to a target domain. The classification loss; and training a classifier of the speech conversion model by minimizing the classification loss so that the acoustic features are classified into the corresponding target domain. The method of claim 1, wherein the step of training the speech conversion model according to multiple losses of the speech conversion model includes: calculating the linguistic consistency ( and by minimizing the periodic consistency loss to ensure that the output features are linguistically consistent with the acoustic features. The method of claim 1, wherein the step of training the speech conversion model according to the multiple losses of the speech conversion model comprises: calculating the identity mapping loss used to describe the components of the input features preserved in the converted output features; and preserving the components of the input features in the output features by minimizing the identity mapping loss. The method of claim 1, wherein the step of training the speech conversion model according to the multiple losses of the speech conversion model comprises: respectively multiplying the calculated classification loss, the cycle consistency loss and the identity mapping loss by corresponding weights to weight the importance of the classification loss, the cycle consistency loss and the identity mapping loss relative to the adversarial loss. A device for designing a personalized virtual singer through singing voice conversion, including: a connection device; a storage device to store a computer program; and a processor, coupled to the connection device and the storage device, configured to load and execute the storage The computer program in the device is used to: parse a music score file represented by a symbol to extract multiple lyrics and multiple notes; use the connecting device to connect to an audio database to load each extracted description Audio data of musical notes; use a vocoder to decompose the waveform of the audio data into a basic frequency, a spectral envelope and a non-periodic envelope, and model the spectral envelope and the non-periodic envelope according to the basic frequency as the using the plurality of acoustic features of the speech conversion model, applying the basic frequency to adjust the pitch of the audio data corresponding to each note and splicing the adjusted audio data to generate singing data; and using a voice The generator of the transformation model converts the multiple acoustic The features are converted into output features conditioned on the target audio attributes of the virtual singer, and the speech conversion model is trained according to multiple losses of the speech conversion model, so that the loss is minimized or maximized according to the type of each of the calculated losses. ization to obtain synthetic singing material with optimized output features, wherein the loss includes one of adversarial loss, classification loss, period consistency loss, identity mapping loss, or a combination thereof. As described in claim 8, the device for designing a personalized virtual singer through voice conversion, wherein the processor further records multiple audio data of each of multiple words in a language and records them in the audio database. The device for designing a personalized virtual singer through singing voice conversion as described in claim 8, wherein the processor includes calculating the adversarial loss used to describe the degree of distinction between the output features after conversion and the acoustic features, and maximizing the adversarial loss so that the discriminator of the speech conversion model learns the optimal decision boundary between the output features after conversion and the acoustic features conditioned on the target audio attributes. The device for designing a personalized virtual singer through singing voice conversion as claimed in claim 8, wherein the processor includes calculating the classification loss used to describe the acoustic features synthesized by the speech conversion model belonging to a target domain, and training a classifier of the speech conversion model by minimizing the classification loss so that the acoustic features are classified into the corresponding target domain. The device for designing a personalized virtual singer through singing voice conversion as described in claim 8, wherein the processor includes calculating the cycle consistency loss used to describe the language consistency of the output features and the acoustic features after conversion, and ensuring the language consistency of the output features and the acoustic features by minimizing the cycle consistency loss. The device for designing a personalized virtual singer through singing voice conversion as described in claim 8, wherein the processor includes calculating the identity mapping loss used to describe the component of the converted output feature that retains the input feature, and by The identity mapping loss is minimized to preserve components of the input features in the output features. The device for designing a personalized virtual singer through singing voice conversion as described in claim 8, wherein the processor includes multiplying the calculated classification loss, the cycle consistency loss and the identity mapping loss by corresponding weights respectively , to weight the importance of the classification loss, the period consistency loss and the identity mapping loss relative to the adversarial loss.

Images