JP2022065566A - Method for synthesizing voice and program - Google Patents

Abstract

To provide a method for synthesizing voice that can easily add acoustic data with the same sound quality as that of acoustic data with a specific sound quality to the acoustic data with the specific sound quality.SOLUTION: The method for synthesizing voice includes: preparing a musical score encoder 111 for generating an intermediate feature amount MF1 from a musical score feature amount of musical score data D1, an acoustic encoder 121 for generating an intermediate feature amount MF2 from an acoustic feature amount of acoustic data D2, and an acoustic decoder 133 for generating an acoustic feature amount AFS based on the MF1 and the MF2; receiving acoustic data D2_T for auxiliary learning and training the acoustic decoder 133 to generate an acoustic feature amount AFS close to the acoustic feature amount of D2_T by using the MF2 generated from the acoustic feature amount of the D2_T by the acoustic encoder and by using the acoustic feature amount of the D2_T; and generating an acoustic feature amount AFS by processing the MF1 generated from the musical score data D1 arranged on the time axis of the D2_T by using the musical score encoder, by the trained acoustic decoder 133.SELECTED DRAWING: Figure 2

Description

The present invention relates to speech synthesis methods and programs. The "voice" in the present specification means a general "sound" and is not limited to a "human voice".

A voice synthesizer that synthesizes the singing voice of a specific singer or the playing sound of a specific musical instrument is known. A speech synthesizer using machine learning learns acoustic data with musical score data of a specific singer or musical instrument as teacher data. A voice synthesizer that has learned the acoustic data of a specific singer or musical instrument synthesizes and outputs the singing voice of a specific singer or the playing sound of a specific musical instrument by being given musical score data by the user. The following Patent Document 1 discloses a technique for synthesizing a singing voice using machine learning. Further, a technique for converting the voice quality of a singing voice by using a technique for synthesizing a singing voice is known.

Japanese Unexamined Patent Publication No. 2019-101094

You may want to add a few singing voices or lines with the same sound quality as that singer to a track on which a singer's singing voice is recorded, or you may want to modify the singing voice of that track a little. In addition, there are cases where it is desired to add a little performance sound having the same sound quality as the instrument sound to a track on which the performance sound of the instrument is recorded, or to modify the performance sound of the track a little. In those cases, it was necessary to re-record the singer's singing and the instrument's playing sound for that part of the track.

A voice synthesizer using machine learning can learn and synthesize the singing voice of a desired singer and the playing sound of a musical instrument. However, for the learning, it is necessary to perform labeling work in addition to the acoustic data of the singing of a specific singer or the performance sound of a musical instrument, and prepare the musical score data corresponding to the acoustic data.

An object of the present invention is to easily add acoustic data having the same sound quality to recorded acoustic data having a specific sound quality, or to partially modify the acoustic data while maintaining the sound quality. It is to provide a synthesis method.

The voice synthesis method according to one aspect of the present invention is a voice synthesis method realized by a computer, the score encoder that generates the first intermediate feature from the score feature of the score data, and the second from the acoustic feature of the acoustic data. Prepare an acoustic encoder that generates intermediate features and an acoustic decoder that generates acoustic features based on the first intermediate features or the second intermediate features, receive auxiliary learning acoustic data, and use the acoustic encoder. Using the second intermediate feature amount generated from the acoustic feature amount of the auxiliary learning acoustic data and the acoustic feature amount of the auxiliary learning acoustic data, an acoustic feature amount close to the acoustic feature amount of the auxiliary learning acoustic data is generated. First, the acoustic decoder is assisted and trained, the score data arranged on the time axis of the auxiliary learning acoustic data is received via the user interface, and the score data is generated from the score data arranged by using the score encoder. An acoustic feature is generated by processing the intermediate feature with an auxiliary trained acoustic decoder.

A voice synthesis program according to another aspect of the present invention is a program that causes a computer to execute a voice synthesis method, and based on the program, the computer generates a first intermediate feature amount from the score feature amount of the score data. An acoustic encoder that generates a second intermediate feature amount from the score feature amount of acoustic data, and an acoustic decoder that generates an acoustic feature amount based on the first intermediate feature amount or the second intermediate feature amount are prepared, and the sound for auxiliary learning is prepared. The sound of the auxiliary learning acoustic data is received using the second intermediate feature amount generated from the acoustic feature amount of the auxiliary learning acoustic data using the acoustic encoder and the acoustic feature amount of the auxiliary learning acoustic data. The acoustic decoder is assisted and trained to generate acoustic features close to the features, and the score data arranged on the time axis of the auxiliary learning acoustic data is received via the user interface and arranged using the score encoder. The first intermediate feature amount generated from the score data is processed by the auxiliary trained acoustic decoder to generate the acoustic feature amount.

INDUSTRIAL APPLICABILITY The present invention is a voice synthesis method capable of easily adding acoustic data of the same sound quality to recorded acoustic data of a specific sound quality and partially modifying the acoustic data while maintaining the sound quality. I will provide a.

It is a block diagram of the speech synthesizer which concerns on embodiment. It is a functional block diagram of the speech synthesizer which concerns on embodiment. It is a figure which shows the data used by a speech synthesizer. It is a flowchart which shows the basic training method which concerns on embodiment. It is a flowchart which shows the voice synthesis method which concerns on embodiment. It is a figure which shows the user interface of a speech synthesizer. It is a figure which shows the user interface of a speech synthesizer. It is a flowchart which shows the acoustic decoder training method which concerns on embodiment. It is a figure which shows the user interface of a speech synthesizer. It is a figure which shows the user interface of a speech synthesizer. It is a figure which shows the user interface of a speech synthesizer.

(1) Configuration of Speech Synthesizer Hereinafter, the speech synthesizer according to the embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a configuration diagram showing a speech synthesizer 1 according to an embodiment. As shown in FIG. 1, the voice synthesizer 1 includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, an operation unit 14, a display unit 15, a storage device 16, and a sound. It includes a system 17, a device interface 18, and a communication interface 19. As the voice synthesizer 1, for example, a personal computer, a tablet terminal, a smartphone, or the like is used.

The CPU 11 is composed of one or a plurality of processors, and controls the entire speech synthesizer 1. The RAM 12 is used as a work area when the CPU 11 executes a program. The ROM 13 stores a control program and the like. The operation unit 14 inputs a user's operation on the voice synthesizer 1. The operation unit 14 is, for example, a mouse or a keyboard. The display unit 15 displays the user interface of the speech synthesizer 1. The operation unit 14 and the display unit 15 may be configured as a touch panel type display. The sound system 17 includes a sound source, a function of D / A conversion and amplification of an audio signal, a speaker that outputs an analog-converted audio signal, and the like. The device interface 18 is an interface for the CPU 11 to access a storage medium RM such as a CD-ROM or a semiconductor memory. The communication interface 19 is an interface for the CPU 11 to connect to a network such as the Internet.

The storage device 16 stores a speech synthesis program P1, a training program P2, musical score data D1, and acoustic data D2. The voice synthesis program P1 is a program for generating voice-synthesized acoustic data or sound quality-converted acoustic data. The training program P2 is a program for training an encoder and an acoustic decoder used for speech synthesis or sound quality conversion.

The musical score data D1 is data that defines a musical piece. The musical score data D1 includes information on the pitch and intensity of each note, information on the tone within each note (only in the case of singing), information on the pronunciation period of each note, information on performance symbols, and the like. The acoustic data D2 is voice waveform data. The acoustic data D2 is, for example, singing waveform data, musical instrument sound waveform data, or the like. In the voice synthesizer 1, the content of one song is generated by using the score data D1 and the acoustic data D2.

(2) Functional Configuration of Speech Synthesizer FIG. 2 is a functional block diagram of the speech synthesizer 1. As shown in FIG. 2, the speech synthesizer 1 includes a control unit 100. The control unit 100 includes a conversion unit 110, a score encoder 111, a pitch model 112, an analysis unit 120, an acoustic encoder 121, a switching unit 131, a switching unit 132, an acoustic decoder 133, and a vocoder 134. In FIG. 2, the control unit 100 is a functional unit realized by executing the speech synthesis program P1 by the CPU 11 while using the RAM 12 as a work area. That is, in the conversion unit 110, the score encoder 111, the pitch model 112, the analysis unit 120, the acoustic encoder 121, the switching unit 131, the switching unit 132, the acoustic decoder 133, and the vocoder 134, the voice synthesis program P1 is executed by the CPU 11. It is a functional part to be realized. Further, the score encoder 111, the acoustic encoder 121, and the acoustic decoder 133 learn their functions by being executed by the CPU 11 while the training program P2 uses the RAM 12 as a work area.

The conversion unit 110 reads the score data D1 and generates various score feature data SFs from the score data D1. The conversion unit 110 outputs the score feature data SF to the score encoder 111 and the pitch model 112. The musical score feature data SF acquired from the conversion unit 110 by the musical score encoder 111 is a factor that controls the sound quality at each time point, and is, for example, a context such as pitch, intensity, and phoneme label. The musical score feature data SF acquired by the pitch model 112 from the conversion unit 110 is a factor that controls the pitch at each time point, and is, for example, the context of the note specified by the pitch and the pronunciation period. The context includes, in addition to the data at each point in time, at least one of the data before and after.

The score encoder 111 generates intermediate feature data MF1 from the score feature data SF. The trained score encoder 111 is a statistical model that generates intermediate feature data MF1 from the score feature data SF, and is defined by a plurality of variables 111_P stored in the storage device 16. In the present embodiment, the score encoder 111 uses a generation model that outputs intermediate feature data MF1 according to the score feature data SF. As a generative model constituting the score encoder 111, for example, a convolutional neural network (CNN), a recurrent neural network (RNN), a combination thereof, or the like is used. It may be an autoregressive model or a model with attention.

The pitch model 112 reads the score feature data SF and generates the fundamental frequency F0 of the sound in the music from the score feature data SF. The pitch model 112 outputs the acquired fundamental frequency F0 to the switching unit 132. The trained pitch model 112 is a statistical model that generates the fundamental frequency F0 of the sound in the music from the musical score feature data SF, and is defined by a plurality of variables 112_P stored in the storage device 16. As the pitch model 112, in the present embodiment, a generation model that outputs a fundamental frequency F0 corresponding to the musical score feature data SF is used. As the generative model constituting the pitch model 112, for example, CNN, RNN, a combination thereof, or the like is used. It may be an autoregressive model or a model with attention. Conversely, a simpler hidden Markov or random forest model may be used.

The analysis unit 120 reads the acoustic data D2 and performs frequency analysis on the acoustic data D2. The analysis unit 120 generates the fundamental frequency F0 and the acoustic feature data AF of the sound indicated by the acoustic data D2 by performing frequency analysis on the acoustic data D2. The acoustic feature data AF shows the frequency spectrum of the sound indicated by the acoustic data D2, and is, for example, a Mel-Scale Log-Spectrum (MSLS). The analysis unit 120 outputs the fundamental frequency F0 to the switching unit 132. The analysis unit 120 outputs the acoustic feature data AF to the acoustic encoder 121.

The acoustic encoder 121 generates intermediate feature data MF2 from the acoustic feature data AF. The trained acoustic encoder 121 is a statistical model that generates intermediate feature data MF2 from acoustic feature data AF, and is defined by a plurality of variables 121_P stored in the storage device 16. In the present embodiment, the acoustic encoder 121 uses a generation model that outputs intermediate feature data MF2 corresponding to the acoustic feature data AF. As the generative model constituting the acoustic encoder 121, for example, CNN, RNN, a combination thereof, or the like is used.

The switching unit 131 receives the intermediate feature data MF1 from the score encoder 111. The switching unit 131 receives the intermediate feature data MF2 from the acoustic encoder 121. The switching unit 131 selectively outputs either the intermediate feature data MF1 from the score encoder 111 or the intermediate feature data MF2 from the acoustic encoder 121 to the acoustic decoder 133.

The switching unit 132 receives the fundamental frequency F0 from the pitch model 112. The switching unit 132 receives the fundamental frequency F0 from the analysis unit 120. The switching unit 132 selectively outputs either the fundamental frequency F0 from the pitch model 112 or the fundamental frequency F0 from the analysis unit 120 to the acoustic decoder 133.

The acoustic decoder 133 generates acoustic feature data AFS based on the intermediate feature data MF1 or the intermediate feature data MF2. The acoustic feature data AFS is a frequency amplitude spectrum, for example, a mel frequency logarithmic spectrum. The acoustic decoder 133 is a statistical model that generates acoustic feature data AFS, and is defined by a plurality of variables 133_P stored in the storage device 16. In the present embodiment, the acoustic decoder 133 uses a model that outputs acoustic feature data AFS corresponding to the intermediate feature data MF1 or the intermediate feature data MF2. As a model constituting the acoustic decoder 133, for example, CNN, RNN, a combination thereof, or the like is used. It may be an autoregressive model or a model with attention.

The vocoder 134 generates synthetic acoustic data D3 based on the acoustic feature data AFS input from the acoustic encoder 121. If the acoustic feature data AFS is a mel frequency logarithmic spectrum, the vocoder 134 converts the mel frequency logarithmic spectrum input from the acoustic encoder 121 into an acoustic signal in the time domain to generate synthetic acoustic data D3.

(3) Information used by the speech synthesizer FIG. 3 shows data used by the speech synthesizer 1. The voice synthesizer 1 uses the score data D1 and the acoustic data D2 as the data related to the voice synthesis. As described above, the musical score data D1 is data that defines a musical piece. The musical score data D1 includes information on the pitch of each note, information on the melody in each note (only in the case of singing), information on the pronunciation period of each note, information on performance symbols, and the like. As described above, the acoustic data D2 is audio waveform data. The acoustic data D2 is, for example, singing waveform data, musical instrument sound waveform data, or the like. The waveform data of each singing is given a sound source ID indicating the singer who sang the song, and the waveform data of each musical instrument sound is given a sound source ID indicating the musical instrument. The sound source ID indicates the source of the sound indicated by the waveform data.

The score data D1 used by the voice synthesizer 1 includes the score data D1_R for basic learning and the score data D1_S for synthesis. The acoustic data D2 used by the speech synthesizer 1 includes basic learning acoustic data D2_R, synthetic acoustic data D2_S, and auxiliary learning acoustic data D2_T corresponding thereto. The basic learning musical score data D1_R corresponding to the basic learning acoustic data D2_R indicates a musical score (note string or the like) corresponding to the performance in the basic learning acoustic data D2_R. The synthetic musical score data D1_S corresponding to the synthetic acoustic data D2_S indicates a musical score (note string or the like) corresponding to the performance in the synthetic acoustic data D2_S. The storage device 16 of FIGS. 1 and 2 shows the score data D1 and the acoustic data D2, but in reality, the score data D1 stores the basic learning score data D1_R and the composite score data D1_S. As the acoustic data D2, the basic learning acoustic data D2_R, the synthetic acoustic data D2_S, and the auxiliary learning acoustic data D2_T are stored.

The musical score data D1_R for basic learning is data used for training the musical score encoder 111, the acoustic encoder 121, and the acoustic decoder 133. The basic learning acoustic data D2_R is data used for training the musical score encoder 111, the acoustic encoder 121, and the acoustic decoder 133. By learning the score encoder 111, the acoustic encoder 121, and the acoustic decoder 133 using the basic learning score data D1_R and the basic learning acoustic data D2_R, the speech synthesizer 1 produces a voice with a sound quality specified by the sound source ID. It is set to a state where it can be synthesized.

The score data D1_S for synthesis is data given to the voice synthesizer 1 in a state in which voice of a specific sound quality can be synthesized. The voice synthesizer 1 generates synthetic acoustic data D3 of voice having a sound quality specified by a sound source ID based on the musical score data D1_S for synthesis. For example, in the case of singing synthesis, the voice synthesizer 1 can synthesize and output the singing voice of the singer specified by the sound source ID by being given lyrics (sounds) and melody (note strings). In the case of musical instrument sound synthesis, by giving a melody (note string), the performance sound of the musical instrument specified by the sound source ID can be synthesized and output.

The synthetic acoustic data D2_S is data given to the voice synthesizer 1 in a state in which voice of a specific sound quality can be synthesized. The voice synthesizer 1 generates the synthetic sound data D3 of the sound quality specified by the sound source ID based on the sound data D2_S for synthesis. For example, the voice synthesizer 1 synthesizes and outputs the singing voice of a singer or the performance sound of a musical instrument specified by a different sound source ID by being given the acoustic data D2_S for synthesizing the singer or musical instrument sound of an arbitrary sound source ID. .. By utilizing this function, the speech synthesizer 1 functions as a kind of sound quality converter.

The auxiliary learning acoustic data D2_T is data used for training the acoustic decoder 133. The auxiliary learning acoustic data D2_T is learning data for changing the sound quality synthesized by the acoustic decoder 133. By learning by the acoustic decoder 133 using the auxiliary learning acoustic data D2_T, the speech synthesizer 1 is set to a state in which the singing voice of another new singer can be synthesized.

(4) Basic training method Next, the basic training method of the speech synthesizer 1 according to the present embodiment will be described. FIG. 4 is a flowchart showing a basic training method of the speech synthesizer 1 according to the present embodiment. In the basic training, the score encoder 111, the acoustic encoder 121, and the acoustic decoder 133 included in the speech synthesizer 1 are trained. The basic training method shown in FIG. 4 is realized by executing the training program P2 by the CPU 11 for each processing step of machine learning. In one processing step, acoustic data corresponding to a plurality of frames of frequency analysis is processed.

Before executing the basic training method of FIG. 4, a plurality of sets of basic learning musical score data D1_R and corresponding basic learning acoustic data D2_R are prepared as teacher data for each sound source ID and stored in the storage device 16. The basic learning score data D1_R and the basic learning acoustic data D2_R prepared as teacher data are data prepared for basic training of a musical piece having a sound quality specified by each sound source ID. Here, a case where the score data D1_R for basic learning and the acoustic data D2_R for basic learning are data prepared for basic training of the singing voices of a plurality of singers specified by a plurality of sound source IDs will be described as an example.

In step S101, the CPU 11 as the conversion unit 110 generates the score feature data SF based on the score data D1_R for basic learning. In the present embodiment, for example, data indicating a phoneme label is used as the musical score feature data SF indicating the characteristics of the musical score for generating the acoustic features. Next, in step S102, the CPU 11 as the analysis unit 120 generates acoustic feature data AF showing a frequency spectrum based on the basic learning acoustic data D2_R whose sound quality is specified by the sound source ID. In this embodiment, for example, a mel frequency logarithmic spectrum is used as the acoustic feature data AF. The process of step S102 may be executed before the process of step S101.

Next, in step S103, the CPU 11 processes the score feature data SF using the score encoder 111 to generate the intermediate feature data MF1. Next, in step S104, the CPU 11 processes the acoustic feature data AF using the acoustic encoder 121 to generate the intermediate feature data MF2. The process of step S104 may be executed before the process of step S103.

Next, in step S105, the CPU 11 processes the sound source ID of the fundamental learning acoustic data D2_R, the fundamental frequency F0, and the intermediate feature data MF1 using the acoustic decoder 133 to generate the acoustic feature data AFS1. The sound source ID, the fundamental frequency F0, and the intermediate feature data MF2 are processed to generate the acoustic feature data AFS2. In the present embodiment, for example, a mel frequency logarithmic spectrum is used as the acoustic feature data AFS showing the frequency spectrum. The acoustic decoder 133 inputs the fundamental frequency F0 from the switching unit 132 when executing the acoustic decoding. The fundamental frequency F0 is generated by the pitch model 112 when the input data is the basic learning musical score data D1_R, and is generated by the analysis unit 120 when the input data is the basic learning acoustic data D2_R. Further, the acoustic decoder 133 inputs a sound source ID as an identification for identifying the singer when executing the acoustic decoding. These fundamental frequency F0 and sound source ID are input values to the generation model constituting the acoustic decoder 133 together with the intermediate feature data MF1 and MF2.

Next, in step S106, the score encoder 111, acoustically, the CPU 11 so that the intermediate feature data MF1 and the intermediate feature data MF2 approach each other and the acoustic feature data AFS approaches the acoustic feature data AF which is the correct answer. Train the encoder 121 and the acoustic decoder 133. That is, the intermediate feature data MF1 is generated from the score feature data SF (for example, indicating a phonetic label), and the intermediate feature data MF2 is generated from a frequency spectrum (for example, a mel frequency logarithmic spectrum). The generative model of the score encoder 111 and the generative model of the acoustic encoder 121 are trained so that the distances of MF1 and MF2 are close to each other.

Specifically, the back provacation of the difference is executed so as to reduce the difference between the intermediate feature data MF1 and the intermediate feature data MF2, and the variable 111_P of the score encoder 111 and the variable 121_P of the acoustic encoder 121 are updated. .. As the difference between the intermediate feature data MF1 and the intermediate feature data MF2, for example, the Euclidean distance of the vector representing these two data is used. In parallel, an error back-progress is executed so that the acoustic feature data AFS generated from the acoustic decoder 133 approaches the acoustic feature data AF generated from the basic learning acoustic data D2_R, which is the teacher data, and the score encoder. The variable 111_P of 111, the variable 121_P of the acoustic encoder 121, and the variable 133_P of the acoustic decoder 133 are updated.

By repeatedly executing the learning process of one processing step (steps S101 to S106) for the plurality of teacher data, the basic learning score data D1_R and the basic learning acoustic data D2_R, the score encoder 111, the acoustic encoder 121, and the sound encoder 121 The acoustic decoder 133 is trained to be able to synthesize acoustic data (corresponding to the singing voice of a singer and the playing sound of an instrument), which has a specific sound quality specified by each sound source ID and whose sound quality changes according to the amount of musical score features. Will be done. Specifically, the trained speech synthesizer 1 can synthesize a trained voice (singing voice or musical instrument sound) having a specific sound quality trained by using the score encoder 111 and the acoustic decoder 133 based on the score data D1. be. Further, the trained voice synthesizer 1 can synthesize a trained voice (singing voice or musical instrument sound) having a specific sound quality by using the sound encoder 121 and the sound decoder 133 based on the sound data D2.

As described above, in the training of the acoustic decoder 133, the sound source ID assigned to each basic learning acoustic data D2_R is used as an input value. Therefore, the acoustic decoder 133 can learn the singing voices of a plurality of singers and the playing sounds of a plurality of musical instruments by using the basic learning acoustic data D2_R of a plurality of sound source IDs.

(5) Speech Synthesis Method Next, a method of synthesizing the sound quality of the designated sound source ID by the speech synthesizer 1 according to the present embodiment will be described. FIG. 5 is a flowchart showing a speech synthesis method by the speech synthesizer 1 according to the present embodiment. The speech synthesis method shown in FIG. 5 is realized by executing the speech synthesis program P1 by the CPU 11 every time corresponding to a frame of frequency analysis. For the sake of simplification of the description, it is assumed here that the generation of the fundamental frequency F0 from the musical score data D1_S for synthesis and the generation of the fundamental frequency F0 from the acoustic data D2_S for synthesis have been completed in advance. The generation of the fundamental frequency F0 may be executed in parallel with the process of FIG.

In step S201, the CPU 11 as the conversion unit 110 acquires the score data D1_S for composition arranged before and after the time of the frame on the time axis of the user interface. Alternatively, the analysis unit 120 acquires the synthetic acoustic data D2_S arranged before and after the time of the frame on the time axis of the user interface. FIG. 6 is a diagram showing a user interface 200 displayed on the display unit 15 by the voice synthesis program P1. In this embodiment, as the user interface 200, for example, a piano roll having a time axis and a pitch axis is used. As shown in FIG. 6, the user operates the operation unit 14 to perform the composition score data D1_S (note or text) and the composition sound data D2_S at the positions corresponding to the desired time and pitch on the piano roll. Place (waveform data). In the periods T1, T2 and T4 in the figure, the score data D1_S for composition is arranged on the piano roll by the user. In the period T1, the user arranges only the text (narrative in the song) without pitch (TTS function). In the periods T2 and T4, the user arranges a time series of notes (pitch and pronunciation period) and lyrics sung by each note (song voice synthesis function). In the figure, block 201 represents the pitch and duration of the note. Further, below the block 201, the lyrics (phonology) sung by the note are displayed. Further, in the periods T3 and T5, the user arranges the synthetic acoustic data D2_S at a desired time position of the piano roll (sound quality conversion function). In the figure, the waveform 202 is a waveform indicated by the synthetic acoustic data D2_S (waveform data), and the position in the pitch axis direction is arbitrary. Alternatively, the waveform 202 may be automatically arranged at a position corresponding to the fundamental frequency F0 of the synthetic acoustic data D2_S. Also, in the figure, lyrics are arranged in addition to the notes for singing synthesis, but in instrumental sound synthesis, it is not necessary to arrange lyrics and text.

Next, in step S202, the CPU 11 which is the control unit 100 determines whether or not the data acquired at the current time is the musical score data D1_S for synthesis. If the acquired data is the score data D1_S (musical notes) for composition, the process proceeds to step S203. In step S203, the CPU 11 generates the score feature data SF from the score data D1_S for synthesis, and processes the score feature data SF using the score encoder 111 to generate the intermediate feature data MF1. The musical score feature data SF shows the characteristics of the phonology in the case of singing synthesis, and the sound quality of the generated singing is controlled according to the phonology. Further, in the case of musical instrument sound synthesis, the musical score feature data SF indicates the pitch and intensity of the note, and the sound quality of the generated musical instrument sound is controlled according to the pitch and intensity.

Next, in step S204, the CPU 11 as the control unit 100 determines whether or not the data acquired at the current time is the synthetic acoustic data D2_S. If the acquired data is the synthetic acoustic data D2_S (waveform data), the process proceeds to step S205. In step S205, the CPU 11 generates an acoustic feature amount AF (frequency spectrum) from the synthetic acoustic data D2_S, processes the acoustic feature amount AF using the acoustic encoder 121, and generates intermediate feature data MF2.

After executing step S203 or step S205, the process proceeds to step S206. In step S206, the CPU 11 uses the acoustic decoder 133 to obtain the sound source ID specified at that time, the fundamental frequency F0 at that time, and the intermediate feature data MF1 or the intermediate feature data MF2 generated at that time. It is processed to generate acoustic feature data AFS. Since the two intermediate feature data generated by the basic training are trained to approach each other, the intermediate feature data MF2 generated from the acoustic feature data AF is similar to the intermediate feature data MF1 generated from the musical score feature data. Reflects the characteristics of the corresponding score. In the present embodiment, the acoustic decoder 133 combines the sequentially generated intermediate feature data MF1 and intermediate feature data MF2 on the time axis, executes decoding processing, and generates acoustic feature data AFS.

Next, in step S207, the CPU 11 as the vocoder 134 basically has the sound quality indicated by the sound source ID based on the acoustic feature data AFS indicating the frequency spectrum, and the sound quality further changes according to the tone and pitch. Synthetic acoustic data D3, which is waveform data, is generated. Since the acoustic feature data AFS is generated after the intermediate feature data MF1 and the intermediate feature data MF2 are combined on the time axis, the content of the synthetic acoustic data D3 in which the connection in the song is natural is generated. FIG. 7 is a diagram showing a user interface 200 that displays a speech synthesis processing result. In FIG. 7, the generated fundamental frequency (F0) 211 is displayed throughout the periods T1 to T5. In the period T1, the waveform 212 of the synthetic acoustic data D3 is displayed superimposed on the fundamental frequency. In the periods T3 and T5, the waveform 213 of the synthetic acoustic data D3 is displayed superimposed on the fundamental frequency.

(6) Acoustic Decoder Training Method FIG. 8 is a flowchart showing an auxiliary training method for the speech synthesizer 1 according to the present embodiment. In the auxiliary training, the acoustic decoder 133 included in the speech synthesizer 1 is trained. The auxiliary training method shown in FIG. 8 is realized by executing the training program P2. Before executing the auxiliary training method of FIG. 8, auxiliary learning acoustic data D2_T having a new sound quality specified by a predetermined sound source ID is prepared as teacher data and stored in the storage device 16. The auxiliary learning acoustic data D2_T prepared as teacher data is data prepared for changing the sound quality of the basic trained acoustic decoder 133. The auxiliary learning acoustic data D2_T is usually acoustic data different from the basic learning acoustic data D2_R used for basic training, but the sound source ID of the acoustic data is the same as the basic learning acoustic data D2_R, that is, the same singer. It may be acoustic data of the same instrument. That is, the sound decoder 133 can be made to learn the sound quality of a new singer or musical instrument, or the sound quality of a singer or musical instrument that has already been learned can be improved.

First, in step S301, the CPU 11 which is the analysis unit 120 generates the fundamental frequency F0 and the acoustic feature data AF based on the auxiliary learning acoustic data D2_T. In the present embodiment, for example, a mel frequency logarithmic spectrum is used as the acoustic feature data AF showing the frequency spectrum of the auxiliary learning acoustic data D2_T. In this acoustic decoder training, another sound quality (for example, the singing voice of a new singer) is trained by the generative model using only the auxiliary learning acoustic data D2_T. Therefore, the score data D1 is unnecessary in the acoustic decoder training. That is, the CPU 11 trains the acoustic decoder 133 using the auxiliary learning acoustic data D2_T without a phoneme label.

Next, in step S302, the CPU 11 processes the acoustic feature data AF using the acoustic encoder 121 to generate the intermediate feature data MF2. Subsequently, in step S303, the CPU 11 processes the sound source ID of the auxiliary learning acoustic data D2_T, the fundamental frequency F0, and the intermediate feature data MF2 using the acoustic decoder 133 to generate the acoustic feature data AFS. Subsequently, in step S304, the CPU 11 trains the acoustic decoder 133 so that the acoustic feature data AFS approaches the acoustic feature data AF generated from the auxiliary learning acoustic data D2_T. That is, the score encoder 111 and the acoustic encoder 121 are not trained, but only the acoustic decoder 133 is trained. As described above, according to the auxiliary training method of the present embodiment, since the auxiliary learning acoustic data D2_T without a phoneme label can be used for training, the acoustic decoder 133 can be trained without the trouble and cost of preparing teacher data. ..

FIG. 9 is a diagram showing a user interface 200 according to a training method for an acoustic decoder. In response to the user's recording instruction, the CPU 11 newly records, for example, the singing voice of a singer for one song or the playing sound of a musical instrument, and assigns a sound source ID. If the sound source has been learned, the same sound source ID is assigned, and if it has not been learned, a new sound source ID is assigned. The recorded waveform data for one track is the acoustic data D2_T for auxiliary learning. This recording may be performed while playing the accompaniment track. In FIG. 9, the waveform 221 is the waveform indicated by the auxiliary learning acoustic data D2_T. After the auxiliary training of the acoustic decoder, the voice sung by the user or the sound of the musical instrument played may be directly captured through the microphone connected to the voice synthesizer 1 and the sound quality conversion processing may be performed in real time. When the CPU 11 performs the auxiliary training process of FIG. 8 using the auxiliary learning acoustic data D2_T, the acoustic decoder 133 learns the properties of a new singing voice or musical instrument sound for, for example, one song, and the singing voice of the voice quality or the like. Musical instrument sounds can be synthesized. FIG. 9 further shows how the CPU 11 arranges three notes (composite score data D1_S) in the period T12 on the time axis of the recorded waveform data in response to the note arrangement instruction of the user. In the figure, the lyrics of each note are input for singing synthesis, but if it is musical instrument sound synthesis, the lyrics are unnecessary. For the period T12, the CPU 11 processes the score data D1_S for synthesis using the auxiliary trained speech synthesizer 1, and performs speech synthesis of the sound quality indicated by the sound source ID of the acoustic data D2_T for auxiliary learning. In the CPU 11, the period T12 is the synthetic acoustic data D3 voice-synthesized with the sound quality indicated by the sound source ID, and the section T11 is the content that is the auxiliary learning acoustic data D2_T. Alternatively, the period T12 is the synthetic acoustic data D3 voice-synthesized with the sound quality indicated by the sound source ID, and the section T11 is the sound quality of the sound source ID synthesized by the voice synthesizer 1 with the auxiliary learning acoustic data D2_T as an input. Content that is synthetic audio data D3 may be generated.

(7) Other Embodiments By using the voice synthesizer 1 of the present embodiment described above, the user's singing voice and the played musical instrument sound are inserted into the voice-synthesized song based on the score data D1_S for synthesis. It is also possible to do. FIG. 10 shows a user interface 200 that reproduces a voice-synthesized song in the voice synthesizer 1. In the periods T21 and T23, the score data D1_S for synthesis is arranged by the user, and the CPU 11 executes singing synthesis with the sound quality indicated by the sound source ID specified by the user. When the user instructs the start of overdubbing while the user interface 200 shown in FIG. 10 is displayed, the CPU 11 executes the voice synthesis program P1 to reproduce the synthetic sound data D3 having the sound quality indicated by the sound source ID. conduct. At this time, the current time position is indicated by the time bar 214 in the user interface 200. The user sings while looking at the position of the time bar 214. The voice sung by the user is picked up through a microphone connected to the voice synthesizer 1 and recorded as synthetic acoustic data D2_S. In the figure, the waveform 202 shows the waveform of the acoustic data D2_S for synthesis. The CPU 11 processes the synthetic acoustic data D2_S using the acoustic encoder 121 and the acoustic decoder 133, and generates the synthetic acoustic data D3 of the sound quality indicated by the sound source ID. FIG. 11 shows a user interface 200 to which the waveform 215 of the synthetic acoustic data D3 is combined. In the CPU 11, the periods T21 and T23 are synthetic acoustic data D3 of the sound quality indicated by the sound source ID sung and synthesized from the score data D1_S for synthesis, and the period T22 is the sound quality indicated by the sound source ID sung and synthesized from the user singing. Generate content that is synthetic acoustic data D3.

In the above-described embodiment, the case where the voice synthesizer 1 synthesizes the singing voice of the singer specified by the sound source ID has been described as an example. The voice synthesizer 1 of the present embodiment can be used for synthesizing voices having various sound qualities in addition to synthesizing the singing voice of a specific singer. For example, the voice synthesizer 1 can be used for synthesizing the performance sound of a musical instrument specified by a sound source ID.

In the above-described embodiment, the intermediate feature data MF1 generated based on the synthetic score data D1_S and the intermediate feature data MF2 generated based on the synthetic acoustic data D2_S are combined on the time axis. , Acoustic feature data AFS was generated. As another embodiment, the acoustic feature data AFS generated based on the intermediate feature data MF1 and the acoustic feature data AFS generated based on the intermediate feature data MF2 are combined to generate the synthetic acoustic data D3. You may. Alternatively, as another embodiment, the synthetic acoustic data D3 is generated from the acoustic feature data AFS generated based on the intermediate feature data MF1, and the synthetic acoustic data is generated from the acoustic feature data AFS generated based on the intermediate feature data MF2. D3 may be generated and these two synthetic acoustic data D3 may be combined.

The voice synthesizer 1 of the present embodiment can synthesize the singing voice of a singer specified by a sound source ID by using the synthetic acoustic data D2_S without a phoneme label. This makes it possible to use the speech synthesizer 1 as a cross-language synthesizer. That is, even if the sound decoder 133 is trained only with Japanese sound data for the sound source ID, if it is trained with English sound data with another sound source ID, it is for synthesizing English lyrics. By giving the acoustic data D2_S, it is possible to generate a singing in an English language with the sound quality of the sound source ID.

In the above embodiment, the speech synthesis program P1 and the training program P2 have been described by taking the case where they are stored in the storage device 16 as an example. The speech synthesis program P1 and the training program P2 are provided in a form stored in a computer-readable recording medium RM, and may be installed in the storage device 16 or the ROM 13. Further, when the voice synthesizer 1 is connected to the network via the communication interface 19, even if the voice synthesis program P1 or the training program P2 distributed from the server connected to the network is installed in the storage device 16 or the ROM 13. good. Alternatively, the CPU 11 may access the storage medium RM via the device interface 18 and execute the speech synthesis program P1 or the training program P2 stored in the storage medium RM.

(8) Effect of the Embodiment As described above, the voice synthesis method according to the present embodiment is a voice synthesis method realized by a computer, and is a first intermediate feature amount from the score feature amount of the score data D1. The score encoder 111 that generates (intermediate feature data MF1), the acoustic encoder 121 that generates the second intermediate feature amount (intermediate feature data MF2) from the acoustic feature amount of the acoustic data D2, and the first intermediate feature amount (intermediate feature data). Prepare an acoustic decoder 133 that generates an acoustic feature amount (acoustic feature data AFS) based on the MF1) or the second intermediate feature amount (intermediate feature data MF2), receive the auxiliary learning acoustic data D2_T, and use the acoustic encoder 121. Using the second intermediate feature amount (intermediate feature data MF2) generated from the acoustic feature amount of the auxiliary learning acoustic data D2_T and the acoustic feature amount of the auxiliary learning acoustic data D2_T, the auxiliary learning acoustic data D2_T The acoustic decoder 133 is assisted and trained so as to generate an acoustic feature amount (acoustic feature data AFS) close to the acoustic feature amount of the above, and the score is arranged on the time axis of the auxiliary learning acoustic data D2_T via the user interface 200. By receiving the data D1 and processing the first intermediate feature amount (intermediate feature data MF1) generated from the score data D1 arranged using the score encoder 111 by the auxiliary trained acoustic decoder 133, the acoustic feature amount (Acoustic feature data AFS) is generated. This makes it easy to add acoustic data of the same sound quality to the recorded acoustic data of a specific sound quality, and to partially modify the acoustic data while maintaining the sound quality.

To prepare, the first intermediate feature amount (intermediate feature data MF1) generated by the score encoder 111 based on the basic learning score data D1_R and the second intermediate generated by the acoustic encoder 121 based on the basic learning acoustic data D2_R. The feature amount (intermediate feature data MF2) approaches, and the acoustic feature amount (acoustic feature data AFS) generated by the acoustic decoder 133 approaches the acoustic feature amount acquired from the basic learning acoustic data D2_R. , Score encoder 111, acoustic encoder 121 and acoustic decoder 133 may be included. The acoustic decoder 133 can generate acoustic feature data AFS for either the intermediate feature data MF1 generated based on the score data D1 or the intermediate feature data MF2 generated based on the acoustic data D2.

The acoustic decoder 133 may be trained based on a first identifier that specifies a timbre. It is possible to generate synthetic speech with sound quality according to the identifier.

Based on the second identifier that specifies the timbre, the acoustic decoder 133 may be trained with a timbre different from the timbre specified by the first identifier. It is possible to generate synthetic speech with different sound quality depending on the identifier.

The voice synthesis program of the acoustic decoder according to the present embodiment is a program that causes a computer to execute a voice synthesis method, and based on the program, the computer determines the first intermediate feature amount (intermediate feature) from the score feature amount of the score data D1. A score encoder 111 that generates data MF1), an acoustic encoder 121 that generates a second intermediate feature amount (intermediate feature data MF2) from the score feature amount of acoustic data D2, and a first intermediate feature amount (intermediate feature data MF1) or An acoustic decoder 133 that generates an acoustic feature amount (acoustic feature data AFS) based on the second intermediate feature amount (intermediate feature data MF2) is prepared, an auxiliary learning acoustic data D2_T is received, and auxiliary learning is performed using the acoustic encoder 121. Using the second intermediate feature amount (intermediate feature data MF2) generated from the acoustic feature amount of the auxiliary learning acoustic data D2_T and the acoustic feature amount of the auxiliary learning acoustic data D2_T, the acoustic feature amount of the auxiliary learning acoustic data D2_T. The acoustic decoder 133 is assisted and trained so as to generate an acoustic feature amount (acoustic feature data AFS) close to, and the score data D1 arranged on the time axis of the auxiliary learning acoustic data D2_T is received via the user interface 200. , The first intermediate feature amount (intermediate feature data MF1) generated from the score data D1 arranged using the score encoder 111 is processed by the auxiliary trained acoustic decoder 133 to obtain an acoustic feature amount (acoustic feature data). AFS) is generated. This makes it easy to add acoustic data of the same sound quality to the recorded acoustic data of a specific sound quality, and to partially modify the acoustic data while maintaining the sound quality.

100 ... Control unit, 110 ... Conversion unit, 111 ... Score encoder, 120 ... Analysis unit, 121 ... Acoustic encoder, 131 ... Switching unit 133 ... Acoustic decoder, 134 ... Bocoder, D1 ... Score data, D2 ... Acoustic data, D3 ... Synthetic acoustic data, SF ... Score feature data, AF ... Acoustic feature data, MF1, MF2 ... Intermediate feature data, AFS ... Acoustic feature data

Claims (5)

It is a speech synthesis method realized by a computer.
A musical score encoder that generates a first intermediate feature amount from a musical score feature amount of musical score data, an acoustic encoder that generates a second intermediate feature amount from an acoustic feature amount of acoustic data, and the first intermediate feature amount or the second intermediate feature amount. Prepare an acoustic decoder that generates acoustic features based on the quantity,
Receives acoustic data for auxiliary learning
Using the second intermediate feature amount generated from the acoustic feature amount of the auxiliary learning acoustic data using the acoustic encoder and the acoustic feature amount of the auxiliary learning acoustic data, the auxiliary learning acoustic data can be obtained. The acoustic decoder is assisted and trained to generate acoustic features that are close to the acoustic features.
The musical score data arranged on the time axis of the auxiliary learning acoustic data is received via the user interface, and the score data is received.
A speech synthesis method for generating acoustic features by processing the first intermediate features generated from the arranged musical score data using the score encoder with the auxiliary trained acoustic decoder. The above preparation is
The first intermediate feature amount generated by the score encoder based on the basic learning score data and the second intermediate feature amount generated by the acoustic encoder based on the basic learning acoustic data approach each other and the acoustic. Training the score encoder, the acoustic encoder, and the acoustic decoder so that the acoustic features generated by the decoder approach the acoustic features acquired from the basic learning acoustic data.
The voice synthesis method according to claim 1. The speech synthesis method according to claim 1 or 2, wherein the acoustic decoder is trained based on a first identifier that specifies a timbre. The speech synthesis method according to claim 3, wherein the acoustic decoder is trained with a tone different from the tone specified by the first identifier based on the second identifier that specifies the tone. A program that causes a computer to execute a speech synthesis method, and the computer is based on the program.
A score encoder that generates a first intermediate feature from a score feature of score data, an acoustic encoder that generates a second intermediate feature from a score feature of acoustic data, and the first intermediate feature or the second intermediate feature. Prepare an acoustic decoder that generates acoustic features based on the quantity,
Receives acoustic data for auxiliary learning
Using the second intermediate feature amount generated from the acoustic feature amount of the auxiliary learning acoustic data using the acoustic encoder and the acoustic feature amount of the auxiliary learning acoustic data, the auxiliary learning acoustic data can be obtained. The acoustic decoder is assisted and trained to generate acoustic features that are close to the acoustic features.
The musical score data arranged on the time axis of the auxiliary learning acoustic data is received via the user interface, and the score data is received.
A speech synthesis program that generates acoustic features by processing the first intermediate features generated from the arranged musical score data using the score encoder with the auxiliary trained acoustic decoder.

Images