JP2017097332A - Voice synthesizer and voice synthesizing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve operability when editing an addition parameter generated from a configuration parameter.SOLUTION: A voice synthesizer comprises: a storage section which preliminarily stores an acoustic model including an acoustic parameter learned through a statistical method; a parameter generation section which receives input of a musical score generated from the acoustic parameter and generates both a first parameter series representing a change in the acoustic parameter value corresponding to the musical score and a second parameter series generated from the first parameter; a parameter presentation section which presents at least the second parameter series in an editable form to a user; and a synthesis section which synthesizes singing voice using at least either respective acoustic parameter values included in the first parameter series or respective acoustic parameter values included in the second parameter series. When the second parameter series is edited, the parameter generation section regenerates the first parameter series using the edited second parameter series.SELECTED DRAWING: Figure 10

Description

  The present invention relates to a speech synthesis technique.

  Conventionally, various devices for synthesizing natural speech by a computer have been proposed. As a method for enabling such speech synthesis, a waveform connection method and a method using a hidden Markov model (HMM) are known.

  Among voice synthesis, in particular, in singing voice synthesis for synthesizing a singing voice, it is known that an acoustically natural synthesized sound can be generated by changing the pitch (pitch) and volume (volume of sound). Conventionally, as a method of changing the pitch and volume, various parameters constituting the pitch and volume (for example, fundamental frequency parameters, parameters of singing expression such as vibrato, hereinafter also referred to as “configuration parameters”) are GUI (Graphical User Interface). ) Is known (for example, Patent Document 1). In such an apparatus, the user can change the pitch and volume by changing the configuration parameters using the GUI.

Japanese Patent Laying-Open No. 2015-049253

  However, in the above-described conventional technique, the change in pitch and volume only follows the change of the configuration parameter. That is, the conventional technique has a problem that the user cannot directly edit the pitch and volume and cannot perform intuitive operations. Such a problem is common to the case where either one of the pitch and the volume is to be changed and the case where both the pitch and the volume are to be changed. Further, such a problem is not limited to the pitch and volume, and is a problem common to various parameters generated from the configuration parameters (hereinafter also referred to as “addition parameters”).

  An object of the present invention is to improve operability when editing an addition parameter generated from a configuration parameter.

  SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.

(1) According to an aspect of the present invention, a speech synthesizer is provided. The speech synthesizer includes: a storage unit that stores in advance an acoustic model including an acoustic parameter learned by a statistical method; a parameter generation unit; a first parameter series generated from the stored acoustic parameter A parameter generation unit for generating a first parameter series representing a change in acoustic parameter value corresponding to the input score; and a second parameter series generated from the first parameter series; A parameter presenting unit that presents at least the second parameter series to the user in an editable manner; each acoustic parameter value included in the first parameter series; and each included in the second parameter series A synthesis unit that synthesizes a singing voice using at least one of acoustic parameter values; and the parameter generation unit; If the parameter sequence is edited to regenerate the first parameter sequence using said second parameter sequence after editing.
According to the speech synthesizer of this aspect, the first parameter series (that is, the constituent parameter series) representing the change in the acoustic parameter value corresponding to the input score, and the second parameter generated from the first parameter series A series (that is, an addition parameter series) is handled. According to the speech synthesizer of this embodiment, the second parameter series (addition parameter series) is presented in an editable manner by the parameter presentation unit. For this reason, the user can directly edit each value in the second parameter series (addition parameter series) to a desired value. Therefore, when editing the second parameter series (addition parameter series) The operability can be improved.

(2) In the speech synthesizer according to the above aspect; the parameter presentation unit; further presents the first parameter series to the user in addition to the second parameter series; and the first parameter series is regenerated. If it has been done, the first parameter series after regeneration may be presented to the user.
According to the speech synthesizer of this embodiment, the user can check both the first and second parameter series, and can also check the first parameter series after regeneration. Convenience can be improved.

(3) In the speech synthesizer according to the above aspect; the parameter presenting unit may present the first parameter series and the second parameter series in a manner in which the user can visually recognize the first parameter series and the second parameter series simultaneously.
According to the speech synthesizer of this embodiment, the user can confirm the first and second parameter series at the same time, and therefore the convenience for the user can be further improved.

(4) In the speech synthesizer according to the above aspect; the parameter presentation unit; and the first parameter series in a state where visibility of the second parameter series is higher than that of the first parameter series. The second parameter series may be presented.
According to the speech synthesizer of this form, the user can more easily confirm (view) the second parameter series between the first and second parameter series presented simultaneously. Therefore, the user can intuitively understand that the second parameter series is an editing target.

(5) In the speech synthesizer of the above aspect; the parameter presentation unit presents the first parameter series to the user in an editable manner in addition to the second parameter series; the parameter generation unit Further, when the first parameter series is edited, the second parameter series is regenerated using the edited first parameter series; the parameter presenting unit further includes the second parameter series When the parameter series is regenerated, the second parameter series after regeneration may be presented to the user.
According to the speech synthesizer of this aspect, the user can directly edit both the first and second parameter series to desired values, and the first and second parameter series after regeneration can be edited. It can also be confirmed. That is, since the user can edit the first and second parameter series in both directions and check the result in real time, the convenience for the user can be further improved.

(6) In the speech synthesizer of the above aspect, further comprising: a parameter acquisition unit that acquires the contents of the editing; the parameter acquisition unit; performing an operation according to a first operation method assigned in advance as the first parameter Acquired as editing contents for a certain parameter series included in the series; a second operation method assigned in advance, wherein an operation by a second operation method different from the first operation method is performed by the first operation method; You may acquire as edit content with respect to another parameter series contained in a parameter series.
According to the speech synthesizer of this aspect, the user can designate one parameter series intended for editing from among various parameter series included in the first parameter series by using different operation methods. Therefore, convenience for the user can be further improved.

(7) In the speech synthesizer of the above aspect, further comprising: a parameter acquisition unit that acquires the contents of the editing; the parameter acquisition unit; performing an operation according to a third operation method assigned in advance as the first parameter Acquired as editing contents for a series; a fourth operation method assigned in advance, which is an operation according to a fourth operating method different from the third operating method, is acquired as editing contents for the second parameter series May be.
According to the speech synthesizer of this aspect, the user can designate a parameter series intended for editing from among the first and second parameter series by using different operation methods. The sex can be further improved.

(8) In the speech synthesizer of the above aspect; the first parameter series includes at least: a pitch parameter series representing a change in pitch parameter value; and a song expression parameter representing a change in parameter value of a song expression The parameter presenting unit may present all parameter series included in the first parameter series to the user.
According to the speech synthesizer of this aspect, the user can confirm all the parameter series included in the first parameter series, so that the convenience for the user can be further improved.

(9) In the speech synthesizer of the above aspect; the singing expression may include at least one of vibrato, sneeze, fist, attack, and release.

(10) In the speech synthesizer of the above form; an image display device including a pointing device is connected to the speech synthesizer; the parameter presenting unit is capable of being operated using the graphical user interface and the pointing device In this manner, the second parameter series may be displayed on the image display device.
According to the speech synthesizer of this form, the user can easily edit the second parameter series displayed on the GUI by using the pointing device.

(11) In the speech synthesizer of the above aspect; the second parameter series includes at least a parameter series relating to pitch, a parameter series relating to loudness, a parameter series relating to voice age, and a blurring condition of sound. Any one of the parameter series may be included.

  The present invention can be realized in various modes. For example, a singing voice synthesizing apparatus, a singing voice synthesizing system using the singing voice synthesizing apparatus, a singing voice synthesizing apparatus, and a singing voice synthesizing system to realize the functions of the information processing system. The present invention can be realized in the form of a method executed in the apparatus, a computer program, a server apparatus for distributing the computer program, a non-temporary storage medium storing the computer program, and the like.

The schematic block diagram of embodiment of a singing voice synthesizer. Explanatory drawing which shows the acoustic model using HMM, and the principle of the learning. The flowchart which shows a singing voice synthetic | combination preparation routine. Explanatory drawing which shows the typical parameter extracted from data. Explanatory drawing which shows the context-dependent phoneme which is a basic unit of learning using an acoustic model. Explanatory drawing which shows a mode that the set of the state of HMM is clustered. Explanatory drawing which shows the mode continuation length model and the mode of decision tree of each parameter. The flowchart which shows the procedure of a singing voice synthetic | combination routine. The figure which shows an example of an edit screen. The figure which shows the 1st example of the editing method of a parameter series. The figure which shows the 2nd example of the editing method of a parameter series. The figure which shows the 3rd example of the editing method of a parameter series. The figure which shows the 4th example of the editing method of a parameter series. The figure which shows the 5th example of the editing method of a parameter series.

A. Embodiment:
Several embodiments of the present invention will be described with reference to the drawings.

A-1. Composition of singing voice processing device:
FIG. 1 is a schematic configuration diagram showing a first embodiment of the singing voice processing apparatus of the present invention. The singing voice processing apparatus 100 of the present embodiment is a device that synthesizes a singing voice from an acoustic model including acoustic parameters learned by a statistical method, and a user can adjust the pitch of the synthesized sound (hereinafter, “pitch”). Alternatively, the singing voice processing apparatus 100 is improved in operability when editing the pitch and the volume of sound (hereinafter also referred to as “volume”). The singing voice processing apparatus 100 includes both a configuration for learning acoustic parameters for synthesizing a singing voice in advance and a configuration for actually synthesizing a singing voice (that is, a configuration as a singing voice synthesis device). If only singing voice synthesis is performed, the former configuration is not necessary. Here, both will be described together. However, if only singing voice synthesis is performed, a learned acoustic model may be stored in a storage unit such as a hard disk, and speech synthesis may be performed using this acoustic model.

  The singing voice processing apparatus 100 shown in FIG. 1 includes a computer PC1 connected via a network NW, and a server 30 and another computer PC2 connected to the server 30. The singing voice processing apparatus 100 can be configured by a single computer, or can be configured by a plurality of computers distributed on a network.

  The computer PC1 is provided for inputting a singing voice and includes a score input unit 10 and a voice input unit 20 for inputting a singing (speech as voice). The score input unit 10 generally uses a keyboard, and the voice input unit 20 uses a microphone. When a singer sings into the microphone as the voice input unit 20 and inputs a score including the lyrics from the keyboard which is the score input unit 10, the score and the singing voice are associated with each other and stored in the computer PC1. When inputting a score, it may be input in the form of a score written on a staff, or may be input using a keyboard type keyboard. In the latter case, a piano keyboard type keyboard is used as the score input unit 10 in addition to a text input keyboard, and the keyboard and keyboard for inputting pitches and sound lengths using a keyboard type keyboard. The input of lyrics by Kana (in the case of Japanese, a kana character string corresponding to each sound) may be performed in association with each other. Singing sheet music and singing voice data are accumulated at least several minutes at a time. As will be described later, the musical score and singing voice data are analyzed by an acoustic parameter learning unit in the server 30. In order to analyze, not all phonemes and their combinations and all singing expressions need to be included in the score and singing voice data, but the types of phonemes and combinations that allow statistical learning, It is desirable that various singing expressions are included. Therefore, generally, a singing voice of about several minutes to several tens of minutes is required.

  The reason why the computer PC1 for singing voice input is separated from the server 30 is to facilitate data input of a plurality of singers. The computer PC1 can be realized by a notebook personal computer equipped with a microphone, and can be easily carried to collect and store musical scores and singing voices. In this example, the singing voice is directly input from the voice input unit 20 such as a microphone. However, the sound source does not have to be a live song and may be collected from a singing voice recorded on a CD or DVD. There is no problem. Or it is good also as what inputs via network NW.

  The singing voice data collected and recorded in this way is sent to the server 30 via the network NW and stored in the hard disk 31 in the server 30. In the server 30, a score analysis unit 33, an acoustic parameter learning unit 40, and an acoustic model storage unit 50 are provided. The acoustic model storage unit 50 functions as a “storage unit”. In addition to this, the server 30 is provided with a parameter acquisition unit 55, a parameter presentation unit 56, a score analysis unit 57, and a speech synthesis unit 60, and constitutes a singing voice synthesis apparatus together with the acoustic model storage unit 50. The speech synthesizer 60 functions as a “synthesizer”.

  The parameter acquisition unit 55, the parameter presentation unit 56, and the score analysis unit 57 exchange data with the computer PC2. The computer PC2 is provided with a keyboard 51, a pointing device 52 such as a mouse, and a display unit 53. From the keyboard 51, mainly the musical score data of the singing voice to be synthesized is input. In addition, the display unit 53 displays a GUI (Graphical User Interface) representing an acoustic parameter described later. The computer PC2 can specify or modify this acoustic parameter by using the GUI and the pointing device 52. Details will be described later.

  The learning unit 40 in the server 30 will be described. The learning unit performs learning for constructing an acoustic model from a set of each score and voice data stored in the hard disk 31. Since this learning is performed in order to finally synthesize a singing voice, the outline of the technique of singing voice synthesis used in the present embodiment will be described first. In the present embodiment, various organs such as vocal cords and palate used by a person to generate a singing voice are regarded as a filter having a sound source (excitation source) and a predetermined transfer characteristic, and this is simulated by a digital filter. To do. At this time, a sequence along the time axis of the acoustic parameters including the spectral parameters extracted from the speech waveform, the fundamental frequency, and the period / non-period information is used. If these acoustic parameter strings can be estimated from the score, the corresponding speech can be synthesized from the score. Therefore, the relationship between the acoustic parameter sequence and the corresponding score is learned from the actual singing voice data and the score using a predetermined acoustic model. As such an acoustic model, in this embodiment, a case where a hidden Markov model (HMM, Hidden Markov Model) is employed is illustrated.

  FIG. 2 is an explanatory diagram showing an acoustic model using the HMM and the principle of learning thereof. In FIG. 2, the phoneme is a minimum unit, and within this minimum unit, the temporal variation of the observation sequence is shown as an example in three states, with “1” being the first position and “1” being the center position. “2” is added, and “3” is added to the last position.

In FIG. 2, a _ij indicates a transition probability. When i = j, it indicates the probability of staying in the same part of the phoneme, and when j = i + 1, it indicates the probability of transition to the next part. At this time, the observation sequence o has a value obtained by the output probability density function b _q (o _t ). As described above, in this embodiment, the HMM considering the context is learned from the score and the singing voice data. Once the HMM learned for each singer is completed, the singing voice is obtained from the score using the HMM. It is synthesized. The state transition probability a _ij and the output probability density function b _q (o _t ) learned by such an HMM can be estimated using an expected value maximization (EM) algorithm which is one of the maximum likelihood estimation methods.

  Next, acoustic parameters used in HMM learning will be described. It has already been explained that spectral parameters, fundamental frequencies, and period / aperiodic information extracted from speech waveforms are basically assumed as acoustic parameters. Here, a mel cepstrum, a line spectrum pair (LSP), or the like is used as the spectrum parameter. In this embodiment, a mel cepstrum was used. The mel cepstrum is information obtained by further inversely Fourier transforming the logarithm of the Fourier transform of an audio signal and having a large amount of information in the low frequency region in accordance with human auditory characteristics. In general, a logarithmic value is used as the fundamental frequency. Period / non-period is a distinction between periodic speech such as vowels and non-periodic speech such as consonants. In addition, parameters called dynamic features are also used. The dynamic feature is a parameter corresponding to a first-order derivative (delta) or a second-order derivative (delta delta) in the time direction of parameters such as a fundamental frequency and a mel cepstrum. These parameters are used to compensate for the fact that the HMM is difficult to model the correlation in the time axis direction of time series data. By handling dynamic features, the joints when synthesizing phoneme sequences are smoothed.

Up to this point, the acoustic model has been described as being an HMM using spectral parameters and fundamental frequencies, but the model actually used is more complex. Hereinafter, the model actually introduced will be briefly described.
(A) State duration model: Since the length of each phoneme included in the singing voice varies depending on the singing style, etc., the temporal structure of the speech (how long the phoneme lasts) is more accurate. An explicit state duration distribution is used to model. This is a model called a hidden semi-Markov model. In the present embodiment, simply “HMM” means an HMM with a state duration model.
(B) Context-dependent model: Acoustic parameters such as speech spectrum, fundamental frequency, and duration are easily affected by linguistic information contained in the lyrics and how to sing based on the score. For this reason, modeling is performed in consideration of linguistic language information and contexts such as pitch, tempo, tonality, and time signature obtained from the score.
(C) Multi-spatial probability distribution HMM: The voice has a silent part, and there is no time-series data of the fundamental frequency. In the present embodiment, a multi-space probability distribution HMM (MSD-HMM) is used to handle such special time series.

(D) Singing expression model: The singing voice has various deviations when viewed from the score. This is called broad singing expression. Since singing expressions characterize singing with a specific singing style, they are also used for learning. Hereafter, what is included in a broad sense of singing expression is listed. All of these need not be included in the singing expression.
(1) Timing: The actual singing voice may be unintentionally or intentionally deviated from the position on the time axis of the note calculated from the score. For example, a consonant is often uttered slightly before the start timing of the note. In addition, there are singing expressions that intentionally shift the timing of utterances, such as “before”, “after”, and “tame”. For this reason, the temporal deviation from the actual utterance based on the absolute time calculated from the score is modeled in units of phonemes.
(2) Pitch vibrato: Pitch vibrato is a singing expression that periodically fluctuates the pitch. The timing at which the pitch vibrato is applied to the singing voice, the period, and the change in amplitude differ for each singing style, and are therefore used for learning the acoustic model for each singing style. The pitch vibrato is further treated as two parameters of its period and amplitude, and is incorporated into the acoustic model.
(3) Sound loudness vibrato: Sound loudness (volume) vibrato is a singing expression that periodically fluctuates the loudness. Similar to the pitch vibrato, the timing at which the loudness vibrato in the singing voice is applied, the period, and the change in amplitude differ for each singing style. For this reason, the vibrato of the loudness is also used for learning the acoustic model for each singing style. The loudness vibrato is further treated as two parameters of its period and amplitude, and is incorporated into the acoustic model.
(4) Other singing expressions: There are various singing expressions other than the above-mentioned vibrato. For example, singing expressions that affect the pitch (pitch) include “shakuri” and “fist”. Here, “scribbing” includes raising and lowering. In addition, “attack release” is a singing expression that affects the volume of sound. Such a singing expression can be treated as a variation in the middle of a pitch or volume note, and is incorporated into an acoustic model.
In this specification, the above-mentioned models are referred to as HMMs. Note that it is not always necessary to use all the acoustic models described above, and some of them may be omitted.

  Returning to FIG. 1, the configuration in the server 30 will be further described. As described above, in order to learn the acoustic model for singing voice synthesis, the F0 extraction unit 41 that extracts the fundamental frequency of the singing voice and its derivative (delta parameter) from the singing voice data stored in the hard disk 31, the singing voice SP extraction unit 43 that extracts the included spectral parameters and their derivatives (delta parameters), singing P extraction unit 44 that extracts the above-described broad singing expression parameters, and an HMM that learns the HMM using these extracted acoustic parameters A learning unit 45 is included. As described with reference to FIG. 2, these parameters are learned based on phoneme string data (context-dependent model) that is an array of phonemes.

  With reference to FIG. 3, processing executed for preparation for singing voice synthesis will be described. The first half (steps S110 to S120) of the singing voice synthesis preparation routine shown in FIG. 3 is executed by the computer PC1. The second half (steps S140 to S160) is executed by the server.

A-2. Singing voice synthesis preparation routine:
When this singing voice synthesis preparation routine is started, singing voice data is first input (step S110). The singing voice data is input by inputting at least a few minutes of singing through the voice input unit 20 such as a microphone and storing it as digital data. Subsequently, the score input unit 10 inputs a score (step S120). The pitch extracted from the inputted score and the lyrics (pronunciation) are associated with the singing voice data.

  Next, the server 30 that receives the singing voice data and the score performs data analysis (step S140). Data analysis is performed by sequentially extracting singing voices stored in the hard disk 31. For data analysis, the score analysis using the score analysis unit 33, the analysis of the fundamental frequency and its related parameters by the F0 extraction unit 41 of the learning unit 40, the spectral parameter (SP) and its related parameters by the SP extraction unit 43, Analysis and further analysis of parameters related to singing expression are included. FIG. 4 illustrates various parameters extracted by such analysis.

Fundamental frequency, typically are handled as logarithmic fundamental frequency p _t, as the related parameters, distinction between voiced / unvoiced, first derivative of the logarithmic fundamental frequency (Delta] p _t) and second derivative (Δ ^{2 p} _t) is considered It is done. These are sometimes called sound source information. It should be noted that the silent portion does not have a value of logarithmic fundamental frequency p _t. For this reason, the voiced / unvoiced distinction is made by a method such as putting a predetermined constant in the voiceless part. As the spectral parameter, cepstrum _{c t} and its first derivative (.DELTA.c _t), second derivative (Δ ² c _t), and the like. Temporary derivative and second derivative are used to take into account temporal variations. These are sometimes referred to as spectral information. Furthermore, in addition to such sound source information and spectrum information, in this embodiment, singing expression information is handled.

The singing expression information, and the pitch of the vibrato of the period V1f _t and amplitude V1a _t, and the size of the vibrato of the period V2f _t and amplitude V2a _t of sound, a parameter set S1~S6 about jerking, parameters for the attack released The sets AR1 to AR6 are modeled and included in phonemes. The pitch vibrato period, the pitch vibrato amplitude, the loudness vibrato period, and the loudness vibrato amplitude are the temporary derivative (Δ) and secondary derivative (Δ ² ) For convenience of illustration, in FIG. 4, illustration of temporary differentiation and secondary differentiation for these periods and amplitudes is omitted. In the present embodiment, the three parameters of “length”, “height”, and “steepness” are provided at the beginning and the end of the note, respectively, with respect to scrambling and attack / release. Therefore, each consists of six parameters. A method for learning parameters such as the parameters of scouring will be described later. Among the above parameters, first derivative and second derivative of each parameter including the mel cepstrum c _t is used to account for time-varying. By considering the dynamic features, the connection between the sounds during the synthesis of the singing voice becomes smooth. Description of the speech synthesis method using dynamic features is omitted.

  Subsequently, a context-dependent model is constructed using the analyzed data (step S150). The context-dependent model is constructed by learning a hidden semi-Markov model using each extracted parameter. As described above, the context-dependent model is constructed for each phoneme included in the basic singing voice data. However, instead of handling the phoneme alone, HMM learning is performed together with many factors that cause speech fluctuations in speech synthesis. . Factors that cause speech fluctuations in the speech to be synthesized include, for example, a combination of phonemes before and after the phoneme (a phoneme triphone that is a combination of the phonemes before and after the phoneme and the phonemes before and after the phoneme) Phone), music score information and language information. The musical score information includes the pitches of the front and back phonemes, the length of rests, and the like. The language information includes various information such as the part of speech of the word to which the phoneme belongs, the utilization form or accent position, and the accent type. These factors are collectively referred to as context.

When performing smooth speech synthesis, there are many factors to be considered, but in order to outline the learning method, FIG. 5 shows the above triphone as an example of a phoneme with context. FIG. 5 is an explanatory diagram showing a case where a triphone is taken out, taking as an example a singing voice “I don't know at all”. In the voice data “I don't know at all”, the phoneme a appears a plurality of times, but even if it is the same phoneme, the acoustic characteristics of the voice will be different if the context of the phonemes before and after is different. Therefore, even if the phoneme is the same a, it is separately modeled as a triphone that considers the phonemes before and after. When attention is paid to a specific phoneme, there may be a case where there is no preceding or following phoneme. In this case, it is assumed that there is a phoneme representing silence such as “sil”. Phonemes are sequentially extracted from the singing voice stored in the hard disk 31 in consideration of the context. A phoneme in which context is taken into account is hereinafter referred to as a context-dependent phoneme. The number of context-dependent phonemes taken from minutes to tens of minutes of voices ranges from hundreds to tens of thousands. The state transition probability a _ij and the output probability density function b _q (o _t ) shown in FIG. 2 are learned for all the context-dependent phonemes extracted from the hard disk 31. That is, the parameters shown in FIG. 4 are extracted for each frame to which the context-dependent phoneme belongs, and the HMM of each context-dependent phoneme is learned.

  Subsequently, the context-dependent model is clustered to obtain a representative Gaussian distribution for each cluster (step S160). Specifically, the context-dependent models constructed in the context-dependent model construction (step S150) are classified by binary trees. First, all the context-dependent models constructed in the construction of the context-dependent model (step S150) are defined as one cluster. Select the best question from the list of context questions prepared for that cluster in advance and apply that question to the context-dependent model in the cluster (actually the same state number) to To divide. A question is selected and applied to the classified clusters in the same manner, and the clusters are further divided. Classification is performed by repeating this. An example of a decision tree subjected to clustering is shown in FIG. In FIG. 6, a thick arrow indicates that the determination for the branch condition of each binary tree is “YES”, and a thin arrow indicates that the determination in each binary tree is “NO”. Thus, if a representative Gaussian distribution is obtained for each cluster, a model (context dependent model) capable of speech synthesis for each clustered context is obtained. In other words, clustering is performed to construct a decision tree for selecting a context-dependent model used for speech synthesis. In speech synthesis, it is desirable to use a context-dependent model with the same context if possible. However, as described above, context-dependent phonemes corresponding to all context combinations cannot be obtained from limited audio data, and context-dependent models corresponding to all context combinations cannot be learned. . Therefore, a decision tree is created by clustering and prepared so that the most suitable context-dependent model can be selected during speech synthesis.

  Clustering and a process for obtaining a representative Gaussian distribution are performed for all statistically learned features. This state is schematically shown in FIG. Focusing on one context-dependent phoneme, each state is given a length for which each state continues by a state duration model. A binary tree that determines this state duration is learned from a number of context-dependent phonemes. This is called a state continuation length decision tree. In addition, a binary tree that determines the difference between the time information of the score and the actual singing timing is learned from a timing model that considers the context. This is called a timing decision tree. Similarly, a decision tree of squealing is learned from a squealing model including squealing up and squeezing down, and an attack / release decision tree is learned from an attack / release model. Further, a mel cepstrum decision tree, a fundamental frequency decision tree, and a decision tree for each singing expression (pitch vibrato, sound loudness vibrato, etc.) are constructed for each analyzed parameter.

  The model for singing expression and how to make the decision tree will be briefly explained below using sukuri as an example. First, an acoustic model is created ignoring the parameters of squealing, and singing voice synthesis is performed using this model. What is synthesized is a singing voice that does not include sneezing. Then, the basic frequency sequence of the voice data stored in advance as including the screaming is compared with the basic frequency sequence of the singing voice generated from the acoustic model not including the screaming. Since the difference between the two is the presence or absence of squealing, the "height", "length", and "steepness" of squealing are extracted for each of the beginning and end of each note, and a context-dependent model is created for each note. can do. Then, a sneeze model is constructed considering the extracted context, and a sneeze decision tree is created by context clustering. For attack / release, the same processing is performed to obtain a difference with respect to the volume, and an attack / release model is constructed from this, and an attack / release decision tree is created. Thus, the fact that a set of these decision trees is obtained based on the singing voice data of a specific singing style is nothing other than the learning of the acoustic model of that singing style.

  The acoustic model learned in this way is stored in the acoustic model storage unit 50. This completes the preparation for singing voice synthesis. Although the present embodiment has been described from the preparation of the acoustic model, singing voice synthesis may be performed using an acoustic model prepared in advance. The learning of the acoustic model is not limited to the above method, and other methods may be used. When performing the singing voice synthesis of this embodiment using an acoustic model prepared in advance, the PC 1, the hard disk 31, the score analysis unit 33, the FO extraction unit 41, the SP extraction unit 43, the singing P extraction unit 44, and the HMM learning in FIG. The part 45 may be omitted.

  When the above-described singing voice synthesis preparation routine (FIG. 3) is executed, the acoustic model storage unit 50 of the server 30 is in a state where the acoustic model is stored. Singing voice synthesis is performed by using the acoustic model and the server 30 and the computer PC2. The server 30 includes a parameter acquisition unit 55, a parameter presentation unit 56, a score analysis unit 57, and a speech synthesis unit 60. The parameter acquisition unit 55 and the parameter presentation unit 56 are provided to enable the user to adjust the acoustic parameters. Details will be described later. The score analysis unit 57 analyzes a score representing a singing voice to be synthesized and outputs a phoneme string (combination of voiced pitch and phoneme) to be synthesized. The speech synthesizer 60 receives the outputs from the parameter acquisition unit 55 and the score analysis unit 57 and synthesizes speech.

  The voice synthesis unit 60 includes a parameter generation unit 61, a sound source generation unit 63, a synthesis filter 65, and the like. The parameter generation unit 61 receives the output of the score analysis unit 57 and generates various acoustic parameters such as a fundamental frequency, a mel cepstrum parameter, and a song expression parameter from the learned acoustic model. The sound source generation unit 63 receives parameters related to the pitch, such as the fundamental frequency, pitch vibrato, shackle, and fist, and generates the excitation source parameter along the time axis. The synthesizing filter 65 is a filter that synthesizes speech mainly by a mel cepstrum. As such a filter, for example, an MLSA filter is known. Of the singing expression parameters, parameters not involved in sound source generation by the sound source generation unit 63 are input to the synthesis filter 65 as part of the mel cepstrum.

A-3. Singing voice synthesis routine:
FIG. 8 is a flowchart showing the procedure of the singing voice synthesis routine. The singing voice synthesis routine is a process for synthesizing a singing voice using the acoustic model stored in the acoustic model storage unit 50. The singing voice synthesis routine is started in response to a user instruction, and is executed by the cooperation of the server 30 and the computer PC2.

  First, the server 30 analyzes the input score (step S210). The user inputs the musical score of the song to be synthesized from the computer PC2. Specifically, for example, a piano roll screen is displayed on the display unit 53 of the computer PC2. The user inputs a note and a sound length (that is, a melody line) by tracing a predetermined portion of the piano roll screen using the pointing device 52. In addition, the user inputs lyrics corresponding to each note by assigning a character string to the note input on the piano roll screen using the keyboard 51. The melody line may be input using a keyboard type keyboard. The score input may be replaced by reading a score file in a predetermined format. The server 30 acquires the score inputted in this way, and the score analysis unit 57 analyzes the acquired score. By the analysis by the score analysis unit 57, context-dependent phoneme string data (FIG. 2) corresponding to the input score is generated. The generated phoneme string data includes pitch information.

  Next, the server 30 generates a configuration parameter series (step S220). Specifically, the generation of the configuration parameter series can be performed by the following procedures a1 to a4, for example.

(A1) The parameter generation unit 61 acquires an acoustic model corresponding to each phoneme sequence obtained in step S210 from the acoustic models stored in the acoustic model storage unit 50.
(A2) The parameter generation unit 61 sets the boundary of each phoneme on the time axis (that is, the separation of each phoneme) according to the state duration model among the acoustic models acquired in step a1. The parameter generation unit 61 corrects the boundaries of each phoneme set on the time axis back and forth according to the timing model of the acoustic models acquired in step a1.
(A3) The parameter generation unit 61 arranges a frame storing various acoustic parameters (FIG. 4) included in the acoustic model acquired in the procedure a1 on the time axis. That is, “frame” means a set of various acoustic parameters per unit time. Note that the parameter generator 61 follows the boundary of each phoneme set in step a2 when arranging the frame on the time axis.

(A4) The parameter generation unit 61 generates a configuration parameter series. Specifically, the parameter generation unit 61 changes the parameter value between phonemes in consideration of the dynamic feature amount for one of the acoustic parameters (for example, the fundamental frequency) included in the frame arranged in step a3. Generate a smoothed parameter series. In other words, the “parameter series” is information representing changes in parameter values on the time axis. The audio parameter generation unit 61 generates the parameter series respectively corresponding to all of the acoustic parameters by repeating the above-described processing for all of the acoustic parameters included in the frame arranged in the procedure a3. Thus, a spectrum parameter series, a fundamental frequency parameter series, a singing expression parameter series, and the like are generated. The parameter series of singing expression parameters includes, for example, a pitch vibrato parameter series, a sound loudness vibrato parameter series, a shawl parameter series, a fist parameter series, an attack release parameter series, and the like. Hereinafter, a set of parameter series for acoustic parameters is also referred to as a “configuration parameter series”. The configuration parameter series functions as a “first parameter series”.

  Next, the server 30 generates a pitch parameter series and a loudness parameter series from the constituent parameter series (step S230). Specifically, the parameter generation unit 61 adds the parameter series of the parameters related to the pitch (for example, fundamental frequency, pitch vibrato, shackle, fist, etc.) among the constituent parameter series, thereby adding the pitch. The parameter series of is generated. In addition, the parameter generation unit 61 adds the parameter series of the parameters related to the sound volume (for example, spectrum parameter, sound volume vibrato, attack / release, etc.) among the constituent parameter series, thereby generating the sound. A parameter series of the size of is generated. That is, the pitch parameter series and the loudness parameter series are parameter series obtained by adding the elements of the constituent parameter series. Therefore, the pitch parameter series and the loudness parameter series are collectively referred to as “addition parameter series”. The addition parameter series functions as a “second parameter series”. Note that “addition” in step S230 includes both simply adding each listed parameter and adding each listed parameter in a logarithmic domain.

  FIG. 9 is a diagram illustrating an example of the editing screen. In FIG. 9, the illustration of the piano roll is omitted. In the singing voice synthesis routine (FIG. 8), the parameter presentation unit 56 of the server 30 generates a screen W1 representing each generated parameter series and displays it on the display unit 53 of the computer PC2 (step S240).

  In the present embodiment shown in FIG. 9, the screen W1 is configured as a screen suitable for editing the addition parameter series. The screen W1 includes a toolbar TB, two main tabs MT1 and MT2, a main window MW, four sub tabs ST1 to ST4, and a sub window SW.

  The tool bar TB includes various tools used for editing each parameter series and various tools (or buttons) used for operating the screen W1. For example, in the example of FIG. 9, a pen tool T1, an eraser tool T2, a selection tool T3, and an edit end button Tn are included. The pen tool T1 is a tool for drawing the shape of each parameter series by dragging the pointing device 52. The eraser tool T2 is a tool for canceling the drawing contents. The selection tool T3 is a tool for selecting an arbitrary point (or range) in the parameter series. The editing end button Tn is a button for instructing the server 30 to finish the editing operation and synthesize a singing voice using each parameter series displayed on the screen W1. The tool bar TB can include various tools (buttons) other than those illustrated.

  The main tabs MT1 and MT2 are used for designating an addition parameter series to be displayed on the main window MW. For example, in the example of FIG. 9, a pitch (pitch) is assigned to the main tab MT1, and a loudness (volume) is assigned to the main tab MT2.

  The main window MW is used to display the addition parameter series selected by the main tabs MT1 and MT2. In the example of FIG. 9, when the main tab MT1 (pitch (pitch)) is selected, a pitch parameter series is displayed in the main window MW. When the main tab MT2 (sound volume (volume)) is selected, a sound volume parameter series is displayed in the main window MW. As the pitch parameter series and the loudness parameter series, the one generated in step S230 can be used. In either case, a piano roll screen showing the melody line and lyrics of the score is displayed behind the parameter series in the main window MW. As the melody line and the lyrics, those acquired in step S210 can be used.

  The parameter value in the parameter series displayed on the main window MW is changed using the keyboard 51 or the pointing device 52 according to the editing method (for example, drawing, deleting, etc.) with the tool selected on the toolbar TB. Can do.

  The sub tabs ST1 to ST4 are used for designating a configuration parameter series to be displayed on the sub window SW. For example, in the example of FIG. 9, subtab ST1 has basic pitch data (FIG. 9: basic), subtab ST2 has a vibrato period (FIG. 9: Vf), and subtab ST3 has a vibrato amplitude (FIG. 9: Va). ) And scissors (S) are respectively assigned to the subtabs ST4. Note that the items displayed on each sub tab and the number of all sub tabs vary depending on the item selected in the main tabs MT1 and MT2. For example, when the main tab MT2 (sound volume (volume)) is selected, the subtab ST1 has basic sound volume data, the subtab ST2 has a sound volume vibrato period, and the subtab ST3 has a sound volume. An attack / release is assigned to each of the amplitudes of the vibrato and the sub tab ST4.

  The sub window SW is used to display the configuration parameter series selected by the sub tabs ST1 to ST4. In the example of this embodiment shown in FIG. 9, when basic data (basic) is selected in the sub tab ST1, a parameter series of pitch basic data is displayed in the sub window SW. Basic data means a pitch (pitch) to which no singing expression such as vibrato is added. Similarly, when the vibrato period (Vf) is selected in the sub-tab ST2, the sub-window SW has a pitch vibrato period parameter series, and the sub-tab ST3 has the vibrato amplitude (Va) selected. Is the parameter series of the amplitude of the vibrato of the pitch, and when the chatter (S) is selected in the subtab ST4, the parameters of the chatter are displayed in the subwindow SW. Each of the parameter series described above can use the one generated in step S220. Note that the volume parameter series is often modeled as part of the spectral parameters, and can be obtained from the spectral parameter series (step S220).

  In the example of this embodiment shown in FIG. 9, the parameter series displayed in the subwindow SW is used only for display and cannot be changed.

  In the main window MW and the sub window SW, the horizontal axis represents the front-rear direction on the time axis, and the time units of both are the same. In addition, the vertical axes of the main window MW and the subwindow SW change depending on the parameter series displayed in the window. For example, in the case of a window that displays the pitch, the vertical axis is the pitch of the sound (for example, logarithmic Hz), and in the case of the window that displays the volume of the sound, the vertical axis is the volume of the sound (for example, db). Become.

  FIG. 10 is a diagram illustrating a first example of a parameter series editing method. In FIG. 10, the illustration of the piano roll is omitted. As described above, the user can edit the parameter series (pitch parameter series P1 in the example of FIG. 10) displayed in the main window MW using the input device of the computer PC2. For example, as shown by an arrow with (1) in FIG. 10, the user selects a pen tool T1 and then draws a desired trajectory by a drag operation of the pointing device 52. Each value can be moved up, down, and time back and forth.

  In the singing voice synthesis routine (FIG. 8), the server 30 determines whether or not the editing has been completed (step S250). Specifically, the server 30 determines that the editing has been completed when the pressing of the editing end button Tn on the screen W1 is acquired. When editing is completed (step S250: YES), the server 30 shifts the process to step S280. Details will be described later.

  If the editing has not ended (step S250: NO), that is, if the pressing of the editing end button Tn has not been detected, the server 30 shifts the process to step S260. The parameter acquisition unit 55 of the server 30 acquires the editing content performed in the main window MW of the screen W1 (step S260). The editing content can be specified by, for example, the amount of change from the original parameter series.

  Next, the parameter generation unit 61 of the server 30 changes the configuration parameter series in accordance with the editing content acquired in step S260 (specifically, the amount of change from the original parameter series) (step S270). Thereafter, the process proceeds to step S230. As a result, the addition parameter series is regenerated based on the changed configuration parameter series (step S230), and the regenerated addition parameter series is displayed on the main window MW of the screen W1 and the regenerated configuration is displayed. The parameter series is displayed on the subwindow SW of the screen W1 (step S240). In step S240, the trajectory of the addition parameter series and the constituent parameter series (that is, the trajectory of each parameter series based on the default value) displayed when step S240 is executed for the first time is left in broken line notation, color changed notation, or the like. It is preferable to keep it.

  This process will be described with reference to a specific example of FIG. After the user has edited the pitch parameter series P1 (FIG. 10: arrow with (1)), the parameter acquisition unit 55 acquires the edited content for the pitch parameter series P1 (step S260). ). Thereafter, the parameter generation unit 61 changes the configuration parameter series according to the edited content (step S270). At this time, in step S270, at least a part corresponding to the edited content is selected from at least all the constituent parameter series related to the pitch (basic data, pitch vibrato period, pitch vibrato amplitude, squeak, fist). Each value of the configuration parameter series is changed. Further, the pitch parameter series P1 is regenerated based on the changed configuration parameter series. Finally, the regenerated pitch parameter series P1 is displayed in the main window MW of the screen W1, and among the regenerated constituent parameter series, the parameter series P11 of the basic data selected in the sub tab is It is displayed on the subwindow SW of the screen W1 (step S240). In this way, as indicated by the arrow with (2) in FIG. 10, the edited content of the pitch parameter series P1 is reflected in the parameter series P11 of the basic data.

  As described above, in the singing voice synthesis routine (FIG. 8), the generation of the addition parameter series (step S230), display (step S240), acquisition of the edited content (step S260), and reflection of the edited content (step S270). By repeating the above, on the screen W1 on the computer PC2, the edited content for the parameter series (pitch parameter series P1 in the examples of FIGS. 9 and 10) displayed in the main window MW is displayed in the subwindow SW. The reflected parameter series (in the example of FIGS. 9 and 10, the basic data parameter series P11) is reflected in real time.

  When editing is completed in the singing voice synthesis routine (FIG. 8) (step S250: YES), the server 30 synthesizes and outputs the singing voice (step S280). Specifically, first, the parameter acquisition unit 55 of the server 30 acquires the latest configuration parameter series and addition parameter series on the screen W1. Next, the parameter generation unit 61 sets the sound source generation unit 63 and the synthesis filter 65 using the acquired parameter series. Thereafter, the singing voice processing apparatus 100 outputs the singing voice synthesized using the parameter series set in the sound source generation unit 63 and the synthesis filter 65 to the speaker 70. As a result, the singing voice synthesized using each parameter series displayed on the screen W1 is reproduced from the speaker 70. In the example of FIG. 8, the post-processing is ended. However, when the user does not desire the singing voice reproduced from the speaker 70, the user can continue the editing operation by changing the processing to step S <b> 230. Good.

  FIG. 11 is a diagram illustrating a second example of a parameter series editing method. In FIG. 11, the illustration of the piano roll is omitted. As shown by the arrow with (1) in FIG. 11, after the user selects the selection tool T3, the user can drag the pointing device 52 in the addition parameter series (pitch parameter series in the example in the figure). Select any range. Thereafter, the user drags an arbitrary portion within the selection range left and right. Thereby, the user can change the cycle of the addition parameter series. The edited content is also reflected in the configuration parameter series by the above-described processing, and the configuration parameter series (the parameter series of the pitch vibrato period in the example in the figure) is also changed.

  FIG. 12 is a diagram illustrating a third example of the parameter series editing method. In FIG. 12, the illustration of the piano roll is omitted. As shown by the arrow with (1) in FIG. 12, the user selects the selection tool T3 and then drags the pointing device 52 in the addition parameter series (pitch parameter series in the example in the figure). Select any range. Thereafter, the user drags an arbitrary part within the selection range up and down. As a result, the user can change the amplitude of the addition parameter series. The edited content is also reflected in the constituent parameter series by the above-described processing, and the constituent parameter series (the parameter series of the pitch vibrato amplitude in the example in the figure) is also changed.

  FIG. 13 is a diagram illustrating a fourth example of the parameter series editing method. In FIG. 13, the illustration of the piano roll is omitted. As shown by the arrow with (1) in FIG. 13, the user selects the selection tool T3 and then clicks on the pointing device 52, and then in the added parameter series (pitch parameter series in the example in the figure). Select an arbitrary point. Thereafter, the user drags the selected point left and right. Thereby, the user can change the cycle of the addition parameter series. The edited content is also reflected in the constituent parameter series by the above-described processing, and the constituent parameter series (the parameter series of the pitch vibrato amplitude in the example in the figure) is also changed.

  FIG. 14 is a diagram illustrating a fifth example of the parameter series editing method. In the example of the present embodiment, the parameter sequence of “shearing” constituting the pitch parameter sequence is modeled in note units as described above. For this reason, when the sub tab ST4 (shaking) is selected, as shown in the figure, the sub window SW displays, for each note, the height, length, and steepness values corresponding to the beginning of the note, The height, length, and steepness values corresponding to the end of the note are displayed. The same applies to the “attack release” that constitutes the parameter series of sound volume.

  After selecting the selection tool T3, the user selects an arbitrary point corresponding to the note desired to be edited in the addition parameter series by clicking the pointing device 52. Thereafter, the user can change the height assigned to the note by dragging the selected point in the vertical direction (FIG. 14: D1 direction). Similarly, the user can change the length assigned to the note by dragging the selected point in the left-right direction (FIG. 14: D2 direction). 14: D3 direction), the steepness assigned to the note can be changed. The edited content is also reflected in the configuration parameter series by the above-described processing, and each value in the configuration parameter (corresponding note scoring parameter in the example in the figure) is also changed.

  Instead of the operation of selecting the toolbar TB, an operation of pressing a predetermined key assigned in advance to the keyboard 51 may be employed. In this case, for example, the pointing device 52 that is operated without pressing the key is operated by the selection tool T3, the pointing device 52 that is operated while the Ctrl key is pressed is operated by the pen tool T1, and the pointing device that is pressed while the Alt key is pressed. The operation 52 can be properly used with the eraser tool T2, etc., and the operability can be improved. Further, the operation of selecting the sub tabs ST1 to ST4 may be realized by an operation of pressing a predetermined key previously assigned to the keyboard 51. In this case, for example, the sub tab ST1 may be selected by pressing the F1 key, and the sub tab ST2 may be selected by pressing the F2 key.

  In addition, the designation of which configuration parameter series the editing contents in the main window MW should be reflected may be realized by pressing a predetermined key assigned in advance to the keyboard 51. In this case, an operation for selecting the sub tabs ST1 to ST4 is not necessary. For example, when the user moves the pitch parameter series of the main window MW left and right while pressing the Ctrl key, the operation is acquired as an edit for the pitch vibrato cycle, and when the user moves up and down, the operation Is obtained as an edit to the amplitude of the vibrato of the pitch. Further, for example, when the user operates a curve on the parameter series of the pitch of the main window MW without pressing a key, the operation is acquired as editing for the parameter series of basic data. In this case, the parameter series displayed in the sub window SW (and the sub tabs ST1 to ST4 in the active state) may be automatically switched according to the user's operation.

  Further, when the user places the pointing device 52 on a vibrato part of the pitch parameter series displayed on the main window MW, for example, a guide image such as an arrow or an icon is displayed. The configuration parameter series may be edited in accordance with the guidance image (for example, dragging the arrow image is editing with respect to the pitch or vibrato period of the pitch).

  As described above, according to the speech synthesizer (singing voice processing apparatus 100) of the above embodiment, the user can set each value in the parameter series related to the pitch (pitch) and the parameter series related to the loudness (volume). Among these values, at least one of the values presented in an editable manner by the parameter presentation unit 56 (specifically, displayed on the screen W1) is directly used, for example, by using the various editing methods described above. It can be edited to a desired value. As described above, the user can change the pitch and volume by an intuitive operation. Therefore, according to the speech synthesizer of the above embodiment, the operability when editing at least one of the pitch and the volume. Can be improved.

  In addition, according to the speech synthesizer (singing voice processing apparatus 100) of the above embodiment, the user uses the screen W1 to display both the first and second parameter series (that is, the constituent parameter series and the addition parameter series). Both) can be confirmed, and the first parameter series (configuration parameter series) after regeneration can be confirmed in real time, so that convenience for the user can be improved.

  Furthermore, according to the speech synthesizer (singing voice processing device 100) of the above embodiment, the user uses the screen W1 to display the second parameter series (addition parameter series) by the main window MW and the first by the subwindow SW. Since the parameter series (configuration parameter series) can be confirmed at the same time, the convenience for the user can be further improved.

  Furthermore, according to the speech synthesis apparatus (singing voice processing apparatus 100) of the above embodiment, the first parameter series (configuration parameter series) is displayed in the main window MW in which the second parameter series (addition parameter series) is displayed. The display area occupied on the screen is large (in other words, the visibility is high) compared to the sub window SW. Therefore, the user can more easily confirm the second parameter series (addition parameter series) between the first and second parameter series (that is, the constituent parameter series and the addition parameter series) presented at the same time. . Therefore, the user can intuitively understand that the second parameter series (addition parameter series) is an editing target.

A-4. Modified editing screen:
The editing screen described in the above embodiment can be variously modified as exemplified below. The deformations b1 to b8 may be employed alone or in combination. Moreover, you may switch the presence or absence of adoption of modification b1-b8 by the designation | designated from a user.

(B1) Modification 1: Bidirectional editing using main window and subwindow In the screen W1 described above, the parameter series displayed in each subwindow SW is used only for display, and the parameter value cannot be changed. However, the parameter values displayed on each sub-window SW may be changed using the keyboard 51 and the pointing device 52 as in the main window MW. In this case, designation of the window to be edited (main window MW / subwindow SW) can be realized by key assignment. For example, an operation of the pointing device 52 that is performed without pressing a key can be an operation for the main window MW, and an operation of the pointing device 52 that is performed while the Shift key is pressed can be an operation for a sub window SW. Also, the designation of which item of the subtabs ST1 to ST4 is to be edited can be realized by pressing a predetermined key assigned in advance to the keyboard 51 as described above.

  Note that the editing contents for the parameter series displayed in the sub-window SW are implemented by repeating the generation of the addition parameter series (step S230) to the reflection of the editing contents (step S270) in the singing voice synthesis routine (FIG. 8). Similar to the embodiment, the addition parameter series displayed in the main window MW and the configuration parameter series displayed in the sub window SW are reflected in real time. According to the first modification, the user can edit the addition parameter series and the configuration parameter series in both directions using the main window MW and the sub window SW, and check the result in real time. As a result, the convenience for the user can be further improved.

(B2) Modification 2: Editing of only one addition parameter series One of the main tabs MT1 and MT2 described above may be omitted. For example, when the main tab MT2 is omitted, the processing related to the sound volume parameter series in the singing voice synthesis routine (FIG. 8) may be omitted. For example, when the main tab MT1 is omitted, the processing related to the pitch parameter series in the speech synthesis routine may be omitted. According to the modification 2, the process in the speech synthesis routine can be simplified.

(B3) Modification 3: Editing Three or More Addition Parameter Series The main tabs MT1 and MT2 described above are merely examples, and various modes can be adopted. For example, a new main tab as shown below may be provided.

“Gender” tab: When the gender tab is selected, the addition parameter is a gender parameter series. The gender parameter series is a parameter series that indicates whether the voice is childish or adult. The components of the gender parameter series (constituent parameter series) include, for example, the parameter series of the basic data of gender parameters and the height, length, and steepness for the beginning and end of the notes modeled in note units. It is a parameter.

“Voice / unvoiced ratio” tab: When the voiced / unvoiced ratio tab is selected, the addition parameter is a voiced / unvoiced ratio parameter series. The voiced / unvoiced ratio parameter series is a parameter series representing the degree of voice fading. In order to realize the voiced / unvoiced ratio parameter series, the learning unit 40 of the server 30 is further configured to include a “voiced / unvoiced ratio extracting unit”. In step S140 of the singing voice synthesis preparation routine (FIG. 3), the voiced / unvoiced ratio extraction unit extracts the voiced / unvoiced ratio included in the singing voice and its derivative. As a result, in addition to the parameters described above, the various parameters extracted by the analysis (FIG. 4) further include, as voiced / unvoiced ratio information, a static feature quantity of voiced / unvoiced ratio, its first derivative, And second derivative. Further, the statistically learned acoustic model (FIG. 7) includes a voiced / unvoiced ratio decision tree in addition to the above-described binary trees. The constituent elements of the voiced / unvoiced ratio parameter series (constituent parameter series) are the height, length, and steepness of the basic data series of the voiced / unvoiced ratio and the beginning and end of the notes modeled in note units. Parameter.

  According to the third modification, the user can edit three or more addition parameter series by switching each main tab described above, and can check the result in real time. As a result, the convenience for the user can be further improved.

(B4) Modification 4: Omission of subwindow In the screen W1, the subtabs ST1 to ST4 and the subwindow SW may be omitted. In this case, display / redisplay of the configuration parameter series corresponding to the sub-window SW can be omitted. In addition, display / non-display of the sub tabs ST1 to ST4 and the sub window SW may be switched on the screen W1. According to the modification 4, the editing screen can be simplified.

(B5) Modification 5: Omission of subwindow In the screen W1, the subwindow SW may be omitted, and the configuration parameter series corresponding to the subwindow SW may all be displayed in the main window MW. In this case, all items of the piano roll, the addition parameter, and the configuration parameter are displayed on the main window MW. In this case, it is preferable that the visibility of the addition parameter is higher in the main window MW than the configuration parameter. As a method of making a difference in visibility, for example, a method of changing the thickness and color of a drawn waveform line can be adopted. According to the modification 5, the user can confirm all items simultaneously in one window.

(B6) Modification 6: Different Screen Display of Main Window and Subwindow Main window MW and subwindow SW may be displayed as separate screens. According to the modification 7, the user can perform an editing operation while referring to only a necessary screen.

(B7) Modification 7: Simultaneous display in sub-window In the screen W1, the sub-tabs ST1 to ST4 may be omitted. In this case, for example, all the configuration parameters are displayed in the sub window SW. According to the modification 7, the user can confirm all items simultaneously in one sub-window SW. Further, for example, the subtabs ST1 to ST4 may be omitted, and four subwindows SW respectively corresponding to the subtabs ST1 to ST4 may be displayed side by side.

(B8) Modification 8: Display of Configuration Parameters in Main Window In the screen W1 described above, the main window MW displays the piano roll and the addition parameter series. However, a part of the configuration parameter series may be further displayed in the main window MW. In this case, for example, the constituent parameter series that fluctuates following the editing of the addition parameter series (for example, the parameter series of the pitch vibrato period when the pitch parameter series period is edited) is displayed in the main window. Display on MW. In order to improve visibility for the user, it is preferable to distinguish between the display mode of the addition parameter series and the display mode of the configuration parameter series in the main window WM. For example, it is preferable that the addition parameter series is a solid line (or thick line, dark color) and the constituent parameter series is a broken line (or thin line, light color). According to the modification 8, the user can confirm the configuration parameter series affected by the change of the addition parameter series in the main window MW.

B. Variations:
In the above embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. Good. In addition, the following modifications are possible.

・ Modification 1:
In the said embodiment, the structure of the singing voice processing apparatus was illustrated. However, the configuration of the singing voice processing apparatus in the above embodiment is merely an example, and any aspect can be adopted. For example, the deformation | transformation which abbreviate | omits a part of the component, adds a further component, or changes a part of component is possible. For example, each function of the above-described singing voice processing device may be realized by cooperation of a plurality of devices. For example, the acoustic model may be distributed and stored in a plurality of devices.

  In the above embodiment, the speech synthesis employing the HMM method has been described. However, the present invention can be applied to speech synthesis using any method. Specifically, for example, a DNN (Deep Neural Network) method may be adopted instead of the HMM method, and a waveform connection method may be adopted instead of the HMM method.

Modification 2
In the singing voice synthesis routine (FIG. 8) of the above embodiment, a part of each parameter series exemplified as the constituent parameter series may be omitted, or another parameter series may be added. For example, the release (singing expression in which the volume decreases at the end of a note) may be omitted for “attack” which is one of the constituent parameter series. Moreover, you may employ | adopt either singing expression of an attack and release. Similarly, one of the pitch parameter series and the loudness parameter series exemplified as the addition parameter series may be omitted, or another parameter series may be added. For example, the above-described gender parameter series or voiced / unvoiced ratio parameter series may be employed instead of the pitch / sound volume parameter series.

  In the singing voice synthesizing routine (FIG. 8) of the above embodiment, the pitch parameter series and the loudness parameter series are targeted for generation and editing. Here, the “pitch parameter series” includes not only the parameter series for the pitch itself but also the overall parameter series for the pitch (for example, the parameter series for the logarithm of the pitch, the pitch of the pitch). Parameter series for variables proportional to Similarly, the “sound loudness parameter” may include not only a parameter series relating to the loudness itself but also a general parameter series relating to the loudness.

  In the singing voice synthesizing routine (FIG. 8) of the above embodiment, an example of an editing method of each parameter series using a keyboard or a pointing device is shown. However, each parameter series can be edited by an arbitrary method. For example, as an input device, a touch pad, a tablet, a microphone, wireless communication, or the like can be employed instead of the keyboard or the pointing device or together with the keyboard or the pointing device. When a microphone is used, each parameter series can be edited using voice commands. When wireless communication is used, each parameter series can be edited using a reception command. Further, the use method (drag, click) of the pointing device exemplified in the above embodiment can be changed to any method. For example, a mouse wheel press or a shortcut assigned to the mouse may be used.

・ Modification 3:
Although several embodiments and modifications of the present invention have been described above, the present invention is not limited to these embodiments, and can of course be implemented in various modes within the scope not changing the gist of the present invention. It is. For example, it may be implemented only as a singing voice synthesizing device that does not have a singing voice learning function. Also, the singing voice is not limited to that based on the equal temperament, but may be one that follows a specific temperament such as folk music. For example, the present invention may be applied to synthesis of Japanese music, kyoku, statement, sutra, singing voice that follows the pre-equilibrium temperament such as Gregorian chant in Europe.

DESCRIPTION OF SYMBOLS 10 ... Musical score input part 20 ... Voice input part 30 ... Server 31 ... Hard disk 33 ... Musical score analysis part 40 ... Learning part 41 ... F0 extraction part 43 ... SP extraction part 44 ... Singing P extraction part 45 ... HMM learning part 50 ... Acoustic model Storage unit 51 ... Keyboard 52 ... Pointing device 53 ... Display unit 55 ... Parameter acquisition unit 56 ... Parameter presentation unit 57 ... Score analysis unit 60 ... Speech synthesis unit 61 ... Parameter generation unit 63 ... Sound source generation unit 65 ... Synthesis filter 100 ... Singing voice Processing equipment

Claims (12)

A speech synthesizer,
A storage unit for storing in advance an acoustic model including acoustic parameters learned by a statistical method;
A parameter generator,
A first parameter series generated from the stored acoustic parameters, the first parameter series representing a change in acoustic parameter values corresponding to the input score;
A second parameter series generated from the first parameter series;
A parameter generator for generating
A parameter presenting unit that presents at least the second parameter series to the user in an editable manner;
A synthesis unit that synthesizes a singing voice using at least one of each acoustic parameter value included in the first parameter series and each acoustic parameter value included in the second parameter series;
With
The parameter generator is
A speech synthesizer that regenerates the first parameter series using the second parameter series after editing when the second parameter series is edited. The speech synthesizer according to claim 1,
The parameter presentation unit
In addition to the second parameter series, further presenting the first parameter series to the user,
A speech synthesizer that presents the regenerated first parameter series to a user when the first parameter series is regenerated. The speech synthesizer according to claim 2,
The parameter presentation unit
A speech synthesizer in which a user presents the first parameter series and the second parameter series in a manner that can be viewed simultaneously. The speech synthesizer according to claim 3,
The parameter presentation unit
A speech synthesizer that presents the first parameter series and the second parameter series in a state in which the visibility of the second parameter series is higher than that of the first parameter series. The speech synthesizer according to any one of claims 1 to 4,
The parameter presentation unit presents the first parameter series to the user in an editable manner in addition to the second parameter series,
The parameter generation unit further regenerates the second parameter series using the edited first parameter series when the first parameter series is edited,
The parameter presenting unit further presents the second parameter series after regeneration to the user when the second parameter series is regenerated. The speech synthesizer according to any one of claims 1 to 4, further comprising:
A parameter acquisition unit for acquiring the content of the edit;
The parameter acquisition unit
An operation according to the first operation method assigned in advance is acquired as an editing content for a certain parameter series included in the first parameter series,
An operation according to a second operation method that is assigned in advance and is different from the first operation method is acquired as editing content for another parameter sequence included in the first parameter sequence. A speech synthesizer. The speech synthesizer according to claim 5, further comprising:
A parameter acquisition unit for acquiring the content of the edit;
The parameter acquisition unit
An operation according to the third operation method assigned in advance is acquired as an editing content for the first parameter series,
A speech synthesizer that acquires a fourth operation method that is assigned in advance and that is a fourth operation method that is different from the third operation method, as edit contents for the second parameter series. The speech synthesizer according to any one of claims 2 to 7,
The first parameter series includes at least
A parameter series of pitches representing changes in pitch parameter values,
Singing expression parameter series representing changes in parameter values of singing expression,
Contains
The parameter presentation unit
A speech synthesizer that presents all parameter series included in the first parameter series to a user. The speech synthesizer according to claim 8, wherein
The speech synthesizer in which the singing expression includes at least one of vibrato, shackle, fist, attack, and release. The speech synthesizer according to any one of claims 1 to 9,
An image display device including a pointing device is connected to the speech synthesizer,
The parameter presentation unit
A speech synthesizer that uses a graphical user interface and displays the second parameter series on the image display device in a manner that allows operation by the pointing device. The speech synthesizer according to any one of claims 1 to 10,
The second parameter series includes at least one of a parameter series related to pitch, a parameter series related to loudness, a parameter series related to voice age, and a parameter series related to sound blur. A speech synthesizer. A speech synthesis method,
A step of generating a first parameter series representing a change in an acoustic parameter value corresponding to an input musical score, from acoustic parameters included in an acoustic model stored in advance and learned by a statistical method When,
Generating a second parameter series from the first parameter series;
Presenting at least the second parameter series to the user in an editable manner;
Synthesizing a singing voice using at least one of each acoustic parameter value included in the first parameter series and each acoustic parameter value included in the second parameter series;
With
In the step of generating the first parameter series,
A speech synthesis method in which, when the second parameter series is edited, the first parameter series is regenerated using the edited second parameter series.

Images