JP6822075B2 - Speech synthesis method - Google Patents

Description

The present invention relates to speech synthesis.

A technique for synthesizing voice such as singing is known. In order to generate a more expressive singing voice, it is attempted not only to output the voice of the given lyrics in a given scale but also to give a musical singing expression to this voice. Patent Document 1 discloses a technique for converting voice quality by adjusting two voice signals so that the frequency bands of the tuning components are located close to each other.

Japanese Unexamined Patent Publication No. 2014-2338

In the technique described in Patent Document 1, the singing expression given to the synthetic singing may not be sufficient. On the other hand, the present invention provides a technique for giving richer speech expression.

The present invention includes a step of acquiring the pitch of a synthetic sound, a step of acquiring a pitch representing a piece of speech expression given to the synthetic sound, a pitch representing the piece, and the synthetic sound. Provided is a speech synthesis method including a step of shifting the pitch of the element piece and a step of adding the element piece to which the pitch is shifted to the synthesized sound so as to match the pitch of.

The element piece has a characteristic portion having a large fluctuation and a non-characteristic portion having a small fluctuation on the time axis, and the pitch of the element piece may be determined based on the non-characteristic portion.

In the element piece, the pitch of the element piece is shifted so that the pitch of the representative of the element piece and the pitch of the synthetic sound match while maintaining the relative pitch change in the characteristic portion. May be done.

According to the present invention, a richer speech expression can be provided.

The figure which illustrates GUI related to the related technology. The figure which shows the concept of singing expression addition which concerns on one Embodiment. The figure which illustrates the functional structure of the voice synthesis apparatus 1 which concerns on one Embodiment. The figure which illustrates the hardware configuration of the voice synthesizer 1. The schematic diagram which shows the structure of database 10. The figure which illustrates the reference time in the singing expression of the attack standard. The figure which illustrates the reference time in the singing expression of the release reference. The figure which illustrates the functional structure of a synthesizer 20. The figure which illustrates the mapping function in the example which the time length of the element piece of a singing expression is short. The figure which illustrates the mapping function in the example which the time length of the element piece of a singing expression is long. The figure which illustrates the relationship between the spectrum envelope and the spectrum envelope outline. The figure which exemplifies the process of shifting the fundamental frequency of the element piece of a singing expression. The figure which illustrates the functional structure of the synthesis means 24 for synthesis in a frequency domain. A sequence chart illustrating the operation of the synthesizer 20. The figure which illustrates the functional structure of the synthesis means 24 for synthesis in the time domain. The figure which illustrates the functional structure of the UI unit 30. The figure which illustrates the GUI used in the UI part 30. The figure which illustrates the UI which selects a singing expression. The figure which shows another example of the UI which selects a singing expression. An example of a table that makes the angle of rotation of the dial correspond to the morphing coefficient. Another example of a UI for editing parameters related to singing expressions.

1. 1. Speech synthesis technology Various techniques for speech synthesis are known. Of the voices, those with changes in scale and rhythm are called singing (singing voice). As singing synthesis, elemental connection type singing synthesis and statistical singing synthesis are known. In the piece-connected singing synthesis, a database containing a large number of singing pieces is used. Singing elements (an example of audio elements) are mainly classified by phonemes (single phonemes or phoneme chains). In singing composition, these singing elements are connected after adjusting the fundamental frequency, timing, and continuation length according to the musical score information. The singing element pieces used for the element piece connection type singing composition are required to have the same sound quality as possible over all the phonemes registered in the database. This is because if the sound quality is not constant, unnatural voice fluctuations will occur when singing is synthesized. In addition, the part of the dynamic acoustic changes contained in these pieces that corresponds to the singing expression (an example of the audio expression) needs to be processed so that it does not appear at the time of synthesis. This is because the singing expression should be given to the singing depending on the musical context, and should not be directly associated with the type of phonology. If the same singing expression is always expressed for a particular phoneme, the resulting synthetic singing will be unnatural. Therefore, in elemental connection type singing synthesis, for example, changes in fundamental frequency and volume are generated based on musical score information and predetermined rules, rather than directly using what is contained in the singing element. Changes in fundamental frequency and volume are used. If the database contains singing pieces that correspond to all combinations of phonemes and singing expressions, singing pieces that correspond to both phonological pieces that match the musical score information and singing pieces that are natural to the musical context can be obtained. It becomes possible to select. However, it takes a huge amount of time and effort to record a singing element corresponding to every singing expression for every phoneme, and the capacity of the database becomes enormous. Also, since the number of combinations of pieces increases explosively with respect to the number of pieces, it is difficult to guarantee that the singing will not be unnatural for any connection between pieces.

On the other hand, in statistical singing synthesis, the relationship between musical score information and the acoustic characteristics of singing is learned in advance as a statistical model using a large amount of training data. At the time of composition, the most plausible acoustic features are estimated from the input score information, and the singing is synthesized using it. Statistical singing synthesis has the advantage that it is possible to learn a statistical model that includes various singing expressions by constructing training data for each of various singing styles. However, there are two main problems with statistical singing synthesis. The first problem is oversmoothing. Since the process of learning a statistical model from a large number of training data essentially involves data averaging and dimensionality reduction, the synthetically output acoustic features are inevitably more feature-distributed than a normal single song. Becomes smaller. As a result, the expressiveness and realism of the synthetic sound are impaired. The second problem is that the types of acoustic features that can be trained in statistical models are limited. In particular, phase information has a cyclic range, so statistical modeling is difficult. For example, the phase relationship between tonal components or specific tonal components and the components existing in the vicinity and their temporal fluctuations can be determined. It is difficult to model properly. However, in reality, it is necessary to appropriately use phase information in order to synthesize expressive singing including muddy voice and hoarseness.

VQM (Voice Quality Modification) described in Patent Document 1 is known as a technique for synthesizing various voice qualities in singing synthesis. In VQM, a first voice signal having a voice quality corresponding to a certain singing expression and a second voice signal obtained by singing synthesis are used. The second voice signal may be a piece-connected singing composition or a statistical singing composition. By using these two audio signals, it is possible to synthesize a singing including phase information. As a result, it is possible to synthesize a singing that is more realistic and expressive than a normal singing synthesis. However, in this technique, it is not clear how to reflect the time change of the acoustic characteristics of the first speech signal in the singing synthesis. The time change mentioned here does not mean a high-speed change in acoustic characteristics that is observed even when a muddy voice or a hoarse voice is uttered constantly, but such a change immediately after the start of utterance, for example. It corresponds to a relatively macroscopic transition of voice quality, in which the degree of high-speed fluctuation is large, then gradually attenuates with the passage of time, and then stabilizes to a certain degree with the passage of time. Such changes in voice quality make a big difference depending on the type of singing expression.

FIG. 1 is a diagram illustrating a GUI related to the related technology. This GUI is used in a singing synthesis program related to a related technique. This GUI includes a score display area 911, a window 912, and a window 913. The score display area 911 is an area in which a score related to speech synthesis is displayed, and in this example, the score is represented in a format corresponding to a so-called piano roll. In the score display area 911, the horizontal axis represents time and the vertical axis represents scale. The window 912 is a pop-up window displayed according to a user operation, and includes a list of singing expressions that can be given to the synthetic singing. The user selects the applicable singing expression from this list. The window 913 displays a graph showing the degree of application of the selected singing expression. In the window 913, the horizontal axis represents time and the vertical axis represents the strength of application of the singing expression. The user edits the graph in window 913 and inputs the time variation of the degree of application of VQM. However, in this example, it is difficult to synthesize a natural and expressive singing because the user has to manually input the time change of the degree of application of VQM.

2. 2. Configuration FIG. 2 is a diagram showing the concept of singing expression assignment according to one embodiment. In the following, "synthetic singing" refers to a synthesized voice, and in particular, a voice to which a scale and lyrics are given. Unless otherwise specified, the term "synthetic singing" simply refers to a synthetic voice to which the singing expression according to the present embodiment is not added. "Singing expression" refers to a musical expression given to a synthetic voice, and includes, for example, expressions such as vocal fly (fry), growl (growl), and roar (rough). In the present embodiment, adding a pre-recorded element (sample) of a local singing expression to a normal synthetic singing (without a singing expression) by morphing is described as "singing expression for synthetic singing". Is given. " Here, the element of the singing expression is temporally local to the entire singing voice or one note. Temporally local means that the time occupied by the singing expression is partial to the entire singing voice or one note. A piece of singing expression is a pre-recorded piece of singing expression by a singer, and is a piece of singing expression (musical expression) made at a local time during singing. A piece is a data of a part of the voice waveform emitted by a singer. Further, morphing refers to a process of multiplying at least one of a piece of a singing expression and a synthetic singing by a coefficient that increases or decreases with the passage of time and adds both. A piece of singing expression and a normal synthetic singing are morphed in time. In morphing, the temporal variation of acoustic features in singing expression remains preserved. When the elements of the singing expression are added by morphing, morphing is performed on the synthetic singing at a local time in the normal synthetic singing.

In this example, the reference time for addition of the synthetic singing and the element of the singing expression is the start time of the note (that is, the note) and the end time of the note. Hereinafter, setting the start time of a note as a reference time is referred to as an "attack reference", and setting the end time as a reference time is referred to as a "release reference".

FIG. 3 is a diagram illustrating a functional configuration of the speech synthesizer 1 according to the embodiment. The voice synthesizer 1 has a database 10, a synthesizer 20, and a UI (User Interface) unit 30. In this example, a piece-connected singing composition is used. The database 10 is a database in which singing elements and singing expression elements are recorded. The synthesizer 20 reads a singing element piece and a singing expression element piece from the database 10 based on the musical score information and the information instructing the singing expression, and synthesizes a singing voice with the singing expression using these. The UI unit 30 is an interface for inputting, editing, and outputting score information, singing expression, and singing voice.

FIG. 4 is a diagram illustrating a hardware configuration of the speech synthesizer 1. The speech synthesizer 1 is a computer device having a CPU (Central processing unit) 101, a memory 102, a storage 103, an input / output IF 104, a display 105, and an input device 106, specifically, for example, a tablet terminal. The CPU 101 is a control device that executes a program to control other elements of the speech synthesizer 1. The memory 102 is a main storage device, and includes, for example, a ROM (Read Only Memory) and a RAM (Random Access Memory). The ROM stores a program or the like for starting the voice synthesizer 1. The RAM functions as a work area when the CPU 101 executes a program. The storage 103 is an auxiliary storage device and stores various data and programs. The storage 103 includes, for example, at least one of an HDD (Hard Disk Drive) and an SSD (Solid State Drive). The input / output IF 104 is an interface for inputting / outputting information to / from another device, and includes, for example, a wireless communication interface or a NIC (Network Interface Controller). The display 105 is a device for displaying information, and includes, for example, an LCD (Liquid Crystal Display). The input device 106 is a device for inputting information to the speech synthesizer 1, and includes, for example, at least one of a touch screen, a keypad, a button, a microphone, and a camera.

In this example, the storage 103 stores a program (hereinafter referred to as “singing synthesis program”) that causes the computer device to function as the speech synthesis device 1. When the CPU 101 executes the singing synthesis program, the function of FIG. 3 is implemented in the computer device. The storage 103 is an example of a storage means for storing the database 10. The CPU 101 is an example of the synthesizer 20. The CPU 101, the display 105, and the input device 106 are examples of the UI unit 30. The details of the functional elements of FIG. 3 will be described below.

2-1. Database 10
The database 10 includes a database containing singing elements (elementary piece database) and a database containing singing expression elements (singing expression database). The elemental piece database is a conventionally known elemental connection. Since it is the same as that used in type singing synthesis, detailed description is omitted. Hereinafter, unless otherwise specified, the singing expression database is simply referred to as database 10. In database 10, in order to reduce the calculation load during singing synthesis and prevent estimation errors of acoustic features, the acoustic features of the elementary pieces of the singing expression are estimated in advance, and the estimated acoustic features are estimated. Is preferably recorded in the database. The acoustic features recorded in the database 10 may be manually modified.

FIG. 5 is a schematic diagram illustrating the structure of the database 10. In order to make it easy for the user or the program to find the desired singing expression, the pieces of the singing expression are organized and recorded in the database 10. FIG. 5 shows an example of a tree structure. Each leaf at the end of the tree structure corresponds to one singing expression. For example, "Attack-Fry-Power-High" means a singing expression that has a strong voice quality and is suitable for a high range among attack-based singing expressions that mainly consist of fly vocalization. Singing expressions may be placed in the nodes as well as the leaves at the ends of the tree structure. For example, in addition to the above example, a singing expression corresponding to "Attack-Fry-Power" may be recorded.

The database 10 contains at least one piece for each singing expression. Two or more pieces may be recorded depending on the phoneme. It is not necessary to record a unique piece of singing expression for every phoneme. This is because the elements of the singing expression are morphed with synthetic singing, so the basic quality of singing is already ensured by synthetic singing. For example, in order to obtain a good quality singing in a piece-connected singing composition, it is necessary to record a piece for each phoneme of a two-phoneme chain (for example, a combination of / ai / or / ao /). .. However, the elements of the singing expression may be unique for each single note element (for example, / a / or / o /), or the number may be further reduced so that each singing expression is represented by the singing expression. Only one piece (for example, / a / only) may be recorded. The number of pieces to be recorded for each singing expression is determined by the database creator in consideration of the balance between the man-hours for creating the singing expression database and the quality of synthetic singing. To obtain a higher quality (realistic) synthetic singing, record a piece of original singing expression for each phoneme. To reduce the man-hours required to create a singing expression database, reduce the number of pieces per singing expression.

When two or more pieces are recorded for each singing expression, it is necessary to define the mapping (correlation) between the pieces and the phoneme. As an example, for a singing expression, the elemental file "S0000" is mapped to the phonemes / a / and / i /, and the elemental file "S0001" is mapped to the phonemes / u /, / e /, and / o /. Will be done. Such a mapping is defined for each singing expression. The number of pieces recorded in the database 10 may differ for each singing expression. For example, one singing expression may contain two pieces, and another singing expression may contain five pieces.

In the database 10, information indicating a reference time (expression reference time) is recorded for each element of the singing expression. This reference time is a feature point on the time axis in the waveform of the elementary piece of the singing expression. The reference time includes at least one of a singing expression start time, a singing expression end time, a note-on-set start time, a note-offset start time, a note-on-set end time, and a note-offset end time.

6 and 7 are diagrams illustrating each reference time. In this example, the time domain of the voice waveform of the elemental piece of the singing expression is divided into a pre-section, an onset section, a sustain section, an offset section, and a post-section. These sections are divided by, for example, the creator of database 10. FIG. 6 shows an attack-based singing expression, and FIG. 7 shows a release-based singing expression.

Attack-based singing expressions are divided into pre-sections, onset sections, and sustain sections. The sustain section is a section in which acoustic characteristics (for example, fundamental frequency) are stable within a predetermined range. The fundamental frequency of the sustain section corresponds to the pitch of this singing expression. The onset section is a section before the sustain section, and is a section in which the acoustic characteristics change with time. The pre-section is the section before the on-set section. In the attack-based singing expression, the beginning of the pre-section is the singing expression start time. The beginning of the onset section is the note onset start time. The end of the onset section is the note onset end time. The end of the sustain section is the end time of the singing expression.

Release-based singing expressions are divided into sustain sections, offset sections, and post sections. The offset section is a section after the sustain section, in which the acoustic characteristics change with time. The post section is the section after the offset section. The beginning of the sustain section is the singing expression start time. The end of the sustain section is the note offset start time. The end of the offset section is the note offset end time. The end of the post section is the end time of the singing expression.

Templates of parameters applied to singing synthesis are recorded in the database 10. The parameters referred to here include, for example, the time transition and application time of the morphing coefficient (application rate), and the speed of singing expression. For example, a plurality of templates may be created by the database creator, and the database creator may determine in advance which template is applied for each singing expression. That is, which template is applied to which singing expression may be determined in advance. Alternatively, the template itself is included in the database 10, and the user may select which template to use when singing.

2-2. Synthesizer 20
FIG. 8 is a diagram illustrating the functional configuration of the synthesizer 20. The synthesizer 20 includes a timing calculation means 21, a time expansion / contraction mapping means 22, a short-time spectrum manipulation means 23, a synthesis means 24, a specific means 25, and an acquisition means 26.

The timing calculation means 21 calculates the timing (position on the time axis) to match the element piece of the singing expression with the synthetic singing by using the reference time recorded for the element piece of the singing expression. For example, the timing calculation means 21 makes the note-on-set start time (an example of the synthetic sound reference time) match the vowel start time of the synthetic singing with respect to the element of the attack-based singing expression. For a piece of release-based singing expression, either match the note offset end time (another example of synthetic sound reference time) with the synthetic singing vowel end time, or set the singing expression end time to the synthetic singing pronunciation. Match the end time.

The time expansion / contraction mapping means 22 calculates the time expansion / contraction mapping of the element of the singing expression (performs the expansion process on the time axis). Here, the time expansion / contraction mapping means 22 calculates a mapping function indicating the time correspondence between the synthetic singing and the element of the singing expression. The mapping function used here is a non-linear function that divides the characteristics for each reference time of the elementary piece of the singing expression. By using such a function, it is possible to add to the synthetic singing without impairing the property of the singing expression contained in the element piece as much as possible. The time expansion / contraction mapping means 22 performs time expansion of the characteristic portion of the elementary piece of the singing expression by an algorithm different from that of the portion other than the characteristic portion (that is, using a different mapping function). The characteristic portion is, for example, a pre-section and an on-set section in an attack-based singing expression as described later.

FIG. 9 is a diagram illustrating a mapping function in an example in which the time length of the elemental piece of the singing expression is shorter than that of the synthetic singing. This is used, for example, when an attack-based singing expression is applied to a specific note, and the time length of the singing expression element is shorter than that of the synthetic singing. First, the basic concept of the mapping function will be explained. In the elemental piece of the singing expression, the pre-section and the on-set section contain a lot of dynamic fluctuations of the acoustic characteristics as the singing expression. Therefore, if this section is expanded or contracted over time, the nature of the singing expression will change. Therefore, the time expansion / contraction mapping means 22 does not perform time expansion / contraction as much as possible in the pre-section and the onset section, and obtains a desired time expansion / contraction mapping by extending the sustain section.

FIG. 9A shows an example in which the slope of the mapping function is made gentle in the sustain interval, that is, the time of the entire element piece is extended by slowing down the data reading speed of the element piece of the singing expression. FIG. 9B shows an example in which the time of the entire piece is extended by repeatedly returning the data read position to the front while keeping the read speed constant even in the sustain section. This utilizes the characteristic that the acoustic characteristics that are generally stationary are maintained in the sustain section. At this time, it is preferable that the time for returning the data read position and the time for returning the data correspond to the start position and the end position of the temporal periodicity appearing in the acoustic feature. By adopting such a data reading position, it is possible to obtain a synthetic singing with a natural singing expression. These start and end positions can be obtained, for example, by obtaining an autocorrelation function with respect to the time series of the acoustic features of the elements of the singing expression and adopting the peaks thereof. FIG. 9C shows an example in which a so-called random mirror loop (Random-Mirror-Loop) is applied in the sustain section to extend the time of the entire element piece. The random mirror loop is a method of extending the time of the entire element piece by reversing the sign of the data reading speed many times during reading. The time to invert the sign is determined based on a pseudo-random number so that artificial periodicity that is not originally included in the singing expression sample does not occur.

9 (a) to 9 (c) show an example in which the data reading speed in the pre-section and the on-set section is not changed, but the user may want to adjust the speed of the singing expression. As an example, there is a case where the singing expression of "hiccups" is desired to be faster than the singing expression recorded as a piece. In such a case, the data read speed in the pre-section and the on-set section may be changed. Specifically, if you want to make it faster than the raw piece, increase the data read speed. FIG. 9D shows an example of increasing the data read speed in the pre-section and the on-set section. In the sustain section, the data read speed is slowed down and the time of the entire piece is extended.

FIG. 10 is a diagram illustrating a mapping function in an example in which the elemental piece of the singing expression has a longer time length than the synthetic singing. This is used, for example, when applying an attack-based singing expression to a specific note, when the elemental piece of the singing expression has a longer time length than the synthetic singing. In these examples as well, the time expansion / contraction mapping means 22 obtains the desired time expansion / contraction mapping by shortening the sustain interval without performing time expansion / contraction in the pre-section and the onset section as much as possible.

FIG. 10A shows an example in which the slope of the mapping function is steep in the sustain interval, that is, the time of the entire element is shortened by increasing the data reading speed of the element of the singing expression. FIG. 10B shows an example in which the time of the entire element is shortened by discontinuing the data reading in the middle of the sustain section while the reading speed remains constant even in the sustain section. Since the sonic characteristics of the sustain section are stationary, it is more natural to obtain a synthetic singing by simply not using the end of the piece while keeping the data reading speed constant rather than changing the data reading speed. FIG. 10C shows a mapping function used when the time of synthetic singing is shorter than the sum of the time lengths of the pre-interval and the onset interval of the elemental pieces of the singing expression. In this example, the time expansion / contraction mapping means 22 increases the data read speed in the onset section so that the end of the onset section coincides with the end of the synthetic singing. FIG. 10 (d) shows another example of the mapping function used when the time of the synthetic singing is shorter than the sum of the time lengths of the pre-interval and the onset interval of the element of the singing expression. In this example, the time expansion / contraction mapping means 22 shortens the time of the entire element piece by stopping the data reading in the middle of the onset section while keeping the data reading speed constant even in the onset section. In the example of FIG. 10D, it is necessary to pay attention to the determination of the fundamental frequency. Since the pitch of the onset section is often different from the pitch of the note, the fundamental frequency of the synthetic singing does not reach the pitch of the note unless the end of the onset section is used, and the sound is off (tone deafness). It may be heard. In order to avoid this, the time expansion / contraction mapping means 22 determines a representative value of the fundamental frequency corresponding to the pitch of the note in the onset section, and expresses the singing so that this fundamental frequency matches the pitch of the note. Shift the fundamental frequency of the entire piece. As a representative value of the fundamental frequency, for example, the fundamental frequency at the end of the onset section is used.

Although FIGS. 9 and 10 exemplify the time expansion / contraction mapping for the attack-based singing expression, the idea is the same for the time expansion / contraction mapping for the release-based singing expression. That is, in the release-based singing expression, the offset section and the post section are characteristic parts, and the time extension mapping is performed by an algorithm different from the other parts.

The short-time spectrum manipulation means 23 decomposes the short-time spectrum of the element of the singing expression into several components (acoustic features). The short-time spectrum manipulation means 23 morphs a part of the components obtained by decomposition to the same component of the synthetic singing to obtain a series of short-time spectra of the synthetic singing to which the singing expression is given. The short-time spectrum manipulation means 23 decomposes the short-time spectrum of the element of the singing expression into, for example, one or more of the following components.
(A) Spectral Envelope (b) Outline of Spectral Envelope (c) Phase Spectral Envelope (d) Temporal Fine Fluctuations of Spectral Envelope (or Harmonic Amplitude) Fluctuation (f) Fundamental frequency In order to morph these components independently between the elemental piece of the singing expression and the synthetic singing, the above decomposition must be performed on the synthetic singing as well. In the synthesizer, this information may be generated in the middle of synthesis, so it may be used. Each component will be described below.

Spectral envelopes are an outline of the amplitude spectrum and are primarily concerned with the perception of phonology and individuality. Many methods for estimating spectral envelopes have been proposed, for example, estimation by low-order cepstrum coefficients can be used. In this embodiment, it is of special significance to treat the spectral envelope independently of other components. That is, even if a piece of singing expression whose phonology or individuality is different from that of synthetic singing is used, if the morphing application rate for spectral envelope is set to zero, the phonology and individuality of synthetic singing will appear 100%. Therefore, it is possible to divert a piece of singing expression having a different phoneme or individuality (for example, another phoneme of the person or a piece of another person at all). In addition, in a singing expression that intentionally changes the phonology or individuality, this component may be morphed independently in order to control the degree.

The spectral envelope outline is a more general representation of the amplitude spectral envelope, and is mainly related to the brightness of the voice. The spectral envelope outline can be obtained by various methods, for example, by a cepstrum coefficient having a lower order than the spectral envelope. Unlike the spectral envelope, the spectral envelope scheme contains little phonological or personality information. Therefore, even when the spectral envelope is not morphed, the brightness of the voice included in the singing expression and its temporal movement can be maintained by performing the morphing only for the spectral envelope approximately formed portion.

The phase spectrum envelope is an outline of the phase spectrum. The phase spectrum envelope can be determined by various methods. For example, a short-time spectrum is analyzed at a frame interval synchronized with the period of the signal, then only the phase value of each tuning component is adopted, unwrapped at this stage, and frequencies other than the tuning component (tuning). In (between waves and tuning), it is possible to obtain a phase spectrum encapsulation rather than a simple phase spectrum by performing nearest-neighbor interpolation or linear or higher-order curve interpolation.

FIG. 11 is a diagram illustrating the relationship between the spectral envelope and the spectral envelope outline. The temporal variation of the spectral envelope and the temporal variation of the phase spectrum envelope correspond to the components that fluctuate at high speed in the speech spectrum in a very short time, and correspond to the rough feeling of muddy voice and hoarseness. The temporal fine variation of the spectral envelope is obtained by taking the difference on the time axis with respect to these estimated values, or by taking the difference between these values smoothed within a certain time interval and the value in the frame of interest. be able to. The temporal variation of the phase spectrum envelope is obtained by taking the difference on the time axis with respect to the phase spectrum envelope, or by taking the difference between these values smoothed within a certain time interval and the value in the frame of interest. Fine fluctuations can be obtained. All of these processes correspond to some kind of high-pass filter.

When both spectral envelopes and spectral envelope schemes are used as acoustic features, the actual morphing does not use the spectral envelope itself (eg, FIG. 11).
Two acoustic features are used: (a') the difference between the spectral envelope scheme and the spectral envelope, and (b) the spectral envelope scheme. For example, when the spectrum envelope and the spectrum envelope outline are separated as shown in FIG. 11, the spectrum envelope includes the information of the spectrum envelope outline, so that both are treated separately. When separated in this way, information about the absolute volume is included in the spectral envelope scheme. When changing the intensity of human voice, individuality and phonological characteristics can be maintained to some extent, but since the volume and the overall slope of the spectrum often change at the same time, the volume information in the spectrum envelope outline. It can be said that it is natural to include.

Note that the wave tuning amplitude and the wave tuning phase may be used instead of the spectrum envelope and the phase spectrum envelope. The choice of whether to use spectral envelopes and phase spectral envelopes or to use tuned amplitude and tuned phase depends on the choice of synthesis method. Spectral envelopes and phase spectral envelopes are used when synthesizing pulse trains or with time-varying filters, and sine wave model-based synthesis schemes such as SMS, SPP, or WBHSM use tuned amplitude and phase. Is used.

The fundamental frequency is mainly related to the perception of pitch. Unlike other acoustic features, the fundamental frequency cannot be determined by simple interpolation based on the conversion rate. This is because the pitch of the note in the element of the singing expression and the pitch of the note of the synthetic singing are generally different, and the fundamental frequency of the element of the singing expression and the fundamental frequency of the synthetic singing are simply interpolated. Even so, the pitch will be completely different from the pitch to be synthesized. Therefore, in the present embodiment, the short-time spectrum manipulation means 23 first bases the entire element piece of the singing expression so that the pitch of the note given to the element piece of the singing expression matches the pitch of the note of the synthetic singing expression. Shift the frequency by a certain amount. This process does not match the fundamental frequency of each time of the element of the singing expression with the synthetic sound, and the dynamic fluctuation of the fundamental frequency contained in the element of the singing expression is retained.

FIG. 12 is a diagram illustrating a process of shifting the fundamental frequency of a piece of a singing expression. In FIG. 12, the broken line shows the characteristics of the elemental pieces of the singing expression before the shift (that is, recorded in the database 10), and the solid line shows the characteristics after the shift. In this process, the entire characteristic curve of the element piece is shifted as it is so that the fundamental frequency of the sustain section becomes a desired frequency while maintaining the fluctuation of the fundamental frequency in the pre-section and the on-set section. When the parameter of the application rate of the singing expression is applied to the fundamental frequency, the short-time spectrum manipulation means 23 interpolates the fundamental frequency obtained by this processing and the fundamental frequency in the normal singing synthesis at each time.

The synthesis means 24 synthesizes the synthetic singing and the element pieces of the singing expression to obtain a synthetic singing to which the singing expression is given. There are various methods for synthesizing a synthetic singing and a piece of singing expression and finally obtaining it as a waveform in the time domain, but these methods are roughly divided into two types depending on the expression method of the spectrum to be input. it can. One is a method based on the tuning component, and the other is a method based on the spectral envelope.

For example, SMS is known as a synthesis method based on the wave-tuning component (Serra, Xavier, and Julius Smith. "Spectral modeling synthesis: A sound analysis / synthesis system based on a deterministic plus stochastic decomposition." Computer Music Journal 14.4 (1990): 12-24.). The spectrum of voiced sounds is represented by the frequency, amplitude, and phase of the sinusoidal component at the fundamental frequency and its approximately integral multiples. When the spectrum is generated by SMS and the inverse Fourier transform is performed, a waveform for several cycles multiplied by the window function is obtained. After dividing the window function, only the vicinity of the center of the composite result is cut out by another window function, and the output result buffer is overlap-added. By repeating this process at frame intervals, a long-term continuous waveform can be obtained.

As a synthesis method based on the spectrum envelope, for example, NBVPM (Bonada, Jordi. "High quality voice transformations based on modeling radiated voice pulses in frequency domain." Proc. Digital Audio Effects (DAFx). 2004.) is known. In this example, the spectrum is represented by an amplitude spectrum envelopment and a phase spectrum encapsulation, and does not include frequency information of the fundamental frequency or the tuning component. By inverse Fourier transforming this spectrum, a pulse waveform corresponding to the vocal cord vibration for one cycle and the vocal tract response to it can be obtained. This is superimposed and added to the output buffer. At this time, if the phase spectrum envelopes in the spectra of adjacent pulses have substantially the same value, the reciprocal of the time interval to be superimposed and added to the output buffer becomes the fundamental frequency of the final synthesized sound.

There are two methods for synthesizing singing voice and singing expression, one is performed in the frequency domain and the other is performed in the time domain. Regardless of which method is used, the composition of the singing voice and the singing expression is basically performed by the following procedure. First, the singing voice and the singing expression are morphed for the components other than the temporal fine fluctuation components of the amplitude and the phase. Next, a synthetic singing with a singing expression is generated by adding the temporal fine fluctuation components of the amplitude and phase of each tuning component (or its peripheral frequency band).

When synthesizing the singing voice and the singing expression, a time expansion / contraction mapping different from the other components may be used only for the temporal minute fluctuation component. This is effective in the following two cases, for example.

The first is the case where the user intentionally changes the speed of the singing expression. The temporal fine fluctuation component is deeply related to the nature of the texture of the voice such as "roughness", "gritty", or "shwashwa" in the speed and periodicity of the fluctuation, and it is said that the fluctuation speed is changed. The nature of the texture of is changed. For example, when the user inputs an instruction to increase the speed in a singing expression in which the pitch is lowered at the end as shown in FIG. 7, the user specifically lowers the pitch and changes the timbre and texture accordingly. Although it has the intention of increasing the speed of the song, it is presumed that it does not intend to change the nature of the texture of the singing expression itself. Therefore, in order to obtain the singing expression as intended by the user, it is sufficient to increase the data reading speed of the post section by linear time expansion and contraction for the components such as the fundamental frequency and the spectral envelope, but it is appropriate for the time minute fluctuation component. It is looped in a cycle (similar to the sustain section of FIG. 9B) or a random mirror loop (similar to the sustain section of FIG. 9C).

The second is the case of synthesizing a singing expression in which the fluctuation period of the temporal fine fluctuation component should depend on the fundamental frequency. It has been empirically found that in a singing expression in which the amplitude and phase of the wave tuning component are periodically modulated, it may sound more natural to correlate the amplitude and phase fluctuation period with the fundamental frequency. There is. A singing expression having such a texture is called, for example, "rough" or "growl". As a method of correlating the fluctuation period of amplitude and phase with the fundamental frequency, the same ratio as the conversion ratio of the fundamental frequency applied when synthesizing the singing expression waveform is applied to the data reading speed of the temporal fine fluctuation component. Can be used.

The synthesis means 24 synthesizes the synthetic singing and the singing expression waveform. That is, the synthesis means 24 gives a singing expression to the synthetic singing. The synthesis of the synthetic singing and the singing expression waveform is performed using at least one of the above-mentioned acoustic features (a) to (f). Which of the acoustic features (a) to (f) is used is set for each singing expression. For example, the singing expression crescendo or decrescendo in musical terms is mainly related to changes in vocal strength over time. Therefore, the main acoustic feature to be morphed is the spectral envelope scheme. Phonology and individuality are not considered to be the main acoustic features that make up a crescendo or decrescendo. Therefore, if the morphing application amount (coefficient) of the spectral envelope is set to zero, every singer can obtain a piece of crescendo's singing expression recorded from the singing of only one phonology of one singer. It can also be applied to any phoneme of. In another example, in a singing expression such as vibrato, the fundamental frequency fluctuates periodically, and the volume fluctuates in synchronization with it. Therefore, the acoustic features to be morphed are the fundamental frequency and spectral envelope schemes.

Moreover, since the spectral envelope is an acoustic feature related to phonology, it is possible to impart a singing expression that does not depend on phonology by excluding the spectral envelope from the target of morphing. For example, a singing expression in which a fragment is recorded only for a specific phoneme (for example, / a /) can be used for a synthetic song of a phoneme other than the specific phoneme by excluding the spectral envelope from the target of morphing. Can also morph a piece of that singing expression.

In this way, the acoustic features to be morphed can be limited for each type of singing expression. In this way, the acoustic features targeted for morphing may be limited, or all acoustic features may be dealt with by morphing regardless of the type of singing expression. When many acoustic features are targeted for morphing, a synthetic singing that is close to the element of the original singing expression can be obtained, and the naturalness of that part is improved. However, since the difference in sound quality from the part to which the singing expression is not given becomes large, there is a possibility that a sense of incongruity may occur when listening to the whole singing. Therefore, when template the acoustic features to be morphed, the acoustic features to be morphed are determined in consideration of the balance between naturalness and discomfort.

FIG. 13 is a diagram illustrating a more detailed functional configuration of the synthesizing means 24 for synthesizing a singing voice and a piece of a singing expression in the frequency domain. In this example, the synthesis means 24 has a spectrum generation means 2401, an inverse Fourier transform means 2402, a synthesis window application means 2403, and an overlay addition means 2404.

FIG. 14 is a sequence chart illustrating the operation of the synthesizer 20. In step S1400, the specifying means 25 identifies the element piece used for generating the synthetic song and the element piece used for giving the singing expression from the element piece database and the singing expression database included in the database 10. The spectrum generation means 2401 identifies these elements based on the information supplied from the UI unit 30.

In step S1401, the acquisition means 26 acquires the time variation of the acoustic features used to generate the synthetic singing. The acoustic features acquired here are the spectral wrapping H (f), the spectral wrapping outline G (f), the phase spectral wrapping P (f), the temporal fine variation of the spectral wrapping I (f), and the phase spectral wrapping. It includes at least one of the temporal fine variation Q (f) and the fundamental frequency F0. The acquisition means 26 acquires these acoustic features from, for example, the short-time spectrum manipulation means 23 that has processed the element pieces identified in step S1400.

In step S1402, the acquisition means 26 acquires the time change of the acoustic feature used for imparting the singing expression. The acoustic features acquired here are the same as those used to generate synthetic singing. When distinguishing between the acoustic characteristics of a synthetic singing and the acoustic characteristics of a singing expression, the acoustic characteristics of the synthetic singing are subscript v, the acoustic characteristics of the singing expression are subscript p, and the synthetic singing with the singing expression is added. The subscript bp is added to each. The acquisition means 26 acquires these acoustic features from, for example, the short-time spectrum manipulation means 23 that has processed the element pieces identified in step S1400.

In step S1403, the acquisition means 26 acquires the reference time set for the element of the singing expression to be given. The reference time acquired here is at least one of the singing expression start time, the singing expression end time, the note-on-set start time, the note offset start time, the note-on-set end time, and the note offset end time, as described above. Includes one.

In step S1404, the timing calculation means 21 uses the reference time recorded for the element piece of the singing expression to match the element piece of the singing expression with the note (synthetic singing) (position on the time axis). ) Is calculated.

In step S1405, the time expansion / contraction mapping means 22 performs time expansion / contraction mapping on the element piece of the singing expression according to the relationship between the time length of the target note and the time length of the element piece of the singing expression.

In step S1406, the time expansion / contraction mapping means 22 of the singing expression element piece so that the reference frequency F0v of the singing voice and the reference frequency F0p of the singing expression match (that is, the pitches of both match). Shift the pitch.

In step S1407, the spectrum generating means 2401 multiplies each of the synthetic singing and the singing expression by a morphing coefficient and then adds each acoustic feature. As an example, for the spectral envelope outline G (f), the spectral envelope H (f), and the temporal fine variation I (f) of the spectral envelope.
Gvp (f) = (1-aG) Gv (f) + aG · Gp (f) ... (1)
Hvp (f) = (1-aH) Hv (f) + aH · Hp (f) ... (2)
Ivp (f) = (1-aI) Iv (f) + aI · Ip (f)… (3)
Morphs synthetic singing and singing expressions. Note that aG, aH, and aI are morphing coefficients for the spectral envelope outline G (f), the spectral envelope H (f), and the temporal fine variation I (f) of the spectral envelope, respectively. Each of these may be set independently.

In step S1408, the spectrum generating means 2401 outputs the spectrum obtained by adding the acoustic features. When the spectrum is input, the inverse Fourier transform means 2402 performs an inverse Fourier transform on the input spectrum (step S1409) and outputs a waveform in the time domain. When the waveform in the time domain is input, the composite window application means 2403 applies a predetermined window function to the back-input waveform (step S1410), and outputs the result. The overlap-add method 2404 superimposes and adds the waveform to which the window function is applied (step S1411). By repeating this process at each frame interval, a continuous waveform for a long time can be obtained.

The method of synthesizing in the frequency domain has an advantage that the amount of calculation can be suppressed because it is not necessary to execute a plurality of synthesizing processes. However, in order to morph the fine fluctuation components of amplitude and phase, the singing synthesis means (not shown in FIG. 13) must also use these acoustic features.

FIG. 15 is a diagram illustrating a more detailed functional configuration of the synthesizing means 24 for synthesizing a singing voice and a piece of a singing expression in the time domain. In this example, the synthesizing means 24 has a spectrum generating means 2411, an inverse Fourier transform means 2412, a synthesizing window application means 2413, an overlay addition means 2414, a singing synthesis means 2415, a multiplication means 2416, a multiplication means 2417, and an addition means 2418. ..

In this example, the spectrum generating means 2411 has the spectrum envelopment H (f) of the synthetic song, the spectrum envelopment outline G (f), the phase spectrum envelopment P (f), and the fundamental frequency F0, and the elementary pieces of the singing expression. The temporal fine variation I (f) of the spectral inclusion and the temporal fine variation Q (f) of the phase spectrum inclusion are input. The spectrum generating means 2411 obtains a spectrum from the input acoustic features.

The inverse Fourier transform means 2412 performs an inverse Fourier transform on the input spectrum to obtain a waveform in the time domain. The composite window application means 2413 applies a predetermined window function to the waveform obtained by the inverse Fourier transform. The overlap-add method 2414 superimposes and adds the waveform to which the window function is applied. By repeating this process at each frame interval, a continuous waveform for a long time can be obtained. This waveform shows the waveform of a piece of singing expression in which the fundamental frequency is shifted to the fundamental frequency of synthetic singing.

The spectrum envelope H (f), the spectrum envelope outline G (f), the phase spectrum envelope P (f), and the fundamental frequency F0 of the synthetic song are input to the song synthesis means 2415. The singing synthesis means 2415 generates a waveform in the time domain of the synthetic singing from these acoustic features, for example, by using a known technique.

The multiplication means 2416 multiplies the output of the superposition addition means 2414 by the application coefficient a of the fine variation component. The multiplication means 2417 multiplies the output of the singing synthesis means 2415 by a coefficient (1-a). The adding means 2418 adds the output of the multiplying means 2416 and the output of the multiplying means 2417.

In the method of synthesizing in the time domain, the minute fluctuation component is dealt with only in the part where the waveform of the singing expression is synthesized (the right half of FIG. 15). According to this method, the singing synthesis means 2415 does not have to be of a method using minute fluctuation components of amplitude and phase. In this case, in the singing synthesis means 2415, for example, SPP (Spectral Peak Processing) (Bonada, Jordi, Alex Loscos, and H. Kenmochi. "Sample-based singing voice synthesizer by spectral concatenation." Proceedings of Stockholm Music Acoustics Conference. 2003 .) Can be used. In SPP, a component corresponding to the texture of voice is synthesized by the spectral shape around the tonal peak, not by the minute fluctuation with time. When adding a singing expression to an existing singing synthesis means adopting such a method, it is convenient to adopt a method of synthesizing in the time domain in that the existing singing synthesis means can be used as it is. In the case of synthesizing in the time domain, if the phases of the singing composition and the singing expression composition are different, the waveforms cancel each other out or a beat occurs. In order to prevent such a problem from occurring, it is necessary that the phase spectrum envelopes match in both composites and the reference positions (so-called pitch marks) of the voice pulses for each period match.

Since the value of the phase spectrum obtained by analyzing the voice by short-time Fourier transform or the like generally has indefiniteness with respect to θ + n2π, that is, the integer n, morphing the phase spectrum envelope may be difficult. .. Since the influence of the phase spectrum envelopment on the perception of sound is smaller than that of other acoustic feature components, the phase spectrum envelopment does not necessarily have to be interpolated, and an arbitrary value may be given. The simplest and most natural method for determining the phase spectrum envelope is to use the minimum phase calculated from the amplitude spectrum envelope. In this case, the spectrum envelope H (f) + G (f) excluding the fine variation component is first obtained from H (f) and G (f) of FIG. 13 or 15, and the corresponding minimum phase is obtained and P ( f). As a method of calculating the minimum phase corresponding to an arbitrary amplitude spectrum envelope, for example, a method via cepstrum (Oppenheim, Alan V., and Ronald W. Schafer. Discrete-time signal processing. Pearson Higher Education, 2010.) is used. be able to.

2-3. UI part 30
2-3-1. Functional configuration FIG. 16 is a diagram illustrating the functional configuration of the UI unit 30. The UI unit 30 has a display means 31, a reception means 32, and a sound output means 33. The display means 31 displays the UI screen. The receiving means 32 receives an operation via the UI. The sound output means 33 outputs a synthetic singing according to an operation received via the UI. The UI displayed by the display means 31 includes, for example, an image object for simultaneously changing the values of a plurality of parameters used for synthesizing the singing expression given to the synthetic singing, as will be described later. The receiving means accepts an operation on this image object.

2-3-2. UI example (overview)
FIG. 17 is a diagram illustrating a GUI used in the UI unit 30. This GUI is used in the singing synthesis program according to the embodiment. This GUI includes a score display area 511, a window 512, and a window 513. The score display area 511 is an area in which a score related to singing composition is displayed, and in this example, the score is represented in a format corresponding to a so-called piano roll. In the score display area 511, the horizontal axis represents time and the vertical axis represents scale. In this example, image objects corresponding to the five notes of notes 5111-5115 are displayed. Lyrics are assigned to each note. In this example, notes 5111-5115 are assigned the lyrics "I", "love", "you", "so", and "much". The user can add a new note at any position on the score by clicking on the piano roll. For the notes set on the score, attributes such as the position, scale, or length of the notes on the time axis can be edited by operations such as so-called drag and drop. As for the lyrics, the lyrics for one song may be input in advance, and the lyrics may be automatically assigned to each note according to a predetermined algorithm, or the user may manually assign the lyrics to each note.

Window 512 and window 513 display image objects showing controls for imparting an attack-based singing expression and a release-based singing expression to one or more notes selected in the score display area 511, respectively. The area. The note selection in the score display area 511 is performed by a predetermined operation (for example, left button click of the mouse).

2-3-3. UI example (selection of singing expression)
FIG. 18 is a diagram illustrating a UI for selecting a singing expression. This UI uses a pop-up window. When the user performs a predetermined operation (for example, right-clicking of the mouse) on the note to which the singing expression is to be added, the pop-up window 514 is displayed. The pop-up window 514 is a window for selecting the first layer of the singing expressions organized in a tree structure, and includes display of a plurality of options. When the user performs a predetermined operation (for example, clicking the left mouse button) on the first-ranked option among the plurality of options included in the pop-up window 514, the pop-up window 515 is displayed. The pop-up window 515 is a window for selecting the second layer of the organized singing expression. When the user selects one option for the pop-up window 515, the pop-up window 516 is displayed. The pop-up window 516 is a window for selecting the third layer of the organized singing expression. The UI unit 30 outputs information for identifying the selected singing expression via the UI of FIG. 18 to the synthesizer 20. In this way, the user can select the desired singing expression from among the organized structures.

In the score display area 511, icons 5116 and 5117 are displayed around the note 5111. The icon 5116 is an icon (an example of an image object) for instructing the editing of the attack-based singing expression, and the icon 5117 is an icon for instructing the editing of the release-based singing expression. For example, when the user clicks the right mouse button with the mouse pointer on the icon 5116, a pop-up window 514 for selecting an attack-based singing expression is displayed.

FIG. 19 is a diagram showing another example of a UI for selecting a singing expression. In this example, in window 512, an image object for selecting an attack-based singing expression is displayed. Specifically, a plurality of icons 5121 are displayed in the window 512. Each icon represents a singing expression. In this example, 10 types of singing expressions are recorded in the database 10, and 10 types of icons 5121 are displayed in the window 512. The user selects an icon corresponding to the singing expression to be given from the icons 5121 of the window 512 in a state where one or more target notes are selected in the score display area 511. Similarly, for the release-based singing expression, the user selects an icon in window 513. The UI unit 30 outputs information for identifying the selected singing expression via the UI of FIG. 19 to the synthesizer 20. The synthesizer 20 generates a synthetic singing with a singing expression based on this information. The sound output means 33 of the UI unit 30 outputs the generated synthetic singing.

2-3-4. UI example (parameter input of singing expression)
In the example of FIG. 19, the window 512 displays an image object of the dial 5122 for changing the degree of singing expression based on the attack. Dial 5122 is an example of a single operator for simultaneously changing the values of a plurality of parameters used for assigning a singing expression assigned to synthetic singing. Further, the dial 5122 is an example of an operator that is displaced according to a user operation. In this example, a single dial 5122 is used to simultaneously adjust a plurality of parameters related to the singing expression. The degree of release-based singing expression is also adjusted via dial 5132, which is also displayed in window 513. The plurality of parameters related to the singing expression are, for example, the maximum value of the morphing coefficient of each acoustic feature. The maximum value of the morphing coefficient is the maximum value when the morphing coefficient changes with the passage of time in each note. In the example of FIG. 2, the attack-based singing expression has the maximum morphing coefficient at the beginning of the note, and the release-based singing expression has the maximum morphing coefficient at the end of the note. The UI unit 30 has information (for example, a table) for changing the maximum value of the morphing coefficient according to the rotation angle of the dial 5122 from the reference position.

FIG. 20 is a diagram illustrating a table in which the rotation angle of the dial 5122 and the maximum value of the morphing coefficient are associated with each other. This table is defined for each singing expression. Multiple acoustic features (spectral wrapping H (f), spectral wrapping outline G (f), phase spectral wrapping P (f), temporal variability of spectral wrapping I (f), temporal variability of phase spectral wrapping For each of Q (f) and the fundamental frequency F0), the maximum value of the morphing coefficient is defined in association with the rotation angle of the dial 5122. For example, when the angle of rotation is 30 °, the maximum value of the morphing coefficient of the spectral envelope H (f) is zero, and the maximum value of the morphing coefficient of the spectral envelope outline G (f) is 0.3. In this example, the value of each parameter is defined only for the discrete value of the rotation angle, but the value of each parameter is specified by interpolation for the rotation angle not defined in the table.

The UI unit 30 detects the rotation angle of the dial 5122 according to the user's operation. The UI unit 30 specifies the maximum value of the morphing coefficient corresponding to the detected rotation angle with reference to the table of FIG. The UI unit 30 outputs the maximum value of the specified morphing coefficient to the synthesizer 20. The parameters related to the singing expression are not limited to the maximum value of the morphing coefficient. Other parameters may be adjusted, such as the rate of increase or decrease in the morphing coefficient. The user selects which singing expression portion of which note is to be edited on the score display area 511. At this time, the UI unit 30 sets the table corresponding to the selected singing expression as the table to be referred to in response to the operation of the dial 5122.

FIG. 21 is a diagram showing another example of the UI for editing the parameters related to the singing expression. In this example, the shape of the graph showing the time variation of the morphing coefficient applied to the acoustic features of the singing expression for the selected note in the score display area 511 is edited. The singing expression to be edited is designated by the icon 616. The icon 611 is an image object for designating the start of the period in which the morphing coefficient takes the maximum value in the attack-based singing expression. The icon 612 is an image object for designating the end of the period in which the morphing coefficient takes the maximum value in the attack-based singing expression. The icon 613 is an image object for designating the maximum value of the morphing coefficient in the attack-based singing expression. The user can adjust the period during which the morphing coefficient takes the maximum value and the maximum value of the morphing coefficient by moving the icons 611 to 613 by operations such as drag and drop. The dial 614 is an image object for adjusting the shape of the curve (profile of the rate of increase of the morphing coefficient) from the start of application of the singing expression to the maximum of the morphing coefficient. When the dial 614 is operated, the curve from the start of application of the singing expression to the maximum of the morphing coefficient changes from, for example, a downwardly convex profile to a linear profile and then to an upwardly convex profile. The dial 615 is an image object for adjusting the shape of the curve (profile of the reduction rate of the morphing coefficient) from the end of the maximum period of the morphing coefficient to the end of application of the singing expression. The user can adjust the shape of the change curve of the morphing coefficient with time in the note by operating the dials 614 and 615. The UI unit 30 outputs the parameters specified by the graph of FIG. 21 to the synthesizer 20. The synthesizer 20 generates a synthetic singing to which the elements of the singing expression controlled by using these parameters are added. The “synthetic singing to which the elements of the singing expression controlled by the parameters are added” means, for example, the synthetic singing to which the elements processed by the process of FIG. 14 are added. As described above, this addition may be performed in the time domain or the frequency domain. The sound output means 33 of the UI unit 30 outputs the generated synthetic singing.

3. 3. Modifications The present invention is not limited to the above-mentioned mobile phones, and various modifications can be made. Some modifications will be described below. Two or more of the following modifications may be used in combination.

The target to which the singing expression is given is not limited to the singing voice, and may be a voice without singing. That is, the singing expression may be a voice expression. Further, the sound to which the voice expression is given is not limited to the synthetic sound synthesized by the computer device, and may be an actual human singing sound. Further, the object to which the singing expression is given may be a sound that is not based on the human voice.

The functional configuration of the voice synthesizer 1 is not limited to that illustrated in the embodiment. Some of the functions illustrated in the embodiments may be omitted. For example, the speech synthesizer 1 may omit at least some of the functions of the timing calculation means 21, the time expansion / contraction mapping means 22, and the short-time spectrum manipulation means 23.

The hardware configuration of the voice synthesizer 1 is not limited to that illustrated in the embodiment. The speech synthesizer 1 may have any hardware configuration as long as it can realize the required functions. For example, the voice synthesizer 1 may be a client device that cooperates with a server device on the network. That is, the function as the voice synthesizer 1 may be distributed to the server device on the network and the local client device.

The program executed by the CPU 101 or the like may be provided by a storage medium such as an optical disk, a magnetic disk, or a semiconductor memory, or may be downloaded via a communication line such as the Internet.

1 ... Speech synthesizer, 10 ... Database, 20 ... Synthesizer, 21 ... Timing calculation means, 22 ... Time expansion / contraction mapping means, 23 ... Short-time spectrum manipulation means, 24 ... Synthesis means, 25 ... Specific means, 26 ... Acquisition means , 30 ... UI unit, 31 ... Display means, 32 ... Reception means, 33 ... Sound output means, 101 ... CPU, 102 ... Memory, 103 ... Storage, 104 ... Input / output IF, 105 ... Display, 106 ... Input device, 911 ... Score display area, 912 ... Window, 913 ... Window, 2401 ... Spectral generation means, 2402 ... Inverse Fourier conversion means, 2403 ... Synthesis window application means, 2404 ... Overlap-add method, 2411 ... Spectrum generation means, 2412 ... Inverse Fourier conversion Means, 2413 ... Composite window application means, 2414 ... Overlap-add method, 2415 ... Singing synthesis means, 2416 ... Multiplication means, 2417 ... Multiplication means, 2418 ... Addition means

Claims (3)

A speech synthesis method performed by a computer device.
The step of acquiring the pitch of the synthetic sound to which no voice expression is given, and
A step of acquiring the spectrum entrainment outline of the elemental piece of the speech expression given to the synthetic sound and the pitch representing the elemental piece,
A step of shifting the pitch of the element so that the pitch representing the element and the pitch of the synthetic sound match.
By performing morphing using the spectral envelopment outline and the shifted pitch, the pitch-shifted element is added to the synthetic sound to obtain a synthetic sound to which the speech expression is added. A speech synthesis method with steps to generate. The element piece has a feature portion having a large fluctuation and a non-feature portion having a small fluctuation on the time axis.
The voice synthesis method according to claim 1, wherein the pitch of the element piece is determined based on the non-characteristic portion. The pitch of the element is shifted so that the pitch representing the element and the pitch of the synthetic sound match while maintaining the relative pitch change in the characteristic portion of the element. The voice synthesis method according to claim 2.

Images