JP6683103B2 - Speech synthesis method - Google Patents

Description

  The present invention relates to speech synthesis.

  A technique for synthesizing a voice such as a song is known. In order to generate a more expressive singing voice, it has been attempted not only to output a voice of a given lyrics in a given scale, but also to add a musical singing expression to this voice. Patent Document 1 discloses a technique of converting voice quality by adjusting two audio signals so that frequency bands of harmonic components are located close to each other.

JP, 2014-2338, A

  In the technique described in Patent Document 1, the singing expression given to the synthetic singing may not be sufficient. On the other hand, the present invention provides a technique for giving a richer voice expression.

  The present invention comprises a step of obtaining a temporal change of a spectral envelope outline used for synthesis of a synthetic voice, and a temporal change of the spectral envelope outline used for synthesis of an expression element of a voice expression given to the synthetic voice. And multiplying the temporal change of the spectrum envelope outline of the synthesized sound by a first coefficient, multiplying the time change of the spectrum envelope outline of the expression segment by a second coefficient, and adding both. And a step of generating a synthetic sound to which the speech expression is added, using a time change of the spectrum envelope outline obtained by addition.

  This speech synthesis method includes a step of acquiring a temporal change of a spectrum envelope used for synthesizing the synthesized sound, a step of acquiring a time change of the spectrum envelope used for synthesizing a voice expression, and the spectrum of the synthesized sound. Multiplying the temporal change of the envelope by the first coefficient, multiplying the temporal change of the spectral envelope of the expression element by the second coefficient, and adding both; and the spectral envelope outline obtained by the addition. And a step of generating a synthetic sound to which a voice expression is added, using the time change of the spectrum envelope.

  This speech synthesis method includes a step of obtaining a temporal fine variation of a spectrum envelope rough shape used for synthesis of a synthesized voice, and a time fine variation of the spectral envelope rough shape used for synthesis of a speech expression expression segment. A step of obtaining, a step of multiplying the temporal fine variation of the spectral envelope of the synthesized sound by the first coefficient, multiplying the spectral envelope of the expression element by the second coefficient, and adding both. Generating a synthesized sound to which a voice expression is added, using the temporal change of the spectrum envelope outline obtained by addition and the temporal fine variation of the spectrum envelope outline.

  This speech synthesis method includes a step of obtaining a pitch that is a reference of the expression segment, and the addition of the expression segment so that the pitch of the expression segment and the pitch of the synthesized speech match before addition. Shifting the pitch of the element.

  According to the present invention, richer voice expression can be given.

The figure which illustrates the GUI which concerns on related technology. The figure which shows the concept of singing expression provision which concerns on one Embodiment. The figure which illustrates the functional structure of the speech synthesizer 1 which concerns on one Embodiment. The figure which illustrates the hardware constitutions of the speech synthesizer 1. The schematic diagram which shows the structure of the database 10. The figure which illustrates the reference | standard time in the song expression of an attack reference | standard. The figure which illustrates the reference | standard time in the song expression of release reference | standard. The figure which illustrates the functional structure of the synthesizer 20. The figure which illustrates the mapping function in the example in which the time length of the segment of a singing expression is short. The figure which illustrates the mapping function in the example in which the time length of the segment of a singing expression is long. The figure which illustrates the relationship of a spectrum envelope and a spectrum envelope outline. The figure which illustrates the process which shifts the fundamental frequency of the element of a singing expression. The figure which illustrates the functional structure of the synthetic | combination means 24 for synthesize | combining in a frequency domain. 9 is a sequence chart illustrating the operation of the synthesizer 20. The figure which illustrates the functional structure of the synthetic | combination means 24 for synthesize | combining in a time domain. The figure which illustrates the function structure of UI part 30. The figure which illustrates the GUI used in the UI part 30. The figure which illustrates UI which selects a singing expression. The figure which shows another example of UI which selects a singing expression. An example of the table which matches the rotation angle of a dial and a morphing coefficient. Another example of UI for editing the parameter concerning a singing expression.

1. Speech synthesis technology Various techniques for speech synthesis are known. Singing (singing voice) is a voice that accompanies a change in scale and rhythm. As the singing voice synthesis, there are known segment-connecting singing voice synthesis and statistical singing voice synthesis. A database that contains a large number of singing pieces is used in the singing piece connection type singing synthesis. Singing pieces (an example of speech pieces) are mainly classified by phonemes (single phonemes or phoneme chains). In synthesizing a song, these song pieces are connected after the fundamental frequency, timing, and duration are adjusted according to the score information. The singing pieces used for the singing piece connection type singing voice synthesis are required to have a constant sound quality as much as possible over all phonemes registered in the database. This is because if the sound quality is not constant, an unnatural change in voice will occur when singing a song. In addition, a portion corresponding to a singing expression (an example of a voice expression) among the dynamic acoustic changes included in these pieces needs to be processed so as not to appear during synthesis. This is because the singing expression should be given to the singing depending on the musical context and should not be directly associated with the phonological type. If the same singing expression is always expressed for a particular phoneme, the resulting synthetic singing will be unnatural. Therefore, in the singing voice synthesis with segment connection, for example, changes in the fundamental frequency and the volume are generated based on the score information and a predetermined rule, instead of directly using the one included in the singing segment. Changes in fundamental frequency and volume are used. If singing pieces corresponding to all combinations of phonemes and singing expressions are recorded in the database, singing pieces corresponding to both phonological elements that match the score information and singing expressions that are natural to the musical context will be obtained. It becomes possible to choose. However, it takes a great deal of time and effort to record a singing element corresponding to any singing expression for any phoneme, and the capacity of the database also becomes enormous. Also, since the number of combinations of pieces increases explosively with respect to the number of pieces, it is difficult to guarantee that unnatural synthetic singing will not occur for every connection of pieces.

  On the other hand, in the statistical singing voice synthesis, a lot of training data is used to preliminarily learn the relationship between the score information and the acoustic features of the singing as a statistical model. At the time of synthesis, the most probable acoustic feature quantity is estimated from the input musical score information, and the song is synthesized using it. The statistical singing voice synthesis has an advantage that a statistical model including various singing expressions can be learned by constructing training data for each of various singing styles. However, there are two main problems with statistical singing synthesis. The first problem is over-smoothing. Since the process of learning a statistical model from a large number of training data essentially involves averaging and dimension reduction of the data, the acoustic features that are synthesized and output are inevitably more distributed than the usual single singing. Becomes smaller. As a result, the expressiveness and realism of the synthetic sound are impaired. The second problem is that the types of acoustic feature quantities that can learn the statistical model are limited. In particular, since phase information has a cyclical range, it is difficult to statistically model it. For example, the phase relationship between harmonic components or a specific harmonic component and its surrounding components and their temporal fluctuations can be calculated. It is difficult to model properly. However, in reality, in order to synthesize expressive songs including dull voices and hoarse voices, it is necessary to appropriately use the phase information.

  VQM (Voice Quality Modification) described in Patent Document 1 is known as a technique that enables various voice qualities to be synthesized in singing voice synthesis. In VQM, a first voice signal having a voice quality corresponding to a certain singing expression and a second voice signal by singing voice synthesis are used. The second audio signal may be based on voice synthesis with unit segment connection or may be based on statistical voice synthesis. By using these two audio signals, it is possible to synthesize a song including phase information. As a result, it is possible to synthesize a song that is more realistic and expressive than ordinary song synthesis. However, in this technique, it is not clear how to reflect the temporal change of the acoustic characteristics of the first audio signal in the singing voice synthesis. It should be noted that the term "time change" here does not mean a high-speed change in acoustic characteristics that is observed even when a voiced voice or hoarse voice is steadily uttered. This is a relatively macroscopic change in voice quality, in which the degree of rapid change is large, then gradually attenuates with the passage of time, and stabilizes to a certain degree with the passage of time. Such changes in voice quality greatly differ depending on the type of singing expression.

  FIG. 1 is a diagram illustrating a GUI according to related art. This GUI is used in a singing voice synthesis program according to related art. This GUI includes a score display area 911, a window 912, and a window 913. The score display area 911 is an area in which a score related to voice synthesis is displayed, and in this example, the score is represented in a format corresponding to what is called a piano roll. In the score display area 911, the horizontal axis represents time and the vertical axis represents scale. The window 912 is a pop-up window that is displayed according to a user's operation, and includes a list of song expressions that can be given to the synthetic song. The user selects the applicable singing expression from this list. In the window 913, a graph showing the degree of application of the selected song expression is displayed. In the window 913, the horizontal axis represents time and the vertical axis represents application strength of singing expressions. The user edits the graph in window 913 and inputs the time change of the degree of application of VQM. However, in this example, it is difficult to synthesize a natural and expressive song because the user has to manually input the time change of the degree of application of VQM.

2. Configuration FIG. 2 is a diagram showing a concept of giving a song expression according to an embodiment. In the following, "synthetic singing" is a synthesized voice, particularly a voice to which a scale and lyrics are given. Unless otherwise specified, the term “synthetic singing” simply refers to synthetic speech to which no singing expression according to the present embodiment is given. “Singing expression” refers to a musical expression given to synthetic speech, and includes expressions such as vocal fly (fry), growl (growl), and hoarse voice (rough). In the present embodiment, adding a segment (sample) of a pre-recorded local singing expression to a normal (singing expression not added) synthetic singing by morphing means “singing expression for synthetic singing”. Is given ”. Here, the segment of the singing expression is temporally local to the entire singing voice or one note. Temporally local means that the time occupied by the singing expression is partial with respect to the entire singing voice or one note. The singing expression fragment is a singing expression recorded by the singer in advance, and is a fragment of the singing expression (musical expression) performed at a local time during singing. The segment is a part of the voice waveform emitted by the singer. Morphing refers to a process of multiplying at least one of a singing expression segment and a synthetic singing by a coefficient that increases or decreases with the passage of time, and adds both. Morphing is performed at the same timing between the singing expression segment and the normal synthetic singing. In morphing, the temporal variation of acoustic features in singing expressions remains. When adding pieces of a singing expression by morphing, morphing is performed on a synthetic singing at a local time among ordinary synthetic singing.

  In this example, the reference time of addition of the synthetic singing and the singing expression segment is the start time of the note (that is, the note) and the end time of the note. Hereinafter, using the start time of the note as the reference time is referred to as “attack reference”, and using the end time as the reference time is referred to as “release reference”.

  FIG. 3 is a diagram illustrating a functional configuration of the voice synthesis device 1 according to the embodiment. The speech synthesizer 1 has a database 10, a synthesizer 20, and a UI (User Interface) unit 30. In this example, segment-connected singing voice synthesis is used. The database 10 is a database in which singing pieces and singing expression pieces are recorded. The synthesizer 20 reads a singing segment and a segment of the singing expression from the database 10 based on the score information and the information indicating the singing representation, and synthesizes the singing voice with the singing representation using these. The UI unit 30 is an interface for inputting, editing, and outputting score information, singing expression, and singing voice.

  FIG. 4 is a diagram illustrating a hardware configuration of the speech synthesizer 1. The speech synthesis device 1 is a computer device having a CPU (Central Processing Unit) 101, a memory 102, a storage 103, an input / output IF 104, a display 105, and an input device 106, specifically, for example, a tablet terminal. The CPU 101 is a control device that executes a program and controls other elements of the speech synthesis device 1. The memory 102 is a main storage device and includes, for example, a ROM (Read Only Memory) and a RAM (Random Access Memory). The ROM stores a program or the like for starting the speech synthesizer 1. The RAM functions as a work area when the CPU 101 executes the program. The storage 103 is an auxiliary storage device and stores various data and programs. The storage 103 includes, for example, at least one of an HDD (Hard Disk Drive) and an SSD (Solid State Drive). The input / output IF 104 is an interface for inputting / outputting information to / from another device, and includes, for example, a wireless communication interface or a NIC (Network Interface Controller). The display 105 is a device that displays information, and includes, for example, an LCD (Liquid Crystal Display). The input device 106 is a device for inputting information to the voice synthesizer 1, and includes, for example, at least one of a touch screen, a keypad, a button, a microphone, and a camera.

  In this example, the storage 103 stores a program that causes a computer device to function as the voice synthesis device 1 (hereinafter referred to as a “singing voice synthesis program”). The functions of FIG. 3 are implemented in the computer device by the CPU 101 executing the singing voice synthesis program. The storage 103 is an example of a storage unit that stores the database 10. The CPU 101 is an example of the synthesizer 20. The CPU 101, the display 105, and the input device 106 are an example of the UI unit 30. Details of the functional elements in FIG. 3 will be described below.

2-1. Database 10
The database 10 includes a database (single piece database) in which singing pieces are recorded and a database (singing expression database) in which pieces of singing expressions are recorded. Regarding the piece data base, conventionally known piece connection. Since it is the same as that used in type singing voice synthesis, detailed description will be omitted. Hereinafter, unless otherwise noted, the singing expression database is simply referred to as the database 10. In the database 10, in order to achieve both reduction of calculation load at the time of singing voice synthesis and prevention of acoustic feature estimation error, the acoustic features of the singing expression segments are estimated in advance, and the estimated acoustic features are estimated. Is preferably recorded in the database. The acoustic features recorded in the database 10 may be modified by human hands.

  FIG. 5 is a schematic diagram illustrating the structure of the database 10. In order to make it easy for the user or program to find the desired singing expression, the singing expression pieces are organized and recorded in the database 10. FIG. 5 shows an example of a tree structure. Each leaf in the tree structure corresponds to one singing expression. For example, "Attack-Fry-Power-High" means a singing expression that has a strong voice quality and is suitable for a high range, among the singing expressions based on the attack that mainly consists of fly vocalization. Singing expressions may be placed not only at the leaves at the end of the tree structure but also at the nodes. For example, a singing expression corresponding to "Attack-Fry-Power" may be recorded in addition to the above example.

  The database 10 stores at least one segment for each singing expression. Two or more pieces may be recorded depending on the phoneme. It is not necessary to record a unique piece of singing expression for every phoneme. This is because the segment of the singing expression is morphed as a synthetic singing, and the basic quality as a singing is already secured by the synthetic singing. For example, in order to obtain a good quality singing in singing voice synthesis with unit connected, it is necessary to record a unit for each phoneme of a two-phoneme chain (for example, a combination such as / a-i / or / a-o /). . However, as the singing expression segment, a unique one may be recorded for each single phoneme (for example, / a / or / o /), or the number of singing expressions may be further reduced, and the singing expression may be changed for each singing expression. Only one element (for example, only / a /) may be recorded. The number of pieces to be recorded for each singing expression is decided by the database creator in consideration of the balance between the man-hours for creating the singing expression database and the quality of the synthetic singing. In order to obtain a higher quality (realistic) synthetic singing, each phoneme contains a unique singing expression segment. In order to reduce the man-hours required to create a singing expression database, the number of pieces per singing expression is reduced.

  When two or more phonemes are recorded for one singing expression, it is necessary to define a mapping (association) between phonemes and phonemes. As an example, regarding a certain singing expression, the segment file "S0000" is mapped to the phonemes / a / and / i /, and the segment file "S0001" is mapped to the phonemes / u /, / e /, and / o /. To be done. Such mapping is defined for each singing expression. The number of pieces included in the database 10 may be different for each singing expression. For example, two pieces may be recorded for one singing expression, and five pieces may be recorded for another singing expression.

  In the database 10, information indicating a reference time (expression reference time) is recorded for each segment of the singing expression. This reference time is a characteristic point on the time axis in the waveform of the singing expression segment. The reference time includes at least one of a song expression start time, a song expression end time, a note-on set start time, a note offset start time, a note-on set end time, and a note offset end time.

  6 and 7 are diagrams illustrating each reference time. In this example, the time domain of the voice waveform of the singing expression segment is divided into a pre section, an onset section, a sustain section, an offset section, and a post section. These sections are divided by the creator of the database 10, for example. 6 shows an attack-based singing expression, and FIG. 7 shows a release-based singing expression.

  The attack-based singing expression is divided into a pre-section, an onset section, and a sustain section. The sustain section is a section in which acoustic characteristics (for example, fundamental frequency) are stable within a predetermined range. The fundamental frequency of the sustain section corresponds to the pitch of this singing expression. The onset section is a section before the sustain section and has acoustic characteristics that change with time. The pre-section is a section before the onset section. In the attack-based singing expression, the beginning of the pre-section is the singing expression start time. The start of the onset section is the note onset start time. The end of the onset section is the note onset end time. The end of the sustain section is the singing expression end time.

  The release-based singing expression is divided into a sustain section, an offset section, and a post section. The offset section is a section subsequent to the sustain section and has acoustic characteristics that change with time. The post section is a section subsequent to the offset section. The beginning of the sustain section is the singing expression start time. The end of the sustain section is the note offset start time. The end of the offset section is the note offset end time. The end of the post section is the singing expression end time.

  The database 10 records a template of parameters applied to singing voice synthesis. The parameters mentioned here include, for example, the time transition and application time of the morphing coefficient (application rate), and the speed of singing expression. For example, the database creator may create a plurality of templates, and the database creator may determine in advance which template is applied to each singing expression. That is, it may be predetermined which template is applied to which singing expression. Alternatively, the template itself may be included in the database 10 and the user may select which template to use during singing.

2-2. Synthesizer 20
FIG. 8 is a diagram illustrating a functional configuration of the combiner 20. The synthesizer 20 has a timing calculation means 21, a time expansion / contraction mapping means 22, a short-time spectrum manipulation means 23, a synthesis means 24, an identification means 25, and an acquisition means 26.

  The timing calculation means 21 calculates the timing (position on the time axis) at which the element of the singing expression and the synthetic singing coincide with each other using the reference time recorded for the element of the singing expression. For example, the timing calculation unit 21 matches the note-on-set start time (an example of the synthetic sound reference time) with the vowel starting time of the synthetic singing for the attack-based singing expression segment. For the fragment of the release-based singing expression, the note offset end time (another example of the synthetic sound reference time) is matched with the vowel ending time of the synthetic singing, or the singing expression ending time is pronounced in the synthetic singing. Match the end time.

  The time expansion / contraction mapping unit 22 calculates time expansion / contraction mapping of the singing expression segment (performs expansion processing on the time axis). Here, the time expansion / contraction mapping unit 22 calculates a mapping function indicating the correspondence of the time between the synthetic singing and the singing expression segment. The mapping function used here is a non-linear function in which the characteristics are divided for each reference time of the segment of the singing expression. By using such a function, the property of the singing expression included in the segment can be added to the synthetic singing without deteriorating as much as possible. The time expansion / contraction mapping unit 22 expands the time of the characteristic part of the singing expression segment by an algorithm different from that of the part other than the characteristic part (that is, by using a different mapping function). The characteristic portion is, for example, a pre-section and an onset section in the attack-based singing expression as described later.

  FIG. 9: is a figure which illustrates the mapping function in the example in which the time length of the segment of a singing expression is shorter than the synthetic singing. This is used, for example, when the attack-based singing expression is applied to a specific note when the duration of the singing expression segment is shorter than that of the synthetic singing. First, the basic idea of the mapping function will be explained. In the segment of singing expression, the pre-section and the onset section include many dynamic fluctuations of acoustic features as the singing expression. Therefore, if this section is expanded or contracted with time, the nature of the singing expression will change. Therefore, the time-varying mapping unit 22 obtains a desired time-varying mapping by extending the sustain period without performing time-varying in the pre-section and the onset period as much as possible.

  FIG. 9A shows an example of extending the time of the entire segment by slowing the slope of the mapping function in the sustain section, that is, by slowing the data reading speed of the segment of the singing expression. FIG. 9B shows an example in which the reading speed is kept constant even in the sustain section and the data reading position is repeatedly returned to the front to extend the time of the entire segment. This utilizes the characteristic that acoustic characteristics that are almost constant are maintained in the sustain section. At this time, it is preferable that the return time and the return time of the data read position correspond to the start position and the end position of the temporal periodicity appearing in the acoustic feature. By adopting such a data read position, it is possible to obtain a synthetic singing to which a natural singing expression is added. The start position and the end position can be obtained, for example, by obtaining an autocorrelation function with respect to the time series of the acoustic feature amount of the element of the singing expression, and adopting its peak. FIG. 9C shows an example in which a so-called random-mirror-loop is applied in the sustain section to extend the time of the entire segment. The random mirror loop is a method of prolonging the time of the entire segment by reversing the sign of the data reading speed many times during reading. In order to prevent generation of artificial periodicity that is not originally included in the singing expression sample, the time at which the code is inverted is determined based on a pseudo random number.

  9A to 9C show an example in which the data reading speed in the pre-section and the onset section is not changed, but the user may want to adjust the speed of singing expression. As an example, there is a case where the singing expression of "shakuri" is desired to be faster than the singing expression recorded as a segment. In such a case, the data read speed in the pre-section and the onset section may be changed. Specifically, if it is desired to be faster than the unit, the data reading speed is increased. FIG. 9D shows an example of increasing the data reading speed in the pre-section and the onset section. In the sustain section, the data reading speed is slowed down, and the time of the entire segment is extended.

  FIG. 10: is a figure which illustrates the mapping function in the example where the time length of the segment of a singing expression is longer than the synthetic singing. This is used, for example, when the attack-based singing expression is applied to a specific note when the duration of the singing expression segment is longer than that of the synthetic singing. In these examples as well, the time-varying mapping unit 22 obtains a desired time-varying mapping by shortening the sustain period without performing time-varying in the pre-section and the onset section as much as possible.

  FIG. 10A shows an example in which the time of the entire segment is shortened by making the slope of the mapping function steep in the sustain section, that is, by increasing the data reading speed of the segment of the singing expression. FIG. 10B shows an example in which the reading time is kept constant even in the sustain section, and the data reading is terminated in the middle of the sustain section to shorten the time of the entire segment. Since the phonological characteristics of the sustain section are constant, it is more natural to simply use the end of the segment without changing the data read rate and to obtain a natural synthesized singing. FIG.10 (c) shows the mapping function used when the time of a synthetic singing is shorter than the sum of the time lengths of the pre-section and the onset section of the singing expression segment. In this example, the time expansion / contraction mapping unit 22 increases the data read speed in the onset section so that the end of the onset section matches the end of the synthetic singing. FIG. 10D shows another example of the mapping function used when the time of the synthetic singing is shorter than the sum of the time lengths of the pre-section and the onset section of the singing expression segment. In this example, the time expansion / contraction mapping unit 22 shortens the time of the entire segment by stopping the data reading in the middle of the onset section while keeping the data reading speed constant in the onset section. Note that in the example of FIG. 10D, attention must be paid to the determination of the fundamental frequency. Since the pitch of the onset section is often different from the pitch of the note, unless the end of the onset section is used, the fundamental frequency of the synthetic singing does not reach the pitch of the note, and the sound seems to be out of sync ( In some cases, you may hear it. In order to avoid this, the time expansion / contraction mapping unit 22 determines a representative value of the fundamental frequency corresponding to the pitch of the note within the onset section, and the singing expression is made so that this fundamental frequency matches the pitch of the note. Shift the fundamental frequency of the entire unit. As the representative value of the basic frequency, for example, the basic frequency at the end of the onset section is used.

  Although FIG. 9 and FIG. 10 exemplify the time stretch mapping for the attack-based singing expression, the idea is the same for the time-stretch mapping for the release-based singing expression. That is, in the release-based singing expression, the offset section and the post section are the characteristic portions, and the time extension mapping is performed by an algorithm different from the other portions.

The short-time spectrum manipulation means 23 decomposes the short-time spectrum of the singing expression segment into several components (acoustic features). The short-time spectrum manipulating unit 23 morphs a part of the components obtained by the decomposition with respect to the same component of the synthetic singing voice to obtain a series of the short-time spectrum of the synthetic singing voice to which the singing expression is added. The short-time spectrum manipulation unit 23 decomposes the short-time spectrum of the singing expression element into, for example, one or more of the following components.
(A) Spectral envelope (b) Spectral envelope rough shape (c) Phase spectral envelope (d) Temporal fine fluctuation of spectral envelope (or harmonic amplitude) (e) Temporal fineness of phase spectral envelope (or harmonic phase) Variation (f) fundamental frequency In order to independently morph these components between the singing expression segment and the synthetic singing, the above decomposition also needs to be performed on the synthetic singing. In the combiner, these pieces of information may be generated in the middle of the combination, so that it may be used. Each component will be described below.

  Spectral envelope is a general shape of the amplitude spectrum and mainly relates to the perception of phoneme and individuality. Many methods for estimating the spectrum envelope have been proposed, and for example, estimation using a low-order cepstrum coefficient can be used. In this embodiment, it is of special significance to treat the spectrum envelope independently of other components. That is, even if a singing expression segment whose phonological or individuality is different from that of the synthetic singing is used, if the morphing application rate regarding the spectrum envelope is zero, the phonological and individuality of the synthetic singing appears 100%. Therefore, it is possible to divert a phoneme or a segment of a singing expression having a different individuality (for example, the other phoneme of the person or a segment of a completely other person). In addition, in a singing expression that intentionally changes the phoneme or individuality, this component may be independently morphed to control the degree.

  The spectrum envelope outline is a rough representation of the amplitude spectrum envelope, and mainly relates to the brightness of the voice. The spectral envelope rough shape can be obtained by various methods, for example, by a cepstrum coefficient of a lower order than the spectral envelope. Unlike the spectral envelope, the spectral envelope outline contains very little phoneme or personality information. Therefore, even if the spectral envelope morphing is not performed, the brightness of the voice included in the singing expression and its temporal movement can be retained by performing the morphing only on the spectral envelope rough formation.

  The phase spectrum envelope is an outline of the phase spectrum. The phase spectrum envelope can be obtained by various methods. For example, a short-time spectrum is analyzed at frame intervals that are synchronized with the signal cycle, and then only the phase value of each harmonic component is adopted, unwrapped at this stage, and the frequency (harmonic (Between waves and harmonics), a phase spectrum envelope can be obtained by performing nearest neighbor interpolation, linear or higher-order curve interpolation, and the like, rather than a simple phase spectrum.

  FIG. 11: is a figure which illustrates the relationship of a spectrum envelope and a spectrum envelope outline. The temporal variation of the spectrum envelope and the temporal variation of the phase spectrum envelope correspond to a fast varying component in the voice spectrum in a very short time, and correspond to the harshness of a dull voice or hoarse voice. The temporal fine fluctuation of the spectral envelope is obtained by taking the difference on the time axis with respect to these estimated values, or by taking the difference between these values smoothed within a certain time interval and the value in the frame of interest. be able to. The temporal variation of the phase spectrum envelope is calculated by taking the difference on the time axis with respect to the phase spectrum envelope, or by taking the difference between these values smoothed within a certain time interval and the value in the frame of interest. Fine variations can be obtained. All of these processes correspond to some sort of high-pass filter.

If both spectral envelopes and spectral envelope outlines are used as acoustic features, the actual morphing is not the spectral envelope itself (eg FIG. 11) but
Two acoustic features are used: (a ') the difference between the spectral envelope and the spectral envelope, and (b) the spectral envelope. For example, when the spectrum envelope and the spectrum envelope outline are separated as shown in FIG. 11, the spectrum envelope also includes the information of the spectrum envelope outline, so that the two are treated separately. With this separation, information about absolute loudness is included in the spectral envelope outline. When varying the strength of a human voice, while individuality and phonology can be maintained to some extent, the volume and the overall slope of the spectrum often change simultaneously. It is natural to include.

  Note that the harmonic amplitude and the harmonic phase may be used instead of the spectrum envelope and the phase spectrum envelope. The choice of using spectral envelopes and phase spectral envelopes or harmonic amplitudes and phases depends on the choice of synthesis scheme. Spectral envelopes and phase spectral envelopes are used when pulse trains or time-varying filters are combined, and harmonic amplitudes and phases are used in sinusoidal model-based combining schemes such as SMS, SPP, or WBHSM. To use.

  The fundamental frequency mainly relates to pitch perception. Unlike other acoustic features, the fundamental frequency cannot be obtained by simple interpolation with the conversion application rate. This is because the pitch of the note in the singing voice segment and the pitch of the note in the synthetic singing voice are generally different, and the basic frequency of the singing voice and the basic frequency of the synthetic singing are simply interpolated. However, the pitch will be completely different from the pitch to be synthesized. Therefore, in the present embodiment, the short-time spectrum operation unit 23 firstly sets the basics of the entire singing expression unit so that the pitch of the note given to the singing expression unit matches the pitch of the synthetic singing note. Shift the frequency by a fixed amount. This processing does not match the fundamental frequency of each element of the singing expression with the synthesized sound, but the dynamic fluctuation of the fundamental frequency included in the element of the singing expression is retained.

  FIG. 12: is a figure which illustrates the process which shifts the fundamental frequency of the segment of a singing expression. In FIG. 12, the broken line shows the characteristic of the segment of the singing expression before the shift (that is, recorded in the database 10), and the solid line shows the characteristic after the shift. In this process, the entire characteristic curve of the segment is shifted as it is so that the fundamental frequency of the sustain section becomes a desired frequency while the fluctuation of the fundamental frequency in the pre-section and the onset section is maintained. When applying the parameter of the application rate of the singing expression to the fundamental frequency, the short-time spectrum manipulation means 23 interpolates the fundamental frequency obtained by this processing and the fundamental frequency in the normal singing synthesis at each time.

  The synthesizing unit 24 synthesizes the synthetic singing song and the singing expression segment to obtain a synthetic singing song to which the singing expression is added. There are various methods for synthesizing a synthetic singing song and a singing expression segment, and finally obtaining a waveform in the time domain. These methods are roughly classified into two types depending on the representation method of the input spectrum. it can. One is a method based on harmonic components, and the other is a method based on spectral envelopes.

  As a synthesis method based on harmonic components, for example, SMS is known (Serra, Xavier, and Julius Smith. "Spectral modeling synthesis: A sound analysis / synthesis system based on a deterministic plus stochastic decomposition." Computer Music Journal 14.4 (1990): 12-24.). The spectrum of voiced sound is represented by the frequency, amplitude, and phase of sinusoidal components at the fundamental frequency and frequencies that are approximately integral multiples thereof. When a spectrum is generated by SMS and an inverse Fourier transform is performed, a waveform for several cycles multiplied by a window function is obtained. After the window function is divided, only the vicinity of the center of the combined result is cut out by another window function, and the result is superimposed and added to the output result buffer. By repeating this process for each frame interval, a long-time continuous waveform can be obtained.

  As a synthesis method based on the spectrum envelope, for example, NBVPM (Bonada, Jordi. "High quality voice transformations based on modeling radiated voice pulses in frequency domain." Proc. Digital Audio Effects (DAFx). 2004.) is known. In this example, the spectrum is represented by an amplitude spectrum envelope and a phase spectrum envelope, and does not include frequency information of the fundamental frequency and harmonic components. Inverse Fourier transform of this spectrum gives a pulse waveform corresponding to one cycle of vocal cord vibration and a vocal tract response thereto. This is superimposed and added to the output buffer. At this time, if the phase spectrum envelopes in the spectra of adjacent pulses are approximately the same value, the reciprocal of the time interval to be superimposed and added to the output buffer becomes the final fundamental frequency of the synthesized voice.

  There are two methods for synthesizing the singing voice and the singing expression in the frequency domain and the time domain. Whichever method is used, the synthesis of the singing voice and the singing expression is basically performed by the following procedure. First, the singing voice and the singing expression are morphed with respect to the components other than the temporal fine fluctuation component of the amplitude and the phase. Next, a synthetic singing to which a singing expression is added is generated by adding temporal fine fluctuation components of amplitude and phase of each harmonic component (or its peripheral frequency band).

  In addition, when synthesizing the singing voice and the singing expression, a time expansion / contraction mapping different from the other components may be used only for the temporal fine variation component. This is effective in the following two cases, for example.

  First, there is a case where the user intentionally changes the speed of singing expression. The temporal fine fluctuation component is such that the speed and periodicity of the fluctuation are deeply related to the nature of the texture of the voice such as “ragged”, “rigid”, or “shwashwa”. The nature of the texture changes. For example, when the user inputs an instruction to increase the speed in a singing expression in which the pitch decreases at the end as shown in FIG. 7, the user specifically decreases the pitch while changing the tone color and the texture. Although it has the intention of speeding up the speed of, it is presumed that it does not intend to change the nature of the texture of the singing expression. Therefore, in order to obtain the singing expression as intended by the user, it is sufficient to increase the data reading speed in the post section by linear time expansion / contraction for components such as the fundamental frequency and spectral envelope, but it is appropriate for the temporal fine fluctuation component. It may be looped in a cycle (similar to the sustain section in FIG. 9B) or a random mirror loop (similar to the sustain section in FIG. 9C).

  Second, there is a case of synthesizing a singing expression in which the fluctuation cycle of the temporal fine fluctuation component depends on the fundamental frequency. It has been empirically known that, in a singing expression having a periodic modulation in the amplitude and phase of a harmonic component, it may sound more natural to correlate the fluctuation period of the amplitude and phase with the fundamental frequency. There is. A singing expression having such a texture is called, for example, "rough" or "growl". As a method to correlate the fluctuation cycle of amplitude and phase with the fundamental frequency, the same ratio as the conversion ratio of the fundamental frequency applied when synthesizing the singing expression waveform is applied to the data reading speed of the temporal fine fluctuation component. Can be used.

  The synthesizing unit 24 synthesizes the synthesized singing voice and the singing expression waveform. That is, the synthesizing unit 24 gives a singing expression to the synthetic singing. The synthesis of the synthetic singing and the singing expression waveform is performed by using at least one of the acoustic features (a) to (f) described above. Which of the acoustic features (a) to (f) is used is set for each singing expression. For example, the singing expression of crescendo or decrescendo in the musical term is mainly related to a temporal change in vocal strength. Therefore, the main acoustic feature to be morphed is the spectral envelope shape. Phonology and personality are not considered to be the major acoustic features that make up the crescendo or decrescendo. Therefore, if the morphing application amount (coefficient) of the spectrum envelope is set to be zero, the fragment of the crescendo singing expression recorded from the singing of only one phoneme of one singing person can be used for all singing singers. It can also be applied to all phonemes of. In another example, in a singing expression such as vibrato, the fundamental frequency fluctuates periodically, and the volume fluctuates in synchronization with it. Therefore, the acoustic features to be morphed are the fundamental frequency and the spectral envelope shape.

  Further, since the spectrum envelope is an acoustic feature related to the phoneme, it is possible to add a singing expression that does not depend on the phoneme by excluding the spectrum envelope from the morphing target. For example, even for a singing expression in which a segment is recorded only for a certain specific phoneme (for example, / a /), by excluding the spectrum envelope from the morphing target, a synthetic singing of a phoneme other than the specific phoneme is performed. Can also morph the fragment of the singing expression.

  In this way, the acoustic features to be morphed can be limited for each type of singing expression. In this way, the acoustic features targeted for morphing may be limited, or all acoustic features may be treated as morphing regardless of the type of singing expression. When many acoustic features are targeted for morphing, a synthetic singing similar to the original singing expression segment is obtained, and the naturalness of that part is improved. However, since the difference in sound quality from the part to which the singing expression is not added becomes large, there is a possibility that an uncomfortable feeling may appear when listening through the entire singing. Therefore, when the acoustic features to be morphed are templated, the acoustic features to be morphed are determined in consideration of the balance between naturalness and discomfort.

  FIG. 13: is a figure which illustrates the more detailed functional structure of the synthetic | combination means 24 for synthesize | combining a singing voice and the element of a singing expression in a frequency domain. In this example, the synthesizing unit 24 has a spectrum generating unit 2401, an inverse Fourier transforming unit 2402, a synthesizing window applying unit 2403, and a superposition adding unit 2404.

  FIG. 14 is a sequence chart illustrating the operation of the combiner 20. In step S1400, the specifying unit 25 specifies a segment used for generating a synthetic singing song and a segment used for assigning a singing expression from the segment database and the singing expression database included in the database 10. The spectrum generation unit 2401 identifies these pieces based on the information supplied from the UI unit 30.

  In step S1401, the acquisition unit 26 acquires the time change of the acoustic feature used to generate the synthetic singing voice. The acoustic features acquired here are the spectrum envelope H (f), the spectrum envelope outline G (f), the phase spectrum envelope P (f), the temporal fine variation I (f) of the spectrum envelope, and the phase spectrum envelope. At least one of the temporal fine variation Q (f) and the fundamental frequency F0 is included. The acquisition unit 26 acquires these acoustic features from the short-time spectrum manipulation unit 23 that has processed the segment identified in step S1400, for example.

  In step S1402, the acquisition unit 26 acquires the time change of the acoustic feature used for adding the singing expression. The acoustic features obtained here are the same as those used to generate the synthetic singing. When distinguishing the acoustic features of the synthetic singing from the acoustic features of the singing expression, the subscript v is added to the acoustic features of the synthetic singing, the subscript p is added to the acoustic features of the singing expression, and the singing expression is added. To the subscript vp. The acquisition unit 26 acquires these acoustic features from the short-time spectrum manipulation unit 23 that has processed the segment identified in step S1400, for example.

  In step S1403, the acquisition unit 26 acquires the reference time set for the assigned singing expression segment. As described above, the reference time acquired here is at least one of the song expression start time, the song expression end time, the note-on set start time, the note offset start time, the note-on set end time, and the note offset end time. Including one.

  In step S1404, the timing calculation means 21 uses the reference time recorded for the singing expression segment to make the singing expression segment and the note (composite singing) coincide with each other (position on the time axis). ) Is calculated.

  In step S1405, the time expansion / contraction mapping unit 22 performs time expansion / contraction mapping on the singing expression segment according to the relationship between the time length of the target note and the singing expression fragment.

  In step S1406, the time expansion / contraction mapping unit 22 determines that the reference frequency F0v of the singing voice and the reference frequency F0p of the singing expression match (that is, the pitches of the two singing expressions match each other). Shift pitch.

In step S1407, the spectrum generation unit 2401 multiplies each of the synthetic singing and the singing expression by a morphing coefficient and then adds the acoustic singing. As an example, for the spectral envelope rough shape G (f), the spectral envelope H (f), and the temporal fine variation I (f) of the spectral envelope,
Gvp (f) = (1-aG) Gv (f) + aG · Gp (f) (1)
Hvp (f) = (1-aH) Hv (f) + aH · Hp (f) (2)
Ivp (f) = (1-aI) Iv (f) + aI · Ip (f) (3)
Morphs synthetic singing and singing expressions by. Note that aG, aH, and aI are morphing coefficients for the spectrum envelope outline G (f), the spectrum envelope H (f), and the temporal fine variation I (f) of the spectrum envelope, respectively. These may be set individually.

  In step S1408, the spectrum generation unit 2401 outputs the spectrum obtained by adding the acoustic features. When the spectrum is input, the inverse Fourier transform unit 2402 performs an inverse Fourier transform on the input spectrum (step S1409) and outputs a time domain waveform. When the time domain waveform is input, the synthesis window applying unit 2403 applies a predetermined window function to the inversely input waveform (step S1410) and outputs the result. The superposition addition unit 2404 superimposes and adds the waveform to which the window function is applied (step S1411). By repeating this processing for each frame interval, a long-time continuous waveform can be obtained.

  The method of synthesizing in the frequency domain has an advantage that the amount of calculation can be suppressed because it is not necessary to execute a plurality of synthesizing processes. However, in order to morph the minute fluctuation components of the amplitude and the phase, the singing voice synthesizing means (not shown in FIG. 13) must also use these acoustic feature quantities.

  FIG. 15 is a diagram illustrating a more detailed functional configuration of the synthesizing unit 24 for synthesizing a singing voice and a singing expression segment in the time domain. In this example, the synthesizing unit 24 has a spectrum generating unit 2411, an inverse Fourier transforming unit 2412, a synthesizing window applying unit 2413, a superposition adding unit 2414, a singing voice synthesizing unit 2415, a multiplying unit 2416, a multiplying unit 2417, and an adding unit 2418. .

  In this example, the spectrum generating means 2411 uses the spectrum envelope H (f) of the synthetic singing, the spectrum envelope outline G (f), the phase spectrum envelope P (f), and the fundamental frequency F0, and the segment of the singing expression. The temporal fine variation I (f) of the spectrum envelope and the fine temporal variation Q (f) of the phase spectrum envelope are input. The spectrum generation means 2411 obtains a spectrum from the input acoustic features.

  The inverse Fourier transform unit 2412 performs an inverse Fourier transform on the input spectrum to obtain a time domain waveform. The synthesis window applying means 2413 applies a predetermined window function to the waveform obtained by the inverse Fourier transform. The superposition adding means 2414 superimposes and adds the waveforms to which the window function is applied. By repeating this processing for each frame interval, a long-time continuous waveform can be obtained. This waveform shows a waveform of a fragment of a singing expression in which the fundamental frequency is shifted to the fundamental frequency of the synthetic singing.

  The singing voice synthesizing means 2415 receives the spectral envelope H (f), the spectral envelope outline G (f), the phase spectral envelope P (f), and the fundamental frequency F0 of the synthetic singing voice. The singing voice synthesizing means 2415 generates a time-domain waveform of the synthetic singing voice from these acoustic features using, for example, a known method.

  The multiplication means 2416 multiplies the output of the superposition addition means 2414 by the application coefficient a of the fine fluctuation component. The multiplying means 2417 multiplies the output of the singing voice synthesizing means 2415 by a coefficient (1-a). The adding means 2418 adds the output of the multiplying means 2416 and the output of the multiplying means 2417.

  As for the method of synthesizing in the time domain, only the portion (the right half of FIG. 15) for synthesizing the waveform of the singing expression handles the minute fluctuation component. According to this method, the singing voice synthesizing means 2415 does not need to be of the type that uses minute fluctuation components of amplitude and phase. In this case, in the singing voice synthesizer 2415, for example, SPP (Spectral Peak Processing) (Bonada, Jordi, Alex Loscos, and H. Kenmochi. "Sample-based singing voice synthesizer by spectral concatenation." Proceedings of Stockholm Music Acoustics Conference. 2003 .) Can be used. In SPP, a component corresponding to the texture of a voice is synthesized by the spectral shape around the harmonic peak instead of the temporal fine fluctuation. When adding a singing expression to the existing singing voice synthesizing means that employs such a method, it is easier to use the synthesizing method in the time domain in that the existing singing voice synthesizing means can be used as it is. In the case of synthesizing in the time domain, if the phases of singing voice synthesis and singing voice synthesis are different, the waveforms may cancel each other or a beat may occur. In order to prevent such a problem from occurring, it is necessary that the phase spectrum envelopes be the same in the combination of both, and the reference position (so-called pitch mark) of the voice pulse for each period be the same.

  In addition, since the value of the phase spectrum obtained by analyzing the voice by a short-time Fourier transform or the like generally has indefiniteness with respect to θ + n2π, that is, the integer n, morphing the phase spectrum envelope may be difficult. . Since the influence of the phase spectrum envelope on the perception of sound is smaller than that of other acoustic feature components, the phase spectrum envelope does not necessarily have to be interpolated, and an arbitrary value may be given. The simplest and most natural method for determining the phase spectrum envelope is to use the minimum phase calculated from the amplitude spectrum envelope. In this case, the spectral envelope H (f) + G (f) excluding the fine fluctuation component is first obtained from H (f) and G (f) in FIG. 13 or FIG. 15, and the minimum phase corresponding to this is obtained to obtain P ( f). As a method of calculating the minimum phase corresponding to an arbitrary amplitude spectrum envelope, for example, a method via a cepstrum (Oppenheim, Alan V., and Ronald W. Schafer. Discrete-time signal processing. Pearson Higher Education, 2010.) is used. be able to.

2-3. UI unit 30
2-3-1. Functional Configuration FIG. 16 is a diagram illustrating a functional configuration of the UI unit 30. The UI unit 30 has a display unit 31, a reception unit 32, and a sound output unit 33. The display unit 31 displays a UI screen. The accepting unit 32 accepts an operation via the UI. The sound output unit 33 outputs a synthetic singing voice in response to an operation received via the UI. As will be described later, the UI displayed by the display unit 31 includes, for example, an image object for simultaneously changing the values of a plurality of parameters used for composition of a singing expression given to a composition singing. The accepting means accepts an operation for this image object.

2-3-2. UI example (outline)
FIG. 17 is a diagram illustrating a GUI used in the UI unit 30. This GUI is used in the singing voice synthesis program according to one embodiment. This GUI includes a score display area 511, a window 512, and a window 513. The score display area 511 is an area in which a score related to singing voice synthesis is displayed, and in this example, the score is represented in a format corresponding to a so-called piano roll. In the score display area 511, the horizontal axis represents time and the vertical axis represents scale. In this example, image objects corresponding to the five notes 5111 to 5115 are displayed. Lyrics are assigned to each note. In this example, the lyrics “I”, “love”, “you”, “so”, and “much” are assigned to the notes 5111 to 5115. The user can add a new note at any position on the score by clicking on the piano roll. With respect to the notes set on the score, attributes such as the position on the time axis, the scale, or the length of the notes can be edited by an operation such as so-called drag and drop. As the lyrics, one song's worth of lyrics may be inputted in advance and automatically assigned to each note according to a predetermined algorithm, or the lyrics may be manually assigned to each note by the user.

  In each of the window 512 and the window 513, an image object indicating an operator for giving an attack-based singing expression and a release-based singing expression to one or more notes selected in the score display area 511 is displayed. Area. Selection of a note in the score display area 511 is performed by a predetermined operation (for example, clicking the left button of the mouse).

2-3-3. UI example (selection of singing expression)
FIG. 18 is a diagram illustrating a UI for selecting a singing expression. This UI uses a pop-up window. When the user performs a predetermined operation (for example, clicking the right button of the mouse) on the note to which the singing expression is given, a pop-up window 514 is displayed. The pop-up window 514 is a window for selecting the first layer of the song expressions organized in a tree structure, and includes a display of a plurality of options. When the user performs a predetermined operation (for example, clicking the left button of the mouse) on the first option among the plurality of options included in the pop-up window 514, the pop-up window 515 is displayed. The pop-up window 515 is a window for selecting the second hierarchy of organized song expressions. When the user operates the pop-up window 515 to select one option, the pop-up window 516 is displayed. The pop-up window 516 is a window for selecting the third hierarchy of organized song expressions. The UI unit 30 outputs information specifying the singing expression selected via the UI of FIG. 18 to the synthesizer 20. In this way, the user can select the desired singing expression from the organized structure.

  In the score display area 511, an icon 5116 and an icon 5117 are displayed around the note 5111. The icon 5116 is an icon (an example of an image object) for instructing to edit the attack-based singing expression, and the icon 5117 is an icon for instructing to edit the release-based singing expression. For example, when the user holds the mouse pointer on the icon 5116 and clicks the right mouse button, a pop-up window 514 for selecting the attack-based singing expression is displayed.

  FIG. 19 is a diagram showing another example of a UI for selecting a singing expression. In this example, an image object for selecting an attack-based singing expression is displayed in the window 512. Specifically, the window 512 displays a plurality of icons 5121. Each icon represents a singing expression. In this example, the database 10 stores 10 types of singing expressions, and the window 512 displays 10 types of icons 5121. The user selects one or more target notes in the score display area 511 and then selects an icon corresponding to the singing expression to be added from the icons 5121 of the window 512. Similarly for the release-based singing expression, the user selects the icon in the window 513. The UI unit 30 outputs information specifying the singing expression selected via the UI of FIG. 19 to the synthesizer 20. The synthesizer 20 generates a synthetic singing song to which a singing expression is added based on this information. The sound output unit 33 of the UI unit 30 outputs the generated synthetic singing voice.

2-3-4. UI example (parameter input of singing expression)
In the example of FIG. 19, an image object of the dial 5122 for changing the degree of attack-based singing expression is displayed in the window 512. The dial 5122 is an example of a single operator for simultaneously changing the values of a plurality of parameters used for giving a song expression given to a synthetic song. Further, the dial 5122 is an example of an operator that is displaced according to a user operation. In this example, by operating a single dial 5122, a plurality of parameters related to singing expressions are adjusted simultaneously. The degree of release-based singing expression is also adjusted via the dial 5132 displayed in the window 513 as well. The plurality of parameters related to the singing expression are, for example, the maximum value of the morphing coefficient of each acoustic feature. The maximum value of the morphing coefficient is the maximum value when the morphing coefficient changes with the passage of time in each note. In the example of FIG. 2, the attack-based singing expression has the maximum morphing coefficient at the beginning of the note, and the release-based singing expression has the maximum morphing coefficient at the end of the note. The UI unit 30 has information (for example, a table) for changing the maximum value of the morphing coefficient according to the rotation angle of the dial 5122 from the reference position.

  FIG. 20 is a diagram illustrating a table that associates the rotation angle of dial 5122 with the maximum value of the morphing coefficient. This table is defined for each song expression. A plurality of acoustic features (spectral envelope H (f), spectral envelope outline G (f), phase spectral envelope P (f), temporal fine variation I (f) of the spectral envelope, temporal fine variation of the phase spectral envelope For each of Q (f) and the fundamental frequency F0, the maximum value of the morphing coefficient is defined in association with the rotation angle of the dial 5122. For example, when the rotation angle is 30 °, the maximum value of the morphing coefficient of the spectrum envelope H (f) is zero, and the maximum value of the morphing coefficient of the spectrum envelope outline G (f) is 0.3. In this example, the value of each parameter is defined only for the discrete value of the rotation angle, but the value of each parameter is specified by interpolation for the rotation angle not defined in the table.

  The UI unit 30 detects the rotation angle of the dial 5122 according to the user's operation. The UI unit 30 specifies the maximum value of the morphing coefficient corresponding to the detected rotation angle with reference to the table of FIG. The UI unit 30 outputs the maximum value of the identified morphing coefficient to the synthesizer 20. The parameter relating to the singing expression is not limited to the maximum value of the morphing coefficient. Other parameters may be adjusted, such as the rate of increase or decrease of the morphing factor. The user selects, on the score display area 511, which song expression part of which note is to be edited. At this time, the UI unit 30 sets the table corresponding to the selected singing expression as the table referred to according to the operation of the dial 5122.

  FIG. 21 is a diagram showing another example of the UI for editing the parameter relating to the singing expression. In this example, the shape of the graph showing the time change of the morphing coefficient applied to the acoustic feature of the singing expression for the note selected in the score display area 511 is edited. The song expression to be edited is designated by the icon 616. The icon 611 is an image object for designating the start of a period in which the morphing coefficient takes the maximum value in the attack-based singing expression. The icon 612 is an image object for designating the end of the period in which the morphing coefficient takes the maximum value in the attack-based singing expression. The icon 613 is an image object for designating the maximum value of the morphing coefficient in the attack-based singing expression. The user can adjust the period during which the morphing coefficient has the maximum value and the maximum value of the morphing coefficient by moving the icons 611 to 613 by an operation such as drag and drop. The dial 614 is an image object for adjusting the shape of the curve (the profile of the increasing rate of the morphing coefficient) from the start of applying the singing expression to the maximum morphing coefficient. When the dial 614 is operated, the curve from the start of applying the singing expression to the maximum morphing coefficient changes from a downward convex profile to a linear profile to an upward convex profile. The dial 615 is an image object for adjusting the shape of the curve (the profile of the morphing coefficient decrease rate) from the end of the maximum period of the morphing coefficient to the end of application of the singing expression. By operating the dials 614 and 615, the user can adjust the shape of the change curve of the morphing coefficient with time in the note. The UI unit 30 outputs the parameters specified by the graph of FIG. 21 to the synthesizer 20. The synthesizer 20 generates a synthetic singing to which the singing expression segment controlled by using these parameters is added. The “composite singing in which the pieces of the singing expression controlled by using the parameters are added” means, for example, a synthetic singing in which the pieces processed by the process of FIG. 14 are added. As described above, this addition may be performed in the time domain or the frequency domain. The sound output unit 33 of the UI unit 30 outputs the generated synthetic singing voice.

3. MODIFIED EXAMPLE The present invention is not limited to the above-described embodiment, and various modified implementations are possible. Hereinafter, some modified examples will be described. Two or more of the following modifications may be used in combination.

  The target to which the singing expression is given is not limited to the singing voice, and may be a voice without singing. That is, the singing expression may be an audio expression. Further, the sound to which the voice expression is added is not limited to the synthesized sound synthesized by the computer device, and may be an actual human singing voice. Furthermore, the target to which the singing expression is added may be a sound that is not based on a human voice.

  The functional configuration of the voice synthesizer 1 is not limited to the one exemplified in the embodiment. Some of the functions illustrated in the embodiments may be omitted. For example, at least a part of the functions of the timing calculation unit 21, the time expansion / contraction mapping unit 22, and the short time spectrum operation unit 23 may be omitted in the speech synthesis device 1.

  The hardware configuration of the speech synthesizer 1 is not limited to the one exemplified in the embodiment. The voice synthesizer 1 may have any hardware configuration as long as it can realize the required function. For example, the speech synthesizer 1 may be a client device that cooperates with a server device on the network. That is, the function of the voice synthesizer 1 may be distributed to the server device and the local client device on the network.

  The program executed by the CPU 101 or the like may be provided by a storage medium such as an optical disk, a magnetic disk, or a semiconductor memory, or may be downloaded via a communication line such as the Internet.

DESCRIPTION OF SYMBOLS 1 ... Speech synthesizer, 10 ... Database, 20 ... Synthesizer, 21 ... Timing calculation means, 22 ... Time expansion / contraction mapping means, 23 ... Short-time spectrum operating means, 24 ... Synthesizing means, 25 ... Specification means, 26 ... Acquisition means , 30 ... UI unit, 31 ... Display means, 32 ... Receiving means, 33 ... Sound output means, 101 ... CPU, 102 ... Memory, 103 ... Storage, 104 ... Input / output IF, 105 ... Display, 106 ... Input device, 911 ... Score display area, 912 ... Window, 913 ... Window, 2401 ... Spectrum generation means, 2402 ... Inverse Fourier transform means, 2403 ... Synthesis window application means, 2404 ... Superposition addition means, 2411 ... Spectrum generation means, 2412 ... Inverse Fourier transform Means, 2413 ... Composition window applying means, 2414 ... Superposition adding means, 2415 ... Singing Forming means, 2416 ... multiplication means, 2417 ... multiplication means, 2418 ... adding means

Claims (3)

Obtaining a temporal change of a spectral envelope outline used for synthesis of a synthetic sound,
Obtaining a temporal change of the spectrum envelope outline used for synthesizing an expression segment of a voice expression given to the synthesized sound;
Multiplying the time change of the spectrum envelope outline of the synthesized sound by a first coefficient, multiplying the time change of the spectrum envelope outline of the expression element by a second coefficient, and adding both.
Using the temporal change of the spectrum envelope outline obtained by addition, and generating a synthesized voice to which the voice expression is added. Obtaining a temporal change of a spectral envelope used for the synthesis of the synthetic sound,
Obtaining a temporal change of the spectral envelope used for synthesizing a speech expression,
Multiplying a temporal change of the spectral envelope of the synthesized sound by a third coefficient, multiplying a temporal change of the spectral envelope of the expression element by a fourth coefficient, and adding both.
Generating a synthesized speech to which a speech expression is added, using the spectral envelope outline obtained by addition and the temporal change of the spectral envelope. Acquiring a reference pitch of the expression element,
Prior to the addition, a step of shifting the pitch of the expression element so that the pitch of the expression element and the pitch of the synthesized sound coincide with each other. The speech synthesis method described in.

Images