JP6167503B2 - Speech synthesizer - Google Patents

Description

  The present invention relates to speech synthesis technology, and more particularly to real-time speech synthesis technology.

  A speech synthesis technique for synthesizing a voice signal representing a guidance voice in voice guidance, a reading voice of a literary work, or a singing voice of a song by electrical signal processing using a plurality of types of synthesis information has become widespread. For example, in the case of synthesis of singing voice, prosodic information (for example, musical note information indicating the pitch or duration of each note constituting the melody of the singing song) indicating prosody change in the singing voice synthesis target song Music expression information such as information representing the phoneme string of the lyrics of the song is used as the synthesis information. When synthesizing the voice signal of the guidance voice in the voice guidance or the reading voice of the literary work, information indicating the phonological sequence of the sentence of the guidance sentence or the literary work and the prosody information indicating the prosody change such as intonation and accent are combined information. Used as Conventionally, in this type of speech synthesis, various types of synthesis information over the entire speech to be synthesized are input to a speech synthesizer in advance, and a speech signal representing the sound waveform of the entire speech to be synthesized is based on the synthesis information. A so-called batch processing method in which batch generation is performed is common. However, in recent years, a real-time speech synthesis technique has also been proposed (see, for example, Patent Document 1).

  As an example of real-time speech synthesis, information indicating the phonological sequence of the lyrics of the entire song is input to the singing synthesizer in advance, and the sound when the lyrics are pronounced for each note by operating the keyboard simulating a piano keyboard There is a technique for synthesizing a singing voice for each note by sequentially inputting note information representing a height or duration. Also, in recent years, a singing in which a phoneme information input unit in which operators for inputting each phoneme (consonant and vowel) constituting a phoneme string of lyrics are arranged and a note information input unit imitating a piano keyboard are arranged side by side. Using the synthesis keyboard, the user can sequentially input the note information for each note and the phoneme sequence information indicating the phoneme sequence of the lyrics to be pronounced according to the note, and synthesize the singing voice for each note. Proposed.

Japanese Patent No. 3879402

  Some electronic keyboard instruments, such as an electronic piano, can specify the sound intensity (velocity) of each note by the key-pressing speed. It can be carried out. Some singing synthesis keyboards allow the velocity to be specified by the key-pressing speed. However, in the case of singing voice, it is often impossible to obtain sufficient expressive power simply by changing the sound intensity for each note. The same applies to real-time synthesis of guidance voices and reading voices.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique that makes it possible to synthesize a speech having a richer expressiveness than before in a real-time manner.

  In order to solve the above problem, the present invention changes phonological sequence information indicating a phonological sequence of a speech to be synthesized, prosodic information indicating a prosodic change of the speech, and a part of the phonological sequence indicated by the phonological sequence information. And at least one of the phoneme sequence information, the prosodic information, and the phoneme control information is acquired by operating an operator. A sound obtained by changing a part of the phoneme of the phoneme sequence indicated by the phoneme sequence information included in the plurality of types of synthesis information acquired by the acquisition unit according to the phoneme control information included in the plurality of types of synthesis information There is provided a speech synthesizer characterized by comprising speech synthesis means for performing synthesis.

  In the speech synthesizer of the present invention, speech synthesis by the speech synthesizer is performed when a plurality of types of synthesis information is acquired by the acquisition unit. Since at least one of phonological sequence information, prosodic information, and phonological control information among the plurality of types of synthetic information is acquired by operation of an operator, real-time speech synthesis is performed in the speech synthesizer of the present invention. Done. The speech synthesizer of the speech synthesizer of the present invention performs speech synthesis by changing a part of the phoneme of the phoneme sequence indicated by the phoneme sequence information according to the phoneme control information. Here, when the synthesis target is Japanese singing voice and the phoneme string represented by the phoneme string information is “consonant + vowel”, a specific example of the process of changing a part of the phoneme string is continuation of consonant Processing to change the length (for example, shortening), processing to remove consonants, processing to replace consonants with other consonants that approximate the sound, and generate consonant repetition (ie, one or more consonants before vowels) Insertion) and the like. When singing synthesis is performed by changing a part of the phoneme of the phoneme string by these processes, the consonant is less easily heard in the synthesized singing voice. Since a person's singing voice and reading voice tend to be difficult to listen to consonants as the pronunciation becomes quicker (the ease of listening to consonants decreases), according to the present invention, It is possible to reproduce the tendency peculiar to reading speech, and it is possible to synthesize speech with richer expressiveness than before in a real-time manner. In addition, when the synthesis target is a singing voice of a language other than Japanese, such as English, a consonant may be located at the end of the lyrics sung along with the note. , A process of changing the duration, a process of deleting the consonant, a process of replacing the consonant with another consonant that approximates the sound, and generating a repetition of the consonant (for example, the consonant between the consonant and the preceding phoneme) (One or a plurality of insertions) may be performed.

  In a more preferable aspect, an aspect in which the speech synthesizer executes a process of changing the duration of the consonant at a frequency according to the phoneme control information, an adjustment amount of the consonant duration, the number of missing or inserted consonants, and replacement An aspect is conceivable in which the number of consonants is a variable value corresponding to the phoneme control information. According to such an aspect, it becomes possible to finely control the ease of listening of consonants in the synthesized singing voice, and the expressive power of the synthesized singing voice can be further improved.

  When the present invention is applied to a singing voice synthesizing device that synthesizes a singing voice based on a plurality of types of synthesis information, a singing voice synthesis keyboard is used as an input device for allowing a user to input phoneme string information and prosodic information. Just do it. In this case, a series of note information input using the singing synthesis keyboard plays the role of prosodic information. Also, a dedicated operator for inputting phonological control information may be provided in either the phonological information input unit or the musical note information input unit so that the phonological control information can be input using the singing synthesis keyboard. For example, an operator for inputting note information may be assigned the role of inputting phonological control information. Specifically, it is conceivable that the velocity corresponding to the key pressing speed when the pitch is designated plays a role of the phoneme control information.

  As another preferred mode, the plurality of types of synthesis information includes prosodic control information that instructs to synthesize speech by generating a shift from an ideal prosodic change, so that the shift occurs. In addition, a mode in which speech synthesis means executes a process of performing speech synthesis while adjusting prosodic changes according to prosodic control information is conceivable. Here, as a specific example of a method for causing a deviation from an ideal prosody change, a mode of changing the depth of attack (transient prosody change from consonant to vowel), a mode of changing the duration of the attack A mode in which a lack of attack is generated. As another specific example of the adjustment of prosody change, the depth of undershoot (prosody change from silence to consonant) or the height (or both) of overshoot (prosody change from consonant to vowel) is changed. Examples include an aspect, an aspect in which the duration of undershoot or overshoot (or both) is changed, and an aspect in which an undershoot or overshoot (or both) is missing. Thus, the ease of listening of consonants in the synthesized speech can also be adjusted by adjusting the prosody change.

It is a figure which shows the structural example of the song synthesizing | combining apparatus 1 of 1st Embodiment of this invention. It is a figure for demonstrating operation | movement of the song synthesizing | combining apparatus 1. FIG. It is a figure for demonstrating the aspect which changes the phoneme of a part of phoneme string by adjustment of the continuation length of a consonant. It is a figure for demonstrating the aspect which changes the phoneme of a part of phoneme string by generating the loss of a consonant. It is a figure for demonstrating the aspect which changes the one part phoneme of a phoneme string by replacement | exchange of a consonant. It is a figure for demonstrating the aspect which changes the one part phoneme of a phoneme string by insertion of a consonant. It is a figure for demonstrating the aspect which produces the shift | offset | difference from an ideal prosodic change by adjustment of the depth of an attack. It is a figure for demonstrating the aspect which produces the shift | offset | difference from an ideal prosodic change by adjustment of the continuation length of an attack. It is a figure for demonstrating the aspect which produces the shift | offset | difference from an ideal prosody change by missing an attack. It is a figure for demonstrating the aspect which produces the shift | offset | difference from an ideal prosodic change by adjusting the depth of an undershoot and the height of an overshoot. It is a figure for demonstrating the aspect which produces the shift | offset | difference from an ideal prosodic change by adjustment of the continuation length of an undershoot and an overshoot. It is a figure for demonstrating the aspect which produces the shift | offset | difference from an ideal prosodic change by missing undershoot and overshoot.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(A: 1st Embodiment)
FIG. 1 is a block diagram showing a configuration example of a singing voice synthesizing apparatus 1 according to an embodiment of the voice synthesizing apparatus of the present invention. The singing voice synthesizing apparatus 1 is a device that allows a user to input a plurality of types of synthetic information such as phoneme string information and prosodic information, and performs real-time singing synthesis using the synthetic information. As shown in FIG. 1, the singing voice synthesizing apparatus 1 includes a control unit 110, an operation unit 120, a display unit 130, an audio output unit 140, an external device interface (hereinafter abbreviated as “I / F”) unit 150, and a storage unit 160. , And a bus 170 that mediates data exchange between these components.

  The control unit 110 is, for example, a CPU (Central Processing Unit). The control unit 110 operates as a voice synthesis unit that synthesizes a singing voice based on the plurality of types of synthesis information by operating according to the song synthesis program stored in the storage unit 160. Details of processing executed by the control unit 110 in accordance with this singing synthesis program will be made clear later. In the present embodiment, a CPU is used as the control unit 110, but a digital signal processor (DSP) may of course be used.

  The operation unit 120 is the above-described singing synthesis keyboard, and includes a phoneme information input unit and a note information input unit. The user of the singing voice synthesizing device 1 can specify the phonological string of the lyrics that are pronounced in accordance with the notes and the notes constituting the melody of the song tune to be synthesized by operating the operation unit 120. For example, when the word “sa” is designated, an operator corresponding to the consonant “s” and an operator corresponding to the vowel “a” are sequentially selected from the plurality of operators provided in the phonological information input unit. When “C4” is designated as the pitch of the note corresponding to the lyrics, a key corresponding to the pitch is selected from a plurality of operators (keys) provided in the note information input unit. It is only necessary to instruct the start of sound generation by pressing the key and to end the sound generation by releasing the key. That is, the length of time that the key is pressed is the duration of the note. In addition, the user can specify the velocity at which the lyrics are pronounced in accordance with the note by the key pressing speed corresponding to the note. In addition, what is necessary is just to employ | adopt the thing in the conventional electronic keyboard musical instrument as a mechanism which makes it possible to input the note information containing designation | designated of a velocity by key pressing operation.

  When an operation for designating a phoneme sequence is performed, the operation unit 120 provides the control unit 110 with phoneme sequence information indicating the phoneme sequence. In addition, when a key pressing operation for instructing the start of sounding is performed, the operation unit 120 generates a note-on event (MIDI (Musical Instrument Digital Interface) event) corresponding to the pressed key for instructing the start of sounding. Information is given to the control unit 110. This note-on event includes information indicating the pitch corresponding to the pressed key and information indicating the velocity magnitude corresponding to the key pressing speed (an integer value of 1 to 127). Then, the operation unit 120 gives a note-off event (MIDI event) corresponding to the key to the control unit 110 as musical note information instructing the end of the sounding when the key depression is released. In this way, the note information input by operating the operator of the note information input unit plays the role of the prosodic information described above.

  Although details will be described later, the user of the singing voice synthesizing apparatus 1 can adjust (decrease) the ease of listening to the consonant sound in the synthesized singing voice by the key pressing speed of the key corresponding to the note. The adjustment of the ease of listening of the consonant is performed by changing a part of the phoneme of the phoneme sequence designated by the operation of the phoneme information input unit (that is, the phoneme sequence indicated by the phoneme sequence information input via the operation unit 120). It is realized by changing the song composition. In the present embodiment, the velocity included in the note information instructing the start of pronunciation includes the role of phonological control information instructing to perform singing synthesis by changing a part of the phoneme sequence indicated by the phoneme sequence information. Is given. That is, the operation unit 120 serves as an acquisition unit for causing the control unit 110 to acquire a plurality of types of synthesis information (in this embodiment, phoneme string information, prosodic information, and phoneme control information) used for synthesis of the singing voice. .

The display unit 130 is, for example, a liquid crystal display and a driving circuit thereof, and displays various images such as a menu image that prompts the use of the singing voice synthesizing device 1 under the control of the control unit 110. As shown in FIG. 1, the audio output unit 140 includes a D / A converter 142, an amplifier 144, and a speaker 146. The D / A converter 142 performs D / A conversion on the digital audio data (audio data representing the sound waveform of the synthesized singing voice) given from the control unit 110, and gives an analog audio signal as a conversion result to the amplifier 144. . The amplifier 144 amplifies the signal level (that is, the volume) of the audio signal supplied from the D / A converter 142 to a level suitable for driving the speaker and supplies the amplified signal to the speaker 146. The speaker 146 outputs the audio signal given from the amplifier 144 as sound.

  The external device I / F unit 150 is a collection of interfaces for connecting other external devices to the song synthesizer 1, such as a USB (Universal Serial Bus) interface and an audio interface. In the present embodiment, the case where the singing voice synthesizing keyboard (operation unit 120) and the voice output unit 140 are components of the singing voice synthesizing device 1 will be described. However, the singing voice synthesizing keyboard and the voice output unit 140 are connected to the external device I / F. Of course, an external device connected to the unit 150 may be used.

  The storage unit 160 includes a nonvolatile storage unit 162 and a volatile storage unit 164. The nonvolatile storage unit 162 is configured by a nonvolatile memory such as a ROM (Read Only Memory), a flash memory, or a hard disk, for example. The volatile storage unit 164 is configured by a volatile memory such as a RAM (Random Access Memory), for example. Has been. The volatile storage unit 164 is used by the control unit 110 as a work area when executing various programs. On the other hand, as shown in FIG. 1, the non-volatile storage unit 162 stores a song synthesis library 162a and a song synthesis program 162b in advance.

  The singing synthesis library 162a is a database storing segment data representing speech waveforms of various phonemes and diphones (transitions from phonemes to different phonemes (including silence)). Note that the singing synthesis library 162a may be a database storing triphone segment data in addition to a monophone or a diphone, and may also be a phoneme stationary part of a speech waveform or a transition part (transient part to another phoneme). Part) may be stored in the database. The song synthesis program 162b is a program for causing the control unit 110 to perform song synthesis using the song synthesis library 162a. The control unit 110 operating in accordance with the singing synthesis program 162b executes two types of processes, a clarity adjustment process and a singing synthesis process.

  The singing synthesis process is a process of synthesizing and outputting voice data representing the sound waveform of the singing voice based on a plurality of types of synthesis information acquired via the operation unit 120. For example, as shown in FIG. 2A, it is assumed that “ma” is designated as the composition target lyrics and “C4” is designated as the pitch when the lyrics are pronounced. In this case, phoneme string information representing consonant “m” + vowel “a” and note information for instructing the start of sound generation of pitch “C4” are provided from the operation unit 120 to the control unit 110. The control part 110 performs the process which produces | generates the phoneme sequence which the said phoneme string information shows as pre-processing of a song synthesis process. As shown in FIG. 2 (a), when “ma” is designated as the composition target lyrics, the control unit 110, as shown in FIG. 2 (b), is silent (in FIG. 2 (b), The transition from the consonant “m” to the consonant “m”, the transition from the consonant “m” to the vowel “a”, the transition from the vowel “a”, and the transition from the vowel “a” to the silence is arranged. Generated phoneme strings. In this preprocessing, the control unit 110 generates the pitch curve shown in FIG. 2C based on the note information instructing the start of sound generation. In the singing synthesis process, the control unit 110 reads out the segment data of each phoneme (or diphone) constituting the phoneme sequence from the singing synthesis library 162a, converts it into frequency domain data, and converts it into frequency domain data. The converted piece data is subjected to pitch conversion according to the pitch curve and combined, and then converted back to time domain data to generate voice data of synthesized singing voice.

  The pitch curve shown in FIG. 2 (c) is a pitch curve that shows an ideal prosody change in that a singing voice with a natural audibility can be obtained. In the pitch curve shown in FIG. 2C, the prosody change in the section T1 corresponds to a transitional transition (undershoot) from silence to the consonant “m”. The prosody change in the section T2 corresponds to the attack in the consonant “m”. The prosody change in the section T3 corresponds to a transition (overshoot) from the consonant “m” to the vowel “a”. The change in pitch in the section T4 corresponds to the steady change (sustain) of the pitch in the vowel “a”. A change in pitch in the same section T5 corresponds to a transition (release) from the vowel “a” to silence. In the present embodiment, pitch curve data that characterizes the ideal pitch curve shown in FIG. 2C (the duration (duration) of each section of sections T1 to T5), the depth of undershoot (D), and the gradient of the attack (Α, data indicating each of the overshoot heights H) are stored in advance in the nonvolatile storage unit 162, and the control unit 110 performs pitch curve data and pitches indicated by note information instructing the start of sound generation. Based on the above, a pitch curve is generated, and the pitch conversion of each piece data is executed according to the pitch curve. Similarly, the volume of the synthesized singing voice is stored in advance in the non-volatile storage unit 162 in such a manner that data indicating an ideal volume change in terms of obtaining a natural audibility, and the volume is adjusted according to this data. A synthetic singing voice may be output while being controlled.

  The articulation adjustment process is a process for adjusting the ease of listening to the consonant of the lyrics that are pronounced in accordance with the note according to the velocity value v included in the note information instructing the start of pronunciation. As described above, in the present embodiment, the consonant ease of listening can be adjusted by changing a part of the phoneme sequence indicated by the phoneme sequence information (the phoneme sequence generated by the preprocessing). . This intelligibility adjustment process is a process executed prior to the song synthesis process (in parallel with the pre-process or after the pre-process), and (a) changing (for example, shortening) the duration of the consonant. ), (B) generating a consonant loss, (c) replacing the consonant with another consonant that approximates the sound, and (d) generating a repetition of the consonant. The phoneme of a part of the phoneme string shown is changed. Hereinafter, as in the case shown in FIG. 2A, an example in which phoneme string information indicating “ma” (that is, consonant “m” + vowel “a”) is given from the operation unit 120 to the control unit 110 is an example. Therefore, specific processing contents of the clarity adjustment processing will be described.

(A-1: A mode in which a part of phonemes in a phoneme sequence is changed by shortening the duration of consonants)
As a specific example of a mode in which a part of the phoneme sequence indicated by the phoneme sequence information is changed by shortening the duration of the consonant, as shown in FIG. 3, a transition from silent to consonant (see FIG. 3). In the example shown, [# -m]) and the transition length of each transition from a consonant to a vowel (in the example shown in FIG. 3, [m-a]) are adjusted according to the velocity value v ( For example, the larger the velocity value v is, the larger the value is) and the vowel continuation length is extended by the shortened amount. If the transition from silence to consonant and the transition from consonant to vowel are shortened, the consonant is less likely to be heard as if it was pronounced quickly. In this way, the adjustment to reduce the ease of listening of consonants is performed because the singing voice that is easy to listen to consonants is synthesized according to the conventional singing synthesis technology, so the ease of listening is reduced. This is because it is possible to synthesize a singing voice as if it was pronounced quickly by adjusting as described above (that is, making it difficult to hear).

  In this embodiment, the adjustment amount of the consonant continuation length is a variable value according to the value of velocity v. However, when the adjustment amount is a fixed value and the velocity value v exceeds a predetermined threshold th1, The duration of the consonant may be adjusted. In addition, the control unit 110 may execute control that causes the duration of the consonant to be shortened at a frequency according to the velocity value v (that is, the frequency increases as the velocity increases). Specifically, the pseudo-random number generated in the range of 1 to 127 is compared with the velocity, and if the former is equal to or less than the latter, the control unit 110 executes the process of reducing the duration of the consonant. good. In the case where the shortening of the consonant duration is generated at a frequency corresponding to the velocity value v, the amount of reduction of the consonant duration may be a fixed value or a variable value corresponding to the velocity value v. There may be.

(A-2: A mode in which a partial phoneme of a phoneme sequence is changed by generating a missing consonant)
As a specific example of a mode in which a partial phoneme of the phoneme string is changed by generating a consonant loss, a mode in which a consonant loss is generated at a frequency according to the velocity value v can be given. Specifically, as shown in FIG. 4, from the phoneme sequence indicated by the phoneme sequence information, the diphones corresponding to the transitional transition from silence to consonant and the transition from consonant to vowel are deleted. Instead, the control unit 110 is caused to execute the process of compensating for the diphone corresponding to the transition from the silence to the vowel with a frequency corresponding to the velocity value v. The reason why the diphone corresponding to the transition from silent to vowel is supplemented is to make the singing voice rise smoothly. According to such an aspect, the consonant is no longer pronounced. Note that consonant loss may not occur at a frequency corresponding to the velocity value v, but may be always generated when the velocity value v exceeds a predetermined threshold th1. Further, when the consonant part is composed of a plurality of phonemes (for example, when [# −m] in FIG. 5 is replaced with [# −m] + [m]), a plurality of consonant parts are formed. Consonants may be lost by deleting the number of phonemes corresponding to the velocity value v of the phonemes (or deleting the number of phonemes at a frequency corresponding to the velocity value v). .

(A-3: A mode in which a part of the phoneme sequence is changed by replacing consonants)
As a specific example of an aspect in which a part of the phoneme of the phoneme sequence is changed by the consonant replacement, an aspect in which the consonant replacement is generated at a frequency corresponding to the velocity value v can be given. Specifically, as shown in FIG. 5, the control unit 110 is caused to execute a process of replacing a consonant in the phoneme sequence indicated by the phoneme sequence information with another consonant that approximates the sound at a frequency according to the velocity value v. is there. FIG. 5 illustrates the case where the consonant “m” is replaced with the consonant “n”. In order to enable such replacement of consonants, replacement control information indicating consonants that are replacement candidates for each consonant is stored in the nonvolatile storage unit 162 in advance, and the consonant is stored in the control unit 110 according to the replacement control information. It is sufficient to let them be replaced. Instead of generating a consonant replacement at a frequency corresponding to the velocity value v, a consonant replacement may always be generated when the velocity value v exceeds a predetermined threshold th1. When the consonant part is composed of a plurality of phonemes, the number of phonemes corresponding to the velocity value v among the plurality of phonemes constituting the consonant part is replaced with another phoneme approximating the sound (or The number of phonemes may be replaced at a frequency corresponding to the velocity value v).

(A-4: A mode in which part of the phoneme sequence is changed by inserting consonants)
As a specific example of the manner in which a part of the phoneme sequence is changed by inserting a consonant, as shown in FIG. 6, diphones related to consonants included in the phoneme sequence indicated by the phoneme sequence information (in the example shown in FIG. The process of inserting a diphone D1 corresponding to a transitional change from consonant to silence and a diphone D2) corresponding to a transitional change from silence to consonant according to the value of velocity v. An example of causing the control unit 110 to execute at a frequency is given. By inserting such a consonant, it is possible to synthesize a singing voice as if it is pronounced while biting, and to reduce the ease of listening to the consonant. Note that the number of diphones to be inserted may be a predetermined fixed value or a variable value corresponding to the velocity value v (a value that increases as the velocity value v increases). Also, instead of causing consonant insertion to occur at a frequency corresponding to the velocity value v, if the velocity value v exceeds a predetermined threshold th1, a predetermined number (or depending on the velocity value v) is used. The control unit 110 may always execute a process of inserting a fixed number of consonants, or the control unit 110 may execute a process of inserting a number of consonants determined according to the velocity value v.

  As described above, according to the singing voice synthesizing apparatus 1 according to the present embodiment, the phoneme sequence indicated by the phoneme sequence information has a frequency (or adjustment amount) according to the velocity value v designated by the operation for instructing the start of sound generation. After changing some of the phonemes, the singing voice is synthesized. Therefore, according to this embodiment, the ease of listening to consonants in the singing voice is controlled by the key-pressing speed at the time of instructing the start of pronunciation, and the singing voice as if it is pronounced quickly or pronounced while chewing It becomes possible to synthesize various types of singing voices, such as singing voices as if they were playing, and the expressive power of singing synthesis is improved. Note that changing some phonemes of the phoneme sequence indicated by the phoneme sequence information includes (a) shortening the consonant duration, (b) generating missing consonants, (c) replacing consonants, and ( d) Which of the consonant insertions is realized may be determined in advance, or may be selected by the user.

  Further, according to any one of the above aspects (a) to (d), a part of the phoneme string is not changed, but a plurality of combinations (for example, shortening of the consonant duration) Some phonemes of the phoneme string may be changed by a combination of consonant replacements, or a combination of consonant replacements and consonant insertions). In addition, not only consonants but also vowels may be subjected to continuation length adjustment, omission, replacement with other vowels, or insertion. According to such an embodiment, when synthesizing the singing voice of the lyrics “Saita”, the singing voice of “Sa, Sa, Sa, Ita” (repeat consonant + vowel part) is synthesized, It is possible to synthesize various singing voices, such as synthesizing a singing voice of “A, Aita” (missing consonants + repeated vowels), and the expression power can be further enhanced. In addition, since lyrics written in a language other than Japanese, such as English, may have a consonant at the end (for example, fan), the consonant at the end may be any one of the above-described aspects (a) to (d). The ease of listening may be adjusted by (or a combination of these). The point is that any part of the phoneme indicated by the phoneme string information may be changed by adjusting, missing, replacing, or inserting the phoneme duration.

(B: Second embodiment)
In the intelligibility adjustment process of the first embodiment, the consonant is less easily heard by synthesizing the singing voice by changing a part of the phoneme (consonant) of the phoneme string shown in the phoneme string information. On the other hand, in the clarity adjustment processing of the present embodiment, the ease of listening to consonants is reduced by synthesizing the singing voice so that the deviation from the ideal prosody change shown in FIG. There is a feature. Here, specific examples of a method for causing a deviation from an ideal prosodic change include an aspect in which the depth and duration of an attack are changed from an ideal prosodic change, an aspect in which an attack is lost, an overshoot The aspect which changes the height of a chute | shoot, the depth and continuation length of an undershoot, and the aspect which generate | occur | produces the loss of an overshoot and an undershoot are mentioned. The configuration of the singing voice synthesizing apparatus of the present embodiment is not particularly different from that in the first embodiment, and thus detailed description thereof will be omitted. Hereinafter, a specific adjustment mode of prosody change will be described.

(B-1: A mode in which deviation from an ideal prosodic change is caused by adjusting the attack)
FIG. 7 illustrates a mode in which a deviation from an ideal prosody change is caused by adjusting the attack depth. In FIG. 7, the pitch curve in an ideal prosody change is drawn with a broken line, and the pitch curve used in this aspect is drawn with a solid line. In FIG. 7, the two pitch curves are overlapped after the overshoot (parts of both pitch curves are also overlapped in FIGS. 8 to 12). In this aspect, the control unit 110 adjusts the pitch curve data (more precisely, data representing the attack gradient α) according to the velocity value v included in the note information instructing the start of sounding. Adjustment is performed to change (for example, make shallow) the attack depth by the amount, and a pitch curve is generated based on the pitch curve data after the adjustment and the pitch indicated by the note information. This is because making the attack depth shallower than the ideal pitch curve reduces the ease of listening to consonants. In this embodiment, the ease of listening to consonants is reduced by making the attack depth shallower than the ideal pitch curve, but the attack depth is made deeper than the ideal pitch curve. Even so, the ease of listening to consonants decreases. Therefore, the attack may be deepened in accordance with the velocity value v included in the note information instructing the start of sound generation. As a mode for changing the attack depth, it is possible to determine in advance whether to adopt a mode in which the depth is shallower than the ideal pitch curve or a mode in which the depth is deeper than the ideal pitch curve. It may be good or let the user choose.

  In the present embodiment, the velocity included in the note information instructing the start of sound generation is synthesized from the singing voice by causing a deviation from an ideal prosody change, and the prosody control information instructing the magnitude of the deviation. Have a role. It should be noted that the depth of the attack in the ideal pitch curve is a predetermined amount (or an amount corresponding to the velocity value v) with a frequency corresponding to the velocity value v included in the note information instructing the start of sounding. The control unit 110 may execute the process to be changed. Further, the velocity value v included in the note information instructing the start of sound generation exceeds a predetermined threshold th2 (may be the same value as the threshold th1 in the first embodiment or may be a different value). In this case, the control unit 110 may execute processing for changing the attack depth from a depth in the ideal pitch curve by a predetermined amount (or an amount corresponding to the velocity value v).

  FIG. 8 illustrates a mode in which a deviation from an ideal prosody change is caused by adjusting the attack duration. In FIG. 8, as in FIG. 7, the pitch curve in the ideal prosody change is drawn with a broken line, and the pitch curve used in this aspect is drawn with a solid line. In this aspect, the control unit 110 changes the attack duration from the duration in the ideal pitch curve by an adjustment amount corresponding to the velocity value v included in the note information instructing the start of sound generation (specifically, , The pitch curve is generated by changing the sustain in the vowels (specifically, by extending). Thus, the ease of listening of consonants also decreases by changing the duration of the attack from the duration of the ideal pitch curve. Here, as a specific method for generating the pitch curve indicated by the solid line in FIG. 8, pitch curve data (more precisely, data indicating the length of the section T2 in FIG. 2C, the attack gradient α) is used. The section T2 is shortened by a length corresponding to the velocity value v, and the section T4 is lengthened by that amount. A method of generating a pitch curve based on the pitch curve data and the pitch indicated by the note information instructing the start of sound generation can be considered. Also in this aspect, the attack duration is ideally set by a predetermined amount (or an amount corresponding to the velocity value v) at a frequency corresponding to the velocity value v included in the note information instructing the start of sound generation. It is also possible to cause the control unit 110 to execute a process of changing from the continuation length in a simple pitch curve. In addition, when the velocity value v included in the note information instructing the start of sound generation exceeds a predetermined threshold th2, the attack duration is set to a predetermined amount (or an amount corresponding to the velocity value v). You may make the control part 110 perform the process changed from the continuation length in an ideal pitch curve.

  FIG. 9 exemplifies a mode in which a deviation from an ideal prosodic change is generated by omitting an attack (generating a pitch curve with an attack duration of zero). In FIG. 9, the pitch curve in the ideal prosody change is drawn with a broken line as in FIG. 7, and the pitch curve used in this aspect is drawn with a solid line. In this way, the ease of listening to consonants is also reduced by making adjustments that eliminate the attack. As for the absence of an attack, an attack is generated when the velocity is generated at a frequency corresponding to the velocity value v included in the note information instructing the start of sound generation, or when the velocity value v exceeds a predetermined threshold th2. A mode in which a loss occurs can be considered.

(B-2: Mode in which deviation from ideal prosodic change is caused by adjusting undershoot and overshoot)
FIG. 10 illustrates an aspect in which a deviation from an ideal prosody change is generated by adjusting the undershoot depth D and the overshoot height H. In FIG. 10, the pitch curve in the ideal prosody change is drawn with a broken line, and the pitch curve used in this aspect is drawn with a solid line. In this aspect, the control unit 110 changes the undershoot depth D from the ideal prosodic change depth by an adjustment amount corresponding to the velocity value v included in the note information instructing the start of sounding ( For example, the pitch curve is generated by changing the height H of the overshoot by an adjustment amount according to the velocity value v from the ideal height of the prosody change (for example, by lowering). Such an aspect also reduces the ease of listening to consonants. As with the attack depth adjustment, the undershoot depth D is increased by an adjustment amount corresponding to the velocity value v included in the note information instructing the start of sounding, and the overshoot height is set. A mode in which H is increased by an adjustment amount according to the velocity value v may be employed. Further, the undershoot depth D and the overshoot height H are set to a predetermined amount (or an amount corresponding to the velocity value v) at a frequency corresponding to the velocity value v included in the note information instructing the start of sound generation. ) May be executed by the control unit 110. Also, when the velocity value v exceeds the predetermined threshold th2, the undershoot depth D and the overshoot height H are changed by a predetermined amount (or an amount corresponding to the velocity value v). You may make the control part 110 perform a process.

  FIG. 11 illustrates a mode in which a deviation from an ideal prosody change is generated by adjusting the duration of undershoot and overshoot. In FIG. 11, the pitch curve in an ideal prosody change is drawn with a broken line, and the pitch curve used in this aspect is drawn with a solid line. In this aspect, the control unit 110 sets the continuation length of each undershoot and overshoot by an adjustment amount corresponding to the velocity value v included in the note information instructing the start of sound generation, and the continuation length in an ideal prosodic change. The pitch curve is generated by changing (specifically, shortening) and changing the continuation length of the vowel sustain by the change (specifically, extending). Such an aspect also reduces the ease of listening to consonants. It should be noted that the duration of undershoot and overshoot is ideal by a predetermined amount (or an amount corresponding to the velocity value v) at a frequency corresponding to the velocity value v included in the note information instructing the start of sound generation. You may make the control part 110 perform the process changed from the thing in a pitch curve. In addition, when the velocity value v exceeds the predetermined threshold th2, the control unit 110 performs a process of changing the undershoot and overshoot duration by a predetermined amount (or an amount corresponding to the velocity value v). May be executed.

  In FIG. 12, undershoot and overshoot are eliminated (a pitch curve is generated with both the undershoot duration and overshoot duration set to zero), thereby causing a deviation from an ideal prosody change. Embodiments are illustrated. In FIG. 12, the pitch curve in an ideal prosody change is drawn with a broken line, and the pitch curve used in this aspect is drawn with a solid line. Such an aspect also reduces the ease of listening to consonants. The lack of undershoot and overshoot may be generated at a frequency according to the velocity value v included in the note information instructing the start of sound generation, and the velocity value v has a predetermined threshold th2. It may be generated when the number exceeds.

  As explained above, listening to consonants in a singing voice can also be achieved by synthesizing a singing voice while adjusting the prosodic change according to the prosodic control information that instructs to synthesize a voice by generating a deviation from an ideal prosodic change. The ease of taking can be controlled. In the present embodiment, since the velocity value included in the note information instructing the start of pronunciation is used as the prosody control information, the user can select the synthesized singing voice according to the key pressing speed when designating the pitch of the singing voice. This makes it possible to control the ease of listening of consonants in, and to synthesize singing voices rich in expressiveness in real time. It should be noted that as the adjustment mode of the attack, which of the adjustment of the depth, the adjustment of the continuation length, and the occurrence of the omission may be determined in advance or may be selected by the user. Similarly, with regard to undershoot and overshoot adjustment, it may be determined in advance whether the adjustment of the depth and height, the adjustment of the continuation length, or the occurrence of omission is adopted, or the user can select it. good. Further, it is possible to adjust either one of the overshoot and the undershoot at the same time, instead of adjusting both of them.

(C: deformation)
Although each embodiment of the present invention has been described above, it goes without saying that the following modifications may be added to these embodiments.
(1) In the first embodiment, an aspect of adjusting the ease of listening of consonants by changing a part of the phoneme of the phoneme sequence indicated by the phoneme sequence information will be described. In the second embodiment, an ideal prosody is described. The aspect which adjusts the ease of listening of a consonant by generating the shift | offset | difference from a change was demonstrated. However, as a plurality of types of synthesis information, phonological sequence information, prosodic information, phonological control information and prosodic control information are input to the singing synthesizer, and a part of the phonological sequence is changed according to the phonological control information. Of course, it is possible to adjust the ease of listening of the consonant by using together with the process of generating the deviation from the ideal prosodic change according to the prosodic control information. This is because, by using both in combination, it is expected that more various singing voices can be synthesized as compared with the case where only one of them is performed. Note that the phoneme control information and the prosody control information may be different from each other, or one information may serve as the role of the phoneme control information and the prosody control information. For example, as an example of the latter mode, a mode in which both the role of the phoneme control information and the role of the prosody control information can be assigned to the velocity included in the note information (note-on event) instructing the start of pronunciation.

(2) In the first embodiment, the velocity included in the note information instructing the start of sounding plays a role of phonological control information, and in the second embodiment, the velocity plays a role of prosodic control information. It was. However, as a matter of course, the combination of the velocity and the pitch indicated by the note information may play the role of the phoneme control information or the prosody control information (or both). In addition, a dedicated operation element for allowing the user to input phonological control information or prosodic control information is provided on the singing voice synthesis keyboard, and the phonological control information or the phonological control information or Prosodic control information may be input.

(3) In each of the embodiments described above, a plurality of types of synthesis information (phoneme sequence information, prosody information (note information indicating the pitch of a note, the start timing of pronunciation and the end timing of pronunciation) used in song synthesis, The singing voice synthesizing apparatus 1 includes an operation unit 120 serving as an acquisition unit that causes the singing voice synthesizing apparatus 1 to acquire phonological control information (or prosodic control information) and a voice output unit 140 for outputting synthesized singing voice. It was. However, an aspect in which one or both of the operation unit 120 and the audio output unit 140 are connected to the external device I / F unit 150 of the singing voice synthesizing apparatus 1 may be employed. In an aspect in which the operation unit 120 is connected to the singing voice synthesizing apparatus 1 via the external device I / F unit 150, the external device I / F unit 150 serves as the acquisition unit.

  As an example of a mode in which both the operation unit 120 and the audio output unit 140 are connected to the external device I / F unit 150, an Ethernet (registered trademark) interface is used as the external device I / F unit 150. A mode in which a telecommunication line such as a LAN (Local Area Network) or the Internet is connected to the unit 150, and the operation unit 120 and the audio output unit 140 are connected to the telecommunication line. According to such an aspect, it is possible to provide a so-called cloud computing type song synthesis service. Specifically, a plurality of types of synthesis information input by operation of various operators provided in the operation unit 120 is given to the singing synthesizer via the telecommunication line, and the singing synthesizer is provided via the telecommunication line. The singing synthesis process is executed based on the plurality of types of synthesis information acquired in the above. In this way, the voice data of the synthesized singing voice synthesized by the singing voice synthesizer is given to the voice output unit 140 via the telecommunication line, and the sound corresponding to the voice data is output from the voice output unit 140.

(4) In each said embodiment, the keyboard for song synthesis | combination was used as the operation part 120 for inputting multiple types of synthetic | combination information into the song synthesizing | combining apparatus 1. FIG. However, a combination of a general keyboard on which numeric keys, cursor keys, keys corresponding to letters of the alphabet, and the like are arranged, and a so-called MIDI keyboard may be used as the operation unit 120. When a combination of a general keyboard and a MIDI keyboard is used as the operation unit 120, the MIDI keyboard may serve as the note information input unit, and the general keyboard may serve as the phoneme information input unit. Further, a note information input unit or a phoneme information input unit may be realized by a combination of a pointing device such as a mouse and a GUI. When the note information input unit is realized by a combination of a pointing device and a GUI, the operation unit 120 may be configured by a combination of a general keyboard serving as a phoneme information input unit and the note information input unit. When the phoneme information input unit is realized by a combination of a pointing device and a GUI, the operation unit 120 may be configured by a combination of a MIDI keyboard serving as a note information input unit and the phoneme information input unit.

(5) In each of the above embodiments, the song synthesis program 162b that causes the control unit 110 to perform the song synthesis process and the clarity adjustment process is stored in the nonvolatile storage unit 162 of the song synthesis apparatus 1 in advance. However, this singing synthesis program 162b may be distributed by being written on a computer-readable recording medium such as a CD-ROM (Compact Disk-Read Only Memory), or by downloading via a telecommunication line such as the Internet. May be. This is because by causing a general computer such as a personal computer to execute the distributed program in this way, the computer can function as the singing voice synthesizing apparatus 1 of the above embodiment. Of course, the present invention may be applied to a game program of a game partially including a real-time singing synthesis process. Specifically, the song synthesis program included in the game program may be replaced with a song synthesis program 162b. According to such an aspect, it becomes possible to improve the expressive power of the singing voice synthesized as the game progresses.

(6) In the above embodiments, application examples of the present invention to a real-time singing voice synthesizing apparatus have been described. However, the application target of the present invention is not limited to the real-time singing synthesizer. For example, the present invention may be applied to a voice synthesizer that synthesizes guidance voices in voice guidance in real time, or a voice synthesizer that synthesizes reading sounds of literary works such as novels and poems in real time. The application target of the present invention may be a toy having a singing voice synthesis function or a voice synthesis function (a toy incorporating a singing voice synthesis device or a voice synthesis device).

(7) In the first embodiment, phoneme sequence information, prosodic information, and phoneme control information are used as a plurality of types of synthesis information used for synthesis of singing voice, and in the second embodiment, a plurality of types used for synthesis of singing voice Phonological sequence information, prosodic information, and prosodic control information were used as composite information. However, the plurality of types of synthesis information used for synthesizing the singing voice is not limited to only phoneme string information, prosody information, and phoneme control information (or prosody control information). For example, in the case where a plurality of types of voice quality segment data are classified and stored for each voice quality in the song synthesis library 162a, in addition to the phoneme string information, prosody information and phoneme control information (or prosody control information) Voice quality designation information for designating voice quality may be included in the synthesis information, and the singing voice may be synthesized using the segment data of the voice quality designated by the voice quality designation information.

  In each of the above embodiments, phonological sequence information, prosodic information, and phonological control information (or prosodic control information) are input to the singing voice synthesizing apparatus 1 by operating the operator of the operation unit 120, but at least one of these is input. One may be input by operating the operator of the operation unit 120, and the other may be stored in the singing voice synthesizing apparatus 1 in advance. Specifically, phonological sequence information indicating the phonological sequence of the lyrics of the entire song to be synthesized is stored in advance in the nonvolatile storage unit 162, and note information and phonological control information serving as prosodic information are stored. It may be made to input for every note by operation with respect to a manipulator of operation part 120. In this case, the operation unit 120 and a unit (for example, the control unit 110) that reads out phoneme string information stored in the nonvolatile storage unit 162 serve as an acquisition unit that acquires a plurality of types of synthesis information. In addition, prosody information and phonological control information (prosodic control information instead of phonological control information, or both phonological control information and prosodic control information) are stored in advance in the singing synthesizer 1, and only the phonological string information is stored in the operation unit 120. You may make it input into the song synthesizing | combining apparatus 1 by operation with respect to this operator.

(8) In each of the above-described embodiments, the adjustment for reducing the ease of listening of consonants has been described. However, the adjustment for improving the ease of listening of consonants is performed according to the magnitude of velocity or the like. Also good. For example, when pitch curve data representing a pitch curve having an attack depth or the like different from the ideal pitch curve shown in FIG. 2C is stored in the nonvolatile storage unit 162, the ideal pitch curve The speech synthesis may be performed after the pitch curve data is subjected to the process of adjusting the attack depth or the like according to the prosodic control information so as to approach the pitch.

DESCRIPTION OF SYMBOLS 1 ... Singing synthesis apparatus, 110 ... Control part, 120 ... Operation part, 130 ... Display part, 140 ... Audio | voice output part, 142 ... D / A converter, 144 ... Amplifier, 146 ... Speaker, 150 ... External apparatus I / F , 160 ... storage unit, 162 ... nonvolatile storage unit, 162a ... singing synthesis library, 162b ... singing synthesis program, 164 ... volatile storage unit, 170 ... bus.

Claims (6)

Means for acquiring a plurality of types of synthesis information including phonological sequence information indicating a phonological sequence of a speech to be synthesized and prosodic information indicating a prosodic change of the speech , and further including a velocity , comprising: Obtaining means for obtaining at least one of the prosody information and the velocity by operation of an operator;
A means for synthesizing speech using a plurality of types of synthesis information acquired by the acquisition means , wherein the consonant included in the phoneme sequence indicated by the phoneme sequence information included in the plurality of types of synthesis information is easily listened to. , Speech synthesis means for performing speech synthesis by lowering according to the velocity value included in the plurality of types of synthesis information;
A speech synthesizer characterized by comprising: The speech synthesizer according to claim 1, wherein the speech synthesizer lowers the ease of listening of the consonant by changing the synthesized content of the consonant.   The process of changing the consonant synthesis content is predetermined among the process of changing the duration of the consonant, the process of deleting the consonant, the process of replacing the consonant, and the process of inserting the consonant into the phoneme string indicated by the phoneme string information One or more of
  The speech synthesis means controls the duration according to the velocity value in the process of changing the duration of consonants, and the number of consonants to be inserted in the process of inserting consonants into the phoneme sequence indicated by the phoneme sequence information Is controlled according to the velocity value, and in the process of deleting consonants, the number of consonants to be deleted is controlled according to the velocity value, and in the process of replacing consonants, the number of consonants to be replaced is set to the velocity value. The speech synthesizer according to claim 2, wherein the speech synthesizer is controlled accordingly.   The speech synthesizer according to claim 2, wherein the speech synthesizer generates a change in consonant synthesis content at a frequency corresponding to the velocity value. The speech synthesizer according to claim 1, wherein the speech synthesizer reduces the ease of listening of consonants by generating a deviation from a prosodic change indicated by the prosodic information.   The speech synthesizer adjusts the depth of attack or the duration of the attack in the pitch curve representing the prosodic change according to the velocity value, or the height or duration of the overshoot in the pitch curve. Predetermined one or more of processing for adjusting according to the value of velocity, or processing for adjusting the depth or duration of undershoot in the pitch curve according to the value of velocity, from the prosody change 6. The speech synthesizer according to claim 5, wherein the speech synthesizer is executed as a process for generating a deviation.

Images