JP6044284B2 - Speech synthesizer - Google Patents

Description

  The present invention relates to speech synthesis technology, and more particularly to real-time speech synthesis technology.

  A speech synthesis technique for synthesizing a voice signal representing a guidance voice in voice guidance, a reading voice of a literary work, or a singing voice of a song by electrical signal processing using a plurality of types of synthesis information has become widespread. For example, in the case of synthesis of singing voice, prosodic information (for example, musical note information indicating the pitch or duration of each note constituting the melody of the singing song) indicating prosody change in the singing voice synthesis target song Music expression information such as information representing the phoneme string of the lyrics of the song is used as the synthesis information. When synthesizing the voice signal of the guidance voice in the voice guidance or the reading voice of the literary work, the information indicating the phonological sequence of the text of the guidance sentence or the literary work is used as the synthesized information indicating the utterance content, and the prosody such as intonation and accent. Prosodic information indicating a change is used as synthetic information indicating an utterance mode. In addition to these, information specifying sound intensity may be used as synthesis information. Conventionally, in this type of speech synthesis, various types of synthesis information over the entire speech to be synthesized are input to a speech synthesizer in advance, and a speech signal representing the sound waveform of the entire speech to be synthesized is based on the synthesis information. A so-called batch processing method in which batch generation is performed is common. However, in recent years, a real-time speech synthesis technique has also been proposed (see, for example, Patent Document 1).

  As an example of real-time speech synthesis, information representing the lyrics of the entire song is input to the singing synthesizer in advance, and the pitch and the like when each lyric is pronounced with a keyboard simulating a piano keyboard is designated as needed. A technique that promotes singing synthesis. Also, in recent years, a singing synthesis keyboard has been used in which a phoneme information input unit in which operators for inputting consonants and vowels representing phonics of lyrics are arranged and a note information input unit imitating a piano keyboard are arranged side by side. Thus, it has also been proposed to synthesize a singing voice by sequentially inputting both phonological information and musical note information to a user in real time.

Japanese Patent No. 3879402

  When performing a real-time singing synthesis using a singing synthesis keyboard, the user operates the phonological information input unit with one hand even when only essential synthesis information (ie, phonological information and prosodic information) is used. Thus, the phoneme information must be input, and the note information input unit must be operated with the other hand to input the prosodic information. On the other hand, in the singing voice synthesizing apparatus, the singing voice synthesizing process is not started unless both phonological information and prosodic information are prepared. For this reason, if there is a difference in the input timing of both phoneme information and prosodic information, the synthesized singing voice is delayed from the input timing of the earlier input (that is, the initial intention display timing for singing voice output). This delay may make the user feel uncomfortable. The same applies to voice synthesis that synthesizes guidance voices and readings of literary works in real time. In addition, when the input order of a plurality of types of composite information is predetermined, each composite information must be input in a predetermined order, and there is a problem that the degree of freedom of input is low.

  The present invention has been made in view of the above problems, and avoids a reduction in the degree of freedom of input of a plurality of types of synthesis information used for speech synthesis in real-time speech synthesis while maintaining a time difference in the input timing of each synthesis information. It is an object of the present invention to provide a technology that makes it possible to output a sound without delay from the earliest input timing even if there is.

  In order to solve the above problems, the present invention provides a plurality of types of synthesis information used for synthesizing speech signals, including a plurality of phoneme information indicating phonemes of speech to be synthesized and prosody information indicating prosodic changes in the speech. Input means for inputting types of synthesis information; and at least the phoneme information and the prosodic information from when the earliest one of the plurality of types of synthesis information is input to the input means. A voice signal is synthesized and output using synthesis information input via the input means, and a dummy sound is output from the time when the earliest synthesis information is input until the output of the voice signal is started. There is provided a speech synthesizer characterized by comprising speech synthesis means for outputting a dummy sound signal to be expressed. Of the multiple types of synthesis information used for synthesizing speech signals, examples of synthesis information other than phonological information and prosodic information include information indicating sound intensity, information indicating the addition of vibrato, and voice quality of synthesized speech. Information to be specified. However, synthesis information other than phonological information and prosodic information is not necessarily essential in the synthesis of a speech signal.

  According to such a speech synthesizer, among the plurality of types of synthesis information, the input from the earliest input to the input means until at least the phoneme information and the prosodic information, which are essential synthesis information, are complete. A voice signal is synthesized based on the inputted synthesis information, and a dummy sound is output from the input of the earliest synthesis information until the output of the voice signal is started. For this reason, the sound is output from the time when the user first expresses the intention for speech synthesis (that is, the input time of the earliest of the plurality of types of synthesis information), so that the user does not feel uncomfortable. It becomes possible to. Further, in the speech synthesizer of the present invention, the input order of a plurality of types of synthesis information is not questioned, so the degree of freedom when inputting these synthesis information does not decrease.

  Various modes can be considered as to what to use as the dummy sound. For example, as a first mode, a mode in which a sound corresponding to the earliest synthesized information is used as a dummy sound can be considered. In addition, as a second aspect, noise sound, breath sound, nasal sound, sound obtained by outputting a predetermined phoneme at a predetermined pitch, or a periodic sound having a predetermined pitch (the sound waveform is represented by a sine wave). In other words, a sound that can be continued and is not related to the earliest synthesized information is used as a dummy sound. For example, as a specific example of the first aspect in which the present invention is applied to real-time singing synthesis and only essential synthesis information is used as a plurality of types of synthesis information, phonological information is input first, and When the head of the phoneme indicated by the phoneme information is a continuable phoneme, the note information that plays the role of the prosody information is input first, while the sound output from the head phoneme at a predetermined pitch is used as a dummy sound. In such a case, it is conceivable to use, as a dummy sound, a sound in which a predetermined continuous phoneme is output at a pitch indicated by the note information or a periodic sound having a pitch indicated by the note information. Here, the continuable phoneme means a vowel or a consonant that can be continued, such as a frictional sound and a nasal sound. If the phoneme information is input first and the head of the phoneme indicated by the phoneme information is not a continuous phoneme, a predetermined continuous sound such as a noise sound may be used as a dummy sound. .

  As a more preferable aspect, the dummy sound signal is adjusted so that the sound represented by the sound signal synthesized using the synthesis information input to the input means and the dummy sound are smoothly connected (or the sound signal and the dummy sound). A mode is conceivable in which the speech synthesizer executes the process of outputting the signal while adjusting the signal levels of both signals. According to such an aspect, since the dummy sound smoothly changes from the synthesized sound, it is possible to prevent the user from feeling uncomfortable due to the sound switching discontinuously. Further, by causing the voice synthesizer to execute the process of outputting the dummy sound signal while gradually increasing the signal level to a predetermined value so that the volume of the dummy sound gradually increases, the user can feel unnecessary discomfort. It is also conceivable to avoid giving to the above. As yet another aspect, the output of the synthesized voice using the synthesis information that has been input after the earliest one of the plurality of types of synthesis information has been input until at least the required synthesis information is available is started. In the meantime, a mode is also conceivable in which the voice synthesizer executes the process of switching the dummy sound signal so that a plurality of types of dummy sounds are sequentially output.

  Further, as another preferred mode, when the phoneme information and the prosodic information are not prepared until a predetermined waiting time elapses after the earliest one of the plurality of types of synthesis information is input, speech synthesis is performed. The output of the dummy sound signal may be stopped by the means. According to such an aspect, even if phonological information is input but note information is not input (or vice versa), even if a part of essential synthesis information is not input and speech synthesis cannot be performed, dummy It is possible to prevent problems such as sound being continuously output. Also, a stop instruction means for instructing the user to stop the output of the dummy sound signal (when using a singing synthesis keyboard as the input means, an operator for instructing the user to stop the output) is provided, and the dummy sound signal is provided. The voice synthesizing unit may be caused to execute a process of stopping the output of the dummy sound signal when the output stop instruction is given. According to such an aspect, it is possible to prevent the dummy sound from being continuously output for the predetermined time even when the output of the dummy sound is triggered by an erroneous input of the synthesis information. become.

  As another aspect of the present invention, a plurality of types of synthesis information used for synthesis of speech signals, including a plurality of phoneme information indicating phonemes of speech to be synthesized and prosody information indicating prosodic changes in the speech. A first process of synthesizing and outputting a speech signal using synthesis information input at least from the time when the earliest one of the types of synthesis information is input until the phoneme information and prosodic information are aligned; Provided is a program for causing a computer to execute a second process for outputting a dummy sound signal representing a dummy sound from when the first synthesis information is input until the output of the sound signal is started. Embodiments are possible. Here, as a specific provision mode of the program, a program is written and distributed on a computer-readable recording medium such as a CD-ROM (Compact Disk-Read Only Memory) or downloaded via an electric communication line such as the Internet. A distribution mode is conceivable.

It is a figure which shows the structural example of the song synthesizing | combining apparatus 1 of 1st Embodiment of this invention. It is a figure for demonstrating operation | movement of the song synthesizing | combining apparatus 1. FIG. It is a figure for demonstrating the dummy sound output process of 2nd Embodiment. It is a figure for demonstrating the dummy sound output process of 3rd Embodiment. It is a figure for demonstrating the dummy sound output process of 4th Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(A: 1st Embodiment)
FIG. 1 is a block diagram showing a configuration example of a singing voice synthesizing apparatus 1 according to an embodiment of the voice synthesizing apparatus of the present invention. The singing voice synthesizing apparatus 1 allows a user to sequentially input a plurality of types of synthesis information (in this embodiment, two types of phonological information and prosodic information, which are essential synthesis information when performing singing synthesis) for each note. It is a device that performs singing synthesis of the method. As shown in FIG. 1, the singing voice synthesizing apparatus 1 includes a control unit 110, an operation unit 120, a display unit 130, an audio output unit 140, an external device interface (hereinafter abbreviated as “I / F”) unit 150, and a storage unit 160. , And a bus 170 that mediates data exchange between these components.

  The control unit 110 is, for example, a CPU (Central Processing Unit). The control unit 110 functions as a control center of the singing voice synthesizing device 1 by operating according to the singing voice synthesis program stored in the storage unit 160. Details of processing executed by the control unit 110 in accordance with this singing synthesis program will be made clear later. In the present embodiment, a CPU is used as the control unit 110, but a digital signal processor (DSP) may of course be used.

  The operation unit 120 is the above-described singing synthesis keyboard. The user of the singing voice synthesizing device 1 can designate the notes constituting the melody of the song to be synthesized of the singing voice and the phonology of the lyrics to be sung in accordance with the notes by operating the operation unit 120. For example, when “sa” is designated as the phoneme of the lyrics, the operator corresponding to the consonant “s” and the operator corresponding to the vowel “a” may be pressed in sequence, and the note sound corresponding to the lyrics When “C4” is designated as a high pitch, a key corresponding to the pitch is pressed to instruct the start of sound generation, and the end of sound generation is indicated by releasing the key. That is, the length of time that the key is pressed is the duration of the note. When an operation for designating a phoneme is performed, the operation unit 120 gives phoneme information indicating the phoneme to the control unit 110. Also. When a key pressing operation instructing the start of sounding is performed, the operation unit 120 uses a note-on event (MIDI (Musical Instrument Digital Interface) event) corresponding to the pressed key as musical note information instructing the start of sounding. A note-off event (MIDI event) corresponding to the key is given to the control unit 110 as musical note information for instructing the end of sounding. Thus, the note information input by the operation with respect to the operator of the note information input unit plays a role of prosody information indicating prosody change in the singing voice. That is, the operation unit 120 serves as an input unit for inputting a plurality of types of synthesis information used for synthesizing the singing voice.

The display unit 130 is, for example, a liquid crystal display and a driving circuit thereof, and displays various images such as a menu image that prompts the use of the singing voice synthesizing device 1 under the control of the control unit 110. As shown in FIG. 1, the audio output unit 140 includes a D / A converter 142, an amplifier 144, and a speaker 146. The D / A converter 142 performs D / A conversion on the digital audio data provided from the control unit 110, and provides an analog audio signal as a conversion result to the amplifier 144. The amplifier 144 amplifies the signal level (that is, the volume) of the audio signal supplied from the D / A converter 142 to a level suitable for driving the speaker and supplies the amplified signal to the speaker 146. The speaker 146 outputs the audio signal given from the amplifier 144 as sound.

  The external device I / F unit 150 is a collection of interfaces for connecting other external devices to the song synthesizer 1, such as a USB (Universal Serial Bus) interface and an audio interface. In this embodiment, the case where the singing voice synthesis keyboard (operation unit 120) and the voice output unit 140 are constituent elements of the singing voice synthesizing device 1 will be described. Of course, an external device connected to the F unit 150 may be used.

  The storage unit 160 includes a nonvolatile storage unit 162 and a volatile storage unit 164. The volatile storage unit 164 is configured by a volatile memory such as a RAM (Random Access Memory). The volatile storage unit 164 is used by the control unit 110 as a work area when executing various programs. On the other hand, the nonvolatile storage unit 162 is configured by a nonvolatile memory such as a ROM (Read Only Memory), a flash memory, or a hard disk. As shown in FIG. 1, the non-volatile storage unit 162 stores a song synthesis library 162a, a song synthesis program 162b, and a dummy sound library 162c in advance.

  The singing synthesis library 162a is a database storing segment data representing speech waveforms such as various phonemes (monophones) and diphones (transitions from phonemes to different phonemes (including silence)). Note that the singing synthesis library 162a may be a database storing triphone segment data in addition to a monophone or a diphone, and may also be a phoneme stationary part of a speech waveform or a transition part (transient part to another phoneme). Part) may be stored in the database. The song synthesis program 162b is a program for causing the control unit 110 to perform song synthesis using the song synthesis library 162a. The control unit 110 operating in accordance with the singing synthesis program 162b executes two types of processing: singing synthesis processing and dummy sound output processing.

  The singing synthesis process is a process of synthesizing and outputting voice data representing the sound waveform of the singing voice based on the phoneme information and the note information. In this singing synthesis process, the control unit 110 reads the segment data corresponding to the phoneme or the diphone represented by the phoneme information from the singing synthesis library 162a and converts it into frequency domain data so that the pitch indicated by the note information is obtained. The voice data representing the sound waveform of the singing voice is synthesized by combining them while performing pitch conversion, and then returning to the data in the time domain. As described above, the singing synthesis process is executed on the premise that the phonological information representing the lyrics of the singing voice and the note information representing the pitch are prepared. . More specifically, the singing synthesis process according to the present embodiment is performed when the other one of the phonological information and the note information is input before the predetermined waiting time TW elapses from the input time of the first input. Executed with the input of The length of the waiting time TW may be determined in advance by conducting an experiment as appropriate, or may be set according to the user's preference. In this embodiment, the unit connection method algorithm is adopted as the singing synthesis algorithm. However, other algorithms may be adopted. When other algorithms are adopted, the algorithm is adapted to the algorithm. What is necessary is just to comprise the library 162a for song synthesis | combination.

  In the dummy sound output process, from the input time point of the phoneme information and note information input first, until the output of the voice represented by the voice data synthesized by the song synthesis process (that is, the synthesized song voice) is started. In this process, a dummy sound is output to the audio output unit 140. Execution of the dummy sound output process of the present embodiment is triggered by the input of the phoneme information and the note information input first. In this dummy sound output process, a predetermined sound that does not need to designate both a phoneme and a pitch (for example, a breath sound, a noise sound or a nasal sound, a sound that continuously outputs a predetermined continuous phoneme at a predetermined pitch, Alternatively, a periodic sound having a predetermined pitch is output as a dummy sound. The dummy sound library 162c described above is used for various dummy sounds such as a breath sound, a noise sound or a nasal sound, a sound that continuously outputs a predetermined continuous phoneme at a predetermined pitch, or a periodic sound having a predetermined pitch. It is the database which stored the waveform data showing a sound waveform. In the dummy sound output process, the control unit 110 reads the waveform data of the sound determined as the dummy sound from the dummy sound library 162c, gives the waveform data to the audio output unit 140, and outputs the dummy sound. Here, which sound is used as the dummy sound may be determined in advance or may be selected by the user. In the present embodiment, a breath sound (hereinafter, br is used as a phoneme symbol of the breath sound) is used as the dummy sound. In this embodiment, the singing synthesis library 162a and the dummy sound library 162c are separate databases. However, even if they are integrated (for example, the dummy sound library 162c is included in the singing synthesis library 162a). good.

  In the dummy sound output process, the control unit 110 performs a process of determining whether both the phoneme information and the note information (that is, all of the synthesis information used for singing synthesis) are prepared. Continue until the time elapses. And when all the synthetic | combination information used for a song synthesis | combination is not prepared before the said waiting time TW passes, the control part 110 stops a dummy sound output process at the time of the said waiting | waiting TW progressing. That is, in this embodiment, when either one of phonological information and note information is input and the other is not input before the predetermined waiting time TW elapses from the input, the control unit 110 performs the waiting time. At the time when TW has elapsed, the execution of the dummy sound output process is stopped, and the output of the dummy sound is stopped. On the other hand, when the other is input before the waiting time TW elapses from the input of any one of the phoneme information and the note information, the execution of the singing synthesis process is started with the other input as an opportunity, The dummy sound output process is continuously executed until the output of the synthesized singing voice is started. When the phoneme information is input for each phoneme such as a consonant and a vowel, the above determination may be made based on the input timing of the phoneme information of the head phoneme. Hereinafter, in order to make the singing synthesizer 1 synthesize a singing sound that pronounces the lyrics “sa” with a pitch “C4”, information indicating each of the consonant “s” and the vowel “a” is input as phonological information, The contents of the dummy sound output process will be described by taking as an example a case where information indicating the start and stop of the sound of the pitch “C4” is input as the note information.

  For example, in FIG. 2A, after the phoneme information indicating the consonant “s” is input at the time TA1, the phoneme information indicating the vowel “a” is input at the time TA2, and further the time TA3 (however, TA3-TA1). > TW), the case where the note information indicating the start of the sound of the pitch “C4” is input is illustrated. In this case, since note information is not input before the waiting time TW elapses from the input timing of the earliest synthesis information (that is, phoneme information about the first phoneme), input of phoneme information indicating the consonant “s” is input. When the waiting time TW elapses from the timing, the output of the dummy sound is stopped and the singing synthesis process is not executed. On the other hand, as shown in FIG. 2B, after the phoneme information indicating the consonant “s” is input at the time TA1, the phoneme information indicating the vowel “a” is input at the time TA2, and further the time TA3. When the note information indicating the start of the sound of the pitch “C4” is input at ′ (where TA3′−TA1 ≦ TW), the dummy sound continues to be output until the output of the synthesized singing voice is started. . In the example shown in FIG. 2B, as a dummy sound, a plurality of phonemic pieces (FIG. 2) such as transition from silence (indicated by # in FIG. 2B) to breath sound and transition to silence again. In the example shown in (b), the case where the output of the synthesized singing voice is started from the silent state using the sound constituted by [# -br], [br] and [br- #]) is illustrated. This is because the dummy sound and the synthesized singing voice are smoothly connected. However, if the smoothness of the joint between the dummy sound and the synthesized singing voice does not matter, a sound composed of one phoneme piece (for example, [# -br]) may be used as the dummy sound. Also, [br- #] in FIG. 2B is omitted, and [br-s] is used instead of [# -s] in FIG. 2B, or [br- #] in FIG. Instead of [-#], [br-s] may be used, and [# -s] may be omitted, for example, and the transition from the dummy sound to the synthesized singing voice may be made directly so that both are connected more smoothly.

  FIG. 2C and FIG. 2D are diagrams illustrating operations when note information is input first. Also in this case, if the phoneme information of the first phoneme is not input before the waiting time TW elapses from the input timing of the earliest synthesis information (note information indicating the start of pronunciation of the pitch “C4”), When the waiting time TW elapses from the input timing of the note information, the output of the dummy sound is stopped and the singing synthesis process is not executed (see FIG. 2C). On the contrary, as shown in FIG. 2D, when the phoneme information of the first phoneme is input from the input timing of the note information indicating the start of sound generation of the pitch “C4” until the waiting time TW elapses. In the same manner as in the case of FIG. 2B described above, the output of the dummy sound is continued until the output of the synthesized singing voice is started. When note information is input first among note information and phoneme information, a transition portion from a silence to a breath sound in the dummy sound (that is, [# -Br]) may be adjusted.

  As described above, in the singing voice synthesizing apparatus 1 according to the present embodiment, both the note information representing the notes constituting the melody of the singing voice and the phonological information representing the phonology of the lyrics to be pronounced in accordance with the notes are prepared. Although singing synthesis is started as an opportunity, it is not different from the conventional real-time singing synthesis technology, but the one input earlier (ie, multiple types of synthesis information used for singing synthesis) A dummy sound is output during the period from the point of input of the earliest input) until the output of synthesized speech is started. For this reason, a sound is output from the point of time when intentions for singing synthesis are performed, and the user does not feel uncomfortable. Further, according to the present embodiment, either phoneme information or note information may be input first, and there is no restriction on the input order of these information.

  In the present embodiment, a case has been described in which the singing synthesis process is not executed when all the synthesis information has not been collected before the waiting time TW elapses from the input timing of the earliest synthesis information. However, although the output of the dummy sound is stopped when the waiting time TW elapses, the subsequent synthesis information input continues waiting regardless of the elapse of the waiting time TW, and the singing is performed when all the synthesis information is gathered. Of course, it is also possible to execute the synthesis process. In order to produce a more natural audibility, in the dummy sound output process, a process of controlling the volume so that the volume gradually increases (a process of gradually increasing the signal level of the dummy sound signal to a predetermined value). May be executed by the control unit 110. Also, instead of smoothing the connection between the dummy sound and the synthesized singing voice according to the generation form of the dummy sound, the signal level of the dummy sound signal and the synthesized voice signal is adjusted so that the dummy sound and the synthesized singing sound crossfade You may make the control part 110 perform the process to perform. Specifically, while the output end time of the phoneme piece [br- #] in FIG. 2B is set to be later than the output start time of the phoneme piece [# -s], the former signal level is gradually increased to silence. And the latter signal level may be gradually increased from silence.

  In addition, an operation element for instructing the user to stop the output of the dummy sound is provided in the operation unit 120, and when the operation element is operated, a predetermined control signal is output to the operation unit 120, while the control unit 110 If the control signal is received during the waiting time TW, the execution of the dummy sound output process may be stopped at that time. According to such an aspect, even when phonological information or note information is erroneously input due to a mistouch to the operation unit 120 and the output of the dummy sound is started, the dummy sound is output by the operation of the operator. It can be stopped, and it can be avoided that the dummy sound continues to be output until the waiting time TW elapses from the occurrence of the erroneous input.

  Further, in this embodiment, regardless of the input order of the phoneme information and the note information, the dummy sound output process is executed by the control unit 110 triggered by the input of the first input, and the singing synthesis is triggered by the other input. The process is executed by the control unit 110. However, there are two types of operation modes of the synthesizer: a singing synthesis performance mode that synthesizes singing voice and an instrument sound performance mode that outputs the musical instrument sound at the pitch indicated by the note information without performing singing synthesis. Alternatively, the operation mode may be switched according to which of phoneme information and note information is input first. For example, when the phonological information is input first in the state of operating in the singing synthesis performance mode, the dummy sound output processing is executed by the control unit 110, and immediately when the musical note information is input first ( Alternatively, when the phonological information is not input before the waiting time TW elapses, the control unit 110 is caused to execute processing for switching the operation mode to the instrument sound performance mode. Similarly, when the phoneme information is input first in the musical instrument sound performance mode, the operation mode is immediately switched to the singing synthesis performance mode (or when the note information is not input before the waiting time TW elapses). The process is executed by the control unit 110, and when the note information is input first, the dummy sound output process is executed by the control unit 110. According to such an aspect, there is an effect that the user can be made to perform the singing synthesis performance and the musical instrument sound performance while switching seamlessly by an operation on the operation unit 120.

(B: Second embodiment)
In the first embodiment, the sound is irrelevant to the phoneme information and note information input by the user, such as a noise sound, breath sound, nasal sound, or a sound obtained by outputting a predetermined continuous phoneme at a predetermined pitch. Thus, the case where a continuable sound is used as a dummy sound has been described. On the other hand, the present embodiment is different from the first embodiment in that a dummy sound corresponding to the earliest synthesis information (that is, the phoneme information and the note information input earlier) is output. Since the hardware configuration of the singing voice synthesizing apparatus of the present embodiment is the same as that of the first embodiment, detailed description thereof is omitted (the same applies to the third and fourth embodiments). In the following, phoneme information is input first. The dummy sound output processing of this embodiment will be described separately for the case where the note information is input and the case where the note information is input first.

(B-1: When phoneme information is received first)
In this case, the control unit 110 determines whether or not the head phoneme indicated by the phoneme information is a continuable phoneme, and if the determination result is “Yes”, the head phoneme is determined as a predetermined predetermined phoneme. The sound that continues to be output at the pitch of is output as a dummy sound. On the other hand, when the determination result is “No”, the control unit 110 outputs a noise sound, a breath sound, a nasal sound, and a predetermined continuable phoneme with a predetermined pitch as in the first embodiment. A sound that continues to be output or a periodic sound having a predetermined pitch is output as a dummy sound.

(B-2: When note information is received first)
In this case, the control unit 110 outputs a periodic sound having a pitch indicated by the note information (or a sound that continues to output a predetermined continuable phoneme at the pitch) as a dummy sound.

  In summary, the dummy sound in the case of outputting a dummy sound according to the earliest synthesis information (one of phoneme information and note information input earlier) is classified into categories as shown in FIG. Can be divided. Even in this embodiment, even if there is a time difference between the input timings of a plurality of types of synthesis information used for singing synthesis, sound is output without delay from the input time of the first synthesis information, and the user does not feel uncomfortable. Can be. Note that the length of the waiting time TW may be different for each category shown in FIG. 3, and the user may be able to set the length of the waiting time TW for each type of phoneme. In addition, according to the present embodiment, since a dummy sound corresponding to the earliest synthesized information is output, compared to the case where a sound irrelevant to the information is used as a dummy sound, the dummy sound is changed to the synthesized singing voice. It is expected that the transition will be smooth and a more natural listening experience can be produced.

(C: Third embodiment)
In the first and second embodiments, the description has been given of the case where the dummy sound is output from the input time of the earliest synthesis information until the output of the synthesized singing voice is started. On the other hand, the present embodiment is characterized in that the dummy sound output from the input time point of the earliest synthesis information to the start of the output of the synthesized singing voice is sequentially switched. Specifically, as shown in FIG. 4A, the control unit 110 of the singing voice synthesizing apparatus according to the present embodiment outputs the dummy sound D1 from the input time of the phoneme information and the note information input earlier. Is started, the output of the dummy sound D1 is stopped with the other input as an opportunity, the output of the dummy sound D2 different from the dummy sound D1 is started, and the output of the dummy sound D2 is output until the output of the synthesized singing voice is started. It will continue.

  For example, FIG. 4A illustrates the case where phonological information is input first. In this case, the sound corresponding to the phoneme information is output as the dummy sound D1, and then the output of the dummy sound D1 is stopped when the note information is input, and the sound corresponding to the note information is output as the dummy sound D2. Can be output. Further, as the dummy sound D2, a sound obtained by converting the pitch of the dummy sound D1 so as to be the pitch indicated by the note information may be used. As described above, in the aspect in which the dummy sound is switched with the input of the phoneme information and the note information input later as a trigger, the dummy sound D1 and the dummy sound D2 may be crossfade, and further, the dummy sound D2 And the synthesized voice may be crossfade.

  When “sa” should be input as phoneme information, if “t” is erroneously input and then “sa” is input, as shown in FIG. 4B, the input of “t” is performed. If the control unit 110 executes the process of outputting the dummy sound D10 triggered by the input of "s", outputting the dummy sound D20 triggered by the input of "s", and outputting the dummy sound D30 triggered by the input of the note information. good. In this case, according to the category classification shown in FIG. 3, “t” cannot be continued, so a breath sound or the like is used as the dummy sound D10, and “s” can be continued, so that the phoneme (that is, “ It is sufficient to use a sound output at a predetermined pitch and use a sound output at “s” at the pitch indicated by the note information as the dummy sound D30.

  Even in this embodiment, even if there is a time difference between the input timings of the plurality of types of synthesis information used for singing synthesis, the sound is output without delay from the input timing of the first synthesis information, and the user feels uncomfortable. I will not let you. Further, if the sound corresponding to the earlier input of the note information and the phoneme information is used as the dummy sound D1, and the sound corresponding to both the note information and the phoneme information is used as the dummy sound D2, the above-mentioned first It is expected that the dummy sound and the synthesized singing voice are connected more smoothly than in the second embodiment, and a more natural hearing feeling can be produced. In the present embodiment, the case where the dummy sound is switched in response to the input of another type of synthesis information has been described. However, the dummy sound D2 is switched when a predetermined time elapses after the output of the dummy sound D1 is started. You may do it. Of course, the dummy sound may be switched a plurality of times during the period from the input of the earliest synthesis information until the output of the synthesized singing voice is started. In short, a mode in which the dummy sound signal is switched so that a plurality of types of dummy sounds are sequentially output from when the first one of the plurality of types of synthesis information is input until the output of the synthesized singing voice is started. If it is good.

(D: 4th Embodiment)
In the first to third embodiments, the dummy sound output process is started in response to the input of the earliest synthesis information (one of note information and phoneme information input first). On the other hand, in the present embodiment, as shown in FIG. 5, the point that the output of the dummy sound is started when a predetermined time TM has elapsed from the input time point of the earliest synthesis information as a starting point. In the present embodiment, when the other of the note information and the phoneme information is input before the predetermined time TM elapses and the head phoneme indicated by the phoneme information is a continuous phoneme, the note information A sound corresponding to both the phoneme information (for example, a sound that continues to output the phoneme at the pitch indicated by the note information) is output as a dummy sound, and in other cases, a sound corresponding to the information input earlier Alternatively, a breath sound or the like is output as a dummy sound. Even in such a mode, since the dummy sound is output prior to the synthesized singing voice, the user's uncomfortable feeling can be reduced. In addition, since there is an individual difference in how much delay is allowed from the input operation until the output of the synthesized singing voice is actually started, the length of the time TM can be appropriately adjusted according to the user's desire. Is preferred.

(E: deformation)
Although each embodiment of the present invention has been described above, it goes without saying that the following modifications may be added to these embodiments.
(1) In each of the embodiments described above, phonological information and note information (prosodic information) have been described as specific examples of a plurality of types of synthetic information used for singing synthesis. However, velocity is added to phonological information and musical note information (prosodic information). Of course, the note control information may be used for the output control of the dummy sound signal. Velocity is information indicating the strength of sound. In MIDI, note information is formed together with pitch information indicating pitch. As a method of using such velocity, there is a mode in which the volume at which the dummy sound begins to be output is controlled according to the velocity included in the note information for the previous note (the volume is increased as the velocity is increased). Conceivable. Examples of note control information include vibrato and attack or release as control data. When note control information indicating vibrato is given, vibrato is given to the dummy sound, and the rising volume of the dummy sound may be changed in accordance with the magnitude of the attack as the control data.

  When velocity is used as synthesis information, velocity constitutes note information together with pitch information. For this reason, velocity is also acquired by acquiring the note information that plays the role of prosodic information. However, the information indicating the addition of vibrato is not always acquired at the same time as any of the essential synthesis information. On the other hand, it is not preferable that the dummy sound continues to be output even though essential synthesis information is available. Therefore, in addition to phonological information and prosodic information, when using information that is not necessarily essential, such as information indicating the addition of vibrato, as synthetic information, essential synthetic information from the input of the first of a plurality of types of synthetic information It is only necessary to synthesize the singing voice by using the synthesis information input until all the voices are arranged. For example, when the earliest synthesis information is phoneme information, when note information is input after phoneme information, synthesis of singing voice is started at that point, while vibrato is added after phoneme information When the information indicating is input, the singing voice to which the vibrato is added is synthesized after waiting for the input of the note information. When the output of a dummy sound that outputs the head phoneme indicated by the phoneme information at a predetermined pitch is started when the phoneme information is input, the dummy sound is triggered by the input of information indicating the addition of vibrato. Vibrato based on the pitch may be given to the sound. If the earliest synthesized information is neither phoneme information nor prosodic information, a predetermined sound such as a noise sound or a breath sound may be output as a dummy sound as in the first embodiment. .

(2) In each of the above embodiments, the operation unit 120 for inputting a plurality of types of synthesis information used for singing synthesis and the audio output unit 140 for outputting a synthesized singing voice are incorporated in the singing synthesis apparatus 1. . However, an aspect in which one or both of the operation unit 120 and the audio output unit 140 are connected to the external device I / F unit 150 of the singing voice synthesizing apparatus 1 may be employed. As an example of a mode in which both the operation unit 120 and the audio output unit 140 are connected to the external device I / F unit 150, an Ethernet (registered trademark) interface is used as the external device I / F unit 150. A mode in which a telecommunication line such as a LAN (Local Area Network) or the Internet is connected to the unit 150, and the operation unit 120 and the audio output unit 140 are connected to the telecommunication line. According to such an aspect, it is possible to provide a so-called cloud computing type song synthesis service. Specifically, the phoneme information and the note information input by the operation of the operation unit 120 are given to the singing synthesizer via the telecommunication line, and the phonological information given via the telecommunication line A singing synthesis process is executed based on the note information. Then, the voice data of the synthesized singing voice synthesized by the singing voice synthesizer is given to the voice output unit 140 via the telecommunication line, and the voice output unit 140 outputs a sound corresponding to the voice data.

(3) In each of the above embodiments, the singing synthesis keyboard is used as an input means (operation unit 120) for inputting a plurality of types of synthesis information to the singing synthesis apparatus. A combination of a general keyboard on which corresponding keys are arranged and a so-called MIDI keyboard may be used as the input means. When a combination of a general keyboard and a MIDI keyboard is used as the input means, the MIDI keyboard may serve as a note information input unit, and the general keyboard may serve as a phoneme information input unit. Further, a note information input unit or a phoneme information input unit may be realized by a combination of a GUI and a pointing device such as a mouse. When the note information input unit is realized by a combination of a GUI and a pointing device such as a mouse, the input means is realized by a combination of a general keyboard serving as a phoneme information input unit and the note information input unit. be able to. When the phoneme information input unit is realized by a combination of a GUI and a pointing device such as a mouse, the input means is realized by a combination of a MIDI keyboard serving as a note information input unit and the phoneme information input unit. be able to.

(4) In each of the above embodiments, the song synthesis program 162b that causes the control unit 110 to perform the song synthesis process and the dummy sound output process is stored in the nonvolatile storage unit 162 of the song synthesis apparatus 1 in advance. However, the song synthesis program 162b may be distributed by being written on a computer-readable recording medium such as a CD-ROM, or may be distributed by download via an electric communication line such as the Internet. This is because by causing a general computer such as a personal computer to execute the distributed program in this way, the computer can function as the singing voice synthesizing apparatus 1 of the above embodiment. Of course, the present invention may be applied to a game program of a game partially including a real-time singing synthesis process. Specifically, the song synthesis program included in the game program may be replaced with a song synthesis program 162b. This is because it is preferable to reduce the time difference between the input timing of the earliest synthesis information and the output timing of the synthesized speech even in the game.

(5) In the above embodiments, application examples of the present invention to a real-time singing voice synthesizing apparatus have been described. However, the application target of the present invention is not limited to the real-time singing synthesizer. For example, the present invention may be applied to a voice synthesizer that synthesizes guidance voices in voice guidance in real time, or a voice synthesizer that synthesizes reading sounds of literary works such as novels and poems in real time. In these speech synthesizers, the speech synthesis process is executed when the phoneme information indicating the utterance content and the prosodic information indicating the utterance form are prepared, which is the same as the singing synthesizers of the above embodiments. Because. The application target of the present invention may be a toy having a singing voice synthesis function or a voice synthesis function (a toy incorporating a singing voice synthesis device or a voice synthesis device).

DESCRIPTION OF SYMBOLS 1 ... Singing synthesis apparatus, 110 ... Control part, 120 ... Operation part, 130 ... Display part, 140 ... Audio | voice output part, 142 ... D / A converter, 144 ... Amplifier, 146 ... Speaker, 150 ... External apparatus I / F , 160 ... storage unit, 162 ... nonvolatile storage unit, 162a ... singing synthesis library, 162b ... singing synthesis program, 162c ... dummy sound library, 164 ... volatile storage unit, 170 ... bus.

Claims (5)

Input means for inputting a plurality of types of synthesis information including a plurality of types of synthesis information used for synthesis of a speech signal, including phonological information indicating the phoneme of the synthesis target speech and prosody information indicating prosodic change in the speech When,
Using the composite information input through the input unit after the earliest one of the plurality of types of composite information is input to the input unit until at least the phonological information and the prosodic information are aligned. A voice synthesizer that outputs a dummy sound signal representing a dummy sound from the time when the earliest synthesis information is input until the output of the voice signal is started.
A speech synthesizer characterized by comprising:   The voice synthesizing unit generates a dummy sound signal or the voice signal so that the voice represented by the voice signal synthesized using the synthesis information input to the input unit and the dummy sound are smoothly connected. The speech synthesizer according to claim 1, wherein the signal levels of both the voice signal and the dummy sound signal are adjusted and output.   3. The speech synthesizer according to claim 1, wherein the speech synthesizer outputs a dummy sound signal while adjusting a signal level so that the volume of the dummy sound gradually increases.   The voice synthesizing unit is a period from when the earliest one of the plurality of types of synthesis information is input to when the output of the voice signal synthesized using the synthesis information input to the input unit is started. The voice synthesizer according to claim 1, wherein the dummy sound signals are switched so that a plurality of types of dummy sounds are sequentially output. The speech synthesis means, when at least the phoneme information and the prosodic information are not aligned before a predetermined waiting time elapses after the first one of the plurality of types of synthesis information is input, 4. The speech synthesizer according to claim 1, wherein output of the dummy sound signal is stopped.

Images