JP6060520B2 - Speech synthesizer - Google Patents

Description

  The present invention relates to a speech synthesizer.

  Various techniques for synthesizing speech imitating human real voice have been proposed. For example, in Patent Document 1, a human voice (hereinafter referred to as “input speech”) is cut out for each phoneme, and speech segments are collected, and arbitrary speech is obtained by connecting these speech segments to each other. Techniques for synthesizing are disclosed. Patent Document 2 proposes a technique for synthesizing a singing voice in accordance with a performance freely performed by a user.

JP 2003-255974 A JP 2008-170592 A

By the way, in the apparatus that synthesizes the singing voice in accordance with the performance freely performed by the user as in the technique described in Patent Document 2, the performance by the user is performed by the decoding processing of the speech segment data, the speech synthesis processing, or the like. There is a delay between operation and sound / mute. In particular, if the timing of mute is delayed, the user who is playing may feel uncomfortable.
The present invention has been made in view of the above-described background, and in an apparatus for generating synthesized speech in accordance with a performance freely performed by a user, until a synthesized speech is muted after a performance operation for silencing is performed. It is an object of the present invention to provide a technique capable of reducing the delay of the system.

In order to solve the above-described problem, the present invention provides a pronunciation instruction acquisition unit that acquires a pronunciation instruction indicating a pronunciation start instruction or a pronunciation end instruction of a phoneme, and when the pronunciation instruction acquisition unit acquires the pronunciation instruction, A phoneme specifying unit that specifies a phoneme corresponding to the acquired pronunciation instruction, and a phoneme unit that is a set of phoneme unit data in which data representing a phoneme waveform for each phoneme and phoneme chain is encoded according to a predetermined rule A speech unit data acquisition unit that acquires speech unit data corresponding to the phoneme specified by the phoneme specifying unit from a database, and a decoding unit that decodes the speech unit data acquired by the speech unit data acquisition unit Generating synthesized speech data representing synthesized speech using the speech segment data decoded by the decoding unit, and outputting the generated synthesized speech data When the pronunciation instruction acquisition unit acquires the pronunciation termination instruction, the speech unit data is registered in the speech unit database and represents a speech waveform of a transition part from a predetermined phoneme to silence. Corresponding to the speech waveform of the transition part from the phoneme specified by the phoneme specifying unit as a phoneme corresponding to the acquired sounding end instruction from the storage unit storing decoded data obtained by decoding the segment data Read the decoded data, generate synthesized speech data using the read decoded data, output the synthesized speech data generated, and timing before the pronunciation instruction acquisition unit acquires the pronunciation termination instruction , The speech unit data representing the speech waveform of the transition from the phoneme to the silence satisfying the predetermined condition is decoded and the decoded data is decoded. It generates and provides a speech synthesis apparatus characterized by the generated decoded data to and a decoded data storage control unit for storing in the storage unit.

  Further, in a further preferred aspect of the present invention, the decoded data storage control unit is specified by the phoneme specifying unit as a phoneme corresponding to the pronunciation start instruction when the pronunciation instruction acquisition unit acquires the pronunciation start instruction. The decoded speech data may be generated by decoding the speech segment data representing the speech waveform of the transition from the phoneme to the silence, and the generated decoded data may be stored in the storage unit.

Further, in a further preferred aspect of the present invention, the decoded data storage control unit selects one or a plurality of phonemes based on the processing history in the speech synthesis unit, and the speech of the transition portion from the selected phonemes to silence. The speech unit data representing the waveform may be decoded to generate decoded data, and the generated decoded data may be stored in the storage unit .

Further, in a further preferred aspect of the present invention, the speech synthesizer has the pronunciation instruction acquisition unit acquire the pronunciation start instruction, and decoded data corresponding to a phoneme corresponding to the acquired pronunciation start instruction is stored. When stored in the storage unit, information for notifying the user may be output.
Further, in a further preferred aspect of the present invention, the storage unit may store the decoded data over a period until the speech synthesis process by the speech synthesizer ends.

  ADVANTAGE OF THE INVENTION According to this invention, in the apparatus which produces | generates a synthetic | combination voice according to the performance performed freely by the user, the delay until a synthetic | combination voice is muted after performing performance operation for mute can be reduced.

Block diagram showing an example of the configuration of a speech synthesis system A diagram showing an example of speech segments The figure which shows an example of the contents of the speech segment database Block diagram showing an example of the functional configuration of a speech synthesizer Timing chart showing timing of processing performed by speech synthesizer Timing chart showing timing of processing performed by speech synthesizer Block diagram showing an example of system configuration

  Embodiments of the present invention will be described with reference to the drawings. Each embodiment shown below is the aspect which applied this invention in order to synthesize | combine the song voice of a music.

1. Configuration FIG. 1 is a block diagram showing a configuration of a singing voice synthesis system according to an embodiment of the present invention. This singing voice synthesizing system is formed by connecting a performance data generating apparatus 100 and a voice synthesizing apparatus 200 via a communication cable 300 such as a MIDI cable. In this singing synthesis system, the performance data generating device 100 is a device that generates a MIDI message as performance data in response to a user's performance operation, and is, for example, a MIDI keyboard instrument. The speech synthesizer 200 is a device that generates synthesized speech, such as a personal computer or a mobile communication terminal. The voice synthesizer 200 sequentially uses the lyric data stored in advance while changing the singing position in response to the MIDI messages sequentially given from the performance data generator 100, and sings with a pitch and a pronunciation period determined by the MIDI message. Synthesize speech in real time.

  The performance data generating apparatus 100 includes a control unit 110, a storage unit 120, an operation unit 130, a display unit 140, an interface unit 150, a keyboard unit 160, and a sound source unit 170, which are connected via a bus 190. . The control unit 110 includes a processor such as a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. In the control unit 110, a processor such as a CPU reads out a computer program stored in the ROM or the storage unit 120, loads it into the RAM, and executes it, whereby each unit of the performance data generating apparatus 100 is controlled.

  The storage unit 120 is a storage unit for storing various data. As the storage unit 120, various storage devices such as a hard disk device incorporating a magnetic disk and a device for driving a portable recording medium represented by a CD-ROM are employed. The operation unit 130 includes various operators and outputs an operation signal representing the content of an operation performed by the user to the control unit 110. The display unit 140 includes a liquid crystal panel, for example, and displays various images under the control of the control unit 110. The interface unit 150 includes a MIDI interface for transmitting a MIDI message as performance data, and a driver for transferring data to and from an external recording medium such as a magnetic disk or a CD-ROM.

  The keyboard unit 160 includes a plurality of keys pressed by the user and a key switch circuit that supplies the control unit 110 with a signal indicating whether each key is pressed or released. The sound source unit 170 is a device that forms a digital musical tone signal in accordance with a command from the control unit 110. The sound system 180 includes a D / A converter that converts the digital musical tone signal formed in the sound source unit 170 into an analog musical tone signal, an amplifier that amplifies the analog musical tone signal, and a speaker that outputs the output signal of the amplifier as sound. Etc.

  The voice synthesizer 200 includes a control unit 210, a storage unit 220, an operation unit 230, a display unit 240, an interface unit 250, and a sound system 260, which are connected via a bus 270. The control unit 210 includes a processor such as a CPU, a RAM, a ROM, and the like. In the control unit 210, a processor such as a CPU reads out a computer program stored in the ROM or the storage unit 220, loads it into the RAM, and executes it, whereby each unit of the speech synthesizer 200 is controlled.

  The storage unit 220 is a storage unit for storing various data. As the storage unit 220, various storage devices such as a hard disk device incorporating a magnetic disk and a device for driving a portable recording medium represented by a CD-ROM are employed. The operation unit 230 includes various operators and outputs an operation signal representing the content of the operation performed by the user to the control unit 210. The display unit 240 includes a liquid crystal panel, for example, and displays various images under the control of the control unit 210. The interface unit 250 includes a MIDI interface for receiving a MIDI message as performance data, a driver for transferring data to and from an external recording medium such as a magnetic disk and a CD-ROM, and the like. The sound system 260 outputs the voice synthesized by the voice synthesizer 200, a D / A converter that converts singing voice data that is sample data of the synthesized voice into an analog voice signal, and the analog voice signal. The amplifier includes an amplifier that amplifies and a speaker that outputs the output signal of the amplifier as sound.

  The storage unit 220 includes a speech segment database DB1 storage area 221 and a lyrics data storage area 222, as shown. The speech unit database DB1 storage area 221 stores a speech unit database DB1, which is a set of data representing speech units composed of one or more phonemes (hereinafter referred to as “speech unit data”). Has been. The speech segment data is data used when generating synthesized speech. In the present embodiment, the “phoneme” corresponds to a minimum unit of language semantic distinction (for example, vowels and consonants), and is set in consideration of the actual articulation of a certain language and the entire phonological system. It is the smallest phonological unit of the language. The speech segment is obtained by cutting out a section corresponding to a desired phoneme or phoneme chain from the input speech uttered by a specific speaker. The speech segment data in the present embodiment is data indicating the frequency spectrum of the speech segment. In the following description, the term “speech segment” is used to include a single phoneme (for example, a monophone) or a phoneme chain (for example, a diphone or a triphone).

  Specifically, in the speech unit database DB1, speech unit data of single phonemes such as [a] and [i], and phoneme chains such as [ai] and [ap] are stored. Are stored. In this embodiment, phonetic symbols defined by SAMPA (Speech Assessment Methods Phonetic Alphabet) are used as phonetic symbols representing phonemes. The symbols representing phonemes are not limited to this, and may be in accordance with other standards such as X-SAMPA (Extended SAM Phonetic Alphabet) and IPA (International Phonetic Symbols).

  FIG. 2 is a diagram illustrating an example of a speech element. (A1) in FIG. 2 shows a waveform of a phoneme chain [s−a] in which a phoneme [s] of a consonant and a phoneme [a] of a vowel following the consonant are combined. In the waveform shown in (a1) of FIG. 2, the time point Ta1 is specified as the start point of the phoneme [s], the time point Ta3 is specified as the end point of the phoneme [a], and the time point Ta2 is further specified as the phoneme [s] and the phoneme [a. ] Is assumed as the boundary. As shown in (a1) of FIG. 2, the waveform of the phoneme [a] gradually increases in amplitude from the time point Ta2 so as to correspond to the action of the speaker who opens his / her mouth for the utterance. When the speaker exceeds the time Ta0 when the mouth is fully opened, the amplitude is maintained substantially constant. As the end point Ta3 of the phoneme [a], a time point after the waveform of the phoneme [a] transitions to a steady state (that is, a time point after the time point Ta0 shown in (a1) of FIG. 2) is selected.

  On the other hand, (b1) of FIG. 2 shows the waveform of the speech segment [a- #] including the vowel phoneme [a]. [#] Represents silence. The waveform of the phoneme [a] contained in the speech segment [a- #] is that the voice is gradually closed after the speaker speaks with the mouth fully open, and finally the mouth is completely closed. The shape corresponds to the utterance action of being closed. That is, the amplitude of the phoneme [a] waveform is initially maintained substantially constant, and the amplitude gradually decreases from the time point Tb0 when the speaker starts the closing operation (steady point) Tb0. The start point Tb1 of such a speech element is selected as a point in time during which the waveform of the phoneme [a] is maintained in a steady state (that is, a point before the steady point Tb0).

  The speech segment whose range on the time axis is defined as described above is divided into frames F having a predetermined time length (for example, 5 ms to 10 ms). As shown in FIG. 2 (a1), each frame F is selected so that the subsequent frame partially overlaps the previous frame on the time axis. These frames F are simply set as sections having the same time length, but the time length of each frame F may be changed according to the pitch of the speech segment, for example. A frequency spectrum including FFT (Fast Fourier Transform) processing is performed on the waveform of each frame F divided in this way to specify a frequency spectrum, and data representing these frequency spectra is encoded as speech unit data. Is stored in the speech unit database DB1. Therefore, as shown in (a2) and (b2) of FIG. 2, the speech unit data of each speech unit is composed of a plurality of unit data D (D1, D2,. ……)including.

  FIG. 3 is a diagram illustrating an example of the contents of the speech unit database DB1. As shown in the figure, speech unit data (all or a part thereof) is encoded and stored for each phoneme or phoneme chain in the speech unit database DB1. A plurality of speech element data may be stored for each pitch, for each gain, and for each wavelength for one phoneme or one phoneme chain. For example, with respect to the pitch, three speech element data, that is, speech element data for high sounds, speech element data for medium sounds, and speech element data for bass sounds, are stored for one phoneme (or one phoneme chain). May be. As described above, different speech segment data may be stored for each attribute of one phoneme or one phoneme chain in the speech segment database DB1.

  In the lyrics data storage area 222, a character string input by the operation of the operation unit 230 is stored as lyrics data. When the character string input by the operation of the operation unit 230 is stored as lyrics data in the RAM and a storage command is input by the operation of the operation unit 230, the control unit 210 converts the lyrics data in the RAM to the lyrics data storage area 222. To store. The lyrics data storage area 222 stores the lyrics data stored in this way and a set of lyrics data input from the outside via the interface unit 250.

  Next, an example of a functional configuration of the speech synthesizer 200 will be described with reference to the block diagram shown in FIG. In FIG. 4, an initial setting unit 211, a MIDI message reception processing unit 212, and a singing synthesis unit 213 read out the computer program stored in the ROM or the storage unit 220 by the processor of the control unit 210 and load it into the RAM. It is realized by executing.

  The initial setting unit 211 performs various initial settings at the timing when an application for generating a singing voice (hereinafter referred to as a “song synthesis application”) is activated. In particular, the initial setting unit 211 reads out speech unit data representing a speech waveform of a transition portion from a predetermined phoneme (in this embodiment, a vowel and a nasal sound) to silence, from the speech unit database DB1, and reads out the speech Decode the fragment data. The initial setting unit 211 stores the decoded speech segment data (decoded data) in a predetermined storage area in the RAM (hereinafter referred to as “decoded data storage area 231”). In the decoded data storage area 231, each of the speech units [a- #], [i- #], [u- #], [e- #], [o- #], and [n- #] is stored. Decoded data obtained by decoding speech element data representing a speech waveform is stored. The initial setting unit 211 is an example of a decoded data storage control unit according to the present invention. In the following description, for the convenience of explanation, speech segment data representing a speech waveform of a transition portion from a predetermined phoneme to silence is referred to as “silence transition data”.

  In addition, for one phoneme or one phoneme chain, when speech unit data different for each attribute (pitch, gain, wavelength, etc.) is stored in the speech unit database DB1, initial setting is performed. The unit 211 may generate decoded data by performing a decoding process on speech unit data having all attributes corresponding to predetermined phonemes. For example, the initial setting unit 211 has three speech element data, that is, a speech element data for high sounds, a speech element data for medium sounds, and a speech element data for low sounds for one phoneme or one phoneme chain. Is stored, these three speech element data may be decoded. In addition, the initial setting unit 211 performs decoding processing on the speech unit data of some attributes (predetermined attributes) of the speech unit data corresponding to the predetermined phonemes, and the decoded data May be generated. In this way, when speech unit data different for each attribute is recorded in the database, the conversion processing unit 216 described later may select speech unit data corresponding to the attribute of the sound to be synthesized. For example, when there is speech unit data for high, medium, and low sounds regarding the pitch, the conversion processing unit 216 may select speech unit data having a pitch closest to the note. By doing so, a high-quality synthesized sound can be obtained.

  The decoded data storage area 231 is released as the song synthesis application ends. That is, the decoded data storage area 231 stores the decoded silence transition data (decoded data) over a period until the singing voice synthesis process is completed.

  The MIDI message reception processing unit 212 is an interrupt routine that is activated when the MIDI interface in the interface unit 250 receives a MIDI message. The MIDI message reception processing unit 212 interprets the MIDI message received by the MIDI interface, and supplies control information to each unit in the song synthesis unit 213 according to the interpretation result. The MIDI message reception processing unit 212 is an example of a sound generation instruction acquisition unit that acquires a sound generation instruction (MIDI message) indicating note-on (phoneme sounding start) or note-off (phoneme sounding end).

  The singing voice synthesizing unit 213 includes a singing position update processing unit 214, a segment selection unit 215, a conversion processing unit 216, and a segment coupling unit 219. The singing position update processing unit 214 is a routine for updating the position of the singing target portion for performing singing synthesis among the lyrics indicated by the lyric data, that is, the singing position. The singing position update processing unit 214 is an example of a phoneme specifying unit that specifies a phonetic character (phoneme) corresponding to the acquired MIDI message when the MIDI message reception processing unit 212 receives (acquires) the MIDI message. . The singing position update processing unit 214 notifies the segment selection unit 215 and the segment coupling unit 219 of the phonogram that is the singing target portion of the lyrics indicated by the lyrics data. There are the following two types of opportunities for the singing position update processing unit 214 to update the singing position.

  The first opportunity is when a note-on message is received as a MIDI message by the MIDI interface, and the note-on message is delivered from the MIDI message reception processing unit 212 to the singing position update processing unit 214. In this case, for example, in the case of Japanese singing, the singing position update processing unit 214 selects a phonogram next to the phonic character that is the current singing target portion of the phonic character string constituting the lyrics. It becomes a new singing target part. In the case of English singing, a unit of pronunciation is usually a vowel letter or a set of consonant letters and vowel letters that are uttered according to one note. Therefore, in the case of English singing, the singing position update processing unit 214 is located immediately after the current singing target portion in the phonogram string constituting the lyrics, and is a vowel character or consonant character and vowel as a pronunciation unit. A character set is a new singing target part.

  The second opportunity is that a system exclusive message including a song position change instruction is received as a MIDI message by the MIDI interface, and the song position change instruction is delivered from the MIDI message reception processing unit 212 to the song position update processing unit 214. Is the time. The singing position change instruction includes various instructions such as returning to the previous note, returning to the previous note,..., Returning to the beginning of the song. The singing position update processing unit 214 changes the singing position in the lyrics data in accordance with the singing position changing instruction.

  The segment selection unit 215 is a routine that reads the speech segment data corresponding to the singing position updated by the singing position updating unit 213. The element selection unit 215 refers to a phonetic character (a vowel character or a set of consonant characters and a vowel character in the case of English) corresponding to the current singing position. By referring to the phonetic character that was a part, one or a plurality of speech elements for which the speech waveform is synthesized is determined. The unit selection unit 215 reads out the speech unit data corresponding to the determined speech unit from the speech unit database DB1, and delivers it to the conversion processing unit 216. The segment selection unit 215 is an example of a speech segment data acquisition unit according to the present invention.

  The conversion processing unit 216 includes a decoding unit 217 and a speech synthesis unit 218. The decoding unit 217 decodes the speech unit data delivered from the unit selection unit 215 and develops it in the RAM. For example, the decoding unit 217 may decode waveform compressed data that is compressed (encoded) by an audio compression algorithm such as ADPCM (Adaptive Differential Pulse Code Modulation) or MP3 (MPEG Audio Layer-3), Also, a configuration that decodes what is encoded by a method such as speech segment data indicating each component divided into harmonic components (Deterministic Component) and non-harmonic components (Stochastic Component) by SMS (Spectral Modeling Synthesis) technology It may be. Of course, encoding / decoding may be performed using a method capable of handling speech segment data, such as a speech synthesis method or a waveform compression method. The speech synthesizer 218 generates synthesized speech data (singing speech data) representing the synthesized speech using the speech segment data delivered from the segment selector 215. That is, the speech synthesizer 218 performs pitch conversion and inverse FFT on the segment corresponding to voiced sounds such as vowels and voiced consonants in the speech segment data decoded by the decoder 217, and outputs a MIDI message. It is output as singing voice data (digital voice signal in the time domain) having the pitch indicated by the pitch data delivered from the reception processing unit 212.

  The voice synthesizer 218 also serves to adjust the duration of the singing voice data of consonants and vowels. Furthermore, the speech synthesizer 218 has a function of changing the pitch and volume of the singing voice data for adding a musical expression such as vibrato, pitch bend, and sound intensity. An instruction for musical expression is supplied as a MIDI message from the performance data generating apparatus 100 to the speech synthesizer 200 and is given to the speech synthesizer 218 via the MIDI message reception processor 212.

  The voice synthesizer 218 terminates the pronunciation of the singing target portion triggered by reception of a note-off message or reception of a note-on message of a subsequent note. That is, the speech synthesizer 218 stops outputting the generated synthesized speech data when receiving a note-off message or when receiving a note-on message of a subsequent note. At this time, the speech synthesizer 218 receives a subsequent note-on message before receiving the note-off message or a predetermined time after receiving the note-off message (for example, 1/4 measure, 0. 5 seconds, etc. When the note-on message of the subsequent note is received within the following “predetermined time T3”), the synthesized speech data is generated using the speech segment data corresponding to the subsequent note. To do. That is, when the speech synthesis unit 218 receives the subsequent note-on message, the speech synthesis unit 218 causes the decoding unit 217 to decode the speech unit data corresponding to the subsequent note, and uses the speech unit data decoded by the decoding unit 217. To generate synthesized voice data and output the generated synthesized voice data. Specifically, for example, when a note-on message corresponding to phoneme [i] is received after receiving a note-on message corresponding to phoneme [a] without receiving a note-off message, It is assumed that a note-on message corresponding to phoneme [i] is received within a predetermined time T3 after receiving a corresponding note-off message. In such a case, the speech synthesizer 218 reads speech unit data representing the speech waveform of the speech unit [ai] from the speech unit database DB1, and uses the read speech unit data to synthesize speech. Generate data.

  On the other hand, the speech synthesizer 218 receives the note-on message from the decoded data storage area 231 if it does not receive the note-on message of the subsequent note within a predetermined time T3 after receiving the note-off message. The phoneme silence transition data corresponding to the note-off message is read, synthesized speech data is generated using the read silence transition data, and the generated synthesized speech data is output. Specifically, for example, when a note-on message for the next phoneme is not received within a predetermined time T3 after receiving a note-off message corresponding to the phoneme [a], the speech synthesizer 218 reads decoded data representing the speech waveform of the speech unit [a- #] from the decoded data storage area 231 and generates synthesized speech data using the read speech unit data.

  The segment connecting unit 219 adjusts the singing voice data that has undergone the processing of the conversion processing unit 216 so that the finally obtained singing voice is a series of speech units smoothly connected. Furthermore, the segment linking unit 219 performs a delay process on the singing voice data using the buffer area in the RAM under the control of the MIDI message reception processing unit 212, and gives the singing voice data to the sound system 260. As a function to perform timing control for sound emission.

  FIG. 5 is a timing chart showing the timing of decoding processing and speech synthesis processing according to this embodiment. FIG. 6 is a timing chart showing the timing of the decoding process and the voice synthesizing process in the conventional voice synthesizing apparatus (the apparatus that does not perform the silence transition data decoding process in advance). In the example shown in FIG. 5 and FIG. 6, the processing timing in the case where the synthesized speech is generated for the phonetic character “a” is shown. In the present embodiment, as shown in FIG. 5, at the timing T <b> 2-1 when the system is activated, the initial setting unit 211 decodes the silence transition data (T <b> 2-2) and stores it in the decoded data storage area 231. Keep it. After that, when the MIDI message reception processing unit 212 receives a note-on message for the phonetic character “a” at the timing T2-3, the segment selection unit 215 performs the speech unit [# -a] speech unit. The segment data and the speech unit data of the speech unit [a] are sequentially read out from the speech unit database DB1 and decoded (T2-4, T2-5). The speech synthesis unit 218 sequentially performs speech synthesis processing using the speech segment data that has been subjected to decoding processing (T2-7, T2-8). At this time, a delay time T2-6 as shown in FIG. 5 occurs due to the processing time of the decoding process between the reception of the note-on message and the start of the speech synthesis process.

  The speech synthesis data generated by the speech synthesizer 218 is sequentially supplied to the segment linking unit 219, and after the segments are connected to each other, the speech synthesis data is supplied to the sound system 260 and output as sound (T2- 10, T2-11). At this time, a delay time T2-9 as shown in FIG. 5 occurs due to the processing time of the speech synthesis process between the start of the speech synthesis process and the start of the speech process (sound generation process). To do.

  Thereafter, the MIDI message reception processing unit 212 receives a note-off message at timing T2-12, and receives a note-on message of the next phoneme within a predetermined time T3 after receiving the note-off message. If not, the speech synthesizer 218 reads [a- #] decoded speech element data (silence transition data) from the decoded data storage area 231 and performs speech synthesis processing (T2-13). ). The speech synthesis data generated by the speech synthesis processing is supplied to the segment connecting unit 219, and after the segments are connected to each other, the speech synthesis data is supplied to the sound system 260 and output as sound (T2-15). As shown in FIG. 5, the period from the start of the speech synthesis process to the start of the speech process (sound generation process) for the speech unit data [a- #] is attributed to the processing time of the speech synthesis process. A delay time T2-14 occurs.

  Here, the timing chart shown in FIG. 5 is different from the timing chart shown in FIG. 6 in that the timing at which [a- #] speech segment data is decoded is different from that in [a- #]. There is a difference in whether or not a delay time (time T1-12 in FIG. 6) occurs between the start of the decoding process for the segment data and the start of the speech synthesis process. In the example shown in FIG. 6, after the MIDI message reception processing unit 212 receives the note-off message at timing T1-10, the decoding unit 217 performs decoding processing of speech unit data for [a- #] (time T1 -11) After that, speech synthesis processing is performed (time T1-13). Therefore, in the example illustrated in FIG. 6, the delay time T1− is from the time when the decoding process for [a− #] is started (that is, after the note-off message is received) until the voice synthesis process is started. 12 occurs.

  As is apparent from a comparison between FIG. 5 and FIG. 6, in this embodiment, this occurs from the start of decoding of speech segment data including a transition portion from a vowel (or nasal sound) to silence until the start of synthesis processing. The possible delay time (time (T1-12) in FIG. 6) can be suppressed. Thereby, it is possible to reduce the delay from when the note-off message is received until the synthesized speech is muted. Therefore, for example, when synthesizing a song, it can be avoided that the note that is sounding (note off) and the note on the next note are connected to the next note, and the note-off tracking performance is improved. Can be raised. In particular, in the performance of real-time speech (singing) synthesis, there is a case where it is desired to accurately control the timing of the end of the singing, and improving the follow-up performance of note-off has a great effect in such a case.

2. Operation Next, the operation of this embodiment will be described.

2-1. Preparatory work for singing synthesis The user starts up a necessary application when the voice synthesizer 200 performs singing synthesis. That is, by operating the operation unit 230, the control unit 210 is instructed to start an application for inputting lyrics data (hereinafter referred to as “lyric data input application”) or a song composition application. Further, the user instructs the control unit 110 to start an application for generating a MIDI message by operating the operation unit 130.

  When the application is started, first, the initial setting unit 211 performs various initial settings and also performs silence transition data decoding processing, and stores the silence transition data subjected to the decoding processing in the decoded data storage area 231. To do.

  Next, the user prepares lyrics data to be sung. That is, the user inputs a phonetic character string or a phonetic symbol string constituting the lyrics by operating the operation unit 230 while the lyrics data input application is activated. The control unit 210 generates lyric data using the input phonogram string and phonetic symbol string, and stores them in the RAM. When the lyrics data to be sung is stored in the lyrics data storage area 222, the lyrics data may be transferred from the lyrics data storage area 222 to the RAM by the operation of the operation unit 230.

  Next, the user operates the operation unit 230 to instruct the control unit 210 to start singing synthesis. This completes preparation for singing synthesis.

2-2. Speech Synthesis Operation Next, the speech synthesis operation performed by the speech synthesizer 200 will be described. When preparation for singing synthesis is completed as described above, the user performs a keyboard performance using the keyboard section 160 of the performance data generating apparatus 100 and operates the operation section 130 as necessary. In the performance data generating apparatus 100, operation events of the keyboard unit 160 and the operation unit 130 are delivered to the control unit 110. The control unit 110 generates a MIDI message corresponding to the operation event, and transmits the generated MIDI message to the speech synthesizer 200 via the MIDI interface in the interface unit 150.

  In the speech synthesizer 200, every time the MIDI interface in the interface unit 250 receives a MIDI message, an interrupt request to the control unit 210 is generated, and the MIDI message received by the MIDI interface is delivered to the MIDI message reception processing unit 212. It is. The MIDI message reception processing unit 212 interprets the MIDI message, and controls each unit of the song synthesis unit 213 according to the interpretation result.

  When a note-on message or a note-off message is delivered, the MIDI message reception processing unit 212 sends the message to the singing position update processing unit 214, the segment selection unit 215, the conversion processing unit 216, and the segment connection unit 219. To hand over. The singing position update processing unit 214 manages the singing position in the lyrics data to be sung in the RAM, and basically updates the singing position when the note-on message is delivered. That is, for example, when the lyrics data indicates the lyrics “Saisai ...”, the singing position update processing unit 214 sets the singing position to the first table of the lyrics when the first note-on message is delivered. When the second note-on message is delivered, the singing position is moved from “sa” to “yes”, and when the third note-on message is delivered, the singing position is changed to “yes”. ”To“ ta ”, and so on, the singing position is advanced by one phonogram. The singing position update processing unit 214 advances the singing position by one syllable (for example, Sep-tem-ber) in a language such as English.

  The segment selection unit 215 basically has one phonetic character (one in Japanese, one in English) at the song position specified by the song position update processing unit 214 in the phonetic character string indicated by the lyrics data. One or a plurality of speech units used for synthesizing one or a plurality of speech waveforms is determined, speech unit data corresponding to the speech unit is read from the speech unit database DB1, and delivered to the conversion processing unit 216. .

  When the note-on message is delivered from the MIDI message reception processing unit 212, the conversion processing unit 216 takes out a note number that specifies the pitch of the sound to be sounded from the note-on message. Then, pitch conversion to a pitch corresponding to the note number is performed on the speech segment data delivered from the segment selection unit 215 at a time corresponding to voiced sounds such as vowels and voiced consonants. . As described above, when the speech unit data includes the spectrum data of the sample sequence of the speech unit as the waveform data, this pitch conversion is indicated by the pitch indicated by the note number and the unit pitch data included in the speech unit data, for example. This can be executed by obtaining a difference from the pitch and moving the spectrum distribution indicated by the waveform data in the frequency direction by a frequency corresponding to the difference. The conversion processing unit 216 performs inverse FFT on the speech segment data that has undergone such pitch conversion, and outputs singing speech data that is a digital speech signal in the time domain.

  At that time, the conversion processing unit 216 adjusts the duration of the singing voice data. More specifically, in the present embodiment, basically, in response to the reception of the note-on message, the singing target portion is started to sound, and the reception of the note-off message paired with the note-on message or the note-on of the subsequent note is performed. The pronunciation of the singing target part is terminated with the reception of the message as a trigger.

  On the other hand, the conversion processing unit 216 receives a note-off message from the MIDI message reception processing unit 212 and delivers a note-on message of the next phoneme within a predetermined time T3 after the delivery of the note-off message. If not, the silence transition data of the phoneme corresponding to the note-off message is read from the decoded data storage area 231 and the speech synthesis process is performed using the read silence transition data. If the silence transition data of the phoneme corresponding to the note-off message received by the MIDI message reception processing unit 212 is not stored in the decoded data storage area 231, the conversion processing unit 216 displays the silence transition data of the phoneme. Are read out from the speech unit database DB1, and the read out speech unit data are subjected to decoding processing and speech synthesis processing.

  The segment linking unit 219 receives the singing voice data (time domain signal) that has undergone the processing of the conversion processing unit 216, and stores a digital voice signal obtained by linking the singing voice data on the time axis in a buffer area in the RAM. Then, an appropriate delay process is applied to the sound system 260.

  As described above, in this embodiment, in unit-connected speech synthesis, segment waveform data that is frequently used (particularly, speech unit data representing a speech waveform of a transition portion from vowels and consonants to silence). ) In advance (decoded encoded speech segment data is decoded and expanded in the memory), the delay from when the user performs a performance operation for mute to when the synthesized speech is muteed is reduced.

3. Modification The above embodiment can be modified as follows. In addition, you may implement the following modifications suitably combining.

3-1. Modification (1)
The form of the speech segment data is not limited to that shown in the above embodiment. For example, data indicating a spectrum envelope (spectrum envelope) for each frame F of each speech unit may be used as speech unit data, or data indicating a waveform on the time axis of each speech unit may be used as speech unit data. Good. Further, the waveform of a speech unit may be divided into a harmonic component (Deterministic Component) and an inharmonic component (Stochastic Component) by SMS (Spectral Modeling Synthesis) technology, and data indicating each component may be used as speech unit data. In this case, processing by the speech synthesis unit 218 is executed for both the harmonic component and the non-harmonic component, and the harmonic component and the non-harmonic component after this processing are added by the subsequent addition unit of the speech synthesis unit 218. Will be. Further, after dividing each speech unit into frames F, a plurality of feature quantities (for example, the frequency and gain of the peak of the spectrum envelope, or the overall inclination of the spectrum envelope) are extracted in advance. A set of a plurality of parameters representing these feature quantities may be used as speech segment data. Thus, it does not matter how the speech unit is held in the present invention.

3-2. Modification (2)
In the said embodiment, although the case where this invention was applied to the apparatus for synthesize | combining singing voice was illustrated, of course, this invention can be applied also to apparatuses other than this. For example, the present invention is applied to an apparatus that reads out a character string of a document based on document data (for example, a text file) indicating various documents. In other words, the unit acquisition unit 215 may read the speech unit data from the speech unit database DB1 based on the character code included in the text file, and the speech may be synthesized based on the speech unit data. In this case, the control unit of the apparatus connects a sound generation instruction indicating the sound generation start timing and sound generation end timing of each phoneme via an operation unit that outputs an operation signal according to an operation by the user, a communication cable, or the like. It is good also as a structure acquired from the other apparatus made.

The details of the speech synthesis process are not limited to those described in the embodiment. As long as a note and a phonetic symbol (character) are given, any processing may be used as long as it synthesizes a sound corresponding to the note and the phonetic symbol. Further, instead of a musical note, a process may be used in which data representing the arrangement of sounds such as a pitch curve is given, and a process of synthesizing a sound corresponding to the pitch curve.
In the above-described embodiment, a MIDI keyboard instrument having a keyboard is exemplified as the performance data generation apparatus 100. However, the performance data generation apparatus is not limited to a MIDI keyboard instrument, and other apparatuses such as a personal computer that outputs a MIDI message. It may be. Further, the message output by the performance data generating apparatus 100 is not limited to a MIDI message, and any apparatus can be used as long as it outputs a sound generation instruction indicating the sound generation start timing and sound generation end timing. Good.

3-3. Modification (3)
The hardware configuration of the speech synthesizer 200 is not limited to that described with reference to FIG. As long as the function shown in FIG. 4 can be implemented, the speech synthesizer 200 may have any hardware configuration. For example, the speech synthesizer 200 may have dedicated hardware (circuit) corresponding to each of the functional elements shown in FIG. For example, the conversion processing unit 216 illustrated in FIG. 4 may be realized by hardware dedicated to audio processing such as a DSP.

  In the above-described embodiment, a plurality of devices may share functions related to the speech synthesis device 200 of the above-described embodiment. For example, the first device that realizes the functions of the singing position update processing unit 214 and the segment selection unit 215 in FIG. 4 and the second device that realizes the functions of the conversion processing unit 216 and the segment connection unit 219 are different. It may be configured as a body. Further, two or more devices connected via a communication network share functions related to the speech synthesizer 200 of the above-described embodiment, and a system including these devices implements the speech synthesizer 200 of the same embodiment. You may do it.

  In the above-described embodiment, the configuration in which the speech synthesizer 200 stores the speech unit database DB1 has been described. However, the present invention is not limited to this, and for example, as shown in FIG. 200 may be configured as a separate body. In the example shown in FIG. 7, the performance data generation device 100, the speech synthesizer 200, and the speech segment database 400 are connected by a network 500.

  In the above-described embodiment, the configuration in which the units illustrated in FIG. 1 are connected by the bus 270 has been described. However, a configuration in which the units are connected by a communication network instead of the bus 270 may be employed. In the above-described embodiment, the configuration in which the performance data generation device 100 and the speech synthesis device 200 are connected by the communication cable 300 has been described. However, the present invention is not limited to this, and the performance data generation device 100 and the speech synthesis device 200 are connected. It may be configured as an integrated device.

3-4. Modification (4)
In the above-described embodiment, the user inputs the lyrics data in advance before performing the singing synthesis process. However, the timing at which the lyrics data is input is not limited to this, and the lyrics data is input in real time. May be. In other words, the lyrics data may be input in parallel with the MIDI message being input to the speech synthesizer 200. For example, data representing lyrics (phonemes) may be included in the MIDI message.

3-5. Modification (5)
In the above-described embodiment, when the singing synthesis application is activated, the initial setting unit 211 decodes speech segment data representing a speech waveform of a transition portion from a predetermined phoneme to silence, and stores decoded data. The area 231 is stored. The timing for decoding the silent transition data is not limited to the timing shown in the above embodiment. For example, the silence transition data may be decoded at a timing when a note-on message corresponding to a predetermined phoneme (vowel or nasal sound) is received (time T2-16 indicated by a dotted line in FIG. 5). . In this case, for example, if the phonetic character specified by the singing position updating unit 213 as the current singing position is a vowel, the unit selecting unit 215 utters the speech unit data corresponding to this vowel. In addition to reading from the unit database DB1, speech unit data representing the speech waveform of the transition from vowel to silence is read from the speech unit database DB1 and delivered to the conversion processing unit 216. Specifically, for example, when the singing position update unit 213 identifies the phonetic character “a”, the segment selection unit 215 obtains the speech unit data [a] from the speech unit database DB1. In addition to reading, the speech unit data [a- #] is read from the speech unit database DB 1 and supplied to the decoding unit 217 of the conversion processing unit 216.

  The timing for decoding the silence transition data may be the timing when the device is turned on or the time when the speech synthesis application is started, or a note-on message for a predetermined phoneme is received. It may be timing. To explain the predetermined phonemes, the predetermined phonemes are determined by the database used. More specifically, the predetermined phonemes are, for example, vowels and nasal sounds in the case of a Japanese database, and vowels and consonants in the case of an English database. In addition, the types of phonemes that are predetermined according to the database recording status and creation policy may increase or decrease. In short, any timing may be used as long as the decoding process of silence transition data is started before receiving a sound generation end instruction (note-off message) for a predetermined phoneme. By starting the silence transition data decoding process before the note-off message is received, the delay time from when the note-off message is received until the mute process is performed can be reduced.

  In the above-described embodiment, the initial setting unit 211 performs a decoding process when the singing voice synthesis application is activated and stores the decoded data in the decoded data storage area 231. However, the configuration is not limited thereto. For example, the decoded data may be stored in another device (for example, the speech unit database 400) connected to the speech synthesis device 200 via the network 500. In this case, the decoded data may be stored in advance in another device, and the control unit 210 of the speech synthesizer 200 may have a predetermined timing (for example, when the singing synthesis application is activated). A configuration may be adopted in which decoded data is generated by performing a decoding process, and the generated decoded data is transmitted to another device.

3-6. Modification (6)
In the above-described embodiment, the decoded data storage area 231 is configured to store the decoded silence transition data for a period until the singing voice synthesizing process ends. However, the storage area is opened. The timing at which this is performed is not limited to that shown in the above embodiment. For example, the silence transition data is decoded at a timing when a note-on message corresponding to a predetermined phoneme is received, and the decoded silence transition data is stored in the decoded data storage area 231. The storage area in which the decoded silence transition data is stored may be opened at the timing when the sound generation process ends.

3-7. Modification (7)
In the above-described embodiment, it may be possible for the user to select, using the operation unit 230, which phoneme for which silence transition data is to be decoded in advance. In this case, the initial setting unit 211 selects one or a plurality of phonemes according to information output from the operation unit 230 operated by the user, and generates decoded data by decoding the silence transition data corresponding to the selected phonemes. The generated decoded data may be stored in the decoded data storage area 231.

  In addition, the method of selecting the silence transition data to be decoded in advance is not limited to the above-described method. For example, the control unit 210 is configured to statistically calculate the frequency of silence after the phoneme for each phoneme in the speech synthesis process. A configuration may be adopted in which phonemes satisfying a predetermined frequency (high frequency) are identified, and silence transition data of the identified phonemes is decoded in advance. As another example, for example, the control unit 210 may select phonemes to be decoded in advance from a history of speech synthesis processing performed by the user in the past. As another example, for example, a configuration in which the history of speech synthesis processing of each device is stored in a server device connected to a plurality of devices via a communication network or the like, and the control unit 210 stores the history stored in the server device. A phoneme to be decoded in advance may be selected based on this.

  In the above-described embodiment, after the note-on message is received, the control unit 210 has corresponding decoded data (or creates decoded data), and the speech synthesis process is performed using the decoded silence transition data. The user may be notified that the low-latency support is available (as opposed to the next note-off) by turning on / flashing the notification lamp or flashing / vibrating the target key. . That is, when the control unit 210 receives (acquires) a note-on message and the decoded data corresponding to the received note-on message is stored in the decoded data storage area 231, the control unit 210 receives the note-on message. Information for performing some notification may be output. Specifically, for example, in the example shown in FIG. 5, the control unit 210 uses the performance data as control information for starting lighting of the notification lamp when a predetermined time T3 has elapsed from the timing T2-3. Output to the generator 100. Further, the control unit 210 outputs control information indicating that the lighting is ended to the performance data generating apparatus 100 at the timing when the user takes note off (timing T2-12 when the note off message is received). The control unit 110 of the performance data generating apparatus 100 performs notification processing (for example, control of lighting / flashing of the notification lamp, vibration on / off, etc.) according to the control information received from the speech synthesizer 200. Further, in this aspect, the user may be able to perform a cancel operation for this lamp lighting. In this case, when the user performs a cancel operation using the operation unit 130, the control unit 110 stops the notification process according to the information output from the operation unit 130 (for example, the notification lamp is turned off, the vibration is turned off, etc.). To do.

3-8. Modification (8)
In the above-described embodiment, the conversion processing unit 216 receives the note-off message from the MIDI message reception processing unit 212, and the note of the next phoneme within a predetermined time T3 after the note-off message is delivered. When the on message was not delivered, the silence transition data was read from the decoded data storage area 231 and the speech synthesis process was performed using the read silence transition data. Instead, when the conversion processing unit 126 receives a note-off message from the MIDI message reception processing unit 212, the conversion processing unit 126 converts the silence transition data into a decoded data storage area (without waiting for a predetermined time T3). The speech synthesis process may be performed using the decoded silence transition data read out from H.231. Specifically, for example, when there is a phrase delimiter such as “Today is” or “...” as lyrics data, it is determined whether or not a note-on message of the next phoneme is received. Alternatively, the speech synthesis process may be performed using the decoded silence transition data.

  In the above-described embodiment, the user may be allowed to set a predetermined time T3 using the operation unit 230 or the like. In this case, the control unit 210 sets the time T3 according to the signal output from the operation unit 230.

3-9. Modification (9)
In addition to the speech synthesizer, the present invention can be understood as a method for realizing these and a program for causing a computer to realize a speech synthesis function. Such a program may be provided in the form of a recording medium such as an optical disk storing the program, or may be provided in the form of being downloaded to a computer via the Internet or the like and installed and used.

DESCRIPTION OF SYMBOLS 100 ... Performance data generator 110 ... Control part 120 ... Memory | storage part 130 ... Operation part 140 ... Display part 150 ... Interface part 160 ... Keyboard part 170 ... Sound source part 180 ... Sound system 190 ... Bus , 200 ... voice synthesizer, 210 ... control unit, 211 ... initial setting unit, 212 ... MIDI message reception processing unit, 213 ... singing synthesis unit, 214 ... singing position update processing unit, 215 ... segment selection unit, 216 ... conversion Processing unit, 217 ... decoding unit, 218 ... speech synthesis unit, 219 ... unit connection unit, 220 ... storage unit, 221 ... speech unit database DB1 storage region, 222 ... lyric data storage region, 230 ... operation unit, 231 ... Decoded data storage area, 240 ... display section, 250 ... interface section, 260 ... sound system, 270 ... bus, 300 ... Shin cable, DB1 ... speech unit database

Claims (4)

A pronunciation instruction acquisition unit for acquiring a pronunciation instruction indicating a phoneme pronunciation start instruction or a pronunciation end instruction;
When the pronunciation instruction acquisition unit acquires the pronunciation instruction, a phoneme specifying unit that specifies a phoneme corresponding to the acquired pronunciation instruction;
A speech element corresponding to the phoneme specified by the phoneme specifying unit is obtained from a speech unit database which is a set of speech unit data in which data representing speech waveforms for each phoneme and each phoneme chain is encoded according to a predetermined rule. A speech segment data acquisition unit for acquiring fragment data;
A decoding unit that decodes the speech unit data acquired by the speech unit data acquisition unit;
When the synthesized speech data representing the synthesized speech is generated using the speech unit data decoded by the decoding unit, the generated synthesized speech data is output, and the pronunciation instruction acquisition unit acquires the pronunciation termination instruction From a storage unit for storing decoded data obtained by decoding speech unit data registered in the speech unit database and decoded from speech unit data representing a speech waveform of a transition portion from a predetermined phoneme to silence. , Read out the decoded data corresponding to the speech waveform of the transition part from the phoneme specified by the phoneme specifying unit as a phoneme corresponding to the acquired pronunciation end instruction, and use the decoded data A voice synthesizer that generates data and outputs the generated synthesized voice data ;
At the timing before the sound generation instruction acquisition unit acquires the sound generation end instruction, the speech unit data representing the sound waveform of the transition from the phoneme to the silence satisfying the predetermined condition is decoded to generate decoded data And a decoded data storage control unit that stores the generated decoded data in the storage unit. Before SL decoded data storage control unit, transition portion of the if sound instruction acquisition unit acquires the sounding start instruction, to the silence phoneme identified by the phoneme identification part as a sound element corresponding to the emitting sound start instruction 2. The speech synthesizer according to claim 1 , wherein the speech unit data representing the speech waveform is decoded to generate decoded data, and the generated decoded data is stored in the storage unit. The decoded data storage control unit selects one or a plurality of phonemes based on the processing history in the speech synthesis unit, and decodes speech unit data representing a speech waveform of a transition portion from the selected phonemes to silence. 3. The speech synthesizer according to claim 1 , wherein decoded data is generated and the generated decoded data is stored in the storage unit. Before Symbol speech synthesis unit, the sound instruction acquisition unit acquires the sounding start instruction, and, if the decoded data corresponding to the phoneme corresponding to the acquired sounding start instruction is stored in the storage unit In addition, the voice synthesizer according to any one of claims 1 to 3 , wherein information for notifying a user is output.

Images