JP5810947B2 - Speech segment specifying device, speech parameter generating device, and program - Google Patents

Description

  The present invention relates to an utterance interval specifying device and program for specifying an utterance interval in a speech waveform, and an audio parameter generation device for generating an audio parameter from audio data in the specified utterance interval.

  2. Description of the Related Art Conventionally, there has been known a speech synthesizer that includes a speech parameter generation device that prepares speech parameters from input sound and generates speech of specified content by synthesizing speech parameters prepared by the speech parameter generation device. (For example, refer to Patent Document 1).

  The speech parameter generation device in the speech synthesizer described in Patent Document 1 includes a sound separation unit that separates an input sound (hereinafter referred to as speech data) into a harmonic component and a non-harmonic component, and a sound separation. On the basis of the harmonic components separated in the section, the speech data is segmented into a phoneme segmentation section and a phoneme segmentation section that are segmented into sections estimated to be uttered for each phoneme (hereinafter referred to as utterance sections). And a parameter generation unit that generates a speech parameter from speech data in each utterance section.

  And in the method of dividing the speech data in the phoneme segmentation part described in Patent Document 1 for each utterance section, the waveform of the speech data is displayed, and then the displayed waveform is used by the user of the speech parameter generation device. , The utterance section is specified by designating the utterance start time and utterance end time of each phoneme by a switch operation while visually confirming.

JP 2004-038071 A

  In the method of separating each utterance section from the speech data described in Patent Document 1, the start (speech start time) and the end (speech end time) of each utterance section are defined by the user of the speech parameter generation device. Has to be specified while visually confirming, and there has been a problem that the accuracy when the voice data is divided for each utterance section is low.

Furthermore, in the method described in Patent Document 1, there is a problem that it is difficult to separate each voice data for each utterance section of a large amount of voice data as long as it relies on human hands.
In view of this, an object of the present invention is to make it possible to separate each piece of voice data for each voice section with respect to a large amount of voice data.

  In the utterance section specifying device of the present invention made to achieve the above object, the content information acquisition means acquires the utterance content information, the timing information acquisition means acquires the utterance timing information, and the score data acquisition means The music score data is acquired, and the audio data acquisition means acquires the audio data. In the present invention, the utterance content information is information representing a character string of the content to be uttered in the target music that is one music, and the utterance timing information is the utterance content information ( Hereinafter, the information specifies the utterance start timing of the character represented by the specific content information. Furthermore, in the present invention, the music score data represents at least the music score of the singing melody in the target music, and for each output sound constituting the singing melody, at least the pitch and performance start timing are defined, Voice data is data representing a voice waveform uttered for a character string represented by specific content information.

  And in the utterance section specifying device of the present invention, the power transition deriving unit is based on the audio data acquired by the audio data acquiring unit, for each unit time defined to be continuous along the time axis in the audio data. A voice power transition representing a time transition of the calculated power is derived. Then, the utterance section specifying means specifies the time when the voice power transition changes along the time axis as the utterance start time and the utterance end time, and the utterance start time continuous in the order of the utterance start time and the utterance end time; Each utterance section specified by the pair with the utterance end time is specified.

Further, in the utterance section specifying device of the present invention, the musical note lyrics associating means includes the specific content information acquired by the content information acquiring means, the utterance timing information acquired by the timing information acquiring means, and the music score acquired by the score data acquiring means. Based on the data, for each syllable of the character string represented by the specific content information, the output sound having the performance start timing that minimizes the time difference from the utterance start timing of the character corresponding to the syllable is specified, Each note syllable group is generated by associating the output sound with the contents of the syllable. After that, the note singing voice integration means has a performance start timing that minimizes the time difference from the utterance start time that defines the utterance section for each utterance section specified by the utterance section specifying means, and is compatible with note lyrics The note syllable set generated by the appending means is specified, and at least syllable data in which the utterance section and the note syllable set are associated with each other is generated.
Then, the utterance section specifying means in the present invention sets the timing at which the power is equal to or higher than a predetermined threshold value in the time progression of the voice power transition as the utterance start time and ends the timing at which the power is equal to or lower than the predetermined threshold value. Each utterance section may be specified as time.
Further, the utterance section specifying means in the present invention may identify each utterance section with the maximal timing as the utterance start time and the maximal timing as the utterance end time as a result of time differentiation of the voice power transition. good.

  According to such an utterance section specifying device, the utterance start time and the utterance end time, and thus the utterance section are automatically determined based on the timing at which the voice power changes while the uttered voice waveform changes along the time axis. Can be identified.

  As a result, according to the utterance section specifying device of the present invention, unlike the device described in Patent Document 1, it is not necessary for the user of the device to specify the utterance start time and utterance end time, and a large amount of audio data For each of the voice data, the voice data can be divided for each utterance section.

Moreover, in the utterance section specifying device of the present invention, syllable data is generated by associating each specified utterance section with a note syllable group corresponding to the utterance section.
Therefore, according to the utterance section specifying device of the present invention, when generating a speech parameter from speech data in the utterance section included in the syllable data, the content of the highly reliable syllable is added to the speech parameter. As a result, various information can be added to the speech parameters necessary for speech synthesis. As a result, according to the utterance section specifying device of the present invention, it is possible to easily realize a voice desired by a person using the voice synthesis at the time of voice synthesis.

  In the utterance section specifying device of the present invention, the pitch transition deriving means is a pitch time in which the pitch in the voice waveform has shifted along the time axis based on at least the voice data acquired by the voice data acquiring means. The transition may be derived, and the pitch specifying means may specify the pitch in each utterance section specified by the utterance section specifying means as the utterance pitch in the pitch time transition.

In this case, the note singing voice integration means in the present invention associates each utterance pitch specified by the pitch specification means, the utterance interval corresponding to the utterance pitch, and the note syllable group corresponding to the utterance interval. in, but it may also be generating a syllable data.

  According to such an utterance section specifying device, the pitch of each utterance section can be added to each syllable data. As a result, when generating the speech parameter from the speech data in the utterance section included in the syllable data, it is possible to add information on the pitch with high reliability in the utterance section to the speech parameter.

Furthermore, the pitch specifying means in the present invention is based on the autocorrelation value of the frequency spectrum calculated for each unit time defined to be continuous along the time axis in the audio data, and the output sound represented by the music score data. Of these, the pitch corresponding to the frequency that maximizes the result of multiplying the weight function that is defined along the frequency axis so that the frequency corresponding to the pitch of the output sound corresponding to the unit time has a greater weight. the, but it may also be identified as utterance pitch.

According to such an utterance section specifying device, the utterance pitch can be specified as a pitch close to the pitch to be uttered in the target music piece.
And the pitch specifying means in the present invention, the pitch corresponding to the frequency at which the autocorrelation value of the frequency spectrum calculated for each unit time defined to be continuous along the time axis in the voice data is maximum, It is identified as utterance pitch not good.

According to such an utterance section specifying device, the pitch of the actually uttered voice can be specified as the utterance pitch.
By the way, the note singing voice integration means in the present invention generates syllable data by associating the power at each time in the sound power transition, the utterance section corresponding to the time, and the note syllable group corresponding to the utterance section. and even good.

  According to such an utterance section specifying device, power in each utterance section can be added to each syllable data. As a result, when generating a speech parameter from speech data in the utterance section included in the syllable data, it is possible to add information on the power (strength) of the speech with high reliability in the utterance section to the speech parameter. It becomes.

In time and, application of the present invention may be a voice parameter generating device. However, the speech parameter generation device to which the present invention is applied needs to include the utterance section identification device described in claim 1 and parameter derivation means. The parameter deriving means here refers to deriving a speech parameter that is at least one feature quantity defined in advance from speech data in the utterance section in the syllable data generated by the note singing voice integration means in the utterance section specifying device. means der to Apply predicates.

  According to such a speech parameter generation device, when generating speech parameters from speech data in the utterance section included in the syllable data, it is possible to add highly reliable syllable content to the speech parameters, As a result, various information can be added to the speech parameters required for speech synthesis. As a result, according to the utterance section specifying device of the present invention, it is possible to easily realize the voice desired by the person using the voice synthesis at the time of voice synthesis.

  Note that the feature amount as the speech parameter mentioned here is a feature amount necessary for executing speech synthesis by formant synthesis. For example, the fundamental frequency, the mel frequency cepstrum (MFCC), the power, etc. Each time difference is included.

Furthermore, the present invention may be implemented as a program that is executed by a computer. In this case, the program to which the present invention is applied includes a content information acquisition procedure for acquiring utterance content information, a timing information acquisition procedure for acquiring utterance timing information, a score data acquisition procedure for acquiring music score data, and audio data. Voice data acquisition procedure for acquiring voice power transition, power transition derivation procedure for deriving voice power transition, utterance interval specifying procedure for identifying each utterance segment, note lyrics corresponding procedure for generating each note syllable group, and syllable data It must be programmed to execute the note voice integration procedure resulting in the computer Ru.
In the utterance section specifying procedure, in the time progression of the voice power transition, each timing when the power is equal to or higher than a predetermined threshold value is set as the utterance start time, and each timing when the power is equal to or lower than the specified threshold value is set as the utterance end time. Each utterance section may be specified, or each time when the speech power transition is differentiated as a result of time differentiation, each maximal timing is set as the utterance start time, and each minimum timing is set as the utterance end time. good.

The present invention may be a program for causing a computer to function as an utterance section specifying device.
If the program of the present invention is made in this way, for example, it can be recorded on a computer-readable recording medium such as a DVD-ROM, CD-ROM, hard disk, etc. If necessary, it can be used by being acquired and activated by a computer via a communication line. And by making a computer perform each procedure, the computer can be functioned as an utterance area identification device described in claim 1.

It is a block diagram which shows schematic structure of a speech synthesis system. It is a flowchart which shows the process sequence of an audio | voice parameter registration process. It is a flowchart which shows the process sequence of an utterance area estimation process. It is a figure which illustrates audio | voice data. It is a figure which illustrates the derivation method of a pitch time change. It is a figure which illustrates pitch time change. It is a figure which illustrates the derivation method of a power time change. It is a figure which illustrates the identification method of an utterance area. It is a figure which shows the modification of the identification method of an utterance area. It is a figure which shows the process sequence of a speech synthesis process.

Embodiments of the present invention will be described below with reference to the drawings.
<About the speech synthesis system>
FIG. 1 is a diagram showing a schematic configuration of a speech synthesis system to which the present invention is applied.

  The speech synthesis system 1 to which the present invention has been applied is a speech synthesized based on speech parameters registered in advance (that is, synthesis) so that speech having contents designated by the user of the speech synthesis system 1 is output. Sound).

  In order to realize this, the speech synthesis system 1 is used for speech input device 10 for inputting speech, speech input through speech input device 10 (hereinafter referred to as speech waveform data SV), and karaoke. And a MIDI storage server 25 for storing various data (hereinafter referred to as music data MD). Furthermore, the speech synthesis system 1 includes an information processing device 30 that executes processing for generating speech parameters based on the speech waveform data SV and the music data MD stored in the MIDI storage server 25, and the information processing device 30. And a data storage server 50 for storing the generated voice parameters. In addition, the speech synthesis system 1 includes a speech output terminal 60 that outputs a synthesized sound synthesized by speech based on speech parameters stored in the data storage server 50. Note that the speech synthesis system 1 in this embodiment includes a plurality of speech output terminals 60.

  That is, in the speech synthesis system 1 of the present embodiment, the information processing apparatus 30 generates at least the speech parameter PM based on the speech waveform data SV and the music data MD stored in the MIDI storage server 25 and stores the data. Store in the server 50. Then, the voice output terminal 60 performs synthesis by voice synthesis based on the voice parameter PM stored in the data storage server 50 so that the voice having the content specified by the user is output via the voice output terminal 60. Output sound.

The speech parameter PM referred to here is a feature amount of speech used for so-called formant synthesis, which will be described in detail later. For example, the fundamental frequency, mel frequency cepstrum (MFCC), power, And the time difference between them.
<About the MIDI storage server>
First, the MIDI storage server 25 is a device mainly composed of a storage device configured to be able to read and write stored contents, and is connected to the voice input device 10 via a communication network.

  The MIDI storage server 25 stores at least music data MD prepared in advance for each piece of music. The music data MD includes music MIDI data DM (corresponding to music score data in the claims) and a lyric data group DL. The music MIDI data DM and the lyric data group DL correspond to each other. Each song is associated with each other.

  Of these, the music MIDI data DM is data representing the score of one music according to the well-known MIDI (Musical Instrument Digital Interface) standard, and is prepared in advance for each music. Each of the music MIDI data DM includes at least identification data that is data for discriminating music and a score track that represents a score for each instrument used in the music.

  The musical score track defines at least a pitch (so-called note number) and a period during which the sound module outputs the output sound (hereinafter, note length) for each output sound output from the MIDI sound source. Yes. However, the note length of the musical score track is the performance start timing (so-called note-on timing) indicating the time from the start of the performance of the music until the output of the output sound is started, and the output of the output sound is ended. It is defined by the performance end timing (so-called note-off timing) that represents the time from the start of performance of the music.

  Note that the score track includes, for example, a keyboard instrument (eg, piano or pipe organ), a stringed instrument (eg, violin, viola, guitar, bass guitar, koto), or a percussion instrument (eg, vibraphone, drum, cymbal, timpani, Xylophone, etc.) and wind instruments (eg, clarinet, trumpet, flute, shakuhachi, etc.). Among these, in this embodiment, the vibraphone is defined as an instrument in charge of singing melody (melody line) in the music. In the following, the output sound defined in the score track corresponding to the musical instrument in charge of singing melody will be referred to as singing output sound.

  On the other hand, the lyrics data group DL is data relating to lyrics displayed on a display device that constitutes a well-known karaoke device, and lyrics telop data DT ( Lyric output data DO (corresponding to the utterance content information in the present invention) and lyric output data DO in which the timing correspondence relationship for associating the lyric output timing, which is the output timing of the lyrics constituent characters, with the performance of the music MIDI data DM is defined. Equivalent).

Specifically, the timing correspondence relationship in the present embodiment is such that the timing of starting the output of the lyrics telop data DT is associated with the timing of starting the performance of the music MIDI data DM, and then along the time axis of the music. The lyrics output timing of each lyrics constituent character is defined by the elapsed time from the start of the performance of the music MIDI data DM. Note that the elapsed time referred to here is, for example, a time indicating the timing for executing color change of displayed lyrics constituent characters, and is defined by the speed of color change. Further, the lyric constituent characters referred to here may be each of the characters constituting the lyric, or may be a phrase or a phrase grouped according to a specific rule along the time axis.
<About the configuration of the voice input device>
Next, the voice input device 10 will be described.

  The voice input device 10 includes a communication unit 11, an input receiving unit 12, a display unit 13, a voice input unit 14, a voice output unit 15, a sound source module 16, a storage unit 17, and a control unit 20. ing. That is, the voice input device 10 in the present embodiment is configured as a so-called well-known karaoke device.

  Among these, the communication unit 11 communicates with the outside through the communication network (for example, a public wireless communication network or a network line). The input reception unit 12 is an input device (for example, a key, a switch, a remote control reception unit, or the like) that receives input of information and commands in accordance with external operations.

  The display unit 13 is a display device (for example, a liquid crystal display or a CRT) that displays an image including at least information indicated by a character code. The voice input unit 14 is a device (so-called microphone) that converts sound into an electrical signal and inputs the signal to the control unit 20. The audio output unit 15 is a device (so-called speaker) that converts an electrical signal from the control unit 20 into sound and outputs the sound. Furthermore, the sound module 16 is a device (for example, a MIDI sound source) that outputs a sound (that is, an output sound) that simulates a sound from a sound source, based on data defined by the MIDI (Musical Instrument Digital Interface) standard. .

The storage unit 17 is a non-volatile storage device (for example, a hard disk device or a flash memory) configured to be able to read and write stored contents.
The control unit 20 is stored in the ROM 21 that stores processing programs and data that need to retain stored contents even when the power is turned off, the RAM 22 that temporarily stores processing programs and data, and the ROM 21 and RAM 22. It is mainly configured by a known computer having at least a CPU 23 that executes each process (various operations) according to the processing program.

  Then, the ROM 21 stores a processing program for executing a well-known karaoke performance process by the control unit and a voice input via the voice input unit 14 during a period when one piece of music is being played by the karaoke performance process. As the waveform data SV, a processing program in which the control unit 20 executes an audio storage process stored in the MIDI storage server 25 in association with music identification information for identifying the target music is stored.

  That is, the voice input device 10 acquires music data MD corresponding to one piece of music (hereinafter referred to as a target song) designated via the input receiving unit 12 from the MIDI storage server 25 according to the karaoke performance process, The target music is played based on the music MIDI data DM in the music data MD, and the lyrics of the target music are displayed on the display unit 13 based on the lyrics data group DL in the music data MD.

Furthermore, in the voice input device 10, the voice waveform data SV is uttered to identify music identification information (here, the music data MD itself) for identifying the target music and a person who has input the voice (hereinafter referred to as a speaker). Stored in the MIDI storage server 25 in association with the person identification information (hereinafter referred to as the speaker ID). The voice waveform data SV stored in the MIDI storage server 25 is also associated with speaker feature information representing the features of the speaker. The speaker feature information includes, for example, the gender and age of the speaker. Etc.
<Configuration of information processing device>
Next, the information processing apparatus 30 will be described.

The information processing apparatus 30 includes a communication unit 31, an input reception unit 32, a display unit 33, a storage unit 34, and a control unit 40.
Among these, the communication unit 31 communicates with the outside via a communication network (for example, a public wireless communication network or a network line). The input receiving unit 32 is an input device (for example, a keyboard or a pointing device) that receives input of information and commands in accordance with an external operation. The display unit 33 is a display device (for example, a liquid crystal display or a CRT) that displays an image.

  The storage unit 34 is a non-volatile storage device (for example, a hard disk device or a flash memory) configured to be able to read and write stored contents. The control unit 40 is configured around a known computer having at least a ROM 41, a RAM 42, and a CPU 43.

  Then, in the ROM 41 of the information processing apparatus 30, a voice parameter registration process for storing the voice parameter PM generated based on the voice waveform data SV and the music data MD stored in the MIDI storage server 25 in the data storage server 50 is performed. A processing program to be executed by the control unit 40 is stored.

The data storage server 50 is a device that is mainly configured of a storage device that is configured to be able to read and write stored contents, and is connected to the information processing device 30 via a communication network.
<Voice parameter registration process>
Next, the voice parameter registration process executed by the information processing apparatus 30 will be described.

  As shown in FIG. 2, when the voice parameter registration process is started, the music MIDI data DM of the music specified via the input receiving unit 32 (that is, the target music) is acquired (S110). Subsequently, the lyrics data group DL of the target music is acquired (S120), and one speech waveform data SV corresponding to the target music and corresponding to the speaker ID designated via the input receiving unit 32 is acquired (S120). S130).

  Furthermore, in the speech waveform data SV acquired in S130, a section estimated to be uttered corresponding to each syllable included in the utterance content of the speech waveform data SV (hereinafter referred to as utterance section) is specified, and each utterance is identified. An utterance section estimation process for generating syllable data in which various information is associated with the section is executed (S140).

  Then, the speech parameter PM is derived from the speech waveform (hereinafter referred to as a syllable waveform) in the utterance section represented by each syllable data (S150). In S150 of this embodiment, the fundamental frequency, the mel frequency cepstrum (MFCC), the power, and their time differences are each derived as a speech parameter PM. Since these fundamental frequency, MFCC, and power derivation methods are well known, detailed description thereof is omitted here. For example, if the fundamental frequency is used, autocorrelation along the time axis of the syllable waveform, syllable waveform What is necessary is just to derive | lead-out using methods, such as an autocorrelation of a frequency spectrum, or a cepstrum method. In addition, in the case of MFCC, a result obtained by applying a time analysis window to a syllable waveform and performing frequency analysis (for example, FFT) for each time analysis window is further obtained by logarithmizing the size for each frequency. It can be derived by frequency analysis. The power may be derived by applying a time analysis window to the syllable waveform and integrating the square of the amplitude in the time direction.

  Subsequently, voice parameter registration for storing the voice parameter PM derived in S150 is executed (S160). Note that the speech parameter PM stored in the data storage server 50 in S160 of the present embodiment is associated with the content (type) of the uttered syllable, the speaker ID, and speaker characteristic information.

Thereafter, the voice parameter registration process is terminated.
<About the utterance interval estimation process>
Next, the speech segment estimation process activated in S140 in the speech parameter registration process will be described.

  As shown in FIG. 3, when the utterance interval estimation process is started, the pitch time change in which the pitch in the speech waveform has shifted along the time axis is based on the speech waveform data SV acquired in S <b> 130. Is calculated (S210).

  Specifically, in S210 of the present embodiment, the speech waveform x (n) represented by the speech waveform data SV is shifted from the speech waveform data SV as shown in FIG. 4 by a time window having a certain time width LW. Then, the waveform segment xw (n) is cut out (see the following formula (1)). Here, x is a discrete signal sampled at the sampling frequency FS, and n is an index representing time. Further, the symbol “si” in the equation (1) is an index indicating the first position of the time window, and changes at a constant interval (for example, 50% of LW).

  Each of the cut out waveform segments xw (n) is subjected to DFT (Discrete Fourier Transform) to derive a frequency spectrum X (k) as shown in the following equation (2). As shown in FIG. 5A, the frequency spectrum X (k) shows the frequency included in the unit time on the horizontal axis and the level (amplitude) of each frequency on the vertical axis. However, k in the equation (2) is a value from 0 to “LW−1”.

  Further, for the frequency spectrum X (k), an autocorrelation function CORX (p) along the frequency axis is calculated according to the following equation (3). As shown in FIG. 5B, the autocorrelation function CORX (p) shows the amount of frequency deviation on the horizontal axis and the correlation value corresponding to the frequency deviation on the vertical axis. However, the sign p in the equation (3) is a frequency index shift, and Abs is a function that takes an absolute value of a complex number.

  Subsequently, the autocorrelation function CORX (p) calculated in this way is multiplied by a weight function wf (p) defined in advance. This weighting function wf (p) is expressed by the following equation (4). As shown in FIG. 5C, the pitch of the singing output sound corresponding to each time window (hereinafter referred to as model sound). The weight is defined along the frequency axis so that the frequency corresponding to “high” becomes a larger weight. However, the sign “σ” in the equation (4) is a standard deviation indicating a pitch distribution with respect to the model pitch.

  The result of multiplying the autocorrelation function CORX (p) by this weighting function wf (p) (hereinafter referred to as the final calculation result), as shown in FIG. 5D, the frequency index shift that maximizes the final calculation result. p is specified, and the pitch in the time window is derived based on the specified frequency index shift p. Specifically, the pitch in the time window is derived by pitch = Fs · p / LW.

  In S210 of this embodiment, this series of processing is executed while sliding the time window with respect to the entire time of the audio waveform data SV, and the derived pitches are arranged along the time axis. The pitch time change as shown in FIG. 6 is calculated.

  However, in S210 of the present embodiment, it may be determined that the singing voice is not included when there is no protruding peak in the final calculation result. In this case, the condition for determining that the singing voice is not included may be that “the peak level in the final calculation result / the average level in the final calculation result” is equal to or less than a predetermined threshold value.

  In the utterance interval estimation process, subsequently, for each time window, the pitch obtained in S210 and the frequency of its harmonic component (hereinafter referred to as singing voice component) are extracted, and a speech waveform (hereinafter referred to as the waveform) composed of the extracted frequency is extracted. , Referred to as a singing voice waveform) (S220). Specifically, in S220 of the present embodiment, as a technique for extracting the frequency of the singing voice component, a known technique using a comb filter as shown in FIG. 7 is used for the voice waveform data SV. In S220 of the present embodiment, it is not necessary to extract the pitch and the frequency of the harmonic component for the time window determined in S210 that the singing voice is not included.

Moreover, in S220, you may extract the frequency of a singing voice component from the result of FFT with respect to audio | voice waveform data SV.
Then, a power time transition (hereinafter referred to as voice power transition) in the singing voice waveform extracted in S220 is derived (S230). Specifically, in S230 of the present embodiment, power is calculated for each time window i defined to be continuous along the time axis with respect to the singing voice waveform, and the calculated power is arranged along the time axis. Thus, a voice power transition as shown in FIG.

  When the frequency of the singing voice component is extracted by a comb filter, this power is calculated by accumulating the square value of the amplitude of the singing voice waveform in the time window i in the time axis direction. Deriving the power at i. On the other hand, when the frequency of the singing voice component is extracted by FFT, the method for calculating the power is to calculate the squared amplitude value of the frequency component of the singing voice component extracted by FFT, the cumulative value in the frequency direction, Derived as power in time window i.

  Here, the following expression (5) is an expression for calculating the square value sum (ie, power) pw of each amplitude value of the frequency component of the singing voice component. However, the symbol m in the equation (5) is an index indicating what number of harmonics, and p0 is an index indicating the pitch obtained in S210.

  Subsequently, based on the voice power time transition derived in S230, each utterance section in the voice waveform data SV acquired in the previous S130 is specified (S240). Specifically, in S240 of the present embodiment, the utterance start time vs and the utterance end time ve are specified from the time when the sound power transition changes along the time axis, and the utterance start time vs and the utterance end time ve are in this order. Each section defined by a pair of the continuous utterance start time vs and the utterance end time ve is specified as the utterance section.

  In the present embodiment, as shown in FIG. 8B, the method for specifying the utterance start time vs and the utterance end time ve is that the power pw is equal to or greater than a predetermined threshold value as time progresses in the transition of the voice power. Each timing may be specified as the utterance start time vs, and each timing at which the power pw is equal to or less than a specified threshold may be specified as the utterance end time ve.

  Further, the method for specifying the utterance start time vs and the utterance end time ve is to derive the time derivative dpw (i) of the sound power transition according to the following equation (6), and as shown in FIG. Each of i) may be specified as the utterance start time vs at each of the maximum times, and each of the minimum times as the utterance end time ve.

  In the utterance interval estimation process, a well-known morphological analysis is performed on the lyrics represented by the lyrics telop data DT in the lyrics data group DL acquired in the previous S120, and further, referring to a dictionary prepared in advance. Then, the result of morphological analysis is converted into reading (S250). That is, in S250 of the present embodiment, the lyrics of the target music are converted to be expressed in syllable (phoneme) units, and the contents of each syllable are specified.

  Then, for each syllable of the lyrics converted in S250, a note syllable group in which the output sound corresponding to the syllable is associated with the content of the syllable is generated (S260). In S260 of the present embodiment, specifically, the lyric output of the syllable represented by the lyric output data DO, which is the performance start timing of the output sound constituting the singing melody in the music MIDI data acquired in the previous S110. The performance start timing that minimizes the time difference from the timing is specified, and the output sound having the specified performance start timing is associated with the content of the syllable.

  Subsequently, the time gap between the speech waveform data SV acquired at the previous S130 and the music MIDI data acquired at the previous S110 is corrected, and each utterance section in the speech waveform data SV corresponds to the utterance section. Syllable data is generated by associating each set of note syllables to be performed (S270).

  Specifically, in S270 of the present embodiment, a deviation between the first lyrics output timing along the time axis of the lyrics represented by the lyrics output data DO and the first utterance section along the time axis in the speech waveform data SV, The average value of the deviation between the last lyrics output timing along the time axis of the lyrics expressed by the lyrics output data DO and the last utterance section along the time axis in the speech waveform data SV is calculated. Then, for each utterance section, a note syllable group having a performance start timing that minimizes the time difference from the utterance start time vs. the calculated average value is specified, and at least the utterance section and the note syllable group Is generated.

  Furthermore, in the pitch time transition calculated in S210, the pitch in the section corresponding to each utterance section is specified as the utterance pitch, and each specified utterance pitch and the utterance corresponding to the utterance pitch are specified. Corresponding to the syllable data having a section (S280). The utterance pitch associated in S280 may be a note number in the MIDI standard or a musical scale.

  Then, the power at each time in the sound power transition derived in S230 is associated with the syllable data having the utterance section corresponding to the time (S290). The power associated in S290 may be a velocity in the MIDI standard, or a dynamic symbol (for example, a piano or a forte) described in a musical score.

Thereafter, the utterance section estimation process is terminated, and the process proceeds to S150 of the voice parameter registration process.
<Configuration of audio output terminal>
Next, the audio output terminal will be described (see FIG. 1).

  The voice output terminal 60 includes an information receiving unit 61, a display unit 62, a sound output unit 63, a communication unit 64, a storage unit 65, and a control unit 67. As the audio output terminal 60 in the present embodiment, for example, a known portable terminal (a mobile phone or a portable information terminal) or a known information processing apparatus (a so-called personal computer) may be assumed.

  Among these, the information reception part 61 receives the information input via the input device (not shown). The display unit 62 displays an image based on a command from the control unit 67. The sound output unit 63 is a known device that outputs sound, and includes, for example, a PCM sound source and a speaker.

  The communication unit 64 is for the voice output terminal 60 to perform information communication with the outside via a communication network (for example, a public wireless communication network or a network line). The storage unit 65 is a non-volatile storage device (for example, a hard disk device or a flash memory) configured to be able to read and write stored contents, and stores various processing programs and various data.

The control unit 67 is mainly configured by a known computer having at least a ROM, a RAM, and a CPU.
<About voice synthesis processing>
Next, speech synthesis processing executed by the control unit 67 of the speech output terminal 60 will be described.

This voice synthesis process is started when a start command is input via the information receiving unit 61 of the voice output terminal 60.
As shown in FIG. 10, when the speech synthesis process is started, first, information input via the information receiving unit 61 (hereinafter referred to as input information) is acquired (S510). The input information acquired in S510 includes, for example, output text indicating the content (word) of the sound output as synthesized sound, and output property information indicating the nature of the sound output as synthesized sound. Note that the sound property (that is, output property information) mentioned here includes characteristics of the voice of the speaker such as the gender of the speaker and the age of the speaker.

  Subsequently, the speech parameter PM corresponding to each syllable necessary to output the output word acquired in S510 as a synthesized sound and associated with the information most similar to the output property information acquired in S510, Extracted from the data storage server 50 (S520).

  Then, the speech parameter PM acquired in S520 is set so that the synthesized sound is output with the content of the output word acquired in S510 (S530). Subsequently, speech synthesis is performed based on the speech parameter PM set in S530 (S540). For the speech synthesis in S540, a well-known speech synthesis method using formant synthesis other than Patent Document 1 may be used.

Furthermore, the synthesized sound generated by the voice synthesis in S540 is output from the sound output unit 63 (S550).
Thereafter, the speech synthesis process ends.
[Effect of the embodiment]
As described above, according to the utterance interval estimation process of the present embodiment, the utterance start time and the utterance end are based on the timing at which the voice power changes while the uttered voice waveform changes along the time axis. It is possible to automatically specify the time and thus the utterance section.

  More specifically, according to the utterance interval estimation process of the present embodiment, each time when the power becomes greater than or equal to a predefined threshold value in the time progression of the voice power transition is set as the utterance start time, and the power is equal to or less than the defined threshold value. The utterance section can be automatically specified by setting each of the following timings as the utterance end time. Further, according to the utterance interval estimation process of the present embodiment, as a result of time differentiation of the voice power transition, each of the maximal timings is set as the utterance start time, and each of the minimum timings is set as the utterance end time. Can be identified automatically.

  As a result, according to the information processing apparatus of this embodiment, unlike the apparatus described in Patent Document 1, there is no need for the user of the apparatus to specify the utterance start time and utterance end time, and a large amount of speech waveforms With respect to data, it is possible to segment each speech waveform data for each utterance section.

  Moreover, in the utterance interval estimation process of the present embodiment, syllable data is generated by associating each specified utterance interval with a note syllable group corresponding to the utterance interval. Furthermore, in the utterance interval estimation process of the present embodiment, the pitch and power in each utterance interval included in each syllable data are added to each syllable data.

  As a result, in the speech parameter registration processing of the present embodiment, when generating speech parameters from speech waveform data in the utterance interval included in the syllable data, the utterance content with high reliability in the utterance interval, the pitch Information such as power can be added to the audio parameters.

  In other words, according to the speech parameter registration processing of the present embodiment, various information can be added to the speech parameters necessary for speech synthesis. As a result, when executing speech synthesis processing at the speech output terminal, The voice desired by the person using the voice synthesis can be easily realized.

In the utterance interval estimation process of the present embodiment, the pitch corresponding to the frequency that maximizes the result of multiplying the autocorrelation value of the frequency spectrum calculated per unit time by the weight function is specified as the utterance pitch. Yes. For this reason, according to the utterance section estimation process of this embodiment, the utterance pitch can be specified as a pitch close to the pitch to be uttered in the target music.
[Other Embodiments]
As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, In the range which does not deviate from the summary of this invention, it is possible to implement in various aspects.

  For example, in the above embodiment, the weighting function wf is defined by equation (4). However, the weighting function wf is not limited to this, and the user of the voice input device 10 can perform the singing melody of the target song. Thus, the weight function wf may be set in consideration of the case where the pitch is sung so that the pitch becomes 1 octave higher, or the case where the pitch is sung so that the pitch becomes 1 octave lower.

  In this case, the weighting function wf (p) may be defined as the following equation (7) if the former, and may be defined as the following equation (8) if the latter.

  In S230 in the utterance interval estimation process of the above embodiment, the sum of squares of each amplitude value (ie, power) pw of the frequency component of the singing voice component is calculated according to the equation (5). Is not limited to this. For example, the power pw may be calculated in consideration of the spread of the singing voice component on the spectrum and considering components in the vicinity of the fundamental wave and the harmonic component. Specifically, as shown in the following equation (9), the filter FP (k) is expressed by a mixed Gaussian distribution, and the square of the result of multiplying the weight by the filter FP by the frequency spectrum (level) X (k) The power pw may be calculated by integrating the sum with the frequency index k according to the following equation (10).

  In the above embodiment, the voice waveform data SV is generated based on the voice input during the period in which the voice input device 10 performs the karaoke performance process and plays the target music. The SV is not limited to this.

  That is, in the present invention, the voice input data 10 may be generated in the voice input device 10 using a well-known after-recording function in a karaoke machine or the like. That is, in the case of a voice input device (karaoke device) having an after-recording function, dialogue telop data (that is, lyrics telop) representing characters constituting dialogue (hereinafter referred to as dialogue constituent characters) as data relating to dialogue to be uttered. Data) and dialogue output data that defines the timing for displaying dialogue constituent characters on the display unit 13 (that is, data similar to the lyrics output data). Therefore, when the speech waveform data SV is acquired using the after-recording function, the speech input device 10 displays the speech based on the speech telop data on the display unit 13 and performs speech during the period in which the speech is displayed on the display unit 13. A speech waveform input via the input unit 14 may be stored in the MIDI storage server 25 as speech waveform data SV.

In this case, the information processing apparatus 30 may use the speech waveform data SV generated using the after-recording function as a processing target of the speech parameter registration process.
Moreover, in the said embodiment, although the karaoke apparatus was assumed as the audio | voice input apparatus 10, the apparatus assumed as the audio | voice input apparatus 10 is not restricted to a karaoke apparatus, For example, a well-known portable terminal (a mobile phone or portable information) Terminal) or a known information processing apparatus (so-called personal computer) may be assumed.

  In the speech synthesis system of the above embodiment, the MIDI storage server 25 is provided. However, in the present invention, the MIDI storage server 25 may not be provided. In this case, the music data MD and the voice waveform data SV may be stored in the storage unit 17 of the voice input device 10, may be stored in the data storage server 50, and further stored in the information processing device 30. It may be stored in the unit 34.

Similarly, in the speech synthesis system of the above-described embodiment, the data storage server 50 is provided. However, in the present invention, the data storage server 50 may not be provided. In this case, the voice parameter PM and the expression table TD may be stored in the storage unit 34 of the information processing device 30, may be stored in the storage unit 17 of the voice input device 10, or may be a MIDI storage server. 25 may be stored.
[Correspondence between Embodiment and Claims]
Finally, the relationship between the description of the above embodiment and the description of the scope of claims will be described.

  S120 in the voice parameter registration process of the above embodiment corresponds to the content information acquisition unit and the timing information acquisition unit in the description of the claims, and S110 corresponds to the score data acquisition unit in the description of the claims. S130 corresponds to the sound data acquisition means in the claims. Further, S230 in the utterance section estimation process of the above embodiment corresponds to the power transition deriving means in the description of the claims, S240 corresponds to the utterance section specifying means in the description of the claims, and S250, S260. Is equivalent to the note lyrics corresponding means in the description of the claims, and S270 to S290 correspond to the note singing voice integration means in the claims.

  Further, S210 in the utterance section estimation process of the above embodiment corresponds to a pitch transition deriving unit in the description of the claims, and S280 corresponds to a pitch specifying unit in the description of the claims. Note that S160 in the voice parameter registration process of the above embodiment corresponds to the parameter derivation means in the claims.

  DESCRIPTION OF SYMBOLS 1 ... Speech synthesis system 10 ... Voice input device 11 ... Communication part 12 ... Input reception part 13 ... Display part 14 ... Voice input part 15 ... Voice output part 16 ... Sound source module 17 ... Memory | storage part 20 ... Control part 21 ... ROM 22 ... RAM 23 ... CPU 25 ... MIDI storage server 30 ... Information processing device 31 ... Communication unit 32 ... Input receiving unit 33 ... Display unit 34 ... Storage unit 40 ... Control unit 41 ... ROM 42 ... RAM 43 ... CPU 50 ... Data storage server 60 ... Voice output terminal

Claims (10)

Content information acquisition means for acquiring utterance content information representing a character string of content to be uttered in a target music that is one music;
Timing information acquisition means for acquiring utterance timing information defining the utterance start timing of the character represented by the specific content information which is the utterance content information acquired by the content information acquisition means;
Representing at least the musical score of the singing melody in the target musical piece, and for each output sound constituting the singing melody, musical score data acquisition means for acquiring musical score data in which at least the pitch and performance start timing are defined;
Voice data acquisition means for acquiring voice data representing a voice waveform uttered for the character string represented by the specific content information;
Based on the voice data acquired by the voice data acquisition means, power transition for deriving a voice power transition representing a time transition of power calculated for each unit time defined to be continuous along the time axis in the voice data Deriving means;
The time when the voice power transition derived by the power transition deriving means changes along the time axis is specified as the utterance start time and the utterance end time, and the utterance start time that is continuous in the order of the utterance start time and the utterance end time An utterance section specifying means for specifying each utterance section defined by a pair of the utterance end time and the utterance end time;
The character string represented by the specific content information based on the specific content information acquired by the content information acquisition means, the utterance timing information acquired by the timing information acquisition means, and the music score data acquired by the score data acquisition means For each syllable, the output sound having the performance start timing that minimizes the time difference from the utterance start timing of the character corresponding to the syllable is identified, and the output sound and the content of the syllable are associated Note syllable correspondence means for generating each attached note syllable group;
For each utterance section specified by the utterance section specifying means, the performance start timing has a minimum time difference from the utterance start time defining the utterance section, and is generated by the note lyrics association means. A note singing syllable set, and at least a note singing voice integration means for generating syllable data in which the utterance section and the note syllable set are associated with each other .
The utterance section specifying means is
In the time progress of the voice power transition, each time when the power becomes equal to or higher than a predetermined threshold value is set as the utterance start time, and each timing when the power becomes equal to or lower than the specified threshold value is set as the utterance end time. Identify each section
An utterance section identification device characterized by the above.   Content information acquisition means for acquiring utterance content information representing a character string of content to be uttered in a target music that is one music;
  Timing information acquisition means for acquiring utterance timing information defining the utterance start timing of the character represented by the specific content information which is the utterance content information acquired by the content information acquisition means;
  Representing at least the musical score of the singing melody in the target musical piece, and for each output sound constituting the singing melody, musical score data acquisition means for acquiring musical score data in which at least the pitch and performance start timing are defined;
  Voice data acquisition means for acquiring voice data representing a voice waveform uttered for the character string represented by the specific content information;
  Based on the voice data acquired by the voice data acquisition means, power transition for deriving a voice power transition representing a time transition of power calculated for each unit time defined to be continuous along the time axis in the voice data Deriving means;
  The time when the voice power transition derived by the power transition deriving means changes along the time axis is specified as the utterance start time and the utterance end time, and the utterance start time that is continuous in the order of the utterance start time and the utterance end time An utterance section specifying means for specifying each utterance section defined by a pair of the utterance end time and the utterance end time;
  The character string represented by the specific content information based on the specific content information acquired by the content information acquisition means, the utterance timing information acquired by the timing information acquisition means, and the music score data acquired by the score data acquisition means For each syllable, the output sound having the performance start timing that minimizes the time difference from the utterance start timing of the character corresponding to the syllable is identified, and the output sound and the content of the syllable are associated Note syllable correspondence means for generating each attached note syllable group;
  For each utterance section specified by the utterance section specifying means, the performance start timing has a minimum time difference from the utterance start time defining the utterance section, and is generated by the note lyrics association means. Note singing syllable sets, and at least note singing voice integration means for generating syllable data in which the utterance section and the note syllable set are associated with each other
  With
  The utterance section specifying means is
  As a result of time differentiation of the voice power transition, each of the utterance sections is identified with each of the maximum timings as the utterance start time and each of the minimum timings as the utterance end time.
  An utterance section identification device characterized by the above. At least a pitch transition deriving unit for deriving a pitch time transition in which the pitch in the voice waveform has shifted along the time axis based on the voice data acquired by the voice data acquiring unit;
A pitch specifying means for specifying a pitch in each of the utterance sections specified by the utterance section specifying means in a pitch time transition derived by the pitch transition derivation means;
The note singing voice integration means is
The syllable data is generated by associating each utterance pitch specified by the pitch specifying means, the utterance interval corresponding to the utterance pitch, and the note syllable group corresponding to the utterance interval. vocal section specifying device according to claim 1 or claim 2, characterized in. The pitch specifying means is:
The output corresponding to the unit time among the output sounds represented by the musical score data is used as the autocorrelation value of the frequency spectrum calculated for each unit time defined to be continuous along the time axis in the audio data. The pitch corresponding to the frequency that maximizes the result of multiplying the weight function with the weights defined along the frequency axis so that the higher the frequency corresponding to the pitch of the sound is, the higher is the speech pitch. The utterance section identification device according to claim 3 . The pitch specifying means is:
The pitch corresponding to the frequency at which the autocorrelation value of the frequency spectrum calculated for each unit time defined so as to be continuous along the time axis in the voice data is specified as the utterance pitch. The utterance section specifying device according to claim 3 . The note singing voice integration means is
By associating the power at each time in the sound power transition derived by the power transition deriving unit, the utterance section corresponding to the time, and the note syllable group corresponding to the utterance section, the syllable data speech section detection apparatus according to any one of claims 1 or claim 2, characterized in that to generate.   Content information acquisition means for acquiring utterance content information representing a character string of content to be uttered in a target music that is one music;
  Timing information acquisition means for acquiring utterance timing information defining the utterance start timing of the character represented by the specific content information which is the utterance content information acquired by the content information acquisition means;
  Representing at least the musical score of the singing melody in the target musical piece, and for each output sound constituting the singing melody, musical score data acquisition means for acquiring musical score data in which at least the pitch and performance start timing are defined;
  Voice data acquisition means for acquiring voice data representing a voice waveform uttered for the character string represented by the specific content information;
  Based on the voice data acquired by the voice data acquisition means, power transition for deriving a voice power transition representing a time transition of power calculated for each unit time defined to be continuous along the time axis in the voice data Deriving means;
  The time when the voice power transition derived by the power transition deriving means changes along the time axis is specified as the utterance start time and the utterance end time, and the utterance start time that is continuous in the order of the utterance start time and the utterance end time An utterance section specifying means for specifying each utterance section defined by a pair of the utterance end time and the utterance end time;
  The character string represented by the specific content information based on the specific content information acquired by the content information acquisition means, the utterance timing information acquired by the timing information acquisition means, and the music score data acquired by the score data acquisition means For each syllable, the output sound having the performance start timing that minimizes the time difference from the utterance start timing of the character corresponding to the syllable is identified, and the output sound and the content of the syllable are associated Note syllable correspondence means for generating each attached note syllable group;
  For each utterance section specified by the utterance section specifying means, the performance start timing has a minimum time difference from the utterance start time defining the utterance section, and is generated by the note lyrics association means. Note syllable set, and at least the utterance interval and the note syllable set
Note singing voice integration means for generating syllable data associated with
  Parameter deriving means for deriving a speech parameter, which is at least one feature amount defined in advance, from the speech data in the utterance section in the syllable data generated by the note singing voice integration unit;
  With
  The utterance section specifying means is
  In the time progress of the voice power transition, each time when the power becomes equal to or higher than a predetermined threshold value is set as the utterance start time, and each timing when the power becomes equal to or lower than the specified threshold value is set as the utterance end time. Identify each section
  An audio parameter generation device characterized by the above. Content information acquisition means for acquiring utterance content information representing a character string of content to be uttered in a target music that is one music;
Timing information acquisition means for acquiring utterance timing information defining the utterance start timing of the character represented by the specific content information which is the utterance content information acquired by the content information acquisition means;
Representing at least the musical score of the singing melody in the target musical piece, and for each output sound constituting the singing melody, musical score data acquisition means for acquiring musical score data in which at least the pitch and performance start timing are defined;
Voice data acquisition means for acquiring voice data representing a voice waveform uttered for the character string represented by the specific content information;
Based on the voice data acquired by the voice data acquisition means, power transition for deriving a voice power transition representing a time transition of power calculated for each unit time defined to be continuous along the time axis in the voice data Deriving means;
The time when the voice power transition derived by the power transition deriving means changes along the time axis is specified as the utterance start time and the utterance end time, and the utterance start time that is continuous in the order of the utterance start time and the utterance end time An utterance section specifying means for specifying each utterance section defined by a pair of the utterance end time and the utterance end time;
The character string represented by the specific content information based on the specific content information acquired by the content information acquisition means, the utterance timing information acquired by the timing information acquisition means, and the music score data acquired by the score data acquisition means For each syllable, the output sound having the performance start timing that minimizes the time difference from the utterance start timing of the character corresponding to the syllable is identified, and the output sound and the content of the syllable are associated Note syllable correspondence means for generating each attached note syllable group;
For each utterance section specified by the utterance section specifying means, the performance start timing has a minimum time difference from the utterance start time defining the utterance section, and is generated by the note lyrics association means. A note singing syllable set, and at least a note singing voice integration means for generating syllable data in which the utterance section and the note syllable set are associated with each other,
Parameter deriving means for deriving a speech parameter, which is at least one feature amount defined in advance, from the speech data in the utterance section in the syllable data generated by the note singing voice integration unit;
With
The utterance section specifying means is
As a result of time differentiation of the voice power transition, each of the utterance sections is identified with each of the maximum timings as the utterance start time and each of the minimum timings as the utterance end time.
An audio parameter generation device characterized by the above. A content information acquisition procedure for acquiring utterance content information representing a character string of content to be uttered in a target music that is one music,
Timing information acquisition procedure for acquiring utterance timing information that defines the utterance start timing of the character represented by the specific content information that is the utterance content information acquired in the content information acquisition procedure;
Representing at least the musical score of the singing melody in the target musical piece, and for each output sound constituting the singing melody, a musical score data acquisition procedure for acquiring musical score data in which at least the pitch and the performance start timing are defined;
A voice data acquisition procedure for acquiring voice data representing a voice waveform uttered for the character string represented by the specific content information;
Based on the voice data acquired in the voice data acquisition procedure, power transition for deriving a voice power transition representing a time transition of power calculated for each unit time defined to be continuous along the time axis in the voice data Derivation procedure;
The time when the voice power transition derived in the power transition deriving procedure changes along the time axis is specified as the utterance start time and the utterance end time, and the utterance start time that is continuous in the order of the utterance start time and the utterance end time An utterance section specifying procedure for specifying each of the utterance sections defined by the pair of the utterance end time and the utterance end time;
The character string represented by the specific content information based on the specific content information acquired in the content information acquisition procedure, the utterance timing information acquired in the timing information acquisition procedure, and the music score data acquired in the score data acquisition procedure For each syllable, the output sound having the performance start timing that minimizes the time difference from the utterance start timing of the character corresponding to the syllable is identified, and the output sound and the content of the syllable are associated A note lyrics procedure for generating each note syllable set,
For each utterance section identified by the utterance section identification procedure, the performance start timing that minimizes the time difference from the utterance start time defining the utterance section is generated, and is generated by the procedure for associating notes and lyrics. And at least a note singing voice integration procedure for generating syllable data in which the utterance section and the note syllable group are associated with each other .
In the utterance section identification procedure,
In the time progress of the voice power transition, each time when the power becomes equal to or higher than a predetermined threshold value is set as the utterance start time, and each timing when the power becomes equal to or lower than the specified threshold value is set as the utterance end time. Identify each section
A program characterized by that.   A content information acquisition procedure for acquiring utterance content information representing a character string of content to be uttered in a target music that is one music,
  Timing information acquisition procedure for acquiring utterance timing information that defines the utterance start timing of the character represented by the specific content information that is the utterance content information acquired in the content information acquisition procedure;
  Representing at least the musical score of the singing melody in the target musical piece, and for each output sound constituting the singing melody, a musical score data acquisition procedure for acquiring musical score data in which at least the pitch and the performance start timing are defined;
  A voice data acquisition procedure for acquiring voice data representing a voice waveform uttered for the character string represented by the specific content information;
  Based on the voice data acquired in the voice data acquisition procedure, power transition for deriving a voice power transition representing a time transition of power calculated for each unit time defined to be continuous along the time axis in the voice data Derivation procedure;
  The time when the voice power transition derived in the power transition deriving procedure changes along the time axis is specified as the utterance start time and the utterance end time, and the utterance start time that is continuous in the order of the utterance start time and the utterance end time An utterance section specifying procedure for specifying each of the utterance sections defined by the pair of the utterance end time and the utterance end time;
  The character string represented by the specific content information based on the specific content information acquired in the content information acquisition procedure, the utterance timing information acquired in the timing information acquisition procedure, and the music score data acquired in the score data acquisition procedure For each syllable, the output sound having the performance start timing that minimizes the time difference from the utterance start timing of the character corresponding to the syllable is identified, and the output sound and the content of the syllable are associated A note lyrics procedure for generating each note syllable set,
  For each utterance section identified by the utterance section identification procedure, the performance start timing that minimizes the time difference from the utterance start time defining the utterance section is generated, and is generated by the procedure for associating notes and lyrics. A note singing syllable set, and at least a note singing voice integration procedure for generating syllable data that associates the utterance section with the note syllable set;
  To the computer,
  In the utterance section identification procedure,
  As a result of time differentiation of the voice power transition, each of the utterance sections is identified with each of the maximum timings as the utterance start time and each of the minimum timings as the utterance end time.
  A program characterized by that.

Images