JP5845857B2 - Parameter extraction device, speech synthesis system - Google Patents

Description

  The present invention relates to a speech synthesis system that performs speech synthesis and a parameter extraction device that extracts speech parameters necessary for speech synthesis from speech.

  2. Description of the Related Art Conventionally, in a speech synthesizer, text that is input from the outside and speech that represents an emotion designated by an external operation is generated and output by speech synthesis (see Patent Document 1).

  In order to realize this, the speech synthesizer described in Patent Document 1 stores a plurality of emotion expression patterns composed of language attribute vectors, acoustic attribute vectors, and emotion vectors. Then, an emotion expression pattern that represents the emotion closest to the emotion specified by the external input is extracted from a plurality of accumulated emotion expression patterns, and input according to the extracted emotion expression pattern Speech synthesis is performed on the generated text.

  In Patent Document 1, a language attribute vector indicates an attribute that builds a social relationship between a speaker and a listener, such as emotions such as linguistic content, meaning, or concept, and requests. Or speaker's attitude, whether it is equal or superior, whether it is a master-slave relationship, etc. The acoustic attribute vector indicates the acoustic features used to express the social relationship between the speaker and the listener. The acoustic features include the average pitch, the dynamic range of the pitch, and the glottis. They are feelings like openness, speaker's attitudes like requests and orders, whether they are equal or superior, or a master-slave relationship. The emotion vector indicates the social relationship between the speaker and the listener. The emotion expressed as a whole voice, the speaker's attitude such as request or command, whether it is equal or superior, whether it is a master-slave relationship, etc. It is.

JP 2007-183421 A

  By the way, the synthesized sound output from the speech synthesizer described in Patent Document 1 is obtained by synthesizing (changing) speech parameters of one standard voice quality prepared in advance according to an emotion expression pattern.

  Therefore, in the speech synthesizer described in Patent Document 1, although the emotion expressed by the synthesized sound output from the speech synthesizer can be changed and diversified, the gender of the person who uttered the synthesized sound, age, It is difficult to make voice quality diverse.

  In other words, since the speech synthesizer described in Patent Document 1 synthesizes speech based on speech parameters of one standard voice quality, it is difficult to diversify the features of the speaker with respect to the sound output as synthesized sound. There was a problem of being.

  Accordingly, an object of the present invention is to diversify a speaker's characteristics with respect to a synthesized sound generated by speech synthesis.

  In the parameter extraction device of the present invention made to achieve the above object, the content information acquisition means acquires utterance content information representing the character string of the content to be uttered, and the timing information acquisition means is the content information acquisition means. Among the character strings represented by the specific content information that is the acquired utterance content information, obtain utterance timing information that specifies the utterance start timing of at least one character, and the waveform acquisition means uses the character represented by the specific content information. A target waveform that is a speech waveform uttered for a column is acquired.

  In the syllable waveform extracting means, at least for each syllable forming the character string represented by the specific content information from the target waveform acquired by the waveform acquiring means based on the utterance timing information acquired by the timing information acquiring means. Extract a syllable waveform that is a speech waveform uttered. Further, the parameter deriving means derives a speech parameter which is at least one feature quantity defined in advance from each extracted syllable waveform.

  Further, in the metadata generation means, based on the specific content information and the utterance timing information acquired by the timing information acquisition means, the property of the voice represented by the syllable waveform is estimated, and the estimation result is generated as metadata, The parameter registering means stores the speech parameter derived by the parameter deriving means and the metadata generated by the metadata generating means in association with each corresponding syllable in the first storage device.

  Note that the feature quantity as a speech parameter in the present invention is a feature quantity required when executing speech synthesis by formant synthesis, and includes, for example, a fundamental frequency, a mel frequency cepstrum (MFCC), power, and the like. Each time difference is included.

  According to such a parameter extraction device, speech parameters of various speakers are derived by deriving speech parameters from each target waveform obtained by causing many people to utter the content of the character string represented by the utterance content information. it can. As a result, according to the parameter extraction device of the present invention, the types of voice parameters can be diversified.

  Moreover, in the parameter extraction apparatus of this invention, metadata can be estimated automatically based on specific content information and utterance timing information. For this reason, according to the parameter extraction device of the present invention, unlike the conventional speech synthesizer, when the content of the character string represented by the utterance content information is uttered, the property of the speech as metadata is used. It is possible to eliminate the need for the users to input.

From the above, if speech synthesis is performed using speech parameters extracted by the parameter extraction apparatus of the present invention, the features of a speaker who can be regarded as having uttered the synthesized speech can be diversified.
In addition, the nature of speech in the present invention includes at least the emotion of the speaker when the speech is uttered, and is a concept that includes, for example, emotion and atmosphere. Furthermore, information necessary for estimating emotions may be included in the nature of speech.

  In the parameter extraction device of the present invention, the score data acquisition means represents the score of the target song that is one of the songs, and for each output sound output from the sound module, at least the pitch and the performance start timing are present. Get the specified musical score data.

  And the content information acquisition means in this invention acquires the character string of the lyric constituent character which comprises the lyrics of the object music as utterance content information, and the timing information acquisition means has an output timing for at least one of the lyric constituent characters. Then, the lyrics output timing associated with the performance start timing of the output sound corresponding to the lyrics constituent character is acquired as utterance timing information. At this time, the waveform acquisition means in the present invention acquires, as the target waveform, a waveform in which the voice input during the performance of the target music based on the score data has shifted along the time axis, and the syllable waveform extraction means , The speech waveform in the section corresponding to each output sound may be extracted as a syllable waveform.

  According to such a parameter extraction device, speech waveforms can be collected using a karaoke device or the like in which speech is input during a period in which the target music is being played based on the musical score data.

And according to such a parameter extraction apparatus, a phoneme parameter can be produced | generated from the audio | voice sung in the karaoke apparatus etc.
Generally, if the key in a music is a major key, a bright impression is received, and if it is monotonous, a sad impression is received. In the same manner, the lyrics have many bright impressions when the key of the music is major, and many sad lyrics when the key of the music is monotonous.

Therefore, if the musical score data in the present invention is transposed in the music of the target music, the musical score data includes a transposition flag indicating the position of the time of transposition on the time axis when the performance start time of the target music is used as the origin. But it ’s okay.
In this case, in the metadata generation means of the present invention, based on the score data acquired by the score data acquisition means, the section specifying means specifies the same key section, which is each section in which the same key is continued in the target music. The main sound specifying means is included in each key-same section specified by the section specifying means, and specifies the last output sound along the time axis in each key-same section as the main sound. Then, the pitch name frequency deriving means starts from the pitch name of the main sound specified by the main sound specifying means, with the appearance pitch name frequency indicating the frequency of the output sound of the same pitch name included in the same key interval specified by the section specifying means. The key estimation means derives each of the appearance sounds derived by the pitch name frequency deriving means into a key template prepared in advance for each key as a template representing the distribution of pitch names that can be used in each key. As a result of collating the name frequency, the sound property for each key having the highest correlation is defined as metadata.
Further, the key estimation means uses the sound property of being bright as the metadata if the key with the highest correlation is the major key, and the sound property of dark as the metadata when the key with the highest correlation is the minor key. Data.

  With such a parameter extraction device, the key of each tone-same section in the target music can be used as metadata, and by extension, the emotion of the speaker when the lyrics corresponding to each key-same section are uttered is metadata. It can be. Moreover, according to such a key specifying means, the key in each key same section can be specified reliably.

  Further, in the metadata generating means in the present invention, the word dividing means divides the character string represented by the utterance content information acquired by the content information acquiring means for each word character that is a character string constituting the word, The data extracting means may extract the property information corresponding to the word represented by each word character divided by the word dividing means as metadata from the word property table.

According to such a parameter extraction device, the property of each word can be used as metadata.
The word property table in the present invention is a table in which property information representing the property of each word is associated with identification information of the word, and is prepared in advance. Furthermore, the word property mentioned here includes the meaning of the word and the emotion represented by the word.

  Furthermore, the content information acquisition means in this invention may acquire the sentence structure character which is a character string which comprises at least 1 sentence as utterance content information. In this case, the timing information acquisition means acquires, as utterance timing information, information indicating the output timing of outputting at least one character constituting the sentence constituent character to the outside, and the waveform acquisition means includes the utterance content information. A waveform in which a voice input during output of a character string constituting a sentence-constituting character based on the time axis may be acquired as a target waveform.

  According to such a parameter extraction device, a speech waveform when a sentence is read out can be acquired as a target waveform. That is, according to the parameter extraction device of the present invention, the target waveform can be acquired via a karaoke device having a so-called after-recording function.

Note that outputting the character to the outside here includes at least displaying the character.
By the way, the present invention may be implemented as a speech synthesis system.
In this case, the speech synthesis system of the present invention preferably includes a parameter extraction device and a synthesized sound output device.

  Among these, the parameter extraction device has content information acquisition means, timing information acquisition means, waveform acquisition means, syllable waveform extraction means, parameter derivation means, metadata generation means, and parameter registration means, A metadata analysis table in which the parameter analysis means analyzes the audio parameter stored in the first storage device for each metadata associated with the audio parameter, and represents a range of each audio parameter corresponding to the metadata May be generated and stored in the second storage device.

  Further, in the synthesized sound output device, the output property information acquisition unit acquires output property information that is input from the outside and represents the sound property, and the word acquisition unit acquires an output statement that represents the word input from the outside. . Then, the table acquisition unit acquires the metadata correspondence table including the metadata corresponding to the output property information acquired by the output property information acquisition unit from the second storage device, and the voice having information corresponding to the output property information The parameter is acquired from the first storage device.

Then, the speech synthesizer synthesizes the speech parameter acquired by the table acquisition unit according to the metadata correspondence table so that the output message acquired by the word acquisition unit is obtained, and the output unit synthesizes the speech by the speech synthesizer. The synthesized sound generated in this way is output.
The parameter extraction device represents a musical score of a target musical piece that is one of musical pieces, and obtains musical score data in which at least a pitch and a performance start timing are defined for each output sound output from the sound source module. Data acquisition means are provided.
The content information acquisition means acquires a character string of lyric constituent characters constituting the lyrics of the target music as the utterance content information, and the timing information acquisition means has an output timing for at least one of the lyric constituent characters. Then, the lyrics output timing associated with the performance start timing of the output sound corresponding to the lyrics constituent characters is acquired as the utterance timing information.
And the said waveform acquisition means acquires the waveform which the voice input during the performance of the said target music based on the said score data changed along the time axis as said target waveform.
Further, the syllable waveform extracting means extracts a speech waveform in a section corresponding to each output sound as the syllable waveform in the target waveform, and the score data is transposed in the music of the target music. Then, a modulation flag indicating a position at the time of transposing on the time axis when the starting point of performance of the target music is used as the origin is included.
Further, the metadata generation means, based on the musical score data acquired by the musical score data acquisition means, section specifying means for specifying the same key section that is each section in which the same key is continued in the target music, The main tone specifying means for specifying the last output sound along the time axis in each key same section as the main sound included in each key same section specified by the section specifying means, and the same key specified by the section specifying means A pitch name frequency deriving means for deriving an appearance pitch name frequency representing a frequency of output sounds of the same pitch name included in a section for each key-same section starting from the pitch name of the main sound identified by the main sound identifying means; As a result of comparing each appearance name name frequency derived by the above-described sound name frequency deriving means with a key template prepared in advance for each key as a template representing the distribution of pitch names that can be used in the key, the tone with the highest correlation is obtained. The nature of the speech corresponding to, respectively, and a tone estimation means and the metadata.
The key estimation means uses the sound property of bright as the metadata if the key with the highest correlation is the major key, and the sound property of dark as the key with the highest correlation is the minor key. Use metadata.

According to such a speech synthesis system, various synthesized sounds can be generated based on the metadata correspondence table and the speech parameters.
That is, according to the speech synthesis system of the present invention, it is possible to diversify the features of a speaker who can be regarded as having synthesized speech.

It is a block diagram which shows the whole structure of a speech synthesis system. It is a flowchart which shows the process sequence of an audio | voice parameter registration process. It is a flowchart which shows the process sequence of the metadata estimation process in 1st embodiment. It is a figure which shows the processing content of a metadata estimation process. It is a figure which shows the processing content of a metadata estimation process. It is a figure which shows the processing content of a metadata estimation process. It is a figure which shows the outline | summary of the audio | voice parameter registration process in 1st embodiment. It is a flowchart which shows the process sequence of an audio | voice analysis process. It is a figure which illustrates a metadata corresponding | compatible table. It is a flowchart which shows the process sequence of a speech synthesis process. It is a flowchart which shows the process sequence of the metadata estimation process in 2nd embodiment. It is a figure which shows the outline | summary of the audio | voice parameter registration process in 2nd embodiment.

Embodiments of the present invention will be described below with reference to the drawings.
[First embodiment]
<About the speech synthesis system>
FIG. 1 is a diagram showing a schematic configuration of a speech synthesis system to which the present invention is applied.

  The speech synthesis system 1 to which the present invention has been applied is a speech synthesized based on speech parameters registered in advance (that is, synthesis) so that speech having contents designated by the user of the speech synthesis system 1 is output. Sound).

  In order to realize this, the speech synthesis system 1 is used for speech input device 10 for inputting speech, speech input through speech input device 10 (hereinafter referred to as speech waveform data SV), and karaoke. MIDI storage server 25 for storing various types of data (hereinafter referred to as music data MD), and processing for generating at least audio parameters based on audio waveform data SV and music data MD stored in MIDI storage server 25 And an information processing apparatus 30 to be executed. Furthermore, the speech synthesis system 1 outputs a synthesized sound obtained by synthesizing speech based on the speech parameters stored in the data storage server 50 and the speech storage stored in the data storage server 50. And an audio output terminal 60. Note that the speech synthesis system 1 in this embodiment includes a plurality of speech output terminals 60.

  That is, in the speech synthesis system 1 of the present embodiment, the information processing apparatus 30 generates at least the speech parameter PM based on the speech waveform data SV and the music data MD stored in the MIDI storage server 25 and stores the data. Store in the server 50. Then, the voice output terminal 60 performs synthesis by voice synthesis based on the voice parameter PM stored in the data storage server 50 so that the voice having the content specified by the user is output via the voice output terminal 60. Output sound.

The speech parameter PM referred to here is a feature amount of speech used for so-called formant synthesis, which will be described in detail later. For example, the fundamental frequency, mel frequency cepstrum (MFCC), power, And the time difference between them.
<About the MIDI storage server>
First, the MIDI storage server 25 is a device mainly composed of a storage device configured to be able to read and write stored contents, and is connected to the voice input device 10 via a communication network.

  The MIDI storage server 25 stores at least music data MD prepared in advance for each piece of music. The music data MD includes music MIDI data DM (corresponding to the score data in the claims) and the lyrics data group DL. The music MIDI data DM and the lyrics data group DL are respectively corresponding music data. Are associated with each other.

  Of these, the music MIDI data DM is data representing the score of one music according to the well-known MIDI (Musical Instrument Digital Interface) standard, and is prepared in advance for each music. Each of the music MIDI data DM includes identification data that is data for discriminating music, a score track that represents a score for each musical instrument used in the music, and a modulation flag that represents the time at which the key changes in the music. Have at least.

  The musical score track defines at least a pitch (so-called note number) and a period during which the sound module outputs the output sound (hereinafter, note length) for each output sound output from the MIDI sound source. Yes. However, the note length of the musical score track is the performance start timing (so-called note-on timing) indicating the time from the start of the performance of the music until the output of the output sound is started, and the output of the output sound is ended. It is defined by the performance end timing (so-called note-off timing) that represents the time from the start of performance of the music.

  Note that the score track includes, for example, a keyboard instrument (eg, piano or pipe organ), a stringed instrument (eg, violin, viola, guitar, bass guitar, koto), or a percussion instrument (eg, vibraphone, drum, cymbal, timpani, Xylophone, etc.) and wind instruments (eg, clarinet, trumpet, flute, shakuhachi, etc.).

  On the other hand, the lyrics data group DL is data relating to lyrics displayed on a display device that constitutes a well-known karaoke device, and lyrics telop data DT representing characters constituting the lyrics of the music (hereinafter referred to as lyrics constituent characters); And lyrics output data DO in which a timing correspondence relationship for associating the lyrics output timing, which is the output timing of the lyrics constituent characters, with the performance of the music MIDI data DM is provided.

Specifically, the timing correspondence relationship in the present embodiment is based on the timing of starting the output of the lyrics telop data DT to the timing of starting the performance of the music MIDI data DM, and along the time axis of the specific music. The lyrics output timing of each lyrics constituent character is defined by the elapsed time from the start of the performance of the music MIDI data DM. The elapsed time referred to here is, for example, a time representing the timing of executing color change of the displayed lyrics constituent characters, and is defined by the color change speed. Further, the lyric constituent characters here may be each of the characters constituting the lyric, or may be a phrase or a phrase in which each of the characters is grouped according to a specific rule along the time axis.
<About the configuration of the voice input device>
Next, the voice input device 10 will be described.

  The voice input device 10 includes a communication unit 11, an input receiving unit 12, a display unit 13, a voice input unit 14, a voice output unit 15, a sound source module 16, a storage unit 17, and a control unit 20. ing. That is, the voice input device 10 in the present embodiment is configured as a so-called well-known karaoke device.

  Among these, the communication unit 11 communicates with the outside through the communication network (for example, a public wireless communication network or a network line). The input reception unit 12 is an input device (for example, a key, a switch, a remote control reception unit, or the like) that receives input of information and commands in accordance with external operations.

  The display unit 13 is a display device (for example, a liquid crystal display or a CRT) that displays an image including at least information indicated by a character code. The voice input unit 14 is a device (so-called microphone) that converts sound into an electrical signal and inputs the signal to the control unit 20. The audio output unit 15 is a device (so-called speaker) that converts an electrical signal from the control unit 20 into sound and outputs the sound. Furthermore, the sound module 16 is a device (for example, a MIDI sound source) that outputs a sound (that is, an output sound) that simulates a sound from a sound source, based on data defined by the MIDI (Musical Instrument Digital Interface) standard. .

The storage unit 17 is a non-volatile storage device (for example, a hard disk device or a flash memory) configured to be able to read and write stored contents.
The control unit 20 is stored in the ROM 21 that stores processing programs and data that need to retain stored contents even when the power is turned off, the RAM 22 that temporarily stores processing programs and data, and the ROM 21 and RAM 22. It is mainly configured by a known computer having at least a CPU 23 that executes each process (various operations) according to the processing program.

  The ROM 21 stores a processing program for executing a well-known karaoke performance process by the control unit, and a voice waveform inputted through the voice input unit 14 during a period in which the target music piece is being played by the karaoke performance process. As the data SV, a processing program is stored that is associated with the music identification information for identifying the target music and for which the control unit 20 executes the audio storage processing stored in the MIDI storage server 25.

  That is, the voice input device 10 acquires music data MD corresponding to one piece of music (hereinafter referred to as a target song) designated via the input receiving unit 12 from the MIDI storage server 25 according to the karaoke performance process, The target music is played based on the music MIDI data DM in the music data MD, and the lyrics of the target music are displayed on the display unit 13 based on the lyrics data group DL in the music data MD.

Furthermore, in the voice input device 10, during the karaoke performance process, the voice input via the voice input unit 14 is used as voice waveform data SV, and music identification information (here, music data MD) for identifying the target music. Itself) and speaker identification information (hereinafter referred to as “speaker ID”) for identifying the person who has input the voice (hereinafter referred to as “speaker”), and stores it in the MIDI storage server 25. The voice waveform data SV stored in the MIDI storage server 25 is also associated with speaker feature information representing the features of the speaker. The speaker feature information includes, for example, the gender and age of the speaker. Etc.
<Configuration of information processing device>
Next, the information processing apparatus 30 will be described.

The information processing apparatus 30 includes a communication unit 31, an input reception unit 32, a display unit 33, a storage unit 34, and a control unit 40.
Among these, the communication unit 31 communicates with the outside via a communication network (for example, a public wireless communication network or a network line). The input receiving unit 32 is an input device (for example, a keyboard or a pointing device) that receives input of information and commands in accordance with an external operation.

The display unit 33 is a display device (for example, a liquid crystal display or a CRT) that displays an image.
The storage unit 34 is a non-volatile storage device (for example, a hard disk device or a flash memory) configured to be able to read and write stored contents. The control unit 40 is configured around a known computer having at least a ROM 41, a RAM 42, and a CPU 43.

  In the ROM 41 of the information processing apparatus 30, the voice parameter PM generated based on the voice waveform data SV and the music data MD stored in the MIDI storage server 25 is stored in the nature of the voice that is the generation source of the voice parameter PM. A processing program for the control unit 40 to execute a voice parameter registration process stored in the data storage server 50 in association with the metadata representing the.

  Further, in the ROM 41 of the information processing apparatus 30, metadata representing a tendency of the voice parameter PM corresponding to the metadata based on the result of statistical processing of the voice parameter PM stored in the data storage server 50 by the voice parameter registration process. A correspondence table (hereinafter referred to as an expression table TD) is created for each type of metadata associated with the voice parameter PM, and the control unit 40 executes voice analysis processing stored in the data storage server 50. A processing program is stored.

  In the present embodiment, the metadata represents the nature of speech including at least the emotion of the speaker when the speech is uttered, and is a concept that includes, for example, emotion and atmosphere. Furthermore, information necessary for estimating emotions may be included in the nature of speech.

The data storage server 50 is a device that is mainly configured of a storage device that is configured to be able to read and write stored contents, and is connected to the information processing device 30 via a communication network.
<Voice parameter registration process>
Next, the voice parameter registration process executed by the information processing apparatus 30 will be described.

Here, FIG. 2 is a flowchart showing the processing procedure of the voice parameter registration processing.
As shown in FIG. 2, when the voice parameter registration process is started, the music MIDI data DM of the music specified through the input receiving unit 32 (hereinafter referred to as the target music) is acquired (S110). Subsequently, the lyrics data group DL of the target music is acquired (S120), and one speech waveform data SV corresponding to the target music and corresponding to the speaker ID designated via the input receiving unit 32 (claims) (Corresponding to the target waveform in the range) is acquired (S130).

  Furthermore, in the speech waveform data SV acquired in S130, a speech waveform (hereinafter referred to as a syllable waveform) in a section corresponding to each syllable included in the utterance content of the speech waveform data SV is specified (S140).

  Specifically, in S140 of the present embodiment, the performance start timing and performance end of each output sound defined in a musical score track (hereinafter referred to as a melody track) representing the singing melody in the music MIDI data DM acquired in S110. The timing is extracted, and the syllables of the lyrics constituent characters associated with each output sound are specified. Then, in the speech waveform data SV, the speech waveform in the section corresponding to each section from the performance start timing to the performance end timing of each output sound is specified as a syllable waveform. Note that each syllable waveform specified in S140 of the present embodiment is associated with the content of the syllable uttered by the syllable waveform.

  Further, the speech parameter PM is derived from each syllable waveform (S150). In S150 of the present embodiment, the fundamental frequency, the mel frequency cepstrum (MFCC), the power, and the time difference between them are each derived as a voice parameter PM. Since these fundamental frequency, MFCC, and power derivation methods are well known, detailed description thereof is omitted here. For example, if the fundamental frequency is used, autocorrelation along the time axis of the syllable waveform, syllable waveform What is necessary is just to derive | lead-out using methods, such as an autocorrelation of a frequency spectrum, or a cepstrum method. In addition, in the case of MFCC, a result obtained by applying a time analysis window to a syllable waveform and performing frequency analysis (for example, FFT) for each time analysis window is further obtained by logarithmizing the size for each frequency. It can be derived by frequency analysis. The power may be derived by applying a time analysis window to the syllable waveform and integrating the square of the amplitude in the time direction.

Subsequently, metadata estimation processing for estimating metadata about each syllable waveform specified in S140 is executed (S160).
Here, the metadata estimation process will be described in detail with reference to FIG. FIG. 3 is a flowchart showing the processing procedure of the metadata estimation processing.

  When this metadata estimation process is started, first, based on the music MIDI data acquired in the previous S110, the same key section that is the section in which the same key is continued in the target music is specified (S310). ). Specifically, in S310 of the present embodiment, as shown in FIG. 4, based on the modulation flag included in the music MIDI data, the section between the modulation flags adjacent to each other along the time axis is identified as the same key section. To do.

  Subsequently, the main sound in the same key interval specified in S310 is specified (S320). Specifically, in S320 of this embodiment, as shown in FIG. 5, in the same key interval, the last output sound along the time axis is specified as the main sound in the same key interval. In the present embodiment, the main sound is specified for each of the same key intervals specified in S310.

  Then, a histogram (hereinafter referred to as a pitch) in which the pitch name of the main tone specified in S320 is a starting point, the pitch names of the output sounds included in the same key interval in which the main tone is specified is a rank, and the number of appearances of each pitch name is a frequency. , Referred to as frequency of appearance sound name) (S330). Specifically, as shown in FIG. 6 (A), the appearance name frequency derived in S330 of this embodiment is the total number of appearances (appearance frequency) of output sounds with the same pitch name included in the same key interval. It is a thing. And in this embodiment, even if it is an output sound from which an octave differs, if the pitch name is the same, it totals as an output sound of the same pitch name. In the present embodiment, the appearance name frequency is derived for the same section.

  Subsequently, based on the result of matching the key name frequency derived in step S330 with a key template prepared in advance for each key as a template representing the distribution of pitch names that can be used in each key, A key is specified (S340). Specifically, in S340 of the present embodiment, a major template (see FIG. 6B) that represents the distribution of pitch names that can be used in major music, and the distribution of pitch names that can be used in minor music. A minor template to be expressed (see FIG. 6C) is prepared in advance, and the frequency of appearance note names derived in S330 is collated with each key template. As a result, the key corresponding to the key template showing the highest correlation is specified as the key in the key same section. In S340 of the present embodiment, the key for each key-same section is specified.

  Furthermore, the sound property corresponding to the key of the music in the same key section identified in S340 is specified as metadata (S350). Specifically, in S350 of the present embodiment, if the key in the same key section is a major key, the property of the voice representing the emotion that the lyrics (that is, the utterance content) in the same key section is “bright” is used as metadata. Identify. If the key in the same key section is a minor key, the property of the voice representing the feeling that the lyrics in the same key section are “dark” is specified as metadata. In the present embodiment, metadata corresponding to the same key interval is assigned to all syllables included in the same key interval.

Thereafter, the process proceeds to S170 of the voice parameter registration process.
In S170 of the speech parameter registration process, speech parameter registration is performed in which the speech parameter PM derived in S150 and the metadata estimated in S160 are associated with each corresponding syllable and stored in the data storage server 50. (S170). Note that the data associated with the speech parameter PM stored in the data storage server 50 in S170 of the present embodiment includes, in addition to metadata, the content (type) of the uttered syllable, the speaker ID, and speaker characteristics. Contains information.

Thereafter, the voice parameter registration process is terminated.
As described above, as shown in FIG. 7, in the voice parameter registration process of the present embodiment, the voice waveform input during the performance period of the target music is the processing target. Then, the speech waveform data SV based on the speech waveform is divided into sections corresponding to the performance periods of the output sounds constituting the melody line of the target music (that is, each syllable included in the utterance content) to obtain a syllable waveform. And a speech parameter PM is derived from each syllable waveform.

  At the same time, in the voice parameter registration process, each period (that is, the same key section) in which the same key continues in the target music is specified, and the key (tonality) in each key same section is specified. And the property of the voice previously defined as the emotion imaged from the specified key is specified as metadata.

In addition, in the voice parameter registration process, the voice parameter PM and the metadata are associated with each other syllable and stored in the data storage server 50.
<About voice analysis processing>
Next, speech analysis processing executed by the control unit 40 of the information processing apparatus 30 will be described with reference to FIG.

  As shown in FIG. 8, when the voice analysis process is started, first, all voice parameters associated with metadata representing the same content (hereinafter referred to as target metadata) (hereinafter referred to as voice parameters). (Referred to as a group) is acquired from the data storage server 50 (S410). That is, the speech parameter group acquired in S410 of the present embodiment is all speech parameters PM associated with the target metadata among the speech parameters stored in the data storage server 50. Furthermore, the audio parameter PM here includes each of a fundamental frequency, a mel frequency cepstrum (MFCC), power, and their time differences.

Subsequently, the facial expression table TD is generated based on the voice parameter group acquired in S410 (S420).
Specifically, in S420 of the present embodiment, the average value for each voice parameter (that is, each of fundamental frequency, mel frequency cepstrum (MFCC), power, and their time difference) included in the voice parameter group acquired in S410. Is calculated. Then, a parameter difference that is a difference between the calculated average value and each voice parameter PM included in the voice parameter group acquired in S410 is determined for each speaker ID associated with the voice parameter PM, and Derived for each syllable associated with the speech parameter PM.

  Furthermore, in S420 of the present embodiment, the facial expression table TD is generated by associating the derived parameter difference with the metadata, the speaker ID, and the syllable corresponding to the parameter difference. That is, as shown in FIG. 9, the expression table TD generated in S420 is obtained by classifying the content of metadata for each speaker ID, and the syllable content and the parameter difference corresponding to the syllable. Are associated with each other.

Then, the facial expression table TD generated in S420 is stored in the data storage server 50 (S430).
Thereafter, the voice analysis process is terminated.
<Configuration of audio output terminal>
Next, the audio output terminal will be described (see FIG. 1).

  The voice output terminal 60 includes an information receiving unit 61, a display unit 62, a sound output unit 63, a communication unit 64, a storage unit 65, and a control unit 67. As the audio output terminal 60 in the present embodiment, for example, a known portable terminal (a mobile phone or a portable information terminal) or a known information processing apparatus (a so-called personal computer) may be assumed.

  Among these, the information reception part 61 receives the information input via the input device (not shown). The display unit 62 displays an image based on a command from the control unit 67. The sound output unit 63 is a known device that outputs sound, and includes, for example, a PCM sound source and a speaker.

  The communication unit 64 is for the voice output terminal 60 to perform information communication with the outside via a communication network (for example, a public wireless communication network or a network line). The storage unit 65 is a non-volatile storage device (for example, a hard disk device or a flash memory) configured to be able to read and write stored contents, and stores various processing programs and various data.

The control unit 67 is mainly configured by a known computer having at least a ROM, a RAM, and a CPU.
<About voice synthesis processing>
Next, speech synthesis processing executed by the control unit 67 of the speech output terminal 60 will be described.

Here, FIG. 10 is a flowchart showing the procedure of the speech synthesis process.
This voice synthesis process is started when a start command is input via the information receiving unit 61 of the voice output terminal 60.

  As shown in FIG. 10, when the speech synthesis process is started, first, information input via the information receiving unit 61 (hereinafter referred to as input information) is acquired (S510). The input information acquired in S510 includes, for example, output text indicating the content (word) of the sound output as synthesized sound, and output property information indicating the nature of the sound output as synthesized sound. Note that the sound property (ie, output property information) here refers to metadata such as the emotion when the speaker utters in addition to the features of the speaker's voice such as the gender of the speaker and the age of the speaker. It contains information prescribed as

  Subsequently, the facial expression table TD including information most similar to information such as emotion to be defined as metadata in the output property information acquired in S510 is extracted from the data storage server 50 (S520). Furthermore, the speech parameter corresponding to each syllable necessary for outputting the output word acquired in S510 as a synthesized sound and having information most similar to the voice feature in the output property information acquired in S510 PM is extracted from the data storage server 50 (S530).

  Then, the voice parameter PM acquired in S530 is set according to the facial expression table TD acquired in S520 so that the synthesized sound is output with the content of the output word acquired in S510 (S540). Subsequently, speech synthesis is performed based on the speech parameter PM set in S540 (S550). For the speech synthesis in S550, in addition to Patent Document 1, a known speech synthesis method using formant synthesis may be used.

Furthermore, the synthesized sound generated by the voice synthesis in S550 is output from the sound output unit 63 (S560).
Thereafter, the speech synthesis process ends.
[Effect of the first embodiment]
As described above, in the present embodiment, the voice input device 10 is constituted by a karaoke device. For this reason, the result (voice waveform) sung (spoken) by the user (that is, the speaker) of the voice input device 10 can be collected as the voice waveform data SV, and many users can sing. Various voice waveform data SV can be collected.

  The information processing apparatus 30 according to the present embodiment can derive various voice parameters PM uttered by various persons by deriving the voice parameters PM from the various voice waveform data SV of many speakers. That is, according to the information processing apparatus 30 of the present embodiment, the voice parameter PM can be diversified.

  Further, in the metadata estimation process of the voice parameter registration process in the present embodiment, the metadata can be automatically estimated based on the music data MD. For this reason, according to the speech parameter registration process of the present embodiment, unlike the speech synthesizer described in Patent Document 1, when the target music is sung, the voices as metadata are input to the speakers. The need can be eliminated.

  Then, in the voice output terminal 60 according to the present embodiment, the output inputted from the voice parameters PM of various speakers registered in the voice parameter registration process and the facial expression table TD generated in the voice analysis process. A facial expression table TD and a speech parameter PM that match the property information are extracted, and speech synthesis is performed so that the input output text is realized.

Therefore, according to the speech synthesis system 1 of the present embodiment, it is possible to diversify the features of a speaker who can be regarded as having synthesized speech.
In addition, in the metadata estimation process of this embodiment, the emotion of a singer who is highly likely to be represented by the key of each key in the target music is used as metadata. That is, according to the metadata estimation process of the present embodiment, it is possible to use the emotion of the speaker when the lyrics corresponding to the same key interval are uttered as metadata, and to ensure the key in the same key interval. Can be specified.
[Second Embodiment]
Next, a second embodiment of the present invention will be described.

The speech synthesis system of the second embodiment mainly differs from the speech synthesis system 1 of the first embodiment in the processing content of the metadata estimation process. For this reason, in the present embodiment, the same configurations and processes as those in the first embodiment are denoted by the same reference numerals, description thereof will be omitted, and description will be made focusing on metadata estimation processing different from that in the first embodiment. .
<About metadata estimation processing>
Here, FIG. 11 is a flowchart showing a processing procedure of the metadata estimation processing of the present embodiment.

  As shown in FIG. 11, when activated in S160 of the voice parameter registration process, the morphological analysis is performed on the lyrics expressed by the lyrics telop data DT included in the lyrics data group DL acquired in the previous S120. (S710). That is, in S710 of this embodiment, the character string which comprises a lyrics is divided | segmented for every word character which is the character string which comprises the word in the said lyrics by performing a morphological analysis. Note that the morphological analysis performed in S710 is a well-known process, and a detailed description thereof is omitted here.

  Subsequently, word property information is acquired for each word that is a result of the morphological analysis of S710 from a word metadata database (DB in the figure) 100 in which a word property table prepared in advance is stored (S720). However, the word property table referred to here is a table in which word property information representing the property of each word is associated with the identification information of the word, and the word property referred to here is the meaning of the word, the word, Including emotions represented by

Then, the word property information acquired in S720 is assigned as metadata to the section where the word is uttered (S730).
Thereafter, the metadata estimation process is terminated, and the process returns to the voice parameter registration process.

As described above, in the metadata estimation process of the present embodiment, as shown in FIG. 12, morphological analysis is performed on the lyrics of the target music, and the lyrics of the target music are character strings that constitute words. Divide by word character. In addition, word property information corresponding to each word is acquired from the word property information included in the word property table stored in the word metadata database 100 prepared in advance, and each of the word property information is obtained. , Metadata for the speech parameter PM of the corresponding syllable.
[Second Embodiment]
As described above, according to the metadata estimation process of the present embodiment, the meaning of the word uttered by the speaker and the emotion represented by the word can be used as metadata.
[Other Embodiments]
As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, In the range which does not deviate from the summary of this invention, it is possible to implement in various aspects.

  For example, in the voice synthesis process of the above embodiment, the voice input device 10 executes the karaoke performance process and generates the voice waveform data SV based on the voice input during the period in which the target music is played. The voice waveform data SV in the present invention is not limited to this.

  That is, in the present invention, the voice input data 10 may be generated in the voice input device 10 using a well-known after-recording function in a karaoke machine or the like. That is, in the case of a voice input device (karaoke device) having an after-recording function, dialogue telop data (that is, lyrics telop) representing characters constituting dialogue (hereinafter referred to as dialogue constituent characters) as data relating to dialogue to be uttered. Data) and dialogue output data that defines the timing for displaying dialogue constituent characters on the display unit 13 (that is, data similar to the lyrics output data).

  Therefore, when the speech waveform data SV is acquired using the after-recording function, the speech input device 10 displays the speech based on the speech telop data on the display unit 13 and performs speech during the period in which the speech is displayed on the display unit 13. A speech waveform input via the input unit 14 may be stored in the MIDI storage server 25 as speech waveform data SV.

  In this case, the information processing apparatus 30 may use the speech waveform data SV generated using the after-recording function as a processing target of the speech parameter registration process. That is, in the voice parameter registration process, after S110 is omitted, in S120, the dialogue telop data and dialogue output data are obtained, and in S140, based on the obtained dialogue telop data and dialogue output data, What is necessary is just to specify a syllable waveform. At this time, in S140, as a method of specifying the syllable waveform, the dialogue constituent characters in the speech waveform data SV are uttered at the timing when the dialogue constituent characters defined by the dialogue output data are displayed on the display unit 13. The syllable waveform may be specified as being present.

  Moreover, in the said embodiment, although the karaoke apparatus was assumed as the audio | voice input apparatus 10, the apparatus assumed as the audio | voice input apparatus 10 is not restricted to a karaoke apparatus, For example, a well-known portable terminal (a mobile phone or portable information) Terminal) or a known information processing apparatus (so-called personal computer) may be assumed.

  In the speech synthesis system of the above embodiment, the MIDI storage server 25 is provided. However, in the speech synthesis system of the present invention, the MIDI storage server 25 may not be provided. In this case, the music data MD and the voice waveform data SV may be stored in the storage unit 17 of the voice input device 10, may be stored in the data storage server 50, and further stored in the information processing device 30. It may be stored in the unit 34.

  Similarly, in the speech synthesis system of the above embodiment, the data storage server 50 is provided. However, in the speech synthesis system of the present invention, the data storage server 50 may not be provided. In this case, the voice parameter PM and the expression table TD may be stored in the storage unit 34 of the information processing device 30, may be stored in the storage unit 17 of the voice input device 10, or may be a MIDI storage server. 25 may be stored.

Note that the processing content of the speech synthesis processing executed by the speech output terminal 60 is not limited to the content described in the above embodiment. For example, the content of the speech synthesis process executed by the speech output terminal 60 may be only two steps of S310 and S360. However, in this case, it is necessary to execute each step of S320 to S350 in the information processing apparatus 30 or the like. That is, the speech synthesis process executed by the speech output terminal 60 is a process for acquiring input information and transmitting it to the information processing apparatus 30 and outputting a result of speech synthesis by the information processing apparatus 30 (ie, synthesized sound). The information processing apparatus 30 may execute the process of acquiring the facial expression table TD and the voice parameter PM and executing the voice synthesis so as to match the input information.
[Correspondence between Embodiment and Claims]
Finally, the relationship between the description of the above embodiment and the description of the scope of claims will be described.

  S110 in the voice parameter registration process of the above embodiment corresponds to the timing information acquisition means in the description of the claims, in particular, the score data acquisition means, and S120 corresponds to the content information acquisition means in the description of the claims. , S130 corresponds to the waveform acquisition means in the claims. Further, S140 in the parameter registration process corresponds to the syllable waveform extracting means in the description of the claims, S150 corresponds to the parameter deriving means in the description of the claims, and S160 is the description of the claims. S170 corresponds to the parameter registration means in the claims.

  Further, S310 in the metadata estimation process of the first embodiment corresponds to the section specifying means in the description of the claims, S320 corresponds to the main sound specifying means in the description of the claims, and S330, It corresponds to the pitch name frequency deriving means in the description of the claims, and S640 and S650 correspond to the key estimating means. Further, S710 in the metadata estimation process of the second embodiment corresponds to the word dividing means in the claims, and S720 corresponds to the metadata extracting means.

  Note that S510 in the speech synthesis process of the above embodiment corresponds to the output property information acquisition unit and the word acquisition unit in the description of the claims, and S520 corresponds to the table acquisition unit in the description of the claims. S540 and S550 correspond to the speech synthesis means in the description of the claims, and S560 corresponds to the output means in the description of the claims.

  DESCRIPTION OF SYMBOLS 1 ... Voice synthesis system 10 ... Voice input device 11 ... Communication part 12 ... Input reception part 13 ... Display part 14 ... Voice input part 15 ... Voice output part 16 ... Sound source module 17 ... Memory | storage part 20 ... Control part 21,41 ... ROM 22, 42 ... RAM 23, 43 ... CPU 25 ... MIDI storage server 30 ... Information processing device 31 ... Communication unit 32 ... Input reception unit 33 ... Display unit 34 ... Storage unit 40 ... Control unit 50 ... Data storage server 60 ... Audio output Terminal 61 ... Information receiving unit 62 ... Display unit 63 ... Sound output unit 64 ... Communication unit 65 ... Storage unit 67 ... Control unit 100 ... Word metadata database

Claims (3)

Content information acquisition means for acquiring utterance content information representing a character string of content to be uttered;
Timing information acquisition means for acquiring utterance timing information that specifies the utterance start timing of at least one character among the character string represented by the specific content information that is the utterance content information acquired by the content information acquisition means;
Waveform acquisition means for acquiring a target waveform which is a speech waveform uttered for the character string represented by the specific content information;
A speech waveform uttered to each syllable forming the character string represented by the specific content information from the target waveform acquired by the waveform acquisition unit based on at least the utterance timing information acquired by the timing information acquisition unit A syllable waveform extracting means for extracting a syllable waveform,
Parameter derivation means for deriving a speech parameter which is at least one feature quantity defined in advance from each syllable waveform extracted by the syllable waveform extraction means;
Based on the specific content information and the utterance timing information acquired by the timing information acquisition unit, the property of the voice represented by the syllable waveform is estimated, and a metadata generation unit that generates the estimation result as metadata;
One of the musical composition parameters and the parameter registration means for storing the speech parameter derived by the parameter deriving means and the metadata generated by the metadata generating means for each corresponding syllable and storing them in the first storage device A score data acquisition unit that represents a score of a target music piece and acquires score data in which at least a pitch and a performance start timing are defined for each output sound output from the sound module;
The content information acquisition means includes
Acquiring a character string of lyric constituting characters constituting the lyrics of the target music as the utterance content information,
The timing information acquisition means includes
The output timing for at least one of the lyrics constituent characters is acquired as the utterance timing information, the lyrics output timing associated with the performance start timing of the output sound corresponding to the lyrics constituent characters,
The waveform acquisition means includes
A waveform in which a voice input during the performance of the target music based on the score data has shifted along the time axis is acquired as the target waveform,
The syllable waveform extracting means includes
In the target waveform, a speech waveform in a section corresponding to each output sound is extracted as the syllable waveform,
The musical score data is
If transposing in the song of the target music , including a modulation flag indicating the position of the time to transpose on the time axis when the starting time of the performance of the target music is the origin ,
The metadata generation means includes
Based on the musical score data acquired by the musical score data acquiring means, section specifying means for specifying the same key section, which is each section in which the same key is continued in the target music,
A main sound specifying means for specifying the last output sound along the time axis in each key same section as the main sound included in each key same section specified by the section specifying means;
An appearance pitch name frequency representing a frequency of output sounds of the same pitch name included in the same pitch section specified by the section specifying means is set for each key same section starting from the pitch name of the main sound specified by the main sound specifying means. A pitch name deriving means for deriving;
As a result of comparing each appearance name frequency derived by the sound name frequency deriving means with a key template prepared in advance for each key as a template representing the distribution of pitch names that can be used in each key, each key having the highest correlation is obtained. And a key estimation means using the property of speech corresponding to the metadata as the metadata,
The key estimation means includes
If the key with the highest correlation is a major key, the sound property of bright is the metadata, and if the key with the highest correlation is a minor key, the property of the voice is dark is the metadata. Parameter extraction device to do. The content information acquisition means includes
A sentence composing character that is a character string constituting at least one sentence is acquired as the utterance content information;
The timing information acquisition means includes
Information indicating the output timing of outputting at least one character constituting the sentence constituent character to the outside is acquired as the utterance timing information,
The waveform acquisition means includes
According to claim 1, characterized in that to obtain the waveform voice input in the output of the character string constituting the sentence structure characters based on the utterance contents information has remained along the time axis as the target waveform Parameter extraction device. Content information acquisition means for acquiring utterance content information representing a character string of content to be uttered;
Timing information acquisition means for acquiring utterance timing information that specifies the utterance start timing of at least one character among the character string represented by the specific content information that is the utterance content information acquired by the content information acquisition means;
Waveform acquisition means for acquiring a target waveform which is a speech waveform uttered for the character string represented by the specific content information;
A speech waveform uttered to each syllable forming the character string represented by the specific content information from the target waveform acquired by the waveform acquisition unit based on at least the utterance timing information acquired by the timing information acquisition unit A syllable waveform extracting means for extracting a syllable waveform,
Parameter derivation means for deriving a speech parameter which is at least one feature quantity defined in advance from each syllable waveform extracted by the syllable waveform extraction means;
Based on the specific content information and the utterance timing information acquired by the timing information acquisition unit, the property of the voice represented by the syllable waveform is estimated, and a metadata generation unit that generates the estimation result as metadata;
A parameter registration unit that associates the speech parameter derived by the parameter deriving unit with the metadata generated by the metadata generation unit for each corresponding syllable and stores it in the first storage device;
The audio parameters stored in the first storage device are analyzed for each metadata associated with the audio parameters, and a metadata correspondence table representing a range of each audio parameter corresponding to the metadata is generated. A parameter extraction device having parameter analysis means for storing in the second storage device,
An output property information acquisition means for acquiring output property information that is input from the outside and represents a sound property;
A word acquisition means for acquiring an output word representing a word input from the outside;
The metadata correspondence table including the metadata corresponding to the output property information acquired by the output property information acquisition means is acquired from the second storage device, and an audio parameter having information corresponding to the output property information is Table acquisition means for acquiring from the first storage device;
Voice synthesis means for synthesizing the voice parameters acquired by the table acquisition means according to the metadata correspondence table so as to be the output text acquired by the word acquisition means;
A synthesized sound output device comprising: output means for outputting a synthesized sound generated by voice synthesis by the voice synthesis means;
The parameter extraction device includes:
Representing the score of the target music that is one of the music, for each output sound output from the sound source module, comprising score data acquisition means for acquiring score data in which at least the pitch and the performance start timing are defined,
The content information acquisition means includes
Acquiring a character string of lyric constituting characters constituting the lyrics of the target music as the utterance content information,
The timing information acquisition means includes
The output timing for at least one of the lyrics constituent characters is acquired as the utterance timing information, the lyrics output timing associated with the performance start timing of the output sound corresponding to the lyrics constituent characters,
The waveform acquisition means includes
A waveform in which a voice input during the performance of the target music based on the score data has shifted along the time axis is acquired as the target waveform,
The syllable waveform extracting means includes
In the target waveform, a speech waveform in a section corresponding to each output sound is extracted as the syllable waveform,
The musical score data is
If transposing in the song of the target music , including a modulation flag indicating the position of the time to transpose on the time axis when the starting time of the performance of the target music is the origin ,
The metadata generation means includes
Based on the musical score data acquired by the musical score data acquiring means, section specifying means for specifying the same key section, which is each section in which the same key is continued in the target music,
A main sound specifying means for specifying the last output sound along the time axis in each key same section as the main sound included in each key same section specified by the section specifying means;
An appearance pitch name frequency representing a frequency of output sounds of the same pitch name included in the same pitch section specified by the section specifying means is set for each key same section starting from the pitch name of the main sound specified by the main sound specifying means. A pitch name deriving means for deriving;
As a result of comparing each appearance name frequency derived by the sound name frequency deriving means with a key template prepared in advance for each key as a template representing the distribution of pitch names that can be used in each key, each key having the highest correlation is obtained. And a key estimation means using the property of speech corresponding to the metadata as the metadata,
The key estimation means includes
If the key with the highest correlation is a major key, the sound property of bright is the metadata, and if the key with the highest correlation is a minor key, the property of the voice is dark is the metadata. A speech synthesis system.

Images