JP6252420B2 - Speech synthesis apparatus and speech synthesis system - Google Patents

Description

  The present invention relates to a technique for generating synthesized speech.

  Conventionally, a portable terminal having a voice synthesis function for performing voice synthesis based on voice data prepared in advance is known (see Patent Document 1). In the portable terminal described in Patent Document 1, speech synthesis is performed so that the voice is similar to that of a specific person in accordance with the received voice signal.

JP 2010-166324 A

  By the way, in a karaoke apparatus, it is calculated | required to produce | generate and output the exemplary vocal which sang appropriately the song melody of the designated music by speech synthesis. It is preferable that this exemplary vocal reproduces the characteristics of the user's way of singing with sound similar to the voice quality of the user of the karaoke apparatus.

  Singing features are unique to singing and are therefore rarely expressed in conversation. For this reason, even if the technique described in Patent Document 1 for synthesizing speech for normal conversation is used for generating model vocals, it is similar to the voice quality of the user and imitates the characteristics of the user's singing method. There was a problem that it was difficult to generate the model vocal.

That is, the conventional technique has a problem that it is difficult to synthesize a voice that is similar to the voice quality of the user and imitates the characteristics of the user's way of singing.
Therefore, an object of the present invention is to provide a technique for synthesizing a singing voice that resembles the voice quality of a user and imitates the characteristics of the user's singing method.

The present invention, which has been made to achieve the above object, relates to a speech synthesizer comprising speech data acquisition means, analysis means, search means, and synthesis means.
The voice data acquisition means in the present invention acquires singing voice data of singing a song in which lyrics are assigned to at least a part of a plurality of notes consisting of combinations of pitches and note values. The analysis means is an input voice quality representing a voice quality feature amount of the singing voice data acquired by the voice data acquisition means, and an interval and at least one of the amplitude and the fundamental frequency of the singing voice data within a musical note interval constituting the music piece. An input singing time representing a transition (singing, hereinafter referred to as a singing time) is derived.

  And the search means is similar to the input voice quality and the input song time derived by the analysis means in the voice quality song data indicating the voice quality information and song time information stored in the first storage device. The singer identification information included in the voice quality singing data including the voice quality feature value and the singing time feature value satisfying the stipulated condition including being equal to or higher than a predetermined reference value is specified. The voice quality song data stored in the first storage device is a voice quality feature quantity that is a voice quality feature quantity in the sound source voice data, and the sound source voice data of at least one of the amplitude and the fundamental frequency of the sound source voice data. This is data in which the singing feature amount representing the transition in the note-corresponding section corresponding to the note of the song is associated with the singer identification information for identifying the person who has spoken. In addition, the sound source voice data referred to here is a voice waveform that utters lyrics assigned to at least a part of a plurality of notes composed of a combination of pitch and note value, and is different from each other. is there.

  Further, the synthesizing means in the present invention acquires sound source data including specific identification information that is the singer identification information specified by the search means from the sound source data stored in the second storage device, and the acquired sound source data In accordance with the sound source data included in the sound and the singing voice data acquired by the voice data acquisition means, a singing voice singing the specified music that is the specified music is generated and output by voice synthesis. The sound source data stored in the second storage device is data in which sound source sound data is associated with each singer identification information.

  The singing voice data is a voice that sang a part of the music piece, and the singing voice data alone is not sufficient as the data amount of the sound source necessary for executing the voice synthesis.

  Therefore, in the speech synthesizer of the present invention, the singing voice data is analyzed, and the voice quality and singing characteristics of the person who performed the singing for generating the singing voice data (ie, the user) are specified. Then, sound source data generated from the voice of another person who is similar to the voice quality and singing characteristics of the specified user and different from the user is specified, and singing voice is generated from the specified sound source data. Used as at least part of a sound source for speech synthesis.

  According to such a speech synthesizer of the present invention, the voice of the user himself and the voice of another person whose characteristics are similar to the voice of the user are used to sing the singing voice singing the specified music. Speech synthesis is possible.

As a result, according to the speech synthesizer of the present invention, it is possible to synthesize a singing voice that resembles the voice quality of the user and imitates the feature of the user's singing.
The present invention may be implemented as a speech synthesis system including speech data acquisition means, analysis means, search means, and synthesis means.

According to such a speech synthesis system, an effect similar to that of the speech synthesis apparatus according to claim 1 can be obtained.
Furthermore, the search means in this invention may specify specific identification information as satisfying a prescription | regulation condition that it is from a thing with the highest similarity to the prescription | regulation number prescribed | regulated previously.

According to such a speech synthesis system, it is possible to specify a specified number of other persons having high voice quality and high similarity in singing characteristics.
In the present invention, the notes sung by the singing voice data acquired by the voice data acquisition means among the notes that constitute the specified music and the lyrics are assigned are singing notes, the specified music is configured and the lyrics are Of the assigned notes, notes other than singing notes may be non-singing notes.

  And the synthesizing means in the present invention generates the singing voice of the lyrics assigned to the singing note by synthesizing the voice based on the singing voice data acquired by the voice data acquiring means, specified by the searching means, and Singing voices of lyrics assigned to non-singing notes may be generated by performing voice synthesis based on sound source voice data associated with the specific identification information.

  According to such a speech synthesis system, a sound source necessary for speech synthesis can be specified for each note constituting the designated music and assigned lyrics, and speech synthesis can be performed using the sound source for each specified note. . As a result, according to the speech synthesis system of the present invention, it is possible to generate, by speech synthesis, a singing voice that resembles the voice quality of the user and more accurately imitates the characteristics of the user's singing method.

Note that the speech synthesis system of the present invention may include an acquisition unit, an extraction unit, an identification unit, a first derivation unit, a second derivation unit, a generation unit, and a storage control unit.
The obtaining means obtains music data including at least the voice waveform of the performance sound of the music including the vocal sound and the identification information representing the speaker of the vocal sound as the singer identification information. The extracting means extracts vocal sounds included in the music data acquired by the acquiring means as sound source sound data.

  Further, the specifying unit specifies a note vocal which is a section of the sound source sound data corresponding to each note corresponding section among the sound source sound data extracted by the extracting unit. Then, the first deriving unit derives a transition in the note-corresponding section of at least one of the amplitude and the fundamental frequency of the note vocal specified by the specifying unit as a song feature amount. The second deriving unit derives a voice quality feature amount for each note vocal for each note vocal specified by the specifying unit, and derives a representative value of the voice quality feature amount as a voice quality feature amount.

  The generation means generates voice quality song data by associating the song feature quantity derived by the first derivation means, the voice quality feature quantity derived by the second derivation means, and the singer identification information. The storage control means stores the voice quality song data generated by the generation means in the first storage device.

  According to such a speech synthesis system, voice quality song data can be generated and stored in the first storage device.

1 is a block diagram showing a schematic configuration of a karaoke system as a speech synthesis system to which the present invention is applied. It is a flowchart which shows the process sequence of a sound source data generation process. It is explanatory drawing explaining the outline | summary of a song time feature-value, (A) is a figure explaining the outline | summary of a song amplitude vector, (B) is the figure explaining the outline | summary of a song pitch vector. It is a figure which shows the outline | summary of voice quality song times data. It is a flowchart which shows the process sequence of a speech synthesis process.

Embodiments of the present invention will be described below with reference to the drawings.
<Speech synthesis system>
A speech synthesis system 1 shown in FIG. 1 is a system that generates and outputs a synthesized speech singing a song designated by a user (hereinafter referred to as a designated song) with a voice similar to the user.

In order to realize this, the speech synthesis system 1 includes an information processing device 2, an information processing server 10, and a karaoke device 30.
The information processing device 2 generates sound source data SD necessary for generating synthesized speech (that is, speech synthesis) based on speech waveform data WD including speech uttered by a person and MIDI music MD representing the uttered content. .

The information processing server 10 stores at least sound source data SD and MIDI music MD generated by the information processing apparatus 2.
The karaoke apparatus 30 plays the MIDI music MD stored in the information processing server 10, and generates and outputs a synthesized voice in which the music corresponding to the MIDI music MD is sung according to the sound source data SD. Note that the speech synthesis system 1 includes a plurality of karaoke apparatuses 30.
<Audio waveform data>
The audio waveform data WD is audio data representing a performance sound of playing a music, and is associated with music management information in which information related to the music is described. The music management information includes music identification information (hereinafter referred to as music ID) for identifying music.

  The audio waveform data WD of the present embodiment includes, as performance sounds, accompaniment sounds that are played by at least one musical instrument and singing sounds that are sung by humans. Note that the voice waveform data WD differs in the sung person or song (lyric) for each voice waveform data WD.

  The audio waveform data WD may be data configured by an audio file in an uncompressed audio file format, or may be data configured by an audio file in an audio compression format. The voice waveform data WD may be generated by recording voice when the user sings a song, or may be generated by other methods.

The audio waveform data WD in the present embodiment is an example of sound source audio data described in the claims.
<MIDI music>
The MIDI music MD is prepared in advance for each music and has music data and lyrics data.

  Among these, the music data is data representing the score of one music according to the well-known MIDI (Musical Instrument Digital Interface) standard. This music data has at least a music ID and a music score track representing a music score for each musical instrument used in the music.

  The musical score track defines at least the pitch (so-called note number) and the period during which the MIDI sound source outputs the performance sound (hereinafter referred to as the note length) for each performance sound output from the MIDI sound source. Has been. The note length in the score track is the performance start timing (so-called note-on timing) indicating the time from the start of the performance of the music until the output of the performance sound and the music until the output of the performance sound ends. Performance end timing (so-called note-off timing) representing the time from the start of the performance.

  That is, in the score track, one note NO is defined by the note number and the note length represented by the note-on timing and note-off timing. The musical score track functions as one musical score by arranging note NO in the order of performance. Note that the musical score track is prepared for each instrument such as a keyboard instrument, a stringed instrument, a percussion instrument, and a wind instrument, for example. Among these, in this embodiment, a specific musical instrument (for example, vibraphone) is defined as a musical instrument responsible for singing melody in music.

  On the other hand, the lyrics data is data relating to the lyrics of the music, and includes lyrics telop data and lyrics output data. The lyrics telop data represents characters that constitute the lyrics of the music (hereinafter referred to as lyrics component characters). The lyrics output data is data in which a timing correspondence relationship that associates the lyrics output timing, which is the output timing of the lyrics constituent characters, with the performance of the music data is defined.

Specifically, in the timing correspondence relationship in the present embodiment, the timing for starting the output of the lyrics telop data is associated with the timing for starting the performance of the music data. Furthermore, in the timing correspondence relationship, the lyrics output timing of each lyrics constituent character along the time axis of the music is defined by the elapsed time from the start of performance of the music data. Thereby, each performance sound (namely, note NO) prescribed | regulated to the score track | truck and each lyric component character are matched.
<Information processing device>
The information processing apparatus 2 is a known information processing apparatus (for example, a personal computer) including an input receiving unit 3, an external output unit 4, a storage unit 5, and a control unit 6.

  The input receiving unit 3 is an input device that receives input of information and commands from the outside. The input device here is, for example, a key or switch, a reading drive for reading data stored in a portable storage medium (for example, CD, DVD, flash memory), or a communication port for acquiring information via a communication network. Etc. The external output unit 4 is an output device that outputs information to the outside. Here, the output device is a writing drive that writes data to a portable storage medium, a communication port that outputs information to a communication network, or the like.

  The storage unit 5 is a known storage device configured to be able to read and write stored contents. The storage unit 5 stores at least two or more speech waveform data WD in association with a MIDI music piece MD representing the utterance content in the speech waveform data WD. In addition, the code | symbol "l" in FIG. 1 is an identifier which identifies the audio | voice waveform data WD, and is allocated for every user and every music which the said user sang. This code “l” is a natural number of 2 or more. Further, the symbol “o” in FIG. 1 is an identifier for identifying the MIDI music piece MD, and is assigned to each music piece. This code “o” is a natural number of 2 or more.

  The control unit 6 is a known control device that is configured around a known microcomputer including a ROM 7, a RAM 8, and a CPU 9. The ROM 7 stores processing programs and data that need to retain stored contents even when the power is turned off. The RAM 8 temporarily stores processing programs and data. The CPU 9 executes each process according to a processing program stored in the ROM 7 or RAM 8.

The ROM 7 of the present embodiment has a processing program for the control unit 6 to execute sound source data generation processing for generating sound source data SD based on the audio waveform data WD and the MIDI music piece MD stored in the storage unit 5. It is remembered.
<Information processing server>
The information processing server 10 includes a communication unit 12, a storage unit 14, and a control unit 16.

  Among these, the communication unit 12 performs communication between the information processing server 10 and the outside via a communication network. That is, the information processing server 10 is connected to the karaoke apparatus 30 via a communication network. The communication network referred to here may be a wired communication network or a wireless communication network.

  The storage unit 14 is a known storage device configured to be able to read and write stored contents. The storage unit 14 stores at least a plurality of MIDI music pieces MD. 1 is an identifier for identifying the MIDI music piece MD stored in the storage unit 14 of the information processing server 10, and is assigned to each music piece. This code “n” is a natural number of 1 or more. Further, the storage unit 14 stores sound source data SD generated by the information processing apparatus 2 executing sound source data generation processing. The code “m” shown in FIG. 1 is an identifier for identifying the sound source data SD stored in the storage unit 14 of the information processing server 10, and is assigned to each group to be described in detail later. This code “m” is a natural number of 2 or more.

The control unit 16 is a known control device that is configured around a known microcomputer including a ROM 18, a RAM 20, and a CPU 22. The ROM 18 stores processing programs and data that need to retain stored contents even when the power is turned off. The RAM 20 temporarily stores processing programs and data. The CPU 22 executes each process according to a processing program stored in the ROM 18 or the RAM 20.
<Karaoke equipment>
The karaoke apparatus 30 includes a communication unit 32, an input reception unit 34, a music playback unit 36, a storage unit 38, an audio control unit 40, a video control unit 46, and a control unit 50.

  In the communication unit 32, the karaoke apparatus 30 communicates with the outside via a communication network. The input receiving unit 34 is an input device that receives input of information and commands in accordance with external operations. Here, the input device is, for example, a key, a switch, a reception unit of a remote controller, or the like.

  The music playback unit 36 performs a music performance based on the MIDI music MD downloaded from the information processing server 10. The music reproducing unit 36 is, for example, a MIDI sound source. The voice control unit 40 is a device that controls voice input / output, and includes an output unit 42 and a microphone input unit 44.

  A microphone 62 is connected to the microphone input unit 44. As a result, the microphone input unit 44 acquires the sound input via the microphone 62. A speaker 60 is connected to the output unit 42. The output unit 42 outputs the sound source signal of the music reproduced by the music reproducing unit 36 and the sound source signal of the singing sound from the microphone input unit 44 to the speaker 60. The speaker 60 outputs the sound source signal output from the output unit 42 instead of sound.

  The video control unit 46 outputs a video or an image based on the video data sent from the control unit 50. The video control unit 46 is connected to a display unit 64 that displays video or images.

  The control unit 50 is configured around a known computer having at least a ROM 52, a RAM 54, and a CPU 56. The ROM 52 stores processing programs and data that need to retain stored contents even when the power is turned off. The RAM 54 temporarily stores processing programs and data. The CPU 56 executes each process according to a processing program stored in the ROM 52 or the RAM 54.

The ROM 52 of the present embodiment stores a processing program for the control unit 50 to execute speech synthesis processing. The voice synthesizing process is a process of generating and outputting a synthesized voice in which a user's voice and voice similar to the voice of the user are sung on a song designated by the user.
<Sound source data generation processing>
A sound source data generation process executed by the control unit 6 of the information processing apparatus 2 will be described.

  As shown in FIG. 2, when the sound source data generation process is activated, the control unit 6 acquires a MIDI music piece MD including the music piece ID designated via the input receiving unit 3 (S110). Subsequently, the control unit 6 acquires one audio waveform data WD associated with the music ID acquired in S110 from all the audio waveform data WD stored in the storage unit 5 (S120). .

  In the sound source data generation process, the control unit 6 suppresses the accompaniment sound included in the speech waveform data WD acquired in S120 (S130). In the present embodiment, a known method may be used as a method for suppressing the accompaniment sound. The accompaniment sound suppression method in the present embodiment may be a method of emphasizing the singing sound included in the audio waveform data WD, or removing the performance sound of the musical instrument represented by the MIDI music piece MD from the audio waveform data WD. It may be a technique.

  Further, in the sound source data generation process, the control unit 6 specifies the note vocal Vo (a, i) based on the voice waveform data WD in which the accompaniment sound is suppressed in S130 and the MIDI music piece MD acquired in S110. (S140). The note vocal Vo (a, i) is a section corresponding to each note NO (a, i) that constitutes a singing melody and is assigned lyrics in the voice waveform data WD. In S140, the control unit 6 collates the voice waveform data WD acquired in S120 with the performance start timing nnt (a, i) and the performance end timing nft (a, i) in the MIDI music piece MD, so that a musical note vocal is obtained. Specify Vo (a, i).

The code “a” in the present embodiment is a code that identifies a music piece, and the code “i” is a code that identifies a note NO of a singing melody in the music piece.
Further, in the sound source data generation process, the control unit 6 sets a plurality of analysis windows for each of the note vocals Vo (a, i) (S150). In setting the analysis window in S150, the control unit 6 sets the plurality of analysis windows so as to be adjacent to each other along the time axis. This analysis window is a section having a time length shorter than the time length of the note NO (a, i).

  Subsequently, in the sound source data generation process, the control unit 6 performs the singing amplitude vector A (a, i) representing the transition in the section corresponding to the note NO (a, i) of the amplitude in the note vocal Vo (a, i). ) Is calculated (S160). In S160, the control unit 6 first calculates the amplitude value of the note vocal Vo (a, i) in each analysis window set in S150, as shown in FIG. And the control part 6 arrange | positions the amplitude value calculated for every those analysis windows along a time-axis, produces | generates the arrangement | sequence of an amplitude value, and arrange | positions the arrangement | sequence of the amplitude value to singing amplitude vector A (a, a, Calculate as i). The amplitude for each analysis window calculated in S160 may be, for example, an arithmetic average of the discrete values of the note vocal Vo (a, i) in each analysis window.

  Further, in the sound source data generation process, the control unit 6 sings a pitch vector F (a, i) representing the transition in the section corresponding to the note NO (a, i) of the fundamental frequency in the note vocal Vo (a, i). i) is calculated (S170). In S170, the control unit 6 first calculates a fundamental frequency f0 of the note vocal Vo (a, i) in each analysis window set in S150, as shown in FIG. 3B. And the control part 6 produces | generates the arrangement | sequence of the fundamental frequency f0 by arrange | positioning the fundamental frequency f0 calculated for every those analysis windows along a time-axis, and the arrangement | sequence of the fundamental frequency f0 is used as a song pitch vector. Calculated as F (a, i). Various known methods can be considered as a method of calculating the fundamental frequency f0 in the present embodiment. As an example, in S170, the control unit 6 performs frequency analysis (for example, DFT) for each analysis window set in the note vocal Vo (a, i), and uses the strongest frequency component as a result of autocorrelation. The frequency f0 can be considered.

  At least one of the singing amplitude vector A calculated in S160 and the singing pitch vector F calculated in S170 is an example of the singing feature amount described in the claims.

  Subsequently, in the sound source data generation process, the control unit 6 calculates a voice quality feature amount M (a, i) in each note vocal Vo (a, i) (S180). The voice quality feature amount M referred to here is a feature amount representing the voice quality of the person who uttered the sound represented by the speech waveform data WD acquired in S120. In S180, the control unit 6 first performs frequency analysis (for example, DFT) for each analysis window of the note vocal Vo (a, i) set in S150. The control unit 6 calculates a mel frequency cepstrum (MFCC) of each analysis window as a voice quality feature amount M (a, i) by performing cepstrum analysis on the result of frequency analysis (frequency spectrum).

  In the sound source data generation process, the control unit 6 specifies the note property p (a, i) of the note NO (a, i) corresponding to the note vocal Vo (a, i) acquired in S120 (S190). ). In S190 of the present embodiment, specifically, the control unit 6 uses, as a note property p (a, i), information on each note NO (a, i) defined in the MIDI song MD from the MIDI song MD. Extract and identify.

  The note property p (a, i) mentioned here includes a target note attribute, a previous note attribute, and a rear note attribute. The target note attribute is information representing the attribute of the note NO (a, i). The target note attributes include the scale (pitch) of note NO (a, i), note length, and syllable of lyrics. The previous note attribute is information representing the attribute of the note NO (a, i-1) immediately preceding the note NO (a, i) (hereinafter referred to as the previous note) along the time axis. The previous note attribute includes the scale (pitch), note length, syllable of the previous note NO (a, i-1), and the previous note NO (a, i-1) and note NO (a, i). Including the length of time between.

  Further, the back note attribute is information representing the attribute of the note (hereinafter referred to as a back note) NO (a, i + 1) immediately after the target note NO (a, i) along the time axis. The post-note attributes include a musical scale (pitch), a note length, a syllable of lyrics, and a time length between the note NO (a, i) and the subsequent note NO (a, i + 1). Note that the note length in the note property p (a, i) and the time length between notes may be quantized to a predetermined class.

  In the sound source data generation process, the control unit 6 calculates the singing amplitude vector A (a, i) calculated in S160, the singing pitch vector F (a, i) calculated in S170, and S180. Temporal song data T (a, i) in which the voice quality feature value M (a, i) is associated with the note property p (a, i) is generated (S200).

  Subsequently, in the sound source data generation processing, the control unit 6 determines whether or not the processing from S120 to S190 has been executed for all the audio waveform data WD associated with the MIDI music piece MD acquired in S110. (S210). If the result of determination in S210 is that processing has not been executed for all audio waveform data WD (S210: NO), the controller 6 returns the sound source data generation processing to S120. In S120, one audio waveform data WD is acquired from the audio waveform data WD that is associated with the MIDI musical piece MD acquired in S110 and that has not been subjected to the processing from S120 to S200. Thereafter, the control unit 6 executes steps from S130 to S200.

  On the other hand, if the result of determination in S210 is that processing has been executed for all audio waveform data WD (S220: YES), the controller 6 shifts the sound source data generation processing to S220. In S220, the control unit 6 calculates representative values of the singing amplitude vector A, the singing pitch vector F, and the voice quality feature amount M for each group having the common note property p.

  Specifically, in S220 of the present embodiment, the control unit 6 acquires the provisional song data T having the same note property p among all the provisional song data T. Then, representative values of the singing amplitude vector A, the singing pitch vector F, and the voice quality feature amount M included in the acquired provisional song data T are calculated. The representative value referred to here may be an arithmetic mean result, a median value, or a mode value.

  In the sound source data generation process, the control unit 6 generates voice quality song data V and stores it in the storage unit 5 (S230). As shown in FIG. 4, the voice quality song data V generated in S230 includes, for each note property p, the note property p, the representative value of the song amplitude vector A calculated in S220, and the singing pitch. This is data in which a representative value of the vector F, a representative value of the voice quality feature amount M, and singer identification information for identifying a singer (hereinafter referred to as “singer ID”) are associated with each other.

  Further, in the sound source data generation process, the control unit 6 generates sound source data SD (S240). In S240, the control unit 6 generates the sound source data SD by associating each note vocal Vo with the note property p corresponding to the sound represented by the note vocal Vo and the singer ID.

  Subsequently, in the sound source data generation process, the control unit 6 determines whether or not all the MIDI music pieces MD stored in the storage unit 5 have been acquired (S250). As a result of the determination in S250, if the steps from S110 to S240 have not been executed for all the MIDI music pieces MD (S250: NO), the control unit 6 returns the sound source data generation processing to S110. In S110, the control unit 6 acquires one MIDI music MD from the MIDI music MD that has not executed the steps from S110 to S240. Thereafter, in the sound source data generation process, S120 to S240 are repeated.

  By the way, as a result of the determination in S250, if the steps from S110 to S240 have been executed for all the MIDI music pieces MD (S250: YES), the control unit 6 ends the sound source data generation process and starts up. Wait until the command is input.

  As described above, in the sound source data generation process, at least one of the voice quality feature amount M representing the voice quality of the voice waveform data, the amplitude of the voice waveform data, and the fundamental frequency according to the result of analyzing the voice waveform data WD prepared in advance. A voice quality song by associating a song feature (ie, song amplitude vector A, song pitch vector F), note property p, and singer ID representing the transition of notes constituting one piece of music. Data V is generated. In the sound source data generation process, sound source data SD is generated by associating each note vocal Vo, the note property p corresponding to each note vocal Vo, and the singer ID.

The voice quality song data V and the sound source data SD generated by the control unit 6 of the information processing apparatus 2 executing the sound source data generation process are stored in the storage unit 14 of the information processing server 10 using a portable storage medium. May be remembered. When the information processing device 2 and the information processing server 10 are connected via a communication network, the voice quality song data V and the sound source data SD stored in the storage unit 5 of the information processing device 2 are stored in the communication network. May be stored in the storage unit 14 of the information processing server 10.
<Speech synthesis processing>
Next, the speech synthesis process executed by the control unit 50 of the karaoke apparatus 30 will be described.

As shown in FIG. 5, when the speech synthesis process is activated, the control unit 50 acquires a song ID corresponding to a song (designated song) designated via the input receiving unit 34 (S510).
In the voice synthesis process, subsequently, the control unit 50 acquires singing voice data representing the waveform of the voice input via the microphone 62 connected to the microphone input unit 44 (S520). The singing voice data acquired in S520 is a voice sung by the user of the karaoke apparatus in a part of the specified music piece.

Then, the control unit 50 analyzes the voice quality of the singing voice data acquired in S520, and calculates the input voice quality Yk that represents the feature quantity of the voice quality of the singing voice data (S530).
The input voice quality Yk is a voice quality feature value representing a mel frequency cepstrum (MFCC) for each vowel. The method for calculating the input voice quality Yk is the same as S150 and S180 in the sound source data generation process except that “note vocal Vo” is replaced with “singing voice data”, and thus detailed description thereof is omitted here. To do.

  Subsequently, in the speech synthesis process, the control unit 50 derives an input singing time representing a transition in a musical note section constituting at least one of the amplitude and the fundamental frequency of the singing voice data (S540). This input singing time is the singing amplitude vector A and the singing pitch vector F in the singing voice data. The calculation method of the input song times is the same as S150 to S170 in the sound source data generation process except that “note vocal Vo” is replaced with “singing voice data”, and detailed description thereof is omitted here. To do.

  Further, in the speech synthesis process, the control unit 50 has a specified condition including that the similarity between the input voice quality Yk calculated in S530 and the input song time calculated in S540 is equal to or higher than a predetermined reference value. The singer ID included in the voice quality song time data including the voice quality feature value and the song time feature value to be satisfied is specified (S550). In S550, the control unit 50 calculates the correlation coefficient between the input singing time and the singing time feature amount as the singing time similarity. Further, in S550, the control unit 50 calculates the correlation coefficient between the input voice quality Yk and the voice quality feature amount M as the voice quality similarity. And the control part 50 specifies singer ID contained in the voice quality song time data in which both a song time similarity degree and a voice quality similarity degree become more than a reference value. In S550 in the present embodiment, the singer IDs included in the voice quality song data for a predetermined number (the specified number is an integer equal to or greater than “1”) in order from the highest similarity are listed. Identify.

  Further, in the speech synthesis process, the control unit 50 acquires the MIDI music MD corresponding to the music ID acquired in S510 from the information processing server 10 (S560). Subsequently, in the speech synthesis process, the control unit 50 analyzes the MIDI music piece MD acquired in S560 (S570). In the analysis of the MIDI musical piece MD in S570, the control unit 50 specifies the note property p of the melody note NO constituting the song melody of the designated musical piece as the synthesis target information according to the arrangement order of each melody note NO.

In the speech synthesis process, the control unit 50 generates and outputs a synthesized speech singing the singing melody according to the synthesis target information specified in S570 (S580).
Specifically, in S580 of the present embodiment, the control unit 50 acquires sound source data SD including the singer ID specified in S550 if the predetermined setting conditions are satisfied. And the control part 50 produces | generates the singing voice which sang the lyric assigned to the note which should produce | generate a synthetic | combination voice by speech synthesis | combination according to the note vocal Vo contained in the acquired sound source data SD at the present time. The setting condition referred to here is that the note property p of the note for which the synthesized speech is to be generated at the present time does not match the note property p of the note issued to the speech represented by the singing speech data acquired in S520. is there.

  In S280 when the setting condition is satisfied, the control unit 50 includes the singer ID specified in S550 and the sound source data associated with the note property p of the note for which the synthesized speech is to be generated at the present time. The SD is searched in order from the highest similarity. In this search, if there is no sound source data SD that includes the singer ID having the highest similarity and is currently assigned to the note for which the synthesized speech is to be generated, the singer ID having the next highest similarity is selected. The sound source data SD that is included and assigned to the note for which the synthesized speech is to be generated is searched.

  On the other hand, in S580 of the present embodiment, if the setting condition is not satisfied, the control unit 50 sings the lyrics assigned to the notes that should be generated at this time according to the singing voice data acquired in S520. Generate speech by speech synthesis.

  That is, in the present embodiment, among the notes constituting the designated music and assigned lyrics, the notes sung by the singing voice data acquired in S520 are referred to as singing notes, the designated music is constituted and the lyrics Suppose that notes other than the singing note are referred to as non-singing notes among the notes to which is assigned.

  In this case, in S580 in the present embodiment, the control unit 50 generates a singing voice of the lyrics assigned to the singing note by performing voice synthesis based on the singing voice data acquired in S520. Moreover, the control part 50 produces | generates the singing voice | voice of the lyrics allocated to the non-singing musical note by carrying out voice synthesis | combination based on the sound source data SD containing the singer ID acquired in S550.

  Note that the speech synthesis in the present embodiment may be realized by so-called formant synthesis. That is, in S580 of the present embodiment, the control unit 50 calculates the fundamental frequency (f0), mel frequency cepstrum (MFCC), and power in each syllable from the note vocal Vo and singing voice data included in the sound source data SD. Thus, it may be used for speech synthesis (formant synthesis).

  Subsequently, the control unit 50 outputs the synthesized speech generated by synthesizing the speech in S580 to the output unit 42 (S590). The output unit 42 emits synthesized speech from the speaker 60.

Thereafter, the control unit 50 ends the speech synthesis process.
[Effect of the embodiment]
As described above, in the speech synthesis process of the present embodiment, the voice quality of the person (that is, the user) who performed the singing for analyzing the singing voice data and generating the singing voice data is characterized. Is identified. Then, sound source data generated from the voice of another person who is similar to the voice quality and singing characteristics of the specified user and different from the user is specified, and singing voice is generated from the specified sound source data. Used as at least part of a sound source for speech synthesis.

  According to such a voice synthesis process, voice synthesis is performed on a singing voice that sings a specified musical piece using the voice of the user himself and the voice of another person whose characteristics are similar to the voice of the user. can do.

  Further, in the speech synthesis process of the present embodiment, if the singing voice data acquired in S520 is for the synthesis target note, the voice of the lyrics assigned to the singing note is synthesized by voice synthesis based on the singing voice data. Generate singing voice. On the other hand, if the singing voice data acquired in S520 is not for the synthesis target note, it is assigned to a non-singing note by performing voice synthesis based on the sound source voice data associated with the singer ID acquired in S550. Generate singing voice of the lyrics.

According to such a speech synthesis process, a sound source necessary for speech synthesis can be specified for each note constituting the designated music and assigned lyrics, and speech synthesis can be performed using the sound source for each specified note. . As a result, according to the speech synthesis process of the present invention, it is possible to generate, by speech synthesis, a singing voice that is similar to the voice quality of the user and more accurately mimics the characteristics of the user's singing.
[Other Embodiments]
As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, In the range which does not deviate from the summary of this invention, it is possible to implement in various aspects.

  For example, the audio waveform data WD in the above embodiment includes accompaniment sounds that played at least one instrument and at least singing sounds sung by a person as performance sounds, but the audio waveform data WD in the present invention. May contain only the singing sound.

  The aspect which abbreviate | omitted a part of structure of the said embodiment is also embodiment of this invention. Further, an aspect configured by appropriately combining the above embodiment and the modification is also an embodiment of the present invention. Moreover, all the aspects which can be considered in the limit which does not deviate from the essence of the invention specified by the wording described in the claims are the embodiments of the present invention.

In addition to the speech synthesizer and speech synthesis system described above, the present invention includes a program executed by a computer to output singing speech by speech synthesis, a speech synthesis method for outputting singing speech by speech synthesis, etc. It can be realized in various forms.
[Correspondence between Embodiment and Claims]
Finally, the relationship between the description of the above embodiment and the description of the scope of claims will be described.

  The function obtained by executing S520 in the speech synthesis process of the above embodiment is an example of the voice data acquisition means described in the claims, and the function obtained by executing S530 and S540 is a patent. It is an example of the analysis means described in the claims. Further, the function obtained by executing S550 in the speech synthesis process is an example of the search means described in the claims, and the function obtained by executing S580 and S590 is included in the claims. It is an example of the described synthetic | combination means.

  Furthermore, the function obtained by executing S120 in the sound source data generation process of the above embodiment is an example of the acquisition means described in the claims, and the function obtained by executing S130 is claimed. It is an example of the extraction means described in the range. Further, the function obtained by executing S140 in the sound source data generation process is an example of the specifying unit described in the claims, and the function obtained by executing S150 to S170 is described in the claims. It is an example of the 1st derivation | leading-out means described in (1). The function obtained by executing S180 in the sound source data generation process is an example of the second derivation means described in the claims, and is obtained by executing S200 and S220 in the sound source data generation process. The function is an example of the generation unit described in the claims, and the function obtained by executing S230 in the sound source data generation process is an example of the storage control unit described in the claims.

  DESCRIPTION OF SYMBOLS 1 ... Speech synthesis system 2 ... Information processing device 3 ... Input reception part 4 ... External output part 5, 14, 38 ... Storage part 6, 50, 16 ... Control part 7, 18, 52 ... ROM 8, 20, 54 ... RAM 9, 22, 56 ... CPU 10 ... Information processing server 12, 32 ... Communication unit 30 ... Karaoke device 34 ... Input reception unit 36 ... Music playback unit 40 ... Audio control unit 42 ... Output unit 44 ... Microphone input unit 46 ... Video control Unit 60 ... Speaker 62 ... Microphone 64 ... Display unit

Claims (5)

Voice data acquisition means for acquiring singing voice data for singing a song in which lyrics are assigned to at least a part of a plurality of notes consisting of a combination of pitch and note value;
The input voice quality representing the voice quality feature quantity of the singing voice data acquired by the voice data acquisition means, and the transition in the interval of the notes constituting the music of at least one of the amplitude and the fundamental frequency of the singing voice data. An analysis means for deriving an input song time to represent,
The sound source sound data represents a sound waveform obtained by uttering lyrics assigned to at least a part of a plurality of notes composed of a combination of pitch and sound value, and the person who uttered is derived for each different sound source sound data. Voice quality feature quantity that is a voice quality feature quantity and a song feature quantity representing a transition in a note-corresponding section corresponding to a note in the sound source voice data of at least one of the amplitude and fundamental frequency of the sound source voice data And voice quality song data associated with singer identification information for identifying the person who uttered the voice quality song data stored in the first storage device, and the input voice quality derived by the analysis means and Singers included in the voice quality feature quantity satisfying a prescribed condition including that the similarity to the input song time is equal to or higher than a predetermined reference value and the voice quality song time data including the song feature quantity Search means for identifying the different information,
Specific identification information that is the singer identification information specified by the search means from among the sound source data in which the sound source audio data is associated with each singer identification information and stored in the second storage device And synthesizes the singing voice singing the designated music as the designated music according to the sound source data included in the obtained sound source data and the singing voice data obtained by the voice data obtaining means. And a synthesizing unit that generates and outputs the voice. Voice data acquisition means for acquiring singing voice data for singing a song in which lyrics are assigned to at least a part of a plurality of notes consisting of a combination of pitch and note value;
The input voice quality representing the voice quality feature quantity of the singing voice data acquired by the voice data acquisition means, and the transition in the interval of the notes constituting the music of at least one of the amplitude and the fundamental frequency of the singing voice data. An analysis means for deriving an input song time to represent,
The sound source sound data represents a sound waveform obtained by uttering lyrics assigned to at least a part of a plurality of notes composed of a combination of pitch and sound value, and the person who uttered is derived for each different sound source sound data. Voice quality feature quantity that is a voice quality feature quantity and a song feature quantity representing a transition in a note-corresponding section corresponding to a note in the sound source voice data of at least one of the amplitude and fundamental frequency of the sound source voice data And voice quality song data associated with singer identification information for identifying the person who uttered the voice quality song data stored in the first storage device, and the input voice quality derived by the analysis means and Singers included in the voice quality feature quantity satisfying a prescribed condition including that the similarity to the input song time is equal to or higher than a predetermined reference value and the voice quality song time data including the song feature quantity Search means for identifying the different information,
Specific identification information that is the singer identification information specified by the search means from among the sound source data in which the sound source audio data is associated with each singer identification information and stored in the second storage device And synthesizes the singing voice singing the designated music as the designated music according to the sound source data included in the obtained sound source data and the singing voice data obtained by the voice data obtaining means. A speech synthesis system comprising: synthesis means for generating and outputting at the above. The search means includes
The speech synthesis system according to claim 2, wherein the specific identification information is specified by satisfying the specified condition that the degree of similarity is from the highest to a specified number specified in advance. The synthesis means includes
Among the notes constituting the designated music and assigned lyrics, the notes sung by the singing voice data obtained by the voice data obtaining means are singing notes, and the designated music is constituted and the lyrics are assigned. Notes other than the singing notes are non-singing notes,
By synthesizing speech based on the singing voice data acquired by the voice data acquiring means, the singing voice of the lyrics assigned to the singing note is generated, specified by the search means, and corresponding to the specific identification information The speech synthesis system according to claim 3, wherein speech synthesis of the lyrics assigned to the non-singing notes is generated by performing speech synthesis based on the attached sound source speech data. An acquisition means for acquiring music data including at least the voice waveform of the performance sound of the music including the vocal sound and the identification information representing the utterer of the vocal sound as the singer identification information;
Extracting means for extracting the vocal sound included in the music data acquired by the acquiring means as the sound source sound data;
Identifying means for identifying a note vocal that is a section of the sound source sound data corresponding to each of the note-corresponding sections of the sound source sound data extracted by the extracting means;
First derivation means for deriving a transition in the note-corresponding section of at least one of the amplitude and fundamental frequency of the note vocal specified by the specifying means as the song feature amount;
Second derivation means for deriving a voice quality feature amount for each note vocal for each note vocal identified by the specification means, and deriving a representative value of the voice quality feature amount as the voice quality feature amount;
Generating means for generating the voice quality song data by associating the singing feature quantity derived by the first derivation means, the voice quality feature quantity derived by the second derivation means, and the singer identification information; ,
The voice synthesis according to any one of claims 2 to 4, further comprising storage control means for storing the voice quality song data generated by the generation means in the first storage device. system.