JP2016051036A - Voice synthesis system and voice synthesis device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for simplifying generation of a virtual voice sound.SOLUTION: In a voice source data generation process, a voice waveform data WD is grouped by similarity in voice quality, and a representative value of the voice parameter P of a note vocal Vo included in each group is derived for each common note property p. Together with this, a classification key Xq which is the representative value of a voice feature quantity M in the group is associated with the representative value of the voice parameter P derived for each note property p, thereby generating voice data SD. On the other hand, in the voice synthesis process, an input voice input via a microphone is analyzed so as to derive an input voice quality Yk (S530). In the voice data SD stored in a storage part, the voice source data SD having the classification key Xq most approximate to the input voice quality Yk is specified (S540), and the synthesized voice which synthesizes the voice according to the specified voice source data SD is outputted (S570, S580).SELECTED DRAWING: Figure 4

Description

  The present invention relates to a technique for generating synthesized speech.

  2. Description of the Related Art Conventionally, there is known a karaoke apparatus that plays music, and generates and outputs a voice that sang at least one singing melody of a plurality of singing melody as a synthesized voice (Patent Document 1). reference). In this karaoke apparatus, the user is in charge of singing at least one of a plurality of singing melody, and the synthesized voice is in charge of other singing melody.

JP 2011-154290 A

  By the way, in the karaoke apparatus, it is considered that the synthesized voice (hereinafter referred to as “virtual vocalization sound”) sung by the user himself / herself is generated and outputted without the user of the karaoke apparatus singing. Yes.

  The generation of sound source data is performed by analyzing voice actually uttered by a person. For this reason, in order to generate | occur | produce a virtual utterance sound based only on the sound source data produced | generated by one user himself of a karaoke apparatus, a huge amount of sound source data is needed. In order to generate an enormous amount of sound source data, it is necessary to collect an enormous amount of audio data uttered by one user of the karaoke apparatus.

Generally, since it is difficult to collect an enormous amount of voice data of one user, there is a problem that it is difficult to generate a virtual utterance sound in the conventional technology.
That is, the conventional technique has a problem that it is difficult to generate a virtual voice.

  Therefore, an object of the present invention is to provide a technique that simplifies the generation of a virtual vocal sound.

  The speech synthesis system of the present invention made to achieve the above object includes speech data acquisition means, analysis means, classification means, storage control means, input reception means, voice quality analysis means, search means, Synthesizing means.

  Among these, the voice data acquisition means acquires at least two or more voice data in which at least one of the uttered person and the lyrics is different. The voice data referred to here represents a voice waveform obtained by uttering lyrics assigned to at least a part of a plurality of notes composed of combinations of pitches and note values.

  The analysis means derives a voice quality feature amount representing voice quality in the voice data acquired by the voice data acquisition means. The classifying unit classifies the voice data into at least two groups based on the distribution of the voice quality feature amount derived by the analyzing unit.

  The storage control means derives a representative value of the speech feature amount representing the speech feature amount for each syllable of lyric and note type from the speech data included in each group of the speech data classified by the classification means. Then, the sound source data in which the representative value of the derived voice feature quantity and the voice quality feature quantity corresponding to the representative value of the voice feature quantity are associated is stored in the storage device.

  The input receiving means receives voice input. The voice quality analyzing unit analyzes the input voice received by the input receiving unit and derives an input voice quality that is a voice quality feature amount of the input voice. Further, the search means specifies sound source data having a voice quality feature amount most similar to the input voice quality derived by the voice quality analysis means from the sound source data stored in the storage device. The synthesizing unit outputs a synthesized voice synthesized by voice according to the sound source data specified by the search unit.

  In the speech synthesis system of the present invention, synthesized speech is generated using speech parameters (sound source data) that are similar in voice quality to the user's own voice. In the present invention, such sound source data is derived based not only on the sound data of one user but also on the sound data of a plurality of persons.

  That is, according to the speech synthesis system of the present invention, not only speech data uttered by the user of the karaoke apparatus but also other people who are different from the user uttered as speech data necessary for generating sound source data. Audio data is used.

  Therefore, according to the speech synthesis system of the present invention, it is possible to easily collect speech data necessary for speech synthesis, and to reduce the difficulty of generating virtual utterances. As a result, according to the present invention, it is possible to simplify the generation of the virtual vocal sound.

  Further, in the speech synthesis system of the present invention, the music data acquisition means is music data representing the designated music, and at least one of the music data and the musical score data in which a plurality of notes are arranged along the time axis. You may acquire the music data containing the lyric data showing the lyrics allocated to some notes.

In this case, the synthesizing unit in the present invention may output a synthesized voice in which the lyrics included in the song data are sung based on the song data acquired by the song data acquiring unit.
According to such a speech synthesis system, it is possible to sing a song with synthesized speech using a voice close to the voice quality of the user.

By the way, the present invention may be implemented as a speech synthesizer. The speech synthesizer according to the present invention includes an input receiving unit, a voice quality analyzing unit, a searching unit, and a synthesizing unit.
The input receiving means receives voice input. The voice quality analyzing unit analyzes the input voice received by the input receiving unit, and derives an input voice quality that is a voice quality feature amount representing the voice quality of the input voice.

  The search means specifies sound source data having a voice quality feature amount most similar to the input voice quality derived by the voice quality analysis means from the sound source data stored in the storage device. However, the sound source data is stored in the storage device by executing an audio data acquisition step, an analysis step, a classification step, and a storage control step.

  In the voice data acquisition step, at least two or more voice data in which at least one of the uttered person and the lyrics is different are acquired. In the analysis step, a voice quality feature amount representing the voice quality in the voice data acquired in the voice data acquisition step is derived. In the classification step, the speech data is classified into at least two groups based on the distribution of voice quality feature values derived in the analysis step. In the storage control step, for each group of voice data classified in the classification step, a representative value of a voice feature amount representing a voice feature amount for each type of syllable and note is derived from the voice data included in each group, Sound source data in which the representative value of the derived voice feature quantity and the voice quality feature quantity corresponding to the representative value of the voice feature quantity are associated is stored in the storage device.

Then, the synthesizing means in the speech synthesizer outputs synthesized speech synthesized by voice according to the sound source data specified by the search means.
With such a speech synthesizer, the same effect as that of the first aspect of the invention can be obtained.

1 is a block diagram showing an overall configuration of a speech synthesis system to which the present invention is applied. It is a flowchart which shows the process sequence of a sound source data generation process. It is explanatory drawing which illustrated sound source data. It is a flowchart which shows the process sequence of a speech synthesis process.

Embodiments of the present invention will be described below with reference to the drawings.
<Speech synthesis system>
A speech synthesis system 1 shown in FIG. 1 is a system that generates and outputs a synthesized speech singing a song designated by a user (hereinafter referred to as a designated song) with a voice similar to the user.

In order to realize this, the speech synthesis system 1 includes an information processing device 2, an information processing server 10, and a karaoke device 30.
The information processing device 2 generates sound source data SD necessary for generating synthesized speech (that is, speech synthesis) based on speech waveform data WD including speech uttered by a person and MIDI music MD representing the uttered content. . The sound source data SD referred to here is data including speech parameters used for so-called formant synthesis, as will be described in detail later. The speech parameters include, for example, the fundamental frequency (f0), mel frequency cepstrum (MFCC), and power at each syllable in the uttered speech. The voice parameter mentioned here is an example of a voice feature amount described in the claims.

The information processing server 10 stores at least sound source data SD and MIDI music MD generated by the information processing apparatus 2.
The karaoke apparatus 30 plays the MIDI music MD stored in the information processing server 10, and generates and outputs a synthesized voice in which the music corresponding to the MIDI music MD is sung according to the sound source data SD. Note that the speech synthesis system 1 includes a plurality of karaoke apparatuses 30.
<Audio waveform data>
The audio waveform data WD is audio data representing a performance sound of playing a music, and is associated with music management information in which information related to the music is described. The music management information includes music identification information (hereinafter referred to as music ID) for identifying music.

  The audio waveform data WD of the present embodiment includes, as performance sounds, accompaniment sounds that are played by at least one musical instrument and singing sounds that are sung by humans. Note that the voice waveform data WD differs in the sung person or song (lyric) for each voice waveform data WD.

  The audio waveform data WD may be data configured by an audio file in an uncompressed audio file format, or may be data configured by an audio file in an audio compression format. The voice waveform data WD may be generated by recording voice when the user sings a song, or may be generated by other methods.

The audio waveform data WD in the present embodiment is an example of audio data described in the claims.
<MIDI music>
The MIDI music MD is prepared in advance for each music and has music data and lyrics data.

  Among these, the music data is data representing the score of one music according to the well-known MIDI (Musical Instrument Digital Interface) standard. This music data has at least a music ID and a music score track representing a music score for each musical instrument used in the music.

  The musical score track defines at least the pitch (so-called note number) and the period during which the MIDI sound source outputs the performance sound (hereinafter referred to as the note length) for each performance sound output from the MIDI sound source. Has been. The note length in the score track is the performance start timing (so-called note-on timing) indicating the time from the start of the performance of the music until the output of the performance sound and the music until the output of the performance sound ends. Performance end timing (so-called note-off timing) representing the time from the start of the performance.

  That is, in the score track, one note NO is defined by the note number and the note length represented by the note-on timing and note-off timing. The musical score track functions as one musical score by arranging note NO in the order of performance. Note that the musical score track is prepared for each instrument such as a keyboard instrument, a stringed instrument, a percussion instrument, and a wind instrument, for example. Among these, in this embodiment, a specific musical instrument (for example, vibraphone) is defined as a musical instrument responsible for singing melody in music.

  On the other hand, the lyrics data is data relating to the lyrics of the music, and includes lyrics telop data and lyrics output data. The lyrics telop data represents characters that constitute the lyrics of the music (hereinafter referred to as lyrics component characters). The lyrics output data is data in which a timing correspondence relationship that associates the lyrics output timing, which is the output timing of the lyrics constituent characters, with the performance of the performance data is defined.

Specifically, in the timing correspondence relationship in the present embodiment, the timing for starting the output of the lyrics telop data is associated with the timing for starting the performance of the performance data. Furthermore, in the timing correspondence relationship, the lyrics output timing of each lyrics constituent character along the time axis of the music is defined by the elapsed time from the performance start of the performance data. Thereby, each performance sound (namely, note NO) prescribed | regulated to the score track | truck and each lyric component character are matched.
<Information processing device>
The information processing apparatus 2 is a known information processing apparatus (for example, a personal computer) including an input receiving unit 3, an external output unit 4, a storage unit 5, and a control unit 6.

  The input receiving unit 3 is an input device that receives input of information and commands from the outside. The input device here is, for example, a key or switch, a reading drive for reading data stored in a portable storage medium (for example, CD, DVD, flash memory), or a communication port for acquiring information via a communication network. Etc. The external output unit 4 is an output device that outputs information to the outside. Here, the output device is a writing drive that writes data to a portable storage medium, a communication port that outputs information to a communication network, or the like.

  The storage unit 5 is a known storage device configured to be able to read and write stored contents. The storage unit 5 stores at least two or more speech waveform data WD in association with a MIDI music piece MD representing the utterance content in the speech waveform data WD. In addition, the code | symbol "l" in FIG. 1 is an identifier which identifies the audio | voice waveform data WD, and is allocated for every user and every music which the said user sang. This code “l” is a natural number of 2 or more. Further, the symbol “o” in FIG. 1 is an identifier for identifying the MIDI music piece MD, and is assigned to each music piece. This code “o” is a natural number of 2 or more.

  The control unit 6 is a known control device that is configured around a known microcomputer including a ROM 7, a RAM 8, and a CPU 9. The ROM 7 stores processing programs and data that need to retain stored contents even when the power is turned off. The RAM 8 temporarily stores processing programs and data. The CPU 9 executes each process according to a processing program stored in the ROM 7 or RAM 8.

The ROM 7 of the present embodiment has a processing program for the control unit 6 to execute sound source data generation processing for generating sound source data SD based on the audio waveform data WD and the MIDI music piece MD stored in the storage unit 5. It is remembered.
<Information processing server>
The information processing server 10 includes a communication unit 12, a storage unit 14, and a control unit 16.

  Among these, the communication unit 12 performs communication between the information processing server 10 and the outside via a communication network. That is, the information processing server 10 is connected to the karaoke apparatus 30 via a communication network. The communication network referred to here may be a wired communication network or a wireless communication network.

  The storage unit 14 is a known storage device configured to be able to read and write stored contents. The storage unit 14 stores at least a plurality of MIDI music pieces MD. 1 is an identifier for identifying the MIDI music piece MD stored in the storage unit 14 of the information processing server 10, and is assigned to each music piece. This code “n” is a natural number of 1 or more. Further, the storage unit 14 stores sound source data SD generated by the information processing apparatus 2 executing sound source data generation processing. The code “m” shown in FIG. 1 is an identifier for identifying the sound source data SD stored in the storage unit 14 of the information processing server 10, and is assigned to each group to be described in detail later. This code “m” is a natural number of 2 or more.

The control unit 16 is a known control device that is configured around a known microcomputer including a ROM 18, a RAM 20, and a CPU 22. The ROM 18 stores processing programs and data that need to retain stored contents even when the power is turned off. The RAM 20 temporarily stores processing programs and data. The CPU 22 executes each process according to a processing program stored in the ROM 18 or the RAM 20.
<Karaoke equipment>
The karaoke apparatus 30 includes a communication unit 32, an input reception unit 34, a music playback unit 36, a storage unit 38, an audio control unit 40, a video control unit 46, and a control unit 50.

  In the communication unit 32, the karaoke apparatus 30 communicates with the outside via a communication network. The input receiving unit 34 is an input device that receives input of information and commands in accordance with external operations. Here, the input device is, for example, a key, a switch, a reception unit of a remote controller, or the like.

  The music playback unit 36 performs a music performance based on the MIDI music MD downloaded from the information processing server 10. The music reproducing unit 36 is, for example, a MIDI sound source. The voice control unit 40 is a device that controls voice input / output, and includes an output unit 42 and a microphone input unit 44.

  A microphone 62 is connected to the microphone input unit 44. As a result, the microphone input unit 44 acquires the sound input via the microphone 62. A speaker 60 is connected to the output unit 42. The output unit 42 outputs the sound source signal of the music reproduced by the music reproducing unit 36 and the sound source signal of the singing sound from the microphone input unit 44 to the speaker 60. The speaker 60 outputs the sound source signal output from the output unit 42 instead of sound.

  The video control unit 46 outputs a video or an image based on the video data sent from the control unit 50. The video control unit 46 is connected to a display unit 64 that displays video or images.

  The control unit 50 is configured around a known computer having at least a ROM 52, a RAM 54, and a CPU 56. The ROM 52 stores processing programs and data that need to retain stored contents even when the power is turned off. The RAM 54 temporarily stores processing programs and data. The CPU 56 executes each process according to a processing program stored in the ROM 52 or the RAM 54.

The ROM 52 of the present embodiment stores a processing program for the control unit 50 to execute speech synthesis processing. The voice synthesis process is a process of generating and outputting a synthesized voice in which the voice quality is similar to the voice of the user and singing the music designated by the user.
<Sound source data generation processing>
A sound source data generation process executed by the control unit 6 of the information processing apparatus 2 will be described.

  As shown in FIG. 2, when the sound source data generation process is activated, the control unit 6 acquires a MIDI music piece MD including the music piece ID designated via the input receiving unit 3 (S110). Subsequently, the control unit 6 acquires one audio waveform data WD associated with the music ID acquired in S110 from all the audio waveform data WD stored in the storage unit 5 (S120). .

  In the sound source data generation process, the control unit 6 suppresses the accompaniment sound included in the speech waveform data WD acquired in S120 (S130). In the present embodiment, a known method may be used as a method for suppressing the accompaniment sound. The accompaniment sound suppression method in the present embodiment may be a method of emphasizing the singing sound included in the audio waveform data WD, or removing the performance sound of the musical instrument represented by the MIDI music piece MD from the audio waveform data WD. It may be a technique.

  Further, in the sound source data generation process, the control unit 6 specifies the note vocal Vo (a, i) based on the voice waveform data WD in which the accompaniment sound is suppressed in S130 and the MIDI music piece MD acquired in S110. (S140). The note vocal Vo (a, i) is a section corresponding to each note NO (a, i) that constitutes a singing melody and is assigned lyrics in the voice waveform data WD. In S140, the control unit 6 collates the voice waveform data WD acquired in S120 with the performance start timing nnt (a, i) and the performance end timing nft (a, i) in the MIDI music piece MD, so that a musical note vocal is obtained. Specify Vo (a, i).

  The symbol a in the present embodiment is a code for identifying a music piece, and the code i is a code for identifying a note NO of a singing melody in the music piece. The note vocal Vo (a, i) specified in S140 is stored in the storage unit 5 after the vowel of the lyrics assigned to the note NO (a, i) is associated.

  Further, in the sound source data generation process, the control unit 6 sets a plurality of analysis windows for each of the note vocals Vo (a, i) (S150). In setting the analysis window in S150, the control unit 6 sets the plurality of analysis windows so as to be adjacent to each other along the time axis. This analysis window is a section having a time length shorter than the time length of the note NO (a, i).

  Subsequently, in the sound source data generation process, the control unit 6 calculates a voice quality feature amount M (a, i) in each note vocal Vo (a, i) (S160). The voice quality feature amount M referred to here is a feature amount representing the voice quality of the person who uttered. In S160, the control unit 6 first performs frequency analysis (for example, DFT) for each analysis window of the note vocal Vo (a, i) set in S150. The control unit 6 calculates a mel frequency cepstrum (MFCC) of each analysis window as a voice quality feature amount M (a, i) by performing cepstrum analysis on the result of frequency analysis (frequency spectrum).

  In the sound source data generation process, the control unit 6 assigns the classification key Xq to each of the voice waveform data WD acquired in S120, and consequently, the note vocal Vo (a, i) specified in S140 (S170). The classification key Xq here is a vector representing a representative value of the voice quality feature amount M for each vowel of the lyrics.

  Specifically, in S <b> 170, the control unit 6 first extracts all the voice quality feature values M that are common to the vowels of the lyrics, and calculates an arithmetic average of the extracted voice quality feature values M. And the control part 6 makes the result of the arithmetic mean the representative value of the voice quality feature-value M. FIG. The calculation of the representative value of the voice quality feature amount M is executed for each vowel of the lyrics. Further, the control unit 6 generates vector data in which representative values of the voice quality feature amount M are arranged in the order of the vowels of the lyrics as the classification key Xq. The control unit 6 assigns the generated classification key Xq to each of the voice waveform data WD acquired in S120, and thus each of the note vocals Vo (a, i) specified in S140.

  Subsequently, in the sound source data generation process, the control unit 6 determines whether or not the processes from S120 to S170 have been performed on all the audio waveform data WD associated with the MIDI music piece MD acquired in S110. (S180). If the result of determination in S180 is that processing has not been executed for all audio waveform data WD (S180: NO), the controller 6 returns the sound source data generation processing to S120. In S120, one audio waveform data WD is acquired from the audio waveform data WD that is associated with the MIDI musical piece MD acquired in S110 and that has not been subjected to the processing from S120 to S180. Thereafter, the control unit 6 executes steps from S130 to S180.

  On the other hand, as a result of the determination in S180, if the processing has been executed for all the audio waveform data WD (S180: YES), the control unit 6 stores all the MIDI musical pieces MD stored in the storage unit 5. It is determined whether or not it has been acquired (S190). As a result of the determination in S190, if the process is not executed for all the MIDI music pieces MD (S190: NO), the control unit 6 returns the sound source data generation process to S110. In S110, the control unit 6 acquires one MIDI music MD from the MIDI music MD that has not been processed. Thereafter, in the sound source data generation process, S120 to S190 are repeated.

  Incidentally, as a result of the determination in S190, if the processing has been executed for all the MIDI music pieces MD (S190: YES), the control unit 6 shifts the sound source data generation processing to S200.

  In S200, the control unit 6 groups the speech waveform data WD stored in the storage unit 5 and the note vocal Vo in accordance with the classification key Xq. This grouping may be executed by using known clustering (for example, k-means method) using the classification key Xq as data. Therefore, the grouping in S200 forms a group (cluster) in which the speech waveform data WD (and thus the note vocal Vo) is classified for each speech waveform data WD whose voice quality feature value M approximates.

  Subsequently, in the sound source data generation process, the control unit 6 selects one of the plurality of groups generated in S200 (S210). Further, the control unit 6 acquires one note vocal Vo (a, i) included in the group selected in S210 (S220).

  Further, in the sound source data generation process, the control unit 6 calculates the speech parameter P (a, i) of the note vocal Vo (a, i) acquired in S220 (S230). The audio parameter P derived in S230 of the present embodiment includes at least a fundamental frequency (f0), a mel frequency cepstrum (MFCC), power, and each time difference. Since these fundamental frequency, MFCC, and power deriving methods are well known, detailed description thereof is omitted here. For example, if the fundamental frequency is used, autocorrelation along the time axis of the note vocal Vo, note vocal What is necessary is just to derive | lead-out using methods, such as the autocorrelation of the frequency spectrum of Vo, or a cepstrum method. In the case of MFCC, the result of frequency analysis (for example, FFT) for each time analysis window by applying a time analysis window to the note vocal Vo, and the result of logarithmizing the size for each frequency, Furthermore, it may be derived by frequency analysis. The power may be derived by integrating the result of squaring the amplitude by applying a time analysis window to the note vocal Vo and integrating it in the time direction.

  In the sound source data generation process, the control unit 6 specifies the note property p (a, i) of the note NO (a, i) corresponding to the note vocal Vo (a, i) acquired in S220 (S240). ). In S240 of the present embodiment, specifically, the control unit 6 uses, as a note property p (a, i), information of each note NO (a, i) defined in the MIDI song MD from the MIDI song MD. Extract and identify.

  The note property p (a, i) mentioned here includes a target note attribute, a previous note attribute, and a rear note attribute. The target note attribute is information representing the attribute of the note NO (a, i). The target note attributes include the scale (pitch) of note NO (a, i), note length, and syllable of lyrics. The previous note attribute is information representing the attribute of the note NO (a, i-1) immediately preceding the note NO (a, i) (hereinafter referred to as the previous note) along the time axis. The previous note attribute includes the scale (pitch), note length, syllable of the previous note NO (a, i-1), and the previous note NO (a, i-1) and note NO (a, i). Including the length of time between.

  Further, the back note attribute is information representing the attribute of the note (hereinafter referred to as a back note) NO (a, i + 1) immediately after the target note NO (a, i) along the time axis. The post-note attributes include a musical scale (pitch), a note length, a syllable of lyrics, and a time length between the note NO (a, i) and the subsequent note NO (a, i + 1). Note that the note length in the note property p (a, i) and the time length between notes may be quantized to a predetermined class.

  Subsequently, the control unit 6 associates the note property p (a, i) specified in S240 with the voice parameter P (a, i) calculated in S230 (S250). In S250, the control unit 6 stores the speech parameter P (a, i) associated with the note property p (a, i) in the storage unit 5.

  Further, in the sound source data generation process, the control unit 6 determines whether or not the processes from S230 to S250 have been executed for all the note vocals Vo included in the group set in S210 (S260). As a result of the determination in S260, if there is a note vocal Vo for which the processing from S230 to S250 has not been executed (S260: NO), the control unit 6 returns the sound source data generation processing to S220. In S220, the control unit 6 acquires one note vocal Vo (a, i) from the note vocals Vo that have not been subjected to the processing from S230 to S250.

  On the other hand, if the processing from S230 to S250 has been executed for all note vocals Vo included in the group set in S210 (S260: YES), the control unit 6 sets the representative value of the voice parameter P. Calculate (S270). In S270, the control unit 6 arithmetically averages the speech parameter P for each common note property p in the group set in S210. Then, the control unit 6 calculates each arithmetic mean result as a representative value of the voice parameter P.

  Further, in the sound source data generation process, the control unit 6 generates sound source data SD and stores it in the storage unit 5 (S280). In S280, the control unit 6 associates the classification key Xq associated with the group set in S210 with the representative value of the audio parameter P calculated in S270, so that the sound source data as shown in FIG. SD is generated.

  Subsequently, in the sound source data generation process, the control unit 6 determines whether or not the processes from S220 to S280 have been executed for all the groups generated in S200 (S290). As a result of the determination in S290, if there is a group in which the processes from S220 to S250 are not executed (S290: NO), the control unit 6 returns the sound source data generation process to S210. In S210, the control unit 6 selects one group from the groups in which the processes from S220 to S280 are not executed.

  On the other hand, as a result of the determination in S290, if the processes from S220 to S250 have been executed for all groups (S290: YES), the control unit 6 ends the sound source data generation process.

  As described above, in the sound source data generation process, the note vocals Vo are grouped by those whose voice qualities are approximated according to the result of analyzing the voice waveform data WD prepared in advance. Further, in the sound source data generation process, the representative value of the speech parameter P of the note vocal Vo included in the group whose voice quality is approximated is derived for each common note property p. At the same time, in the sound source data generation process, the sound source data SD is generated by associating the classification key Xq, which is the representative value of the voice quality feature amount M in the group, with the representative value of the speech parameter P derived for each note property p. To do.

Note that the sound source data SD generated by the control unit 6 of the information processing device 2 executing the sound source data generation process may be stored in the storage unit 14 of the information processing server 10 using a portable storage medium. When the information processing device 2 and the information processing server 10 are connected via a communication network, the sound source data SD stored in the storage unit 5 of the information processing device 2 is transferred via the communication network. Thus, the information may be stored in the storage unit 14 of the information processing server 10.
<Speech synthesis processing>
Next, the speech synthesis process executed by the control unit 50 of the karaoke apparatus 30 will be described.

As shown in FIG. 4, when the speech synthesis process is activated, the control unit 50 acquires a song ID corresponding to a song (designated song) designated via the input receiving unit 34 (S510).
In the speech synthesis process, subsequently, the control unit 50 acquires utterance speech data representing the waveform of speech input via the microphone 62 connected to the microphone input unit 44 (S520). Then, the control unit 50 analyzes the voice quality of the voiced voice data acquired in S520, and calculates the input voice quality Yk (S530). The input voice quality Yk is a voice quality feature value representative of the mel frequency cepstrum (MFCC) for each vowel, as with the classification key Xq described above. The method for calculating the input voice quality Yk is the same as S150 to S170 in the sound source data generation process except that “note vocal Vo” is read as “voiced voice data”, and detailed description thereof is omitted here. .

Further, in the speech synthesis process, the control unit 50 specifies a similar group that is a group whose voice quality approximates the voice of the user of the karaoke apparatus 30 based on the input voice quality Yk calculated in S530 (S540). In S540, the control unit 50 calculates an approximate value k _opt according to the following equation (1), and identifies a group having the largest approximate value k _opt as a similar group.

なお、（１）式における符号“ｋ”はグループを識別する識別子である。また、（１）式におけるａｒｇ_ｋｍｉｎは、ｋの関数における最小値を意味する。

Note that the symbol “k” in equation (1) is an identifier for identifying a group. Also, arg _k min in the equation (1) means the minimum value in the function of k.

Further, in the speech synthesis process, the control unit 50 acquires the MIDI music MD corresponding to the music ID acquired in S510 from the information processing server 10 (S550).
In the speech synthesis process, the control unit 50 associates the sound source data SD corresponding to the note property p of the note NO included in the MIDI music piece MD acquired in S550 and corresponding to the similar group specified in S540. Obtain (S560). Based on the sound source data SD acquired in S560, the control unit 50 performs voice synthesis so that the singing sound of the music corresponding to the MIDI music MD acquired in S550 is output (S570). Specifically, the control unit 50 selects sound source data SD (corresponding to the syllable of the lyrics of the music corresponding to the MIDI music MD acquired in S550 from the sound source data SD corresponding to the similar group specified in S540. The voice parameter P) is acquired (S560). And the control part 50 produces | generates a synthetic | combination voice by adjusting the acquired audio | voice parameter P and formant-synthesize | combining so that the lyrics of the music corresponding to the MIDI music MD acquired in S550 may be sung (S570). ).

  Further, the control unit 50 outputs the synthesized voice generated by synthesizing the voice in S560 to the output unit 42 (S580). The output unit 42 emits synthesized speech from the speaker 60.

Thereafter, the control unit 50 ends the speech synthesis process.
That is, in the speech synthesis process, the input speech input via the microphone 62 is analyzed, and the input voice quality Yk that is the voice quality feature quantity of the input speech is derived. In the speech synthesis process, the sound source data SD having the classification key Xq most similar to the input voice quality Yk is specified among the sound source data SD stored in the storage unit 14 of the information processing server 10. Further, in the voice synthesis process, a synthesized voice (that is, a virtual utterance) singing the designated music is generated by voice synthesis according to the specified sound source data SD and output.
[Effect of the embodiment]
As described above, the speech synthesis system 1 generates synthesized speech using the speech parameter P (sound source data SD) whose voice quality is similar to that of the user's own voice. In the speech synthesis system 1, such sound source data SD is generated based on not only the speech waveform data WD of one user but also the speech waveform data WD of a plurality of people.

  That is, according to the speech synthesis system 1, not only the speech waveform data WD uttered by the user of the karaoke apparatus 30 as speech waveform data WD necessary for generating the sound source data SD, but also another person different from the user. Is used as voice waveform data WD.

Therefore, according to the speech synthesis system 1, the speech waveform data WD necessary for speech synthesis can be easily collected, and the difficulty of generating the speech parameter P can be reduced. As a result, according to the speech synthesis system 1, it is possible to reduce the difficulty of generating the synthesized speech (and thus the virtual utterance), and it is possible to simplify the generation of the synthesized speech.
[Other Embodiments]
As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, In the range which does not deviate from the summary of this invention, it is possible to implement in various aspects.

  For example, the audio waveform data WD in the above embodiment includes accompaniment sounds that played at least one instrument and at least singing sounds sung by a person as performance sounds, but the audio waveform data WD in the present invention. May contain only the singing sound.

  In S270 of the sound source data generation process in the above embodiment, the representative value of the voice parameter P is a result of arithmetic averaging of the voice parameter P associated with the note property p common in the same group. The representative value of P is not limited to the arithmetic average result. That is, the representative value of the voice parameter P may be the mode value of the voice parameter P associated with the note property p common in the same group, or may be associated with the note property p common in the same group. The median value of the voice parameter P may be used.

  In addition, the aspect which abbreviate | omitted a part of structure of the said embodiment is also embodiment of this invention. Further, an aspect configured by appropriately combining the above embodiment and the modification is also an embodiment of the present invention. Moreover, all the aspects which can be considered in the limit which does not deviate from the essence of the invention specified by the wording described in the claims are the embodiments of the present invention.

In addition to the above-described speech synthesis system, the present invention can be implemented in various forms such as a program executed by a computer to implement speech synthesis, a speech synthesis method, and the like.
[Correspondence between Embodiment and Claims]
Finally, the relationship between the description of the embodiment and the description of the claims will be described.

  The function obtained by executing S120 of the sound source data generation process in the embodiment is an example of the voice data acquisition means described in the claims, and the function obtained by executing S130 to S160 is analyzed. It is an example of a means. Furthermore, the function obtained by executing S170 to S200 is an example of a classification unit, and the function obtained by executing S210 to S290 is an example of a storage control unit.

  Further, the function obtained by executing S520 of the speech synthesis process is an example of the input receiving means described in the claims, and the function obtained by executing S530 is an example of the voice quality analyzing means. is there. Furthermore, the function obtained by executing S540 is an example of a search unit, and the function obtained by executing S570 and S580 is an example of a synthesis unit. The function obtained by executing S550 is an example of a music data acquisition unit.

  DESCRIPTION OF SYMBOLS 1 ... Speech synthesis system 2 ... Information processing device 3 ... Input reception part 4 ... External output part 5, 14, 38 ... Memory | storage part 6, 16, 50 ... Control part 7, 18, 52 ... ROM 8, 20, 54 ... RAM 9, 22, 56 ... CPU 10 ... Information processing server 12, 32 ... Communication unit 30 ... Karaoke device 34 ... Input reception unit 36 ... Music playback unit 40 ... Audio control unit 42 ... Output unit 44 ... Microphone input unit 46 ... Video control Unit 60 ... Speaker 62 ... Microphone 64 ... Display unit

Claims (3)

Voice data representing a speech waveform uttered by a lyric assigned to at least a part of a plurality of notes consisting of a combination of a pitch and a note value, wherein at least one of the uttered person and the lyrics Audio data acquisition means for acquiring at least two different audio data;
Analysis means for deriving a voice quality feature amount representing voice quality in the voice data acquired by the voice data acquisition means;
Classification means for classifying the voice data into at least two groups based on the distribution of voice quality feature values derived by the analysis means;
For each group of voice data classified by the classification means, a representative value of a voice feature amount representing a voice feature amount for each syllable of the lyrics and each note type is derived from the voice data included in each group. Storage control means for storing sound source data in which the representative value of the derived voice feature quantity and the voice quality feature quantity corresponding to the representative value of the voice feature quantity are associated with each other in a storage device;
An input receiving means for receiving voice input;
Voice quality analysis means for analyzing the input voice received by the input reception means and deriving an input voice quality that is a voice quality feature quantity of the input voice;
Search means for identifying the sound source data having the voice quality feature amount most similar to the input voice quality derived by the voice quality analysis means among the sound source data stored in the storage device;
A speech synthesis system comprising: synthesis means for outputting synthesized speech synthesized by voice according to the sound source data specified by the search means. Music data representing designated music, including musical score data in which a plurality of notes are arranged along a time axis, and lyrics data representing lyrics assigned to at least some of the plurality of notes A music data acquisition means for acquiring music data;
The synthesis means includes
The speech synthesis system according to claim 1, wherein a synthesized speech in which the lyrics included in the song data are sung is output based on the song data acquired by the song data acquisition unit. An input receiving means for receiving voice input;
Voice quality analysis means for analyzing an input voice received by the input reception means and deriving an input voice quality that is a voice quality feature amount representing a voice quality of the input voice;
Voice data representing a speech waveform uttered by a lyric assigned to at least a part of a plurality of notes consisting of a combination of a pitch and a note value, wherein at least one of the uttered person and the lyrics An audio data acquisition step for acquiring at least two different audio data, an analysis step for deriving a voice quality feature amount representing voice quality in the audio data acquired in the audio data acquisition step, and a voice quality feature amount derived in the analysis step A step of classifying the audio data into at least two groups based on the distribution; and for each group of audio data classified in the classification step, from the audio data included in each group, the syllables of the lyrics and Deriving a representative value of the speech feature value representing the feature value of the speech for each type of note, and deriving it The storage control step of storing sound source data in which the representative value of the voice feature amount and the voice quality feature amount corresponding to the representative value of the voice feature amount are associated with each other is executed, so that the storage device stores the sound source data. Search means for specifying the sound source data having the voice quality feature amount most similar to the input voice quality derived by the voice quality analysis means among the sound source data that has been obtained;
A speech synthesizer comprising: synthesis means for outputting synthesized speech synthesized by sound according to the sound source data specified by the search means.

Images