JP5136128B2 - Speech synthesizer - Google Patents

Description

  The present invention relates to a speech synthesizer.

A technique for synthesizing a singing voice based on a human voice by inputting melody and lyrics has been proposed. For example, Patent Document 1 discloses that a database is generated by performing an SMS analysis on a phoneme or two or more phoneme chains using a technique called spectral modeling synthesis (SMS), and a necessary phoneme or phoneme chain is generated. A technique for synthesizing a singing voice by connecting the SMS data is proposed. Patent Documents 2 to 5 propose techniques for performing more natural singing synthesis.
JP 2002-202790 A JP 2002-202788 A JP 2003-323188 A JP 2004-264676 A Japanese Patent Laid-Open No. 2004-4440

  By the way, the singing synthesized sound generated by the singing synthesis may be mechanical and unnatural. Also, there are cases where the user himself / herself wants to adjust the inflection and voice quality of the singing synthesized sound according to the user's preference. Therefore, in order to make the singing synthesized sound desired by the user, there are some that the user can adjust the parameter value and adjust the pitch vent and velocity, and apply various effects.

However, adjustment of such parameter values often depends on empirical rules, and the user needs to repeat trial and error in order to obtain a desired singing synthesized sound. In particular, inexperienced users often have difficulty adjusting to a desired singing synthesized sound.
The present invention has been made under the above-described background, and an object thereof is to provide a technique capable of easily correcting a singing synthesized sound to a mode desired by a user.

In order to solve the above problems, the present invention provides singing score data acquisition means for acquiring singing score data including melody data representing characteristics of each phoneme, which is singing score data representing a melody composed of a sequence of phonemes. The voice waveform corresponding to the singing score data is expressed from the first voice waveform data acquiring means for acquiring the first voice waveform data representing the voice waveform and the singing score data acquired by the singing score data acquiring means. Second voice waveform data generating means for generating second voice waveform data, correspondence means for associating the first voice waveform data and the second voice waveform data in the time axis direction, and the first First feature detecting means for detecting the feature according to the analysis result, and analyzing the second speech waveform data, and analyzing the feature according to the analysis result. The feature data included in the singing score data acquired by the singing score data acquiring means according to the association result of the association means and the first feature detecting means Correction of feature data so that a difference at a corresponding portion between the feature of the first speech waveform data detected by the feature and the feature of the second speech waveform data detected by the second feature detection means is reduced. Means, third voice waveform data generating means for generating third voice waveform data representing a voice waveform corresponding to the singing score data from the singing score data corrected by the feature data correcting means, and the third And an output means for outputting the third voice waveform data generated by the voice waveform data generation means. .
In a preferred aspect of the present invention, second association means for associating the first speech waveform data and the third speech waveform data generated by the third speech waveform data generation means in the time axis direction; Analyzing the third speech waveform data and detecting the feature according to the analysis result, and the feature data correcting unit according to the association result of the second association unit The feature data included in the modified singing score data includes the features of the first speech waveform data detected by the first feature detection unit and the third speech waveform detected by the third feature detection unit. Second feature data correction means for correcting the difference in the corresponding location with the feature of the data to be small, and the third speech waveform data generation means includes the feature data correction means. Ri the modified speech waveform data representing the song score data or the second speech waveform corresponding to the song score data corrected by the characteristic data correcting means is or may be generated as the third speech waveform data.

In a further preferred aspect of the present invention, the characteristics include at least the pronunciation timing of each phoneme constituting the melody, a temporal change in pitch, the phoneme of each phoneme constituting the melody , the speech spectrum, the volume, the sound quality, and the voice quality . Any one of them may be included.

  Further, in a further preferred aspect of the present invention, when the singing score data corrected by the feature data correcting unit satisfies a predetermined condition, the singing score data is supplied to the singing score data acquiring unit. An acquisition control means may be provided.

  Further, in a further preferred aspect of the present invention, the singing score data is divided into a plurality of time sections and includes section correspondence data indicating a correspondence relationship between the plurality of time sections. The feature data included in the singing score data acquired by the singing score data acquiring means according to the association result of the associating means for at least one of the time intervals of So that the difference at the corresponding location between the feature of the first speech waveform data detected by the feature detection unit and the feature of the second speech waveform data detected by the second feature detection unit is reduced. In addition, based on the interval correspondence data, other time intervals corresponding to the time interval are included in the singing score data. The characteristic data may be modified by modifying aspects in said time interval.

In a further preferred aspect of the present invention, the first voice waveform data acquisition means may acquire voice data representing the voice collected by the sound collection means as the first voice data.
Moreover, the further preferable aspect of this invention WHEREIN: The said characteristic data correction means may correct the characteristic data contained in the singing score data acquired by the said singing score data acquisition means with a several different correction | amendment aspect.
Further, in a further preferred aspect of the present invention, the feature data correction means is characterized in that the feature data included in the song score data acquired by the song score data acquisition means is detected by the first feature detection means. The difference at the corresponding location between the feature of the first speech waveform data and the feature of the second speech waveform data detected by the second feature detection means is corrected to be approximately half or a predetermined threshold value. Also good.
In a preferred aspect of the present invention, the information processing apparatus further comprises selection means for selecting one of the plurality of correction aspects according to information output from a user interface, and the third speech waveform data generation means is selected by the selection means. The third speech waveform data may be generated from singing score data including feature data corrected in the corrected mode.
In another preferred aspect of the present invention, the first feature detection means and the second feature detection means execute at least one of formant detection, cepstrum detection, and speech recognition processing. The phoneme may be detected.

  According to the present invention, the singing synthesized sound can be easily corrected to a mode desired by the user.

<A: Configuration>
FIG. 1 is a block diagram illustrating a hardware configuration of a speech synthesizer 1 according to an embodiment of the invention. The speech synthesizer 1 is a device that performs singing synthesis (speech synthesis) from data representing melody and lyrics (hereinafter referred to as “singing score data”) using a database created in advance. In the figure, a CPU (Central Processing Unit) 11 reads a computer program stored in a ROM (Read Only Memory) 12 or a storage unit 14, loads it into a RAM (Random Access Memory) 13, and executes it. The units of the speech synthesizer 1 are controlled. The storage unit 14 is a storage unit that stores a computer program executed by the CPU 11 and various data, and is, for example, a hard disk device. The storage unit 14 may be a CD-ROM device, a magneto-optical disk (MO) device, a digital multipurpose disk (DVD) device, or the like. The display unit 15 includes a liquid crystal display and the like, and displays various screens such as a menu screen for operating the speech synthesizer 1 under the control of the CPU 11. The operation unit 16 includes a mouse and a keyboard, and outputs a signal corresponding to the content operated by the user. The microphone 17 collects sound and outputs a sound signal (analog signal) representing the collected sound. The audio processing unit 18 includes a DAC and an ADC, converts an audio signal (analog signal) output from the microphone 17 into digital data by A / D conversion, and outputs the digital data to the CPU 11. The audio processing unit 18 converts the digital data supplied from the CPU 11 into an analog signal by D / A conversion and supplies the analog signal to the speaker 19. The speaker 19 emits sound with an intensity corresponding to the analog signal output from the sound processing unit 18.

  In this embodiment, the case where the microphone 17 and the speaker 19 are included in the speech synthesizer 1 will be described. However, the speech processing unit 18 is provided with an input terminal and an output terminal, and the input terminal is connected via an audio cable. A configuration may be adopted in which an external microphone is connected. Similarly, an external speaker may be connected to the output terminal via an audio cable. In this embodiment, the audio signal input from the microphone 17 to the audio processing unit 18 and the audio signal output from the audio processing unit 18 to the speaker 19 are analog audio signals. You may make it input / output. In such a case, the audio processing unit 18 does not need to perform A / D conversion or D / A conversion. The same applies to the display unit 15, and an external output terminal may be provided to connect an external monitor.

  The storage unit 14 includes a Timbre database 141, a phonological template database 142, a singing score data storage area 143, a corrected singing score data storage area 144, and an exemplary voice data storage area 145, as shown in the figure. Yes. The Timbre database 141 is a database in which voice parameters having different phoneme names and pitches are collected. This database is a database that the CPU 11 refers to when performing speech synthesis from singing score data. The voice parameters can be classified into, for example, an envelope of an excitation waveform spectrum, an excitation resonance, a formant, and a difference spectrum. These four speech parameters are obtained by decomposing the spectral envelope (original spectrum) of the harmonic component obtained by analyzing actual human speech or the like (original speech). A voice at a certain time can be expressed by a voice parameter (a set of excitation spectrum, excitation resonance, formant, and difference spectrum), and a voice parameter expressing the same voice is different if the pitch is different. The Timbre database 141 has phoneme names and pitches as indexes. Therefore, the CPU 11 can read out the voice parameter at a certain time t using the data belonging to the phonological track and the pitch track of the singing score data as a key.

  The phoneme template database 142 stores phoneme template data. The phoneme template data is data applied to the transition interval between phonemes and phonemes in the singing score data. When a human utters two phonemes in succession, it changes slowly, not suddenly. For example, if the vowel “e” is pronounced continuously without placing a break after the vowel “a”, “a” is pronounced first, and the pronunciation located between “a” and “e” After that, it changes to “E”. Therefore, in order to perform singing synthesis so that the phoneme combination part becomes natural, it is preferable to have some form of speech information of the excitement part for phoneme combinations that can be combined in a certain language. Considering this, preparing the voice data and the amount of pitch fluctuation in the phonological transition section as template data, and applying this template data to the phonological transition section in the singing score data, it is closer to the actual singing Realize speech synthesis.

The phoneme template data includes a sequence of digital values obtained by sampling a speech parameter P and a pitch variation Pitch expressed as a function of time t at a constant time Δt interval, and a section length T (sec.) Between the speech parameter P and the pitch pitch. .)) And can be represented by the following formula (A). In the following formula (A), t = 0, Δt, 2Δt, 3Δt,.
[Equation 1]
Template = [P (t), Pitch (t), T] (A)

  Next, the singing score data storage area 143 is singing score data representing a melody composed of a sequence of phonemes, and features of each phoneme (pronunciation timing of each phoneme, temporal change in pitch, Singing score data including feature data (phoneme data, pronunciation timing data, pitch data, etc.) representing phonemes etc. is stored.

  Fig.2 (a) is a conceptual diagram which shows an example of the content of song score data. This singing score data is composed of a plurality of tracks including a phonological track and a pitch track. In the phoneme track, phoneme data representing phonemes and sounding timing data indicating the sounding start timing and sounding end timing of each phoneme are recorded. Specifically, for example, in the example shown in FIG. 2A, the phoneme of the “sa” phoneme is pronounced between the time t1 and the time t2, and the phoneme of the “i” phoneme is from the time t2 to the time t3. It is shown that it is pronounced between. In the following, for convenience of explanation, when it is not necessary to distinguish between “sound generation start timing” and “sound generation end timing”, these will be referred to as “sound generation timing”. In the pitch track, pitch data indicating temporal changes in the fundamental frequency (pitch) of the sound to be sounded at each time is recorded.

The singing score data may be stored in advance in the singing score data storage area 143 of the storage unit 14 or generated by the CPU 11 executing a predetermined application program in response to a user operation. It may be.
FIG. 2B is a diagram illustrating an example of a screen displayed on the display unit 15 when the CPU 11 performs a song score data generation process. The CPU 11 displays a screen as illustrated in FIG. 2B and prompts the user to input singing score data. In the figure, the singing score data editing screen 600 includes an event display area 601 for displaying note data in a piano roll format. A scroll bar 606 for scrolling up and down the display screen of the event display area 601 is provided on the right side of the event display area 601. A scroll bar 607 for scrolling the display screen of the event display area 601 left and right is provided below the event display area 601.

  On the left side of the event display area 601, a keyboard display 602 (coordinate axis indicating the pitch) simulating a piano keyboard is displayed. Above the event display area 601, a bar display 604 indicating the bar position from the beginning of the music is displayed. The Reference numeral 603 denotes a piano roll display area which displays note data in a horizontally long rectangle (bar) at a time position indicated by a measure display 604 of a pitch indicated by a keyboard display 602. The left end position of the bar indicates the utterance start timing, the bar length indicates the utterance duration time, and the left end position of the bar indicates the utterance end timing.

  The user moves the mouse pointer to a position on the display screen corresponding to the desired pitch and time position and clicks to specify the utterance start position. Then, a note data bar (hereinafter referred to as “note bar”) from the utterance start position to the utterance end position is formed in the event display area 601 by a drag operation, and then the mouse is dropped. For example, in order to form the note bar 611, the mouse pointer is positioned at the beginning of the first beat of the 53rd bar, clicked on the mouse, and dragged until after one night.

  As described above, the user inputs the singing score data using the operation unit 16 while confirming the screen displayed on the display unit 15. The CPU 11 generates singing score data according to the signal output from the operation unit 16, and stores the generated singing score data in the singing score data storage area 143.

  Next, the corrected singing score data storage area 144 of the storage unit 14 stores corrected singing score data generated by the CPU 11 performing a singing score data correcting process to be described later on the singing score data. Since the singing score data correction process executed by the CPU 11 will be described later, detailed description thereof is omitted here.

  Next, in the exemplary audio data storage area 145 of the storage unit 14, for example, audio data representing an audio waveform in the WAVE format, MP3 (MPEG Audio Layer-3) format, etc., representing the singing audio sung by the user or the like. Audio data (first audio waveform data) is stored. In the following description, for convenience of explanation, the voice data stored in the model voice data storage area 145 is referred to as “model voice data”. The exemplary voice data is preferably data representing singing voice that matches the user's preferences (how to sing, how to add favorite inflection, etc.).

  Next, an example of the functional configuration of the speech synthesizer 1 will be described with reference to the block diagram shown in FIG. By executing the song synthesis program stored in the ROM 12 or the storage unit 14, the CPU 11 plays a role as the song synthesis unit 111, the voice reproduction unit 112, the song comparison unit 113, and the song score data correction unit 114.

  The singing voice synthesizing unit 111 reads the singing score data from the singing score data storage area 143, and generates voice waveform data (second voice waveform data) representing a voice waveform corresponding to the singing score data from the read singing score data. To do. More specifically, in this embodiment, the singing voice synthesizing unit 111 refers to pitch data, pronunciation timing data, phonological data, and the like included in the singing score data, and converts the speech parameters corresponding to the pitch and phonology to the phonological template. The database 142 is read from the Timbre database 141 with reference to the database 142, and digital voice waveform data is generated using the read voice parameters. Note that the singing synthesis unit 111 performs various control processes such as singing synthesis start / stop, tempo designation, and the like. These processes are the same as those in the conventional singing synthesis technique, and a detailed description thereof is omitted here. To do. Hereinafter, for convenience of explanation, the speech waveform data generated from the singing score data will be referred to as “synthesized speech data”.

  The synthesized voice represented by the synthesized voice data generated by the singing voice synthesis unit 111 may be mechanical and unnatural. Even if it is not unnatural, there is a case where it is desired to correct the singing method (intonation etc.) desired by the user. Therefore, in the present embodiment, this synthesized speech data is corrected by performing the processing indicated by the following singing score data correction unit 114.

  The audio reproducing unit 112 reads out the exemplary audio data stored in the exemplary audio data storage area 145, and reproduces the read out exemplary audio data. The singing comparison unit 113 compares the model voice data and the singing voice data, detects the singing timing shift, the pitch and the pitch change (curve) shift, and corrects the singing score data to the data representing the detected difference. Output to the unit 114. In the example shown in FIG. 3, a mechanism and an operation system for controlling the singing voice synthesizing unit 111 and the sound reproducing unit 112 are actually required, but the illustration is omitted to prevent the diagram from becoming complicated. doing.

  Here, the detail of the process which the song comparison part 113 performs is demonstrated below, referring drawings. First, the singing comparison unit 113 detects the pitch, power, and spectrum of each voice data from the model voice data and the synthesized voice data in units of frames each having a predetermined time length. For example, FFT (Fast Fourier Transform) is used for spectrum detection.

  Moreover, the song comparison part 113 calculates | requires both correspondence based on the detected spectrum. The voice represented by the model voice data (hereinafter “model voice”) and the voice represented by the synthesized voice data (hereinafter “synthesized voice”) may be shifted in time. For example, when a model singer deliberately shifts the beginning and end of singing, the model voice and the synthesized voice are shifted in time. Thus, even when the model voice and the synthesized voice are shifted forward and backward in time, time normalization (DTW: Dynamic) is performed to expand and contract the time axis of the synthesized voice data so that they can be associated with each other. Time Warping) and align the time axes of both. As a method for performing this DTW, DP (Dynamic programming) is used in this embodiment. Specifically, the processing is as follows.

  The singing comparison unit 113 forms a coordinate plane (hereinafter referred to as a DP plane) as shown in FIG. The vertical axis of the DP plane corresponds to a parameter obtained by performing inverse Fourier transform on the logarithm of the absolute value of the spectrum of each frame of the model voice data, and the horizontal axis is obtained from each frame of the synthesized voice data. This corresponds to a parameter (cepstrum) obtained by applying inverse Fourier transform to the logarithm of the absolute value of the spectrum. In FIG. 4, a1, a2, a3... An are obtained by arranging the frames of the model voice data according to the time axis, and b1, b2, b3. They are arranged according to the axis. The intervals of a1, a2, a3... An on the vertical axis and the intervals of b1, b2, b3... Bn on the horizontal axis all correspond to the time length of the frame. Each lattice point in the DP plane corresponds to a DP matching score which is a value indicating the Euclidean distance of each parameter of a1, a2, a3... And each parameter of b1, b2, b3. It is attached. For example, the grid points positioned by a1 and b1 include the Euclidean parameters of the parameters obtained from the first frame of the series of exemplary voice data and the parameters obtained from the first frame of the series of synthesized voice data. A value indicating the distance is associated. After the singing comparison unit 113 forms the DP plane having such a structure, the entire path from the lattice point (starting end) positioned by a1 and b1 to the lattice point (ending point) positioned by an and bn For each searched route, the DP matching score of each lattice point traced from the beginning to the end is accumulated, and the minimum accumulated value is obtained. The path with the smallest accumulated value of the DP matching score is considered as a scale of expansion / contraction when the time axis of each frame of the synthesized speech data is expanded / contracted in accordance with the time axis of the exemplary speech data.

  Then, the singing comparison unit 113 specifies a path on which the accumulated value of the DP matching score is minimum from the DP plane, and an alignment process that is a process of expanding and contracting the time axis of the synthesized speech data according to the content of the specified path I do. Specifically, the synthesized speech data is such that the DP matching score of each lattice point on the path specified from the DP plane represents the Euclidean distance of the parameter obtained from the frame having the same position on the time axis. After rewriting the contents of the time stamp of each frame, the frames having the same position on the time axis are sequentially associated as a set. For example, in the path marked on the DP plane shown in FIG. 4, it can be seen that the starting point positioned by a1 and b1 advances to the lattice point positioned by upper right a2 and b2. In this case, since the positions on the time axis of the frames a2 and b2 are the same from the beginning, it is not necessary to rewrite the contents of the time stamp of the frame b2. Furthermore, in this route, it can be seen that the grid point positioned by a2 and b2 advances from the grid point positioned by a2 and b3 on the right. In this case, not only the frame b2 but also the frame b3 need to have the same position on the time axis as the frame a2, so that the time stamp paired with the frame b3 is one frame earlier. Replace with As a result, the frame a2 and the frames b2 and b3 are associated as a set of frames having the same position on the time axis. Such time stamp replacement and frame association are performed for all frame sections from b1 to bn. Thereby, even if the sound generation timing of singing synthesis and the sound generation timing of the model voice are shifted, frames (phonemes) having the same position on the time axis can be associated with each other. The above is the mechanism of DP matching.

  FIG. 5 is a diagram illustrating an example of correspondence between model voices and synthesized voices. FIG. 5A shows an example of a graph showing the temporal change of the pitch of the synthesized speech, and FIG. 5B shows an example of a graph showing the temporal change of the pitch of the exemplary speech. In the figure, the sound generation timing t11 of the synthesized voice is associated with the sound generation timing t21 of the model voice, and the sound generation timing t12 of the synthesized voice and the sound generation timing t22 of the model voice are associated with each other.

  Returning to the description of FIG. The singing score data correction unit 114 corrects the singing score data based on the difference detected by the singing comparison unit 113. More specifically, the singing score data correction unit 114 corrects the pitch data and the pronunciation timing data constituting the singing score data in a direction that eliminates the difference between the synthesized voice data and the model voice data. For the pitch, the singing score data correction unit 114 determines the pitch data value included in the singing score data based on the pitch of the model voice data, the pitch of the synthesized voice data, and the corresponding part of the model voice and the synthesized voice. Correction is made so that the difference between the pitch of the data and the pitch of the synthesized speech corresponding to the pitch becomes small. Note that the amount of correction in this process may be, for example, correcting the value of the pitch data so that the pitch of the synthesized speech matches the pitch of the model speech, or, for example, the difference in which the difference between the two is detected You may make it correct so that it may become substantially half. Further, the difference between the pitch of the model voice and the pitch of the synthesized voice may be corrected so as to be equal to or less than a predetermined threshold value. In short, the singing score data correction unit 114 may correct the value of the pitch data included in the singing score data so that the difference between the pitch of the synthesized voice and the pitch of the model voice becomes small.

Further, the singing score data correcting unit 114 sets the value of the pronunciation timing data included in the singing score data so that the difference between the pronunciation timing detected from the model voice data and the pronunciation timing detected from the synthesized voice data becomes small. Correct it. This correction amount is also the same as the pitch correction described above, and the value of the sound generation timing data may be corrected so that the sound generation timing of the synthesized sound matches the sound generation timing of the model sound.
FIG. 6 is a diagram illustrating an example of the content of the corrected singing score data. As shown in the figure, the pitch and the pronunciation start timing and the pronunciation end timing of each phoneme are corrected according to the model voice.
The singing score data correction unit 114 stores the singing score data obtained by correcting each feature data in the corrected singing score data storage area 144 as corrected singing score data.

<B: Operation>
Next, the operation of this embodiment will be described. When the user performs an operation to correct the singing score data using the operation unit 16, the CPU 11 first performs the process of the singing synthesis unit 111 according to a signal output from the operation unit 16. That is, the CPU 11 generates synthesized speech data from the singing score data stored in the singing score data storage area 143 with reference to the Timbre database 141 and the phonological template database 142.

  Next, the CPU 11 performs the processing of the voice reproduction unit 112 and the song comparison unit 113 described above. That is, the CPU 11 reads out and reproduces the model voice data from the model voice data storage area 145, associates the reproduced model voice data with the synthesized voice data in the time axis direction, and sets the characteristics of each voice (for each pitch and phoneme). Are detected and compared.

  Next, the CPU 11 performs processing of the singing score data correction unit 114 described above. In other words, the CPU 11 modifies the feature data included in the singing score data based on the comparison result so that the difference between the model voice and the synthesized voice becomes small. CPU11 memorize | stores the song score data which corrected the characteristic data in the after-correction song score data storage area 144 as after-correction song score data.

When the CPU 11 finishes the singing score data correction process, the user performs an operation for performing singing synthesis using the operation unit 16. In response to the signal output from the operation unit 16, the CPU 11, from the singing score data stored in the post-correction singing score data storage area 144, the synthesized voice data representing the speech waveform corresponding to the singing score data (third Audio waveform data) is generated. This process is the same as the process performed by the singing voice synthesizing unit 111 described above, and a detailed description thereof is omitted here.
The CPU 11 supplies the generated synthesized voice data to the voice processing unit 18 and emits it as a sound from the speaker 19. Thereby, the sound represented by the singing score data corrected based on the model voice is emitted from the speaker 19.

  As described above, according to the present embodiment, the difference between the model voice and the synthesized voice is automatically analyzed, and the synthesized voice data is automatically corrected, thereby making it possible to more easily generate the synthesized voice of the desired quality. Can do. That is, according to the present embodiment, the CPU 11 generates synthesized voice data from the singing score data, compares the generated synthesized voice data with the model voice data representing the actual singing voice, and determines both of them according to the comparison result. The singing score data is corrected so that the difference becomes smaller. At this time, since the model voice data used as the comparison target is data representing the actual singing voice, the singing synthesized sound represented by the modified singing score data is closer to the actual singing voice and is more natural. It will be a thing.

  Moreover, in this embodiment, the singing composition which the corrected singing score data produced | generated by using the audio | voice data sung by the way of singing (intonation, singing technique, etc.) suitable for a user's preference as model audio | voice data represents The sound becomes a singing voice closer to the user's preference. Thus, according to the present embodiment, the user simply prepares the model voice data that suits his / her preference, and performs the singing synthesized sound without performing complicated operations such as correction of various parameters. It can be tailored to your taste.

<C: Modification>
As mentioned above, although embodiment of this invention was described, this invention is not limited to embodiment mentioned above, It can implement with another various form. An example is shown below. In addition, you may combine each following aspect suitably.
(1) In the above-described embodiment, the CPU 11 corrects the singing score data, and stores the corrected singing score data as the corrected singing score data in the corrected singing score data storage area 144. Not limited to this, as illustrated in FIG. 7, the CPU 11 may overwrite the singing score data storage area 143. In this case, the CPU 11 may make correction again using the corrected singing score data.
In the example shown in FIG. 7, the CPU 11 stores the corrected singing score data in the singing score data storage area 143. Then, the CPU 11 reads out the corrected singing score data from the singing score data storage area 143 (that is, acquires the corrected singing score data), generates synthesized voice data using the corrected singing score data, The generated synthesized voice data is used for comparison with the model voice data again, and the singing score data is corrected again using the comparison result.
As described above, if the singing synthesis parameter is repeatedly corrected, the singing score data can be made closer to the model voice, and the singing quality can be improved.
For example, if the user's singing voice is stored as a model voice, the singing synthesized sound can be brought close to the user's singing voice by repeatedly correcting the voice.

(2) In the above-described embodiment, as illustrated in FIG. 8, the user interface 115 may be provided, and the user may select a correction mode. In this case, for example, the CPU 11 has a plurality of correction levels that are different (for example, the pitch of the synthesized voice is matched with the pitch of the model voice, the difference between the pitch of the synthesized voice and the pitch of the model voice is halved, etc.) The feature data may be generated, and the generated feature data list may be displayed on the display unit 15. The user selects a desired correction mode from the displayed list, and the CPU 11 may correct the singing score data according to the selected content.

(3) In the above embodiment, the model voice data representing the pre-recorded singing voice is stored in the model voice data storage area 145 in advance, and the CPU 11 is stored in the model voice data storage area 145. However, the present invention is not limited to this, and the voice sung by the user may be input to the speech synthesizer 1 in real time as illustrated in FIG. In the example shown in FIG. 9, the user's singing voice is picked up by the microphone 17, converted into a voice signal (voice data), and output to the singing comparison unit 113. The singing comparison unit 113 compares the voice data representing the voice collected by the microphone 17 with the synthesized voice data generated by the singing voice synthesis unit 111.

(4) In the example shown in FIG. 9, accompaniment data may be further reproduced. FIG. 10 is a diagram illustrating an example of a functional configuration of the speech synthesizer 1 when reproducing accompaniment data. In this example, an accompaniment data storage area 146 (shown by a chain line in FIG. 1) for storing accompaniment data is provided in the storage unit 14, and accompaniment data is stored in advance in the accompaniment data storage area 146.
The audio mixing unit 119 mixes the signal representing the accompaniment sound supplied from the accompaniment reproduction unit 118 and the audio signal supplied from the microphone 17 and outputs the result to the speaker 19. Thereby, from the speaker 19, the accompaniment sound and the collected user's singing voice are emitted. Note that accompaniment playback and singing synthesis need to be performed at the same timing, and a control mechanism for that is necessary, but illustration of these is omitted to prevent the drawing from becoming complicated.
Thus, when collecting the singing voice, by reproducing the accompaniment sound, the user can sing in time with the singing synthesized sound represented by the singing score data.

(5) In addition, in the example shown in FIG. 10, it is also possible to record the input sound from the microphone. FIG. 11 shows an example of the functional configuration of the speech synthesizer 1 in this case. In the example shown in FIG. 11, the voice recording / playback unit 117 stores the voice data output from the microphone 17 in the model voice data storage area 145. In this case, the speech synthesizer 1 can repeatedly correct the singing score data using the recorded exemplary speech.

(6) In the above-described embodiment, the CPU 11 corrects the pitch data and the pronunciation timing data included in the singing score data, but the feature data to be corrected is not limited to this. For example, the CPU 11 may detect a lyric error. In this case, the CPU 11 may detect the phoneme difference between the model voice data and the synthesized voice data, and correct the phoneme data of the singing score data so that the difference becomes smaller. In this case, as a method for detecting a phoneme difference, for example, a difference between formants and cepstrum may be detected for the model voice data and the synthesized voice data, and voice recognition processing is performed on the model voice data. Thus, the phoneme may be detected.

(7) Further, the CPU 11 may detect the difference between the sound quality and the voice quality and correct the sound quality and the voice quality. In this case, the singing score data includes sound quality data and voice quality data indicating the sound quality and voice quality, and the CPU 11 detects formants from the model voice data and the synthesized voice data so that the difference between the detected formants becomes small. In addition, sound quality data and voice quality data may be corrected.

  As described above, the feature data to be corrected by the CPU 11 may be pitch data or sounding timing data indicating a temporal change in pitch shown in the above-described embodiment, and may be phoneme data, sound quality data, or voice quality data. There may be. As another example, for example, data representing the velocity (strongness) of sound may be used. As described above, the feature data corrected by the CPU 11 may be any data as long as it indicates the features of the melody and the features of each phoneme constituting the melody.

(8) Moreover, in the above-described embodiment, data indicating the composition of the music is included in the singing score data, and when the CPU 11 corrects the singing score data, the corresponding part in the music is corrected at the same time. good. For example, if the parameter at a certain position of No. 1 is corrected, the corresponding positions of No. 2 and No. 3 may be similarly corrected. In this case, the singing score data is divided into a plurality of time sections and includes section correspondence data indicating a correspondence relationship between the plurality of time sections. Then, the CPU 11 corrects the value of the feature data included in the singing score data in the same manner as in the above-described embodiment for at least one of the plurality of time intervals. Thereafter, the CPU 11 corrects the value of the feature data included in the singing score data for the other time intervals corresponding to the corrected time interval based on the interval corresponding data of the singing score data in the same manner as the time interval. To do.
In this way, for example, the correction of the second and third singing score data can be finished at the stage where the first of the music has been corrected, so that the processing time related to the correction can be shortened. it can.

(9) In the above-described embodiment, the singing voice synthesizing unit 111 reads the singing score data from the singing score data storage area 143, but the mode in which the singing voice synthesizing unit 111 acquires the singing score data is not limited thereto. For example, the singing score data may be received via a communication network such as the Internet. For example, the user performs an operation for inputting the singing score data using the operation unit 16, and the CPU 11 operates the operation unit. Singing score data may be generated in accordance with the signal output from 16, and any data may be used as long as the CPU 11 acquires the singing score data.

(10) In the embodiment described above, the speech synthesizer 1 corrects the singing score data generated in advance using the model voice data, but instead of this, the singing score data is obtained from the model voice data. You may make it produce | generate. FIG. 12 is a diagram illustrating an example of a functional configuration of the speech synthesizer 1 in this case. In the figure, the singing composition unit 111, the singing comparison unit 113, and the singing score data correction unit 114 are the same as those shown in FIG. In the figure, a singing analysis unit 116 analyzes audio data output from the microphone 17 and detects a pitch in units of a frame having a predetermined time length. Further, the singing analysis unit 116 analyzes the audio data output from the microphone 17 and detects the sound generation timing for each phoneme. The singing analysis unit 116 generates pitch data indicating the detected pitch, generates sounding timing data indicating the detected sounding timing, and generates song score data including the generated pitch data and sounding timing data. The generated singing score data is stored in the singing score data storage area 143.
By doing in this way, it is not necessary to prepare song score data beforehand, and the voice score input from the microphone 17 can be analyzed and song score data can be automatically generated.

In this embodiment, the speech input to the microphone 17 may be recognized as speech to extract lyrics (phoneme string), and phoneme data may be generated from the extraction result.
In this aspect, sound input to the microphone 17 may be recognized and formants may be detected to generate sound quality data and voice quality data.

(11) The program executed by the CPU 11 of the speech synthesizer 1 in the above-described embodiment is a magnetic recording medium (magnetic tape, magnetic disk, etc.), an optical recording medium (optical disk, etc.), a magneto-optical recording medium, a semiconductor memory, etc. It can be provided in a state where it is recorded on a computer-readable recording medium. It is also possible to download the speech synthesizer 1 via a network such as the Internet.

It is a block diagram which shows an example of the hardware constitutions of a speech synthesizer. It is a figure which shows an example of the content of song score data. It is a figure which shows an example of the screen displayed on a display part. It is a block diagram which shows an example of a functional structure of a speech synthesizer. It is a figure which shows DP matching. It is a figure which shows an example of the correspondence of a model audio | voice and a synthetic | combination audio | voice. It is a figure which shows an example of the content of the song score data after correction. It is a block diagram which shows an example of a functional structure of a speech synthesizer. It is a block diagram which shows an example of a functional structure of a speech synthesizer. It is a block diagram which shows an example of a functional structure of a speech synthesizer. It is a block diagram which shows an example of a functional structure of a speech synthesizer. It is a block diagram which shows an example of a functional structure of a speech synthesizer. It is a block diagram which shows an example of a functional structure of a speech synthesizer.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Voice synthesizer, 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... Memory | storage part, 15 ... Display part, 16 ... Operation part, 17 ... Microphone, 18 ... Speech processing part, 19 ... Speaker, 111 ... Singing Synthesizer, 112 ... voice playback unit, 113 ... singing comparison unit, 114 ... singing score data correction unit, 115 ... user interface, 116 ... singing analysis unit, 117 ... voice recording / playback unit, 118 ... accompaniment playback unit, 119 ... Voice mixing unit, 141 ... Timbre database, 142 ... Phoneme template database, 143 ... Singing score data storage area, 144 ... Modified song score data storage area, 145 ... Model voice data storage area, 146 ... Accompaniment data storage area.

Claims (9)

Singing score data representing melody composed of a sequence of phonemes, and singing score data obtaining means for obtaining singing score data including feature data representing features of each phoneme;
First voice waveform data acquisition means for acquiring first voice waveform data representing a voice waveform;
Second voice waveform data generating means for generating second voice waveform data representing a voice waveform corresponding to the singing score data from the singing score data acquired by the singing score data acquiring means;
Association means for associating the first speech waveform data and the second speech waveform data in a time axis direction;
First feature detection means for analyzing the first speech waveform data and detecting the feature according to an analysis result;
Second feature detection means for analyzing the second speech waveform data and detecting the feature according to an analysis result;
The first speech waveform data detected by the first feature detecting means is the feature data included in the singing score data acquired by the singing score data acquiring means in accordance with the result of the association by the associating means. And a feature data correcting unit that corrects the difference in the corresponding portion between the feature of the second voice waveform data detected by the second feature detecting unit and the feature of the second speech waveform data to be small.
Third voice waveform data generating means for generating third voice waveform data representing a voice waveform corresponding to the singing score data from the singing score data corrected by the feature data correcting means;
Output means for outputting third voice waveform data generated by the third voice waveform data generating means ;
Second correspondence means for associating the first voice waveform data and the third voice waveform data generated by the third voice waveform data generation means in a time axis direction;
Third feature detection means for analyzing the third speech waveform data and detecting the feature according to the analysis result;
The first speech waveform detected by the first feature detecting unit is the feature data included in the singing score data corrected by the feature data correcting unit according to the association result of the second association unit. Second feature data correction means for correcting the difference between the data feature and the feature of the third speech waveform data detected by the third feature detection means so as to be small.
Comprising
The third voice waveform data generating means is voice waveform data representing a voice waveform corresponding to the singing score data corrected by the feature data correcting means or the singing score data corrected by the second characteristic data correcting means. Is generated as the third speech waveform data
A speech synthesizer characterized by the above. The features include at least one of pronunciation timing of each phoneme constituting the melody, temporal change in pitch, phoneme of each phoneme constituting the melody, speech spectrum, volume, sound quality, and voice quality. The speech synthesizer according to claim 1 . The singing score data acquisition control means for supplying the singing score data to the singing score data acquiring means when the singing score data corrected by the characteristic data correcting means satisfies a predetermined condition. The speech synthesizer according to claim 1 or 2 . The singing score data is divided into a plurality of time sections, and includes section correspondence data indicating a correspondence relationship between the plurality of time sections,
The feature data correction means adds the singing score data acquired by the singing score data acquisition means to the singing score data acquisition means for at least one of the plurality of time intervals according to the association result of the association means. Correspondence between the feature data included in the feature of the first speech waveform data detected by the first feature detection unit and the feature of the second speech waveform data detected by the second feature detection unit In addition to making corrections so that the differences at the locations are smaller,
Based on the section corresponding data for other time interval corresponding to said time interval, the first to third aspects of feature data included in the song score data, characterized by modifying a modified embodiment of said time interval The speech synthesis device according to any one of the above. The first speech waveform data acquiring means, the audio data representing sound collected by the sound collecting unit, any one of claims 1 to 4, characterized in that to obtain, as the first sound data The speech synthesizer described in 1. The said feature data correction means corrects the feature data contained in the singing score data acquired by the said singing score data acquisition means by a several different correction | amendment aspect. The any one of Claim 1 thru | or 5 characterized by the above-mentioned. The speech synthesizer described in 1. The feature data correcting means includes feature data included in the singing score data acquired by the singing score data acquiring means, and features of the first speech waveform data detected by the first feature detecting means and the first of claims 1 to 6, characterized in that the difference in the corresponding portion of the feature of the second speech waveform data detected by the second feature detecting means is corrected so as to be substantially half or predetermined threshold The speech synthesis device according to any one of the above. Selecting means for selecting one of the plurality of correction modes according to information output from the user interface;
It said third audio waveform data generating means, according to claim, characterized in that to generate the third speech waveform data from the song score data including characteristic data corrected by the selected modified manner by the selection means 6 The speech synthesizer described in 1. The features include phonology;
The first feature detection unit and the second feature detection unit detect the phoneme by executing at least one of formant detection, cepstrum detection, and speech recognition processing. The speech synthesizer according to claim 1 .

Images