JP3294192B2 - Voice conversion device and voice conversion method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a voice conversion apparatus and a voice conversion method, and more particularly to a singer's singing voice in karaoke or the like, and a singing voice of a specific singer to be converted. The present invention relates to a voice conversion device and a voice conversion method for converting a voice as if another person were singing.

[0002]

2. Description of the Related Art There have been developed various voice converters for changing the frequency characteristics and the like of an input voice and outputting the converted voice. For example, some karaoke devices convert the pitch of a singer's singing voice into a male voice. Some voices are converted to female voices and vice versa (for example, Japanese Translation of International Patent Application No. Hei 8-5085).
No. 81).

[0003]

However, in the conventional voice converter, voice conversion (for example, male voice →
Female voices, female voices → male voices etc. were performed, but only the voice quality was changed, so it could not be converted, for example, to resemble the voice of a specific singer (for example, a professional singer) . Also, if there is a function like imitation that makes not only voice quality but also singing style similar to a specific singer, it is very interesting in karaoke equipment etc. It was impossible.

As a technique for solving these problems, a sine wave (SIN) component representing an audio signal by synthesizing a sine wave and a residual (RES) that cannot be represented by other sine wave components.
By the signal processing represented by an IDUAL) component, a singer's voice signal (sine wave component, residual component) is converted into a specific singer's voice signal (sine wave component, residual component) to be subjected to voice conversion.
, A voice conversion device that generates a voice signal reflecting the voice quality and singing style to be imitated and outputs the voice signal along with the accompaniment.

[0005] When such an audio converter is constructed,
Since the residual component includes a pitch component, the sine wave component and the residual component are subjected to voice conversion processing and synthesized, respectively.
The listener listens to the pitch components included in each of the sine wave component and the residual component. Therefore, when the pitch components included in each of the sine wave component and the residual component have different frequencies, there is a possibility that the naturalness of the voice that has been subjected to the voice conversion processing may be impaired. Therefore, an object of the present invention is to
An object of the present invention is to provide a voice conversion device and a voice conversion method that can perform voice conversion without impairing the naturalness of voice.

[0006]

According to a first aspect of the present invention, there is provided a sine wave component extracting means for extracting a sine wave component from an input voice signal, and the sine wave component extracting means. A residual component extracting means for extracting a residual component other than the sine wave component extracted from the input audio signal, and a sine wave component extracted by the sine wave component extracting means as a sine wave component of the target audio signal. A sinusoidal wave component transforming means for transforming the residual component extracted by the residual component extracting means based on the residual component of the target audio signal; and a residual component transforming means for transforming the residual component extracted by the residual component extracting means. Removing means for removing the pitch component of the residual component obtained by the component deforming means and its overtone component; a sine wave component deformed by the sine wave component deforming means; And it is characterized by comprising synthesizing means for its harmonic component combines the removed residual component, a.

According to a second aspect of the present invention, in the configuration of the first aspect, a pitch of a sine wave component of the input audio signal is obtained.
A pitch determining unit is provided which sets any one of a pitch of a sine wave component of the target audio signal and a pitch of a sine wave component obtained by the sine wave component deforming unit as a pitch of an attenuation peak in the removing unit. And

According to a third aspect of the present invention, in the configuration of the first aspect, when the removing unit holds the residual component on a frequency axis, the removing unit removes the attenuation peak determined by the pitch determining unit. It is a comb filter having a pitch.

According to a fourth aspect of the present invention, in the configuration of the first aspect, when the removing unit holds the residual component on a time axis, the removing unit removes the attenuation peak determined by the pitch determining unit. It is characterized in that it is a comb filter having a delay filter whose delay time is the reciprocal of the pitch.

According to a fifth aspect of the present invention, there is provided a component extracting step of extracting a sine wave component and a residual component other than the sine wave component from the input voice, and converting the extracted sine wave component into the target voice. A sine wave component deformation step of deforming based on a sine wave component, a residual component deformation step of deforming the extracted residual component based on a residual component of the target sound, and a residual component deformation step. A removing step of removing the pitch component and its overtone component of the obtained residual component; removing the sine wave component deformed in the sine wave component deforming step; and removing the pitch component and its harmonic component obtained in the removing step. And a combining step of combining the obtained residual component.

According to a sixth aspect of the present invention, in the configuration of the fifth aspect, a pitch of a sine wave component of the input voice, a pitch of a sine wave component of the target voice, and a sine wave obtained by the sine wave component deforming means. A pitch determining step is provided, in which any one of the pitches of the wave components is set as the pitch of the attenuation peak in the removing means.

According to the present invention, the sine wave component and the residual component extracted from the input audio signal are respectively modified based on the sine wave component or the residual component of the target audio signal. Next, before synthesizing the transformed sine wave component and the residual component, the pitch component of the residual component and its harmonic component are removed. Therefore, finally, only the pitch component of the sine wave component is heard, and the naturalness of the sound can be improved.

[0013]

DESCRIPTION OF THE PREFERRED EMBODIMENTS [1] Outline Processing of Embodiment First, outline processing of an embodiment will be described. [1.1] Step S1 First, the voice (input voice signal) of the singer (me) who wants to imitate is FFT (Fast Fourie Tr) in real time.
SMS (Spectral Modeling Synthesi) including ansform
s) Perform analysis to extract a sine wave component (Sine component) for each frame, and generate a residual component (Residual component) Rme for each frame from the input audio signal and the sine wave component. In parallel with this, it is determined whether or not the input audio signal is unvoiced (including non-voiced sound). If the input audio signal is unvoiced, the following steps S2 to S6 are not performed and the input audio signal is output as it is. Become. In this case,
As the SMS analysis, pitch synchronous analysis that changes the analysis window width according to the pitch in the previous frame is employed.

[1.2] Step S2 Next, when the input audio signal is a voiced sound, pitch (Pitch), amplifier (Amplitude) and spectral which are original attribute data are further extracted from the extracted sine wave component. -Extract a shape (Spectral Shape). Further, the extracted pitch and amplifier are separated into a vibrato component and components other than the vibrato component.

[1.3] Step S3 Attempt to imitate from the attribute data (target attribute data = pitch, amplifier and spectral shape) of the singer to be imitated (Target) stored in advance (target attribute). The target attribute data (= pitch, amplifier, and spectral shape) of the frame corresponding to the frame of the input voice signal of the singer (me) to be extracted is extracted.
In this case, the singer trying to imitate (m
If target attribute data of a frame corresponding to the frame of the input audio signal of e) does not exist, target attribute data is generated in accordance with a predetermined Easy Synchronization Rule, as described in detail later. Then, the same processing is performed.

[1.4] Step S4 Next, the original attribute data corresponding to the singer (me) to be imitated and the target attribute data corresponding to the singer to be imitated are appropriately selected and combined. , New attribute data (new attribute data = pitch, amplifier, and spectral shape). In addition, when using as a simple voice conversion instead of imitation,
It is also possible to obtain new attribute data by calculation based on both the original attribute data and the target attribute data, such as obtaining the new attribute data as an average of the original attribute data and the target attribute data.

[1.5] Step S5 The sine wave component SINnew of the frame is determined based on the new attribute data obtained in the subsequent step. Further, the sine wave component SINnew ′ is modified to generate a sine wave component SINnew ′ by modifying the amplifier, spectral shape and the like. [1.6] Step S6 Further, the residual component R of the input audio signal obtained in step S1
me (f) is deformed based on the residual component Rtar (f) of the target to obtain a new residual component Rnew (f).

[1.7] Step S7 Also, the pitch Pme-str of the sine wave component of the input voice signal obtained in step S1, the pitch Ptar-sta of the singer's sine wave component to be imitated (Target), Either the pitch Pnew of the sine wave component SINnew generated in step S5 or the pitch Patt of the deformed sine wave component SINnew '(basically the pitch Patt), the optimum pitch of the comb filter (the pitch of the comb filter) : Pcomb). [1.8] Step S8 Subsequently, a comb filter is formed based on the obtained pitch Pcomb, and the residual component Rnew obtained in step S6.
By filtering (f), the pitch component and its harmonic components are removed from the residual component Rnew (f), and a new residual component Rnew ′ (f) is obtained.

[1.9] Step S9 Then, the sine wave component SINnew 'obtained in step S5
And the new residual component Rnew '(f) obtained in step S8, and then inverse FFT is performed to obtain a converted audio signal. [1.10] Summary According to the converted audio signal obtained as a result of these processes, the reproduced voice is the singing voice of the singer trying to imitate, as if another singer (the target singer). It becomes like a singing voice. Furthermore, since the pitch component and its overtone component are removed from the residual component Rnew (f), only the pitch component of the sine wave component is ultimately heard, and the naturalness of the sound is not impaired. .

[2] Detailed Configuration of Embodiment Next, an embodiment of the present invention will be described with reference to the drawings. 1 and 2 show detailed configuration diagrams of the embodiment. The present embodiment is an example in which the voice conversion device (voice conversion method) according to the present invention is applied to a karaoke device and configured as a karaoke device that can perform imitation.
In FIG. 1, the microphone 1 collects the voice of a singer (me) who wants to imitate and outputs the voice to the input audio signal cutout unit 3 as an input audio signal Sv.

In parallel with this, the analysis window generator 2 sets a fixed multiple of the pitch cycle detected in the previous frame (for example,
An analysis window (for example, a hamming window) AW having a period of 3.5 times or the like is generated and output to the input audio signal cutout unit 3. When the initial state or the previous frame is an unvoiced sound (including silence), an analysis window having a fixed period set in advance is output to the input audio signal cutout unit 3 as the analysis window AW. Thus, the input audio signal cutout unit 3 multiplies the input analysis window AW by the input audio signal Sv, cuts out the input audio signal Sv in frame units, and outputs it to the fast Fourier transform unit 4 as a frame audio signal FSv. . More specifically, the relationship between the input audio signal Sv and the frame is
As shown in FIG. 3, each frame FL is set so as to partially overlap the previous frame FL.

Then, the frame sound signal FSv is analyzed and processed by the fast Fourier transform unit 4 and a local peak is detected by the peak detecting unit 5 from the frequency spectrum output from the fast Fourier transform unit 4 as shown in FIG. Is detected. More specifically, a local peak marked with “x” is detected in the frequency spectrum as shown in FIG. This local peak is represented as a combination of a frequency value and an amplifier (amplitude) value. That is, as shown in FIG. 4, (F0, A0), (F1, A1),
Local peaks are detected and represented for each frame as (F2, A2),..., (FN, AN).

Then, as schematically shown in FIG. 3, one set (hereinafter, referred to as a local peak set) is output to the unvoiced / voiced detecting unit 6 and the peak linking unit 8 for each frame. The unvoiced / voiced detection unit 6 detects that the voice is unvoiced ('t', 'k', etc.) according to the magnitude of the high-frequency component based on the input local peak for each frame, and performs unvoiced / voiced detection. The signal U / Vme is output to the pitch detection unit 7, the easy synchronization processing unit 22, and the crossfader 30. Alternatively, it is detected on the time axis that the voice is unvoiced according to the number of zero crossings per unit time ('s', etc.), and the original unvoiced / voiced detection signal U / Vme is detected by the pitch detection unit 7 by the easy synchronization processing. It outputs to the section 22 and the crossfader 30.

Further, if the unvoiced / voiced detection unit 6 does not detect that the input frame is unvoiced, it outputs the input local peak set to the pitch detection unit 7 as it is. The pitch detector 7 detects the pitch Pme of the frame corresponding to the local peak set based on the input local peak set. As a more specific method of detecting the pitch Pme of a frame, for example, Mahe
r, RCandJ.W.Beauchamp: "Fundamental Frequency Esti
mation of Musical Signal using a two-way Mismatch
Procedure "(Journal of Acounstical Society of Amer
ica95 (4): 2254-2263).

Next, the local peak set output from the peak detecting unit 5 is determined by the peak linking unit 8 to be linked with the preceding and succeeding frames, and the local peaks recognized as linked are formed into a series of data strings. A linking process for connecting a local peak to the data is performed. Here, this cooperation processing will be described with reference to FIG.
Now, assume that a local peak as shown in FIG. 5A is detected in the previous frame, and a local peak as shown in FIG. 5B is detected in the current frame.

In this case, the peak coordinating unit 8 calculates each local peak (F0, A0) detected in the previous frame,
(F1, A1), (F2, A2), ..., (FN, A
It is checked whether the local peak corresponding to N) has been detected in the current frame. The determination as to whether or not there is a corresponding local peak is made based on whether or not the local peak of the current frame is detected within a predetermined range centered on the frequency of the local peak detected in the previous frame. More specifically, in the example of FIG. 5, the local peaks (F0, A0), (F1, A1), (F2, A2).
, The corresponding local peak is detected, but for the local peaks (FK, AK), (FIG. 5
(See (A)) and the corresponding local peak (see FIG. 5B) is not detected.

When the corresponding local peaks are detected, the peak linking unit 8 connects the local peaks in chronological order and outputs them as a set of data strings. If the corresponding local peak is not detected, the data is replaced with data indicating that there is no corresponding local peak for the frame. Here, FIG. 6 shows an example of changes in the frequency F0 and the frequency F1 of the local peak over a plurality of frames. Such changes are caused by the amplifiers (amplitude) A0, A
1, A2,... Are similarly recognized. In this case, the data string output from the peak linking unit 8 is a discrete value output at every frame interval.

The peak value output from the peak linking unit 8 is hereinafter referred to as a deterministic component. This means a component that is deterministically replaced as a sine wave element in the original signal (that is, the audio signal Sv). Further, each of the replaced sine waves (strictly speaking, frequency and amplifier (amplitude) which are parameters of the sine wave) will be referred to as sine wave components. Next, the interpolation synthesizing unit 9 performs an interpolation process on the deterministic component output from the peak linking unit 8, and performs a waveform synthesis based on the deterministic component after the interpolation using a so-called oscillator method. In this case, the interpolation is performed at intervals corresponding to the sampling rate (for example, 44.1 KHz) of the final output signal output from the output unit 34 described later. The solid line shown in FIG. 6 described above shows an image when the interpolation processing is performed on the frequencies F0 and F1 of the sine wave components.

[2.1] Configuration of Interpolation Synthesis Unit The configuration of the interpolation synthesis unit 9 is shown in FIG. The interpolation / synthesis unit 9 includes a plurality of partial waveform generation units 9a, and each of the partial waveform generation units 9a adjusts a frequency (F0, F1,...) And an amplifier (amplitude) of a designated sine wave component. Generates a corresponding sine wave. However, the sine wave components (F0, A0), (F1, A1), (F2,
Since A2),... Change every moment according to the interpolation interval, the waveform output from each partial waveform generator 9a becomes a waveform according to the change. That is, the sine wave components (F0, A0) from the peak linking unit 8,
(F1, A1), (F2, A2),... Are sequentially output, and interpolation processing is performed for each of the sine wave components. Therefore, each partial waveform generation unit 9a determines the frequency and amplitude within a predetermined frequency domain. Output a waveform that fluctuates. Then, the waveforms output from the respective partial waveform generators 9a are added and synthesized in an adder 9b. Therefore, the output signal of the interpolation / synthesis unit 9 is a sine wave component synthesized signal SSS obtained by extracting a deterministic component from the input audio signal Sv.

[2.2] Operation of Residual Component Detecting Unit Next, the residual component detecting unit 10 calculates a deviation between the sine wave component synthesized signal SSS output from the interpolation synthesizing unit 9 and the input audio signal Sv. A residual component signal SRD (time waveform) is generated. This residual component signal SRD contains a lot of unvoiced components included in the voice. On the other hand, the above-mentioned sine wave component composite signal SSS corresponds to a voiced component. By the way, in order to resemble the voice of the singer who becomes the target (Target), if only the voiced sound is processed, it is not necessary to process the unvoiced sound. Therefore, in the present embodiment, speech conversion processing is performed on a deterministic component corresponding to a voiced vowel component. More specifically, regarding the residual component signal SRD,
The fast Fourier transform unit 11 converts the signal into a frequency waveform, and the obtained residual component signal (frequency waveform) is stored in the residual component storage unit 12 as Rme (f).

[2.3] Operation of Average Amplifier Operation Unit On the other hand, as shown in FIG. 8A, sine wave components (F0, A) output from the peak detection unit 5 via the peak linking unit 8
0), (F1, A1), (F2, A2),..., (F (N
-1), A (N-1)) N sine wave components (hereinafter, these are collectively referred to as Fn and An. N = 0 to (N-1).)
Is held in the sine wave component holding unit 13, and the amplifier An is input to the average amplifier operation unit 14, and the average amplifier Ame is calculated for each frame by the following equation. Ame = Σ (An) / N

[2.4] Operation of Amplifier Normalization Unit Next, in the amplifier normalization unit 15, each amplifier An is normalized by the average amplifier Ame by the following equation, and the normalized amplifier A'n
Ask for. A'n = An / Ame [2.5] Operation of Spectral Shape Computing Unit Then, in the spectral shape computing unit 16,
As shown in FIG. 8B, an envelope (envelope) having a sine wave component (Fn, A'n) obtained by the frequency Fn and the normalizing amplifier A'n as a break point has a spectral shape Sme (f). Generate as In this case, the value of the amplifier at the frequency between the two break points is calculated by, for example, linearly interpolating the two break points. The method of interpolation is not limited to linear interpolation.

[2.6] Operation of Pitch Normalization Unit Subsequently, the pitch normalization unit 17 normalizes each frequency Fn with the pitch Pme detected by the pitch detection unit 7 to obtain a normalized frequency F'n. F′n = Fn / Pme As a result, the original frame information holding unit 18 obtains the average amplifier Ame, the pitch Pme, and the spectral shape Sme (f) which are the original attribute data corresponding to the sine wave component included in the input audio signal Sv. ), And hold the normalized frequency F'n. In this case, the normalized frequency F'n is
It represents the relative value of the frequency of the harmonic train, and need not be retained if the harmonic structure of the frame is treated as a complete harmonic structure. In this case, if a male / female conversion is going to be performed, at this stage, the pitch is raised by an octave when the male to female conversion is performed, and the pitch is lowered by an octave when the female to male conversion is performed. It is preferable to perform a female voice pitch control process.

Subsequently, among the original attribute data held in the original frame information holding unit 18, the average amplifier Ame and the pitch Pme are further subjected to a filtering process and the like by the static change / vibrato change change separation unit 19. Thus, the static change component and the vibrato change component are separately held. In addition, it is also possible to configure so as to further separate a jitter variable component which is a higher frequency change component from a vibrato variable component. More specifically, the average amplifier Ame is separated and held as an average amplifier static component Ame-sta and an average amplifier vibrato component Ame-vib. Further, the pitch Pme is separated and held as a pitch static component Pme-sta and a pitch vibrato component Pme-vib, and the pitch static component Pme-sta is supplied to the pitch determination unit 40.

As a result, the original frame information data INFme of the corresponding frame is, as shown in FIG.
Average amplifier static component Ame-sta, average amplifier vibrato component Ame-vib, pitch static component Pme-sta, pitch vibrato component Pme-vib, which are original attribute data corresponding to the sine wave component of the input audio signal Sv, Spectral Shape S
me (f), the normalized frequency F'n, and the residual component Rme (f).

On the other hand, target frame information data INFtar composed of target attribute data corresponding to the singer to be imitated (target) is preliminarily analyzed and stored in a hard disk or the like constituting the target frame information storage unit 20 in advance. Have been. In this case, of the target frame information data INFtar,
As target attribute data corresponding to the sine wave component,
Average amplifier static component Atar-sta, average amplifier vibrato component Atar-vib, pitch static component Ptar-sta, pitch vibrato component Ptar-vib, spectral shape St
There is ar (f). Further, the target frame information data I
Among the NFtars, target attribute data corresponding to the residual component includes a residual component Rtar (f). Among them, the pitch static component Ptar-sta is also supplied to the pitch determination unit 40.

[2.7] Operation of Key Control / Tempo Change Unit Next, the key control / tempo change unit 21 responds to the synchronization signal SSYNC from the target frame information holding unit 20 based on the synchronization signal SSYNC from the sequencer 31. Of target frame information INFtar of the frame to be read and read target frame information data IN
The target attribute data constituting the Ftar is corrected, and the read target frame information INF is read.
It outputs tar and a target unvoiced / voiced detection signal U / Vtar indicating whether the frame is unvoiced or voiced.

More specifically, a key control unit (not shown) of the key control / tempo change unit 21 operates when a key of the karaoke apparatus is raised or lowered from a reference, and a pitch static component Ptar- which is target attribute data.
For the sta and pitch vibrato-like components Ptar-vib,
A correction process for raising and lowering by the same amount is performed. For example, 50
When the key is raised by [cent], the pitch static component P
The tar-sta and pitch vibrato-like component Ptar-vib must also be increased by 50 [cent]. When the tempo of the karaoke apparatus is raised or lowered, a tempo change unit (not shown) of the key control / tempo change unit 21 needs to read the target frame information data INFtar at a timing corresponding to the changed tempo. is there.

In this case, if the target frame information data INFtar corresponding to the timing corresponding to the required frame does not exist, the target frame information data of the two frames existing before and after the timing of the required frame is obtained. INFtar is read and these two target frame information data I
Interpolation processing is performed by NFtar, and target frame information data IN of the frame at the necessary timing is obtained.
Ftar, and eventually the target attribute data is generated.
In this case, the vibrato component (the average amp vibrato component Atar-vib and the pitch vibrato component Pt
Regarding ar-vib), if it is left untouched, the vibrato cycle itself changes and is inappropriate, so it is necessary to perform interpolation processing so that the cycle does not change. Alternatively, this problem can be avoided by holding the parameters of the vibrato cycle and the vibrato depth instead of the data representing the vibrato trajectory itself as the target attribute data and calculating the actual trajectory by calculation.

[2.8] Operation of Easy Synchronization Processing Unit Next, the easy synchronization processing unit 22 adds the original frame information data INFme to the frame of the singer who wants to imitate (hereinafter referred to as the original frame). Despite the presence of, the singer's frame that is the target of the corresponding singer (hereinafter referred to as the target frame)
If the target frame information data INFtar does not exist in the target frame, the target frame information data INFta
An easy synchronization process is performed using r as the target frame information data INFtar of the target frame.

Then, the easy synchronization processing section 22 generates target attribute data relating to the sine wave component (average amplifier static component Atar-sync-) out of target attribute data included in the replaced target frame information data INFtar-sync to be described later. sta, average amp vibrato component Atar-sync-vib, pitch static component Ptar-sync-st
a, a pitch vibrato-like component Ptar-sync-vib and a spectral shape Star-sync (f)) are output to the sine wave component attribute data selection unit 23. Further, the easy synchronization processing unit 22 stores target attribute data (residual component Rtar-sync (f)) relating to a residual component among target attribute data included in replaced target frame information data INFtar-sync described later. Difference component selection unit 25
Output to

In the processing in the easy synchronization processing section 22, the vibrato cycle itself changes with respect to the vibrato-like components (average amp vibrato-like component Atar-vib and pitch vibrato-like component Ptar-vib). Therefore, it is necessary to perform interpolation processing so that the period does not change. Alternatively, this problem can be avoided by holding the parameters of the vibrato cycle and the vibrato depth instead of the data representing the vibrato trajectory itself as the target attribute data and calculating the actual trajectory by calculation.

[2.8.1] Details of Easy Synchronization Process Here, the easy synchronization process will be described in detail with reference to FIGS. 9 and 10. FIG. 9 is a timing chart of the easy synchronization process, and FIG. 10 is a flowchart of the easy synchronization process. First, the easy synchronization processing unit 22 sets the synchronization mode = “0” indicating the method of the synchronization processing (step S11). This synchronization mode = "0" corresponds to the case of the normal processing in which the target frame information data INFtar exists in the target frame corresponding to the original frame.

Then, the original silent at a certain timing t /
The voiced detection signal U / Vme (t) changes from unvoiced (U) to voiced (V)
Is determined (step S12). For example, as shown in FIG. 9, at timing t = t1, the original unvoiced / voiced detection signal U / Vme (t) changes from unvoiced (U) to voiced (V). If it is determined in step S12 that the original unvoiced / voiced detection signal U / Vme (t) has changed from unvoiced (U) to voiced (V) (step S12; Yes), the previous timing t of timing t The original unvoiced / voiced detection signal U / Vme (t-1) at -1 is unvoiced (U) and the target unvoiced / voiced detection signal U / Vtar
It is determined whether or not (t-1) is silent (U) (step S18).

For example, as shown in FIG.
= T0 (= t1-1), the original unvoiced / voiced detection signal U
/ Vme (t-1) is unvoiced (U) and the target unvoiced / voiced detection signal U / Vtar (t-1) is unvoiced (U). In the determination in step S18, the original unvoiced / voiced detection signal U /
If Vme (t-1) is unvoiced (U) and the target unvoiced / voiced detection signal U / Vtar (t-1) is unvoiced (U) (step S18; Yes), the target frame is Since the target frame information data INFtar does not exist, the synchronization mode is set to "1" and the replacement target frame information data INFhold is set.
Is the target frame information of the frame existing in the backward direction (Backward) of the target frame.

For example, as shown in FIG.
= T1 to t2, since the target frame information data INFtar does not exist, the synchronization mode is set to “1”, and the replacement target frame information data INFhold is set to a frame existing in the backward direction of the target frame (that is, , Timing t
= Frame existing in t2 to t3) (target frame information data backward). Then, the process proceeds to step S
The process proceeds to step S15, and it is determined whether or not the synchronization mode is "0" (step S15).

If it is determined in step S15 that the synchronization mode is "0", the target frame information data INFtar (t) is present in the target frame corresponding to the original frame at the timing t, that is, the normal mode is set. Since this is a process, the replaced target frame information data INFtar-sync is set as target frame information data INFtar (t). INFtar-sync = INFtar (t) For example, as shown in FIG. 9, since the target frame at the timing t = t2 to t3 has target frame information data INFtar, INFtar-sync = INFtar (t) is set.

In this case, the replaced target frame information data INFtar-sync used in the subsequent processing
Target attribute data (average amplifier static component Atar-sync-sta, average amplifier vibrato-like component Atar-sy)
nc-vib, pitch static component Ptar-sync-sta, pitch vibrato-like component Ptar-sync-vib, spectral shape S
The tar-sync (f) and the residual component Rtar-sync (f) have substantially the following contents (step S16). Atar-sync-sta = Atar-sta Atar-sync-vib = Atar-vib Ptar-sync-sta = Ptar-sta Ptar-sync-vib = Ptar-vib Star-sync (f) = Star (f) Rtar-sync (f) = Rtar (f)

If it is determined in step S15 that the synchronization mode = "1" or the synchronization mode = "2", the target frame information data INFtar (t) is added to the target frame corresponding to the original frame at the timing t. ) Does not exist, the replaced target frame information data INF
Target frame information data INF for replacing tar-sync
Hold. INFtar-sync = INFhold For example, as shown in FIG. 9, the target frame at the timing t = t1 to t2 does not have the target frame information data INFtar, and the synchronization mode = “1”. = Target frame information data INF
Since tar exists, a process P1 is performed in which the replaced target frame information data INFtar-sync is set as the replacement target frame information data INFhold which is the target frame information data of the target frame at the timing t = t2 to t3. The target attribute data included in the replaced target frame information data INFtar-sync used is an average amplifier static component Atar-sync-st
a, average amp vibrato component Atar-sync-vib, pitch static component Ptar-sync-sta, pitch vibrato component P
tar-sync-vib, spectral shape Star-sync (f)
And the residual component Rtar-sync (f) (step S1).
6).

As shown in FIG. 9, the timing t =
In the target frame from t3 to t4, the target frame information data INFtar does not exist, and the synchronization mode becomes "2", but the timing t = t2 to t3.
Since the target frame has target frame information data INFtar, the replaced target frame information data INFtar-sync is set at timing t = t2 to
Replacement target frame information data INF, which is target frame information data of the target frame at t3
The processing P2 for setting the hold is performed, and the target attribute data included in the replaced target frame information data INFtar-sync used in the subsequent processing is the average amplifier static component A
tar-sync-sta, average amp vibrato component Atar-sync
-vib, pitch static component Ptar-sync-sta, pitch vibrato-like component Ptar-sync-vib, spectral shape Sta
It becomes r-sync (f) and the residual component Rtar-sync (f) (step S16).

In the determination in step S12, the original silent /
The voiced detection signal U / Vme (t) changes from unvoiced (U) to voiced (V)
(Step S12; No),
It is determined whether or not the target unvoiced / voiced detection signal U / Vtar (t) has changed from voiced (V) to unvoiced (U) (step S13). In the determination in step S13,
If the target unvoiced / voiced detection signal U / Vtar (t) changes from voiced (V) to unvoiced (U) (step S13; Yes), the original unvoiced / voiced signal at the previous timing t-1 of the timing t is output. The voiced detection signal U / Vme (t-1) is voiced (V) and the target unvoiced / voiced detection signal U / Vtar
It is determined whether or not (t-1) is voiced (V) (step S19).

For example, as shown in FIG.
Target unvoiced / voiced detection signal U / Vtar (t) at 3
Changes from voiced (V) to unvoiced (U) at timing t-1
= T2 to t3, the original unvoiced / voiced detection signal U / Vme
(t-1) is voiced (V) and the target unvoiced / voiced detection signal U / Vtar (t-1) is voiced (U). Step S
In the determination of No. 19, the original unvoiced / voiced detection signal U / Vme (t
-1) is voiced (V) and the target unvoiced / voiced detection signal U
If / Vtar (t-1) is voiced (V) (step S19; Yes), the target frame includes
Since the target frame information data INFtar does not exist, the synchronization mode is set to "2", and the replacement target frame information data INFhold is set as the target frame information of the frame existing in the forward direction of the target frame.

For example, as shown in FIG.
= T3 to t4, since the target frame information data INFtar does not exist, the synchronization mode is set to "2" and the replacement target frame information data INFhold is set to a frame existing in the forward direction of the target frame (i.e., , Timing t
= Frame existing at t2 to t3). Then, the process proceeds to step S
The process proceeds to step S15, and it is determined whether or not the synchronization mode is "0" (step S15), and thereafter, the same processing is performed. If it is determined in step S13 that the target unvoiced / voiced detection signal U / Vtar (t) has not changed from voiced (V) to unvoiced (U) (step S1).
3; No), the original unvoiced / voiced detection signal U / Vme (t) at timing t changes from voiced (V) to unvoiced (U),
Alternatively, the target unvoiced / voiced detection signal U / Vtar (t)
Is changed from unvoiced (U) to voiced (V) (step S14).

In the determination in step S14, the original unvoiced / voiced detection signal U / Vme (t) at the timing t changes from voiced (V) to unvoiced (U), and the target unvoiced / voiced detection signal U / Vtar ( If t) changes from unvoiced (U) to voiced (V) (step S14; Ye)
s), the synchronization mode is set to "0", the replacement target frame information data INFhold is initialized (cleared), the process proceeds to step S15, and the same process is performed. In the determination of step S14, the original unvoiced / voiced detection signal U / V at timing t
me (t) does not change from voiced (V) to unvoiced (U), and
If the target unvoiced / voiced detection signal U / Vtar (t) has not changed from unvoiced (U) to voiced (V) (step S14; No), the process proceeds to step S15 as it is, and thereafter the same process is performed. I do.

[2.9] Operation of Sine Wave Component Attribute Data Selector Subsequently, the sine wave component attribute data selector 23 replaces the target frame information data INFtar-sync which has been input from the easy synchronization processor 22. Attribute data on sine wave components (average amplifier static component Atar-sync-s)
ta, average amp vibrato component Atar-sync-vib, pitch static component Ptar-sync-sta, pitch vibrato component P
tar-sync-vib and spectral shape Star-sync
(f)) and the new amplifier component Anew and the new pitch component Pn, which are new sine wave component attribute data, based on the sine wave component attribute data selection information input from the controller 29.
Generate ew and a new spectral shape Snew (f).

That is, the new amplifier component Anew is generated by the following equation. Anew = A * -sta + A * -vib (* is me or tar-sy
nc) More specifically, as shown in FIG. 8D, the new amplifier component Anew is replaced with the average amplifier static component Ame-st of the original attribute data.
a or the average amplifier static component Atar-sync-sta of the target attribute data and the average amplifier vibrato component Ame-vib of the original attribute data or the average amplifier vibrato component Atar-sync-vib of the target attribute data
Is generated as a combination of any one of the above. The new pitch component Pnew is generated by the following equation. Pnew = P * -sta + P * -vib (* is me or tar-sy
nc)

More specifically, as shown in FIG. 8D, the new pitch component Pnew is defined as the pitch static component Pme-sta of the original attribute data or the pitch static component Ptar-sync-sta of the target attribute data. It is generated as a combination of any one of the pitch vibrato component Pme-vib of the original attribute data and the pitch vibrato component Ptar-sync-vib of the target attribute data. The new spectral shape Snew (f) is generated by the following equation. Snew (f) = S * (f) (However, * is me or tar-sync)

By the way, generally, when the amplifier component is large, the sound becomes a bright sound which extends to a high frequency, and when the amplifier component is small, the sound becomes muffled. Therefore, regarding the new spectral shape Snew (f),
In order to simulate such a state, as shown in FIG. 11, the high frequency component of the spectral shape, that is,
A more realistic sound can be reproduced by performing and controlling spectral tilt correction for compensating for the inclination of the spectral shape of the high-frequency component according to the magnitude of the new amplifier component Anew. . Subsequently, the generated new amplifier component A
new, a new pitch component Pnew and a new spectral shape Snew (f)
Further modification is performed by the attribute data transformation unit 24 based on the sine wave component attribute data transformation information input from.
For example, a deformation such as extending the entire spectral shape is performed. The attribute data deforming unit 24 supplies the pitch Patt of the sinusoidal component after the deformation to the pitch determining unit 40.

[2.10] Operation of Residual Component Selection Unit On the other hand, the residual component selection unit 25 includes a target attribute included in the replaced target frame information data INFtar-sync input from the easy synchronization processing unit 22. Target attribute data (residual component Rtar-sync (f)) relating to the residual component of the data, the residual component signal (frequency waveform) Rme (f) held in the residual component holding unit 12 and input from the controller 29 Based on the residual component attribute data selection information to be generated, a new residual component Rnew (f), which is new residual component attribute data, is generated. That is, the new residual component Rnew (f) is generated by the following equation. Rnew (f) = R * (f) (* is me or tar-sync)

In this case, whether to select me or tar-sync is determined by the new spectral shape Sne.
It is more preferable to select the same as w (f). Further, with respect to the new residual component Rnew (f), in order to simulate a state similar to the new spectral shape, FIG.
As shown in FIG. 1, spectral tilt compensation (spec) for compensating the high-frequency component of the residual component, that is, the slope of the residual component in the high-frequency component portion, according to the magnitude of the new amplifier component Anew
By performing tral tilt correction) and controlling, more realistic sound can be reproduced.

[2.11] Operation of Sine Wave Component Generation Unit Subsequently, the sine wave component generation unit 26
4, a new sine wave component (F "0, F" 0, Fnew) in the frame based on the new amplifier component Anew, the new pitch component Pnew and the new spectral shape Snew (f) without or with the deformation. A "0),
(F "1, A" 1), (F "2, A" 2), ..., (F "(N-
1), N sine wave components of A "(N-1)) (hereinafter collectively referred to as F" n, A "n. N = 0 to (N-
1). ). More specifically, a new frequency F "n and a new amplifier A" n are obtained by the following equations. F "n = F'n.times.Pnew A" n = Snew (F "n) .times.Anew If it is considered as a model of a perfect harmonic structure, F" n = (n + 1) .times.Pnew.

[2.12] Operation of Sine Wave Component Deformer Further, the new frequency F "n and new amplifier A" n obtained
, Based on the sine wave component deformation information input from the controller 29 as necessary.
7 make a further deformation. For example, a modification is performed such that only the new-order amplifier A "n (= A" 0, A "2, A" 4,...) Of even-order components is increased (for example, doubled). As a result, it is possible to give the converted speech further variety.

[2.13] Operation of Pitch Determining Unit The pitch determining unit 40 of the comb filter includes the pitch detecting unit 7
Pme-str from the target frame information holding unit 20, pitch Pnew from the sine wave component attribute data selecting unit 23, and pitch Patt from the attribute data deforming unit 24 (basically Is the pitch Pat
t), the optimum pitch of the comb filter (pitch of the comb filter: Pcomb) is supplied to the comb filter processing unit 41. Here, the pitch of the comb filter (Pcomb)
Will be described. In the above description, the pitch Pcomb is generated from the pitch Patt after the attribute conversion by the attribute data deforming unit 24, but the present invention is not limited to this. For example, in the voice conversion processing, when the target pitch Ptar-sta is used for the pitch of the sine wave component and Rme (f) is used for the new residual component Rnew (f), the residual component is unnecessary. Pitch Pme-sta and pitch Pco
The pitch Pme-sta is used as mb. Conversely, in the voice conversion process, when the pitch Pme-sta is used for the pitch of the sine wave component and the target residual component Rtar-sync (f) is used for the new residual component Pnew (f), the pitch Pcomb is The pitch Ptar-sta is used.

Further, in the case of performing a pitch shift such as an octave in the attribute conversion as a final voice conversion process,
As the pitch Pcomb, when the residual component of the input voice is used for the pitch shift, the pitch Pme-sta is used. When the residual component of the target is used, the pitch Ptar-sta is used.
May be used. Further, when the residual components of the input voice and the target voice are interpolated at an arbitrary ratio, the pitch generated by interpolating the pitch Pme-sta and the pitch Ptar-sta at the same ratio is defined as: The pitch of the comb filter is Pcomb. As described above, in order to filter the residual component subjected to the voice conversion processing by the comb filter and remove the pitch component and its overtone component from the residual component, it is necessary to determine the optimum pitch Pcomb for the comb filter to be used. There is.

[2.14] Operation of Comb Filter Processing Unit The comb filter processing unit 41 uses the pitch Pcomb to
By forming a comb filter and filtering the residual component Rnew (f) with the comb filter, the residual component Rnew (f) is obtained.
The pitch component and its harmonic components are removed from (f), and the resulting component is supplied to the inverse fast Fourier transform unit 28 as a new residual component Rnew '(f). Here, FIG. 12 shows that the pitch Pcomb is 2
It is a conceptual diagram which shows the example of a characteristic of the comb filter in the case of 00 Hz. As described above, when the residual component is held on the frequency axis, a comb filter is formed on the frequency axis based on the pitch Pcomb.

[2.15] Operation of Inverse Fast Fourier Transform Unit Next, the inverse fast Fourier transform unit 28 calculates the new frequency F "n and new amplifier A" n (= new sine wave component) and new residual component. Rnew '(f) is stored in the FFT buffer,
Inverse FFT is sequentially performed, and the obtained time axis signals are overlapped so as to partially overlap, and an addition processing of adding them is performed to generate a converted voice signal which is a new voiced sound time axis signal. .

At this time, based on the sine wave component / residual component balance control signal input from the controller 29,
A more realistic voiced signal is obtained by controlling the mixing ratio of the sine wave component and the residual component. In this case, generally,
When the mixing ratio of the residual components is increased, a rough voice is obtained. In this case, when storing the new frequency F "n, the new amplifier A" n (= new sine wave component) and the new residual component Rnew (f) in the FFT buffer, they are converted at different pitches and at an appropriate pitch. Harmony can be obtained as a converted audio signal by further adding the sine wave component. Further, by giving a harmony pitch adapted to the accompaniment sound by the sequencer 31, musical harmony adapted to the accompaniment can be obtained.

[2.16] Operation of Crossfader Next, the crossfader 30 transmits the original unvoiced / voiced detection signal U /
If the input audio signal Sv is unvoiced (U) based on Vme (t), the input audio signal Sv is output to the mixer 30 as it is. When the input audio signal Sv is voiced (V), the converted audio signal output from the inverse fast Fourier transform converter 28 is output to the mixer 33. In this case, the reason why the cross fader 30 is used as the changeover switch is to prevent a click sound from occurring at the time of switch changeover by performing a crossfade operation.

[2.17] Operation of Sequencer, Sound Source Unit, Mixer, and Output Unit On the other hand, the sequencer 31 transmits sound source control information for generating a karaoke accompaniment sound to, for example, a MIDI (Musical Instrument).
rument Digital Interface) sound source section 3 as data etc.
Output to 2. Thereby, the sound source section 32 generates an accompaniment signal based on the sound source control information, and outputs the accompaniment signal to the mixer 33.
The mixer 33 mixes either the input audio signal Sv or the converted audio signal and the accompaniment signal, and outputs the mixed signal to the output unit 34. The output unit 34 has an amplifier (not shown), amplifies the mixed signal, and outputs it as an acoustic signal.

[3] Modifications of Embodiment [3.1] First Modification In the above description, either the original attribute data or the target attribute data is selectively used as the attribute data. It is also possible to obtain a converted audio signal having an intermediate attribute by performing an interpolation process using both the original attribute data and the target attribute data. However, according to such a configuration, a converted voice that is not similar to any of the singer trying to imitate and the singer to be imitated may be obtained. Also, especially when the spectral shape is obtained by interpolation processing, the singer trying to imitate pronounces "a", and the singer to be imitated pronounces "i". There is a possibility that sounds other than "A" or "I" may be output as converted voices, and care must be taken when handling them.

[3.2] Second Modification The extraction of the sine wave component is not limited to the method used in this embodiment. In short, it is only necessary to extract a sine wave included in the audio signal. [3.3] Third Modification In the present embodiment, the sine wave component and the residual component of the target are stored. Instead, the target speech itself is stored, read out, and subjected to real-time processing. The wave component and the residual component may be extracted. That is, processing similar to the processing performed on the voice of the singer trying to imitate in the present embodiment may be performed on the voice of the target singer.

[3.4] Fourth Modification In this embodiment, all of the pitch, amplifier, and spectral shape are handled as attribute data.
It is also possible to handle at least one of them. [3.5] Fifth Modification In the present embodiment, the residual component is held on the frequency axis. However, the present invention is not limited to this, and the residual component may be held on the time axis. FIG. 13 is a block diagram illustrating a configuration (part) of a modification of the above-described embodiment. FIG.
FIG. 3 is a block diagram illustrating an example of a configuration of a comb filter (delay filter). Note that the same reference numerals are given to portions corresponding to FIG. In the figure, a comb filter processing unit 42 constitutes a comb filter (delay filter) that uses a reciprocal of the pitch Pcomb determined by the pitch determination unit 40 as a delay time, and the comb filter has a residual component Rnew ( t) and filter the residual component Rne
It is supplied to the subtractor 43 as w '' (t). The subtracter 43 subtracts the filtered residual component Rnew '' (t) from the residual component Rnew (t) to remove the pitch component and its harmonic component from the residual component Rnew (t), It is supplied to the IFFT processing unit 8 as a residual component Rnew ′ (t). As described above, even when the residual component is processed on the time axis, it is possible to remove the pitch component and its harmonic component from the residual component Rnew (t), as in the above-described embodiment. Therefore, only the pitch component of the sine wave component is heard in the finally output sound, and the naturalness of the sound can be improved.

[4] Effects of Embodiment As a result, the singer's song is output together with the karaoke accompaniment. The voice quality and singing style are greatly affected by the target, and the voice quality and singing of the target itself are obtained. One. In this way, a song as if imitating the target is output. Further, since the pitch component and its harmonic components are removed from the residual component Rnew (f), only the pitch component of the sine wave component is finally heard, and the naturalness of the sound is not impaired. .

[0074]

As described above, according to the present invention, the sine wave component and the residual component extracted from the input voice signal are respectively transformed based on the sine wave component or the residual component of the target voice. Then, before synthesizing the sine wave component and the residual component, the pitch component of the transformed residual component and its overtone component are removed, so that the naturalness of the voice obtained by the synthesis is impaired. Without conversion, a converted voice that reflects the voice quality and singing style of the target singer to be imitated can be easily obtained from the voice of the singer trying to imitate (input voice).

[Brief description of the drawings]

FIG. 1 is a block diagram (part 1) illustrating a configuration of an embodiment of the present invention.

FIG. 2 is a block diagram (part 2) showing a configuration of an embodiment of the present invention.

FIG. 3 is a diagram illustrating a state of a frame according to the embodiment.

FIG. 4 is an explanatory diagram for describing peak detection of a frequency spectrum in the embodiment.

FIG. 5 is a diagram illustrating cooperation of peak values for each frame in the embodiment.

FIG. 6 is a diagram illustrating a change state of a frequency value in the embodiment.

FIG. 7 is a diagram showing a change state of a deterministic component in a process in the embodiment.

FIG. 8 is an explanatory diagram of signal processing in the embodiment.

FIG. 9 is a timing chart of an easy synchronization process.

FIG. 10 is a flowchart of an easy synchronization process.

FIG. 11 is a diagram for explaining spectral tilt compensation of a spectral shape.

FIG. 12 shows characteristics of a comb filter (pitch Pcomb is 2
FIG. 6 is a conceptual diagram for explaining the case of 00 Hz).

FIG. 13 is a block diagram illustrating a configuration (part) of a voice conversion device according to a modification of the present invention.

FIG. 14 is a block diagram illustrating an example of a configuration of a comb filter (delay filter).

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Microphone, 2 ... Analysis window generation part, 3 ... Input audio signal extraction part, 4 ... Fast Fourier transform part, 5 ... Peak detection part, 6 ...
Unvoiced / voiced detection unit, 7: pitch extraction unit, 8: peak linking unit, 9: interpolation / synthesis unit, 10: residual component detection unit, 11: fast Fourier transform unit, 12: residual component holding unit, 13: sine Wave component holding unit, 14: average amplifier calculation unit, 15: amplifier normalization unit, 16: spectral shape calculation unit, 17
... Pitch normalizing section, 18 ... Original frame information holding section, 19
... Static change / vibrato change separation section, 20 ... Target frame information holding section, 21 ... Key control / tempo change section, 22 ... Easy synchronization processing section, 23 ... Sine wave component attribute data selection section, 24 ... Attribute Data transformation unit, 25: Residual component selection unit, 26: Sine wave component generation unit, 27: Sine wave component transformation unit, 28: Inverse fast Fourier transform unit, 29: Controller, 30: Crossfader, 31: Sequencer, 32 ... sound source section, 33 ... mixer, 34 ... output section, 40 ... pitch determination section, 41, 42 ...
Comb-shaped filter processing unit, 43 ... subtractor

────────────────────────────────────────────────── ─── Continuation of the front page Examiner Satoshi Watanabe (56) References JP-A-6-149242 (JP, A) (58) Field surveyed (Int. Cl. ⁷ , DB name) G10L 21/04

Claims (6)

(57) [Claims] 1. A sine wave component extracting unit for extracting a sine wave component from an input audio signal, and a residual component for extracting a residual component other than the sine wave component extracted by the sine wave component extracting unit from the input audio signal. A difference component extracting unit, a sine wave component deforming unit that deforms the sine wave component extracted by the sine wave component extracting unit based on a sine wave component of the target audio signal, and a sine wave component deforming unit extracted by the residual component extracting unit. Residual component deforming means for deforming the residual component based on the residual component of the target audio signal; and removing means for removing the pitch component of the residual component obtained by the residual component deforming means and its harmonic component. Synthesizing means for synthesizing a sine wave component deformed by the sine wave component deforming means and a residual component from which the pitch component and its harmonic component have been removed by the removing means. Speech conversion system which is characterized in that. 2. The audio conversion device according to claim 1, wherein a pitch of a sine wave component of the input audio signal, a pitch of a sine wave component of the target audio signal, and a sine wave component obtained by the sine wave component deformation unit. One of the pitches,
A voice conversion device comprising a pitch determination unit that sets a pitch of an attenuation peak in the removal unit. 3. The audio conversion device according to claim 1, wherein the removing unit has a pitch of an attenuation peak determined by the pitch determining unit when the residual component is held on a frequency axis. A voice conversion device characterized by being a shape filter. 4. The voice conversion device according to claim 1, wherein the removing unit, when holding the residual component on a time axis, calculates a reciprocal of a pitch of the attenuation peak determined by the pitch determining unit. An audio conversion device characterized by being a comb filter having a delay filter for setting a delay time. 5. A component extracting step of extracting a sine wave component and a residual component other than the sine wave component from the input voice, and extracting the extracted sine wave component based on a sine wave component of the target voice. A sinusoidal wave component deforming step of deforming; a residual component deforming step of deforming the extracted residual component based on a residual component of the target voice; and a residual component obtained in the residual component deforming step. A removing step of removing the pitch component and its overtone component, a sine wave component deformed in the sine wave component deforming step, and a residual component from which the pitch component and its harmonic component removed in the removing step are removed. And a synthesizing step of synthesizing. 6. The voice conversion method according to claim 5, wherein a pitch of a sine wave component of the input voice, a pitch of a sine wave component of the target voice, and a pitch of a sine wave component obtained by the sine wave component transforming means. A pitch determination step of setting any one of the above as a pitch of an attenuation peak in said removing means.