JP3949828B2 - Voice conversion device and voice conversion method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To convert a voice without imparing a natural property of the voice. SOLUTION: An input sound signal Sv corresponding to a voice of a singer whose vocal mimicry is carried out is analyzed to extract a sine wave component and a residual component Rme(f). The sine wave component is voice- converting-processed based on a sine wave component of a target sound signal (sound signal of a miming object). The residual component Rme(f) is voice- converting-processed based on a residual component of the target sound signal, and a pitch component and a harmonic are added to a resultant residual component therein by a comb-shaped filter processing part 41. A converted sound signal is provided by conducting inverse FFT(fast Fourier transform) transformation in an inverse FFT processing part 28, after an attribute-converted sine wave component SINnew' and a filter-processed residual component Rnew'(f) are synthesized.

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a voice conversion device and a voice conversion method, and in particular, a singing voice is sung by another person so that the singing voice of a singer becomes a voice of a specific singer who is a target of voice conversion, especially in karaoke or the like. The present invention relates to a voice conversion device and a voice conversion method for converting to a voice.
[0002]
[Prior art]
Various voice conversion devices that change the frequency characteristics of the input voice and the like have been developed. For example, in a karaoke device, the pitch of a singer's singing voice is converted to convert a male voice into a female voice. Some are converted into a voice or vice versa (for example, refer to Japanese Patent Publication No. 8-508581).
[0003]
[Problems to be solved by the invention]
However, in the conventional voice conversion device, although voice conversion (for example, male voice → female voice, female voice → male voice, etc.) is performed, it has only stopped changing the voice quality. For example, a specific singer (for example, It couldn't be converted to resemble the voice of a professional singer.
Also, if you have a function that imitates a specific singer, not only the voice quality, but also the way you sing, it will be very interesting in a karaoke device etc., but such processing is not possible with conventional speech conversion devices It was impossible.
[0004]
As a technique for solving these problems, signal processing represented by a sine wave (SIN) component that represents a speech signal by synthesis of a sine wave and a residual (RESIDUAL) component that cannot be represented by any other sine wave component, The voice signal (sine wave component, residual component) of a singer is transformed based on the voice signal (sine wave component, residual component) of a specific singer that is subject to speech conversion, An audio conversion device that generates an audio signal reflecting how to sing and outputs it together with the accompaniment can be considered.
[0005]
When such a speech conversion device is configured, since the residual component includes a pitch component, when the sine wave component and the residual component are respectively subjected to speech conversion processing and synthesized, the listener can obtain the sine wave component and the residual component. The pitch component included in each of the residual components is listened to.
In addition, when the pitch included in the residual component and its double voice are removed and combined with the sine wave component, the residual component does not have the pitch component, so that the pitch is not maintained, and the sine wave component and the residual sound component are not retained. Each difference component will be listened to.
Therefore, in any of the cases described above, the naturalness of the voice subjected to the voice conversion process may be impaired.
SUMMARY OF THE INVENTION An object of the present invention is to provide a voice conversion device and a voice conversion method that can convert voice without impairing the naturalness of the voice.
[0006]
[Means for Solving the Problems]
In order to solve the above-described problem, the configuration according to claim 1 includes a sine wave component extraction unit that extracts a sine wave component from an input audio signal, and a sine wave component other than the sine wave component extracted by the sine wave component extraction unit. Residual component extraction means for extracting a residual component from the input speech signal, and sine wave component extracted by the sine wave component extraction means based on the sine wave component of the target speech signal And controlling the slope of the deformed sine wave component according to the magnitude of the amp component of the deformed sine wave component. A sine wave component deforming means for transforming the residual component extracted by the residual component extracting means based on the residual component of the target audio signal And controlling the slope of the deformed residual component according to the magnitude of the amp component of the deformed sinusoidal component. A residual component deforming means, an adding means for adding a pitch component and its overtone component to the residual component obtained by the residual component deforming means, and a sine wave component deforming means. The tilt is controlled And a synthesizing unit for synthesizing the sine wave component and the residual component to which the pitch component and its harmonic component are added by the adding unit.
[0007]
According to a second aspect of the present invention, there is provided the pitch determining means according to the first aspect, wherein the pitch of the sine wave component obtained by the sine wave component deforming means is a pitch of a peak of a pass band in the additional means. The adding means adds the pitch component and its overtone component to the residual component by passing the residual component corresponding to the passband in which the pitch of the peak is determined by the pitch determining means. It is characterized by doing.
[0008]
The configuration according to claim 3 is the claim. 2 In the configuration described above, the addition unit is a comb filter having a passband peak pitch determined by the pitch determination unit when the residual component is held on the frequency axis. Yes.
[0009]
The configuration of claim 4 is the 2 In the configuration described above, the addition unit has a delay filter that uses a reciprocal of the pitch of the passband peak determined by the pitch determination unit as a delay time when the residual component is held on the time axis. It is a comb filter.
[0010]
According to a fifth aspect of the present invention, there is provided a component extraction step of extracting a sine wave component and a residual component other than the sine wave component from an input voice, and the extracted sine wave component is converted into a sine wave of a target voice. Deformation based on components And controlling the slope of the deformed sine wave component according to the magnitude of the amp component of the deformed sine wave component. A sine wave component deformation step, and deforming the extracted residual component based on the residual component of the target speech And controlling the slope of the deformed residual component according to the magnitude of the amp component of the deformed sinusoidal component. A residual component transformation step, an addition step of adding a pitch component and its harmonic component to the residual component obtained in the residual component transformation step, and a deformation in the sine wave component transformation step. The tilt is controlled And a synthesizing step for synthesizing the sine wave component and the residual component to which the pitch component and its harmonic component are added in the adding step.
[0011]
The configuration of claim 6 is the configuration of claim 5, wherein the sine wave component deformation Process A pitch determining step in which the pitch of the sine wave component obtained by the step is used as the pitch of the passband peak in the adding step. In the adding step, a pitch component and its overtone component are added to the residual component by passing the residual component corresponding to the passband in which the pitch of the peak is determined in the pitch determining step. Do It is characterized by that.
[0012]
According to the present invention, the sine wave component and the residual component extracted from the input audio signal are each transformed based on the sine wave component or the residual component of the target audio signal.
Next, before synthesizing the modified sine wave component and the residual component, the pitch component and its harmonic component are added to the residual component.
Therefore, finally, only the pitch component of the sine wave component is heard, and the naturalness of the sound can be improved.
[0013]
DETAILED DESCRIPTION OF THE INVENTION
[1] Outline processing of embodiment
First, an outline process of the embodiment will be described.
[1.1] Step S1
First, the voice (input audio signal) of the singer (me) trying to imitate is analyzed in real time by SMS (Spectral Modeling Synthesis) including FFT (Fast Fourie Transform), and the sine wave component (Sine component) in frame units And a residual component (Residual component) Rme in units of frames from the input audio signal and the sine wave component. In parallel with this, it is determined whether or not the input sound signal is an unvoiced sound (including silence). If the input sound signal is an unvoiced sound, the following steps S2 to S6 are not performed, and the input sound signal is output as it is. Become.
In this case, as the SMS analysis, pitch synchronization analysis is adopted in which the analysis window width is changed according to the pitch in the previous frame.
[0014]
[1.2] Step S2
Next, when the input audio signal is a voiced sound, a pitch (Pitch), an amplifier (Amplitude), and a spectral shape (Spectral Shape) which are original attribute data are further extracted from the extracted sine wave component. Further, the extracted pitch and amplifier are separated into components other than the vibrato component and the vibrato component.
[0015]
[1.3] Step S3
The input audio signal of the singer (me) who wants to imitate from the attribute data (target attribute data = pitch, amplifier and spectral shape) of the singer who is the target of the imitation (Target) stored in advance The target attribute data (= pitch, amplifier, and spectral shape) of the frame corresponding to the frame is extracted. In this case, when there is no target attribute data of the frame corresponding to the frame of the input voice signal of the singer (me) who tries to imitate, a predetermined easy synchronization rule as will be described later in detail. In accordance with (Easy Synchronization Rule), target attribute data is generated and the same processing is performed.
[0016]
[1.4] Step S4
Next, new attribute data (new attribute data = pitch) is obtained by appropriately selecting and combining the original attribute data corresponding to the singer (me) trying to imitate and the target attribute data corresponding to the singer to be imitated. , Amplifier and spectral shape). In addition, when used as mere speech conversion instead of imitation, new attribute data is calculated by calculation based on both original attribute data and target attribute data, such as obtaining new attribute data as an average of original attribute data and target attribute data. It is also possible to obtain
[0017]
[1.5] Step S5
Based on the new attribute data obtained subsequently, a sine wave component SINnew of the frame is obtained. Further, the sine wave component SINnew's amplifier, spectral shape, etc. are deformed to generate a sine wave component SINnew '.
[1.6] Step S6
Further, the residual component Rme (f) of the input speech signal obtained in step S1 is transformed based on the target residual component Rtar (f) to obtain a new residual component Rnew (f).
[0018]
[1.7] Step S7
Further, the pitch Patt of the deformed sine wave component SINnew ′ is set to the pitch (Pcomb) of the comb filter.
[1.8] Step S8
Subsequently, a comb filter is constructed based on the obtained pitch Pcomb, and the residual component Rnew (f) obtained in step S6 is filtered, so that the residual component Rnew (f) has a pitch component and its harmonics. The component is added to obtain a new residual component Rnew ′ (f).
[0019]
[1.9] Step S9
Then, after the sine wave component SINnew ′ obtained in step S5 and the new residual component Rnew ′ (f) obtained in step S8 are synthesized, inverse FFT is performed to obtain a converted speech signal.
[1.10] Summary
According to the converted audio signal obtained as a result of these processes, the reproduced voice is like a singing voice of a singer who tries to imitate, as if sung by another singer (target singer). Become. Furthermore, since the pitch component and its harmonic component are added to the residual component Rnew (f), only the pitch component of the sine wave component is finally heard, which may impair the naturalness of the sound. Absent.
[0020]
[2] Detailed configuration of the embodiment
Next, an embodiment of the present invention will be described with reference to the drawings.
1 and 2 are detailed configuration diagrams of the embodiment. This embodiment is an example of a case where the voice conversion device (voice conversion method) according to the present invention is applied to a karaoke device and configured as a karaoke device capable of performing imitation.
In FIG. 1, the microphone 1 collects the voice of a singer (me) trying to imitate and outputs it as an input audio signal Sv to the input audio signal cutout unit 3.
[0021]
In parallel with this, the analysis window generator 2 generates an analysis window (for example, a hamming window) AW having a fixed period (for example, 3.5 times) of the pitch period detected in the previous frame. And output to the input voice signal cutout unit 3. When the initial state or the previous frame is a silent sound (including silent sound), an analysis window having a preset fixed period is output to the input voice signal cutout unit 3 as an analysis window AW.
As a result, the input voice signal cutout unit 3 multiplies the input analysis window AW and the input voice signal Sv, cuts out the input voice signal Sv in units of frames, and outputs it to the fast Fourier transform unit 4 as the frame voice signal FSv. .
More specifically, the relationship between the input audio signal Sv and the frame is as shown in FIG. 3, and each frame FL is set to partially overlap the previous frame FL.
[0022]
The frame audio signal FSv is analyzed in the fast Fourier transform unit 4 and a local peak is detected by the peak detection unit 5 from the frequency spectrum output from the fast Fourier transform unit 4 as shown in FIG. .
More specifically, a local peak marked with x is detected for the frequency spectrum as shown in FIG. This local peak is expressed as a combination of a frequency value and an amplifier (amplitude) value.
That is, as shown in FIG. 4, local peaks are detected and expressed for each frame as (F0, A0), (F1, A1), (F2, A2),..., (FN, AN). It will be.
[0023]
Then, as schematically shown in FIG. 3, each frame is output to the unvoiced / voiced detection unit 6 and the peak link unit 8 as one set (hereinafter referred to as a local peak set). The unvoiced / voiced detection unit 6 detects unvoiced ('t', 'k', etc.) according to the magnitude of the high frequency component based on the input local peak for each frame, and detects unvoiced / voiced. The signal U / Vme is output to the pitch detection unit 7, the easy synchronization processing unit 22 and the cross fader 30. Alternatively, it is detected that the voice is unvoiced according to the number of zero crosses per unit time on the time axis (such as “s”), and the original voiceless / voiced detection signal U / Vme is detected by the pitch detector 7 and the easy synchronization process. Output to the unit 22 and the crossfader 30.
[0024]
Furthermore, if the unvoiced / voiced detection unit 6 does not detect that the input frame is unvoiced, the unvoiced / voiced detection unit 6 outputs the input local peak set to the pitch detection unit 7 as it is.
The pitch detector 7 detects the pitch Pme of the frame corresponding to the local peak set based on the input local peak set.
As a more specific method for detecting the frame pitch Pme, for example, Maher, RC and J.W. Beauchamp: “Fundamental Frequency Estimation of Musical Signal using a two-way Mismatch Procedure” (Journal of Acounstical Society of America 95 (4): 2254-2263).
[0025]
Next, the local peak set output from the peak detection unit 5 is determined in the peak linkage unit 8 for linkage between the previous and subsequent frames, and the local peaks recognized to be linked are local peaks so as to form a series of data strings. Cooperation process to connect
Here, this cooperation processing will be described with reference to FIG.
Assume that a local peak as shown in FIG. 5A is detected in the previous frame, and a local peak as shown in FIG. 5B is detected in the current frame.
[0026]
In this case, the peak link unit 8 performs local peaks corresponding to the local peaks (F0, A0), (F1, A1), (F2, A2),..., (FN, AN) detected in the previous frame. It is checked whether or not is detected even in the current frame. Whether or not there is a corresponding local peak is determined by whether or not the local peak of the current frame is detected within a predetermined range centered on the frequency of the local peak detected in the previous frame.
More specifically, in the example of FIG. 5, corresponding local peaks are detected for local peaks (F0, A0), (F1, A1), (F2, A2). For FK, AK (see FIG. 5A), the corresponding local peak (see FIG. 5B) is not detected.
[0027]
When the peak linking unit 8 detects corresponding local peaks, they are connected in time series order and output as a set of data strings. If no corresponding local peak is detected, the data is replaced with data indicating that there is no corresponding local peak for the frame.
Here, FIG. 6 shows an example of changes in the frequency F0 and the frequency F1 of the local peak over a plurality of frames. Such a change is similarly recognized for the amplifiers (amplitudes) A0, A1, A2,. In this case, the data string output from the peak cooperation unit 8 is a discrete value output at every frame interval.
[0028]
The peak value output from the peak cooperation unit 8 is hereinafter referred to as a deterministic component. This means that the original signal (that is, the audio signal Sv) is a component that is definitely replaced as a sine wave element. Further, each replaced sine wave (strictly speaking, the frequency and amplifier (amplitude) which are parameters of the sine wave) will be referred to as a sine wave component.
Next, the interpolation synthesis unit 9 performs an interpolation process on the deterministic component output from the peak cooperation unit 8 and performs waveform synthesis using a so-called oscillator method based on the deterministic component after the interpolation. In this case, the interpolation interval is performed at an interval corresponding to the sampling rate (for example, 44.1 KHz) of the final output signal output from the output unit 34 described later. The solid line shown in FIG. 6 described above shows an image when the interpolation processing is performed on the frequencies F0 and F1 of the sine wave component.
[0029]
[2.1] Configuration of interpolation / synthesis unit
Here, the configuration of the interpolation / synthesis unit 9 is shown in FIG.
The interpolation / synthesis unit 9 includes a plurality of partial waveform generation units 9a, and each partial waveform generation unit 9a has a frequency (F0, F1,...) And an amplifier (amplitude) of a specified sine wave component. A corresponding sine wave is generated. However, since the sine wave components (F0, A0), (F1, A1), (F2, A2),... In the first embodiment change each time according to the interpolation interval, The waveform output from the partial waveform generator 9a is a waveform according to the change. That is, the sine wave components (F0, A0), (F1, A1), (F2, A2),... Are sequentially output from the peak cooperation unit 8, and interpolation processing is performed for each sine wave component. Each partial waveform generator 9a outputs a waveform whose frequency and amplitude vary within a predetermined frequency region. The waveforms output from the partial waveform generators 9a are added and synthesized in the adder 9b. Therefore, the output signal of the interpolation synthesis unit 9 is a sine wave component synthesis signal SSS obtained by extracting a deterministic component from the input audio signal Sv.
[0030]
[2.2] Operation of residual component detector
Next, the residual component detection unit 10 generates a residual component signal SRD (time waveform) that is a deviation between the sine wave component synthesis signal SSS output from the interpolation synthesis unit 9 and the input speech signal Sv. This residual component signal SRD includes many unvoiced components included in the speech. On the other hand, the aforementioned sine wave component composite signal SSS corresponds to the voiced component.
By the way, in order to resemble a target singer's voice, if only the voiced sound is processed, it is not necessary to process the unvoiced sound.
Therefore, in the present embodiment, the speech conversion process is performed on the deterministic component corresponding to the voiced vowel component. More specifically, the residual component signal SRD is converted into a frequency waveform by the fast Fourier transform unit 11, and the residual component signal (frequency waveform) obtained is set as Rme (f) to the residual component holding unit 12. To keep.
[0031]
[2.3] Operation of average amplifier calculation unit
On the other hand, as shown in FIG. 8 (A), the sine wave components (F0, A0), (F1, A1), (F2, A2),... Output from the peak detection unit 5 via the peak cooperation unit 8. , (F (N-1), A (N-1)) N sine wave components (hereinafter collectively referred to as Fn and An, where n = 0 to (N-1)). While being held in the sine wave component holding unit 13, the amplifier An is input to the average amplifier calculation unit 14, and the average amplifier Ame is calculated by the following equation for each frame.
Ame = Σ (An) / N
[0032]
[2.4] Operation of amplifier normalization unit
Next, the amplifier normalization unit 15 normalizes each amplifier An with the average amplifier Ame according to the following equation to obtain a normalized amplifier A′n.
A'n = An / Ame
[2.5] Operation of the spectral shape calculation unit
Then, in the spectral shape calculation unit 16, as shown in FIG. 8B, an envelope (envelope) with the frequency Fn and the sine wave components (Fn, A′n) obtained by the normalizing amplifier A′n as breakpoints. Line) is generated as a spectral shape Sme (f).
In this case, the value of the amplifier at the frequency between the two breakpoints is calculated by, for example, linearly interpolating the two breakpoints. Note that the interpolation method is not limited to linear interpolation.
[0033]
[2.6] Operation of pitch normalization unit
Subsequently, the pitch normalization unit 17 normalizes each frequency Fn with the pitch Pme detected by the pitch detection unit 7 to obtain a normalized frequency F′n.
F'n = Fn / Pme
As a result, the original frame information holding unit 18 is the average amplifier Ame, the pitch Pme, the spectral shape Sme (f), and the normalized frequency F ′, which are the original attribute data corresponding to the sine wave component included in the input audio signal Sv. n will be held.
In this case, the normalized frequency F′n represents a relative value of the frequency of the harmonic sequence, and if the harmonic structure of the frame is handled as a complete harmonic structure, it is not necessary to hold it.
In this case, if male / female conversion is to be performed, at this stage, the male voice → female voice conversion increases the pitch by an octave, and the female voice → male voice conversion decreases the pitch by an octave. / It is preferable to perform female voice pitch control processing.
[0034]
Subsequently, among the original attribute data held in the original frame information holding unit 18, the average amplifier Ame and the pitch Pme are further filtered by the static change / vibrato change separation unit 19, so as to be static. Separately and keep the target change component and the vibrato change component. It is also possible to separate the jitter changing component, which is a higher frequency changing component, from the vibrato changing component.
More specifically, the average amplifier Ame is separated into an average amplifier static component Ame-sta and an average amplifier vibrato component Ame-vib.
Further, the pitch Pme is separated and held into a pitch static component Pme-sta and a pitch vibrato-like component Pme-vib.
[0035]
As a result, as shown in FIG. 8C, the original frame information data INFme of the corresponding frame is average amplifier static component Ame-sta, which is original attribute data corresponding to the sine wave component of the input audio signal Sv. Average amp vibrato component Ame-vib, pitch static component Pme-sta, pitch vibrato component Pme-vib, spectral shape Sme (f), normalized frequency F'n and residual component Rme (f) Will be held.
[0036]
On the other hand, target frame information data INFtar composed of target attribute data corresponding to a singer who is a target of imitation is preliminarily analyzed and held in advance in a hard disk or the like constituting the target frame information holding unit 20. .
In this case, among target frame information data INFtar, target attribute data corresponding to a sine wave component includes an average amplifier static component Atar-sta, an average amplifier vibrato-like component Atar-vib, a pitch static component Ptar-sta, There is a pitch vibrato-like component Ptar-vib and a spectral shape Star (f).
In the target frame information data INFtar, the target attribute data corresponding to the residual component includes a residual component Rtar (f).
[0037]
[2.7] Operation of key control / tempo change section
Next, the key control / tempo change unit 21 reads out the target frame information INFtar of the frame corresponding to the synchronization signal SSYNC from the target frame information holding unit 20 based on the synchronization signal SSYNC from the sequencer 31 and the read target frame information. The target attribute data constituting the data INFtar is corrected, and the read target frame information INFtar and the target unvoiced / voiced detection signal U / Vtar indicating whether the frame is unvoiced or voiced are output.
[0038]
More specifically, the key control unit (not shown) of the key control / tempo change unit 21 performs pitch static component Ptar-sta and pitch vibrato component Ptar which are target attribute data when the key of the karaoke apparatus is raised or lowered from the reference. For -vib, the same correction process is performed. For example, when the key is raised by 50 [cent], the pitch static component Ptar-sta and the pitch vibrato component Ptar-vib must also be raised by 50 [cent].
Further, when the tempo change unit (not shown) of the key control / tempo change unit 21 increases or decreases the tempo of the karaoke apparatus, it is necessary to read out the target frame information data INFtar at a timing corresponding to the changed tempo. is there.
[0039]
In this case, if the target frame information data INFtar corresponding to the timing corresponding to the necessary frame does not exist, the target frame information data INFtar of the two frames existing before and after the timing of the necessary frame is read. Interpolation processing is performed using these two target frame information data INFtar, and target frame information data INFtar of the frame at the necessary timing, and thus target attribute data are generated.
In this case, the vibrato component (the average amp vibrato component Atar-vib and the pitch vibrato component Ptar-vib) is not suitable as it is because the vibrato cycle itself is changed and is not suitable. It is necessary to perform interpolation processing that does not occur. Alternatively, if the target attribute data is not data representing the trajectory of the vibrato itself but the parameters of the vibrato period and the vibrato depth are held and the actual trajectory is obtained by calculation, this problem can be avoided.
[0040]
[2.8] Operation of the easy synchronization processing unit
Next, the easy synchronization processing unit 22 performs the singing that is the target of the corresponding imitation even though the original frame information data INFme exists in the frame of the singer who tries to imitate (hereinafter referred to as the original frame). If the target frame information data INFtar does not exist in the user's frame (hereinafter referred to as the target frame), the target frame information data INFtar of the frame existing in the front-rear direction of the target frame is used as the target frame information data INFtar of the target frame. The easy synchronization process is performed.
[0041]
Then, the easy synchronization processing unit 22 targets the target attribute data (average amplifier static component Atar-sync-sta, average) among the target attribute data included in the replaced target frame information data INFtar-sync described later. Amp vibrato-like component Atar-sync-vib, pitch static component Ptar-sync-sta, pitch vibrato-like component Ptar-sync-vib, and spectral shape Star-sync (f)) to sine wave component attribute data selection unit 23 Output.
Further, the easy synchronization processing unit 22 leaves target attribute data (residual component Rtar-sync (f)) related to the residual component among the target attribute data included in the replaced target frame information data INFtar-sync described later. The difference component selection unit 25 outputs the result.
[0042]
Even in the processing in the easy synchronization processing unit 22, the vibrato period itself changes with respect to the vibrato component (average amp vibrato component Atar-vib and pitch vibrato component Ptar-vib). Since it is inappropriate, it is necessary to perform an interpolation process so that the cycle does not fluctuate. Alternatively, if the target attribute data is not data representing the trajectory of the vibrato itself but the parameters of the vibrato period and the vibrato depth are held and the actual trajectory is obtained by calculation, this problem can be avoided.
[0043]
[2.8.1] Details of easy synchronization processing
Here, the easy synchronization process will be described in detail with reference to FIGS. 9 and 10.
FIG. 9 is a timing chart of the easy synchronization process, and FIG. 10 is a flowchart of the easy synchronization process.
First, the easy synchronization processing unit 22 sets the synchronization mode = “0” representing the synchronization processing method (step S11). The synchronization mode = “0” corresponds to a normal process in which the target frame information data INFtar exists in the target frame corresponding to the original frame.
[0044]
Then, it is determined whether or not the original unvoiced / voiced detection signal U / Vme (t) at a certain timing t has changed from unvoiced (U) to voiced (V) (step S12).
For example, as shown in FIG. 9, at the timing t = t1, the original unvoiced / voiced detection signal U / Vme (t) changes from unvoiced (U) to voiced (V).
When the original unvoiced / voiced detection signal U / Vme (t) is changed from unvoiced (U) to voiced (V) in the determination in step S12 (step S12; Yes), the previous timing t of timing t. Whether or not the original unvoiced / voiced detection signal U / Vme (t-1) at -1 is unvoiced (U) and the target unvoiced / voiced detection signal U / Vtar (t-1) is unvoiced (U) (Step S18).
[0045]
For example, as shown in FIG. 9, at the timing t = t0 (= t1-1), the original unvoiced / voiced detection signal U / Vme (t-1) is unvoiced (U) and the target unvoiced / voiced detection signal U / Vtar (t-1) is unvoiced (U).
In the determination in step S18, when the original unvoiced / voiced detection signal U / Vme (t-1) is unvoiced (U) and the target unvoiced / voiced detection signal U / Vtar (t-1) is unvoiced (U) (Step S18; Yes), since the target frame information data INFtar does not exist in the target frame, the synchronization mode is set to “1”, and the replacement target frame information data INFhold is set to the backward direction of the target frame. This is the target frame information of the frame existing in (Backward).
[0046]
For example, as shown in FIG. 9, since the target frame information data INFtar does not exist in the target frame at the timing t = t1 to t2, the synchronization mode is set to “1”, and the replacement target frame information data INFhold is set to the target frame information data INFhold. It is assumed that the target frame information data backward of a frame existing in the backward direction of the target frame (that is, a frame existing at timing t = t2 to t3).
Then, the process proceeds to step S15, and it is determined whether or not the synchronization mode = “0” (step S15).
[0047]
If the synchronization mode = “0” in the determination in step S15, the target frame information data INFtar (t) exists in the target frame corresponding to the original frame at the timing t, that is, normal processing. Therefore, the replaced target frame information data INFtar-sync is set as target frame information data INFtar (t).
INFtar-sync = INFtar (t)
For example, as shown in FIG. 9, target frame information data INFtar exists in the target frame at timing t = t2 to t3.
INFtar-sync = INFtar (t)
And
[0048]
In this case, target attribute data (average amplifier static component Atar-sync-sta, average amplifier vibrato component Atar-sync-vib, pitch static value included in the replaced target frame information data INFtar-sync used in the subsequent processing) Component Ptar-sync-sta, pitch vibrato component Ptar-sync-vib, spectral shape Star-sync (f) and residual component Rtar-sync (f)) are substantially as follows ( Step S16).
Atar-sync-sta = Atar-sta
Atar-sync-vib = Atar-vib
Ptar-sync-sta = Ptar-sta
Ptar-sync-vib = Ptar-vib
Star-sync (f) = Star (f)
Rtar-sync (f) = Rtar (f)
[0049]
If it is determined in step S15 that the synchronization mode = “1” or the synchronization mode = “2”, the target frame information data INFtar (t) exists in the target frame corresponding to the original frame at the timing t. Therefore, the replacement target frame information data INFtar-sync is set as the replacement target frame information data INFhold.
INFtar-sync = INFhold
For example, as shown in FIG. 9, the target frame information data INFtar does not exist in the target frame at timing t = t1 to t2, and the synchronization mode = “1”, but the timing t = t2 to t3. Since the target frame has target frame information data INFtar, the replacement target frame information data INFtar-sync is used as replacement target frame information data INFhold which is target frame information data of the target frame at timing t = t2 to t3. The target attribute data included in the replaced target frame information data INFtar-sync used in the subsequent processing after performing the process P1 is an average amplifier static component Atar-sync-sta, an average amplifier vibrato component Atar-sync-vib, Pitch static component Ptar-sync-sta, pitch vibrato The component Ptar-sync-vib, the spectral shape Starr-sync (f), and the residual component Rtar-sync (f) are obtained (step S16).
[0050]
Further, as shown in FIG. 9, the target frame information data INFtar does not exist in the target frame at the timing t = t3 to t4, and the synchronization mode = “2”, but at the timing t = t2 to t3. Since the target frame has target frame information data INFtar, the replacement target frame information data INFtar-sync is used as replacement target frame information data INFhold which is target frame information data of the target frame at timing t = t2 to t3. The target attribute data included in the replaced target frame information data INFtar-sync used in the subsequent processing after performing the process P2 is an average amplifier static component Atar-sync-sta, an average amplifier vibrato component Atar-sync-vib, Pitch static component Ptar-sync-sta, pitch vibrato-like composition The component Ptar-sync-vib, the spectral shape Starr-sync (f), and the residual component Rtar-sync (f) (step S16).
[0051]
If the original unvoiced / voiced detection signal U / Vme (t) does not change from unvoiced (U) to voiced (V) in step S12 (step S12; No), the target unvoiced / voiced detection signal U It is determined whether / Vtar (t) has changed from voiced (V) to unvoiced (U) (step S13).
When the target unvoiced / voiced detection signal U / Vtar (t) is changed from voiced (V) to unvoiced (U) in the determination in step S13 (step S13; Yes), the previous timing t of the timing t. Whether the original unvoiced / voiced detection signal U / Vme (t-1) at -1 is voiced (V) and the target unvoiced / voiced detection signal U / Vtar (t-1) is voiced (V). (Step S19).
[0052]
For example, as shown in FIG. 9, the target unvoiced / voiced detection signal U / Vtar (t) changes from voiced (V) to unvoiced (U) at timing t3, and at timing t-1 = t2 to t3, the original The unvoiced / voiced detection signal U / Vme (t-1) is voiced (V) and the target unvoiced / voiced detection signal U / Vtar (t-1) is voiced (U).
In the determination of step S19, when the original unvoiced / voiced detection signal U / Vme (t-1) is voiced (V) and the target unvoiced / voiced detection signal U / Vtar (t-1) is voiced (V) (Step S19; Yes), since the target frame information data INFtar does not exist in the target frame, the synchronization mode is set to “2”, and the replacement target frame information data INFhold is set in the forward direction of the target frame. The target frame information of the frame existing in (forward) is used.
[0053]
For example, as shown in FIG. 9, since the target frame information data INFtar does not exist in the target frame at the timing t = t3 to t4, the synchronization mode is set to “2”, and the replacement target frame information data INFhold is set to the target frame information data INFhold. It is assumed that the target frame information data forward of a frame existing in the forward direction of the target frame (that is, a frame existing at timing t = t2 to t3).
Then, the process proceeds to step S15, it is determined whether or not the synchronization mode = “0” (step S15), and the same process is performed thereafter.
If the target unvoiced / voiced detection signal U / Vtar (t) does not change from voiced (V) to unvoiced (U) in the determination in step S13 (step S13; No), the original unvoiced / voiced at the timing t. Detection signal U / Vme (t) changes from voiced (V) to unvoiced (U), or target unvoiced / voiced detection signal U / Vtar (t) changes from unvoiced (U) to voiced (V) It is determined whether or not there is (step S14).
[0054]
In the determination of step S14, the original unvoiced / voiced detection signal U / Vme (t) at timing t changes from voiced (V) to unvoiced (U), and the target unvoiced / voiced detection signal U / Vtar (t) is When the state is changed from unvoiced (U) to voiced (V) (step S14; Yes), the synchronization mode is set to “0”, the replacement target frame information data INFhold is initialized (clear), and the process is performed. Is transferred to step S15, and the same processing is performed thereafter.
In step S14, the original unvoiced / voiced detection signal U / Vme (t) at timing t does not change from voiced (V) to unvoiced (U), and the target unvoiced / voiced detection signal U / Vtar (t) Is not changed from unvoiced (U) to voiced (V) (step S14; No), the process proceeds to step S15 as it is, and the same process is performed thereafter.
[0055]
[2.9] Operation of sine wave component attribute data selection unit
Subsequently, the sine wave component attribute data selection unit 23 selects target attribute data (average) among the target attribute data included in the replaced target frame information data INFtar-sync input from the easy synchronization processing unit 22. Amplifier static component Atar-sync-sta, Average amplifier vibrato component Atar-sync-vib, Pitch static component Ptar-sync-sta, Pitch vibrato component Ptar-sync-vib and Spectral shape Star-sync (f) ) And a new sine wave component attribute data, a new amplifier component Anew, a new pitch component Pnew, and a new spectral shape Snew (f).
[0056]
That is, the new amplifier component Anew is generated by the following equation.
Anew = A * -sta + A * -vib (* is me or tar-sync)
More specifically, as shown in FIG. 8D, the new amplifier component Anew is either the average amplifier static component Ame-sta of the original attribute data or the average amplifier static component Atar-sync-sta of the target attribute data. One of the average amp vibrato component Ame-vib of the original attribute data and the average amp vibrato component Atar-sync-vib of the target attribute data is generated.
The new pitch component Pnew is generated by the following equation.
Pnew = P * -sta + P * -vib (* is me or tar-sync)
[0057]
More specifically, as shown in FIG. 8D, the new pitch component Pnew is either the pitch static component Pme-sta of the original attribute data or the pitch static component Ptar-sync-sta of the target attribute data. And a pitch vibrato component Pme-vib of the original attribute data or a pitch vibrato component Ptar-sync-vib of the target attribute data.
The new spectral shape Snew (f) is generated by the following equation.
Snew (f) = S * (f) (where * is me or tar-sync)
[0058]
By the way, in general, when the amplifier component is large, a bright sound extending to a high frequency is obtained, and when the amplifier component is small, the sound is concealed. Therefore, for the new spectral shape Snew (f), in order to simulate such a state, as shown in FIG. 11, the slope of the high frequency component of the spectral shape, that is, the spectral shape of the high frequency component portion is shown. Is controlled by performing spectral tilt correction that compensates according to the magnitude of the new amplifier component Anew, so that more realistic sound can be reproduced. Subsequently, for the generated new amplifier component Anew, new pitch component Pnew, and new spectral shape Snew (f), the attribute data deformation is performed based on the sine wave component attribute data deformation information input from the controller 29 as necessary. Further deformation is performed by the part 24. For example, deformation such as extending the spectral shape as a whole is performed. The attribute data transformation unit 24 supplies the pitch Patt of the sine wave component after the transformation to the pitch determination unit 40.
[0059]
[2.10] Operation of residual component selector
On the other hand, the residual component selecting unit 25 selects target attribute data (residual component Rtar) regarding the residual component among the target attribute data included in the replaced target frame information data INFtar-sync input from the easy synchronization processing unit 22. -sync (f)), a residual component signal (frequency waveform) Rme (f) held in the residual component holding unit 12 and a residual component attribute data selection information inputted from the controller 29, a new residual A new residual component Rnew (f) which is component attribute data is generated.
That is, the new residual component Rnew (f) is generated by the following equation.
Rnew (f) = R * (f) (where * is me or tar-sync)
[0060]
In this case, it is more preferable to select either me or tar-sync, which is the same as the new spectral shape Snew (f).
Further, for the new residual component Rnew (f), in order to simulate the same state as the new spectral shape, as shown in FIG. 11, the high-frequency component of the residual component, that is, the high-frequency component portion A more realistic sound can be reproduced by performing and controlling spectral tilt correction that compensates for the slope of the residual component in accordance with the magnitude of the new amplifier component Anew.
[0061]
[2.11] Operation of sine wave component generator
Subsequently, the sine wave component generation unit 26 is based on the new amplifier component Anew, the new pitch component Pnew, and the new spectral shape Snew (f) that are not accompanied by the deformation, or are accompanied by the deformation, output from the attribute data deformation unit 24. The new sine wave components (F "0, A" 0), (F "1, A" 1), (F "2, A" 2),..., (F "(N-1) ), A ″ (N−1)) N sine wave components (hereinafter collectively referred to as F ″ n and A ″ n. N = 0 to (N−1)).
More specifically, a new frequency F ″ n and a new amplifier A ″ n are obtained by the following equations.
F "n = F'n × Pnew
A "n = Snew (F" n) * Anew
If you think of it as a perfect harmonic structure model,
F "n = (n + 1) × Pnew
It becomes.
[0062]
[2.12] Operation of sine wave component deformation unit
Further, the obtained new frequency F ″ n and the new amplifier A ″ n are further deformed by the sine wave component deformation unit 27 based on the sine wave component deformation information input from the controller 29 as necessary. For example, only the new-order amplifier A ″ n (= A ″ 0, A ″ 2, A ″ 4,...) Of even-order components is increased (for example, doubled). As a result, it is possible to give the converted speech further variety.
[0063]
[2.13] Operation of pitch determination unit
The comb filter pitch determination unit 40 sets the pitch Patt from the attribute data transformation unit 24 as the comb filter pitch (Pcomb) and supplies the comb filter to the comb filter processing unit 41.
[0064]
[2.14] Operation of comb filter processing unit
The comb filter processing unit 41 configures a comb filter using the pitch Pcomb, and filters the residual component Rnew (f) with the comb filter, whereby the residual component Rnew (f). In Pitch component and its harmonic component Append The new residual component Rnew ′ (f) is supplied to the inverse fast Fourier transform unit 28. Here, FIG. 12 is a conceptual diagram showing a characteristic example of the comb filter when the pitch Pcomb is 200 Hz. As described above, when the residual component is held on the frequency axis, a comb filter is configured on the frequency axis based on the pitch Pcomb.
[0065]
[2.15] Operation of inverse fast Fourier transform unit
Next, the inverse fast Fourier transform unit 28 stores the obtained new frequency F ″ n, new amplifier A ″ n (= new sine wave component) and new residual component Rnew ′ (f) in the FFT buffer, and sequentially performs inverse FFT. Further, overlap processing is performed so that the obtained time axis signals partially overlap, and addition processing for adding them is performed to generate a converted voice signal that is a time axis signal of a new voiced sound.
[0066]
At this time, based on the sine wave component / residual component balance control signal input from the controller 29, the mixing ratio of the sine wave component and the residual component is controlled to obtain a more realistic voiced signal. In this case, generally, a rough voice can be obtained by increasing the mixing ratio of the residual components.
In this case, when storing the new frequency F ″ n, the new amplifier A ″ n (= new sine wave component), and the new residual component Rnew (f) in the FFT buffer, they are converted at different pitches and appropriate pitches. By further adding a sine wave component, harmony can be obtained as a converted audio signal. Further, by giving a harmony pitch suitable for the accompaniment sound by the sequencer 31, a musical harmony suitable for the accompaniment can be obtained.
[0067]
[2.16] Operation of crossfader
Next, based on the original unvoiced / voiced detection signal U / Vme (t), the crossfader 30 outputs the input sound signal Sv as it is to the mixer 30 when the input sound signal Sv is unvoiced (U).
When the input sound signal Sv is voiced (V), the converted sound signal output from the inverse fast Fourier transform conversion unit 28 is output to the mixer 33.
In this case, the reason why the cross fader 30 is used as the changeover switch is to prevent the generation of a click sound when the switch is changed by performing a crossfade operation.
[0068]
[2.17] Operation of sequencer, tone generator, mixer, and output unit
On the other hand, the sequencer 31 outputs sound source control information for generating accompaniment sound of karaoke to the sound source unit 32 as, for example, MIDI (Musical Instrument Digital Interface) data.
Thereby, the sound source unit 32 generates an accompaniment signal based on the sound source control information and outputs the accompaniment signal to the mixer 33.
The mixer 33 mixes either the input audio signal Sv or the converted audio signal and the accompaniment signal, and outputs the mixed signal to the output unit 34.
The output unit 34 has an amplifier (not shown) and amplifies the mixed signal and outputs it as an acoustic signal.
[0069]
[3] Modification of embodiment
[3.1] First modification
In the above description, either the original attribute data or the target attribute data is selectively used as the attribute data. However, by performing interpolation processing using both the original attribute data and the target attribute data, It is also possible to obtain a converted audio signal having typical attributes.
However, according to such a configuration, there may be a case where a converted voice that does not resemble either a singer who tries to imitate or a singer who is a target of imitation is obtained.
Also, especially when the spectral shape is obtained by interpolation processing, the singer who wants to imitate pronounces “a” and the singer who is imitated pronounces “yes” May cause a sound that is neither “A” nor “I” to be output as converted speech, and must be handled with care.
[0070]
[3.2] Second modification
The extraction of the sine wave component is not limited to the method used in this embodiment. In short, it is only necessary to extract a sine wave included in the audio signal.
[3.3] Third modification
In this embodiment, the target sine wave component and residual component are stored, but instead, the target speech itself is stored and read out, and the sine wave component and residual component are extracted by real-time processing. May be. That is, you may perform the process similar to the process performed with respect to the audio | voice of the singer who imitates also in this embodiment with respect to the audio | voice of the target singer.
[0071]
[3.4] Fourth modification
In this embodiment, the pitch data, the amplifier, and the spectral shape are all handled as the attribute data, but it is also possible to handle at least one of them.
[3.5] Fifth modification
In the present embodiment, the residual component is held on the frequency axis. However, the present invention is not limited to this, and the residual component may be held on the time axis. FIG. 13 is a block diagram illustrating a configuration (a part) of a modification of the above-described embodiment. FIG. 14 is a block diagram showing an example of the configuration of a comb filter (delay filter). It should be noted that portions corresponding to those in FIG. In the figure, a comb filter processing unit 42 constitutes a comb filter (delay filter) in which the reciprocal of the pitch Pcomb determined by the pitch determination unit 40 is a delay time, and the comb filter has a residual component Rnew ( t) is filtered and supplied to the adder 43 as a residual component Rnew ″ (t). The adder 43 adds the filtered residual component Rnew ″ (t) to the residual component Rnew (t), thereby adding a pitch component and its harmonic component to the residual component Rnew (t), The new residual component Rnew ′ (t) is supplied to the IFFT processing unit 8.
As described above, even when the residual component is processed on the time axis, the pitch component and its harmonic component can be added to the residual component Rnew (t) as in the above-described embodiment. . Therefore, only the pitch component of the sine wave component is heard in the sound that is finally output, and the naturalness of the sound can be improved.
[0072]
[4] Effects of the embodiment
As a result, the song of the singer is output along with the accompaniment of karaoke. The voice quality and singing method are greatly influenced by the target, and the voice quality and singing method of the target itself are obtained. In this way, a song as if imitating the target is output.
Further, since the residual component has a pitch component equal to the sine wave component by adding the pitch component and its harmonic component to the residual component Rnew (f), it is synthesized by synthesizing with the sine wave component. The sound is kept in pitch and does not impair the naturalness of the sound.
[0073]
【The invention's effect】
As described above, according to the present invention, the sine wave component and the residual component extracted from the input speech signal are respectively transformed based on the sine wave component or the residual component of the target speech, and then the sine Before the wave component and the residual component are synthesized, the pitch component and its harmonic component are added to the deformed residual component, so that the pitch of the synthesized voice is maintained and the naturalness is impaired. Without any change, it is possible to easily obtain a converted voice reflecting the voice quality and singing method of the target singer to be imitated from the voice (input voice) of the singer to imitate.
[Brief description of the drawings]
FIG. 1 is a block diagram (part 1) illustrating a configuration of an embodiment of the present invention.
FIG. 2 is a block diagram (part 2) showing a configuration of an embodiment of the present invention.
FIG. 3 is a diagram illustrating a state of a frame in the embodiment.
FIG. 4 is an explanatory diagram for explaining peak detection of a frequency spectrum in the embodiment.
FIG. 5 is a diagram illustrating cooperation of peak values for each frame in the embodiment.
.
FIG. 6 is a diagram illustrating a change state of a frequency value in the embodiment.
FIG. 7 is a diagram illustrating a change state of a deterministic component in a process in the embodiment.
FIG. 8 is an explanatory diagram of signal processing in the embodiment.
FIG. 9 is a timing chart of an easy synchronization process.
FIG. 10 is an easy synchronization process flowchart;
FIG. 11 is a diagram for explaining spectral tilt compensation of a spectral shape.
FIG. 12 is a conceptual diagram for explaining the characteristics of a comb filter (when the pitch Pcomb is 200 Hz).
FIG. 13 is a block diagram showing a configuration (a part) of a voice conversion device according to a modification of the present invention.
FIG. 14 is a block diagram illustrating an example of a configuration of a comb filter (delay filter).
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... Microphone, 2 ... Analysis window production | generation part, 3 ... Input audio | voice signal extraction part, 4 ... Fast Fourier transform part, 5 ... Peak detection part, 6 ... Unvoiced / voiced detection part, 7 ... Pitch extraction part, 8 ... Peak Linking unit, 9 ... interpolation synthesis unit, 10 ... residual component detection unit, 11 ... fast Fourier transform unit, 12 ... residual component holding unit, 13 ... sine wave component holding unit, 14 ... average amplifier calculation unit, 15 ... amplifier Normalization unit, 16 ... Spectral shape calculation unit, 17 ... Pitch normalization unit, 18 ... Original frame information holding unit, 19 ... Static change / vibrato change separation unit, 20 ... Target frame information holding unit, 21 ... key Control / tempo change unit, 22 ... easy synchronization processing unit, 23 ... sine wave component attribute data selection unit, 24 ... attribute data transformation unit, 25 ... residual component selection unit, 26 ... sine wave component generation unit, 27 Sinusoidal component transformation unit, 28 ... Inverse fast Fourier transform unit, 29 ... Controller, 30 ... Crossfader, 31 ... Sequencer, 32 ... Sound source unit, 33 ... Mixer, 34 ... Output unit, 40 ... Pitch determination unit, 41, 42 ... comb filter processing unit, 43 ... adder

Claims (6)

Sine wave component extraction means for extracting a sine wave component from the input audio signal;
Residual component extraction means for extracting residual components other than the sine wave component extracted by the sine wave component extraction means from the input audio signal;
The sine wave component extracted by the sine wave component extracting means is deformed based on the sine wave component of the target audio signal , and the deformed sine wave is changed according to the magnitude of the amplifier component of the deformed sine wave component. Sine wave component deformation means for controlling the inclination of the component;
The residual component extracted by the residual component extraction means is deformed based on the residual component of the target audio signal, and the deformed residual is changed according to the magnitude of the amplifier component of the deformed sine wave component. Residual component deformation means for controlling the slope of the difference component;
An adding means for adding a pitch component and its harmonic component to the residual component obtained by the residual component deforming means;
A voice conversion device comprising: a sine wave component deformed by the sine wave component deforming unit and controlled in inclination; and a synthesizing unit that synthesizes a residual component to which a pitch component and its harmonic component are added by the adding unit. . The voice conversion device according to claim 1,
Pitch determination means comprising the pitch of the sine wave component obtained by the sine wave component deformation means as the peak pitch of the pass band in the additional means ,
The adding means adds the pitch component and its harmonic component to the residual component by passing the residual component corresponding to the passband in which the pitch of the peak is determined by the pitch determining means. A voice conversion device. The voice conversion device according to claim 2 , wherein
The speech converting apparatus according to claim 1, wherein the adding means is a comb filter having a passband peak pitch determined by the pitch determining means when the residual component is held on the frequency axis. The voice conversion device according to claim 2 , wherein
The adding means is a comb filter having a delay filter whose delay time is the reciprocal of the pitch of the passband peak determined by the pitch determining means when the residual component is held on the time axis. A voice conversion device characterized by that. A component extraction step of extracting a residual component that is a component other than the sine wave component and the sine wave component from the input speech;
A sine that transforms the extracted sine wave component based on the sine wave component of the target speech and controls the slope of the transformed sine wave component according to the magnitude of the amplified sine wave component Wave component deformation process;
The extracted residual component is transformed based on the residual component of the target speech, and the slope of the transformed residual component is controlled in accordance with the magnitude of the amplified sine wave component. A residual component deformation process;
An adding step of adding a pitch component and its harmonic component to the residual component obtained in the residual component deformation step;
A speech conversion method comprising: a synthesizing step of synthesizing a sine wave component that is deformed and controlled in inclination in the sine wave component deforming step, and a residual component to which a pitch component and its harmonic component are added in the adding step. . The voice conversion method according to claim 5, wherein
Including a pitch determination step in which the pitch of the sine wave component obtained by the sine wave component deformation step is set as the pitch of the passband peak in the addition step ;
In the adding step, a pitch component and its overtone component are added to the residual component by passing the residual component corresponding to the passband in which the pitch of the peak is determined in the pitch determining step. A voice conversion method characterized by the above.

Images