JP2000010598A - Speech transforming device and method therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily obtain natural acoustic feeling of speech at the time of transforming speech by providing this device with a means for generating a deformed spectral shape and a sine wave component information generating means for generating sine wave component information based on the deformed spectral shape. SOLUTION: A spectral shape extracting means extracts a spectral shape on the frequency axis from an input speech signal. A deformed spectral shape generating means 23 calculates a constant αbased on the pitch of the input speech signal and that of a target speech signal, and generates the deformed spectral shape by shifting the spectral shape α times in the direction of the frequency axis. A sine wave component information generating part 26 generates sine wave component information based on the frequency components contained in the sine wave components extracted from the target speech signal and the deformed spectral shape. And, a transformed speech signal is generated based on the sine wave component information and outputted.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a voice conversion apparatus and a voice conversion method for converting an input voice into another voice and outputting the same, and more particularly to a voice conversion apparatus and a voice conversion method suitable for use in a karaoke apparatus. .

[0002]

2. Description of the Related Art There have been developed various voice converters for changing the frequency characteristics and the like of an input voice and outputting the converted voice. For example, some karaoke devices convert the pitch of a singer's singing voice into a male voice. To convert a female voice into a female voice or vice versa (for example, Japanese Translation of International Patent Publication No. Hei 8-5085).
No. 81).

[0003]

However, in the conventional voice converter, since the pitch of the singing voice is simply converted, there is a problem that a natural sound cannot be obtained in terms of audibility. Therefore, an object of the present invention is to provide a voice conversion device and a voice conversion method capable of easily obtaining a natural sound in audibility when performing voice conversion.

[0004]

In order to solve the above-mentioned problems, a configuration according to claim 1 comprises a spectral shape extracting means for extracting a spectral shape on a frequency axis from an input audio signal, and a pitch of the input audio signal. And a constant α based on the pitch of the target audio signal, and a spectral shape deforming means for generating a modified spectral shape by shifting the spectral shape by α times in the frequency axis direction, and extracting from the target audio signal. A sine wave component information generating means for generating sine wave component information based on the frequency component included in the obtained sine wave component and the deformed spectral shape, and generating and outputting a converted audio signal based on the sine wave component information Voice generating means.

According to a second aspect of the present invention, there is provided a sine wave component extracting means for extracting a sine wave component of an input audio signal, and a constant α is calculated based on a pitch of the input audio signal and a pitch of a target audio signal. The target spectrum on the frequency axis extracted in advance from the target audio signal
Target spectral shape deforming means for generating a deformed target spectral shape by shifting the shape by α times in the frequency axis direction, and a sine wave component based on the frequency component included in the sine wave component and the deformed target spectral shape A sine wave component information generating means for generating information; a sound generating means for generating and outputting a converted sound signal based on the sine wave component information;
It is characterized by having.

According to a third aspect of the present invention, in the configuration of the first or second aspect, a predetermined shift coefficient is β, a pitch of the input audio signal is forg, and a pitch of the target audio signal is ftar. Where α is calculated by the following equation. α = βk where k = log2 (forg / ftar)

According to a fourth aspect of the present invention, there is provided a spectral shape extracting step of extracting a spectral shape on a frequency axis from an input voice, and calculating a constant α based on a pitch of the input voice and a pitch of a target voice.
A spectral shape transformation step of generating a modified spectral shape by shifting the spectral shape by α times in the frequency axis direction, and a frequency component included in a sine wave component extracted from the target sound and the modified spectral shape. A sine wave component information generating step of generating sine wave component information based on the sine wave component information; and a voice generating step of generating a converted voice based on the sine wave component information.

According to a fifth aspect of the present invention, there is provided a sine wave component extracting step of extracting a sine wave component of an input voice, calculating a constant α based on a pitch of the input voice and a pitch of the target voice, and calculating a constant α from the target voice. A target spectral shape deforming step of generating a deformed target spectral shape by shifting the target spectral shape on the frequency axis extracted in advance in the frequency axis direction by α times, the frequency component included in the sine wave component and the deformation A sine wave component information generating step of generating sine wave component information based on the target spectral shape, and a voice generating step of generating a converted voice based on the sine wave component information are provided.

According to a sixth aspect of the present invention, in the configuration of the fourth or fifth aspect, the predetermined shift coefficient is β, the pitch of the input audio signal is forg, and the pitch of the target audio signal is ftar. Where α is calculated by the following equation. α = βk where k = log2 (forg / ftar)

[0010]

Preferred embodiments of the present invention will now be described with reference to the drawings. [1] Outline Processing of Embodiment First, outline processing of the embodiment will be described. [1.1] Step S1 First, a voice (input voice signal) of a singer (hereinafter, referred to as a former singer (me)) is subjected to FFT (Fast Fouri
SMS (Spectral Modeling Synthe
sis) analysis to extract a sine wave component (Sine component) in frame units and generate a residual component (Residual component) in frame units from the input audio signal and the sine wave component. In parallel with this, the input audio signal is unvoiced (including silence)
It is determined whether or not the input voice signal is unvoiced. If the voice is unvoiced, the following steps S2 to S6 are not performed, and the input voice signal is output as it is. In this case, SM
As the S analysis, a pitch synchronous analysis that changes the analysis window width according to the pitch in the previous frame is employed.

[1.2] Step S2 Next, when the input voice signal is a voiced sound, pitch, Amp (Amplitude) and spectral which are original attribute data are further extracted from the extracted sine wave component. -Extract a shape (Spectral Shape). [1.3] Step S3 Attribute data (target attribute data = pitch,
The target attribute data (= pitch, amplifier and target spectral shape) of a frame corresponding to the frame of the input voice signal of the former singer (me) is extracted from the amplifier and the spectral shape.

[1.4] Step S4 Next, based on the original attribute data corresponding to the former singer (me) and the target attribute data corresponding to the target singer, the spectral shape of the former singer (or the target singer). Target spectral shape)
A transform spectral shape (or a transform target spectral shape) is generated based on the above, and the generated transform spectral shape (or the transform target spectral shape) and a frequency included in a sine wave component previously extracted from the target audio signal New sine wave component information is generated based on the component (or the frequency component included in the sine wave component extracted from the input audio signal).

[1.5] Step S5 Inverse FFT of the new sine wave component information obtained in the subsequent step is performed to obtain a converted audio signal. [1.6] Conclusion According to the converted voice signal obtained as a result of these processes, the reproduced voice is as if the singing voice of the original singer is a natural singing voice sung by another singer.

[2] Detailed Configuration of the Embodiment FIGS. 1 and 2 show detailed configuration diagrams of the embodiment. In addition,
The present embodiment is an example in which the voice conversion device (voice conversion method) according to the present invention is applied to a karaoke device and configured as a karaoke device capable of performing more natural voice conversion. In FIG. 1, a microphone 1 collects the voice of a former singer (me) and outputs the voice to an input audio signal cutout unit 3 as an input audio signal Sv. In parallel with this, the analysis window generation unit 2 generates an analysis window (for example, a Hamming window) AW having a period that is a fixed multiple (for example, 3.5 times) of the period of the pitch detected in the previous frame. Is output to the input audio signal cutout unit 3. When the initial state or the previous frame is an unvoiced sound (including silence), an analysis window having a fixed period set in advance is output to the input audio signal cutout unit 3 as the analysis window AW.

Thus, the input audio signal extracting section 3 multiplies the input analysis window AW by the input audio signal Sv, cuts out the input audio signal Sv in units of frames, and outputs a fast Fourier transform section 4 as a frame audio signal FSv. Is output to More specifically, the relationship between the input audio signal Sv and the frames is as shown in FIG. 3, and each frame FL is set so as to partially overlap the previous frame FL. Then, the frame audio signal FSv is analyzed in the fast Fourier transform unit 4, and a local peak is detected by the peak detecting unit 5 from the frequency spectrum output from the fast Fourier transform unit 4, as shown in FIG. . More specifically, a local peak marked with “x” is detected in the frequency spectrum as shown in FIG. This local peak is represented as a combination of a frequency value and an amplifier (amplitude) value.

That is, as shown in FIG. 4, (F0, A
0), (F1, A1), (F2, A2),..., (F
N, AN), the local peak is detected and represented for each frame. Then, as schematically shown in FIG. 3, one set (hereinafter, referred to as a local peak set) is output to the unvoiced / voiced detection unit 6 and the peak coordination unit 8 for each frame. The unvoiced / voiced detection unit 6
Based on the input local peak for each frame, it is detected that the voice is unvoiced ('t', 'k', etc.) according to the magnitude of the high frequency component, and the unvoiced / voiced detection signal U / Vme
Is output to the pitch detection unit 7, the easy synchronization processing unit 22, and the crossfader unit 30. Alternatively, it is detected on the time axis that the voice is unvoiced according to the number of zero crossings per unit time ('s', etc.), and the original unvoiced / voiced detection signal U / Vme is detected by the pitch detection unit 7 by the easy synchronization processing. Output to the section 22 and the crossfader section 30.

Further, when the input frame is not detected as unvoiced, the unvoiced / voiced detection section 6 outputs the input local peak set as it is, and
Output to The pitch detector 7 detects the pitch Pme of the frame corresponding to the local peak set based on the input local peak set. More specific methods for detecting the pitch Pme of a frame include, for example, Maher, RC
andJ.W.Beauchamp: "Fundamental Frequency Estimation
of Musical Signal using a two-way Mismatch Proced
ure "(Journal of Acounstical Society of America95
(4): 2254-2263).

Next, the local peak set output from the peak detecting section 5 is determined by the peak linking section 8 to be linked with the preceding and succeeding frames, and the local peaks recognized as linked are formed into a series of data strings. A linking process for connecting a local peak to the data is performed. Here, this cooperation processing will be described with reference to FIG.
Now, assume that a local peak as shown in FIG. 5A is detected in the previous frame, and a local peak as shown in FIG. 5B is detected in the current frame.

In this case, the peak linking unit 8 determines each local peak (F0, A0) detected in the previous frame,
(F1, A1), (F2, A2), ..., (FN, A
It is checked whether the local peak corresponding to N) has been detected in the current frame. The determination as to whether or not there is a corresponding local peak is made based on whether or not the local peak of the current frame is detected within a predetermined range centered on the frequency of the local peak detected in the previous frame. More specifically, in the example of FIG. 5, the local peaks (F0, A0), (F1, A1), (F2, A2).
, The corresponding local peak is detected, but for the local peaks (FK, AK), (FIG. 5
(See (A)) and the corresponding local peak (see FIG. 5B) is not detected.

When detecting the corresponding local peaks, the peak coordinating unit 8 connects the local peaks in chronological order and outputs them as a set of data strings. If the corresponding local peak is not detected, the data is replaced with data indicating that there is no corresponding local peak for the frame. Here, FIG. 6 shows an example of changes in the frequency F0 and the frequency F1 of the local peak over a plurality of frames. Such changes are caused by the amplifiers (amplitude) A0, A
1, A2,... Are similarly recognized. In this case, the data string output from the peak linking unit 8 is a discrete value output at every frame interval.

The peak value output from the peak linking unit 8 is hereinafter referred to as a deterministic component. This means a component that is deterministically replaced as a sine wave element in the original signal (that is, the audio signal Sv). Further, each of the replaced sine waves (strictly speaking, frequency and amplifier (amplitude), which are parameters of the sine wave), will be referred to as partial components. Next, the interpolation synthesis unit 9
Performs interpolation processing on the deterministic component output from the peak linking unit 8, and performs waveform synthesis based on the deterministic component after interpolation using a so-called oscillator method. In this case, the interpolation is performed at intervals corresponding to the sampling rate (for example, 44.1 KHz) of the final output signal output from the output unit 34 described later. The solid line shown in FIG. 6 described above shows an image when the interpolation processing is performed on the frequencies F0 and F1 of the sine wave components.

[2.1] Configuration of Interpolation / Synthesis Unit The configuration of the interpolation / synthesis unit 9 is shown in FIG. The interpolation / synthesis unit 9 includes a plurality of partial waveform generation units 9a, and each of the partial waveform generation units 9a adjusts a frequency (F0, F1,...) And an amplifier (amplitude) of a designated sine wave component. Generates a corresponding sine wave. However, the sine wave components (F0, A0), (F1, A1), (F2,
Since A2),... Change every moment according to the interpolation interval, the waveform output from each partial waveform generator 9a becomes a waveform according to the change. That is, the sine wave components (F0, A0) from the peak linking unit 8,
(F1, A1), (F2, A2),... Are sequentially output, and interpolation processing is performed for each of the sine wave components. Therefore, each partial waveform generation unit 9a determines the frequency and amplitude within a predetermined frequency domain. Output a waveform that fluctuates. Then, the waveforms output from the respective partial waveform generators 9a are added and synthesized in an adder 9b. Therefore, the output signal of the interpolation / synthesis unit 9 is a sine wave component synthesized signal SSS obtained by extracting a deterministic component from the input audio signal Sv.

[2.2] Operation of Residual Component Detecting Unit Next, the residual component detecting unit 10 calculates the deviation between the sine wave component synthesized signal SSS output from the interpolation synthesizing unit 9 and the input voice signal Sv. A residual component signal SRD (time waveform) is generated. This residual component signal SRD contains a lot of unvoiced components included in the voice. On the other hand, the above-mentioned sine wave component composite signal SSS corresponds to a voiced component. By the way, in order to resemble the voice of the singer who becomes the target (Target), if only the voiced sound is processed, it is not necessary to process the unvoiced sound. Therefore, in the present embodiment, speech conversion processing is performed on a deterministic component corresponding to a voiced vowel component. More specifically, regarding the residual component signal SRD,
The fast Fourier transform unit 11 converts the signal into a frequency waveform, and the obtained residual component signal (frequency waveform) is stored in the residual component storage unit 12 as Rme (f).

[2.3] Operation of Average Amplifier Operation Unit On the other hand, as shown in FIG. 8A, sine wave components (F0, A) output from the peak detection unit 5 through the peak linking unit 8
0), (F1, A1), (F2, A2),..., (F (N
-1), A (N-1)) N sine wave components (hereinafter, these are collectively referred to as Fn and An. N = 0 to (N-1).)
Is held in the sine wave component holding unit 13, and the amplifier An is input to the average amplifier operation unit 14, and the average amplifier Ame is calculated for each frame by the following equation. Ame = Σ (An) / N

[2.4] Operation of Amplifier Normalization Unit Next, in the amplifier normalization unit 15, each amplifier An is normalized by the average amplifier Ame by the following equation, and the normalized amplifier A'n
Ask for. A'n = An / Ame [2.5] Operation of Spectral Shape Computing Unit Then, in the spectral shape computing unit 16,
As shown in FIG. 8B, an envelope (envelope) having a sine wave component (Fn, A'n) obtained by the frequency Fn and the normalizing amplifier A'n as a break point has a spectral shape Sme (f). Generate as In this case, the value of the amplifier at the frequency between the two breakpoints,
It is calculated by linear interpolation. The method of interpolation is not limited to linear interpolation.

[2.6] Operation of Pitch Normalization Unit Subsequently, the pitch normalization unit 17 normalizes each frequency Fn with the pitch Pme detected by the pitch detection unit 7 to obtain a normalized frequency F'n. F′n = Fn / Pme As a result, the original frame information holding unit 18 obtains the average amplifier Ame, the pitch Pme, and the spectral shape Sme (f) which are the original attribute data corresponding to the sine wave component included in the input audio signal Sv. ), And hold the normalized frequency F'n. In this case, the normalized frequency F'n
Represents the relative value of the frequency of the harmonic train, and need not be retained if the harmonic structure of the frame is treated as a complete harmonic structure.

In this case, if male / female conversion is to be performed, at this stage, if male / female conversion is to be performed, the pitch is raised by an octave, and female →
When performing male voice conversion, it is preferable to perform male / female voice pitch control processing for lowering the pitch by an octave.
Subsequently, of the original attribute data held in the original frame information holding unit 18, the average amplifier Ame and the pitch Pme are further subjected to the static change / vibrato change separation unit 1.
9, a filtering process or the like is performed to separate and retain a static variation component and a vibrato variation component. In addition, it is also possible to configure so as to further separate a jitter variable component which is a higher frequency change component from a vibrato variable component.

More specifically, the average amplifier Ame is separately held as an average amplifier static component Ame-sta and an average amplifier vibrato-like component Ame-vib. The pitch Pme is defined as a pitch static component Pme-sta and a pitch vibrato-like component Pme.
-vib and keep separately. As a result, the original frame information data INFme of the corresponding frame is as shown in FIG.
As shown in the figure, the average amplifier static component Ame-sta, the average amplifier vibrato-like component Ame-vib, and the pitch static component P are original attribute data corresponding to the sine wave component of the input audio signal Sv.
It is held in the form of me-sta, pitch vibrato-like component Pme-vib, spectral shape Sme (f), normalized frequency F'n, and residual component Rme (f).

On the other hand, the target frame information data INFtar composed of the target attribute data corresponding to the singer to be imitated (target) is analyzed in advance and stored in a hard disk or the like constituting the target frame information storage unit 20 in advance. Have been. In this case, of the target frame information data INFtar,
As target attribute data corresponding to the sine wave component,
Average amplifier static component Atar-sta, average amplifier vibrato component Atar-vib, pitch static component Ptar-sta, pitch vibrato component Ptar-vib, spectral shape St
There is ar (f). Further, the target frame information data I
Among the NFtars, target attribute data corresponding to the residual component includes a residual component Rtar (f).

[2.7] Operation of Key Control / Tempo Change Unit Next, the key control / tempo change unit 21 responds to the synchronization signal SSYNC from the target frame information holding unit 20 based on the synchronization signal SSYNC from the sequencer 31. Of target frame information INFtar of the frame to be read and read target frame information data IN
The target attribute data constituting the Ftar is corrected, and the read target frame information INF is read.
It outputs tar and a target unvoiced / voiced detection signal U / Vtar indicating whether the frame is unvoiced or voiced. More specifically, the key control unit (not shown) of the key control / tempo change unit 21 is configured such that when the key of the karaoke apparatus is raised or lowered from a reference, the pitch static component Ptar-sta and the pitch vibrato-like component Ptar which are the target attribute data. For -vib, a correction process of raising and lowering the same is performed. For example, when the key is raised by 50 [cent], the pitch static component Ptar-sta and the pitch vibrato-like component Ptar-vib are also increased by 50 [c].
ent].

A tempo change unit (not shown) of the key control / tempo change unit 21 reads the target frame information data INFtar at a timing corresponding to the changed tempo when the tempo of the karaoke apparatus is raised or lowered. There is a need to do. In this case, if there is no target frame information data INFtar corresponding to the timing corresponding to the required frame, the target frame information data INFtar of the two frames existing before and after the timing of the required frame is read out. Interpolation is performed using these two pieces of target frame information data INFtar to generate target frame information data INFtar of the frame at the necessary timing, and furthermore, target attribute data.

In this case, the vibrato-like components (average amp vibrato-like component Atar-vib and pitch vibrato-like component Ptar-vib) are unsuitable as they are, because the vibrato period itself changes and is unsuitable. It is necessary to perform interpolation processing so that the period does not change. Alternatively, this problem can be avoided by holding the parameters of the vibrato cycle and the vibrato depth instead of the data representing the vibrato trajectory itself as the target attribute data and calculating the actual trajectory by calculation.

[2.8] Operation of Easy Synchronization Processing Unit Next, the easy synchronization processing unit 22 adds the original frame information data INFme to the frame of the singer who wants to imitate (hereinafter referred to as the original frame). Despite the presence of, the singer's frame that is the target of the corresponding singer (hereinafter referred to as the target frame)
If the target frame information data INFtar does not exist in the target frame, the target frame information data INFta
An easy synchronization process is performed using r as the target frame information data INFtar of the target frame. Then, the easy synchronization processing unit 22 performs the target attribute data (average amplifier static component Atar-sync-sta, average amplifier static component Atar-sync-sta) of the target attribute data included in the replaced target frame information data INFtar-sync described later. Amp vibrato-like component Atar-sync-vib, pitch static component Ptar-sync-s
ta, a pitch vibrato-like component Ptar-sync-vib, and a spectral shape Star-sync (f)) are output to the modified spectral shape generator 23.

Further, the easy synchronization processing section 22 generates target attribute data (residual component Rtar-sync (f)) relating to a residual component among target attribute data included in replaced target frame information data INFtar-sync to be described later. ) Is output to the residual component selector 25. Also in the processing in the easy synchronization processing unit 22, the vibrato-like components (the average amplifier vibrato-like component Atar-vib and the pitch vibrato-like component Ptar
With respect to -vib), if it is left as it is, the vibrato cycle itself changes and is inappropriate, so it is necessary to perform interpolation processing so that the cycle does not fluctuate. Alternatively, this problem can be avoided by holding the parameters of the vibrato cycle and the vibrato depth instead of the data representing the vibrato trajectory itself as the target attribute data and calculating the actual trajectory by calculation.

[2.8.1] Details of Easy Synchronization Process Here, the easy synchronization process will be described in detail with reference to FIGS. 9 and 10. FIG. 9 is a timing chart of the easy synchronization process, and FIG. 10 is a flowchart of the easy synchronization process. First, the easy synchronization processing unit 22 sets the synchronization mode = “0” indicating the method of the synchronization processing (step S11). This synchronization mode = "0" corresponds to the case of normal processing in which the target frame information data INFtar exists in the target frame corresponding to the original frame. Then, it is determined whether or not the original unvoiced / voiced detection signal U / Vme (t) at a certain timing t has changed from unvoiced (U) to voiced (V) (step S12).

For example, as shown in FIG.
At = t1, the original unvoiced / voiced detection signal U / Vme (t) changes from unvoiced (U) to voiced (V). Step S
In the determination of No. 12, the original unvoiced / voiced detection signal U / Vme
If (t) has changed from unvoiced (U) to voiced (V) (Step S12; Yes), the original unvoiced / voiced detection signal U / Vme at the previous timing t-1 of the timing t.
It is determined whether (t-1) is unvoiced (U) and the target unvoiced / voiced detection signal U / Vtar (t-1) is unvoiced (U) (step S18). For example, as shown in FIG. 9, at timing t = t0 (= t1-1), the original unvoiced / voiced detection signal U / Vme (t-1) is unvoiced (U) and the target unvoiced / voiced detection signal U / Vme (t-1). Vtar (t-1) is silent (U). In the determination in step S18, the original unvoiced / voiced detection signal U / Vme (t-1) is unvoiced (U) and the target unvoiced /
When the voiced detection signal U / Vtar (t-1) is unvoiced (U) (step S18; Yes), the target frame includes the target frame information data INFta.
Since r does not exist, the synchronization mode =
It is set to “1”, and the replacement target frame information data I
Set NFhold in the backward direction of the target frame (Backwar
This is the target frame information of the frame existing in d).

For example, as shown in FIG.
= T1 to t2, since the target frame information data INFtar does not exist, the synchronization mode is set to “1”, and the replacement target frame information data INFhold is set to a frame existing in the backward direction of the target frame (ie, , Target frame information data backward at timing t = frames existing at t2 to t3). Then, the process proceeds to step S15, and the synchronization mode =
It is determined whether it is “0” (step S15). If it is determined in step S15 that the synchronization mode is “0”, the target frame corresponding to the original frame at the timing t includes the target frame information data INFtar (t), that is, the normal process. Therefore, the replaced target frame information data INFtar-sync is set as target frame information data INFtar (t). INFtar-sync = INFtar (t)

For example, as shown in FIG.
Since target frame information data INFtar exists in the target frame from t2 to t3, INFtar-sync = INFtar (t) is set. In this case, the target attribute data (average amplifier static component Ata) included in the replaced target frame information data INFtar-sync used in the subsequent processing
r-sync-sta, average amp vibrato component Atar-sync-v
ib, pitch static component Ptar-sync-sta, pitch vibrato-like component Ptar-sync-vib, spectral shape Star-
The sync (f) and the residual component Rtar-sync (f) have substantially the following contents (step S16). Atar-sync-sta = Atar-sta Atar-sync-vib = Atar-vib Ptar-sync-sta = Ptar-sta Ptar-sync-vib = Ptar-vib Star-sync (f) = Star (f) Rtar-sync (f) = Rtar (f)

If it is determined in step S15 that the synchronization mode = "1" or the synchronization mode = "1", the target frame information data INFtar (t) is added to the target frame corresponding to the original frame at the timing t. ) Does not exist, the replaced target frame information data I
Target frame information data I for replacing NFtar-sync
NFhold. INFtar-sync = INFhold For example, as shown in FIG. 9, the target frame at the timing t = t1 to t2 does not have the target frame information data INFtar, and the synchronization mode = “1”. = Target frame information data IN
Since the Ftar exists, the process P1 of setting the replaced target frame information data INFtar-sync as replacement target frame information data INFhold, which is the target frame information data of the target frame at the timing t = t2 to t3, is performed. The target attribute data included in the replaced target frame information data INFtar-sync used is an average amplifier static component Atar-sync-s
ta, average amp vibrato component Atar-sync-vib, pitch static component Ptar-sync-sta, pitch vibrato component P
tar-sync-vib, spectral shape Star-sync (f)
And the residual component Rtar-sync (f) (step S1).
6).

As shown in FIG. 9, the timing t =
In the target frame from t3 to t4, the target frame information data INFtar does not exist, and the synchronization mode = “2”.
Since the target frame information data INFtar exists in the target frame at t3, the replaced target frame information data INFtar-sync is set at the timing t = t.
Replacement target frame information data IN which is the target frame information data of the target frames from 2 to t3
The target attribute data included in the replaced target frame information data INFtar-sync used in the subsequent processing includes an average amplifier static component Atar-sync-sta and an average amplifier vibrato-like component Atar-sy.
nc-vib, pitch static component Ptar-sync-sta, pitch vibrato-like component Ptar-sync-vib, spectral shape S
The result is a tar-sync (f) and a residual component Rtar-sync (f) (step S16).

In the determination in step S12, the original silent /
The voiced detection signal U / Vme (t) changes from unvoiced (U) to voiced (V)
(Step S12; No),
It is determined whether or not the target unvoiced / voiced detection signal U / Vtar (t) has changed from voiced (V) to unvoiced (U) (step S13). In the determination in step S13,
If the target unvoiced / voiced detection signal U / Vtar (t) changes from voiced (V) to unvoiced (U) (step S13; Yes), the original unvoiced / voiced signal at the previous timing t-1 of the timing t is output. The voiced detection signal U / Vme (t-1) is voiced (V) and the target unvoiced / voiced detection signal U / Vtar
It is determined whether or not (t-1) is voiced (V) (step S19).

For example, as shown in FIG.
Target unvoiced / voiced detection signal U / Vtar (t) at 3
Changes from voiced (V) to unvoiced (U) at timing t-1
= T2 to t3, the original unvoiced / voiced detection signal U / Vme
(t-1) is voiced (V) and the target unvoiced / voiced detection signal U / Vtar (t-1) is voiced (U). Step S
In the determination of No. 19, the original unvoiced / voiced detection signal U / Vme (t
-1) is voiced (V) and the target unvoiced / voiced detection signal U
If / Vtar (t-1) is voiced (V) (step S19; Yes), the target frame includes
Since the target frame information data INFtar does not exist, the synchronization mode is set to “2”, and the replacement target frame information data INFhold is set as the target frame information of the frame existing in the forward direction of the target frame.

For example, as shown in FIG.
= T3 to t4, since the target frame information data INFtar does not exist in the target frame, the synchronization mode is set to “2” and the replacement target frame information data INFhold is set to a frame existing in the forward direction of the target frame (that is, , Target frame information data forward at a timing t = frames t2 to t3). Then, the process proceeds to step S15, and the synchronization mode =
It is determined whether or not it is “0” (step S15), and the same processing is performed thereafter. If it is determined in step S13 that the target unvoiced / voiced detection signal U / Vtar (t) has not changed from voiced (V) to unvoiced (U) (step S13; No), the original unvoiced / voiced at the timing t. The detection signal U / Vme (t) changes from voiced (V) to unvoiced (U), or the target unvoiced / voiced detection signal U /
It is determined whether Vtar (t) has changed from unvoiced (U) to voiced (V) (step S14).

In the determination in step S14, the original unvoiced / voiced detection signal U / Vme (t) at the timing t changes from voiced (V) to unvoiced (U), or the target unvoiced / voiced detection signal U / Vtar ( If t) changes from unvoiced (U) to voiced (V) (step S14; Ye)
s), the synchronization mode is set to "0", the replacement target frame information data INFhold is initialized (cleared), the process proceeds to step S15, and the same process is performed. In the determination of step S14, the original unvoiced / voiced detection signal U / V at timing t
me (t) does not change from voiced (V) to unvoiced (U), and
If the target unvoiced / voiced detection signal U / Vtar (t) has not changed from unvoiced (U) to voiced (V) (step S14; No), the process proceeds to step S15 as it is, and thereafter the same process is performed. I do.

[2.9] Operation of Modified Spectral Shape Generation Unit Subsequently, the modified spectral shape generation unit 23 corresponds to the sine wave component of the input audio signal Sv input from the static change / vibrato change change separation unit 19. Average attribute static component Ame-sta, average amplifier vibrato component Ame-vib, pitch static component Pme-sta, pitch vibrato component Pme-vib, spectral shape Sme
(f), the normalized frequency F′n, the target attribute data regarding the sine wave component among the target attribute data included in the replaced target frame information data INFtar-sync input from the easy synchronization processing unit 22 (average amplifier static Component Atar-sync-sta, average amp vibrato component Atar-sync-vib, pitch static component Ptar-
Based on the sync-sta, the pitch vibrato-like component Ptar-sync-vib, and the spectral shape Star-sync (f)) and the deformed spectral shape generation information input from the controller 29, the deformed spectral shape as a new spectral shape is obtained. Generate the shape Snew (f). The generation of the modified spectral shape is performed by shifting the spectral shape corresponding to the former singer (or the target spectral shape corresponding to the target singer) by a constant α in the frequency axis direction.

In this case, the constant α is obtained as follows. A shift coefficient β (β = β
1.2 to 1.3 / oct), α = βk k = log 2 (forg / ftar) where forg: pitch corresponding to the former singer ftar: pitch corresponding to the target singer Rounds off the decimal part of k. Here, more specifically, the deformed spectral shape Snew
The generation of (f) will be described.

[2.9.1] Case of Male Voice to Female Voice Conversion First, a case where the target singer is a female and the former singer is a male will be described. FIG. 11 shows the spectral shape of a woman who is the target singer. FIG.
As shown in (1), frequency components included in the sine wave component of the target singer are represented by ff0 to ffn. FIG. 12 shows a spectral shape of a man who is a former singer.
As shown in FIG. 12, frequency components included in the sine wave component of the former singer are represented by fm0 to fmn. The amplifiers corresponding to the frequency components fm0 to fmn are Afm0 to Afmn.
It is represented by In this case, the amplifier A (ff) of the target singer = Aff0, Aff1,..., Affn is kept as it is, and only the frequency components ff0 to ffn are multiplied by α, that is, the spectral is equal to the value of the constant α. Generate a deformed spectral shape Snew (f) by shifting the shape to the lower side along the frequency axis. First, the constant α is calculated. In this case, the shift coefficient β is set to 1.2 / oct. In this case, FIG.
As shown in FIG. 12 and FIG. 12, since forg = fm0 ftar = ff0, k = log2 (forg / ftar) = log2 (fm0 / ff0). Thus, α = βk = 1.2k. That is, assuming that the frequency components corresponding to the deformed spectral shape are fh0 to fhn, fh0 = α · ff0 fh1 = α · ff1 fh2 = α · ff2... Fhn = α · ffn, and is shown in FIG. A modified spectral shape Snew (f) specified by a modified sine wave component group (= a group of sine wave components represented by frequency components and amplifiers) is obtained. (Fh0, Aff0) (fh1, Aff1) (fh2, Aff2) ... (fh0, Aff0)

[2.9.2] Conversion from Female to Male Voice Next, a case where the target singer is a male and the former singer is a female will be described. Let the male spectral shape shown in FIG. 12 be the spectral shape of the target singer. In this case, the amplifier A (fm) of the target singer = Afm0, Afm1,..., Afmn is unchanged, and only the frequency components fm0 to fmn are multiplied by α. A deformed spectral shape is generated by shifting the shape to a higher frequency side along the frequency axis. First, the constant α is calculated. In this case, the shift coefficient β = 1.2 /
oct. In this case, as shown in FIGS. 11 and 12, since forg = ff0 and ftar = fm0, k = log2 (forg / ftar) = log2 (ff0 / fm0). Thus, α = βk = 1.2k.

That is, assuming that the frequency components corresponding to the deformed spectral shape are fh0 to fhn, fh0 = α · fm0 fh1 = α · fm1 fh2 = α · fm2... Fhn = α · fmn, and FIG. A modified spectral shape Snew (f) specified by the following modified sine wave component group is obtained. (Fh0, Afm0) (fh1, Afm1) (fh2, Afm2) ... (fh0, Afm0)

By the way, generally, when the amplifier component is large, it becomes a bright sound that extends to a high frequency range, and when the amplifier component is small, it becomes a muffled sound. Therefore, regarding the new spectral shape Snew (f),
In order to simulate such a state, as shown in FIG. 15, the high frequency component of the spectral shape, that is,
A more realistic sound can be reproduced by performing and controlling spectral tilt correction for compensating for the inclination of the spectral shape of the high-frequency component according to the magnitude of the new amplifier component Anew. . Subsequently, based on the deformed spectral shape processing information input from the controller 29 as needed, the generated deformed spectral shape processing unit 2 performs the generated deformed spectral shape Snew (f).
4 is used to further process the waveform. For example, waveform processing such as extending the deformed spectral shape Snew (f) entirely is performed.

[2.10] Operation of Residual Component Selection Unit On the other hand, the residual component selection unit 25 includes a target attribute included in the replaced target frame information data INFtar-sync input from the easy synchronization processing unit 22. Target attribute data (residual component Rtar-sync (f)) relating to the residual component of the data, the residual component signal (frequency waveform) Rme (f) held in the residual component holding unit 12 and input from the controller 29 Based on the residual component attribute data selection information to be generated, a new residual component Rnew (f), which is new residual component attribute data, is generated. That is, the new residual component Rnew (f) is generated by the following equation. Rnew (f) = R * (f) (* is me or tar-sync) In this case, whether to select me or tar-sync is the same as the new spectral shape Snew (f) It is more preferred to select Further, as for the new residual component Rnew (f), in order to simulate the same state as the new spectral shape, as shown in FIG. 10, the high frequency component of the residual component, that is, the high frequency component portion Spectral tilt compensation for compensating the slope of the residual component according to the magnitude of the new amplifier component Anew
section), and by controlling, a more realistic sound can be reproduced.

[2.11] Operation of Sine Wave Component Generation Unit Next, the sine wave component generation unit 26
A new sine wave component (F "0, A" 0) in the frame based on the deformed spectral shape Snew (f) without or with waveform processing output from the shape processing unit 24, (F "1, A" 1),
(F "2, A" 2), ..., (F "(N-1), A" (N-1))
N sinusoidal wave components (hereinafter collectively referred to as F ″ n,
A "n. N = 0 to (N-1).
More specifically, a new frequency f "n and a new amplifier a" n are obtained by the following equations. F "n = F'n.times.Pnew A" n = Snew (f "n) .times.Anew If it is considered as a model of a perfect harmonic structure, F" n = (n + 1) .times.Pnew.

[2.12] Operation of Sine Wave Component Deformation Unit Further, the obtained new frequency F "n and new amplifier A"
n is further modified by the sine wave component transformation unit 27 based on the sine wave component transformation information input from the controller 29 as necessary, and the new frequency F ″ ′ n
And new amplifiers A "'n. For example, new amplifiers A" n (= A "0, A" 2, A ") of even-order components
4,...) Are increased (for example, doubled). As a result, it is possible to give the converted speech further variety.

[2.13] Operation of Inverse Fast Fourier Transform Unit Next, the inverse fast Fourier transform unit 28 calculates the new frequency F ″ ′ n and the new amplifier A ″ ′ n (= new sine wave component) and the new residual The difference component Rnew (f) is stored in the FFT buffer, the inverse FFT is sequentially performed, the obtained time axis signals are overlapped so as to partially overlap, and an addition process of adding them is performed to perform a new addition. A converted voice signal that is a time axis signal of the voice is generated. At this time, based on the sine wave component / residual component balance control signal input from the controller 29, the mixing ratio of the sine wave component and the residual component is controlled to obtain a more realistic voiced signal. In this case, generally, a rough voice is obtained by increasing the mixing ratio of the residual components.

In this case, when the new frequency f "n, the new amplifier a" n (= new sine wave component) and the new residual component Rnew (f) are stored in the FFT buffer, a different pitch and an appropriate pitch are used. By adding the sine wave component converted by the above, harmony can be obtained as a converted audio signal. Further, by giving a harmony pitch adapted to the accompaniment sound by the sequencer 31, musical harmony adapted to the accompaniment can be obtained.

[2.14] Operation of Crossfader Next, the crossfader 30 transmits the original unvoiced / voiced detection signal U /
If the input audio signal Sv is unvoiced (U) based on Vme (t), the input audio signal Sv is output to the mixer 30 as it is. If the input audio signal Sv is voiced (V), the converted audio signal output from the inverse FFT converter 28 is output to the mixer 30. In this case, the reason why the cross fader 30 is used as the changeover switch is to prevent a click sound from occurring at the time of switch changeover by performing a crossfade operation.

[2.15] Operation of Sequencer, Sound Source Unit, Mixer, and Output Unit On the other hand, the sequencer 31 transmits sound source control information for generating a karaoke accompaniment sound to, for example, MIDI (Musical Instrument).
rument Digital Interface) sound source section 3 as data etc.
Output to 2. Thereby, the sound source section 32 generates an accompaniment signal based on the sound source control information, and outputs the accompaniment signal to the mixer 33.
The mixer 33 mixes either the input audio signal Sv or the converted audio signal and the accompaniment signal, and outputs the mixed signal to the output unit 34. The output unit 34 has an amplifier (not shown), amplifies the mixed signal, and outputs it as an acoustic signal.

[3] Modified Example of Embodiment [3.1] First Modified Example In the description of the above embodiment, the spectral shape of the target singer is shifted, but similarly, the original shape is changed. It can also be configured to shift the singer's spectral shape. [3.2] Second Modification In the description of the above-described embodiment, the modified spectral shape generated based on the spectral shape of the former singer and the frequency component included in the sine wave component of the target voice signal of the target singer The sine wave component group is calculated based on the target singer, and the converted sine wave component is obtained.However, the sine wave component of the input voice signal of the original singer and the modified spectral shape generated based on the spectral shape of the target singer are included. It is also possible to adopt a configuration in which a sine wave component group is calculated based on the frequency components to be obtained to obtain a converted voice.

[3.3] Third Modification The extraction of the sine wave component is not limited to the method used in this embodiment. In short, it is only necessary to extract a sine wave component included in the audio signal. [3.4] Fourth Modification In the present embodiment, the sine wave component and the residual component of the target are stored. Instead, the target voice itself is stored, read out, and the sine wave is read out by real-time processing. The wave component and the residual component may be extracted. That is, processing similar to the processing performed on the voice of the singer trying to imitate in the present embodiment may be performed on the voice of the target singer.

[4] Effects of the Embodiment As a result, along with the accompaniment of karaoke, the song of the former singer is output, and its voice quality and singing style are output as converted voices greatly influenced by the target singer. However, unlike the case where fixed effect parameters depending on the singer and the music are used, the obtained converted sound is converted into a sound using a (dynamic) effect parameter based on the raw voice component of the original singer, and the natural sound is audibly natural. It becomes something.

[0061]

As described above, according to the present invention, it is possible to easily obtain a converted sound that is natural in audibility.

[Brief description of the drawings]

FIG. 1 is a block diagram (part 1) illustrating a configuration of an embodiment of the present invention.

FIG. 2 is a block diagram (part 2) showing a configuration of an embodiment of the present invention.

FIG. 3 is a diagram illustrating a state of a frame according to the embodiment.

FIG. 4 is an explanatory diagram for describing peak detection of a frequency spectrum in the embodiment.

FIG. 5 is a diagram illustrating cooperation of peak values for each frame in the embodiment.

FIG. 6 is a diagram illustrating a change state of a frequency value in the embodiment.

FIG. 7 is a diagram showing a change state of a deterministic component in a process in the embodiment.

FIG. 8 is an explanatory diagram of signal processing in the embodiment.

FIG. 9 is a timing chart of an easy synchronization process.

FIG. 10 is a flowchart of an easy synchronization process.

FIG. 11 is a diagram illustrating a female spectral shape.

FIG. 12 is a diagram illustrating a male spectral shape.

FIG. 13 is an explanatory diagram of a male-to-female conversion process.

FIG. 14 is an explanatory diagram of a female-to-male conversion process.

FIG. 15 is a diagram for explaining spectral tilt compensation of a spectral shape.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Microphone, 2 ... Analysis window generation part, 3 ... Input audio signal extraction part, 4 ... Fast Fourier transform part, 5 ... Peak detection part, 6 ...
Unvoiced / voiced detection unit, 7: pitch extraction unit, 8: peak linking unit, 9: interpolation synthesis unit, 10: residual component detection unit, 11: fast Fourier transform unit, 12: residual component holding unit, 13: sine Wave component holding unit, 14: average amplifier calculation unit, 15: amplifier normalization unit, 16: spectral shape calculation unit, 17
... Pitch normalizing section, 18 ... Original frame information holding section, 19
... Static change / vibrato change separation section, 20 ... Target frame information holding section, 21 ... Key control / tempo change section, 22 ... Easy synchronization processing section, 23 ... Deformed spectral shape generation section, 2
4 ... deformed spectral shape processing section, 25 ... residual component selection section, 26 ... sine wave component generation section, 27 ... sine wave component deformation section, 28 ... inverse fast Fourier transform section, 29 ... controller, 30 ... cross fader section , 31 ... sequencer, 3
2 ... sound source section, 33 ... mixer, 34 ... output section

Claims (6)

[Claims] 1. A spectral shape extracting means for extracting a spectral shape on a frequency axis from an input audio signal, a constant α is calculated based on a pitch of the input audio signal and a pitch of a target audio signal, and the spectral shape is calculated. Is shifted by α times in the frequency axis direction to generate a modified spectral shape, and a sine based on the frequency component included in the sine wave component extracted from the target audio signal and the modified spectral shape. Sine wave component information generating means for generating wave component information, generating a converted audio signal based on the sine wave component information,
A voice conversion device, comprising: a voice generation unit for outputting. 2. A sine wave component extracting means for extracting a sine wave component of an input audio signal, and a constant α is calculated based on a pitch of the input audio signal and a pitch of a target audio signal, and is previously extracted from the target audio signal. A target spectral shape deforming means for generating a deformed target spectral shape by shifting the target spectral shape on the frequency axis by α times in the frequency axis direction, a frequency component included in the sine wave component and the deformed target spectrum A sine wave component information generating means for generating sine wave component information based on the shape, and generating a converted audio signal based on the sine wave component information;
A voice conversion device, comprising: a voice generation unit for outputting. 3. The voice converter according to claim 1, wherein a predetermined shift coefficient is β, a pitch of the input voice signal is forg, and a pitch ft of the target voice signal is ft.
A sound conversion apparatus characterized in that when α is set, α is calculated by the following equation. α = βk where k = log2 (forg / ftar) 4. A spectral shape extracting step of extracting a spectral shape on a frequency axis from an input voice, calculating a constant α based on a pitch of the input voice and a pitch of a target voice, and converting the spectral shape into a frequency axis. Spectral shape deformation step of generating a deformed spectral shape by shifting by α times in the direction, and sine wave component information based on the frequency component included in the sine wave component extracted from the target voice and the deformed spectral shape. A voice conversion method, comprising: generating a sine wave component information; and generating a converted voice based on the sine wave component information. 5. A sine wave component extracting step of extracting a sine wave component of the input voice, a constant α is calculated based on the pitch of the input voice and the pitch of the target voice, and a constant α is calculated on the frequency axis previously extracted from the target voice. A target spectral shape deforming step of generating a deformed target spectral shape by shifting the target spectral shape in the frequency axis direction by α times, based on the frequency component included in the sine wave component and the deformed target spectral shape A sine-wave component information generating step of generating sine-wave component information by using the sine-wave component information, and a voice generating step of generating a converted voice based on the sine-wave component information. 6. The voice conversion method according to claim 4, wherein a predetermined shift coefficient is β, a pitch of the input voice signal is forg, and a pitch ft of the target voice signal is ft.
A voice conversion method, wherein, when ar is set, the α is calculated by the following equation. α = βk where k = log2 (forg / ftar)