JP2000003200A - Voice signal processor and voice signal processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily obtain a female converted voice natural on hearing sense by adding an aspirated noise component signal to an original conversion voice signal and outputting it as a conversion voice signal. SOLUTION: A band-pass filter part 106 passes only a white noise signal having a frequency in a prescribed frequency band corresponding to the third formant of the original conversion voice signal detected by a formant frequency detection part 103 among the white noise signals outputted by a white noise generation part 105 under control of a band-pass filter characteristic control part 104 through as an original aspirated noise signal to output it to an amplifier part 109. An aspirated noise level control part 108 outputs the level control signal of the original aspirated noise signal to the amplifier part 109 based on the size of the amplification of the original conversion voice signal detected by an amp/envelope detection part 107. The amplifier part 109 changes the signal level of the original aspirated noise signal based on this signal to output it to a mixer part 110 as an aspirated noise signal.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a voice signal processing apparatus and a voice signal processing method for converting an input voice into another voice and outputting the generated voice, and for generating a synthesized voice. The present invention relates to an audio signal processing apparatus and an audio signal processing method suitable for use in a karaoke apparatus having the apparatus.

[0002]

2. Description of the Related Art There have been developed various voice converters for changing the frequency characteristics and the like of an input voice and outputting the converted voice. For example, some karaoke devices convert the pitch of a singer's singing voice into a male voice. Is converted into a female voice (for example, Japanese Patent Application Laid-Open No. Hei 8-508581).

[0003]

However, in the conventional voice converter, since the pitch of the singing voice is simply converted, a female voice which is natural in terms of audibility can be obtained even when the male voice is converted to the female voice. There was a problem that it could not be done.
Accordingly, an object of the present invention is to provide an audio signal processing device and an audio signal processing method capable of easily obtaining a female voice that is natural in terms of audibility when performing voice conversion from male voice to female voice.

[0004]

According to a first aspect of the present invention, there is provided an audio signal processing apparatus for converting an input voice of a male voice into a converted voice of a female voice and outputting the converted voice.
The apparatus further comprises a breathiness noise adding means for adding a breathiness noise component signal to an original converted voice signal obtained by converting the input voice and outputting the converted voice signal.

According to a second aspect of the present invention, in the configuration of the first aspect, the breathing noise adding means detects a formant frequency of an original converted voice signal obtained by converting the input voice into a female voice. Means, a breathiness noise generation means for generating the breathiness noise component signal having a frequency band corresponding to the detected formant frequency, and the converted speech by superimposing the breathiness noise component signal on the original converted speech signal. And superimposing means for outputting as a signal.

According to a third aspect of the present invention, in the configuration of the second aspect, the breathing noise generating means includes a white noise generating means for generating and outputting a white noise signal, and a detection result of the formant frequency detecting means. On the basis of,
Band-pass filter means for passing only a predetermined frequency band component corresponding to a third formant of the original converted audio signal out of the white noise signal and outputting it as an original breathable noise component signal; hand,
Signal level control means for controlling a signal level of the original breathing noise component signal and outputting the signal as the breathing noise component signal.

A fourth aspect of the present invention is a voice signal processing apparatus for synthesizing and outputting a female voice, wherein the converted voice is added by adding a breathy noise component signal to an original synthesized voice signal obtained by the synthesis. It is characterized by having a breathing noise adding means for outputting as a signal.

[0008] According to a fifth aspect of the present invention, in the configuration of the fourth aspect, the breathiness noise addition means is configured to generate the breathiness noise component signal having a predetermined frequency band corresponding to a formant frequency of the original synthesized speech signal. , And superimposing means for superimposing the breathable noise component signal on the original synthesized speech signal and outputting the converted speech signal as the converted speech signal.

According to a sixth aspect of the present invention, in the configuration of the fifth aspect, the breathing noise generation means generates a white noise signal and outputs the white noise signal; Band-pass filter means for passing only a predetermined frequency band component corresponding to a third formant of the converted audio signal and outputting the signal as an original breathable noise component signal; and based on the original converted audio signal, the original breathable noise component signal And a signal level control means for controlling the signal level of the signal and outputting the signal as the breathing noise component signal.

According to a seventh aspect of the present invention, in the audio signal processing method for converting a male voice input voice into a female voice conversion voice,
The method further comprises a breathiness noise adding step of adding a breathiness noise component to an original converted voice obtained by converting the input voice and obtaining the converted voice.

An eighth aspect of the present invention is a voice signal processing method for synthesizing a female voice, wherein a breathy noise component is added to an original synthesized voice obtained by the synthesis to produce the converted voice. It is characterized by having an addition step.

[0012]

Preferred embodiments of the present invention will now be described with reference to the drawings. [1] Principle Configuration of Embodiment First, the principle of the embodiment will be described with reference to the principle explanatory diagram of FIG. [1.1] Principle Configuration of Embodiment The audio signal processing apparatus 100 includes a male / female conversion unit 102 that converts a male / female voice input from the microphone 101 and outputs the converted voice as an original converted audio signal, and a formant of the original converted audio signal. Formant frequency detector 103 for detecting frequency
And, based on the detected frequency of the third formant,
A band-pass filter characteristic control unit 104 for performing pass band control of a band-pass filter described later; a white noise generation unit 105 for generating white noise and outputting it as a white noise signal; and a band-pass filter characteristic control unit 10
4, a band-pass filter unit 106 that passes only a white noise signal having a frequency in a predetermined frequency band corresponding to the third formant as a breathable noise signal, , An amplifier / envelope detector 107 for detecting the size of the amplifier, and a breathiness noise level controller 108 for outputting a signal level control signal for controlling the signal level of the original breathiness noise signal based on the detected size of the amplifier. And an amplifier unit 109 for changing the signal level of the original breathing noise signal based on the signal level control signal and outputting the signal as a breathing noise signal, and adding the breathing noise signal to the original converted audio signal to obtain a converted audio signal A mixer unit 110 for outputting an audio signal, and a spin unit for performing an electric / acoustic conversion based on the converted audio signal and outputting the converted audio signal. And it is configured to include a mosquito unit 111, a.

[1.2] Principle Operation of Embodiment Next, the principle operation will be described. The male voice input from the microphone 101 is subjected to male-to-female conversion by the male / female converter 102 and is output to the formant frequency detector 103 and the amplifier / envelope detector 107 as an original converted audio signal. The formant frequency detection unit 103 detects a formant frequency (particularly, a third formant) of the original converted audio signal. Bandpass filter characteristic control unit 10
4 controls the pass band of the band-pass filter based on the third formant frequency detected by the formant frequency detection unit 103.

Thus, the band-pass filter section 106
Is under the control of the bandpass filter characteristic control unit 104,
Of the white noise signals output by the white noise generating unit 105, only white noise signals having a frequency in a predetermined frequency band corresponding to the third formant are passed as original breathing noise signals, and output to the amplifier unit 109. On the other hand, the breathiness noise level control unit 108 controls the signal level of the original breathiness noise signal based on the frequency of the original converted audio signal detected by the amplifier / envelope detection unit 107 minus the size of the amplifier on the amplifier axis. The level control signal is output to the amplifier unit 109.

The amplifier unit 109 changes the signal level of the original breathing noise signal based on the signal level control signal,
The signal is output to the mixer unit 110 as a breathing noise signal, and the mixer unit 110 adds the breathing noise signal to the original converted audio signal and outputs the result to the speaker unit 111 as a converted audio signal. Then, the speaker unit 111 performs an electric / acoustic conversion based on the converted audio signal and outputs it as an audio signal.

[1.3] Conclusion In these processings, according to the converted audio signal obtained, the reproduced audio signal (voice) is such that the singing voice (male voice) of the former singer is the same as that of the female singer. It becomes like the singing voice of a natural woman who sang.

[2] Detailed Configuration of the Embodiment FIGS. 1 and 2 show detailed configuration diagrams of the embodiment. In addition,
The present embodiment is an example in which the voice conversion device (voice conversion method) according to the present invention is applied to a karaoke device and configured as a karaoke device capable of performing more natural voice conversion. In FIG. 1, a microphone 1 collects the voice of a former singer (me) and outputs the voice to an input audio signal cutout unit 3 as an input audio signal Sv. In parallel with this, the analysis window generation unit 2 generates an analysis window (for example, a Hamming window) AW having a period that is a fixed multiple (for example, 3.5 times) of the period of the pitch detected in the previous frame. Is output to the input audio signal cutout unit 3. When the initial state or the previous frame is an unvoiced sound (including silence), an analysis window having a fixed period set in advance is output to the input audio signal cutout unit 3 as the analysis window AW.

Thus, the input audio signal extracting section 3 multiplies the input analysis window AW by the input audio signal Sv, cuts out the input audio signal Sv in units of frames, and converts the input audio signal Sv into a frame audio signal FSv. Is output to More specifically, the relationship between the input audio signal Sv and the frames is as shown in FIG. 3, and each frame FL is set so as to partially overlap the previous frame FL. Then, the frame audio signal FSv is analyzed in the fast Fourier transform unit 4, and a local peak is detected by the peak detecting unit 5 from the frequency spectrum output from the fast Fourier transform unit 4, as shown in FIG. .

More specifically, a local peak marked with x is detected in the frequency spectrum as shown in FIG. This local peak is represented as a combination of a frequency value and an amplifier (amplitude) value. That is, as shown in FIG. 4, (F0, A0), (F1, A1), (F
Local peaks are detected and represented for each frame, such as (2, A2),..., (FN, AN). Then, as schematically shown in FIG. 3, one set (hereinafter, referred to as a local peak set) is output to the unvoiced / voiced detection unit 6 and the peak coordination unit 8 for each frame.
The unvoiced / voiced detection unit 6 detects that the voice is unvoiced ('t', 'k', etc.) according to the magnitude of the high-frequency component based on the input local peak for each frame, and performs unvoiced / voiced detection. The signal U / Vme is output to the pitch detection unit 7, the easy synchronization processing unit 22, and the crossfader unit 30. Alternatively, it is detected that there is no voice (such as 's') on the time axis according to the number of zero crosses per unit time,
The original unvoiced / voiced detection signal U / Vme is output to the pitch detection unit 7, the easy synchronization processing unit 22, and the crossfader unit 30.

Further, when the input frame is not detected as unvoiced, the unvoiced / voiced detection unit 6 outputs the input local peak set as it is, and
Output to The pitch detector 7 detects the pitch Pme of the frame corresponding to the local peak set based on the input local peak set. More specific methods for detecting the pitch Pme of a frame include, for example, Maher, RC
andJ.W.Beauchamp: "Fundamental Frequency Estimation
of Musical Signal using a two-way Mismatch Proced
ure "(Journal of Acounstical Society of America95
(4): 2254-2263).

Next, the local peak set output from the peak detecting section 5 is determined by the peak linking section 8 to be linked with the preceding and succeeding frames, and the local peaks recognized to be linked are formed into a series of data strings. A linking process for connecting a local peak to the data is performed. Here, this cooperation processing will be described with reference to FIG.
Now, assume that a local peak as shown in FIG. 5A is detected in the previous frame, and a local peak as shown in FIG. 5B is detected in the current frame.

In this case, the peak coordinating unit 8 calculates each local peak (F0, A0) detected in the previous frame,
(F1, A1), (F2, A2), ..., (FN, A
It is checked whether the local peak corresponding to N) has been detected in the current frame. The determination as to whether or not there is a corresponding local peak is made based on whether or not the local peak of the current frame is detected within a predetermined range centered on the frequency of the local peak detected in the previous frame. More specifically, in the example of FIG. 5, the local peaks (F0, A0), (F1, A1), (F2, A2).
, The corresponding local peak is detected, but for the local peaks (FK, AK), (FIG. 5
(See (A)) and the corresponding local peak (see FIG. 5B) is not detected.

When the corresponding local peaks are detected, the peak linking unit 8 connects the local peaks in chronological order and outputs them as a set of data strings. If the corresponding local peak is not detected, the data is replaced with data indicating that there is no corresponding local peak for the frame. Here, FIG. 6 shows an example of changes in the frequency F0 and the frequency F1 of the local peak over a plurality of frames. Such changes are caused by the amplifiers (amplitude) A0, A
1, A2,... Are similarly recognized. In this case, the data string output from the peak linking unit 8 is a discrete value output at every frame interval.

The peak value output from the peak linking unit 8 is hereinafter referred to as a deterministic component. This means a component that is deterministically replaced as a sine wave element in the original signal (that is, the audio signal Sv). Further, each of the replaced sine waves (strictly speaking, frequency and amplifier (amplitude) which are parameters of the sine wave) will be referred to as sine wave components.

Next, the interpolation synthesizing section 9 performs an interpolation process on the deterministic component output from the peak linking section 8 and synthesizes a waveform based on the deterministic component after the interpolation by a so-called oscillator method. The interpolation interval in this case is determined by the output unit 3 described later.
4 is performed at intervals corresponding to the sampling rate (for example, 44.1 KHz) of the final output signal output. The solid lines shown in FIG. 6 described above are the frequencies F0 and F1 of the sine wave components.
5 shows an image when the interpolation processing is performed for.

[2.1] Configuration of Interpolation Synthesis Unit The configuration of the interpolation synthesis unit 9 is shown in FIG. The interpolation / synthesis unit 9 includes a plurality of partial waveform generation units 9a, and each of the partial waveform generation units 9a adjusts a frequency (F0, F1,...) And an amplifier (amplitude) of a designated sine wave component. Generates a corresponding sine wave. However, the sine wave components (F0, A0), (F1, A1), (F2,
Since A2),... Change every moment according to the interpolation interval, the waveform output from each partial waveform generator 9a becomes a waveform according to the change. That is, the sine wave components (F0, A0) from the peak linking unit 8,
(F1, A1), (F2, A2),... Are sequentially output, and interpolation processing is performed for each of the sine wave components. Therefore, each partial waveform generation unit 9a determines the frequency and amplitude within a predetermined frequency domain. Output a waveform that fluctuates. Then, the waveforms output from the respective partial waveform generators 9a are added and synthesized in an adder 9b. Therefore, the output signal of the interpolation / synthesis unit 9 is a sine wave component synthesized signal SSS obtained by extracting a deterministic component from the input audio signal Sv.

[2.2] Operation of Residual Component Detecting Unit Next, the residual component detecting unit 10 calculates the deviation between the sine wave component synthesized signal SSS output from the interpolation synthesizing unit 9 and the input audio signal Sv. A residual component signal SRD (time waveform) is generated. This residual component signal SRD contains a lot of unvoiced components included in the voice. On the other hand, the above-mentioned sine wave component composite signal SSS corresponds to a voiced component. By the way, in order to resemble the voice of the singer who becomes the target (Target), if only the voiced sound is processed, it is not necessary to process the unvoiced sound. Therefore, in the present embodiment, speech conversion processing is performed on a deterministic component corresponding to a voiced vowel component. More specifically, regarding the residual component signal SRD,
The fast Fourier transform unit 11 converts the signal into a frequency waveform, and the obtained residual component signal (frequency waveform) is stored in the residual component storage unit 12 as Rme (f).

[2.3] Operation of Average Amplifier Operation Unit On the other hand, as shown in FIG. 7A, the sine wave components (F0, A) output from the peak detection unit 5 through the peak linking unit 8
0), (F1, A1), (F2, A2),..., (F (N
-1), A (N-1)) N sine wave components (hereinafter, these are collectively referred to as Fn and An. N = 0 to (N-1).)
Is held in the sine wave component holding unit 13, and the amplifier An is input to the average amplifier operation unit 14, and the average amplifier Ame is calculated for each frame by the following equation. Ame = Σ (An) / N

[2.4] Operation of Amplifier Normalization Unit Next, in the amplifier normalization unit 15, each amplifier An is normalized by the average amplifier Ame by the following equation, and the normalized amplifier A'n
Ask for. A'n = An / Ame [2.5] Operation of Spectral Shape Computing Unit Then, in the spectral shape computing unit 16,
As shown in FIG. 8B, an envelope (envelope) having a sine wave component (Fn, A'n) obtained by the frequency Fn and the normalizing amplifier A'n as a break point has a spectral shape Sme (f). Generate as In this case, the value of the amplifier at the frequency between the two breakpoints,
It is calculated by linear interpolation. The method of interpolation is not limited to linear interpolation.

[2.6] Operation of Pitch Normalization Unit Subsequently, the pitch normalization unit 17 normalizes each frequency Fn with the pitch Pme detected by the pitch detection unit 7 to obtain a normalized frequency F'n. F′n = Fn / Pme As a result, the original frame information holding unit 18 obtains the average amplifier Ame, the pitch Pme, and the spectral shape Sme (f) which are the original attribute data corresponding to the sine wave component included in the input audio signal Sv. ), And hold the normalized frequency F'n. In this case, the normalized frequency F'n
Represents the relative value of the frequency of the harmonic train, and need not be retained if the harmonic structure of the frame is treated as a complete harmonic structure.

In this case, if male / female conversion is going to be performed, at this stage, if male / female conversion is performed, the pitch is raised by an octave, and female →
When performing male voice conversion, it is preferable to perform male / female voice pitch control processing for lowering the pitch by an octave.
Subsequently, of the original attribute data held in the original frame information holding unit 18, the average amplifier Ame and the pitch Pme are further subjected to the static change / vibrato change separation unit 1.
9, a filtering process or the like is performed to separate and retain a static variation component and a vibrato variation component. In addition, it is also possible to configure so as to further separate a jitter variable component which is a higher frequency change component from a vibrato variable component.

More specifically, the average amplifier Ame is separately held as an average amplifier static component Ame-sta and an average amplifier vibrato component Ame-vib. The pitch Pme is defined as a pitch static component Pme-sta and a pitch vibrato-like component Pme.
-vib and keep separately. As a result, the original frame information data INFme of the corresponding frame is as shown in FIG.
As shown in the figure, the average amplifier static component Ame-sta, the average amplifier vibrato-like component Ame-vib, and the pitch static component P are original attribute data corresponding to the sine wave component of the input audio signal Sv.
It is held in the form of me-sta, pitch vibrato-like component Pme-vib, spectral shape Sme (f), normalized frequency F'n, and residual component Rme (f).

On the other hand, target frame information data INFtar composed of target attribute data corresponding to a singer to be imitated (target) is analyzed in advance and stored in advance in a hard disk or the like constituting the target frame information storage unit 20. Have been. In this case, of the target frame information data INFtar,
As target attribute data corresponding to the sine wave component,
Average amplifier static component Atar-sta, average amplifier vibrato component Atar-vib, pitch static component Ptar-sta, pitch vibrato component Ptar-vib, spectral shape St
There is ar (f). Further, the target frame information data I
Among the NFtars, target attribute data corresponding to the residual component includes a residual component Rtar (f).

[2.7] Operation of Key Control / Tempo Change Unit Next, the key control / tempo change unit 21 responds to the synchronization signal SSYNC from the target frame information holding unit 20 based on the synchronization signal SSYNC from the sequencer 21. Of target frame information INFtar of the frame to be read and read target frame information data IN
The target attribute data constituting the Ftar is corrected, and the read target frame information INF is read.
It outputs tar and a target unvoiced / voiced detection signal U / Vtar indicating whether the frame is unvoiced or voiced. More specifically, the key control unit (not shown) of the key control / tempo change unit 21 is configured such that when the key of the karaoke apparatus is raised or lowered from a reference, the pitch static component Ptar-sta and the pitch vibrato-like component Ptar which are the target attribute data. For -vib, a correction process of raising and lowering the same is performed. For example, when the key is raised by 50 [cent], the pitch static component Ptar-sta and the pitch vibrato-like component Ptar-vib are also increased by 50 [c].
ent].

A tempo change unit (not shown) of the key control / tempo change unit 21 reads the target frame information data INFtar at a timing corresponding to the changed tempo when the tempo of the karaoke apparatus is raised or lowered. There is a need to do. In this case, if there is no target frame information data INFtar corresponding to the timing corresponding to the required frame, the target frame information data INFtar of the two frames existing before and after the timing of the required frame is read out. Interpolation is performed using these two pieces of target frame information data INFtar to generate target frame information data INFtar of the frame at the necessary timing, and furthermore, target attribute data. In this case, the vibrato-like component (average amp vibrato-like component A
As for the tar-vib and the pitch vibrato-like component Ptar-vib), the period of the vibrato itself changes and is inappropriate, so that it is necessary to perform interpolation processing so that the period does not fluctuate. Or, if the target attribute data is not data representing the trajectory of the vibrato itself, but the parameters of the vibrato cycle and the vibrato depth are held, and the actual trajectory is obtained by calculation,
This problem can be avoided.

[2.8] Operation of Easy Synchronization Processing Unit Next, the easy synchronization processing unit 22 adds the original frame information data INFme to the frame of the singer who wants to imitate (hereinafter referred to as the original frame). Despite the presence of, the singer's frame that is the target of the corresponding singer (hereinafter referred to as the target frame)
If the target frame information data INFtar does not exist in the target frame, the target frame information data INFta
An easy synchronization process is performed using r as the target frame information data INFtar of the target frame.

Then, the easy synchronization processing section 22 outputs target attribute data relating to a sine wave component (average amplifier static component Atar-sync-) among target attribute data included in replaced target frame information data INFtar-sync, which will be described later. sta, average amp vibrato component Atar-sync-vib, pitch static component Ptar-sync-st
a, a pitch vibrato-like component Ptar-sync-vib and a spectral shape Star-sync (f)) are output to the sine wave component attribute data selection unit 23. Further, the easy synchronization processing unit 22 stores target attribute data (residual component Rtar-sync (f)) relating to a residual component among target attribute data included in replaced target frame information data INFtar-sync described later. Difference component selection unit 25
Output to

This easy synchronization section 2
In the processing in 2, the vibrato-like components (average amp vibrato-like component Atar-vib and pitch vibrato-like component Ptar-vib) are unsuitable as they are because the vibrato period itself changes and is unsuitable.
It is necessary to perform interpolation processing so that the period does not change. Alternatively, this problem can be avoided by holding the parameters of the vibrato cycle and the vibrato depth instead of the data representing the vibrato trajectory itself as the target attribute data and calculating the actual trajectory by calculation.

[2.8.1] Details of Easy Synchronization Process Here, the easy synchronization process will be described in detail with reference to FIGS. 9 and 10. FIG. 9 is a timing chart of the easy synchronization process, and FIG. 10 is a flowchart of the easy synchronization process. First, the easy synchronization unit 22 sets the synchronization mode = "0" indicating the processing method of the synchronization processing (step S11). This synchronization mode = "0" corresponds to the case of normal processing in which the target frame information data INFtar exists in the target frame corresponding to the original frame. Then, it is determined whether or not the original unvoiced / voiced detection signal U / Vme (t) at a certain timing t has changed from unvoiced (U) to voiced (V) (step S12).

For example, as shown in FIG.
At = t1, the original unvoiced / voiced detection signal U / Vme (t) changes from unvoiced (U) to voiced (V). Step S
In the determination of No. 12, the original unvoiced / voiced detection signal U / Vme
If (t) has changed from unvoiced (U) to voiced (V) (Step S12; Yes), the original unvoiced / voiced detection signal U / Vme at the previous timing t-1 of the timing t.
It is determined whether (t-1) is unvoiced (U) and the target unvoiced / voiced detection signal U / Vtar (t-1) is unvoiced (U) (step S18). For example, as shown in FIG. 9, at timing t = t0 (= t1-1), the original unvoiced / voiced detection signal U / Vme (t-1) is unvoiced (U) and the target unvoiced / voiced detection signal U / Vme (t-1). Vtar (t-1) is silent (U). In the determination in step S18, the original unvoiced / voiced detection signal U / Vme (t-1) is unvoiced (U) and the target unvoiced /
When the voiced detection signal U / Vtar (t-1) is unvoiced (U) (step S18; Yes), the target frame includes the target frame information data INFta.
Since r does not exist, the synchronization mode =
It is set to “1”, and the replacement target frame information data I
Set NFhold in the backward direction of the target frame (Backwar
This is the target frame information of the frame existing in d).

For example, as shown in FIG.
= T1 to t2, since the target frame information data INFtar does not exist, the synchronization mode is set to “1”, and the replacement target frame information data INFhold is set to a frame existing in the backward direction of the target frame (ie, , Target frame information data backward at timing t = frames existing at t2 to t3). Then, the process proceeds to step S15, and the synchronization mode =
It is determined whether it is “0” (step S15). If it is determined in step S15 that the synchronization mode is “0”, the target frame corresponding to the original frame at the timing t includes the target frame information data INFtar (t), that is, the normal process. Therefore, the replaced target frame information data INFtar-sync is set as target frame information data INFtar (t). INFtar-sync = INFtar (t)

For example, as shown in FIG.
Since target frame information data INFtar exists in the target frame from t2 to t3, INFtar-sync = INFtar (t) is set. In this case, the target attribute data (average amplifier static component Ata) included in the replaced target frame information data INFtar-sync used in the subsequent processing
r-sync-sta, average amp vibrato component Atar-sync-v
ib, pitch static component Ptar-sync-sta, pitch vibrato-like component Ptar-sync-vib, spectral shape Star-
The sync (f) and the residual component Rtar-sync (f) have substantially the following contents (step S16). Atar-sync-sta = Atar-sta Atar-sync-vib = Atar-vib Ptar-sync-sta = Ptar-sta Ptar-sync-vib = Ptar-vib Star-sync (f) = Star (f) Rtar-sync (f) = Rtar (f)

If it is determined in step S15 that the synchronization mode is "1" or the synchronization mode is "1", the target frame information data INFtar (t) is added to the target frame corresponding to the original frame at the timing t. ) Does not exist, the replaced target frame information data I
Target frame information data I for replacing NFtar-sync
NFhold. INFtar-sync = INFhold For example,
As shown in FIG. 9, the target frame at timing t = t1 to t2 includes target frame information data INF.
If there is no tar, synchronization mode =
Although it becomes "1", the target frame at the timing t = t2 to t3 has the target frame information data INFta
Since r exists, the process P1 is performed to set the replaced target frame information data INFtar-sync as replacement target frame information data INFhold which is the target frame information data of the target frame at the timing t = t2 to t3. The target attribute data included in the replaced target frame information data INFtar-sync used is an average amplifier static component Atar-sync-sta,
Average amp vibrato component Atar-sync-vib, pitch static component Ptar-sync-sta, pitch vibrato component Ptar-
The result is a sync-vib, a spectral shape Star-sync (f), and a residual component Rtar-sync (f) (step S16).

As shown in FIG. 9, the timing t =
In the target frame from t3 to t4, the target frame information data INFtar does not exist, and the synchronization mode = “2”.
Since the target frame information data INFtar exists in the target frame at t3, the replaced target frame information data INFtar-sync is set at the timing t = t.
Replacement target frame information data IN which is the target frame information data of the target frames from 2 to t3
The target attribute data included in the replaced target frame information data INFtar-sync used in the subsequent processing includes an average amplifier static component Atar-sync-sta and an average amplifier vibrato-like component Atar-sy.
nc-vib, pitch static component Ptar-sync-sta, pitch vibrato-like component Ptar-sync-vib, spectral shape S
The result is a tar-sync (f) and a residual component Rtar-sync (f) (step S16). In the determination in step S12, the original silent /
The voiced detection signal U / Vme (t) changes from unvoiced (U) to voiced (V)
(Step S12; No),
It is determined whether or not the target unvoiced / voiced detection signal U / Vtar (t) has changed from voiced (V) to unvoiced (U) (step S13).

If it is determined in step S13 that the target unvoiced / voiced detection signal U / Vtar (t) changes from voiced (V) to unvoiced (U) (step S13; Y)
es), the original unvoiced / voiced detection signal U / Vme (t-1) at the previous timing t-1 of the timing t is voiced (V) and the target unvoiced / voiced detection signal U / Vtar (t-1) is voiced ( V) is determined (step S19). For example, as shown in FIG. 9, at timing t3, the target unvoiced / voiced detection signal U / Vtar (t) changes from voiced (V) to unvoiced (U), and at timing t-1 = t2 to t3, The unvoiced / voiced detection signal U / Vme (t-1) is voiced (V) and the target unvoiced / voiced detection signal U / Vtar (t-
1) is voiced (U).

In the determination in step S18, the original silent /
If the voiced detection signal U / Vme (t-1) is voiced (V) and the target unvoiced / voiced detection signal U / Vtar (t-1) is voiced (V) (step S19; Yes), Since the target frame information data INFtar does not exist in the target frame, the synchronization mode is set to “2” and the replacement target frame information data INFhold is set to the target of the frame existing in the forward direction of the target frame. Frame information. For example, as shown in FIG.
= T3 to t4, since the target frame information data INFtar does not exist in the target frame, the synchronization mode is set to “2” and the replacement target frame information data INFhold is set to a frame existing in the forward direction of the target frame (that is, , Target frame information data forward at a timing t = frames t2 to t3).

Then, the process shifts to step S15.
It is determined whether or not the synchronization mode is "0" (step S15), and the same processing is performed thereafter. In the determination in step S13, the target unvoiced /
The voiced detection signal U / Vtar (t) changes from voiced (V) to unvoiced (U)
(Step S13; No),
Original unvoiced / voiced detection signal U / Vme at timing t
(t) changes from voiced (V) to unvoiced (U), or
It is determined whether or not the target unvoiced / voiced detection signal U / Vtar (t) has changed from unvoiced (U) to voiced (V) (step S14).

In the determination in step S14, the original unvoiced / voiced detection signal U / Vme (t) at the timing t changes from voiced (V) to unvoiced (U), or the target unvoiced / voiced detection signal U / Vtar ( If t) changes from unvoiced (U) to voiced (V) (step S14; Ye)
s), the synchronization mode is set to "0", the replacement target frame information data INFhold is initialized (cleared), the process proceeds to step S15, and the same process is performed. In the determination of step S14, the original unvoiced / voiced detection signal U / V at timing t
When me (t) does not change from voiced (V) to unvoiced (U), or when target unvoiced / voiced detection signal U / Vtar (t) does not change from unvoiced (U) to voiced (V). (Step S14; No), the process proceeds to Step S15, and the same process is performed thereafter.

[2.9] Operation of Modified Spectral Shape Generation Unit Subsequently, the modified spectral shape generation unit 23 responds to the sine wave component of the input audio signal Sv input from the static change / vibrato change change separation unit 19. Average attribute static component Ame-sta, average amplifier vibrato component Ame-vib, pitch static component Pme-sta, pitch vibrato component Pme-vib, spectral shape Sme
(f), the normalized frequency F′n, and the target attribute data relating to the sine wave component among the target attribute data included in the replaced target frame information data INFtar-sync input from the easy synchronization section 22 (average amplifier static Component Atar-sync-sta, average amp vibrato-like component Atar-sync-vib, pitch static component Ptar-sy
Based on the nc-sta, the pitch vibrato-like component Ptar-sync-vib and the spectral shape Star-sync (f)) and the deformed spectral shape generation information input from the controller 29, the deformed spectral shape as a new spectral shape is obtained. Generate the shape Snew (f).

The generation of the modified spectral shape is performed by shifting the spectral shape corresponding to the original singer (or the target spectral shape corresponding to the target singer) by a constant α (0 <α ≦ 2) in the frequency axis direction. It is done by doing. Here, the generation of the modified spectral shape Snew (f) will be described more specifically.

[2.9.1] Specific Method of Generating Modified Spectral Shape Snew (f) FIG. 11 shows a spectral spectrum of a woman who is a target singer.
Indicates a shape. As shown in FIG. 11, frequency components included in the sine wave component of the target singer are represented by ff0 to ffn. FIG. 12 shows a spectral shape of a man who is a former singer. As shown in FIG. 12, frequency components included in the sine wave component of the former singer are represented by fm0 to fmn. Amplifiers corresponding to the frequency components fm0 to fmn are represented by Afm0 to Afmn.

In this case, the former singer's amplifier A
(Fm) = Afm0, Afm1,..., Afmn are unchanged, and only the frequency components fm0 to fmn are multiplied by α (1 ≦ α ≦ 2), that is, the spectral shape is shifted by the frequency corresponding to the value of α. A shifted spectral shape Snew (f) is generated by shifting to higher frequencies along the axis. That is,
The frequency component corresponding to the deformed spectral shape is f
When expressed as h0 to fhn, fh0 = α · fm0 fh1 = α · fm1 fh2 = α · fm2... fhn = α · fmn, and a modified sine wave component group (= frequency component and amplifier A modified spectral shape Snew (f) specified by a group of sinusoidal components represented) is obtained. (Fh0, Afm0) (fh1, Afm1) (fh2, Afm2) ... (fh0, Afm0)

By the way, in general, when the amplifier component is large, the sound extends to a high range and has a good sound, and when the amplifier component is small, the sound is muffled. Therefore,
Regarding the new spectral shape Snew (f), in order to simulate such a state, as shown in FIG. 15, the high-frequency component of the spectral shape, that is, the slope of the spectral shape of the high-frequency component is newly set. By performing and controlling spectral tilt correction (spectral tilt correction) for compensating according to the magnitude of the amplifier component Anew, a more realistic sound can be reproduced. Next, the generated deformed spectral
For the shape Snew (f), based on the deformed spectral shape processing information input from the controller 29 as needed, the deformed spectral shape processing unit 24
To further process the waveform. For example, waveform processing such as extending the deformed spectral shape Snew (f) entirely is performed. Then, the deformed spectral shape processing unit 24 obtains the obtained deformed spectral shape Sne.
A third formant is detected based on w (f).

[2.10] Detection of Third Formant FT3 Next, a method of detecting the third formant FT3 is described by using a normalization amplifier A′fK corresponding to two adjacent sine wave components.
The case where the change is performed based on the change in the difference ΔA ′ (fK−fK−1) of A′fK−1 will be described. FIG. 16 shows the state of the deformed spectral shape in the vicinity of the third formant FT3 and the end of the second formant FT2. Since the range of the frequency of the third formant FT3 is usually 1.5 [kHz] or more and 4 [kHz] or less, K that satisfies fK］ 1.5 [kHz] is determined, and its value is set to KS. Further, K that satisfies fKＨｚ4 [kHz] is determined, and its value is set to KE.

Next, the value of K of fK (K = KS,..., KE) is increased. Then, the value of ΔA ′ (fK−fK−1) is observed, and the value of K when ΔA ′ (fK−fK−1) ≧ 0 from the state of ΔA ′ (fK−fK−1) <0 Is p. This is repeated until K = KE, and the frequency fp closest to the average frequency of the third formant FT3 can be detected by setting it as the third formant FT3. Note that the detection of the third formant FT3 is not limited to the above method, but can be obtained by using, for example, a linear prediction method.
Then, the detected third formant FT3 is output to the band-pass filter characteristic control unit 42. In parallel with the output of the third formant FT3, the amplifier AFT3 at the frequency of the third formant FT3 is detected and output to the level control unit 43.

[2.11] Operation of Residual Component Selection Unit On the other hand, the residual component selection unit 25 includes the target attribute data included in the replaced target frame information data INFtar-sync input from the easy synchronization unit 22. Among the target attribute data (residual component Rtar-sync (f)) relating to the residual component, the residual component signal (frequency waveform) Rme (f) held in the residual component holding unit 12, and the input from the controller 29. A new residual component Rnew (f), which is new residual component attribute data, is generated based on the residual component attribute data selection information. That is, the new residual component Rnew (f) is generated by the following equation. Rnew (f) = R * (f) (* is me or tar-sync) In this case, whether to select me or tar-sync is the same as the new spectral shape Snew (f) It is more preferred to select

Further, as for the new residual component Rnew (f), as shown in FIG. 10, in order to simulate a state similar to the new spectral shape, the high frequency component of the residual component, that is, the high frequency component Performing a spectral tilt correction for compensating the gradient of the residual component of the component part according to the magnitude of the new amplifier component Anew,
By controlling, more realistic sound can be reproduced.

[2.12] Operation of Sine Wave Component Generation Unit Subsequently, the sine wave component generation unit 26
A new sine wave component (F "0, A" 0) in the frame based on the deformed spectral shape Snew (f) without or with waveform processing output from the shape processing unit 24, (F "1, A" 1),
(F "2, A" 2), ..., (F "(N-1), A" (N-1))
N sinusoidal wave components (hereinafter collectively referred to as F ″ n,
A "n. N = 0 to (N-1).
More specifically, if the amplifier of the modified spectral shape Snew (f) in the frequency component X is represented as A (X), each of the sine wave components (F "0, A" 0), (F "1, A "
1), (F "2, A" 2), ..., (F "(N-1), A"
(N-1)) can be expressed as follows. (F "0, A" 0) = (ff0, A (ff0)) (F "1, A" 1) = (ff1, A (ff1)) (F "2, A" 2) = (ff2, A (Ff2)) (F "(N-1), A" (N-1)) = (ffn, A (ffn))

[2.13] Operation of Sine Wave Component Deformation Unit Further, the obtained new frequency f ″ n and new amplifier a ″
n is further modified by the sine wave component transformation unit 27 based on the sine wave component transformation information input from the controller 29 as needed. For example, a new amplifier A "n (= A" 0, A "2, A" 4,...) Of even harmonic components.
..) Are increased (for example, doubled). As a result, it is possible to give the converted speech further variety.

[2.14] Operation of Inverse Fast Fourier Transform Unit Next, the inverse fast Fourier transform unit 28 calculates the new frequency F ″ n and new amplifier A ″ n (= new sine wave component) and new residual component. Rnew (f) is stored in the FFT buffer,
Inverse FFT is sequentially performed, and the obtained time axis signals are overlapped so as to partially overlap, and an addition processing of adding them is performed to generate a converted voice signal which is a new voiced sound time axis signal. . At this time, based on the sine wave component / residual component balance control signal input from the controller 29, the mixing ratio of the sine wave component and the residual component is controlled to obtain a more realistic voiced signal. In this case, generally, a rough voice is obtained by increasing the mixing ratio of the residual components.

In this case, when storing the new frequency f "n, the new amplifier a" n (= new sine wave component) and the new residual component Rnew (f) in the FFT buffer, a different pitch and an appropriate pitch are used. By adding the sine wave component converted by the above, harmony can be obtained as a converted audio signal. Further, by giving a harmony pitch adapted to the accompaniment sound by the sequencer 31, musical harmony adapted to the accompaniment can be obtained.

[2.15] Operation of Crossfader Next, the crossfader 30 transmits the original unvoiced / voiced detection signal U /
If the input audio signal Sv is unvoiced (U) based on Vme (t), the input audio signal Sv is output to the mixer 30 as it is. If the input audio signal Sv is voiced (V), the converted audio signal output from the inverse FFT converter 28 is output to the mixer 30. In this case, the reason why the cross fader 30 is used as the changeover switch is to prevent a click sound from occurring at the time of switch changeover by performing a crossfade operation.

[2.16] Operation of Sequencer and Sound Source Unit On the other hand, the sequencer 31 transmits sound source control information for generating a karaoke accompaniment sound to, for example, MIDI (Musical Instrument).
rument Digital Interface) sound source section 3 as data etc.
Output to 2. Thereby, the sound source section 32 generates an accompaniment signal based on the sound source control information, and outputs the accompaniment signal to the mixer 33. [2.17] Operation of Bandpass Filter Characteristic Control Unit, White Noise Generating Unit, and Bandpass Filter The bandpass filter characteristic control unit 42 performs the following processing based on the third formant FT3 output from the modified spectral shape processing unit 24. Bandpass filter (BPF) 4
The characteristic control signal SBC is output to the band-pass filter 41 so that the pass band of No. 1 is a frequency band near the third formant FT3.

On the other hand, the white noise generator 40 generates a white noise signal SWN and
Output to Under the control of the band-pass filter characteristic control unit 42, the band-pass filter 41 converts only the white noise signal SWN having a frequency of a predetermined frequency band corresponding to the third formant FT3 out of the white noise signal SWN into the original breathing noise. The signal is passed as a signal SBWN and output to the amplifier section 44.

[2.18] Operation of Level Control Unit and Amplifier Unit On the other hand, based on the amplifier AFT3 output from the modified spectral shape processing unit 24, the level control unit 43
A signal level control signal SLC for controlling the signal level of the original breathing noise signal SBWN is output to the amplifier unit 44.
The amplifier 44 changes the signal level of the original breathing noise signal SBWN based on the signal level control signal SLC, and outputs the signal to the mixer 33 as the breathing noise signal SABWN. [2.19] Mixer and Output Unit The mixer 33 mixes either the input audio signal Sv or the converted audio signal, the breathless noise signal SABWN and the accompaniment signal, and outputs the mixed signal to the output unit 34. Output unit 3
Reference numeral 4 has an amplifier (not shown) to amplify the mixed signal and output it as an acoustic signal.

[3] Modified Example of Embodiment [3.1] First Modified Example In the first embodiment (particularly, FIG. 15), the audio signal processing apparatus for converting a male voice into a female voice has been described. However, the present invention is also applicable to a female voice synthesizer capable of synthesizing a female voice. In this case,
In the case of the first embodiment, the frequency of the third formant is detected. However, the present invention is not limited to the configuration in which the frequency of the third formant is detected after the combination, and the frequency of the third formant is set in advance, It is also possible to configure to store.

[3.2] Second Modification In the description of the above embodiment, the modified spectral shape generated based on the spectral shape of the former singer and the sine wave component of the target voice signal of the target singer are included. The sine wave component group is calculated based on the frequency component to be converted, and the converted voice is obtained.However, the modified spectral shape generated based on the spectral shape of the target singer and the sine wave of the input voice signal of the former singer. A configuration is also possible in which a sine wave component group is calculated based on the frequency components included in the components to obtain a converted voice.

[3.3] Third Modification The extraction of the sine wave component is not limited to the method used in this embodiment. In short, it is only necessary to extract a sine wave component included in the audio signal.

[3.4] Fourth Modification In the present embodiment, the sine wave component and the residual component of the target are stored. Instead, the target voice itself is stored and read out to be read in real time. The sine wave component and the residual component may be extracted by the processing. That is, processing similar to the processing performed on the voice of the singer trying to imitate in the present embodiment may be performed on the voice of the target singer.

[4] Effect of Embodiment As a result, the song of the male singer who is the former singer is converted and output together with the accompaniment of karaoke.
The resulting converted voice is a female voice that is natural in hearing.

[0071]

As described above, according to the present invention, when performing male to female conversion, it is possible to easily obtain a female voice that is natural in terms of audibility.

[Brief description of the drawings]

FIG. 1 is a block diagram (part 1) illustrating a configuration of an embodiment of the present invention.

FIG. 2 is a block diagram (part 2) showing a configuration of an embodiment of the present invention.

FIG. 3 is a diagram illustrating a state of a frame according to the embodiment.

FIG. 4 is an explanatory diagram for describing peak detection of a frequency spectrum in the embodiment.

FIG. 5 is a diagram illustrating cooperation of peak values for each frame in the embodiment.

FIG. 6 is a diagram illustrating a change state of a frequency value in the embodiment.

FIG. 7 is a diagram showing a change state of a deterministic component in a process in the embodiment.

FIG. 8 is an explanatory diagram of signal processing in the embodiment.

FIG. 9 is a timing chart of an easy synchronization process.

FIG. 10 is a flowchart of an easy synchronization process.

FIG. 11 is a diagram illustrating a female spectral shape.

FIG. 12 is a diagram illustrating a male spectral shape.

FIG. 13 is an explanatory diagram of a male-to-female conversion process.

FIG. 14 is a diagram for explaining spectral tilt compensation of a spectral shape.

FIG. 15 is a diagram illustrating the principle of the embodiment;

FIG. 16 is an explanatory diagram of a method of detecting a third formant.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Microphone, 2 ... Analysis window generation part, 3 ... Input audio signal extraction part, 4 ... Fast Fourier transform part, 5 ... Peak detection part, 6 ...
Unvoiced / voiced detection unit, 7: pitch extraction unit, 8: peak linking unit, 9: interpolation synthesis unit, 10: residual component detection unit, 11: fast Fourier transform unit, 12: residual component holding unit, 13: sine Wave component holding unit, 14: average amplifier calculation unit, 15: amplifier normalization unit, 16: spectral shape calculation unit, 17
... Pitch normalizing section, 18 ... Original frame information holding section, 19
... Static change / vibrato change separation section, 20 ... Target frame information holding section, 21 ... Key control / tempo change section, 22 ... Easy synchronization processing section, 23 ... Deformed spectral shape generation section, 2
4 ... deformed spectral shape processing section, 25 ... residual component selection section, 26 ... sine wave component generation section, 27 ... sine wave component deformation section, 28 ... inverse fast Fourier transform section, 29 ... controller, 30 ... cross fader section , 31 ... sequencer, 3
2 ... sound source section, 33 ... mixer, 34 ... output section, 40 ... white noise generation section, 41 ... band pass filter (BP)
F), 42: band-pass filter characteristic control unit, 43: level control unit, 44: amplifier unit, AFT3, third formant amplifier, FT3: third formant (frequency)

Claims (8)

[Claims] 1. An audio signal processing apparatus for converting an input voice of a male voice into a converted voice of a female voice and outputting the converted voice, wherein a breathing noise component signal is added to an original converted voice signal obtained by converting the input voice. An audio signal processing device comprising a breathing noise adding means for outputting as a converted audio signal. 2. The audio signal processing device according to claim 1, wherein the breathiness noise adding unit detects a formant frequency of an original converted audio signal obtained by converting the input voice into a female voice, Breathiness noise generation means for generating the breathiness noise component signal having a frequency band corresponding to the detected formant frequency; and superimposing the breathiness noise component signal on the original converted speech signal and outputting the converted speech signal as the converted speech signal. An audio signal processing device comprising: 3. The audio signal processing device according to claim 2, wherein the breathiness noise generation unit generates and outputs a white noise signal, and a white noise generation unit based on a detection result of the formant frequency detection unit. Band-pass filter means for passing only a predetermined frequency band component corresponding to a third formant of the original converted audio signal out of the white noise signal and outputting it as an original breathable noise component signal; Signal level control means for controlling the signal level of the original breathing noise component signal and outputting the signal as the breathing noise component signal. 4. An audio signal processing apparatus for synthesizing and outputting a female voice, wherein a breathiness noise signal is added to an original synthesized audio signal obtained by the synthesis and output as the converted audio signal. An audio signal processing device comprising noise adding means. 5. The audio signal processing device according to claim 4, wherein the breathiness noise adding means generates the breathiness noise component signal having a predetermined frequency band corresponding to a formant frequency of the original synthesized speech signal. An audio signal processing apparatus, comprising: breathiness noise generation means; and superposition means for superimposing the breathiness noise component signal on the original synthesized speech signal and outputting the converted speech signal as the converted speech signal. 6. The audio signal processing device according to claim 5, wherein the breathiness noise generation unit generates and outputs a white noise signal, and the original converted audio signal among the white noise signals. Band-pass filter means for passing only a predetermined frequency band component corresponding to the third formant and outputting the signal as a source breathing noise component signal; and a signal level of the source breathing noise component signal based on the source conversion voice signal. And a signal level control means for controlling the signal to output as the breathing noise component signal. 7. A voice signal processing method for converting an input voice of a male voice into a converted voice of a female voice, wherein a breathiness noise component is added to an original converted voice obtained by converting the input voice, and the breath becomes the converted voice. An audio signal processing method comprising a sexual noise adding step. 8. A voice signal processing method for synthesizing a female voice, comprising a breathiness noise adding step of adding a breathiness noise component to an original synthesized voice obtained by the synthesis to obtain the converted voice. An audio signal processing method characterized by the following.